Artifact Dashboard: Make it pretty

In this blog post I'll cover how to use webjars in conjunction with sbt. The plan is to download bootstrap and maybe other web techniques via maven artefacts. 

This blog post is a successor to another blog post I made only recently, I had the feel it should have better looks, so here we go.

Motivation

I'm well aware that there are package managers for javascript, at least I've heard of them - not overly good things - but I heard of them. Chances are high that if you continue to read to the end of this post there has been a new one invented.

In my bubble, in the past - well, many years - I had to fiddle around with JVM de factor standard build systems like maven or gradle, and if i neglect countless hours of fustration with them I have to admit they served me well. As such, I'll stay with this ecosystem, at least I use it in creative ways. Well, sort of. I'll use sbt to download dependencies and build the example project, I hope I got your head spinning in the meantime.

Reality chck: One thing I would advise to avoid is to have too many package managers in your corporate setting, it is a maintenance burden which gets multiplied by each new system. Naturally, every build system wants to outmatch the other ones, either in complexity or in the wealth of features it provides. (One advice: Sometimes it pays off to bet on the most conservative option, but I doubt you'll find such a thing in javascript land. 😉)

In some bright moments however, different worlds work together, and this blog post wants to shed a light on how to combine web techniques with tools used for corporate settings.

To put it bluntly, I try to download a jar file containing prepacked static content containing a bunch of javascript and other resources and put it at the right place - on demand.

For example, I want to use bootstrap in my project - just go to their website, download it, extract things you are interested in, move on? 

Of course this is a valid - and most probably - the most used approach by a majority of developer teams, but it has some drawbacks. 

First, it is unclear where such frameworks come from. It is also hard to tell 'afterwards' which file belongs to which framework. You are left to best guessing based on directory or zip file names. A concern might be the licensing situation - how to tell if a bunch of code maybe inflicts legal issues in your code? What about if you want to update a certain library - often you can be glad if you even know which version you currently use. 

Of course you can document it, but odds are high that this documentation is not really in sync with your code, at least you have to check manually that this is the case. Another anti pattern which comes to mind is that third party code will be checked in your code repository, no matter how hard you fight it, by some innocent but mindless colleague. The best thing about is, that you'll surely find the same library downloaded and checked in multiple times, maybe also in different versions, open doors for subtle bugs, o my god, in PRODUCTION. Insane! Mind blowing! (Yes I play bass and yes I slap and yes I know davie504)

There has to be a reason why such a zoo of package managers exist, for each programming language several of them, along with their ecosystems, everybody fighting for the attention of the very confused poor Mr. Bob Developer.

As such, for all of the good reasons cited above, there is an excuse to automate it yet again and write a blog post about it, mainly for you who reads this with intense boredom waiting for the anticlimax which is yet to come.

Implementation

I'll continue to work on the artifact-dashboard project, and enhance version 0.4 by adding necessary commands to the sbt build file to download bootstrap.

Before reading on, maybe you want to inform yourself about webjars.org, whose mission statement is 

WebJars are client-side web libraries (e.g. jQuery & Bootstrap) packaged into JAR (Java Archive) files.

This is exactly what we need, what I tried to convey above with my ranting about problems if you don't use any sort of package manager. Webjars.org is a really cool project for all JVM engineers who don't want to get their hands too dirty with front end development, stay cozy in their own world, not to touch those strange javascript package managers alltogether.

Besides, source code management is trivial, right?

Don't worry, to download bootstrap via sbt is far less text than the rant above, but maybe more interesting.

To download and unpack it, not much is necessary, witness it here:

enablePlugins(ScalaJSPlugin)

name := "artifact-dashboard"
scalaVersion := "2.12.10"

// This is an application with a main method
scalaJSUseMainModuleInitializer := true

libraryDependencies ++=
  Seq("org.scala-js" %%% "scalajs-dom" % "1.0.0"
    , "org.webjars" % "bootstrap" % "4.4.1")

// defines sbt name filters for unpacking
val bootstrapMinJs: NameFilter = "**/bootstrap.min.js"
val bootstrapMinCss: NameFilter = "**/bootstrap.min.css"
val bootstrapFilters: NameFilter = bootstrapMinCss | bootstrapMinJs

// magic to invoke unpacking stuff in the compile phase
resourceGenerators in Compile += Def.task {
  val jar = (update in Compile).value
    .select(configurationFilter("compile"))
    .filter(_.name.contains("bootstrap"))
    .head
  val to = (target in Compile).value
  unpackjar(jar, to, bootstrapFilters)
  Seq.empty[File]
}.taskValue

// a helper function which unzips files defined in given namefilter
// to a given directory, along with some reporting
def unpackjar(jar: File, to: File, filter: NameFilter): File = {
  val files: Set[File] = IO.unzip(jar, to, filter)
  // print it out so we can see some progress on the sbt console
  println(s"Processing $jar and unzipping to $to")
  files foreach println
  jar
}

This is part of v0.5 of the artifact-dashboard project, and this is the complete sbt file which is necessary to download a version of bootstrap - at the time of writing the most current one - unpacking it to the right place and make it available for the javascript code.

As usual, you only have to provide the correct paths in your html file to the javascript code, and you can profit from all the goodies bootstrap has to offer.

As a side note, I tweaked also a little bit how Scala.js is used, have a look in the source code / html file.

I think it is a nice combination of technologies, don't know if it is popular, at least it works for me.

Using bootstrap, our download page looks far better, users will surely be more willing to click on a download link than ever before.

Here is a little screenshot of the current state of affairs with v0.5 of artifact dashboard in action for my test instance:

Screenshot of Artifact Dashboard v0.5

Thanks for reading.

Artifact Dashboard: Custom Nexus Summary Page

Motivation

In this post I want to discuss a small project called artifact-dashboard which shows

how to use Nexus Repository Manager, Version 3, and it's Rest API. 

(This article is a part of a mini series of blog posts, see the second part here.)

I use my preferred technologies (and by doing that I almost successfully circumvent using Javascript entirely), namely Scala and Scala.js, maven, sbt powered by severe googling.

Introduced with version 3, Nexus Repo Manager's REST API provides means to query about the state of artifacts which are deployed on the server. (Of course, many other things are possible doing with this API - like for example query licensing status or health status etc.)

My plan is to create a self contained website with some 'dynamic' download links by using information retrieved from Nexus REST API, namely latest snapshots or 'interesting' artifacts like product packages, installers etc. 

In a typical small company setting, this alleviates engineering from daily questions like: 'where is the latest version of our software?...' since everything up to the point of downloading artifacts is automated. (I'm talking about SNAPSHOT / LATEST versions which potentially change permanently and don't have a stable url to download them from Nexus.

(Thinking of it, maybe there is a custom workflow directly baked in to Nexus Repo Manager, I didn't check recent docs, maybe in the process of writing this I'll learn more about it. (UPDATE: I think there is an alternate (or maybe first hand) way to do what I describe by using well configured 'Content Selector's - a concept I didn't yet know starting this post); UPDATE 2: Obviously I'm not the only one who would like such a thing, it was resolved already, but anyway I'll continue with my effort to learn more about it. UPDATE 3: My whole approach reduces itself mainly to a single rest call - which is nice)

Anyway, it will be a way to learn both about Scala.js and creating minimalistic UI's with it. My goal is to create a standalone html page which uses Javascript to automatically create correct download urls, just for the fun of it. So let's dive in.

Prerequisites

If you have not already done so, I would recommend installing IntelliJ Community Edition along with its excellent Scala plugin and checkout the github project which accompanies this post.

LPT: Checkout code and skip the whole blog post!

We need a rest service which we want to query, the whole purpose of this post is to be able to talk to the Nexus Rest API. It is available starting with version 3 of this software package (in fact I learned that Nexus provided an API already in the 2.x series, but here I'll cover v3), which can be downloaded on Sonatype's website for OSS Nexus Repository.

You'll have to register an account, then you can download a version for your platform. You have to do a basic setup, and create an administrative account, we'll see what we need along the way.

You'll need also a small maven project to deploy to your test repository, I would recommend something small and simple without many dependencies or huge build times, or you deploy your artifacts directly to Nexus via the web gui. 

Step 1 - Basic Scala.js project setup

I start from scratch by creating more or less a Scala.js 'Hello world' minimal example.

For convenience, I made some releases in artifact-dashboard which covers different milestones during development, and can be downloaded as zip files.

Release v0.1 contains just a raw setup with more or less current versions of Scala.js and friends at the time of writing this post.

Nothing fancy here, just a minimal sbt build definition. On the sbt console you have to issue a 'fastOptJS' command such that Scala.js compiles (later then do a 'fullOptJS' to obtain the best optimized Javascript available ;-) ).

The most interesting part are the few lines of Scala code shown below:

package net.ladstatt.adash

import org.scalajs.dom.document
import org.scalajs.dom.raw.Element

object ArtifactDashboard {

  def main(args: Array[String]): Unit = {
    document.body.appendChild(p("Paragraph filled via Scala.js"))
  }

  def p(text:String): Element = {
    val p = document.createElement("p")
    p.textContent = text
    p
  }

}

By importing 'org.scalajs.dom' you'll get access to the page dom ... and there you go. We don't need much more than that (ignoring all the magic behind the scenes) to create more or less what we want on the html page. Attention: Don't forget to import a library in your build.sbt (scalajs-dom) otherwise it won't work.

Side note: It would be more coherent if we would only use Scala + Maven (there is a plugin available for this) - sadly enough it doesn't support Scala.js compilation. However, there exists another maven plugin - I don't use for no particular reason - which should support Scala.js compilation. Furthermore I learned that there is a possibility to execute Scala.js cross compiler as a command line tool which would also be a possible approach, but not today. We'll stick to sbt.

Step 2 - setup Nexus for a simple rest call

Ok, up to here you should have a running Nexus 3 Instance somewhere. If you don't have access to one already, make sure to download a free OSS version from Sonatype's website. To make things easy for me and this blog post, I assume from now on that a running Nexus on localhost is present.

When you download a fresh one, make sure that you execute Maven once - configured to use this local installation. This is necessary as to fill your Nexus instance with some data we can query later.

The easiest way for that is to change your settings.xml file to point to your local Nexus, for me it worked to configure the 'mirror's section like shown below:

<?xml version="1.0" encoding="UTF-8"?>
<settings
        xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd"
        xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <mirrors>
        <mirror>
            <id>local-nexus</id>
            <mirrorOf>*</mirrorOf>
            <url>http://127.0.0.1:8081/repository/maven-public/</url>
        </mirror>
    </mirrors>

</settings>

Now we should have some data in our Nexus, to my current understanding Nexus should now return some useful result on its API. Luckily enough Nexus provides itself a nice testbed for API calls, directly baked in in its user interface. The UI for the Rest API can be easily found in the administration console (default address: http://127.0.0.1:8081/#admin/system/api)

Let's try the easiest api call, or is it, at least the first one, '/v1/assets'.

You have to provide which repository to query, just enter 'maven-public' and click on 'execute'.

Do it! Do it now. I'll wait.

Step 3 - implement rest call with Scala.js

At this point you should now gaze with amazement on the Nexus Administrative Console, and see a response for the rest api call. The guys from sonatype even included a directly executable command line invocation for curl, which looks like this in my case:

curl -X GET "http://127.0.0.1:8081/service/rest/v1/assets?repository=maven-public" -H "accept: application/json"

I can copy paste this into a terminal, and it returns a json list of assets which proves that at least something is going on, and thus we can continue with implementing a corresponding call in Scala.js.

... and then I rediscovered a thing called 'CORS' ;-) 

TLDR, the approach of calling a REST api from an html page doesn't work if it is not from the same origin like the api, that means it has to be served from the same server. See this article for more information.

So curl is working perfectly, but having a website calling this rest api is not possible, which is a good thing security wise. There are ways to work around for this problem, the easiest one is to place the final html and javascript in the public folder of the Nexus installation itself, which may not be viable for a variety of reasons.

For educational purposes I pursue this approach however. Keep in mind that the whole point of this blog post is to learn about how to work with Scala.js and communicate with webservices, not necessarily with Nexus, maybe create your own little rest service and play with it.

Version v0.2 of the project is tagged at a state where a minimal example for a webservice setup is given, for completeness and also to show how easy it is I'll include the source code here as well:

package net.ladstatt.adash

import org.scalajs.dom.document
import org.scalajs.dom.ext.Ajax
import org.scalajs.dom.raw.Element

import scala.concurrent.ExecutionContextExecutor
import scala.scalajs.js
import scala.scalajs.js.JSON
import scala.util.{Failure, Success}

@js.native
trait CheckSum extends js.Object {
  def md5: String

  def sha1: String
}

@js.native
trait Asset extends js.Object {
  def id: String

  def path: String

  def downloadUrl: String

  def repository: String

  def format: String

  def checksum: CheckSum
}

@js.native
trait AssetsResult[T] extends js.Object {
  def items: js.Array[T]

  def continuationToken: String
}


object ArtifactDashboard {

  def main(args: Array[String]): Unit = {
    queryNexus()
  }

  def queryNexus(): Unit = {
    // executioncontext needed for ajax call
    implicit val ec: ExecutionContextExecutor = scala.concurrent.ExecutionContext.global

    Ajax.get("http://127.0.0.1:8081/service/rest/v1/assets?repository=maven-public") onComplete {
      case Success(v) =>
        val assetsResult = JSON.parse(v.responseText).asInstanceOf[AssetsResult[Asset]]
        for (a <- assetsResult.items) {
          document.body.appendChild(p(a.id))
        }
        org.scalajs.dom.window.alert(assetsResult.continuationToken)
      case Failure(e) =>
        e.printStackTrace()
        org.scalajs.dom.window.alert(e.getMessage)
    }
  }

  def p(text: String): Element = {
    val p = document.createElement("p")
    p.textContent = text
    p
  }

}

I didn't yet mention two very cool things up there right before your eyes in the sourcecode. Firstly, it demonstrates how easy it is to parse a json response in Scala.js, and with a little more effort than just calling JSON.parse you get full fledged case classes back from your call, quite effortless, by using those magical @js.native annotations (and a little hardcore casting with asInstanceOf ;-))

If you look closely you'll see that the example code uses those case classes innocently, as if this wouldn't be a big deal - but there you have it: a rest call, a json response, conversion to a case class.

Don't bother me with details! Nice.

Step 4 - Setup a test repository and deploy test data to it

For convenience, I've created a dedicated Nexus Maven2 hosted repository for my experiments. I've used my before mentioned toy project to deploy it to my test repository and checked on Nexus UI and API that a deployment was successful.

In order to make this happen you have to declare a distribution management section in your pom:

    <distributionManagement>
        <repository>
            <id>testrepo</id>
            <name>testrepo</name>
            <url>http://127.0.0.1:8081/repository/testrepo/</url>
        </repository>
        <snapshotRepository>
            <id>testrepo-snapshot</id>
            <name>testrepo-snapshot</name>
            <url>http://127.0.0.1:8081/repository/testrepo-snapshot/</url>
        </snapshotRepository>
    </distributionManagement>

As you can see above, in my test instance I've created two repositories: one where I can deploy releases, one where I deploy snapshots. The latter one can now also be queried, and we are done configuring Nexus for our purposes.

Deploying to our testrepo-snapshot repository now only needs a 'mvn deploy' command, which should be issued several times to create more than one snapshot to test it on the server (since we want to have the url of the last SNAPSHOT artifact, remember?)

Step 5 - Give Rest call proper parameters

Nexus Rest API provides a search functionality, which is exactly what we need. It can be configured in a way that we need to call it only once per artifact, like shown below:

http://127.0.0.1:8081/service/rest/v1/search/assets?repository=testrepo-snapshot&maven.classifier=jar-with-dependencies&maven.baseVersion=1.0-SNAPSHOT&direction=asc&maven.extension=jar&maven.groupId=net.ladstatt&maven.artifactId=fx-animations&sort=version

Nexus provides a convenient way to experiment with its search api, the url above yields all snapshot versions deployed on Nexus with Version 1.0-SNAPSHOT, groupId: net.ladstatt, artifactId=fx-animations. We just take the first element of the result list and can retrieve the correct url. See version  v0.3 for a source code snapshot containing described webservice call. Note also the usage of scala's Future or Either construct which can be used in Scala.js as well.

Step 6 - make it configurable

Lets recall what we have accomplished so far:

• setup a Scala.js project from scratch
• setup a Nexus Webserver
• fought against CORS
• call a webservice
• parse json
• map 'native' javascript objects to Scala case classes
• build a website's dom 'dynamically'.
• ...

What is left? The page should get a decent look, but also an easy way to configure different assets, maybe groups of assets ... For this to acheive we could again use Nexus and json parsing, or directly include the meta information (which artifacts should be displayed) in our source code itself. For simplicity, take this route first.

We still have to solve issues like be able to query more than one repository on a nexus. We initially wanted to have this functionality for a way to get a link to the most current SNAPSHOT version of a maven artifact, but it is surely nice to include released artifacts as well.

To further elaborate, we want to include artifacts with a given groupId/artifactId/version scheme, with or without an extension or classifier; things that are not possible with version 0.3. 

See v0.4 for a version which has more artifacts configured and a case class structure prepared for easy configuration.

I'll stop here, maybe this project is of use for somebody or motivates to give Scala.js and / or Nexus a try.

I used this website to hightlight some source code snippets.

Thanks for reading.