A Longer Example

Audience

This example is written for build and devops engineers who are responsible for the builds from a number of projects. These builds produce files that will here be called “artifacts”. The goal in this example will be to declare dependencies between different builds and have the depenencies for each project’s build downloaded automatically as part of the build. Under normal circumstances, a package manager would be used to do this; however, sometimes this is not possible. This example in particular illustrates how degasolv can be used to resolve dependencies between zip files, which do not carry dependency information.

The dependencies

In this example, suppose that you keep the artifacts for your builds, all zip files, stored on an auto-indexed HTTP server called reposerver, which serves the files at the URL http://example.com/repo/. These builds depend on the presence of artifacts from other builds to complete successfully. The dependency tree looks like this:

digraph G {
  a -> b;
  b -> c;
  b -> d;
  d -> e;
  c -> e;
}

For example, in order to build the artifact for a, there must first be artifacts generated by the b, c, d, and e builds present in the build directory.

The complication here is that each project above has generated artifacts at different versions. To be short in writing, we will denote the artifact generated by the build for a at version 1.0.0 as a@1.0.0.

In our example, there was a recent breaking change to a. Where artifacts a@1.9.0 worked fine with all previous versions of artifacts for b, the newer a@2.1.0 only works with b@2.3.0 or greater. Since the 2.0.0 line of b came out, it relies on the newer c@3.5.0, and the ancient-but-still-used d, the only version of which was published as d@0.5.0. The last time d was touched, the newest version of e was 1.1.0; however, the newer c@3.5.0 relies on the fact that the artifact for e must be at least at version 1.8.0 or newer. There are three published artifacts at different versions for e: e@1.8.0, e@2.1.0, and e@2.4.0. Only the e@1.8.0 version of e is backwards compatible with e@1.1.0 and so it is the only version which will satisfy all of the build-time dependencies for a.

Adding e to the degasolv repo

The first step is to build e, since it is at the bottom of our dependency tree. In our example, when we build e, we mean that we are generating the file e-<version>.zip using the source code for e. Let’s say that we have as part of this build already created e, at the version of 1.8.0. We might have a file called degasolv.edn somewhere in our source code repository for e. We can use this file to specify options to degasolv, including repositories, requirements, etc. of the build. The file will be simple for e, though, since e has no other dependencies. It might look like this:

; filename: degasolv.edn
{
    :id "e"
    :version "1.8.0"
}

During the build of e, we push the build artifact e-1.8.0.zip to the reposerver so that it can be downloaded at https://example.com/repo/e-1.8.0.zip. Then, we generate a dscard file for e. This file will represent e in a degasolv repository. It is done like this:

$ java -jar degasolv-<version>-standalone.jar \
    generate-card \
    --location "https://example.com/repo/e-1.8.0.zip" \
    --output-file "e-1.8.0.zip.dscard"

Note that it is good practice to name the output file after the name of the file that the card will be representing in the degasolv respository.

This will create a file called e-1.8.0.zip.dscard. We would then copy this file up to the reposerver:

$ rsync e-1.8.0.zip.dscard user@reposerver:/var/www/repo/

Once the card is added to the repo on the repo server, a command is run on the server to generate (or update) a degasolv repository index:

$ ssh user@reposerver
$ cd /var/www/repo
$ java -jar ./degasolv-<version>-standalone.jar \
    generate-repo-index \
    --search-directory /var/www/repo \
    --output-file /var/www/repo/index.dsrepo

This command takes all of the package information from all of the degasolv card files found under /var/www/repo and adds this information to the repository index /var/www/repo/index.dsrepo. Once this is done, the package e is listed as available in the degasolv respository index. We can check that listing e@1.8.0 as available in the index was successful by querying the index from any machine that can see the index.dsrepo file on the reposerver, like this:

$ java -jar ./degasolv-<version>-standalone.jar \
    query-repo \
    --repository "https://example.com/repo/index.dsrepo" \
    --query "e"

Supposing that multiple versions of e is in the repository, its output will look like this:

e==1.8.0 @ https://example.com/repo/e-1.8.0.zip
e==2.1.0 @ https://example.com/repo/e-2.1.0.zip
e==2.4.0 @ https://example.com/repo/e-2.4.0.zip

We can see that the version of e we were building, namely 1.8.0, is now in the repository index. We now know that the repository index has been properly updated.

Adding d to the degasolv repo

In our example, d is ancient, and not built anymore in our environment; however, it is still used in other builds. We will not use a degasolv.edn file for it, because there is nowhere to commit such a file to source. We will simply generate a dscard file for it using command line options:

$ java -jar degasolv-<version>-standalone.jar \
    generate-card \
    --id "d" \
    --version "0.5.0" \
    --location "https://example.com/repo/d-0.5.0.zip" \
    --requirement "e>=1.00,<2.0.0" \
    --output-file "d-0.8.0.zip.dscard"

Note that we can either use command-line options or config file keys to specify the information that degasolv needs.

We then copy the newly created d-0.5.0.zip.edn file up to the server and use it to update the repository index in the same way as for e above.

Adding c to the degasolv repo

The c artifact (zip file) represents a project that is being actively built and developed, so we will create a degasolv.edn file and commit it to the source repository for c. The build for c relies on the e artifact being present, so we will resolve that dependency before we start the build for c. Then, when we build the c project, we will create its corresponding degasolv card file as part of the build, like we did with e.

First, we commit its degasolv.edn file to source code. It might look like this:

; filename: degasolv.edn
{
    :id "c"
    :version "3.5.0"
    :requirements ["e>=1.8.0"]
    :repositories ["https://example.com/repo/index.dsrepo"]
}

As mentioned earlier, c needs the e artifact in order to build. We will use degasolv as part of c build script to download the most recent version fitting the requirement for e like this:

$ java -jar degasolv-<version>-standalone.jar \
    resolve-locations

This command is run from the same directory where degasolv.edn resides. It will return output looking something like this:

e==1.8.0 @ https://example.com/repo/e-1.8.0.zip

We can use this output in a script to download and unzip the zip file so that it can be used as part of the build for c like this:

#!/bin/sh

java -jar degasolv-<version>-standalone.jar -c ./degasolv.edn \
    resolve-locations | while read pkg
do
  spec=$(echo "${pkg}" | awk -F ' @ ' '{print $1}')
  name=$(echo "${spec}" | awk -F '==' '{print $1}')
  version=$(echo "${spec}" | awk -F '==' '{print $2}')
  url=$(echo "${pkg}" | awk -F ' @ ' '{print $2}')
  curl -o ${name}-${version}.zip -L ${url}
  unzip ${name}.zip
done

This stanza can be used in a build script to download all of the dependencies for c and unzip them in the current directory.

At the end of the build for c, we can create the degasolv card file for c like this:

$ java -jar degasolv-<version>-standalone.jar \
    generate-card \
    --location "https://example.com/repo/c-3.5.0.zip" \
    --output-file "c-3.5.0.zip.dscard"

Then we upload this file to our http server and use it to update the index.dsrepo degasolv repository index file in the same way as what we did during the build for e.

Let us now suppose that we have repeated these steps for the build artifacts of b. Then all of the projects except for a which are mentioned at the beginning of this example will have had artifacts built from their builds and entries created in the degasolv respository index for their artifacts.

Building a

Now suppose that we are building a. In our example, the build artifact for a need not be uploaded to the zip file repository, because a represents our final product, and the build for a will generate an artifact that will be handed off to Project Management or Ops for later release. We don’t need it for any other builds. While we are not (in this trivial example) not interested in uploading it to the repo, we are interested in resolving its dependencies, downloading them, and using them to build the final product.

Just like some of our previously described builds in this example, we will put a file called degasolv.edn in the root of the git repository associated with building a. It might look like this:

; filename: degasolv.edn
{
    :id "a"
    :version "2.1.0"
    :file-name "a-2.1.0.zip"
    :requirements ["b>2.0"]
    :repositories ["https://example.com/repo/index.dsrepo"]
}

Then, as in the script used to build the artifact for a, we resolve its dependencies and download them, just as we did when we built e:

#!/bin/sh

java -jar degasolv-<version>-standalone.jar -c ./degasolv.edn \
    resolve-locations | while read pkg
do
  spec=$(echo "${pkg}" | awk -F ' @ ' '{print $1}')
  name=$(echo "${spec}" | awk -F '==' '{print $1}')
  version=$(echo "${spec}" | awk -F '==' '{print $2}')
  url=$(echo "${pkg}" | awk -F ' @ ' '{print $2}')
  curl -o ${name}-${version}.zip -L ${url}
  unzip ${name}.zip
done

This will resolve all of the dependencies for a, download them, and unzip them. The rest of the build process for a can then continue as normal.