.. _A Longer Example:

A Longer Example
================

Audience
--------

This example is written for build and devops engineers who are
responsible for the builds from a number of projects. These builds
produce files that will here be called "artifacts". The goal in this
example will be to declare dependencies between different builds and
have the depenencies for each project's build downloaded automatically
as part of the build. Under normal circumstances, a package manager
would be used to do this; however, sometimes this is not
possible. This example in particular illustrates how degasolv can be
used to resolve dependencies between zip files, which do not carry
dependency information.

The dependencies
----------------

In this example, suppose that you keep the artifacts for your builds,
all zip files, stored on an auto-indexed HTTP server called
``reposerver``, which serves the files at the URL
``http://example.com/repo/``. These builds depend on the presence of
artifacts from other builds to complete successfully. The dependency
tree looks like this:

.. graphviz::

  digraph G {
    a -> b;
    b -> c;
    b -> d;
    d -> e;
    c -> e;
  }

For example, in order to build the artifact for ``a``, there must
first be artifacts generated by the ``b``, ``c``, ``d``, and ``e``
builds present in the build directory.

The complication here is that each project above has generated
artifacts at different versions.  To be short in writing, we will
denote the artifact generated by the build for ``a`` at version
``1.0.0`` as ``a@1.0.0``.

In our example, there was a recent breaking change to ``a``. Where
artifacts ``a@1.9.0`` worked fine with all previous versions of
artifacts for ``b``, the newer ``a@2.1.0`` only works with ``b@2.3.0``
or greater. Since the ``2.0.0`` line of ``b`` came out, it relies on
the newer ``c@3.5.0``, and the ancient-but-still-used ``d``, the only
version of which was published as ``d@0.5.0``. The last time ``d`` was
touched, the newest version of ``e`` was ``1.1.0``; however, the newer
``c@3.5.0`` relies on the fact that the artifact for ``e`` must be at
least at version ``1.8.0`` or newer. There are three published
artifacts at different versions for ``e``: ``e@1.8.0``, ``e@2.1.0``,
and ``e@2.4.0``. Only the ``e@1.8.0`` version of ``e`` is backwards
compatible with ``e@1.1.0`` and so it is the only version which will
satisfy all of the build-time dependencies for ``a``.

Adding ``e`` to the degasolv repo
---------------------------------

The first step is to build ``e``, since it is at the bottom of our
dependency tree. In our example, when we build ``e``, we mean that we
are generating the file ``e-<version>.zip`` using the source code for
``e``. Let's say that we have as part of this build already created
``e``, at the version of ``1.8.0``. We might have a file called
``degasolv.edn`` somewhere in our source code repository for ``e``. We
can use this file to specify options to degasolv, including
repositories, requirements, etc. of the build. The file will be simple
for ``e``, though, since ``e`` has no other dependencies. It might
look like this::

  ; filename: degasolv.edn
  {
      :id "e"
      :version "1.8.0"
  }

During the build of ``e``, we push the build artifact ``e-1.8.0.zip``
to the ``reposerver`` so that it can be downloaded at
``https://example.com/repo/e-1.8.0.zip``. Then, we generate a
``dscard`` file for ``e``. This file will represent ``e`` in a
degasolv repository. It is done like this::

  $ java -jar degasolv-<version>-standalone.jar \
      generate-card \
      --location "https://example.com/repo/e-1.8.0.zip" \
      --output-file "e-1.8.0.zip.dscard"

Note that it is good practice to name the output file after the name
of the file that the card will be representing in the degasolv
respository.

This will create a file called ``e-1.8.0.zip.dscard``. We would then
copy this file up to the ``reposerver``::

  $ rsync e-1.8.0.zip.dscard user@reposerver:/var/www/repo/

Once the card is added to the ``repo`` on the repo server, a command
is run on the server to generate (or update) a degasolv repository
index::

  $ ssh user@reposerver
  $ cd /var/www/repo
  $ java -jar ./degasolv-<version>-standalone.jar \
      generate-repo-index \
      --search-directory /var/www/repo \
      --output-file /var/www/repo/index.dsrepo

This command takes all of the package information from all of the
degasolv card files found under ``/var/www/repo`` and adds this
information to the repository index
``/var/www/repo/index.dsrepo``. Once this is done, the package ``e``
is listed as available in the degasolv respository index. We can check
that listing ``e@1.8.0`` as available in the index was successful by
querying the index from any machine that can see the ``index.dsrepo``
file on the reposerver, like this::

  $ java -jar ./degasolv-<version>-standalone.jar \
      query-repo \
      --repository "https://example.com/repo/index.dsrepo" \
      --query "e"

Supposing that multiple versions of e is in the repository, its output
will look like this::

  e==1.8.0 @ https://example.com/repo/e-1.8.0.zip
  e==2.1.0 @ https://example.com/repo/e-2.1.0.zip
  e==2.4.0 @ https://example.com/repo/e-2.4.0.zip

We can see that the version of ``e`` we were building, namely
``1.8.0``, is now in the repository index. We now know that the
repository index has been properly updated.

Adding ``d`` to the degasolv repo
---------------------------------

In our example, ``d`` is ancient, and not built anymore in our
environment; however, it is still used in other builds. We will not
use a ``degasolv.edn`` file for it, because there is nowhere to commit
such a file to source. We will simply generate a ``dscard`` file for
it using command line options::

  $ java -jar degasolv-<version>-standalone.jar \
      generate-card \
      --id "d" \
      --version "0.5.0" \
      --location "https://example.com/repo/d-0.5.0.zip" \
      --requirement "e>=1.00,<2.0.0" \
      --output-file "d-0.8.0.zip.dscard"

Note that we can either use command-line options or config file keys
to specify the information that degasolv needs.

We then copy the newly created ``d-0.5.0.zip.edn`` file up to the
server and use it to update the repository index in the same way as
for ``e`` above.

Adding ``c`` to the degasolv repo
---------------------------------

The ``c`` artifact (zip file) represents a project that is being
actively built and developed, so we will create a ``degasolv.edn``
file and commit it to the source repository for ``c``. The build for
``c`` relies on the ``e`` artifact being present, so we will resolve that
dependency before we start the build for ``c``. Then, when we
build the ``c`` project, we will create its corresponding degasolv
card file as part of the build, like we did with ``e``.

First, we commit its ``degasolv.edn`` file to source code. It might
look like this::

  ; filename: degasolv.edn
  {
      :id "c"
      :version "3.5.0"
      :requirements ["e>=1.8.0"]
      :repositories ["https://example.com/repo/index.dsrepo"]
  }

As mentioned earlier, ``c`` needs the ``e`` artifact in order to
build. We will use ``degasolv`` as part of ``c`` build script to
download the most recent version fitting the requirement for ``e``
like this::

  $ java -jar degasolv-<version>-standalone.jar \
      resolve-locations

This command is run from the same directory where ``degasolv.edn``
resides.  It will return output looking something like this::

  e==1.8.0 @ https://example.com/repo/e-1.8.0.zip

We can use this output in a script to download and unzip the zip file
so that it can be used as part of the build for ``c`` like this::

  #!/bin/sh

  java -jar degasolv-<version>-standalone.jar -c ./degasolv.edn \
      resolve-locations | while read pkg
  do
    spec=$(echo "${pkg}" | awk -F ' @ ' '{print $1}')
    name=$(echo "${spec}" | awk -F '==' '{print $1}')
    version=$(echo "${spec}" | awk -F '==' '{print $2}')
    url=$(echo "${pkg}" | awk -F ' @ ' '{print $2}')
    curl -o ${name}-${version}.zip -L ${url}
    unzip ${name}.zip
  done

This stanza can be used in a build script to download all of the
dependencies for ``c`` and unzip them in the current directory.

At the end of the build for ``c``, we can create the degasolv card
file for ``c`` like this::

  $ java -jar degasolv-<version>-standalone.jar \
      generate-card \
      --location "https://example.com/repo/c-3.5.0.zip" \
      --output-file "c-3.5.0.zip.dscard"

Then we upload this file to our http server and use it to update the
``index.dsrepo`` degasolv repository index file in the same way as
what we did during the build for ``e``.

Let us now suppose that we have repeated these steps for the build
artifacts of ``b``.  Then all of the projects except for ``a`` which
are mentioned at the beginning of this example will have had artifacts
built from their builds and entries created in the degasolv
respository index for their artifacts.

Building ``a``
--------------

Now suppose that we are building ``a``. In our example, the build
artifact for ``a`` need not be uploaded to the zip file repository,
because ``a`` represents our final product, and the build for ``a``
will generate an artifact that will be handed off to Project
Management or Ops for later release. We don't need it for any other
builds. While we are not (in this trivial example) not interested in
uploading it to the repo, we are interested in resolving its
dependencies, downloading them, and using them to build the final
product.

Just like some of our previously described builds in this example, we
will put a file called ``degasolv.edn`` in the root of the git
repository associated with building ``a``. It might look like this::

  ; filename: degasolv.edn
  {
      :id "a"
      :version "2.1.0"
      :file-name "a-2.1.0.zip"
      :requirements ["b>2.0"]
      :repositories ["https://example.com/repo/index.dsrepo"]
  }

Then, as in the script used to build the artifact for ``a``,
we resolve its dependencies and download them, just as we did when we built
``e``::

  #!/bin/sh

  java -jar degasolv-<version>-standalone.jar -c ./degasolv.edn \
      resolve-locations | while read pkg
  do
    spec=$(echo "${pkg}" | awk -F ' @ ' '{print $1}')
    name=$(echo "${spec}" | awk -F '==' '{print $1}')
    version=$(echo "${spec}" | awk -F '==' '{print $2}')
    url=$(echo "${pkg}" | awk -F ' @ ' '{print $2}')
    curl -o ${name}-${version}.zip -L ${url}
    unzip ${name}.zip
  done

This will resolve all of the dependencies for ``a``, download them,
and unzip them. The rest of the build process for ``a`` can then
continue as normal.