Differences between revisions 20 and 21
Revision 20 as of 2021-01-10 00:53:22
Size: 22419
Editor: ?Louis-PhilippeVeronneau
Comment: replace with the New Packaging Tutorial
Revision 21 as of 2021-01-11 17:06:00
Size: 22420
Editor: ?Louis-PhilippeVeronneau
Comment: fix typo
Deletions are marked like this. Additions are marked like this.
Line 346: Line 346:
4. The `--java-lib option, that installs the `.jar` in `/usr/share/java` for us. 4. The `--java-lib` option, that installs the `.jar` in `/usr/share/java` for us.

Clojure Packaging Tutorial

Introduction

Clojure packages must be built using JavaHelper and MavenRepoHelper or MavenDebianHelper, and whenever possible, using Leiningen.

Clojure package names

Typically, we use packages' upstream names as Debian names, adding a -clojure suffix. For example, the cgrand/parsley project (on GitHub) has a source package parsley-clojure and binary package libparsley-clojure (on Salsa).

This does not cover all possible Clojure package names. For more information on our detailed naming policy, see the Clojure Packaging Reference.

Creating the package's git repository

Since the majority of Clojure software uses git for source control, you can preserve the upstream git tree in our package history. (If you're updating a repository that includes all upstream source in a single commit, you can use gbp import-orig, but I prefer to preserve upstream history where possible.) Here's how to set up the repository.

First, clone the upstream source:

$ git clone https://salsa.debian.org/ehashman/gibson.git

Then set up a corresponding repository on Salsa. You can request that someone from the Clojure Team with "Master" permissions or higher completes this step for you.

Then you should be able to access the bare repository https://salsa.debian.org/clojure-team/gibson-clojure.git [not a real repo]

You'll need to set that up as a remote repository:

$ git remote add salsa https://salsa.debian.org/clojure-team/gibson-clojure.git

Now you can set up the three required branches:

  • debian/main: Where the debian packaging lives. You will build off this branch.

  • upstream: Where the upstream source tree lives.

  • pristine-tar: Where the "pristine" tarball files live. Once a package is uploaded to the archive, you must use the same original source tarball to upload future packaging changes for the same upstream version. Hence, you will commit that tarball here so anyone can upload a package update against the original "pristine" tar.

The workflow you will use to create the branches depends on whether or not upstream tags their releases.

When upstream tags their releases

Add a new upstream/VERSION tag that matches the existing upstream tag:

$ git tag                    # See all available upstream tags
gibson-0.1.0
$ git checkout gibson-0.1.0  # Check out our target version tag
$ git tag upstream/0.1.0     # Make a new tag in the right format
$ git checkout -b upstream   # Make a new 'upstream' branch tracking the tag

Now the upstream branch is fully set up! We can push these to salsa:

$ git push salsa upstream
$ git push salsa upstream/0.1.0

Next, you'll download the "pristine" original tar and create the pristine-tar branch. You will need the pristine-tar package installed, in order for us to commit the original tarball we downloaded earlier to git.

Download the original tarball (.tar.gz file) corresponding to the upstream tag we're packaging (gibson-0.1.0 in this case). Put it in the directory above our source directory with the name gibson-clojure_0.1.0.orig.tar.gz.

You can now commit the tarball to git with the pristine-tar command:

$ pristine-tar commit ../gibson-clojure_0.1.0.orig.tar.gz upstream

Now the pristine-tar branch is done! Push it to salsa:

$ git push salsa pristine-tar

You'll create the debian/main branch in the next section.

When upstream does NOT tag their releases

You can file a bug against upstream requesting that they tag their releases. But sometimes, the maintainers do not respond. In this case, you must create your own upstream tags and pristine tarball.

You'll need to add a new upstream/VERSION tag that matches the commit with the version release. If you're not sure what commit this is, you can look at the git log of the project.clj file. There should only be one commit in the git history where the version is set to 0.1.0; elsewhere, Clojure developers usually commit against version SNAPSHOTs (e.g. "0.1.1-SNAPSHOT").

Say you've found the right commit, and its ID is abc123def. You can now make a tag:

$ git checkout abc123def     # Check out our target commit
$ git tag upstream/0.1.0     # Make a new tag in the right format
$ git checkout -b upstream   # Make a new 'upstream' branch tracking the tag

Now the upstream branch is fully set up! Push these to salsa:

$ git push salsa upstream
$ git push salsa upstream/0.1.0

You'll create the pristine-tar and debian/main branches in the next section.

Packaging using leiningen

Now we're ready to generate our Debian packaging! To start, we'll need a branch to work off. Let's create one:

$ git checkout upstream   # We'll start on the 'upstream' branch
$ git checkout -b debian/main  # Create a new 'debian/main' branch based on 'upstream' and check it out

Files that are not Clojure-specific like changelog or copyright won't be described in this tutorial, as the Debian New Maintainer Guide already does a pretty good job of it.

Once you are happy with these file, you should commit them to the debian/main branch:

$ git add debian/                      # Add all the debian/* files
$ git commit -am "Package for Debian"  # Commit them (using a message of your choice)
$ git push salsa debian/main         # Push to salsa

For these examples, we will look at the shell-utils-clojure package.

control

Typically, dependencies will be listed in the project.clj file of the Clojure project you are packaging.

Source: shell-utils-clojure
Section: java
Priority: optional
Maintainer: Debian Clojure Maintainers <pkg-clojure-maintainers@lists.alioth.debian.org>
Uploaders:
 Thomas Goirand <zigo@debian.org>,
Build-Depends:
 debhelper-compat (= 13),
 default-jdk,
 javahelper,
 maven-repo-helper,
 libclojure-java,
 libprismatic-schema-clojure (>= 1.1.12),
 libcommons-exec-java,
 libcommons-io-java,
 libslf4j-java,
 libtrapperkeeper-clojure (>= 3.1.0),
 libkitchensink-clojure (>= 3.1.1),
 libpuppetlabs-i18n-clojure (>= 0.9.0-2),
 libcomplete-clojure <!nocheck>,
 leiningen,
Standards-Version: 4.5.1
Vcs-Git: https://salsa.debian.org/clojure-team/shell-utils-clojure.git
Vcs-Browser: https://salsa.debian.org/clojure-team/shell-utils-clojure
Homepage: https://github.com/puppetlabs/clj-shell-utils
Rules-Requires-Root: no

Package: libshell-utils-clojure
Architecture: all
Depends:
 ${java:Depends},
 ${misc:Depends},
 libclojure-java,
 libprismatic-schema-clojure (>= 1.1.12),
 libcommons-exec-java,
 libcommons-io-java,
 libslf4j-java,
 libtrapperkeeper-clojure (>= 3.1.0),
 libkitchensink-clojure (>= 3.1.1),
 libpuppetlabs-i18n-clojure (>= 0.9.0-2),
Recommends:
 ${java:Recommends},
Description: shell execution common to Puppet clojure projects
 This package contains a library for shell execution common to Puppet clojure
 projects

rules

Here is an annotated template one can use to build using Leiningen. If you use this, don't forget to remove comments!

   1 #!/usr/bin/make -f
   2 
   3 include /usr/share/javahelper/java-vars.mk
   4 include /usr/share/dpkg/pkg-info.mk
   5 
   6 # lein uses ~/.lein by default
   7 export LEIN_HOME=$(CURDIR)/.lein
   8 # lein fetches dependencies online by default
   9 export LEIN_OFFLINE=true
  10 NAME=shell-utils-clojure
  11 
  12 %:
  13   dh $@ --with javahelper --with maven_repo_helper
  14 
  15 # We symlink to have all depencies in the debian directory, since by default
  16 # lein won't look at local packages installed. More on this in the section about
  17 # patching the upstream code.
  18 override_dh_auto_configure:
  19   cd debian && ln -sf /usr/share/maven-repo .
  20 
  21 override_dh_auto_build:
  22   # Most of the time, upstream's pom.xml file is either outdated, breaks things
  23   # by adding uncessary links or simply don't exist. Creating the pom.xml file
  24   # with lein also ensures we don't forget to refresh it when we update a
  25   # package.
  26   lein pom debian/pom.xml
  27   # Tells lein to bundle all the relevant source code in a .jar
  28   lein jar
  29   # By default, lein creates a jar with a version number in it. We symlink it so
  30   # we don't need a version in debian/libfoo-clojure.poms
  31   cd target && ln -sf $(NAME)-$(DEB_VERSION_UPSTREAM).jar $(NAME).jar
  32 
  33 override_dh_auto_test:
  34   # Running upstream tests during build is important! How else will you know if
  35   # things are broken?
  36   lein test
  37 
  38 override_dh_clean:
  39   rm -f debian/maven-repo
  40   rm -Rf target
  41   rm -f debian/pom.xml
  42   dh_clean

patches/0001_Lein_Local.patch

Since by default, lein builds by fetching dependencies online, we need to path the project.clj file using quilt to fix a few things:

1. Dependencies need to be changed to use generic versions. Most of the time, this will be a debian version, but some packages like libclojure-java use major versions instead, like 1.10.x.

2. Some dependencies and plugins we don't need need to be removed. For example, some packages will also build ?ClojureScript, which we don't need.

3. Some 2nd level dependencies loaded by 1st level dependencies need to be added, as they don't have a "debian" version and use another generic version like "1.x".

4. Lein needs to be told to use a local repository instead of using the default online one.

For example, with the shell-utils-clojure package, we go from this unpatched project.clj:

   1 (defproject puppetlabs/clj-shell-utils "1.0.2"
   2   :description "Clojure shell execution utilities"
   3 
   4   :min-lein-version "2.9.0"
   5 
   6   :parent-project {:coords [puppetlabs/clj-parent "3.0.0"]
   7                    :inherit [:managed-dependencies]}
   8 
   9   :pedantic? :abort
  10 
  11   :test-paths ["test/unit"]
  12 
  13   :plugins [[lein-project-version "0.1.0"]
  14             [lein-parent "0.3.6"]]
  15 
  16   :source-paths ["src/clj"]
  17   :java-source-paths ["src/java"]
  18 
  19   :dependencies [[org.clojure/clojure]
  20                  [prismatic/schema]
  21                  [org.apache.commons/commons-exec]
  22                  [commons-io]
  23                  [org.slf4j/log4j-over-slf4j]
  24                  [org.slf4j/slf4j-api]
  25                  [puppetlabs/trapperkeeper]
  26                  [puppetlabs/kitchensink]
  27                  [puppetlabs/i18n]]
  28 
  29   :profiles { :test { :dependencies [[puppetlabs/trapperkeeper nil :classifier "test" :scope "test"]]}}
  30 
  31   :deploy-repositories [["releases" {:url "https://clojars.org/repo"
  32                                      :username :env/clojars_jenkins_username
  33                                      :password :env/clojars_jenkins_password
  34                                      :sign-releases false}]
  35                         ["snapshots" "http://nexus.delivery.puppetlabs.net/content/repositories/snapshots/"]]
  36   )

To this patched one:

   1 (defproject puppetlabs/clj-shell-utils "1.0.2"
   2   :description "Clojure shell execution utilities"
   3 
   4   :min-lein-version "2.9.0"
   5 
   6   :pedantic? :abort
   7 
   8   :test-paths ["test/unit"]
   9 
  10   :source-paths ["src/clj"]
  11   :java-source-paths ["src/java"]
  12 
  13   :dependencies [[org.clojure/clojure "1.10.x"]
  14                  [prismatic/schema "debian"]
  15                  [org.apache.commons/commons-exec "debian"]
  16                  [commons-io "debian"]
  17                  [org.slf4j/log4j-over-slf4j "debian"]
  18                  [org.slf4j/slf4j-api "debian"]
  19                  [puppetlabs/trapperkeeper "debian"]
  20                  [puppetlabs/kitchensink "debian"]
  21                  [puppetlabs/i18n "debian"]
  22 
  23                  [org.yaml/snakeyaml "1.x"]
  24                  [com.fasterxml.jackson.core/jackson-core "2.x"]
  25                  [com.fasterxml.jackson.dataformat/jackson-dataformat-smile "2.x"]
  26                  [com.fasterxml.jackson.dataformat/jackson-dataformat-cbor "2.x"]]
  27 
  28   :profiles { :test { :dependencies [[puppetlabs/trapperkeeper "debian" :classifier "test" :scope "test"]]}}
  29 
  30   :deploy-repositories [["releases" {:url "https://clojars.org/repo"
  31                                      :username :env/clojars_jenkins_username
  32                                      :password :env/clojars_jenkins_password
  33                                      :sign-releases false}]
  34                         ["snapshots" "http://nexus.delivery.puppetlabs.net/content/repositories/snapshots/"]]
  35   :local-repo "debian/maven-repo")

libfoo-clojure.classpath

This file will be used when installing the .jar file in Debian to create a Class-Path entry in the META-INF/MANIFEST.MF file inside the .jar. It is important, as it will be used to load needed dependencies when using the .jar.

This file should contain all the first-level dependencies needed in the binary package:

usr/share/java/clj-shell-utils.jar /usr/share/java/clojure.jar /usr/share/java/schema.jar /usr/share/java/commons-exec.jar /usr/share/java/commons-io.jar /usr/share/java/slf4j-api.jar /usr/share/java/log4j-over-slf4j.jar /usr/share/java/trapperkeeper.jar /usr/share/java/kitchensink.jar /usr/share/java/puppetlabs-i18n.jar

Note that:

1. The order is important. Your package's .jar should come first.

2. Your package's .jar entry should not start with /

libfoo-clojure.poms

This file is used by maven-repo-helper to install .jar and .pom files in /usr/share/maven-repo/. In it, your are specifying:

1. The path where to find the pom.xml file

2. The path where to find the .jar file to install

3. The name to use when installing the .jar file

4. The --java-lib option, that installs the .jar in /usr/share/java for us.

debian/pom.xml --artifact=target/clj-shell-utils.jar --usj-name=clj-shell-utils --java-lib

Packaging WITHOUT leiningen

Some packages do not have a project.clj file, and thus cannot be packaged using leiningen. They instead use a deps.edn file to specify dependencies and upstream uses tools currently not in the archive to build the .jar.

This means when this happens, we need to build the package manually using the jar CLI tool. Fortunately, these packages are often simpler than the ones that need to be built using leiningen.

For these examples, we will look at the data-json-clojure package.

For the other important debian files, consult the section about building using leiningen.

rules

Note two things:

1. It is important to specify the right name for the PRODUCED_JAR variable. You want to stick to the name upstream uses for the artifact. Check on https://mvnrepository.com if necessary.

2. When building the .jar, you need to specify the right path to the Clojure files. If you use the wrong path (say src/main/ instead of src/main/clojure), your final .jar won't be usable.

   1 #!/usr/bin/make -f
   2 
   3 include /usr/share/javahelper/java-vars.mk
   4 
   5 PRODUCED_JAR=data.json.jar
   6 
   7 %:
   8         dh $@ --with javahelper --with maven_repo_helper
   9 
  10 override_jh_build:
  11         jar cf $(PRODUCED_JAR) -C src/main/clojure .
  12 
  13 override_dh_auto_test:
  14         debian/run-build-tests
  15 
  16 override_jh_clean:
  17         jh_clean
  18         rm -f $(CURDIR)/$(PRODUCED_JAR)
  19         rm -rf $(CURDIR)/meta

run-build-tests

Since you can't rely on leiningen to run tests, you'll have to write a shell script to run them manually using Clojure.

This script resembles a lot the one used to run unit tests in the autopkgtest. Note that you'll need to specify the entire classpath needed to run the tests.

   1 #!/bin/sh
   2 set -efu
   3 
   4 CLASSPATH=/usr/share/java/clojure.jar:src/main/clojure:src/test/clojure
   5 
   6 clojure \
   7     -cp $CLASSPATH \
   8     -e "(require '[clojure.test])" \
   9     -e "(require '[clojure.data.json-compat-0-1-test])" \
  10     -e "(require '[clojure.data.json-test])" \
  11     -e "(System/exit (if (clojure.test/successful? (clojure.test/run-tests
  12          'clojure.data.json-compat-0-1-test
  13          'clojure.data.json-test)) 0 1))"

Autopkgtests

Autopkgtests are important! How else will we know if your binary package really works, or if an updated dependency broke your package?

Check that you can load the Clojure library

This is a superficial autopkgtest, but it helps you confirm one can load your Clojure library properly.

   1 #!/bin/sh
   2 
   3 CLASSPATH=/usr/share/java/clojure.jar:/usr/share/java/schema.jar:/usr/share/java/commons-exec.jar:/usr/share/java/commons-io.jar:/usr/share/java/slf4j-api.jar:/usr/share/java/log4j-over-slf4j.jar:/usr/share/java/trapperkeeper.jar:/usr/share/java/kitchensink.jar:/usr/share/java/puppetlabs-i18n.jar:/usr/share/java/clj-shell-utils.jar
   4 
   5 clojure -cp $CLASSPATH -e '(use '"'"'puppetlabs.puppetserver.shell-utils)'

Run the upstream testsuite

This is not always possible, but if you can, you should definitely run the upstream testsuite. Note we cannot use Leiningen directly, since we'd need a project.clj file and it is not available in the binary package.

In this particular example, we're also copying the files in dev-resources since the testsuite requires them, but it is not always the case.

   1 #!/bin/sh
   2 set -efu
   3 
   4 CLASSPATH=/usr/share/java/clojure.jar:/usr/share/java/schema.jar:/usr/share/java/commons-exec.jar:/usr/share/java/commons-io.jar:/usr/share/java/slf4j-api.jar:/usr/share/java/log4j-over-slf4j.jar:/usr/share/java/trapperkeeper.jar:/usr/share/java/trapperkeeper-test.jar:/usr/share/java/kitchensink.jar:/usr/share/java/puppetlabs-i18n.jar:/usr/share/java/clj-shell-utils.jar:test
   5 
   6 cp -a test/unit "$AUTOPKGTEST_TMP/test"
   7 cp -a dev-resources "$AUTOPKGTEST_TMP"
   8 cd "$AUTOPKGTEST_TMP"
   9 
  10 clojure \
  11     -cp $CLASSPATH \
  12     -e "(require '[clojure.test])" \
  13     -e "(require '[puppetlabs.puppetserver.shell-utils-test])" \
  14     -e "(System/exit (if (clojure.test/successful? (clojure.test/run-tests
  15          'puppetlabs.puppetserver.shell-utils-test)) 0 1))"

Building the package

Now your are ready to build your package!

You will need git-buildpackage installed in order to build packages, in addition to the rest of the build dependencies specified in debian/control.

When upstream tags their releases

To build the package, run

$ gbp buildpackage -uc -us --git-pristine-tar  # Build package, don't sign source or changes

When upstream does NOT tag their releases

You should run

$ gbp buildpackage -uc -us --git-pristine-tar-commit

to ensure that we generate a pristine tar from the upstream source and that we commit it to the pristine-tar branch during our build. You'll only need to run this once; after your pristine tar is committed, you can build using the command above in the "When upstream tags their releases" section.

Your built package

Assuming all goes well, your packaging files can all be found in the directory above the source directory, looking something like:

.
├── gibson
│   └── ... (source files)
├── gibson-clojure_0.1.0-1.debian.tar.xz
├── gibson-clojure_0.1.0-1.dsc
├── gibson-clojure_0.1.0-1_amd64.build
├── gibson-clojure_0.1.0-1_amd64.buildinfo
├── gibson-clojure_0.1.0-1_amd64.changes
├── gibson-clojure_0.1.0.orig.tar.gz
└── libgibson-clojure_0.1.0-1_all.deb

Yay! You have a package!

When things go wrong

Here are some tips for when you run into build issues.

Issue: GBP refuses to build my package!

Maybe you received an error like this one:

gbp:error: You have uncommitted changes in your source tree:
gbp:error: On branch debian/main
...

GBP will have printed a list of the files that are not committed to the source tree.

If these are build artifacts, you should first remove them by running

$ debian/rules clean

If they are changes to your packaging, you may choose to commit them, or to ignore them for your current build by passing the following flag to gbp:

$ gbp buildpackage -uc -us --git-ignore-new

Issue: No files were listed but it still won't build!

Commonly, upstream .gitignore files on Clojure projects ignore all pom.xml changes. Ensure that you haven't forgotten to commit or reset any changes in your POM.

Issue: My build failed because I'm missing dependencies!

You should make sure you install anything missing. dkpg-checkbuilddeps will let you know which ones are missing:

dpkg-checkbuilddeps: error: Unmet build dependencies: libtools-nrepl-clojure (>= 0.2.12)
dpkg-buildpackage: warning: build dependencies/conflicts unsatisfied; aborting
dpkg-buildpackage: warning: (Use -d flag to override.)
debuild: fatal error at line 1116:
dpkg-buildpackage -rfakeroot -us -uc -i -I failed
gbp:error: 'debuild -i -I -uc -us' failed: it exited with 29

If these dependencies aren't available on your current OS/distro, you may opt to create a special environment for your package builds, to ensure they have access to all the dependencies you need. For example, you could create an LXC or a VM running Debian sid (unstable). You could even get creative and make a Dockerfile with a debian:unstable base.

Leiningen Helper

Dh-leiningen needs your help! At the moment, a lot of these steps are manual, but it would be nice to have a debhelper tool to simplify packaging Clojure libraries and to make them more consistent.

If you're interested, please contact the Clojure Team.


CategoryClojure