Clojure-CLI

This page is intended to help the (re-)packaging of clj (and maybe other tools?)

ITP: 891141

This is a GSoC 2021 Project

Relevant ITPs

(from the accepted proposal)

1. The following ITPs should be fixed:

2. Consider also closing the following RFPs:

We plan to collaborate with upstream during the packaging process, to ensure the package's quality. From what we have already heard from them in the debian-clojure list, they are listening :)

Packaging strategy

  1. Check which packages are already in Debian (some of them already are).
  2. Identify leaf nodes (all)
  3. Identify leaf nodes to work on (probably only a subset of them)

  4. Work on org.clojure leaf nodes

  5. Work on non- org.clojure leaf nodes

clj

Upstream repo: https://github.com/clojure/brew-install.git

In order to package clj, we first need to study the upstream dependencies.

The main branch is named after the current Clojure version (as of 2021.06.11, it is 1.10.3) 

git checkout 1.10.3

As of 2021.06.11, HEAD --> 1.10.3, and HEAD --> 577cb35b0a2475ec5befb93087873770a3ec0c4c.

Conclusions

  1. The only first-level dependency not already in Debian is org.clojure:tools.deps.alpha.

  2. There are two org.clojure leaf nodes: libtools-gitlibs-clojure and libtools-cli-clojure.

Course of action

Proposed:

  1. package org.clojure:tools.gitlibs

  2. consider updating libtools-cli-clojure (already in Debian)

  3. consider packaging org.clojure:data.xml

  4. consider packaging com.cognitec.aws:* packages

After feedback from pollo, ehashman 891136 and Alex Miller:

  1. update org.clojure:data.xml to 0.2.0-beta6

    • lein uses 0.2.0-beta5 ==> open PR upstream so lein uses 0.2.0-beta6

    • lein has many outdated dependencies: maybe also consider opening other PRs upstream so lein uses latest dependencies of other libraries already in Debian?

  2. update libjsch-agent-proxy-java to (TBD: 891136 mentions 0.0.9)

  3. update jgit to (TBD: 891136 mentions 4.10.0)

  4. consider updating libtools-cli-clojure (already in Debian)

  5. package org.clojure:tools.gitlibs

  6. consider packaging com.cognitec.aws:* packages

Detailed Dependency Analysis

UPDATE: The following dependency has a fatal flaw: I did it on master (and not the release tag). Thanks @pollo for noting this! I'm missing one dependency: [org.clojure/data.codec] (not currently on Debian. As this dependency was later replaced upstream, I have two options: a) Also package data.codec (which will eventually be removed), or b) patching the release with the dependency switch. As the first option would imply useless work, I choose the second one.

`data-xml-clojure` dependencies (raw tree)

commit da84d4f3a9151f94a3846d752c9ec1fb8e195764 (HEAD -> master) 

% lein deps :tree
...
Possibly confusing dependencies found:
[org.clojure/clojurescript "1.10.439"] -> [com.google.javascript/closure-compiler-unshaded "v20180805"] -> [args4j "2.0.26"]
 overrides
[figwheel-sidecar "0.5.17"] -> [figwheel "0.5.17" :exclusions [org.clojure/tools.reader]] -> [org.clojure/clojurescript "1.10.238" :exclusions [org.apache.ant/ant]] -> [com.google.javascript/closure-compiler-unshaded "v20180204"] -> [args4j "2.33"]
 and
[figwheel-sidecar "0.5.17"] -> [org.clojure/clojurescript "1.10.238" :exclusions [org.apache.ant/ant]] -> [com.google.javascript/closure-compiler-unshaded "v20180204"] -> [args4j "2.33"]

Consider using these exclusions:
[figwheel-sidecar "0.5.17" :exclusions [args4j]]

[org.clojure/clojurescript "1.10.439"] -> [com.google.javascript/closure-compiler-unshaded "v20180805"] -> [com.google.errorprone/error_prone_annotations "2.0.18"]
 overrides
[org.clojure/clojurescript "1.10.439"] -> [com.google.javascript/closure-compiler-unshaded "v20180805"] -> [com.google.guava/guava "25.1-jre"] -> [com.google.errorprone/error_prone_annotations "2.1.3"]

Consider using these exclusions:
[org.clojure/clojurescript "1.10.439" :exclusions [com.google.errorprone/error_prone_annotations]]

[org.clojure/clojurescript "1.10.439"] -> [com.google.javascript/closure-compiler-unshaded "v20180805"] -> [com.google.code.findbugs/jsr305 "3.0.1"]
 overrides
[org.clojure/clojurescript "1.10.439"] -> [com.google.javascript/closure-compiler-unshaded "v20180805"] -> [com.google.guava/guava "25.1-jre"] -> [com.google.code.findbugs/jsr305 "3.0.2"]

Consider using these exclusions:
[org.clojure/clojurescript "1.10.439" :exclusions [com.google.code.findbugs/jsr305]]

 [binaryage/devtools "0.9.10"]
   [binaryage/env-config "0.2.2"]
 [clojure-complete "0.2.5" :exclusions [[org.clojure/clojure]]]
 [com.cemerick/piggieback "0.2.2"]
 [figwheel-sidecar "0.5.17"]
   [clj-stacktrace "0.2.8"]
   [co.deps/ring-etag-middleware "0.2.0"]
   [com.stuartsierra/component "0.3.2"]
     [com.stuartsierra/dependency "0.2.0"]
   [figwheel "0.5.17" :exclusions [[org.clojure/tools.reader]]]
   [hawk "0.2.11" :exclusions [[org.clojure/clojure]]]
     [net.incongru.watchservice/barbary-watchservice "1.0"]
       [net.java.dev.jna/jna "3.2.2"]
   [http-kit "2.3.0"]
   [org.clojure/core.async "0.4.474" :exclusions [[org.clojure/tools.reader]]]
     [org.clojure/tools.analyzer.jvm "0.7.0"]
       [org.clojure/core.memoize "0.5.9"]
         [org.clojure/core.cache "0.6.5"]
           [org.clojure/data.priority-map "0.0.7"]
       [org.clojure/tools.analyzer "0.6.9"]
       [org.ow2.asm/asm-all "4.2"]
   [ring-cors "0.1.12" :exclusions [[ring/ring-core] [org.clojure/clojure]]]
   [ring/ring-core "1.6.3" :exclusions [[org.clojure/tools.reader] [org.clojure/clojure]]]
     [clj-time "0.11.0"]
       [joda-time "2.8.2"]
     [commons-fileupload "1.3.3"]
     [commons-io "2.5"]
     [crypto-equality "1.0.0"]
     [crypto-random "1.2.0"]
     [ring/ring-codec "1.0.1"]
   [simple-lein-profile-merge "0.1.4"]
   [strictly-specking-standalone "0.1.1"]
     [net.cgrand/parsley "0.9.3" :exclusions [[org.clojure/clojure]]]
       [net.cgrand/regex "1.1.0"]
     [net.cgrand/sjacket "0.1.1" :exclusions [[org.clojure/clojure] [net.cgrand/parsley]]]
   [suspendable "0.1.1" :exclusions [[org.clojure/clojure] [com.stuartsierra/component]]]
 [nrepl "0.6.0" :exclusions [[org.clojure/clojure]]]
 [org.clojure/clojure "1.10.0-beta8"]
   [org.clojure/core.specs.alpha "0.2.44"]
   [org.clojure/spec.alpha "0.2.176"]
 [org.clojure/clojurescript "1.10.439"]
   [com.cognitect/transit-clj "0.8.309" :exclusions [[org.clojure/clojure]]]
     [com.cognitect/transit-java "0.8.332"]
       [com.fasterxml.jackson.core/jackson-core "2.8.7"]
       [commons-codec "1.10"]
       [org.msgpack/msgpack "0.6.12"]
         [com.googlecode.json-simple/json-simple "1.1.1" :exclusions [[junit]]]
         [org.javassist/javassist "3.18.1-GA"]
   [com.google.javascript/closure-compiler-unshaded "v20180805"]
     [args4j "2.0.26"]
     [com.google.code.findbugs/jsr305 "3.0.1"]
     [com.google.code.gson/gson "2.7"]
     [com.google.errorprone/error_prone_annotations "2.0.18"]
     [com.google.guava/guava "25.1-jre"]
       [com.google.j2objc/j2objc-annotations "1.1"]
       [org.checkerframework/checker-qual "2.0.0"]
       [org.codehaus.mojo/animal-sniffer-annotations "1.14"]
     [com.google.javascript/closure-compiler-externs "v20180805"]
     [com.google.jsinterop/jsinterop-annotations "1.0.0"]
     [com.google.protobuf/protobuf-java "3.0.2"]
   [org.clojure/data.json "0.2.6"]
   [org.clojure/google-closure-library "0.0-20170809-b9c14c6b"]
     [org.clojure/google-closure-library-third-party "0.0-20170809-b9c14c6b"]
   [org.clojure/tools.reader "1.3.0"]
   [org.mozilla/rhino "1.7R5"]
 [org.clojure/test.check "0.9.0"]
 [org.clojure/tools.nrepl "0.2.13"]

`lein` dependencies (raw tree, upstream)

commit e270102447fb9ca1f33591d39b72b15d06cdc0ef (HEAD -> master) 

% lein deps :tree
 [clojure-complete "0.2.5" :exclusions [[org.clojure/clojure]]]
 [commons-io "2.8.0"]
 [commons-lang "2.6"]
 [leiningen-core "2.9.5"]
   [clj-commons/pomegranate "1.2.0" :exclusions [[org.slf4j/jcl-over-slf4j]]]
     [org.apache.maven.resolver/maven-resolver-api "1.3.3"]
     [org.apache.maven.resolver/maven-resolver-connector-basic "1.3.3"]
     [org.apache.maven.resolver/maven-resolver-impl "1.3.3"]
     [org.apache.maven.resolver/maven-resolver-spi "1.3.3"]
     [org.apache.maven.resolver/maven-resolver-transport-file "1.3.3"]
     [org.apache.maven.resolver/maven-resolver-transport-http "1.3.3"]
     [org.apache.maven.resolver/maven-resolver-transport-wagon "1.3.3"]
     [org.apache.maven.resolver/maven-resolver-util "1.3.3"]
     [org.apache.maven.wagon/wagon-http "3.3.2"]
       [org.apache.maven.wagon/wagon-http-shared "3.3.2"]
         [org.jsoup/jsoup "1.11.3"]
     [org.apache.maven.wagon/wagon-provider-api "3.3.2"]
     [org.apache.maven/maven-resolver-provider "3.6.1"]
       [javax.inject "1"]
       [org.apache.maven/maven-model-builder "3.6.1"]
         [org.apache.maven/maven-artifact "3.6.1"]
           [org.apache.commons/commons-lang3 "3.8.1"]
         [org.apache.maven/maven-builder-support "3.6.1"]
         [org.codehaus.plexus/plexus-component-annotations "1.7.1" :exclusions [[junit]]]
         [org.codehaus.plexus/plexus-interpolation "1.25"]
       [org.apache.maven/maven-model "3.6.1"]
       [org.apache.maven/maven-repository-metadata "3.6.1"]
       [org.codehaus.plexus/plexus-utils "3.2.0"]
   [com.hypirion/io "0.3.1"]
   [org.clojure/clojure "1.10.1"]
     [org.clojure/core.specs.alpha "0.2.44"]
     [org.clojure/spec.alpha "0.2.176"]
   [org.clojure/tools.macro "0.1.5"]
   [org.flatland/classlojure "0.7.1"]
   [org.slf4j/slf4j-nop "1.7.25"]
     [org.slf4j/slf4j-api "1.7.25"]
   [robert/hooke "1.3.0"]
 [net.cgrand/parsley "0.9.3" :exclusions [[org.clojure/clojure]]]
 [nrepl "0.8.3"]
 [org.apache.httpcomponents/httpclient "4.5.13"]
   [commons-codec "1.11"]
   [commons-logging "1.2"]
 [org.apache.httpcomponents/httpcore "4.4.13"]
 [org.clojars.trptcolin/sjacket "0.1.1.1" :exclusions [[org.clojure/clojure]]]
   [net.cgrand/regex "1.1.0"]
 [org.clojure/data.xml "0.2.0-alpha5"]
   [org.clojure/data.codec "0.1.0"]
 [scout "0.1.1"]
 [stencil "0.5.0" :exclusions [[org.clojure/core.cache]]]
   [quoin "0.1.2"]
 [timofreiberg/bultitude "0.3.0" :exclusions [[org.clojure/clojure]]]
   [org.tcrawley/dynapath "1.0.0"]

`clj` dependencies (raw tree)

According to mvn dependency:tree, we have the following tree:

commit 577cb35b0a2475ec5befb93087873770a3ec0c4c (HEAD -> 1.10.3) 

% mvn dependency:tree
[INFO] Scanning for projects...
[INFO]
[INFO] ---------------------< org.clojure:clojure-tools >----------------------
[INFO] Building clojure-tools 1.10.3.855
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ clojure-tools ---
[INFO] org.clojure:clojure-tools:jar:1.10.3.855
[INFO] +- org.clojure:clojure:jar:1.10.3:compile
[INFO] |  +- org.clojure:spec.alpha:jar:0.2.194:compile
[INFO] |  \- org.clojure:core.specs.alpha:jar:0.2.56:compile
[INFO] +- org.clojure:tools.deps.alpha:jar:0.11.922:compile
[INFO] |  +- org.apache.maven.resolver:maven-resolver-api:jar:1.7.0:compile
[INFO] |  +- org.apache.maven.resolver:maven-resolver-spi:jar:1.7.0:compile
[INFO] |  +- org.apache.maven.resolver:maven-resolver-impl:jar:1.7.0:compile
[INFO] |  |  +- org.apache.maven.resolver:maven-resolver-named-locks:jar:1.7.0:compile
[INFO] |  |  +- org.apache.commons:commons-lang3:jar:3.8.1:compile
[INFO] |  |  \- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  +- org.apache.maven.resolver:maven-resolver-util:jar:1.7.0:compile
[INFO] |  +- org.apache.maven.resolver:maven-resolver-connector-basic:jar:1.7.0:compile
[INFO] |  +- org.apache.maven.resolver:maven-resolver-transport-file:jar:1.7.0:compile
[INFO] |  +- org.apache.maven.resolver:maven-resolver-transport-http:jar:1.7.0:compile
[INFO] |  |  +- org.apache.httpcomponents:httpclient:jar:4.5.13:compile
[INFO] |  |  |  \- commons-codec:commons-codec:jar:1.11:compile
[INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.14:compile
[INFO] |  |  \- org.slf4j:jcl-over-slf4j:jar:1.7.30:runtime
[INFO] |  +- org.apache.maven:maven-resolver-provider:jar:3.8.1:compile
[INFO] |  |  +- org.apache.maven:maven-model:jar:3.8.1:compile
[INFO] |  |  +- org.apache.maven:maven-model-builder:jar:3.8.1:compile
[INFO] |  |  |  \- org.codehaus.plexus:plexus-interpolation:jar:1.25:compile
[INFO] |  |  +- org.apache.maven:maven-repository-metadata:jar:3.8.1:compile
[INFO] |  |  \- org.codehaus.plexus:plexus-utils:jar:3.2.1:compile
[INFO] |  +- org.apache.maven:maven-core:jar:3.8.1:compile
[INFO] |  |  +- org.apache.maven:maven-settings:jar:3.8.1:compile
[INFO] |  |  +- org.apache.maven:maven-settings-builder:jar:3.8.1:compile
[INFO] |  |  |  \- org.sonatype.plexus:plexus-sec-dispatcher:jar:1.4:compile
[INFO] |  |  |     \- org.sonatype.plexus:plexus-cipher:jar:1.4:compile
[INFO] |  |  +- org.apache.maven:maven-builder-support:jar:3.8.1:compile
[INFO] |  |  +- org.apache.maven:maven-artifact:jar:3.8.1:compile
[INFO] |  |  +- org.apache.maven:maven-plugin-api:jar:3.8.1:compile
[INFO] |  |  +- org.apache.maven.shared:maven-shared-utils:jar:3.2.1:compile
[INFO] |  |  |  \- commons-io:commons-io:jar:2.5:compile
[INFO] |  |  +- org.eclipse.sisu:org.eclipse.sisu.plexus:jar:0.3.4:compile
[INFO] |  |  |  \- javax.enterprise:cdi-api:jar:1.0:compile
[INFO] |  |  |     \- javax.annotation:jsr250-api:jar:1.0:compile
[INFO] |  |  +- org.eclipse.sisu:org.eclipse.sisu.inject:jar:0.3.4:compile
[INFO] |  |  +- com.google.inject:guice:jar:no_aop:4.2.1:compile
[INFO] |  |  |  +- aopalliance:aopalliance:jar:1.0:compile
[INFO] |  |  |  \- com.google.guava:guava:jar:25.1-android:compile
[INFO] |  |  |     +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  |  |     +- org.checkerframework:checker-compat-qual:jar:2.0.0:compile
[INFO] |  |  |     +- com.google.errorprone:error_prone_annotations:jar:2.1.3:compile
[INFO] |  |  |     +- com.google.j2objc:j2objc-annotations:jar:1.1:compile
[INFO] |  |  |     \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.14:compile
[INFO] |  |  +- org.codehaus.plexus:plexus-classworlds:jar:2.6.0:compile
[INFO] |  |  \- org.codehaus.plexus:plexus-component-annotations:jar:2.1.0:compile
[INFO] |  +- org.clojure:data.xml:jar:0.2.0-alpha6:compile
[INFO] |  |  \- org.clojure:data.codec:jar:0.1.0:compile
[INFO] |  +- org.clojure:tools.gitlibs:jar:2.3.167:compile
[INFO] |  +- org.clojure:tools.cli:jar:1.0.206:compile
[INFO] |  +- com.cognitect.aws:api:jar:0.8.505:compile
[INFO] |  |  +- org.clojure:data.json:jar:1.0.0:compile
[INFO] |  |  +- org.clojure:tools.logging:jar:1.1.0:compile
[INFO] |  |  +- com.cognitect:http-client:jar:0.1.106:compile
[INFO] |  |  |  +- org.eclipse.jetty:jetty-client:jar:9.4.36.v20210114:compile
[INFO] |  |  |  |  \- org.eclipse.jetty:jetty-io:jar:9.4.36.v20210114:compile
[INFO] |  |  |  +- org.eclipse.jetty:jetty-http:jar:9.4.36.v20210114:compile
[INFO] |  |  |  \- org.eclipse.jetty:jetty-util:jar:9.4.36.v20210114:compile
[INFO] |  |  \- org.clojure:core.async:jar:1.3.610:compile
[INFO] |  |     \- org.clojure:tools.analyzer.jvm:jar:1.1.0:compile
[INFO] |  |        +- org.clojure:tools.analyzer:jar:1.0.0:compile
[INFO] |  |        +- org.clojure:core.memoize:jar:1.0.236:compile
[INFO] |  |        |  \- org.clojure:core.cache:jar:1.0.207:compile
[INFO] |  |        |     \- org.clojure:data.priority-map:jar:1.0.0:compile
[INFO] |  |        +- org.ow2.asm:asm:jar:5.2:compile
[INFO] |  |        \- org.clojure:tools.reader:jar:1.3.2:compile
[INFO] |  +- com.cognitect.aws:endpoints:jar:1.1.11.969:compile
[INFO] |  +- com.cognitect.aws:s3:jar:811.2.858.0:compile
[INFO] |  \- javax.inject:javax.inject:jar:1:compile
[INFO] \- org.slf4j:slf4j-nop:jar:1.7.30:compile
[INFO]    \- org.slf4j:slf4j-api:jar:1.7.30:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.599 s
[INFO] Finished at: 2021-06-09T11:07:08-03:00
[INFO] ------------------------------------------------------------------------

`clj` dependencies (full tree, listed)

`clj` dependencies (first-level)

* org.clojure:clojure:jar:1.10.3:compile (has 2 dependencies, already in Debian as clojure)

* org.clojure:tools.deps.alpha:jar:0.11.922:compile (16 dependencies, not yet in Debian)

* org.slf4j:slf4j-nop:jar:1.7.30:compile (has 1 dependency, already in Debian as libslf4j)

`clj` dependencies (second-level, only those for `org.clojure:tools.deps.alpha`)

Patching release 0.2.0-alpha6

As noted by @pollo, there is a relevant change we should patch (the dependency tree I originally posted was from current master, not from the release tag.)

We identify the commit as e4a50a68c80acf7cd6e48c0438b4bd08acaff331. https://github.com/clojure/data.xml/commit/e4a50a68c80acf7cd6e48c0438b4bd08acaff331 

        git diff data.xml-0.2.0-alpha6..e4a50a68c80acf7cd6e48c0438b4bd08acaff331

Affected files:

This patch is justified because it backports a dependency replacement from a Clojure-specific library _not on Debian_ to a JDK-builtin one (JDK >= 8). If we didn't patch the released package, we should also package [org.clojure/data.codec]. Since the dependency has since been removed upstream, this makes no sense.

HOWTO: Follow steps from https://wiki.debian.org/UsingQuilt

Make sure there are no other relevant changes between the release and current master (da84d4f3a9151f94a3846d752c9ec1fb8e195764, from April 15th, 2021)

        git diff data.xml-0.2.0-alpha6..e4a50a68c80acf7cd6e48c0438b4bd08acaff331

        git diff e4a50a68c80acf7cd6e48c0438b4bd08acaff331..master

In both cases, we see only minor changes.

Do the actual patching

1. apply existing patches to the source

Step one from https://wiki.debian.org/UsingQuilt 

% quilt push -a
The working tree was created by an older version of quilt. Please run 'quilt upgrade'.

% quilt upgrade
Converting meta-data to version 2

Use dch to update the changelog.

Commit.

2. Create a new patch 

% quilt new Replace-data-codec-with-builtin-base64-encoding.diff
Patch patches/Replace-data-codec-with-builtin-base64-encoding.diff is now on top

3. Specify files to be patched 

% quilt add pom.xml project.clj src/main/clojure/clojure/data/xml/impl.clj
File pom.xml added to patch patches/Replace-data-codec-with-builtin-base64-encoding.diff
File project.clj added to patch patches/Replace-data-codec-with-builtin-base64-encoding.diff
File src/main/clojure/clojure/data/xml/impl.clj added to patch patches/Replace-data-codec-with-builtin-base64-encoding.diff

Make sure the applied changes are **only** those we need to apply. If we just replace the files, we will apply changes we don't want to apply.

4. Get the actual files from upstream commit, and modify current versions accordingly.

(The upstream commit is e4a50a68c80acf7cd6e48c0438b4bd08acaff331.)

A non-optimal way of getting the files is via ?GitHub's web interface.

5. Refresh the patches 

% quilt refresh
Refreshed patch patches/Replace-data-codec-with-builtin-base64-encoding.diff

6. Edit the patch header. 

% quilt header -e
Replaced header of patch patches/Replace-data-codec-with-builtin-base64-encoding.diff

7. Finish editing the patch. 

% quilt pop -a
Removing patch patches/Replace-data-codec-with-builtin-base64-encoding.diff
Restoring src/main/clojure/clojure/data/xml/impl.clj
Restoring project.clj
Restoring pom.xml

No patches applied