Notes from the 2017 Debian Med Sprint in Bucharest
Progress on meeting goals
Done
Updated cwltool & schema-salad: done. Added cwl-runner via update-alternatives, also for toil
- Packaged Toil resource distributor for CWL workflows (python-bd2k and essential modules of cgcloud with it)
seqan 2.3.1 is packaged & uploaded
Ongoing
- Preparing package for conda
Establish workflow for transfer of descriptions to bio.tools, working document on https://docs.google.com/document/d/1H44HxQik2RnxWzBui3GuYI9pAMfX5tNDBiwRa5g0WXg/edit?usp=sharing
- Complete dependencies for Galaxy
- reiterating on borked python-bx packaging - now in New Queue
- Remaining missing python dependencies (to be validated by all):
- testfixtures (needed only for testing)
- pulsar-galaxy (is it really needed? it is a runner, so could be disabled/removed from package)
- I (Olivier) created a galaxy git repo in debianmed with a begin of packaging and added to README.source info about missing deps, possible workaround/patches etc.... so that debianmed team can collaborate and update progress. Latest galaxy release is imported. Of course, there is still lots to do.
- Galaxy fails with python-routes/python-webhelpers from Debian, more recent than galaxy requirements. Would need patching (2 lines modification)
Galaxy status: several dependencies are not yet python3 compatible and will impact Galaxy code (https://github.com/galaxyproject/galaxy/issues/1715). Upstream is working on this, but no roadmap yet. No python3 support is blocking. Auto install of dependencies by Galaxy in a virtualenv, shared with installation of toolshed means code modification in Debian to use system libs. This *may* be managed with --system-site-packages option in virtualenv creation.
- run.sh options --skip-venv --skip-eggs --skip-wheels seem to disable virtualenv and packages installation.
- Tested Galaxy execution and unit tests with Debian package and above command. With a few patching to fix routes for runtime execution and run-test.sh for testing, it looks fine!
- Huge number of Javascript libraries, some internals, other from external sources, to be listed and investigated:
For runtime and build
"jquery": "~1.12", <= 3.1 !!
"ravenjs": "~3", NO!
"underscore": "~1", <= 1.8.3
"backbone": "~1.3", <= 1.3.3
"bootstrap": "~3.3.2", <= 3.3.7
"bootstrap-tour": "~0.10.2", <= 0.11
"d3": "~3", <= 3.5.17
"farbtastic": "~2.0.0-alpha.1", <= NO
"toastr": "~2.1.0", <= NO
"jQTouch": "git://github.com/senchalabs/jQTouch#~1.0.0", <= NO
"bib2json": "git://github.com/galaxyproject/bib2json#galaxy", <= NO
"jquery-form": "~3.46.0", <= 11 ??
"jquery-autocomplete": "git://github.com/dyve/jquery-autocomplete", <= NO
"select2": "~3.5.2", <= 4.0.1
"jStorage": "~0.4.12", <= 0.3.1
"jquery.cookie": "~1.4.1", <= 11 ?
"dynatree": "~1.2.5", <= NO
"jquery-mousewheel": "~3.1.12", <= 11?
"wymeditor": "~1.0.0-rc.1", <= NO
"jstree": "~3.0.9", <= NO
"jquery-ui": "git://github.com/jquery/jquery-ui.git#~1.11.2", <= 1.12.1
"jquery.complexify": "git://github.com/danpalmer/jquery.complexify.js.git#~0.5.1", <= NO
"threedubmedia.jquery.event": "*", <= NO
"jquery-migrate": "~1.4", <= 1.4.1
"requirejs": "~2" <= 2.3.2
To build libraries:
"amdi18n-loader": "^0.2.0", <= NO
"grunt": "^0.4.5", <= OK
"grunt-bower-install-simple": "^1.1.0", <= NO
"grunt-check-modules": "^1.0.0", <= NO
"grunt-cli": "^0.1.13", <= 1.2
"grunt-contrib-clean": "^0.6.0", <= 1.0
"grunt-contrib-copy": "^0.5.0", <= 1.0
"grunt-contrib-less": "^1.1.0", <= NO
"grunt-contrib-uglify": "^0.8.0", <= 2.0
"grunt-contrib-watch": "^0.6.1", <= NO
"grunt-exec": "^0.4.6", <= NO
"grunt-spritesmith": "^4.7.1", <= NO
"i18n-webpack-plugin": "^0.2.7", <= NO
"webpack": "^1.10.1", <= NO
"webpack-dev-server": "^1.7.0" <= NO- packaging bcbio: MRC working on it and pybedtools dependency
- snpeff: looked at dependencies needed to package it
apfloat => source at http://repo1.maven.org/maven2/org/apfloat/apfloat/1.6.3/
charts4j => from https://code.google.com/archive/p/charts4j/, code at https://github.com/julienchastang/charts4j
akka-actor => source at http://dl.bintray.com/typesafe/maven-releases/com/typesafe/akka/akka-actor/2.0.1/
trove4j => in debian (libtrove-java) freemarker in Debian
distlib => unknown source , only bin jar on maven, but no reference in code...
samtools => v1.107, too old, code not compatible with htsjdk...
- log4j in Debian
- Main issue seems to be samtools-related with imports like import net.sf.samtools.SAMFileReader
Packaging lobSTR workflow: Bogdan & Michael got the workflow to run again https://github.com/common-workflow-language/workflows/pull/126#issuecomment-272632458
Project ideas
List of demands for development discussed/identified during the meeting
- bash autocomplete from CWL tool descriptions
- manual page generation from CWL tool descriptions
using acd2cwl to generate CWL descriptions for the EMBOSS tools
- [simple] add "Files-Excluded" to dh_make template of debian/copyright
- patch uscan to allow for a recursive grep to determine what files to exclude via Files-Excluded
Demonstrate usage of http://snapshot.debian.org/ in a Dockerfile for reproducible computing
- Develop strategy for automated package updates in a 'pantsonfire' distribution
- Crowd-sourcing logos/icons for research software
establish platform inviting for graphical contributions from people with different skill set, extending http://wiki.debian.org/DebianArt
- running through an organisation that can accept money
- Cross-platform (Debian/Fedora/Conda/...) package tests formalised with terms of the CWL
- Debian gives references to those tests in debian/control
- Position paper on Debian as a natural implementation of best software-development practices to support eHealth
- Proposal to support Debian and independent organisations to support Open Source software maintenance
- distributed maintenance of community software
- knowledege distribution that goes with the writing of tests