cctbx packaging

peoples interested by this packaging effort

you can find the git repository here cctbx repo

the current ITP is here 679905

upstream

the main website is there http://cctbx.sourceforge.net/

build dependencies

mmdb

679982

gpp4

679988

clipper

679990

bundled

patch series status

0001-remove-hardcoded-libtbx_build-env.patch 0002-fix-opengl-header-missing-gltbx.patch 0003-correct-paths-in-dispatcher-creation.patch 0004-upstream-fix-for-declaration-errors-in-gcc4.7.patch 0005-fix-for-gcc4.7-compilation-error.patch 0006-options-for-system-libs-installtarget-and-prefix.patch 0007-adding-shlib-versioning.patch 0008-Fix-to-skip-pycbf-build.patch 0009-build-libann-statically.patch 0010-adding-setup_py.patch 0011-fix-missing-python-lib-during-linking.patch 0012-fix-to-remove-cctbx.python-interpreter.patch 0013-fix-to-support-LDFLAGS-in-use_enviroment_flags.patch 0014-Fix-to-append-CPPFLAGS-to-CXXFLAGS.patch 0015-fix-cif-parser-to-work-with-antlr3c-3.2.patch 0016-autogenerate-pkgconfig-files.patch 0017-Fix-to-use-systems-include-path.patch 0018-Fix-to-skip-build-of-clipper-examples.patch

package organisation

As explain by the upstream

python module organisation generated using this script cctbx-depends.png

libraries generated

Dispatcher scripts

There are 5 categories of dispatcher scripts:

python modules/extensions

solved questions

interfacing between scons and setup.py

We'll use a custom distutils "build_ext", "install_lib" and "install_data" commands which call the scons build system under the hood, by spawning a separate process. We'll also have custom "test" and "clean" commands. That way, "python2.x setup.py build / install" work as usual.

As the upstream build system needs to be run with the same version of python that was configured, we'll call scons with: "python2.x /usr/bin/scons"

importing the extensions

Upstream import the extensions a little bit differently from the typical python project. Usually, extensions .so files are located inside the package directory where they belong. In cctbx, all .so file are in a "lib" directory, which is added to PYTHONPATH, and then an import stub imports the objects to their final place.

The extensions are not meant to be imported directly by the user, and most of them are only imported from one place, however scitbx_array_family_shared_ext.so is imported from several places.

The import stubs use a wrapper around __import__ called boost.python.import_ext, which is part of boost_adaptbx. This is defined in cctbx_sources/boost_adaptbx/boost/python.py. This wrapper does 3 things:

We'll split the extensions between the different debian packages, but keep otherwise the upstream way. This causes a runtime dependency on boost_adaptbx an a little bit of namespace pollution, but at least it doesn't introduce debian-specific problems.

Upstream builds and tests with a global option to the python interpreter: "-Qnew". This sets the int division to produce floats in all modules (i.e. this is the same as adding a "from __future__ import division" line in all modules). Such a global option is inacceptable for Debian, so we would add the "__future__" line in cctbx modules instead.

Adding such lines automatically during build is possible, as is shown by our "build_py" distutils command. This commands adds the "__future__" line in non-empty modules as the first executable line, while respecting comments, whitespace and inline help that may exist at the start of the file.

That beeing said, upstream do not seem to actively remove the "__future__" lines in modules that contain them, and seem to avoid int division altogether. So maybe they are trying to keep their code working without "-Qnew" as well. If they could provide us with such a warranty, we would not need to change the modules, which would of course be the best.

So, when the packaging is ready, we'll have to ask upstream if working without "-Qnew" is part of their development goals. If so, we might be able to remove our "build_py" command.

in Baptiste's TODO list

unbundle "optik"

this is a patched version of a deprecated ancestor of stdlib's "optparse". It is only used in "libtbx/option_parser.py". I plan to change "libtbx/option_parser.py" to use "optparse" instead.

tests

cctbx has 2 kinds of tests: python scripts, which the upstream test system ("libtbx/test_utils.py") executes in-place in the source tree, and compiled test programs, which are built by scons in the build directory and executed from there. Also, many of the python tests are not part of an importable python package, so they are by default not taken into account by distutils.

The "test" command that we have right now knows how to

I plan to add the python tests that are not part of a package as "package_data" in setup.py. Thus they will be copied by distutils to its build directory, and also installed by our packages.

When this is done, I will go through the still failing tests and see if the failure is our fault. I will also skip the tests that do not make sense in our case, and those that take an unreasonably long time.

open questions

TODO

Build system

Problems

ann

upstreams libann is build with with self_includes which makes it incompatible with libann in Debian.

$ grep "const ANNbool" /usr/include/ANN/ANN.h
const ANNboolANN_ALLOW_SELF_MATCH= ANNtrue;

$ grep "const ANNbool" cctbx_sources/annlib/include/ANN/ANN.h
const ANNboolANN_ALLOW_SELF_MATCH= ANNfalse;

annlib version:

If you check ANNmanual_1.1.pdf Section 2.2.3 there is explained what this constant does. The file "cctbx_sources/annlib_adaptbx/ann/ann_adaptor.cpp" can clear things a little bit up.

So cctbx upstream ANNfalse and auto generates code with ANNtrue and namespace annself_include. So when we use annself_include it equals libann in debian.

http://cci.lbl.gov/~hohn/scitbx-tour.html#Installation http://sig9.ecanews.org/pdfs/03%20Ralf%20Grosse-Kunstleve%20-%20library_aspects.pdf