Differences between revisions 33 and 34
Revision 33 as of 2012-07-30 19:31:21
Size: 7268
Editor: ?RadostanRiedel
Comment:
Revision 34 as of 2012-08-05 14:07:55
Size: 7438
Editor: picca
Comment:
Deletions are marked like this. Additions are marked like this.
Line 117: Line 117:
 * ask the upstream for different naming schema between nighty and release source distribution. This way it will be possible to filter nighty built in the watch file.

cctbx packaging

peoples interested by this packaging effort

  • Baptiste Carvello
  • picca

  • Radostan Riedel

you can find the git repository here cctbx repo

the current ITP is here 679905

upstream

the main website is there http://cctbx.sourceforge.net/

build dependencies

mmdb

679982

gpp4

679988

clipper

679990

bundled

  • debhelper (>= 9),

  • python-all-dev,
  • libfftw3-dev,
  • libantlr3c-dev (< 3.4),

  • libcbf-dev,
  • libgl2ps-dev,
  • libann-dev,
  • libclipper-dev,
  • libboost-python-dev(>= 1.49),

  • scons,
  • libboost-thread-dev,
  • python-setuptools,
  • libgl1-mesa-dev,
  • libglu1-mesa-dev,
  • libtool

package organisation

As explain by the upstream

python module organisation generated using this script cctbx-depends.png

  • Red boxes : python modules,
  • green boxes are others (C++ or external).
  • Black arrows are run-time dependancies
  • Red arrows are build-time dependancies
  • Green arrows are optional dependancies.

libraries generated

  • libann.a
  • libcbf.a
  • libccp4io.a
  • libcctbx_sgtbx_asu.so
  • libcctbx.so
  • libiotbx_mtz.so
  • libiotbx_pdb.so
  • libmmtbx_masks.so
  • libomptbx.so
  • librstbx.so
  • libscitbx_boost_python.so
  • libscitbx_minpack.so
  • libscitbx_slatec.so
  • libsmtbx_refinement_constraints.so
  • libspotfinder.so

Dispatcher scripts

There are 5 categories of dispatcher scripts:

  • Pseudo python interpreters *.python
  • Scripts conflicting the namespace of other packages
    • like: boost.lcm
  • Scripts which just show build or dist paths
    • like: annlib.show_build_path
  • Build related scripts
    • like: libtbx.assert_line_count
  • Everything else. (This could get usefull)
    • like: cctbx.lattice_symmetry

python modules/extensions

solved questions

interfacing between scons and setup.py

We'll use a custom distutils "build_ext" command which calls the scons build system under the hood, by spawning a separate process. That way, "python2.x setup.py build / install" work as usual.

As the upstream build system needs to be run with the same version of python that was configured, we'll call scons with: "python2.x /usr/bin/scons"

importing the extensions

Upstream import the extensions a little bit differently from the typical python project. Usually, extensions .so files are located inside the package directory where they belong. In cctbx, all .so file are in a "lib" directory, which is added to PYTHONPATH, and then an import stub imports the objects to their final place.

The extensions are not meant to be imported directly by the user, and most of them are only imported from one place, however scitbx_array_family_shared_ext.so is imported from several places.

The import stubs use a wrapper around __import__ called boost.python.import_ext, which is part of boost_adaptbx. This is defined in cctbx_sources/boost_adaptbx/boost/python.py. This wrapper does 3 things:

  • it checks for a libstdc++ mismatch with the python binary (useless in Debian)
  • it beautifies the import exception messages (nice to have, but unimportant)
  • it calls dlopen with RTLD_GLOBAL. People in the Boost community seem to think that this is necessary for correct operation in corner cases.

We'll split the extensions between the different debian packages, but keep otherwise the upstream way. This causes a runtime dependency on boost_adaptbx an a little bit of namespace pollution, but at least it doesn't introduce debian-specific problems.

open questions

  • do we need to add "from __future__ import division" lines? It looks like most modules using integer division still have them, but we don't have a warranty from upstream. Should we add them to all files, just to be sure?

  • do we package the tests? If so, the setup.py files must be fixed to include all the data files needed for the tests.
  • ask the upstream for different naming schema between nighty and release source distribution. This way it will be possible to filter nighty built in the watch file.

TODO

  • solve the API/ABI stability problems for libs
  • what should be do with this future problem (could you elaborate ?)

  • license check of the remaining files
  • package missing build-dep
  • DFSG free the source package
    • identify the bundeled libraries (boost ...)
    • repack to remove all bundled libraries, it should save some space.
    • create the get-orig-source target for this purpose
    • We can delete scons, boost and most files in clipper directory. We just to need to keep the empty directories for boost, clipper etc. otherwise the build system is not working.
  • remove the cctbx bundled in objcryst-fox once done
  • run all tests at the end of the build (python setup.py check ?)
  • what about the documentation ?
  • provide pkg-config files for public shared libraries (patch for upstream)

Build system

  • Modify configure.py to support "--use-system-libraries"
    • skip shlibs build which are already available in Debian.
    • link with the system libraries.
    • use pkg-config as much as possible for library detection
    • on Debian boost_python library is named according to the python version
      • libboost_python-py26
      • libboost_python-py27
      • libboost_python-py32
  • Build for multiple python versions but build shlibs only one time.
  • version the shared libraries using scons
  • support a --prefix option
  • the install target should support DESTDIR
  • Use upstreams "--use-environment-flags" option to set CFLAGS, CXXFLAGS, CPPFLAGS and LDFLAGS
    • we need -O2 and -g for debugging symbols.
  • add a setup.py for python modules (autogenerated ?)
  • add a clean target ?
  • construct a cctbx-dev package with dispatcher script and libtbx-build system to provide a build-dep for other packages. We will also need a python script that dynamically updates the libtbx_env pickle object.

Problems

  • upstream is linking additional symbols into libcbf. libcbf is built statically so we don't need libcbf-dev
    • can be backported into the cbf upstream ?
    • check with the cbf Debian maintainer
  • upstreams libann is build with with self_includes which makes it incompatible with libann in Debian.
    • can be backported into the ann upstream ?
    • check with the ann Debian maintainer
  • fftw3tbx is missing a header file.
  • useless file in the current source package cctbx_sources/lapack_fem/xerbla.f.orig
  • what should we do with all the dispatcher in the bin directory ?
  • Depend on cbflib python bindings but cbflib in debian is not providing a python package 681997

  • Upstream is linking additional symbols from generated source into their libann.so. They call it ""Generating C++ files for self-inclusive approximate nearest neighbor"

http://cci.lbl.gov/~hohn/scitbx-tour.html#Installation http://sig9.ecanews.org/pdfs/03%20Ralf%20Grosse-Kunstleve%20-%20library_aspects.pdf