Differences between revisions 26 and 27
Revision 26 as of 2012-07-16 12:37:29
Size: 6062
Editor: ?Baptiste Carvello
Comment:
Revision 27 as of 2012-07-17 10:23:04
Size: 6351
Editor: ?RadostanRiedel
Comment:
Deletions are marked like this. Additions are marked like this.
Line 131: Line 131:
 * Build for multiple python versions but build shlibs only one time.
Line 134: Line 135:
 * can we customise the build with the traditionals CFLAGS, CXXFLAGS, CPPFLAGS and LDFLAGS ?  * Use upstreams "--use-environment-flags" option to set CFLAGS, CXXFLAGS, CPPFLAGS and LDFLAGS ?
Line 138: Line 139:
 * construct a cctbx-dev package with dispatcher script and libtbx-build system to provide a build-dep for other packages. We will also need a python script that dynamically updates the libtbx_env pickle object.

cctbx packaging

peoples interested by this packaging effort

  • Baptiste Carvello
  • picca

  • Radostan Riedel

you can find the git repository here cctbx repo

the current ITP is here 679905

upstream

the main website is there http://cctbx.sourceforge.net/

build dependencies

mmdb

679982

gpp4

679988

clipper

679990

bundled

  • libann-dev
  • libboost-python-dev (>= 1.49)

  • libfftw3-dev
  • python-setuptools (>= 0.6.10)

  • python-all-dev
  • debhelper (>= 7)

  • python-all-dev (>= 2.3)

  • python (>= 2.3.5-11)

  • libboost-thread-dev
  • scons
  • python-support
  • libtool
  • coreutils

package organisation

As explain by the upstream

python module organisation generated using this script cctbx-depends.png

  • Red boxes : python modules,
  • green boxes are others (C++ or external).
  • Black arrows are run-time dependancies
  • Red arrows are build-time dependancies
  • Green arrows are optional dependancies.

libraries generated

  • libann.so
  • libcbf.a
  • libccp4io.a
  • libcctbx_sgtbx_asu.so
  • libcctbx.so
  • libiotbx_mtz.so
  • libiotbx_pdb.so
  • libmmtbx_masks.so
  • libomptbx.so
  • librstbx.so
  • libscitbx_boost_python.so
  • libscitbx_minpack.so
  • libscitbx_slatec.so
  • libsmtbx_refinement_constraints.so
  • libspotfinder.so

is it all ?, API/ABI stability ?

python modules/extensions

solved questions

interfacing between scons and setup.py

We'll use a custom distutils "build_ext" command which calls the scons build system under the hood, by spawning a separate process. That way, "python2.x setup.py build / install" work as usual.

As the upstream build system needs to be run with the same version of python that was configured, we'll call scons with: "python2.x /usr/bin/scons"

importing the extensions

Upstream import the extensions a little bit differently from the typical python project. Usually, extensions .so files are located inside the package directory where they belong. In cctbx, all .so file are in a "lib" directory, which is added to PYTHONPATH, and then an import stub imports the objects to their final place.

The extensions are not meant to be imported directly by the user, and most of them are only imported from one place, however scitbx_array_family_shared_ext.so is imported from several places.

The import stubs use a wrapper around __import__ called boost.python.import_ext, which is part of boost_adaptbx. This is defined in cctbx_sources/boost_adaptbx/boost/python.py. This wrapper does 3 things:

  • it checks for a libstdc++ mismatch with the python binary (useless in Debian)
  • it beautifies the import exception messages (nice to have, but unimportant)
  • it calls dlopen with RTLD_GLOBAL. People in the Boost community seem to think that this is necessary for correct operation in corner cases.

We'll split the extensions between the different debian packages, but keep otherwise the upstream way. This causes a runtime dependency on boost_adaptbx an a little bit of namespace pollution, but at least it doesn't introduce debian-specific problems.

open questions

  • do we need to add "from __future__ import division" lines? It looks like most modules using integer division still have them, but we don't have a warranty from upstream. Should we add them to all files, just to be sure?

  • do we package the tests? If so, the setup.py files must be fixed to include all the data files needed for the tests.

TODO

  • solve the API/ABI stability problems for libs
  • what should be do with this future problem (could you elaborate ?)

  • license check of the remaining files
  • package missing build-dep
  • DFSG free the source package
    • identify the bundeled libraries (boost ...)
    • repack to remove all bundled libraries, it should save some space.
    • create the get-orig-source target for this purpose
  • remove the cctbx bundled in objcryst-fox once done
  • take care of the pickle object libtbx_env
  • run all tests at the end of the build (python setup.py check ?)
  • what about the documentation ?
  • provide pkg-config files for public shared libraries (patch for upstream)

Build system

  • Modify configure.py to support "--use-system-libraries"
    • skip shlibs build which are already available in Debian.
    • link with the system libraries.
    • use pkg-config as much as possible for library detection
    • on Debian boost_python library is named according to the python version
      • libboost_python-py26
      • libboost_python-py27
      • libboost_python-py32
  • Build for multiple python versions but build shlibs only one time.
  • version the shared libraries using scons
  • support a --prefix option
  • the install target should support DESTDIR
  • Use upstreams "--use-environment-flags" option to set CFLAGS, CXXFLAGS, CPPFLAGS and LDFLAGS ?
    • we need -O2 and -g for debugging symbols.
  • add a setup.py for python modules (autogenerated ?)
  • add a clean target ?
  • construct a cctbx-dev package with dispatcher script and libtbx-build system to provide a build-dep for other packages. We will also need a python script that dynamically updates the libtbx_env pickle object.

Problems

  • upstream is linking additional symbols into libcbf. libcbf is built statically so we can don't need libcbf-dev
    • can be backported into the cbf upstream ?
    • check with the cbf Debian maintainer
  • upstreams libann is build with with self_includes which makes it incompatible with libann in Debian.
    • can be backported into the ann upstream ?
    • check with the ann Debian maintainer
  • fftw3tbx is missing a header file.
  • useless file in the current source package cctbx_sources/lapack_fem/xerbla.f.orig
  • what should we do with all the dispatcher in the bin directory ?

http://cci.lbl.gov/~hohn/scitbx-tour.html#Installation