Style Guide for Packaging Python Libraries

NOTE: Also see Python/AppStyleGuide for alternative single package debian/rules.

This is a wiki page so of course you are free to edit it. However, if you are making changes in recommendations, it is requested that you first contact the debian-python mailing list to discuss your changes, unless you're sure they are uncontroversial (e.g. fixing typos). Thanks!

The purpose of this document is to provide a quick style guide for packaging new Python libraries, or adding Python 3 support to existing packages (assuming that upstream supports it too). It is not intended to supplant the Debian Python Policy and in fact, if you have not read that document, stop now and go read it. Instead, this page captures in example form, best practices for simple Python libraries that you might find on the Python Package Index. A few assumptions about the Python package you are working on are being made:

For packages which already support Python 2 and want to add Python 3 support, and/or modernize the packaging as described in this page, it's easiest to do so if your package already uses debhelper (v8 or newer), dh_python2 explicitly, and source format 3.0 (quilt).

The design of this document is to capture best-practices that you can cargo cult into your own packages. It will forever be a work in progress.

Overview

There are many ways to build Python libraries for Debian, and each package presents its own corner cases and challenges. In modern Debian (e.g. unstable as of November 2013) and derivatives, the easiest way to build Python library packages is with the pybuild debhelper build system. This link has details on pybuild, including more examples for your cargo culting pleasure.

If you can't use pybuild, say because you want to be able to backport your package to distribution versions before pybuild was introduced, you may be interested in the historical version of this document, which contains all the gory details you had to implement manually, but which pybuild now takes care of for you.

The examples below are taken from the enum34 package, but modified for generality.

Note: If you are building a Python 3-only package (i.e. there's no Python 2 support) then it is highly recommended you explicitly set the dh build system to pybuild. By default, if there is a setup.py, dh will set the build system to python_distutils which only supports Python 2 and you'll get errors during package build as it tries to invoke some nonexistent Python 2 tools.

debian/control

Build-Depends

One very important thing to get right is the Build-Depends line in the source package stanza. setuptools/distribute-based packages have the nasty habit of downloading dependencies from PyPI if they are needed at python setup.py build time. If the package is available from the system (as would be the case when Build-Depends is up-to-date), then distribute will not try to download the package, otherwise it will try to download it. This is a huge no-no, and pybuild internally sets the http_proxy and https_proxy environment variables (to 127.0.0.1:9) to prevent this from happening.

dh_python2 and dh_python3 will correctly fill in the installation dependencies (via ${python:Depends} and ${python3:Depends} respectively), but it cannot fill in the build dependencies. Take extra care in getting this right, and double check your build logs for illegal access to pypi.python.org.

You'll want to have at least the following build dependencies:

If the upstream package supports Python 3, you'll need these dependencies to also build a Python 3 version:

If the package has documentation in reStructuredText format, which often uses Sphinx as well, add these dependencies:

Python versions

If upstream supports Python 3, you'll want both of these lines in the source package stanza:

X-Python-Version: >= 2.6
X-Python3-Version: >= 3.2

If the d/control file already support a Python 2 version of the package, but you now want to add Python 3 support, copy the Python 2 binary package stanza and paste it below for a Python 3 version. Change the binary package name prefix from python- to python3-. Change the Depends line in the new stanza from including ${python:Depends} to ${python3:Depends}. Change the short description of the stanzas to make it clear which binary package is the Python 2 version and which is the Python 3 version. Here is an example of the two stanzas you might have:

Package: python-foo
Architecture: all
Depends: ${python:Depends}, ${misc:Depends}
Suggests: python-foo-doc
Description: Python frobnicator (Python 2)
 This package frobnicates a Python-based doohicky so that you no longer
 need to kludge a dingus to make the doodads work.
 .
 This package installs the library for Python 2.

Package: python3-foo
Architecture: all
Depends: ${python3:Depends}, ${misc:Depends}
Suggests: python-foo-doc
Description: Python frobnicator (Python 3)
 This package frobnicates a Python-based doohicky so that you no longer
 need to kludge a dingus to make the doodads work.
 .
 This package installs the library for Python 3.

Documentation package

Usually the same upstream documentation will serve equally well for users of the Python 2 and Python 3 runtime packages. Notice the Suggests line above. If this is the case, it's best to put those files into a separate python-foo-doc package, like so:

Package: python-foo-doc
Architecture: all
Section: doc
Depends: ${sphinxdoc:Depends}, ${misc:Depends}
Description: Python frobnicator (common documentation)
 This package frobnicates a Python-based doohicky so that you no longer
 need to kludge a dingus to make the doodads work.
 .
 This is the common documentation package.

debian/rules

The contents of this file depends on whatever build system you choose, however we highly recommend the pybuild build system for its simplicity (at least for setup-based packages, i.e. the majority of those on PyPI) and its great support for dual Python 2 and Python 3 builds.

Here is a debian/rules file for you to start with. First, I'll show you the whole thing, then I'll explain it line by line.

#export DH_VERBOSE = 1
export PYBUILD_NAME = foo

%:
        dh $@ --with python2,python3 --buildsystem=pybuild

The file starts with the standard #! line. I like adding the (commented out when uploading) DH_VERBOSE line because it can make build problems easier to debug.

The next line is:

export PYBUILD_NAME = foo

pybuild supports a number of variables which can be used to control such things as the destination directory for build artifacts for both the Python 2 and Python 3 builds. The defaults generally do the right thing, but you do need to at least tell pybuild the name of your package. Here, we're telling it that the name is foo. This should match the module name, so for example, in enum34, you'd see:

export PYBUILD_NAME = enum34

even though the binary packages that get produced are python-enum34 and python3-enum34.

The next line is the standard debhelper-based catch-all rule which is used to get the whole build started:

%:
        dh $@ --with python2,python3 --buildsystem=pybuild

What's important to note here is that both of the dh_python2 and dh_python3 helpers are being invoked, and also that the build system that dh will use is pybuild. There's a lot of magic packed into this line, and the pybuild manpage goes into more detail, but essentially what this does is:

Once all that's done, it properly installs all the files into binary packages.

If upstream does not yet support Python 3, omit the python3 helper for now.

And that's it! You usually won't need debian/python-foo.install or debian/python3-foo.install files even if you have multiple binary packages, because again, pybuild does the magic for you. You might need these if there are some non-standard installation tricks you need to implement.

You may or may not need this file. It can be useful for Sphinx-based documentation, by providing the original reST files used to build the HTML documentation. Here is an example debian/python-foo.links file (it does not need a Python 3 version).

usr/share/doc/python-foo/html/_sources usr/share/doc/python-foo/rst

Sphinx documentation

If your package has Sphinx documentation, add the sphinxdoc helper like so:

%:
        dh $@ --with python2,python3,sphinxdoc --buildsystem=pybuild

If you need to build the Sphinx documentation (usually from .rst or .md files), add:

override_dh_auto_build:
    dh_auto_build
    PYTHONPATH=. http_proxy='127.0.0.1:9' sphinx-build -N -bhtml docs/ build/html # HTML generator
    #PYTHONPATH=. http_proxy='127.0.0.1:9' sphinx-build -N -bman docs/ build/man # Manpage generator

sphinx-build might try to access the Internet to fetch intersphinx inventory files; http_proxy set to 127.0.0.1:9 will prevent that.

debian/watch

Make sure that you have a working debian/watch file, for referencing upstream distribution tarballs. Usually, uscan is sufficient to download tarballs from the declarations in debian/watch (or, if you're using svn-buildpackage, this can be done automatically with svn-buildpackage --svn-download-orig), but if the source needs to be repackaged, you should then provide a get-orig-source (policy) target in debian/rules which does the job.

The URLs many existing packages have been using are, or will soon, be broken. PyPI changed the way index listings for package directories work and in fact, these will disappear at some point in the future. Thus, it is highly recommended that you update your existing packages to the new debian/watch format described here.

Since probably for most of our packages, the original tarball comes from the Python Package Index (a.k.a. PyPI or the Cheeseshop), here is a sample debian/watch file that you might be able to adapt for your own package:

version=3
opts=uversionmangle=s/(rc|a|b|c)/~$1/ \
http://pypi.debian.net/mypackage/mypackage-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

Notice the use of pypi.debian.net instead of pypi.python.org. We're now running a redirector for PyPI to adjust for changes in the format of package listings on the latter site. In fact, it's trivial to get a valid debian/watch file for your package. Just do this from the parent of your debian directory:

$ curl -o debian/watch http://pypi.debian.net/mypackage/watch

Overriding

If you need to override the dh_python2/3 defaults, do it with an override method in your debian/rules file. Something like this will work:

override_dh_python3:
        dh_python3 --shebang=/usr/bin/python3

If you need to override the default testing regime (e.g. to use py.test), you can do it one of two ways. Either:

export PYBUILD_TEST_PYTEST = 1

or you can override the dh_auto_test rule:

override_dh_auto_test:
        dh_auto_test -- --test-pytest

Gotchas

Python 3.3/3.4 unittest fixers for 2to3

Python 3.4 added a new 2to3 fixer which rewrites deprecated unittest assert methods to the current preferred names. If your upstream package uses 2to3 to provide a Python 3 compatible version, and uses these deprecated assert names, what this means is that the Python 3.3 and Python 3.4 version of those files will differ. dh_python3 will rightly refuse to copy files to the common location (i.e. /usr/lib/python3/dist-packages/<pkg>) when the files differ, so you'll end up with Python 3.4 specific paths in your python3- binary package in addition to the common location. lintian will complain.

There are a few ways to solve this, and maintainers may have their own preferences:

/genshi/* 3.4
/Genshi*.egg-info/* 3.4

override_dh_python3:
        dh_python3
        rm -rf debian/python3-foo/usr/lib/python3.?

MANIFEST.in issues and missing files

If the upstream package requires some non-standard files to be installed, it must include a MANIFEST.in file. Sometimes these files are omitted from the original tarball because they're mostly useful for creating the sdist. Case in point: a package contained a bunch of doctests in .rst files but python setup.py sdist does not include .rst files by default. Thus, when pybuild went to build the package locally, the .rst files were omitted and the test suite failed because it could not find the doctest.

The solution for this is to either ensure that upstream includes the MANIFEST.in file, or that you add one before pybuild runs.

TODO