Style Guide for Packaging Python Libraries

NOTE: This is a draft proposal. Comments are welcome and encouraged.

NOTE: Also see Python/AppStyleGuide for alternative single package debian/rules.

The purpose of this document is to provide a quick style guide for packaging new Python libraries, or adding Python 3 support to existing packages (assuming that upstream supports it too). It is not intended to supplant the Debian Python Policy and in fact, if you have not read that document, stop now and go read it. Instead, this page captures in example form, best practices for simple Python libraries that you might find on the Python Package Index. A few assumptions about the Python package you are working on are being made:

The design of this document is to capture best-practices that you can cargo cult into your own packages. It will forever be a work in progress.

debian/control

One very important thing to get right is the Build-Depends line in the source package stanza. setuptools/distribute-based packages have the nasty habit of downloading dependencies from PyPI if they are needed at python setup.py build time. If the package is available from the system (as would be the case when Build-Depends is up-to-date), then distribute will not try to download the package, otherwise it will try to download it. This is a huge no-no, so you need to watch carefully to make sure your local builds don't do this, and be sure to add the http_proxy line in your debian/rules file (see below).

dh_python2 and dh_python3 will correctly fill in the installation dependencies (via ${python:Depends} and ${python3:Depends} respectively), but it cannot fill in the build dependencies. Take extra care in getting this right, and double check your build logs for illegal access to pypi.python.org.

Start with a fairly straightforward debian/control file. If upstream supports Python 3, you'll want both of these lines in the source package stanza:

X-Python-Version: >= 2.6
X-Python3-Version: >= 3.2

If the d/control file already support a Python 2 version of the package, but you now want to add Python 3 support, copy the Python 2 binary package stanza and paste it below for a Python 3 version. Change the binary package name prefix from python- to python3-. Change the Depends line in the new stanza from including ${python:Depends} to ${python3:Depends}. Change the short description of the stanzas to make it clear which binary package is the Python 2 version and which is the Python 3 version. Here is an example of the two stanzas you might have:

Package: python-foo
Architecture: all
Depends: ${python:Depends}, ${misc:Depends}
Suggests: python-foo-docs
Description: Python frobnicator (Python 2)
 This package frobnicates a Python-based doohicky so that you no longer
 need to kludge a dingus to make the doodads work.
 .
 This is the Python 2 version of the package.

Package: python3-foo
Architecture: all
Depends: ${python3:Depends}, ${misc:Depends}
Suggests: python-foo-docs
Description: Python frobnicator (Python 3)
 This package frobnicates a Python-based doohicky so that you no longer
 need to kludge a dingus to make the doodads work.
 .
 This is the Python 3 version of the package.

Notice the Suggests line. If the same set of documentation is available for both Python 2 and Python 3, it's best to put those files into a separate python-foo-docs package, like so:

Package: python-foo-docs
Architecture: all
Section: doc
Depends: ${sphinxdoc:Depends}, ${misc:Depends}
Description: Python frobnicator (common documentation)
 This package frobnicates a Python-based doohicky so that you no longer
 need to kludge a dingus to make the doodads work.
 .
 This is the common documentation package.

debian/rules

Here is a debian/rules file for you to start with. First, I'll show you the whole thing, then I'll explain it line by line.

#DH_VERBOSE=1

PYTHON2_VERSIONS=$(shell pyversions -vr)
PYTHON3_VERSIONS=$(shell py3versions -vr)
PYTHON_VERSIONS=${PYTHON2_VERSIONS} ${PYTHON3_VERSIONS}

# Prevent setuptools/distribute from accessing the internet.
export http_proxy = http://127.0.0.1:9

%:
        dh $@ --with python2,python3,sphinxdoc

ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
test-python%:
        python$* setup.py test -vv

override_dh_auto_test: $(foreach pyversion,${PYTHON_VERSIONS},$(pyversion:%=test-python%))
endif

build-python%:
        python$* setup.py build

override_dh_auto_build: $(PYTHON3:%=build-python%)
        dh_auto_build

install-python%:
        python$* setup.py install --root=$(CURDIR)/debian/tmp --install-layout=deb

override_dh_auto_install: $(PYTHON3:%=install-python%)
        dh_auto_install

override_dh_installchangelogs:
        dh_installchangelogs -k foo/NEWS.rst

override_dh_installdocs:
        python setup.py build_sphinx
        dh_installdocs build/sphinx/html

override_dh_auto_clean:
        dh_auto_clean
        rm -rf build
        rm -rf *.egg-info

The file starts with the standard #! line. I like adding the (commented out when uploading) DH_VERBOSE line because it can make build problems easier to debug.

The next three lines are:

PYTHON2_VERSIONS=$(shell pyversions -vr)
PYTHON3_VERSIONS=$(shell py3versions -vr)
PYTHON_VERSIONS=${PYTHON2_VERSIONS} ${PYTHON3_VERSIONS}

This gives you make variables which will contain (just) the version numbers for the Python versions you are requesting your package be built against. See the manpage for details, but essentially this consults the X-Python-Version field in your debian/control file and matches it against the Python versions available for your distroversion. For example, on Wheezy, PYTHON2_VERSIONS would contain 2.6 2.7.

py3versions does the same, but for Python 3, and it uses the X-Python3-Version field in debian/control. For example, on Wheezy, PYTHON3_VERSIONS would contain 3.2. If upstream does not yet support Python 3, you can still leave this line enabled, or comment it out, but you will have to adjust some of the lines that follow.

The next line is a safety valve:

# Prevent setuptools/distribute from accessing the internet.
export http_proxy = http://127.0.0.1:9

setuptools-based setups have the annoying -- and buggy -- behavior of trying to download dependencies from PyPI when they aren't locally available. Some build environments block this (e.g. by disabling the network), but if not, then the build could appear to succeed, when in fact it's only satisfying its setup.py dependencies external to the archive. To prevent this even in local builds, where downloading may succeed, add export http_proxy = http://127.0.0.1:9/ to your debian/rules file. Port 9 is the Discard Protocol, so this should safely prevent downloading even if something is actually listening on the port. Now if you are missing a dependency, even your local builds will fail early, which is a good thing!

The next line is the standard debhelper-based catch-all rule which is used to get the whole build started:

%:
        dh $@ --with python2,python3,sphinxdoc

What's important to note here is that both of the dh_python2 and dh_python3 helpers are being invoked, as well as the Sphinx documentation helper. If upstream does not yet support Python 3, omit the python3 helper for now. If upstream doesn't have Sphinx-based documentation, omit the sphinxdoc helper for now.

The next few lines enable the test suites to be run for each version of the package that gets built. If upstream has a test suite that passes on Debian, then it's a very good idea to enable it in your build. Testing is always a good thing, and in general (although there are exceptions), you want your build to fail if the test suite fails.

ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
test-python%:
        python$* setup.py test -vv

override_dh_auto_test:  $(foreach pyversion,${PYTHON_VERSIONS},$(pyversion:%=test-python%))
endif

Things to note here.

These next few lines build the package:

build-python%:
        python$* setup.py build

override_dh_auto_build: $(PYTHON3:%=build-python%)
        dh_auto_build

Note that debhelper will automatically build the Python 2 version, but does not yet know how to build Python 3 packages. Thus, the override is necessary to add dependencies that trigger the Python 3 build (first rule above). In the body of the override, the standard dh_auto_build is invoked, which will properly build the Python 2 version of the package. If upstream doesn't yet support Python 3, you can comment out both of these rules and just let the normal dh_auto_build process build the Python 2 version.

The next few lines install the package for all supported versions of Python.

install-python%:
        python$* setup.py install --root=$(CURDIR)/debian/tmp --install-layout=deb

override_dh_auto_install: $(PYTHON3:%=install-python%)
        dh_auto_install

Again, this is only necessary because dh doesn't know how to install the Python 3 versions by default. It works in a similar manner as the build rules above. Like above, you can comment out all of these lines (or omit them) if upstream does not yet support Python 3.

Also note that --install-layout=deb is a Debian-only argument for setup.py. It was added to python-distribute to support Debian's dist-packages convention.

The next lines are useful if upstream has a non-standard change log that needs to be installed. In this case, the foo package has a news file that serves as its log of changes. Consult the dh_installchangelogs manpage for details about whether you need this override or not.

override_dh_installchangelogs:
        dh_installchangelogs -k foo/NEWS.rst

If upstream has documentation that's built by Sphinx, these next few lines will build and install them for the separate python-foo-docs package mentioned above.

override_dh_installdocs:
        python setup.py build_sphinx
        dh_installdocs build/sphinx/html

Finally, you should ensure that your package can be built twice in a row, by properly cleaning up any artifacts of your build. By default dh_clean will do most of the work, but Python packages need a little extra help, thus this override.

override_dh_auto_clean:
        dh_auto_clean
        rm -rf build
        rm -rf *.egg-info

Some packages produce other artifacts, or modify files in the upstream original tarball. Those are harder to clean and may produce build failures when built twice in a row.

debian/*.install files

When the same source is used to build three different binary packages (e.g. the Python 2 version, the Python 3 version, and the common documentation package), you will need debian/*.install files to get everything in the proper place for packaging. For most simple Python packages, this is fairly easy. You will not need an .install file for the documentation (TBD: explain why this works automatically), but you will for the Python 2 and Python 3 code.

Here is the debian/python-foo.install file (i.e. the Python 2 version):

usr/lib/python2*

and here is the debian/python3-foo.install file:

usr/lib/python3

Usually, that is all you need.

You may or may not need this file. It can be useful for Sphinx-based documentation, by providing the original reST files used to build the HTML documentation. Here is an example debian/python-foo.links file (it does not need a Python 3 version).

usr/share/doc/python-foo/html/_sources usr/share/doc/python-foo/rst

debian/*.pyremove

Upstream source may install some files that you don't care about including in your Debian packages. The way to handle this is by adding a debian/python-foo.pyremove file. This should work for both Python 2 and Python 3. For example:

foo/conf.py
foo/README.rst
foo/NEWS.rst
foo*.egg-info/SOURCES.txt

TBD: better explanation of the rationale for removing these.

TODO