Differences between revisions 2 and 3
Revision 2 as of 2012-02-07 22:51:40
Size: 12848
Editor: BarryWarsaw
Comment:
Revision 3 as of 2012-02-08 20:14:36
Size: 12844
Comment: drop "a " drom package's short description
Deletions are marked like this. Additions are marked like this.
Line 38: Line 38:
Description: a Python frobnicator (Python 2) Description: Python frobnicator (Python 2)
Line 48: Line 48:
Description: a Python frobnicator (Python 3) Description: Python frobnicator (Python 3)

Style Guide for Packaging Python Libraries

NOTE: This is a draft proposal. It has not yet been vetted with debian-python. Comments are welcome and encouraged.

The purpose of this document is to provide a quick style guide for packaging new Python libraries, or adding Python 3 support to existing packages (assuming the upstream supports Python 3). It is not intended to supplant the Debian Python Policy and in fact, if you have not read that document, stop now and go read it. Instead, this page captures in example form, best practices for simple Python libraries that you might find on the Python Package Index. A few assumptions about the Python package your working on are being made:

  • The package is setuptools/distribute based. It should have a working setup.py or setup.cfg file and should generally be installable without modification into a virtualenv.

  • Pure-Python or simple extension modules only. If you can do python setup.py install you'll be fine. For more complex building instructions, you will have to adapt these guidelines.

  • The package supports Python 2.6 at least, preferably also 2.7 and 3.2.
  • The package has a test suite, invoked via python setup.py test.

  • The package has some documentation buildable by Sphinx.
  • Your Debian packaging uses debhelper, ?dh_python2, and is preferably source format 3.0 (quilt).

  • debhelper v8 or newer is assumed, but v7 probably works too (not sure about earlier ones, anyway, update your package!)

The design of this document is to capture best-practices that you can cargo cult into your own packages. It will forever be a work in progress.

debian/control

One very important thing to get right is the Build-Depends line in the source package stanza. setuptools/distribute-based have the nasty habit of downloading packages from PyPI if they are needed at python setup.py build time. If the package is available from the system (as would be the case when Build-Depends is up-to-date), then distribute will not try to download the package, but there appears to be no way to prevent distribute explicitly from downloading the package. Some build environments may block that in other ways (e.g. disabling the network), but if not, then the build could appear to succeed, when in fact it's only satisfying its setup.py dependencies external to the archive. This is a big no no.

dh_python2 will correctly fill in the installation dependencies (via ${python:Depends} and ${python3:Depends} lines, but it cannot fill in the build dependencies. Take extra care in getting this right, and double check your build logs for illegal access to pypi.python.org.

Start with a fairly straightforward debian/control file. If upstream supports Python 3, you'll want both of these lines in the source package stanza:

X-Python-Version: >= 2.6
X-Python3-Version: >= 3.2

If the d/control file already support a Python 2 version of the package, but you now want to add Python 3 support, copy the Python 2 binary package stanza and paste it below for a Python 3 version. Change the binary package name prefix from python- to python3-. Change the Depends line in the new stanza from including ${python:Depends} to ${python3:Depends}. Change the short description of the stanzas to make it clear which binary package is the Python 2 version and which is the Python 3 version. Here is an example of the two stanzas you might have:

Package: python-foo
Architecture: all
Depends: ${python:Depends}, ${misc:Depends}
Suggests: python-foo-docs
Description: Python frobnicator (Python 2)
 This package frobnicates a Python-based doohicky so that you no longer
 need to kludge a dingus to make the doodads work.
 .
 This is the Python 2 version of the package.

Package: python3-foo
Architecture: all
Depends: ${python3:Depends}, ${misc:Depends}
Suggests: python-foo-docs
Description: Python frobnicator (Python 3)
 This package frobnicates a Python-based doohicky so that you no longer
 need to kludge a dingus to make the doodads work.
 .
 This is the Python 3 version of the package.

Notice the Suggests line. If the same set of documentation is available for both Python 2 and Python 3, it's best to put those files into a separate python-foo-docs package, like so:

Package: python-foo-docs
Architecture: all
Section: doc
Depends: ${sphinxdoc:Depends}, ${misc:Depends}
Description: a Python frobnicator (common documentation)
 This package frobnicates a Python-based doohicky so that you no longer
 need to kludge a dingus to make the doodads work.
 .
 This is the common documentation package.

debian/rules

Here is a debian/rules file for you to start with. First, I'll show you the whole thing, then I'll explain it line by line.

#DH_VERBOSE=1

PYTHON2=$(shell pyversions -vr)
PYTHON3=$(shell py3versions -vr)

%:
        dh $@ --with python2,python3,sphinxdoc

ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
test-python%:
        python$* setup.py test -vv

override_dh_auto_test: $(PYTHON2:%=test-python%) $(PYTHON3:%=test-python%)
endif

build-python%:
        python$* setup.py build

override_dh_auto_build: $(PYTHON3:%=build-python%)
        dh_auto_build

install-python%:
        python$* setup.py install --root=$(CURDIR)/debian/tmp --install-layout=deb

override_dh_auto_install: $(PYTHON3:%=install-python%)
        dh_auto_install

override_dh_installchangelogs:
        dh_installchangelogs -k foo/NEWS.rst

override_dh_installdocs:
        python setup.py build_sphinx
        dh_installdocs build/sphinx/html

override_dh_auto_clean:
        dh_auto_clean
        rm -rf build
        rm -rf *.egg-info

The file starts with the standard #! line. I like adding the (commented out when uploading) DH_VERBOSE line because it can make build problems easier to debug.

The next two lines are:

PYTHON2=$(shell pyversions -vr)
PYTHON3=$(shell py3versions -vr)

This gives you make variables which will contain (just) the version numbers for the Python versions you are requesting your package be built against. See the manpage for details, but essentially this consults the X-Python-Version header in your d/control file and matches them against the Python versions available for your distroversion. For example, on Wheezy, this would return 2.6 2.7.

py3versions does the same, but for Python 3, and it uses the X-Python3-Version header in debian/control. If upstream does not yet support Python 3, you can still leave this line enabled, or comment it out, but you will have to adjust some of the lines that follow.

The next line is the standard debhelper-based catch-all rule which is used to get the whole build started:

%:
        dh $@ --with python2,python3,sphinxdoc

What's important to note here is that both of the dh_python2 and dh_python3 helpers are being invoked, as well as the Sphinx documentation helper. If upstream does not yet support Python 3, omit the python3 helper for now. If upstream doesn't have Sphinx-based documentation, omit the sphinxdoc helper for now.

The next few lines enable the test suites to be run for each version of the package that gets built. If upstream has a test suite that passes on Debian, then it's a very good idea to enable it in your build. Testing is always a good thing, and in general (although there are exceptions), you want your build to fail if the test suite fails.

ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
test-python%:
        python$* setup.py test -vv

override_dh_auto_test: $(PYTHON2:%=test-python%) $(PYTHON3:%=test-python%)
endif

Things to note here.

  • The make conditional wrapping the rules allows you to set DEB_BUILD_OPTIONS=nocheck to temporarily disable the test suite. This can be useful while your debugging your build if the test suite takes a long time to run.

  • You will need to change the rule's command if the package's test suite is invoked in a way other than python setup.py test. The way it is above enables a higher level of verbose output, which again is useful for general diagnostics.

  • If upstream doesn't yet support Python 3, you might need to adjust the dependencies in the override_dh_auto_test rule.

These next few lines build the package:

build-python%:
        python$* setup.py build

override_dh_auto_build: $(PYTHON3:%=build-python%)
        dh_auto_build

Note that debhelper will automatically build the Python 2 version, but does not yet know how to build Python 3 packages. Thus, the override is necessary to add dependencies that trigger the Python 3 build (first rule above). In the body of the override, the standard dh_auto_build is invoked, which will properly build the Python 2 version of the package. If upstream doesn't yet support Python 3, you can comment out both of these rules and just let the normal dh_auto_build process build the Python 2 version.

The next few lines install the package for all supported versions of Python.

install-python%:
        python$* setup.py install --root=$(CURDIR)/debian/tmp --install-layout=deb

override_dh_auto_install: $(PYTHON3:%=install-python%)
        dh_auto_install

Again, this is only necessary because dh doesn't know how to install the Python 3 versions by default. It works in a similar manner as the build rules above. Like above, you can comment out all of these lines (or omit them) if upstream does not yet support Python 3.

Also note that --install-layout=deb is a Debian-only argument for setup.py. It was added to python-distribute to support Debian's dist-packages convention.

The next lines are useful if upstream has a non-standard change log that needs to be installed. In this case, the foo package has a news file that serves as its log of changes. Consult the dh_installchangelogs manpage for details about whether you need this override or not.

override_dh_installchangelogs:
        dh_installchangelogs -k flufl/enum/NEWS.rst

If upstream has documentation that's built by Sphinx, these next few lines will build and install them for the separate python-foo-docs package mentioned above.

override_dh_installdocs:
        python setup.py build_sphinx
        dh_installdocs build/sphinx/html

Finally, you should ensure that your package can be built twice in a row, by properly cleaning up any artifacts of your build. By default dh_clean will do most of the work, but Python packages need a little extra help, thus this override.

override_dh_auto_clean:
        dh_auto_clean
        rm -rf build
        rm -rf *.egg-info

Some packages produce other artifacts, or modify files in the upstream original tarball. Those are harder to clean and may produce build failures when built twice in a row.

debian/*.install files

When the same source is used to build three different binary packages (e.g. the Python 2 version, the Python 3 version, and the common documentation package), you will need debian/*.install}} files to get everything in the proper place for packaging.  For most simple Python packages, this is fairly easy.  You will not need an {{{.install file for the documentation (TBD: explain why this works automatically), but you will for the Python 2 and Python 3 code.

Here is the debian/python-foo.install file (i.e. the Python 2 version:

usr/lib/python2*

and here is the debian/python3-foo.install file:

usr/lib/python3

Usually, that is all you need.

You may or may not need this file. It can be useful for Sphinx-based documentation, by providing the original reST files used to build the HTML documentation. Here is an example debian/python-foo.links file (it does not need a Python 3 version).

usr/share/doc/python-foo/html/_sources usr/share/doc/python-foo/rst

debian/*.pyremove

Upstream source may install some files that you don't care about including in your Debian packages. The way to handle this is by adding a debian/python-foo.pyremove file. Again, this does not need a Python 3 version. For example:

foo/conf.py
foo/README.txt
foo/NEWS.txt
foo*.egg-info/SOURCES.txt

TBD: better explanation of the rationale for removing these.