Differences between revisions 52 and 53
Revision 52 as of 2014-06-16 20:01:05
Size: 14495
Editor: ArthurDeJong
Comment: -doc is a much more common suffix than -docs
Revision 53 as of 2015-01-09 06:43:33
Size: 14483
Editor: BenFinney
Comment: improve spacing based on makefile syntax
Deletions are marked like this. Additions are marked like this.
Line 105: Line 105:
#!/usr/bin/make -f

#export DH_VERBOSE=1
export PYBUILD_NAME=foo
#! /usr/bin/make -f

#export DH_VERBOSE = 1
export PYBUILD_NAME = foo
Line 119: Line 119:
export PYBUILD_NAME=foo export PYBUILD_NAME = foo
Line 125: Line 125:
export PYBUILD_NAME=enum34 export PYBUILD_NAME = enum34
Line 205: Line 205:
export PYBUILD_TEST_PYTEST=1 export PYBUILD_TEST_PYTEST = 1
Line 211: Line 211:
Line 213: Line 212:
        dh_auto_test -- --test-pytest  dh_auto_test -- --test-pytest
Line 236: Line 235:
        dh_python3
        rm -rf debian/python3-foo/usr/lib/python3.?
 dh_python3
 rm -rf debian/python3-foo/usr/lib/python3.?

Style Guide for Packaging Python Libraries

NOTE: Also see Python/AppStyleGuide for alternative single package debian/rules.

The purpose of this document is to provide a quick style guide for packaging new Python libraries, or adding Python 3 support to existing packages (assuming that upstream supports it too). It is not intended to supplant the Debian Python Policy and in fact, if you have not read that document, stop now and go read it. Instead, this page captures in example form, best practices for simple Python libraries that you might find on the Python Package Index. A few assumptions about the Python package you are working on are being made:

  • The package is setuptools/distribute based. It should have a working setup.py (or for >= Python 3.3 setup.cfg) file and should generally be installable without modification into a virtualenv.

  • Pure-Python or simple extension modules only. If you can do python setup.py install you'll be fine. For more complex building instructions, you will have to adapt these guidelines.

  • The package supports at least Python 2.6.
  • The package has a test suite.
  • The package has some Sphinx-based documentation.

For packages which already support Python 2 and want to add Python 3 support, and/or modernize the packaging as described in this page, it's easiest to do so if your package already uses debhelper (v8 or newer), dh_python2 explicitly, and source format 3.0 (quilt).

The design of this document is to capture best-practices that you can cargo cult into your own packages. It will forever be a work in progress.

Overview

There are many ways to build Python libraries for Debian, and each package presents its own corner cases and challenges. In modern Debian (e.g. unstable as of November 2013) and derivatives, the easiest way to build Python library packages is with the pybuild debhelper build system. This link has details on pybuild, including more examples for your cargo culting pleasure.

If you can't use pybuild, say because you want to be able to backport your package to distribution versions before pybuild was introduced, you may be interested in the historical version of this document, which contains all the gory details you had to implement manually, but which pybuild now takes care of for you.

The examples below are taken from the enum34 package, but modified for generality.

Note: If you are building a Python 3-only package (i.e. there's no Python 2 support) then it is highly recommended you explicitly set the dh build system to pybuild. By default, if there is a setup.py, dh will set the build system to python_distutils which only supports Python 2 and you'll get errors during package build as it tries to invoke some nonexistent Python 2 tools.

debian/control

One very important thing to get right is the Build-Depends line in the source package stanza. setuptools/distribute-based packages have the nasty habit of downloading dependencies from PyPI if they are needed at python setup.py build time. If the package is available from the system (as would be the case when Build-Depends is up-to-date), then distribute will not try to download the package, otherwise it will try to download it. This is a huge no-no, and pybuild internally sets the http_proxy and https_proxy environment variables (to 127.0.0.1:9) to prevent this from happening.

dh_python2 and dh_python3 will correctly fill in the installation dependencies (via ${python:Depends} and ${python3:Depends} respectively), but it cannot fill in the build dependencies. Take extra care in getting this right, and double check your build logs for illegal access to pypi.python.org.

You'll want to have at least the following build dependencies:

  • debhelper (>= 8)

  • dh-python
  • python-all (>= 2.6.6-3~)

  • python-setuptools
  • python3-all
  • python3-setuptools

The python3-all dependency is necessary if you're also going to build a Python 3 version, but you can drop that if upstream doesn't support Python 3 yet.

Some other build dependencies you might want:

  • python-docutils
  • python-sphinx (>= 1.0.7+dfsg-1~)

These are useful if your package has Sphinx-based documentation.

If upstream supports Python 3, you'll want both of these lines in the source package stanza:

X-Python-Version: >= 2.6
X-Python3-Version: >= 3.2

If the d/control file already support a Python 2 version of the package, but you now want to add Python 3 support, copy the Python 2 binary package stanza and paste it below for a Python 3 version. Change the binary package name prefix from python- to python3-. Change the Depends line in the new stanza from including ${python:Depends} to ${python3:Depends}. Change the short description of the stanzas to make it clear which binary package is the Python 2 version and which is the Python 3 version. Here is an example of the two stanzas you might have:

Package: python-foo
Architecture: all
Depends: ${python:Depends}, ${misc:Depends}
Suggests: python-foo-doc
Description: Python frobnicator (Python 2)
 This package frobnicates a Python-based doohicky so that you no longer
 need to kludge a dingus to make the doodads work.
 .
 This is the Python 2 version of the package.

Package: python3-foo
Architecture: all
Depends: ${python3:Depends}, ${misc:Depends}
Suggests: python-foo-doc
Description: Python frobnicator (Python 3)
 This package frobnicates a Python-based doohicky so that you no longer
 need to kludge a dingus to make the doodads work.
 .
 This is the Python 3 version of the package.

Notice the Suggests line. If the same set of documentation is available for both Python 2 and Python 3, it's best to put those files into a separate python-foo-doc package, like so:

Package: python-foo-doc
Architecture: all
Section: doc
Depends: ${sphinxdoc:Depends}, ${misc:Depends}
Description: Python frobnicator (common documentation)
 This package frobnicates a Python-based doohicky so that you no longer
 need to kludge a dingus to make the doodads work.
 .
 This is the common documentation package.

debian/rules

The contents of this file depends on whatever build system you choose, however we highly recommend the pybuild build system for its simplicity (at least for setup-based packages, i.e. the majority of those on PyPI) and its great support for dual Python 2 and Python 3 builds.

Here is a debian/rules file for you to start with. First, I'll show you the whole thing, then I'll explain it line by line.

#export DH_VERBOSE = 1
export PYBUILD_NAME = foo

%:
        dh $@ --with python2,python3 --buildsystem=pybuild

The file starts with the standard #! line. I like adding the (commented out when uploading) DH_VERBOSE line because it can make build problems easier to debug.

The next line is:

export PYBUILD_NAME = foo

pybuild supports a number of variables which can be used to control such things as the destination directory for build artifacts for both the Python 2 and Python 3 builds. The defaults generally do the right thing, but you do need to at least tell pybuild the name of your package. Here, we're telling it that the name is foo. This should match the module name, so for example, in enum34, you'd see:

export PYBUILD_NAME = enum34

even though the binary packages that get produced are python-enum34 and python3-enum34.

The next line is the standard debhelper-based catch-all rule which is used to get the whole build started:

%:
        dh $@ --with python2,python3 --buildsystem=pybuild

What's important to note here is that both of the dh_python2 and dh_python3 helpers are being invoked, and also that the build system that dh will use is pybuild. There's a lot of magic packed into this line, and the pybuild manpage goes into more detail, but essentially what this does is:

  • Builds the Python 2 package for all supported Python 2 versions.
  • Builds the Python 3 package for all supported Python 3 versions.
  • Automatically detects the command to run the test suite.
  • Runs the Python 2 test suite.
  • Runs the Python 3 test suite.

Once all that's done, it properly installs all the files into binary packages.

If upstream does not yet support Python 3, omit the python3 helper for now.

And that's it! You usually won't need debian/python-foo.install or debian/python3-foo.install files even if you have multiple binary packages, because again, pybuild does the magic for you. You might need these if there are some non-standard installation tricks you need to implement.

You may or may not need this file. It can be useful for Sphinx-based documentation, by providing the original reST files used to build the HTML documentation. Here is an example debian/python-foo.links file (it does not need a Python 3 version).

usr/share/doc/python-foo/html/_sources usr/share/doc/python-foo/rst

Sphinx documentation

If your package has Sphinx documentation, add the sphinxdoc helper like so:

%:
        dh $@ --with python2,python3,sphinxdoc --buildsystem=pybuild

If you need to build the Sphinx documentation (usually from .rst or .md files), add:

override_dh_auto_build:
    dh_auto_build
    PYTHONPATH=. http_proxy='127.0.0.1:9' sphinx-build -N -bhtml docs/ build/html # HTML generator
    #PYTHONPATH=. http_proxy='127.0.0.1:9' sphinx-build -N -bman docs/ build/man # Manpage generator

sphinx-build might try to access the Internet to fetch intersphinx inventory files; http_proxy set to 127.0.0.1:9 will prevent that.

debian/watch

Make sure that you have a working debian/watch file, for referencing upstream distribution tarballs. Usually, uscan is sufficient to download tarballs from the declarations in debian/watch (or, if you're using svn-buildpackage, this can be done automatically with svn-buildpackage --svn-download-orig), but if the source needs to be repackaged, you should then provide a get-orig-source (policy) target in debian/rules which does the job.

Since probably for most of our packages, the original tarball comes from the Python Package Index (a.k.a. PyPI or the Cheeseshop), here is a sample debian/watch file that you might be able to adapt for your own package:

version=3
opts=uversionmangle=s/(rc|dev|a|b|c)/~$1/ \
https://pypi.python.org/packages/source/e/enum34/enum34-(.*)\.(?:tar\.gz|zip|tar\.bz2)

Overriding

If you need to override the dh_python2/3 defaults, do it with an override method in your debian/rules file. Something like this will work:

override_dh_python3:
        dh_python3 --shebang=/usr/bin/python3

If you need to override the default testing regime (e.g. to use py.test), you can do it one of two ways. Either:

export PYBUILD_TEST_PYTEST = 1

or you can override the dh_auto_test rule:

override_dh_auto_test:
        dh_auto_test -- --test-pytest

Gotchas

Python 3.3/3.4 unittest fixers for 2to3

Python 3.4 added a new 2to3 fixer which rewrites deprecated unittest assert methods to the current preferred names. If your upstream package uses 2to3 to provide a Python 3 compatible version, and uses these deprecated assert names, what this means is that the Python 3.3 and Python 3.4 version of those files will differ. dh_python3 will rightly refuse to copy files to the common location (i.e. /usr/lib/python3/dist-packages/<pkg>) when the files differ, so you'll end up with Python 3.4 specific paths in your python3- binary package in addition to the common location. lintian will complain.

There are a few ways to solve this, and maintainers may have their own preferences:

  • You can add a debian/<pkg>.pyremove file containing the paths to remove (but see bug 741381). E.g. the contents of python3-genshi.pyremove are:

/genshi/* 3.4
/Genshi*.egg-info/* 3.4
  • You could add an override in your d/rules file which does the removal. Something like this perhaps:

override_dh_python3:
        dh_python3
        rm -rf debian/python3-foo/usr/lib/python3.?
  • You could add a quilt patch that renames the deprecated methods in the upstream source. This would make 2to3 produce the same output for both Python 3.3 and 3.4 and avoid the problem altogether.

MANIFEST.in issues and missing files

If the upstream package requires some non-standard files to be installed, it must include a MANIFEST.in file. Sometimes these files are omitted from the original tarball because they're mostly useful for creating the sdist. Case in point: a package contained a bunch of doctests in .rst files but python setup.py sdist does not include .rst files by default. Thus, when pybuild went to build the package locally, the .rst files were omitted and the test suite failed because it could not find the doctest.

The solution for this is to either ensure that upstream includes the MANIFEST.in file, or that you add one before pybuild runs.

TODO

  • describe egg related information, what should go in the binary package and what shouldn't.