Translation(s): none



Python libraries/application packaging

Debian Women IRC Training Session held by Piotr Ożarowski, 02-Dec-2010

This is a tutorial about packaging Python modules and applications written in Python. The tutorial is based on the python-stdeb tool, a Python-specific alternative to dh-make.

Attention: the contents of this document may be outdated, and it is advised to check instead more recent guides like Python/LibraryStyleGuide (refer to instructions in Python).

Requirements

In this tutorial it is assumed that:

Technical requirements:

It is recommended to always use the latest version of those packages from the Unstable repository.

Policy

First, few notes about Debian Python Policy (AKA boring stuff that you have to know ;) ):

Few things you should do ''before'' packaging

Example 1: Python extension

  1. Let's start:
     $ wget http://pypi.python.org/packages/source/M/MarkupSafe/MarkupSafe-0.11.tar.gz
     $ py2dsc -m 'Piotr Ożarowski <piotr@debian.org>' MarkupSafe-0.11.tar.gz
     $ cd deb_dist/markupsafe-0.11

    py2dsc comes from python-stdeb package.

    -m is needed for now, as I didn't patch it yet to use DEBEMAIL.

  2. If sources are already unpacked, you can use this command instead (to create debian dir in the current directory):

     $ python setup.py --command-packages=stdeb.command debianize
  3. Copy the previously prepared copyright file or prepare it now (again, this can save your time).

  4. Let's take a look at debian directory: there's a debian/patches subdirectory, but we didn't create patches yet... If you open debian/patches/debian-changes-0.11-1 you will see that distutils is not cleaning the .egg-info directory. Let's remove this patch and clean the directory via dh_clean (distutils will regenerate it)

     $ quilt delete debian/patches/debian-changes-0.11-1
     $ rm -rf debian/patches
    We don't need this directory, at least for now.
  5. This should stop dpkg-source from adding this patch again:

     $ echo 'MarkupSafe.egg-info/*' > debian/clean

    dh_clean doesn't remove directories, but removing files inside is enough for us.

  6. In debian/changelog: we already have an ITP bug number, so let's replace the source package ... line with:

     Initial release. Closes: 123456
    where 123456 is our ITP number.
  7. Let's try to build the package for the first time:

     $ debuild
     $ lintian -i ../markupsafe_0.11-1_*.changes

    It worked! no errors and warnings, looks like packaging Python extensions is not that hard after all :-)

    You can invoke lintian with -IE if you want more issues to be reported, but it's enough to have the package clean of errors (E:) and warnings (W:).

    Please note that lintian *changes will check source package and binary packages, there's no need to invoke lintian *deb separately.

  8. Let's try to install all binary packages generated from our source package

     # debi ../markupsafe_0.11-1_*.changes
  9. And finally, we can test the package. There's a standard interface to test Python modules:

     $ python setup.py test
    But this will test files in current directory. Remember to change the current directory as Python tries to search for modules in .

    You'll find information about how to test the module in README or other upstream docs.

Notes about new upstream releases

Example 2: Python application

  1. Download the tarball:

     $ wget http://downloads.sourceforge.net/project/ljcharm/charm/charm-1.9.1/charm-1.9.1.tar.gz
  2. Unpack it and create the debian directory the same way you create it for all other packages. You can do it with py2dsc as well, but we'll need to make few adjustments later:

     $ py2dsc -m 'Piotr Ożarowski <piotr@debian.org>' charm-1.9.1.tar.gz
     $ cd deb_dist/charm-1.9.1
  3. Open debian/control file and:

    1. Rename binary package name to charm. We will not provide public module, so python- prefix is not needed.

    2. Remove Breaks: ${python:Breaks}  line as well, for the same reason.

    3. Replace python-all with python in Build-Depends, we don't need all interpreters.

    4. README.charm mentions that this application needs Python 2.5 so tell dh_python2 that by adding

        X-Python-Version: >= 2.5
      in source stanza.

      For Python 3 applications use X-Python3-Version and dh_python3.

  4. Open debian/rules file and:
    1. Remove --buildsystem=python_distutils, there's no Makefile

    2. Add these three lines:
        override_dh_auto_install:
              python setup.py install --root=debian/charm --install-layout=deb --install-lib=/usr/share/charm --install-scripts=/usr/share/charm
        override_dh_auto_build:

      This is the important part, it will tell distutils that we want to use private directory. This directory is not in sys.path, so charm script is installed (via --install-scripts) to this directory as well --root is distutils ' DESTDIR

      We have to add override_dh_auto_build: line because dh detected setup.py file and wants to build modules for all.

  5. Rename the script if the module has exactly the same name, new name doesn't matter, we'll use the original one in /usr/bin anyway.

  6. Create debian/links file with this line inside:

     /usr/share/charm/charm /usr/bin/charm

    This file will be picked up by dh_link which will create a /usr/bin/charm symlink. Use f.e. /usr/share/charm/run /usr/bin/charm if module name is charm as well. But remember to rename the binary in /usr/share/charm.

  7. Build and install the package:

     $ debuild
     # debi ../charm_1.9.1-1_*.changes
  8. Test it. Changing directory is not needed in this case, but I always open new terminal to do tests, just in case:

     $ charm

    It works :) (altough there are warnings about hashlib module).

Packaging Python stuff is really not that scary, and it will be even easier with Python 3.

Questions and Answers

QUESTION: What exactly does ${python:Breaks} expand to and why do we need to remove it?

ANSWER: It will move python (>= first_supported), python (<< last_supported) from Depends into Breaks: python (<< first_supported), python (>= last_supported). It's used for Python modules.

If you will not add Breaks, dh_python2 will add this dependency in Depends and then lintian will report a warning.

QUESTION: So why do we need a different approach for modules and apps?

ANSWER: You mean why private modules are used in applications? Mainly to not pollute global namespace

QUESTION: No. Why do we need the breaks for modules and not for applications? Is it because apps have private modules?

ANSWER1: I suspect you mean why different way of building. One reason I can think of (and I could be wrong) is that apps don't have to support all python versions

ANSWER2: Two applications that provide "gui" module cannot be installed at the same time if both would use public namespace. Breaks is simply not used in applications. You can leave it in debian/control, but it will do nothing.

Private module can support only one Python version at the same time. dh_python2 will regenerate .pyc files if default python version will change. (Python3 can support multiple Python versions even with private modules, BTW).

QUESTION: Sorry, just a clarification about my previous question. For modules we need to add Breaks because if we don't the module will compile on versions of python that it doesn't support? For apps, there are only private modules and they all use only one version at a time.

ANSWER: Breaks will just move some dependencies from Depends to Breaks. Nothing else. It's easier to override it by users if it's in Breaks. For example, if the package is still usable with pythonX.Y, you can use it although it Breaks pythonX.Z

QUESTION: This isn't Python specific, but it looks like the first tutorial didn't cover it. Is there a reference that describes what ./debian/rules tools like debuild (or dpkg-buildpackage, etc) call for building packages? s/f.e./f.e. if/

ANSWER: debuild is actually calling dpkg-buildpackage... which is invoking debian/rules sometarget. It's described in Debian Policy. sometarget can be f.e. build or binary-indep

./debian/ruiles binary-indep will build all architecture independent packages

./debian/ruiles binary-arch will build all architecture dependent packages

debuild is just a wrapper, debian/rules is "just" a Makefile

dh_python2 tries to translate upstream's requires.txt into Debian's dependencies, but if upstream didn't provide this file you can use tools like find_python_dependencies.py to figure out missing dependencies

See: http://svn.debian.org/viewsvn/python-modules/tools/

(See also http://www.debian.org/doc/debian-policy/ch-source.html#s-debianrules)

QUESTION: I often package internal Python scripts and things for work, targetting the stable release. Is there a wiki page that summarizes the "state-of-the-art" for each release? I.e. for Lenny it might say you can use pycentral or dh_python or pysupport?

ANSWER: dh_python2 is introduced in Squeeze and in Wheezy I plan to drop python-central and pytnon-support. But that's just me :)

For Python 3 we have only one helper dh_python3. I modified dh_pysupport and dh_pycentral to ignore python3-foo packages. dh_python3 ignores python-foo on the other hand, but you still have to tell debhelper which tool should handle private directories, i.e. all tools will try to byte-compile files in /usr/lib/foo/

QUESTION: If we have some problem, how, where we can ask ?

ANSWER: I didn't write much documentation yet... but you have #debian-python channel or debian-python@lists.debian.org, there are many smart people there.

QUESTION: Can you explain us the procedure to join the python team in Debian?

ANSWER: http://wiki.debian.org/Teams/PythonModulesTeam/HowToJoin, you pick a package, start packaging it and contact us when you need help or an sponsor. We try to take a look at all DPMT / PAPT packages but it's mostly your responsibility to care about it. You will also receive comments to your commits from time to time, mostly from me complaining about something ;)

QUESTION: Are there issues that make for common complaints from you? Do you see people repeatedly doing things incorrectly, that you want to mention here?

ANSWER: The only common is about tagging before an upload (most packages are not ready before the first review). Ask on #debian-mentors or #debian-python (for Python related stuff), we don't bite. Well, if you'll go to debconf and tell me I'm your sponsor, I might bite (for all your mistakes from the past ;)

QUESTION: Maybe is a little off topic but it seems we have not so many questions, so... How do you (as team) managed relationship with upstream and downstream?

Well, the only policy we have is: talk with upstream: introduce yourself, forward her/him bugs, send patches. We also have quite some upstreams in the team, we try to help them a little bit more than we help others.

QUESTION: I've never used stdeb, how would you compare it with dh_make?

ANSWER: Well, dh_make doesn't know much about recent changes in Python policy, if any ;)

QUESTION: Do you have any suggestions for upstream authors? Anything that they should do to help the packaging?

ANSWER: http://wiki.debian.org/UpstreamGuide, there's a Python section as well. There's also http://us.pycon.org/media/2010/talkdata/PyCon2010/038/paper.html. Stani (SPE and Phatch author) is one of out team members, BTW. He wrote Cross-platform Application Development and Distribution linked above with Nadia Alramli. You can also ask Runa (here on #debian-women) about Python stuff

About sponsors: you can cheat a little bit and check on this page: http://wiki.debian.org/SponsorChecklist what sponsors usually check.

QUESTION: I'm comparing the debian/ directories build by py2dsc and dh_make for the same package, py2dsc is setting some fields in debian/control to reasonable values, including the Buil-Depends. It fetches some depends from setup.py, why is XS-Python-Version: current set? Does XS-Python-Version: current build for the default Python version or all the supported ones?

COMMENT: stdeb tries to parse some data from setup.py like package description. dh_make simply doesn't know much about distutils. And stdeb is... distutils ' extension written by Andrew Straw... who is DPMT member

ANSWER: Current is deprecated. It was used by pytohn-central to bytecompile for one Python version only. But since it's better to simply use private directories, we dropped current keyword. I uploaded stdeb with few patches yesterday :), unexport CPPFLAGS and so on :D (version dedicated to #debian-women ;) )

I also uploaded new python-defaults to experimental, with new features like .pyinstall or .pyremove to make packaging easier, but there were no questions about it. I mean, there were no problems that .pyinstall or .pyremove could solve.

See also