Differences between revisions 10 and 11
Revision 10 as of 2010-12-04 15:13:35
Size: 20473
Comment: apt-get source
Revision 11 as of 2010-12-04 15:14:45
Size: 20507
Comment:
Deletions are marked like this. Additions are marked like this.
Line 155: Line 155:
 $ debdiff ... | filterdiff -x '*/doc/*' -x '*/po/*'  $ debdiff foo_oldversion.dsc foo_newversion.dsc | filterdiff -x '*/doc/*' -x '*/po/*'

Translation(s): none


This tutorial is still a Draft... We're working on it ;-)


Python libraries/application packaging

Debian Women IRC Training Session held by Piotr Ożarowski, 02-Dec-2010

This is a tutorial about packaging Python modules and applications written in Python.

Requirements

In this tutorial it is assumed that:

Technical requirements:

Policy

First, few notes about Debian Python Policy: (AKA boring stuff that you have to know ;) ):

  • Python interpreter in Debian (since Python 2.6) is modified to use /usr/lib/pythonX.Y/dist-packages instead of site-packages. This way we can avoid conflicts with manually installed interpreter. It will also use /usr/local as --prefix by default (everyone who had to uninstall something ez_install installed (and overwrote our files), knows how important this change is), you can pass --install-layout=deb to setup.py's install to set all the Debian's defaults (f.e. --prefix=/usr).

  • We have few special packages:

    • python: which depends on default Python version, currently python2.6; this package also provides f.e. dh_python2 helper.

    • python-dev: like python, but additionally depends on pythonX.Y-dev package which provides header files used while building extensions.

    • python-dbg: depends on debug version of interpreter.

    • python-all: which depends on all supported Python versions, python2.5 and python2.6 right now, and python2.7 in python-all from experimental.

    • python-all-dev: like python-all, but depends on -dev packages as well.

    • python-all-dbg: depends on all supported debug interpreters.

  • See python-defaults source package for more info.

  • Packages with Python modules should support all Python versions supported by Debian (the ones python-all installs). This can be done by creating symlinks to Python pure modules (.py files) at install time (dh_pycentral and dh_pysupport can take care of that for you) or by shipping all symlinks inside the .deb file (dh_python2 does that). If a package provides public Python extension(s) (.so files), it has to ship them for all supported Python versions.

  • Binary packages with public modules should follow python-modulename naming schema (python-namespace.modulename for namespace packages, f.e. python-pyside.qtgui).

  • If documentation is big (bigger than the module itself), you can consider adding python-foo-doc binary package. I usually ship Sphinx generated docs in /usr/share/doc/python-foo-doc/html directory with a /usr/share/doc/python-foo-doc/rst -> /usr/share/doc/python-foo/html/_sources symlink (and /usr/share/doc/python-foo/html -> /usr/share/doc/python-foo-doc/html).

  • If you want to make the Python extension optional (when module doesn't require it) you can ship it in python-foo-ext binary package and Recommend this package in python-foo.

  • Private modules should be installed in private directory such as /usr/share/packagename/ or /usr/lib/packagename/

  • Python 2.X and Python 3.X should be treated as separated runtime systems.

  • You still can use the same source package name, but all binary packages providing Python 3 modules should be prefixed with python3- (python3-mako, python3-lxml, etc). There are special packages for Python 3 as well (python3, python3-all, python3-all-dev, ...), python3 provides dh_python3 helper.

  • Right now Python 3 modules/extensions are not built by dh so you have to modify your debian/rules file, we'll fix that before Wheezy release. (In CDBS v0.4.90 to support Python 3, you'll have to add include /usr/share/cdbs/1/class/python3-distutils.mk line to debian/rules and python3-foo package(s) to debian/control. dh already invokes dh_python3 when you add --with python3 option, but it doesn't build modules/extensions for Python 3 yet.

  • Python 3 installs all public modules and extensions to /usr/lib/python3/dist-packages (all Python 3 interpreters share the same directory!).

Few things you should do ''before'' packaging

  • Check if the package already exists or current maintainer wants someone else to take over (O, RFA bugs against wnpp package), someone requested for package (RFP) or intends to package it (ITP). Contact ITP submitter if it's been a while and bug is still not closed. Remember to check modulename, python-modulename, pydistname, python-pydistname source package names. pydistname as in what upstream uses for to name the package (PasteDeploy for paste.deploy module, etc). (It's not guaranteed that someone will spot a duplicate after reading your ITP).

  • Download the tarball and check if its license matches DFSG (licensecheck -r unpacked-tarball/ could help). Start with writing debian/copyright file to not waste time - it happened to me once or twice that after doing all the packaging work, I figured out I actually cannot upload it due to license issues. Contact upstream author if you still want the package in Debian.

  • Check if it really is worth packaging; do we have alternatives in Debian? If so, will you be able to convince others that we should have yet another package doing XYZ?

  • File ITP (with final long description of the package) or rename O/RFP bug.

Example 1: Python extension

  1. Let's start:
     $ wget http://pypi.python.org/packages/source/M/MarkupSafe/MarkupSafe-0.11.tar.gz
     $ py2dsc -m 'Piotr Ożarowski <piotr@debian.org>' MarkupSafe-0.11.tar.gz
     $ cd deb_dist/markupsafe-0.11

    py2dsc comes from python-stdeb package.

    -m is needed for now, as I didn't patch it yet to use DEBEMAIL.

  2. If sources are already unpacked, you can use this command instead (to create debian dir in the current directory):

     $ python setup.py --command-packages=stdeb.command debianize
  3. Copy the previously prepared copyright file or prepare it now (again, this can save your time).

  4. Let's take a look at debian directory: there's a debian/patches subdirectory, but we didn't create patches yet... If you open debian/patches/debian-changes-0.11-1 you will see that distutils is not cleaning the .egg-info directory. Let's remove this patch and clean the directory via dh_clean (distutils will regenerate it)

     $ quilt delete debian/patches/debian-changes-0.11-1
     $ rm -rf debian/patches
    We don't need this directory, at least for now.
  5. This should stop dpkg-source from adding this patch again:
     $ echo 'MarkupSafe.egg-info/*' > debian/clean

    dh_clean doesn't remove directories, but removing files inside is enough for us.

  6. In debian/changelog: we already have an ITP bug number, so let's replace the source package ... line with:

     Initial release. Closes: 123456
    where 123456 is our ITP number.
  7. Let's try to build the package for the first time:

     $ debuild
     $ lintian -i ../markupsafe_0.11-1_*.changes

    It worked! no errors and warnings, looks like packaging Python extensions is not that hard after all :-)

    You can invoke lintian with -IE if you want more issues to be reported, but it's enough to have the package clean of errors (E:) and warnings (W:).

    Please note that lintian *changes will check source package and binary packages, there's no need to invoke lintian *deb separately.

  8. Let's try to install all binary packages generated from our source package

     # debi ../markupsafe_0.11-1_*.changes
  9. And finally, we can test the package. There's a standard interface to test Python modules:

     $ python setup.py test
    But this will test files in current directory. Remember to change the current directory as Python tries to search for modules in .

    You'll find information about how to test the module in README or other upstream docs.

Notes about new upstream releases

  • When new upstream version will be available, don't use py2dsc again, just invoke:

     $ uupdate path/to/new-tarball

    in directory with old version. It will create ../python-foo-newversion/ directory and use dch to add new entry in debian/changelog. dch is a nice tool (see man dch for more info). If you don't have sources for previous version on you disk anymore, use apt-get source yourpackage to download them.

  • Always skim through the output of:
     $ debdiff foo_oldversion.dsc foo_newversion.dsc

    Hint:

     $ debdiff foo_oldversion.dsc foo_newversion.dsc | filterdiff -x '*/doc/*' -x '*/po/*'
    will ignore changes in docs and translations (remember to adjust dir names).

    You can find changes in debian directory that you forgot to document in debian/changelog or changes in upstream code that require bumping minimum build/runtime versions or are dangerous and should be patched/reported upstream.

Example 2: Python application

  1. Download the tarball:

     $ wget http://downloads.sourceforge.net/project/ljcharm/charm/charm-1.9.1/charm-1.9.1.tar.gz
  2. Unpack it and create the debian directory the same way you create it for all other packages. You can do it with py2dsc as well, but we'll need to make few adjustments later:

     $ py2dsc -m 'Piotr Ożarowski <piotr@debian.org>' charm-1.9.1.tar.gz
     $ cd deb_dist/charm-1.9.1
  3. Open debian/control file and:

    1. Rename binary package name to charm. We will not provide public module, so python- prefix is not needed.

    2. Remove Breaks: ${python:Breaks} line as well, for the same reason.

    3. Replace python-all with python in Build-Depends, we don't need all interpreters.

    4. README.charm mentions that this application needs Python 2.5 so tell dh_python2 that by adding

        X-Python-Version: >= 2.5
      in source stanza.

      For Python 3 applications use X-Python3-Version and dh_python3.

  4. Open debian/rules file and:
    1. Remove --buildsystem=python_distutils, there's no Makefile

    2. Add these three lines:
        override_dh_auto_install:
              python setup.py install --root=debian/charm --install-layout=deb --install-lib=/usr/share/charm --install-scripts=/usr/share/charm
        override_dh_auto_build:

      This is the important part, it will tell distutils that we want to use private directory. This directory is not in sys.path, so charm script is installed (via --install-scripts) to this directory as well --root is distutils ' DESTDIR

      We have to add override_dh_auto_build: line because dh detected setup.py file and wants to build modules for all.

  5. Rename the script if the module has exactly the same name, new name doesn't matter, we'll use the original one in /usr/bin anyway.

  6. Create debian/links file with this line inside:

     /usr/share/charm/charm /usr/bin/charm

    This file will be picked up by dh_link which will create a /usr/bin/charm symlink. Use f.e. /usr/share/charm/run /usr/bin/charm if module name is charm as well. But remember to rename the binary in /usr/share/charm.

  7. Build and install the package:

     $ debuild
     # debi ../charm_1.9.1-1_*.changes
  8. Test it. Changing directory is not needed in this case, but I always open new terminal to do tests, just in case:

     $ charm

    It works :) (altough there are warnings about hashlib module).

Packaging Python stuff is really not that scary, and it will be even easier with Python 3.

Questions and Answers

QUESTION: What exactly does ${python:Breaks} expand to and why do we need to remove it?

ANSWER: It will move python (>= first_supported), python (<< last_supported) from Depends into Breaks: python (<< first_supported), python (>= last_supported). It's used for Python modules, thet's why I added it to stdeb few days ago.

If you will not add Breaks, dh_python2 will add this dependency in Depends and then lintian will cry.

QUESTION: So why do we need a different approach for modules and apps?

ANSWER: You mean why private modules are used in applications? Mainly to not pollute glogal* namespace

QUESTION: No. Why do we need the breaks for modules and not for applications? Is it because apps have private modules?

ANSWER1: I suspect you mean why different way of building. One reason I can think of (and I could be wrong) is that apps don't have to support all python versions

ANSWER2: Two applications that provide "gui" module cannot be installed at the same time if both would use public namespace. Breaks is simply not used in applications. You can leave it in debian/control, but it will do nothing.

Private module can support only one Python version at the same time. dh_python2 will regenerate .pyc files if default python version will change. Well, it will change with Python 3 (which can support multiple Python versions even with private modules).

QUESTION: Sorry, just a clarification about my previous question. For modules we need to add Breaks because if we don't the module will compile on versions of python that it doesn't support? For apps, there are only private modules and they all use only one version at a time.

ANSWER: Breaks will just move some dependencies from Depends to Breaks. Nothing else. It's easier to override it if it's in Breaks. For example, if the package is still usable with pythonX.Y, you can use it although it Breaks pythonX.Z

QUESTION: This isn't python specific, but it looks like the first tutorial didn't cover it. Is there a reference that describes what ./debian/rules tools like debuild (or dpkg-buildpackage, etc) call for building packages? s/f.e./f.e. if/

ANSWER: debuild is actually calling dpkg-buildpackage... which is invoking debian/rules sometarget. It's described in Debian Policy. sometarget can be f.e. build or binary-indep

./debian/ruiles binary-indep will build all architecture independent packages

./debian/ruiles binary-arch will build all architecture dependent packages

debuild is just a wrapper, debian/rules is "just" a Makefile

dh_python2 tries to translate upstream's requires.txt into Debian's dependencies, but if upstream didn't provide this file you can use tools like find_python_dependencies.py to figure out missing dependencies

See: http://svn.debian.org/viewsvn/python-modules/tools/

(See also http://www.debian.org/doc/debian-policy/ch-source.html#s-debianrules)

QUESTION: I often package internal python scripts and things for work, targetting the stable release. Is there a wiki page that summarises the "state-of-the-art" for each release? I.e. for lenny it might say you can use pycentral or dh_python or pysupport?

ANSWER: dh_python2 is introduced in Squeeze and in Wheezy I plan to drop python-central and pytnon-support. But that's just me :)

For Python 3 we have only one helper dh_python3. I modified dh_pysupport and dh_pycentral to ignore python3-foo packages. dh_python3 ignores python-foo on the other hand, but you still have to tell debhelper which tool should handle private directories, i.e. all tools will try to bytecompile files in /usr/lib/foo/

QUESTION: If we have some problem, how, where we can ask ?

ANSWER: I didn't write much documentation yet... but you have #debian-python channel or debian-python@lists.debian.org, there are many smart guys there.

QUESTION: Can you explain us the procedure to join the python team in Debian?

ANSWER: http://wiki.debian.org/Teams/PythonModulesTeam/HowToJoin, you pick a package, start packaging it and contact us when you need help or an sponsor. We try to take a look at all DPMT / PAPT packages but it's mostly your responsibility to care about it. You will also receive comments to your commits from time to time, mostly from me complaining about something ;)

QUESTION: Are there issues that make for common complaints from you? Do you see people repeatedly doing things incorrectly, that you want to mention here?

ANSWER: The only common is about tagging before an upload (most packages are not ready before the first review). Ask on #debian-mentors or #debian-python (for Python related stuff), we don't bite. Well, if you'll go to debconf and tell me I'm your sponsor, I might bite (for all your mistakes from the past ;)

QUESTION: Maybe is a little off topic but it seems we have not so many questions, so... How do you (as team) managed relationship with upstream and downstream?

Well, the only policy we have is: talk with upstream: introduce yourself, forward her/him bugs, send patches. We also have quite some upstreams in the team, we try to help them a little bit more than we help others.

QUESTION: I've never used stdeb, how would you compare it with dh-make?

ANSWER: Well, dh-make doesn't know much about recent changes in Python policy, if any ;)

QUESTION: Do you have any suggestions for upstream authors? Anything that they should do to help the packaging?

ANSWER: http://wiki.debian.org/UpstreamGuide, there's a Python section as well. There's also http://us.pycon.org/media/2010/talkdata/PyCon2010/038/paper.html. Stani (SPE and Phatch author) is one of out team members, BTW. He wrote Cross-platform Application Development and Distribution linked above with Nadia Alramli. You can also ask Runa_ about Python stuff

About sponsors: you can cheat a littile bit and check on this page: http://wiki.debian.org/SponsorChecklist what sponsors usually check.

QUESTION: I'm comparing the debian/ directories build by py2dsc and dh_make for the same package, py2dsc is setting some fields in debian/control to reasonable values, including the Buil-Depends. It fetches some depends from setup.py, why is XS-Python-Version: current set? Does XS-Python-Version: current build for the default Python version or all the supported ones?

COMMENT: stdeb tries to parse some data from setup.py like package description. dh_make simply doesn't know much about distutils. And stdeb is... distutils ' extension written by Andrew Straw... who is DPMT member

ANSWER: Current is deprecated. It was used by pytohn-central to bytecompile for one Python version only. But since it's better to simply use private directories, we dropped current keyword. I uploaded stdeb with few patches yesterday :), unexport CPPFLAGS and so on :D (version dedicated to #debian-women ;) )

I also uploaded new python-defaults to experimental, with new features like .pyinstall or .pyremove to make pakcaging easier, but there were no questions about it. I mean, there were no problems that .pyinstall or .pyremove could solve.

See also