Differences between revisions 25 and 26
Revision 25 as of 2015-11-23 18:15:28
Size: 21307
Editor: NielsThykier
Comment: Advertise the use of "dir/" in debian/clean with debhelper/9.20151117
Revision 26 as of 2016-04-11 13:23:04
Size: 107
Comment: redirect to Python/LibraryStyleGuide
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
## page was renamed from PythonPackaging
#language en
~-[[DebianWiki/EditorGuide#translation|Translation(s)]]: none-~
----

<<TableOfContents(2)>>
----

== Python libraries/application packaging ==
~- Debian Women IRC Training Session held by Piotr Ożarowski, 02-Dec-2010 -~

This is a tutorial about packaging Python modules and applications written in Python.
The tutorial is based on the DebianPkg:python-stdeb tool, a Python-specific alternative to DebianPkg:dh-make.

'''''Attention: the contents of this document may be outdated, and it is advised to check instead more recent guides like [[Python/LibraryStyleGuide]] (refer to instructions in [[Python]]).'''''

== Requirements ==

'''In this tutorial, it is assumed that:'''

 * you know how Debian packaging works. You should read [[IntroDebianPackaging|Introduction to Debian Packaging]] tutorial.
 * you have some Python knowledge.

Technical requirements:

 * DebianPkg:devscripts, DebianPkg:python-all-dev, DebianPkg:python-stdeb

It is recommended to always use the latest version of those packages from the Unstable repository.

== Policy ==

First, few notes about '''Debian Python Policy''' (AKA boring stuff that you have to know ;) ):
 
 * '''Python interpreter''' in Debian (since ''Python 2.6'') is modified to use {{{/usr/lib/pythonX.Y/dist-packages}}} instead of {{{site-packages}}}. This way we can avoid conflicts with manually installed interpreter. It will also use {{{/usr/local}}} as {{{--prefix}}} by default (everyone who had to uninstall something ez_install installed (and overwrote our files), knows how important this change is), you can pass {{{--install-layout=deb}}} to {{{setup.py}}}'s install to set all the Debian's defaults (f.e. {{{--prefix=/usr}}}).

 * We have few '''special packages''':
  * DebianPkg:python: which depends on default Python version, currently ''python2.6''; this package also provides f.e. {{{dh_python2}}} helper.
  * DebianPkg:python-dev: like ''python'', but additionally depends on ''pythonX.Y-dev'' package which provides header files used while building extensions.
  * DebianPkg:python-dbg: depends on debug version of interpreter.
  * DebianPkg:python-all: which depends on all supported Python versions, ''python2.5'' and ''python2.6'' right now, and ''python2.7'' in ''python-all'' from experimental.
  * DebianPkg:python-all-dev: like ''python-all'', but depends on ''-dev'' packages as well.
  * DebianPkg:python-all-dbg: depends on all supported debug interpreters.
 . See ''python-defaults'' source package for more info.

 * '''Packages with Python modules should support all Python versions supported by Debian''' (the ones ''python-all'' installs). This can be done by creating symlinks to Python pure modules ({{{.py}}} files) at install time ({{{dh_pycentral}}} and {{{dh_pysupport}}} can take care of that for you) or by shipping all symlinks inside the {{{.deb}}} file ({{{dh_python2}}} does that). If a package provides public Python extension(s) ({{{.so}}} files), it has to ship them for all supported Python versions.

 * Binary packages with public modules should follow ''python-modulename'' '''naming schema''' (''python-namespace.modulename'' for namespace packages, f.e. ''python-pyside.qtgui'').
 . If documentation is big (bigger than the module itself), you can consider adding ''python-foo-doc'' binary package. I usually ship Sphinx generated docs in {{{/usr/share/doc/python-foo-doc/html}}} directory with a {{{/usr/share/doc/python-foo-doc/rst}}} -> {{{/usr/share/doc/python-foo/html/_sources symlink}}} (and {{{/usr/share/doc/python-foo/html}}} -> {{{/usr/share/doc/python-foo-doc/html}}}).
 . If you want to make the Python extension optional (when module doesn't require it) you can ship it in ''python-foo-ext'' binary package and ''Recommend'' this package in ''python-foo''.
 
 * Private modules should be installed in '''private directory''' such as {{{/usr/share/packagename/}}} or {{{/usr/lib/packagename/}}}

 * ''Python 2.X'' and ''Python 3.X'' should be treated as '''separated runtime systems'''.
 . You still can use the same source package name, but all binary packages providing ''Python 3'' public modules should be prefixed with ''python3-'' (''python3-mako'', ''python3-lxml'', etc). There are special packages for ''Python 3'' as well (''python3'', ''python3-all'', ''python3-all-dev'', ...), ''dh-python'' package provides {{{dh_python3}}} helper.
 . ''dh'' sequencer doesn't build ''Python 3'' modules/extensions by default, add ''--buildsystem=pybuild'' option to change build system to pybuild which builds 2.x and 3.x extensions (see pybuild's [[http://deb.li/pybuild|manual]] for more information). In CDBS to support ''Python 3'', you have to add {{{include /usr/share/cdbs/1/class/python-distutils.mk}}} line to {{{debian/rules}}} and ''python3-foo'' package(s) to {{{debian/control}}}. ''dh'' invokes ''dh_python3'' when you add {{{--with python3}}} option.

 * ''Python 3'' installs all '''public modules and extensions''' to {{{/usr/lib/python3/dist-packages}}} (all ''Python 3'' interpreters share the same directory!).

== Few things you should do ''before'' packaging ==

 * '''Check if the package already exists''' or current maintainer wants someone else to take over (O, RFA bugs against wnpp package), someone requested for package (RFP) or intends to package it (ITP). Contact ITP submitter if it's been a while and bug is still not closed. Remember to check ''modulename'', ''python-modulename'', ''pydistname'', ''python-pydistname'' source package names. ''pydistname'' as in what upstream uses for to name the package (''!PasteDeploy'' for ''paste.deploy'' module, etc). (It's not guaranteed that someone will spot a duplicate after reading your ITP).
 * '''Download the tarball and check if its license matches DFSG''' ({{{licensecheck -r unpacked-tarball/}}} could help). Start with writing {{{debian/copyright}}} file to not waste time - it happened to me once or twice that after doing all the packaging work, I figured out I actually cannot upload it due to license issues. Contact upstream author if you still want the package in Debian.
 * '''Check if it really is worth packaging'''; do we have alternatives in Debian? If so, will you be able to convince others that we should have yet another package doing XYZ?
 * '''File ITP''' (with ''final'' long description of the package) or rename O/RFP bug.

== Example 1: Python extension ==

 1. Let's start:

 {{{
 $ wget http://pypi.python.org/packages/source/M/MarkupSafe/MarkupSafe-0.11.tar.gz
 $ py2dsc -m 'Piotr Ożarowski <piotr@debian.org>' MarkupSafe-0.11.tar.gz
 $ cd deb_dist/markupsafe-0.11
 }}}

 {{{py2dsc}}} comes from ''python-stdeb'' package.

 {{{-m}}} is needed for now, as I didn't patch it yet to use {{{DEBEMAIL}}}.

 1. If sources are already unpacked, you can use this command instead (to '''create {{{debian}}} dir''' in the current directory):

 {{{
 $ python setup.py --command-packages=stdeb.command debianize
 }}}

 1. Copy the previously prepared '''{{{copyright}}}''' file or prepare it now (again, this can save your time).

 1. Let's take a look at {{{debian}}} directory: there's a {{{debian/patches}}} subdirectory, but we didn't create patches yet... If you open {{{debian/patches/debian-changes-0.11-1}}} you will see that ''distutils'' is not cleaning the {{{.egg-info directory}}}. Let's '''remove this patch''' and clean the directory via {{{dh_clean}}} (''distutils'' will regenerate it)

 {{{
 $ quilt delete debian/patches/debian-changes-0.11-1
 $ rm -rf debian/patches
 }}}

 We don't need this directory, at least for now.

 1. This should stop {{{dpkg-source}}} from adding this patch again:

 {{{
 $ echo 'MarkupSafe.egg-info/' > debian/clean
 }}}

 The above works with debhelper/9.20151117. For Jessie (and older), please consider using:

 {{{
 $ echo 'MarkupSafe.egg-info/*' > debian/clean
 }}}

 Please do note that this variant will *not* remove the directory itself. However, removing files inside is enough for us.

 1. In {{{debian/changelog}}}: we already have an '''ITP bug number''', so let's replace the ''source package ...'' line with:

 {{{
 Initial release. Closes: 123456
 }}}

 where 123456 is our ITP number.

 1. Let's try to '''build''' the package for the first time:

 {{{
 $ debuild
 $ lintian -i ../markupsafe_0.11-1_*.changes
 }}}

 It worked! no errors and warnings, looks like packaging Python extensions is not that hard after all :-)

 You can invoke lintian with {{{-IE}}} if you want more issues to be reported, but it's enough to have the package clean of errors (E:) and warnings (W:).

 Please note that {{{lintian *changes}}} will check source package and binary packages, there's no need to invoke {{{lintian *deb}}} separately.

 1. Let's try to '''install''' all binary packages generated from our source package

 {{{
 # debi ../markupsafe_0.11-1_*.changes
 }}}

 1. And finally, we can '''test''' the package. There's a standard interface to test Python modules:

 {{{
 $ python setup.py test
 }}}

 But this will test files in current directory. Remember to change the current directory as Python tries to search for modules in .

 You'll find information about how to test the module in {{{README}}} or other upstream docs.

=== Notes about new upstream releases ===

 * When new upstream version will be available, don't use {{{py2dsc}}} again, just invoke:
 {{{
 $ uupdate path/to/new-tarball
 }}}
 in directory with old version. It will create {{{../python-foo-newversion/}}} directory and use {{{dch}}} to add new entry in {{{debian/changelog}}}.
 {{{dch}}} is a nice tool (see {{{man dch}}} for more info).
 If you don't have sources for previous version on your disk anymore, use {{{apt-get source yourpackage}}} to download them.

 * Always skim through the output of:
 {{{
 $ debdiff foo_oldversion.dsc foo_newversion.dsc
 }}}
 '''Hint:'''
 {{{
 $ debdiff foo_oldversion.dsc foo_newversion.dsc | filterdiff -x '*/doc/*' -x '*/po/*'
 }}}
 will ignore changes in docs and translations (remember to adjust dir names).
 
 You can find changes in {{{debian}}} directory that you forgot to document in {{{debian/changelog}}} or changes in upstream code that require bumping minimum build/runtime versions or are dangerous and should be patched/reported upstream.

== Example 2: Python application ==

 1. '''Download''' the tarball:

 {{{
 $ wget http://downloads.sourceforge.net/project/ljcharm/charm/charm-1.9.1/charm-1.9.1.tar.gz
 }}}

 1. '''Unpack''' it and '''create the {{{debian}}} directory''' the same way you create it for all other packages. You can do it with {{{py2dsc}}} as well, but we'll need to make few adjustments later:

 {{{
 $ py2dsc -m 'Piotr Ożarowski <piotr@debian.org>' charm-1.9.1.tar.gz
 $ cd deb_dist/charm-1.9.1
 }}}

 1. Open {{{debian/control}}} file and:

  1. '''Rename binary package name''' to ''charm''. We will not provide public module, so ''python-'' prefix is not needed.

  1. '''Remove {{{Breaks: ${python:Breaks} }}} line''' as well, for the same reason.

  1. '''Replace ''python-all'' with ''python''''' in ''Build-Depends'', we don't need all interpreters.
 
  1. {{{README.charm}}} mentions that this application needs ''Python 2.5'' so tell {{{dh_python2}}} that by adding

  {{{
  X-Python-Version: >= 2.5
  }}}

  in source stanza.

  For ''Python 3'' applications use ''X-Python3-Version'' and {{{dh_python3}}}.

 1. Open debian/rules file and:

  1. '''Remove {{{--buildsystem=python_distutils}}}''', there's no Makefile

  1. Add these three lines:

  {{{
  override_dh_auto_install:
   python setup.py install --root=debian/charm --install-layout=deb --install-lib=/usr/share/charm --install-scripts=/usr/share/charm
  override_dh_auto_build:
  }}}

  This is the '''important''' part, it will tell ''distutils'' that we want to use private directory. This directory is not in {{{sys.path}}}, so {{{charm}}} script is installed (via {{{--install-scripts}}}) to this directory as well {{{--root}}} is ''distutils'' ' {{{DESTDIR}}}
 
  We have to add {{{override_dh_auto_build:}}} line because {{{dh}}} detected {{{setup.py}}} file and wants to build modules for all.

 1. '''Rename the script''' if the module has exactly the same name, new name doesn't matter, we'll use the original one in {{{/usr/bin}}} anyway.

 1. '''Create {{{debian/links}}}''' file with this line inside:

 {{{
 /usr/share/charm/charm /usr/bin/charm
 }}}

 This file will be picked up by {{{dh_link}}} which will create a {{{/usr/bin/charm}}} symlink. Use f.e. {{{/usr/share/charm/run /usr/bin/charm}}} if module name is ''charm'' as well. But remember to rename the binary in {{{/usr/share/charm}}}.

 1. '''Build''' and '''install''' the package:

 {{{
 $ debuild
 # debi ../charm_1.9.1-1_*.changes
 }}}

 1. '''Test''' it. Changing directory is not needed in this case, but I always open new terminal to do tests, just in case:

 {{{
 $ charm
 }}}

 It works :) (altough there are warnings about hashlib module).

Packaging Python stuff is really not that scary, and it will be even easier with ''Python 3''.

== Questions and Answers ==

'''QUESTION:''' What exactly does {{{${python:Breaks}}}} expand to and why do we need to remove it?

ANSWER: It will move ''python (>= first_supported), python (<< last_supported)'' from ''Depends'' into ''Breaks: python (<< first_supported), python (>= last_supported)''. It's used for Python modules.

If you will not add ''Breaks'', {{{dh_python2}}} will add this dependency in ''Depends'' and then {{{lintian}}} will report a warning.

'''QUESTION:''' So why do we need a different approach for modules and apps?

ANSWER: You mean why private modules are used in applications? Mainly to not pollute global namespace

'''QUESTION:''' No. Why do we need the breaks for modules and not for applications? Is it because apps have private modules?

ANSWER1: I suspect you mean why different way of building. One reason I can think of (and I could be wrong) is that apps don't have to support all python versions

ANSWER2: Two applications that provide "gui" module cannot be installed at the same time if both would use public namespace. ''Breaks'' is simply not used in applications. You can leave it in {{{debian/control}}}, but it will do nothing.

Private module can support only one Python version at the same time. {{{dh_python2}}} will regenerate {{{.pyc}}} files if default python version will change. (''Python3'' can support multiple Python versions even with private modules, BTW).

'''QUESTION:''' Sorry, just a clarification about my previous question. For modules we need to add Breaks because if we don't the module will compile on versions of python that it doesn't support? For apps, there are only private modules and they all use only one version at a time.

ANSWER: ''Breaks'' will just move some dependencies from ''Depends'' to ''Breaks''. Nothing else. It's easier to override it by users if it's in ''Breaks''. For example, if the package is still usable with ''pythonX.Y'', you can use it although it Breaks ''pythonX.Z''

'''QUESTION:''' This isn't Python specific, but it looks like the first tutorial didn't cover it. Is there a reference that describes what {{{./debian/rules}}} tools like {{{debuild}}} (or {{{dpkg-buildpackage}}}, etc) call for building packages?
s/f.e./f.e. if/

ANSWER: {{{debuild}}} is actually calling {{{dpkg-buildpackage}}}... which is invoking {{{debian/rules sometarget}}}. It's described in Debian Policy. {{{sometarget}}} can be f.e. {{{build}}} or {{{binary-indep}}}

{{{./debian/ruiles binary-indep}}} will build all architecture independent packages

{{{./debian/ruiles binary-arch}}} will build all architecture dependent packages

{{{debuild}}} is just a wrapper, {{{debian/rules}}} is "just" a {{{Makefile}}}

{{{dh_python2}}} tries to translate upstream's requires.txt into Debian's dependencies, but if upstream didn't provide this file you can use tools like {{{find_python_dependencies.py}}} to figure out missing dependencies

See: http://svn.debian.org/viewsvn/python-modules/tools/

(See also http://www.debian.org/doc/debian-policy/ch-source.html#s-debianrules)

'''QUESTION:''' I often package internal Python scripts and things for work, targetting the stable release. Is there a wiki page that summarizes the "state-of-the-art" for each release? I.e. for Lenny it might say you can use {{{pycentral}}} or {{{dh_python}}} or {{{pysupport}}}?

ANSWER: {{{dh_python2}}} is introduced in Squeeze and in Wheezy I plan to drop {{{python-central}}} and {{{pytnon-support}}}. But that's just me :)

For ''Python 3'' we have only one helper {{{dh_python3}}}. I modified {{{dh_pysupport}}} and {{{dh_pycentral}}} to ignore ''python3-foo'' packages. {{{dh_python3}}} ignores ''python-foo'' on the other hand, but you still have to tell ''debhelper'' which tool should handle private directories, i.e. all tools will try to byte-compile files in {{{/usr/lib/foo/}}}

'''QUESTION:''' If we have some problem, how, where we can ask ?

ANSWER: I didn't write much documentation yet... but you have #debian-python channel or debian-python@lists.debian.org, there are many smart people there.

'''QUESTION:''' Can you explain us the procedure to join the python team in Debian?

ANSWER: [[http://wiki.debian.org/Teams/PythonModulesTeam/HowToJoin]], you pick a package, start packaging it and contact us when you need help or an sponsor. We try to take a look at all DPMT / PAPT packages but it's mostly your responsibility to care about it. You will also receive comments to your commits from time to time, mostly from me complaining about something ;)

'''QUESTION:''' Are there issues that make for common complaints from you? Do you see people repeatedly doing things incorrectly, that you want to mention here?

ANSWER: The only common is about tagging before an upload (most packages are not ready before the first review). Ask on #debian-mentors or #debian-python (for Python related stuff), we don't bite. Well, if you'll go to debconf and tell me I'm your sponsor, I might bite (for all your mistakes from the past ;)

'''QUESTION:''' Maybe is a little off topic but it seems we have not so many questions, so... How do you (as team) managed relationship with upstream and downstream?

Well, the only policy we have is: talk with upstream: introduce yourself, forward her/him bugs, send patches. We also have quite some upstreams in the team, we try to help them a little bit more than we help others.

'''QUESTION:''' I've never used ''stdeb'', how would you compare it with ''dh_make''?

ANSWER: Well, ''dh_make'' doesn't know much about recent changes in Python policy, if any ;)

'''QUESTION:''' Do you have any suggestions for upstream authors? Anything that they should do to help the packaging?

ANSWER: [[http://wiki.debian.org/UpstreamGuide]], there's a Python section as well. There's also [[http://us.pycon.org/media/2010/talkdata/PyCon2010/038/paper.html]]. Stani (SPE and Phatch author) is one of out team members, BTW. He wrote ''Cross-platform Application Development and Distribution'' linked above with Nadia Alramli. You can also ask Runa (here on #debian-women) about Python stuff

About sponsors: you can cheat a little bit and check on this page: [[http://wiki.debian.org/SponsorChecklist]] what sponsors usually check.

'''QUESTION:''' I'm comparing the {{{debian/}}} directories build by {{{py2dsc}}} and {{{dh_make}}} for the same package, {{{py2dsc}}} is setting some fields in {{{debian/control}}} to reasonable values, including the {{{Buil-Depends}}}. It fetches some depends from {{{setup.py}}}, why is XS-Python-Version: current set? Does {{{XS-Python-Version:}}} current build for the default Python version or all the supported ones?

COMMENT: ''stdeb'' tries to parse some data from {{{setup.py}}} like package description. {{{dh_make}}} simply doesn't know much about distutils. And stdeb is... ''distutils'' ' extension written by Andrew Straw... who is DPMT member

ANSWER: Current is deprecated. It was used by python-central to bytecompile for one Python version only. But since it's better to simply use private directories, we dropped ''current'' keyword. I uploaded stdeb with few patches yesterday :), unexport CPPFLAGS and so on :D (version dedicated to #debian-women ;) )

I also uploaded new ''python-defaults'' to ''experimental'', with new features like {{{.pyinstall}}} or {{{.pyremove}}} to make packaging easier, but there were no questions about it. I mean, there were no problems that {{{.pyinstall}}} or {{{.pyremove}}} could solve.

== See also ==

 * [[http://wiki.debian.org/Teams/PythonModulesTeam/HowToJoin]]
 * [[http://svn.debian.org/viewsvn/python-modules/tools/]]
 * [[http://people.debian.org/~piotr/dc10_slides.pdf]]
 * [[http://lists.debian.org/debian-python/2010/11/msg00063.html]]
 * TODO: Read and extract helpful infos: [[http://stackoverflow.com/questions/1382569/how-do-i-do-debian-packaging-of-a-python-package]]
#redirect Python/LibraryStyleGuide
This guide is not up to date, redirecting to Python/LibraryStyleGuide

This guide is not up to date, redirecting to Python/LibraryStyleGuide