Differences between revisions 66 and 67
Revision 66 as of 2013-05-22 18:15:42
Size: 18925
Editor: ?markus tornow
Comment: i changed the link, so it points straight to the packaging workflow overview instead of the intro about packaging diaspora (same site, different headline).
Revision 67 as of 2013-05-22 18:18:06
Size: 18922
Editor: ?markus tornow
Comment:
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
This page is quite disorganized, and some of it was written as a "plans for Ruby in wheezy page", not as a general documentation for packaging. Please do not hesitate to improve this page. Another effort to document the workflow of packaging ruby-gems for Debian is made by the debian-diaspora Team: http://wiki.debian.org/Diaspora/Packaging#Step_2_-_Build_a_debian_package , which is focused at beginners and very detailed. This page is quite disorganized, and some of it was written as a "plans for Ruby in wheezy page", not as a general documentation for packaging. Please do not hesitate to improve this page. Another effort to document the workflow of packaging ruby-gems for Debian is made by the debian-diaspora Team: [[Diaspora/Packaging#Step_2_-_Build_a_debian_package/|this wiki page]] , which is focused at beginners and very detailed.

This page describes the current packaging practices for Ruby in Debian. For more general information, see this wiki page.

This page is quite disorganized, and some of it was written as a "plans for Ruby in wheezy page", not as a general documentation for packaging. Please do not hesitate to improve this page. Another effort to document the workflow of packaging ruby-gems for Debian is made by the debian-diaspora Team: this wiki page , which is focused at beginners and very detailed.

Joining the pkg-ruby-extras team

We always welcome people to help us out. If you want to, please follow the following guidelines.

  • Contact us first on our mailing list or on IRC.

  • Register an alioth account account.

  • Go to our project page and request your addition to the team. Please specify what you plan to work on.

  • Wait until we approve your addition. Note that there’s a delay before getting your account on svn.debian.org, see the Alioth FAQ for details.

  • Read all the docs (sorry).
  • Enjoy!

Using Git

We use git together with mr (Multiple Repository management tool) to track the git repositories for each package.

Obtaining the master repository

git clone <alioth-username>@git.debian.org:/git/pkg-ruby-extras/pkg-ruby-extras.git

Then, to obtain all the team packages, do:

cd pkg-ruby-extras
# look at .mrconfig
apt-get install mr
mr checkout

To enable parallel checkouts, use

mr -j 5 checkout

To avoid typing your username and your password see this page.

Note that this does not work with version 0.51 or earlier of mr (0.49 is the version shipped in squeeze). The mr in wheezy (1.03) does work.

Creating a new git repository and pushing an existing package to git.debian.org

Note: please do not leave empty repositories on alioth; only create a repository there just before pushing your package.

On git.debian.org:

cd /git/pkg-ruby-extras
./setup-repository ruby-feedparser

On your local machine:

Get a basic .dsc from your package, using:
gem2deb ruby-feedparser-0.7.gem

At this point, please do not make any change to the package generated by gem2deb. It is important to track all the changes in git. Then:

cd where-you-cloned-pkg-ruby-extras.git
./make-mrconfig # will generate the config for your new repo
mr checkout # will clone your new repo
cd ruby-feedparser
apt-get install git-buildpackage
git import-dsc --pristine-tar path-to/ruby-feedparser_0.7-1.dsc
git tag -d debian/0.7-1 # because the package is not ready yet
git push --all
git push --tags

Then, make the needed changes to the packaging, and then

git push

After upload, tag the package with

git tag debian/0.7-1

And don't forget to

git push --tags

Updating a package to a newer version

Fetch the most recent version of a gem, and make a tarball out of it

gem2tgz <gemname>

where <gemname> is the upstream name of the gem.

If you have already downloaded a specific version of the gem, you can make a tarball out of it

gem2tgz <gemname>-<version>.gem

If the debian/watch file points to the gemwatch service, you can also use uscan to download the new tarball automatically generated by gemwatch.

You should now have a tarball <gemname>-<version>.tar.gz. You can now import this tarball in the Git repository: go to the directory containing the Git repository of the package for gemname, and run

git import-orig --pristine-tar path/to/<gemname>-<version>.tar.gz

This should update the upstream branch with the content of the tarball, tag the new upstream version on the upstream branch, and commit the tarball in the pristine-tar branch. Now update your packaging in the master branch. Once you are done, push the branches and the tags to the team repository

git push --all
git push --tags

Known Issues

If mr checkout gives an error like mr: illegal section "[DEFAULT]" in untrusted:

Add .mrconfig path to ~/.mrtrust

If your local username is different from your Alioth username (that's probably the case for non-DD developers with -guest accounts), add the following to your ~/.ssh/config:

Host alioth.debian.org svn.debian.org git.debian.org
User alioth-username

Handling patches on upstream code

If the package you are working on includes upstream patches (i.e. debian/patches/*), you need to make sure that the git repository is in a sane state after build (i.e. no uncommited changes to the work tree). This can be done in two ways:

1) Tell dpkg-source to unapply the patches after the build (source: blog post by Raphael Hertzog):

$ echo "unapply-patches" >>debian/source/local-options
$ git add debian/source/local-options
$ git commit -m "Unapply patches after build"

2) Also apply the patches to the git tree, so that they do not need to be applied again during build. The patches must be kept under debian/patches/*. This solution is discouraged, as (1) will be the default in a future dpkg version.

References

Guidelines for Ruby packaging

Single package for all Ruby versions

For Wheezy, Ruby software must not come in separate packages per Ruby interpreter version, as we did until Squeeze (libfoo-ruby1.8, libfoo-ruby1.9.1 etc). We are going to use a single binary package per source package no matter what. In the case of native extensions, those single binary packages will contain the compiled extensions (*.so) for all supported Ruby versions. This is supported by gem2deb, so you should use it.

Naming of ruby packages

The current libfoo-ruby naming, inspired for perl, is not really great. It would be better to move to the python naming, with ruby-foo. Since all packages will have to be modified anyway to switch to gem2deb (see below), it is the perfect time to do that.

The guidelines are the following:

  • Binary packages must normally be named "ruby-foo". If the package is mainly used as an application (not as a library), then it can be named "foo". Known examples are rails, chef, rubygems, puppet.
  • Source packages must have the same name as the "main" binary package. (our infrastructure is better at handling this case)

Renaming existing packages

Existing packages must be renamed to the new scheme. This renaming must be done using the standard Debian practices (i.e. Debian Developers' Reference, section 5.9.3). In our case, this means:

  • when libfoo-ruby gets renamed to ruby-foo, then ruby-foo must declare Provides: libfoo-ruby, Replaces: libfoo-ruby and Breaks: libfoo-ruby

  • when libfoo-ruby1.8 gets renamed to ruby-foo, then ruby-foo must declare Provides: libfoo-ruby1.8, Replaces: libfoo-ruby1.8, and Breaks: libfoo-ruby1.8

  • when libfoo-ruby1.8 and libfoo-ruby1.9.1 are turned into a single pure-Ruby package ruby-foo, then it must declare Provides: libfoo-ruby, libfoo-ruby1.8, libfoo-ruby1.9.1, Replaces: libfoo-ruby, libfoo-ruby1.8, libfoo-ruby1.9.1, and Breaks: libfoo-ruby, libfoo-ruby1.8, libfoo-ruby1.9.1

  • ruby-foo must also provide transitional binary packages for libfoo-ruby, libfoo-ruby1.8 and libruby1.9.1 (when it's the case). You can use the gen-ruby-trans-pkgs script to generate the proper snippet to be added to your debian/control.

  • note that this is only needed for packages that were already released with Squeeze. New packages must not keep any references to packages named as lib*-ruby.
  • after the renamed package enters the archive, please request the removal of the old package.

    • I guess this is not needed when using transitional packages - according to Renaming_a_Package the old source package will be removed automatically. If that's the case, the above bullet point, as well as this one, can be removed.

Handling of shebangs

  • applications willing to force the use of ruby1.8 should use /usr/bin/ruby1.8 in shebang, and depend on ruby1.8

  • applications willing to use the selected ruby implementation (whatever it is) should use /usr/bin/env ruby, and depend on ruby | ruby-interpreter

gem2deb currenly does the following:

  • If the XS-Ruby-Versions field contains 'all', all shebangs of programs installed to /usr/bin are rewritten to /usr/bin/env ruby

  • Otherwise, all shebangs of programs installed to /usr/bin are rewritten to the binary corresponding to the first version listed in XS-Ruby-Versions. For example, is XS-Ruby-Versions is ruby1.8, ruby1.9.1, all shebangs will be rewritten to /usr/bin/ruby1.8

Values of XS-Ruby-Versions

Currently the XS-Ruby-Versions values matches the name of the ruby interpreter, e.g. current supported values are:

  • ruby1.8
  • ruby1.9.1

When more ruby interpreters will be added, the list will be expanded.

gem2deb as the preferred packaging tool for ruby software

gem2deb is a replacement for our current cdbs-based packaging suite. It:

  • does almost everything automatically
  • uses dh (much faster)
  • is easier to adapt to our needs, since there's no dependency on an external tool
  • runs the test suite as part of the build, for each ruby implementation (yes, it's possible to override this). See more about running tests

  • uses a single binary package for native libraries, instead of one binary package for each ruby implementation

To test it:

git clone git://github.com/ln/gem2deb.git
cd gem2deb
rake install
then, cd to some temporary place, and do:
gem2deb rdiscount # native gem
or:
gem2deb i18n # pure-ruby gem

It serves two goals:

  • it can be used by users to generate .debs from gems locally
  • it generates Debian source package from which we can do the packaging work

Prerequisites for migrations:

  • we need to decide on a workflow for packaging (switch to git?)
  • we need to decide on a migration plan to avoid breaking the archive for too long

Success metrics for the transition to gem2deb

  • There are no non-transitional binary packages named lib.*-ruby.* in the archive. No package should Depend or Recommend a package named lib.*-ruby.*.
  • The only packages installing stuff to /usr/lib/ruby/{1.8,1.9.1} are interpreters. Software packaged with gem2deb will install to /usr/lib/ruby/vendor_ruby/
  • Upgrades from squeeze work. ;)

  • The Ruby community stops whining about Debian ;-) (just kidding. that's not really an achievable goal)

Howto: converting a package from ruby-pkg-tools to gem2deb

This section describes how to convert the libfeedparser-ruby package to gem2deb. That package is simple, but still gives a good overview of the process. Please improve this section with your own experience.

  1. Get the libfeedparser-ruby source package. (apt-get source libfeedparser-ruby)
  2. Find the name of the rubygems on rubygems.org. It is ruby-feedparser.
  3. Run gem2deb ruby-feedparser. This generates a basic (but working) source package.
  4. Cd to ruby-feedparser-0.7/
  5. Copy the changelog entries from the libfeedparser-ruby source package to debian/changelog
  6. Generate the templates for transitional packages: run gen-ruby-trans-pkgs libfeedparser-ruby > /tmp/templates. Edit debian/control: the Replaces, Breaks, Provides go to the ruby-feedparser binary package, and the other binary packages need to be added at the end of debian/control.

  7. Fill-in debian/control: description, homepage, build-dependencies, ...
  8. Copy debian/copyright from libfeedparser-ruby, and review it. It is a good idea to use that opportunity to convert it to DEP5.
  9. Edit ruby-feedparser.docs
  10. Find how the test suite needs to be run, and edit debian/ruby-tests.rb accordingly. There are many examples in the packages maintained by the pkg-ruby-extras team.
  11. Build the package, make sure everything works (build in a clean chroot, run lintian, etc, etc).
  12. Import it into the pkg-ruby-extras git repository.
  13. Ask for review and sponsorship.

Requesting Sponsorship

Once your package is ready (or at least, once you think it is), a DD will have to review your changes and upload it to the archive. It is important to understand that reviewing and sponsoring packages is a tedious process, and that most DDs would prefer to do something else instead. So, make sure that you make the best possible use of their time, so they will be happy to sponsor you again. :-)

First, make sure that your package is really in good shape. That means answering at least the following questions:

  • Does my package build fine in a clean chroot? The easiest way to test that is using pbuilder.

  • Is my package lintian-clean? If not, did I explain why it isn’t in debian/changelog? Note that real problems should not be hidden by lintian overrides.
  • Did I check that all usages of rubygems have been patched out?
  • Is my debian/watch file correct? That file will be used by your sponsor to fetch the upstream source, so it is very important that it is correct. Please check that uscan --download-current-version does the right thing.

  • Is the Git repo up-to-date? Did I commit and push my changes?
  • Does my package actually work? Please check in a clean chroot, it’s easy to miss some @require@s.

Once your package is ready, you must:

  • Change the distribution from UNRELEASED to unstable.

  • Request sponsorship by mailing debian-ruby@lists.debian.org. Here is an example email:

Subject: RFS: libfoo-ruby 1.2-1, libbar-ruby 2.0-1

Hi,

The following packages are ready to be uploaded (I also verified the
points listed on http://wiki.debian.org/Teams/Ruby/Packaging#Requesting_Sponsorship).

Could you please sponsor them?

 libfoo-ruby 1.2-1
 libbar-ruby 2.0-1

Thank you!

OBSOLETE: Using the Subversion repository (obsolete because we are migrating to Git)

Follow the instructions for building the pkg-ruby-extras packages.

If you need general help with Subversion, an online book is available.

Repository structure

Our repository is structured like this:

pkg-ruby-extras
+-- trunk
    +-- libfoo-ruby
    +-- libbar-ruby
    +-- ...
    +-- tools
    +-- website
+-- branches
+-- tags

All our packages will live inside the trunk/ directory. When uploaded, the version being uploaded is copied to tags/.

First, you have to build your package outside of the SVN tree. You should get something like that:

.
|-- libfeedparser-ruby-0.1/ (build dir)
|-- libfeedparser-ruby1.8-dev_0.1-1_all.deb
|-- libfeedparser-ruby1.8_0.1-1_all.deb
|-- libfeedparser-ruby_0.1-1.diff.gz
|-- libfeedparser-ruby_0.1-1.dsc
|-- libfeedparser-ruby_0.1-1_i386.changes
|-- libfeedparser-ruby_0.1.orig.tar.gz
`-- libxfeedparser-ruby_0.1-1_all.deb

Then just do:

svn-inject -v -l 2 -o libfeedparser-ruby_0.1-1.dsc \
       svn+ssh://LOGIN@svn.debian.org/svn/pkg-ruby-extras/

You should not try to import your package manually: this handles a lot of small gory details you do not want to know.

N.B. Make sure that you have registered a SSH key, else you will be prompted for your password a good dozen of times.

Checking out

You can check a package out by using:

$ svn checkout svn+ssh://svn.debian.org/svn/pkg-ruby-extras/trunk/libfoo-ruby

Alternatively, you can check out all the current versions of all packages (takes less than 50 MB):

$ svn checkout svn+ssh://svn.debian.org/svn/pkg-ruby-extras/trunk

Building with svn-buildpackage

First, you have to change do the trunk directory:

$ cd libfoo-ruby/

You need upstream sources to build the package. Use uscan to download them:

$ uscan --download-current-version --destdir ../tarballs/

or just tell svn-buildpackage to automatically download upstream sources when needed. It can be done by adding svn-download-orig to ~/.svn-buildpackage.conf.

Inside libfoo-ruby/, you can build the package with svn-buildpackage:

$ svn-buildpackage -rfakeroot

Releasing and tagging

After releasing the package, its current state must be tagged to keep the exact source used to build the uploaded package. This can be done from the package directory with (assuming the released version is 0.1-1):

$ svn copy trunk/libfoo-ruby tags/libfoo-ruby/0.1-1

Alternatively, you can tell svn-buildpackage to automatically tag the release after building using the option --svn-tag. You can also optionally use debcommit -r if you have the devscripts package installed.

Attention: use this option only when you are sure that this build will be the final build, just before upload.

Package upload checklist

  1. Update your repository (just in case).
  2. Rebuild the package (without --svn-ignore-new!).

  3. Check everything is alright and do whatever tests you always do.
  4. Change the latest changelog entry distribution to “unstable” (or whatever is appropriate).
  5. Rebuild again with pbuilder!

  6. Perform small checks, final lintian & linda check.

  7. Upload the package.
  8. Tag with svn-buildpackage --svn-tag-only.

  9. Commit the new changelog entry for the UNRELEASED distribution.

We will soon release a bash source file with useful aliases. If you are interested, write to our mailing list, pkg-ruby-extras-maintainers (at lists.alioth.debian.org).

OBSOLETE: Packaging with ruby-pkg-tools (obsolete because we are migrating to gem2deb)

See ?/RubyExtras/RubyPackageTools