Differences between revisions 34 and 35
Revision 34 as of 2015-06-02 18:32:04
Size: 15105
Editor: BarryWarsaw
Comment:
Revision 35 as of 2015-06-03 01:06:38
Size: 15066
Editor: PaulWise
Comment: DebianBug
Deletions are marked like this. Additions are marked like this.
Line 164: Line 164:
 * If you have merge conflicts in your patched branch, you will have to resolve them manually, then continue the rebasing until everything applies cleanly. ''However'' be aware that if this happens, your `pristine-tar` branch will [[http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787558|not have been updated]], despite the `--ptc` flag. Before you try to build your package, you'll need to do `pristine-tar commit <path-to-orig.tar.gz>` manually.  * If you have merge conflicts in your patched branch, you will have to resolve them manually, then continue the rebasing until everything applies cleanly. ''However'' be aware that if this happens, your `pristine-tar` branch will [[DebianBug:787558|not have been updated]], despite the `--ptc` flag. Before you try to build your package, you'll need to do `pristine-tar commit <path-to-orig.tar.gz>` manually.

Using git for team packages

This page is UNOFFICIAL

As of this writing (2014-08-22), the Debian/Python team still officially uses Subversion to manage all team packages. The contents of this page are PROPOSED as a team policy and recommendations for using git to manage team package. It is still under active discussion on the debian-python mailing list. Please provide your feedback there.

There are many advantages to team-based package development. The DPMT and PAPT have very excellent, knowledgeable, and experienced Python and Debian developers and our goal is to bring the very best Python experience to Debian users and developers. We're here to help each other and provide consensus (if not 100% agreement) on best practices. Collaboration has many benefits, and one of the things that aids effective collaboration is common tools and workflows surrounding packaging, while still providing individual maintainers the freedom to adapt those practices to their preferences, and the peculiarities of the packages.

One of the perks of being in a team is that all of our packages are managed in a similar way, using similar tools and workflows. This is great because any team member can immediately get the source of a package to do an update or a bug fix, and they don't have to first try to figure out how the package is version controlled. Plus, we can have nice wiki documentation which help new team members how to do things, and almost anyone on IRC can immediately give you advice. We use a common set of vcs procedures to ensure consistency among all of our packages. Both the DPMT and PAPT teams use the same version control system, layout, and workflows.

It is a team requirement that all packages be managed using the same version control system. As of this writing, that vcs is Subversion (svn).

We recognize however, that modern vcses, especially of the distributed variety, provide a lot of additional benefit for packagers. Such vcses include git, Mercurial (hg), and Bazaar (bzr), among others. git seems to be the most popular, so we are investigating using it as a possible replacement for svn for team managed projects.

We propose to relax this requirement to allow for experimentation with git-based team packages, but will eventually re-mandate one-dvcs-to-rule-them-all. I.e. an official switch to git at some point in the future.

git workflows

Most of the git-based workflow is provided by gbp (formerly git-buildpackage). On top of gbp there's the *patch management regime*. Most Debian developers are familiar with quilt for including Debian-specific patches to upstream pristine tarballs. While it's still possible to use quilt directly of course, there are two predominant git-based patch regimes that integrate well with quilt and are in use within Debian for package management, git-dpm and gbp-pq. While there is some overlap and similarities, there are some subtle differences both technically and in ease-of-use that makes it useful for the team to mandate one or the other regimes. You can gain some background on their applicability to team packages by reading these threads in the mailing list: git-dpm and gbp-pq. dgit is an interesting possibility, since its repositories reflect the exact representation of the package in the Debian archives.

Both git-dpm and gbp-pq impose some restrictions and/or defaults on branch names, but you have some leeway. However, as a team we are strongly recommending you use the default branch names, specifically:

  • master - The Debianized upstream source directory. IOW, this contains upstream source and a debian/ packaging directory. It is a *source-full* checkout.

  • pristine-tar - Contains the standard pristine-tar deltas.

  • upstream - The un-Debianized upstream source. This is what you get when you unpack the upstream tarball.

If you use other branch names, please have a good reason (not just personal preference, remember - take one for the team!), and you must document the differences in debian/README.source. The default branch names will help the repo interoperate better with other tools.

Both git-dpm and gbp-pq create a patch branch when you want to directly edit upstream source files for export to quilt patches, however git-dpm deletes this patch branch when you export back to the master branch. gbp-pq keeps its patch branch.

Q: What should we recommend/mandate about the patch branch? Some possibilities include: use a specific name, defer to the helper tool, or prohibit patch branches from being pushed to the public repos.

Team workflow

Q: Do we even need to pick one? I think we do. The two patch regimes don't interoperate. We want team members to easily know what workflow to use, by common convention. In rare cases, if some deviation from the team default is required, it must be documented in the README.Debian file.

Source-full or source-less branches?

Source-less (or debian/-only) closely mirror the team conventions under svn, but [this assertion needs justification] it seems counter-productive to have source-less branches under git.

  • BenFinney asks: What is the justification for this assertion? A number of Debian Python developers have stated they find the debian/-only workflow to be quite productive — not for performance or disk-space reasons, but for clarity and separation of concerns. So how is this workflow “counter-productive”? Counter to what, exactly?

    BarryWarsaw answers: Source-full checkouts are much more convenient especially in the face of a good patch management regime such as git-dpm. When a git clone gives you everything you need, you can easily view all the code in one place, test changes, enter a patch branch, make changes in place, and turn edits to the upstream code into quilt patches. It's a very natural and convenient way to work on packages, especially compared to the debian/-only style of our svn repos. With the latter, you still need to grab the upstream source somehow, then unpack it, arrange for it to be "debianized", and then deal with the quilt patch stack. It might be worthwhile to have a discussion of workflows on the mailing list, but source-full git repos are amazingly productive, easy and natural to use.

    BenFinney: Well, none of that argues *against* source-less, so it doesn't justify the assertion above. Let's discuss it on the ‘debian-python’ mailing list.

    BarryWarsaw: I realized another super-useful feature of source-full checkouts is that it makes it really easy to see the diffs between upload versions in the upstream code. E.g. if you use git-dpm tag, when you import an new orig.tar.gz (say 0.11.4) you can just git checkout upstream and git diff upstream/0.11.3 (for example).

We require team maintained packages to contain the full upstream source, and we don't believe it will be an overwhelming performance or disk-space burden to do so. It is much easier to hack on packages, including generating the quilt patches, when a checkout of the repo gives you the full source. Source-less branches defeat much of the benefit of managing packages in git.

Where do the team's git branches live?

Alioth, under the python-modules directory. To set up a new repository here, you should follow these instructions. They require ssh access to Alioth, but if you have push permissions, you should have ssh permissions. Ask a team member for help if you are unable to do this. These instructions set up the hooks for commit notifications and such:

$ ssh git.debian.org
$ cd /git/python-modules
$ ./setup-repository <srcpkgname> "<srcpkgname> packaging"
$ exit

Of course, use your own source package name for <srcpkgname>.

We will still require all team packages to be available on git.debian.org. Please do not host your team maintained packaging branches anywhere else; e.g. github is not appropriate for the packaging branches because it is not free software. Q: should we honor pull requests and allow for mirrors on github?

Use these Vcs-* headers in your debian/control file:

Vcs-Git: git://anonscm.debian.org/python-modules/packages/<srcpkgname>.git
Vcs-Browser: http://anonscm.debian.org/cgit/python-modules/packages/<srcpkgname>.git

Tag style

There seems to be several tag styles in common use, and git-dpm by default uses a '-' as separator between the branch and version number. However, we recommend using '/' as a separator, yield tags like debian/0.2.0-1. Here are the commands to set the git-dpm tag style:

$ git config dpm.debianTag debian/%e%v
$ git config dpm.patchedTag patched/%e%v
$ git config dpm.upstreamTag upstream/%e%u

Any migration scripts we implement will select this tag style automatically, but please be sure to set it this way for consistency in any new repositors, or even globally (by adding the --global flag to the git config commands).

Transition plan

The rough steps to migrate the team would then be:

  • Pick a patch regime (maybe), e.g. git-dpm or gbp-pq.

  • Relax the constraint that all packages must be under one vcs. All new packages should be under git. Allow maintainers to migrate packages from svn to git whenever is convenient for them (e.g. when preparing a new upstream).
  • After the grace period, migrate remaining svn packages en mass.

Use cases

Here are some common scenarios and use cases. There may be other ways to accomplish these tasks so if you find a more efficient way of doing something, please update these pages and discuss with the team via the debian-python mailing list. Feel free to add more use cases here too! Your experience is valuable.

Creating a new package

First, Initialize the package directory on git.debian.org to set up the bare repo (with hooks installed) that you will push your branch.

Next, you can either ?import an existing .dsc file or start from scratch. Here's are the steps from the manpage that seem to work well:

$ uscan   # Download your package's upstream original tarball
$ tar -xvf srcpkgname_1.0.orig.tar.gz
$ cd srcpkgname_1.0
$ git init
$ git add .
$ git commit -m "import srcpkgname_1.0.orig.tar.gz"
$ git checkout -b upstream
$ pristine-tar commit ../srcpkgname_1.0.orig.tar.gz upstream
$ git-dpm init ../srcpkgname_1.0.orig.tar.gz

You'll now be left on the master branch, i.e. the packaging branch. You'll also have upstream (raw tarball contents) and pristine-tar branch.

Fill in the rest of your debian/ directory, then do this:

$ git add debian/*
$ debcommit  # assuming you have a d/changelog entry, otherwise git commit
$ git-dpm prepare
$ git-dpm status

You can now build your package using whatever tools you like, including debuild, dpkg-buildpackage, and git-buildpackage.

Push your repo to git.debian.org:

$ git push --set-upstream ssh://git.debian.org/git/python-modules/packages/<srcpkgname>.git --all

At this point, I like to cd to a different directory and do a git clone ssh://git.debian.org/git/python-modules/packages/<srcpkgname>.git and then continue working from the <srcpkgname> directory. To prove that all the branches got pushed correctly, in this fresh clone, checkout the master, pristine-tar, and upstream branches.

Building and tagging

Once you've built and uploaded your package, you should tag the release.

$ git-dpm tag
$ git push --tags

Q: should we mandate git-dpm style tag names or git-bp style tag names? Let's work with those projects to make them more consistent, or at least provide options.

Patching

New upstream release

$ git-dpm import-new-upstream --ptc --rebase-patched path-to-orig-tar-gz

Important notes:

  • Don't forget to update debian/changelog! git-dpm will not do it for you.

  • If you have merge conflicts in your patched branch, you will have to resolve them manually, then continue the rebasing until everything applies cleanly. However be aware that if this happens, your pristine-tar branch will not have been updated, despite the --ptc flag. Before you try to build your package, you'll need to do pristine-tar commit <path-to-orig.tar.gz> manually.

Sponsoring, mentoring, reviewing

- pull requests

More information and resources

Recommended/required configuration settings

Both git itself and git-bp provide ways to configure defaults. Settings that are personal to you should live only in $HOME/.gitconfig or $HOME/.gbp.conf. Settings which affect how the team interoperates should live in <repo>/debian/gbp.conf so that anyone who checks out the branch will see the same settings.

Converting from svn to git

Crazy Use Cases

python-pip

BarryWarsaw is working on an update to pip 7 and had some unrelease commits in the aloith svn. Because pip 7 is blocked on NEW dependencies, he wanted to fix some bugs in 1.5.6 so decided to create a 1.5.6 branch. However, he also wanted to preserve the unreleased svn commits. Here's how he did it, using Stefano's converted git repository.

$ gbp clone ssh://git.debian.org/git/python-modules/svn-migration/python-pip.git gitsvn
$ cd gitsvn
$ git co debian/1.5.6-5
$ git checkout -b release-1.5.6
$ git-dpm init ../src/python-pip_1.5.6.orig.tar.gz
$ git-dpm prepare
$ git co master
$ git merge origin/svn
$ git checkout release-1.5.6

Now he can work on python-pip 1.5.6-6. Then later, when he wants to continue working on pip 7, he'll need to switch to the master branch and do another git-dpm init with the pip 7 orig.tar.gz. Since this will result in conflicts (due to changes in upstream pip), the conflicts need to be resolved and then master should be properly prepared to use git-dpm for a new upstream release.


CategoryGit