Using git for team packages
- git workflows
- Where do the team's git branches live?
- Tag style
- Use cases
- More information and resources
- Crazy Use Cases
- Packages No Longer Maintained By DPMT
Using git for team packages
On 2018-02-24, the DPMT and PAPT transition to Git for version controlling all of our packages has been complete. See below for details, workflow, policies, etc. Write access to the subversion repos has been TURNED OFF! You can still access them in read-only mode. We welcome back all Python (modules and app) developers who have moved their packages away from the team because of our previous use of subversion!
There are many advantages to team-based package development. The DPMT and PAPT have very excellent, knowledgeable, and experienced Python and Debian developers and our goal is to bring the very best Python experience to Debian users and developers. We're here to help each other and provide consensus (if not 100% agreement) on best practices. Collaboration has many benefits, and one of the things that aids effective collaboration is common tools and workflows surrounding packaging, while still providing individual maintainers the freedom to adapt those practices to their preferences, and the peculiarities of the packages.
One of the perks of being in a team is that all of our packages are managed in a similar way, using similar tools and workflows. This is great because any team member can immediately get the source of a package to do an update or a bug fix, and they don't have to first try to figure out how the package is version controlled. Plus, we can have nice wiki documentation which help new team members how to do things, and almost anyone on IRC can immediately give you advice. We use a common set of vcs procedures to ensure consistency among all of our packages. Both the DPMT and PAPT teams use the same version control system, layout, and workflows. The DPMT team migrated to Git on 2015-10-09, while the PAPT migrated on 2018-02-24.
It is a team requirement that all packages be managed using the same version control system.
Most of our git-based workflow is provided by gbp (formerly git-buildpackage). On top of gbp there's the patch management regime. Most Debian developers are familiar with quilt for including Debian-specific patches to upstream pristine tarballs. While it's still possible to use quilt directly of course, there is a better way to integrate the best of the git world with quilt. Of the two predominant git-based patch regimes, the Debian Python team has chosen git-dpm.
git-dpm imposes some restrictions and/or defaults on branch names, but you have some leeway. However, as a team we are strongly recommending you use the default branch names, specifically:
master - The Debianized upstream source directory. IOW, this contains upstream source and a debian/ packaging directory. It is a *source-full* checkout.
pristine-tar - Contains the standard pristine-tar deltas.
upstream - The un-Debianized upstream source. This is what you get when you unpack the upstream tarball.
If you use other branch names, please have a good reason (not just personal preference, remember - take one for the team!), and you MUST document the differences in debian/README.source. The default branch names will help the repo interoperate better with other tools.
git-dpm creates a patch branch when you want to directly edit upstream source files for export to quilt patches, however git-dpm deletes this patch branch when you export back to the master branch.
Q: What if I don't want to use git-dpm? A: Please be nice to your fellow teammates, and reconsider! We want team members to easily know what workflow to use, by common convention. In rare cases, if some deviation from the team default is required, you MUST have a good rationale, and both the reason and workflow MUST be documented in the README.source file.
Q: Source-full or source-less branches? A: Source-full branches please! There are lots of good reasons for this, including the ability to easily diff between upstream versions, say to double check that nothing untoward or undocumented has been added in a new upstream version. E.g. git checkout upstream and git diff upstream/0.11.3 would for example give you an overview of what's new in upstream's 0.11.4.
We require team maintained packages to contain the full upstream source, and we don't believe it will be an overwhelming performance or disk-space burden to do so. It is much easier to hack on packages, including generating the quilt patches, when a checkout of the repo gives you the full source. Source-less branches defeat much of the benefit of managing packages in git.
Where do the team's git branches live?
They can be checked out individually, or managed as a group, with mr. It is configured to not checkout all the team packages by default, but easily manage the ones that are checkout out, collectively.
$ sudo apt install mr $ git clone git+ssh://git.debian.org/git/python-modules/tools/python-modules.git $ cd python-modules $ echo $PWD/.mrconfig >> ~/.mrtrust $ ./checkout PACKAGENAME # For each package you care about
if your username differs between your local machine and Alioth, then you can add this snippet to ~/.ssh/config:
Host git.debian.org User ALIOTH_USERNAME
To update the .mr-config file, run
$ git pull $ ./make-mrconfig
And if there are any, commit and push changes.
To update all your checked out packages, run
To check out packages individually, you might be able to just
$ debcheckout -a <src-pkg-name>
but while the Vcs-* fields have been updated in the repository, new uploads with those fields still need to be done. In the meantime, you can:
$ gbp clone git+ssh://firstname.lastname@example.org/python-team/modules/<src-pkg-name>.git
gbp clone is preferable to git clone because gbp clone ensures that you'll have all the remote tracking branches set up properly. You can still use git clone but you'll have to check out the pristine-tar and upstream branches yourself.
You might also want to add this handy little convenience to your ~/.gitconfig file:
[url "git+ssh://email@example.com/python-team/modules/"] insteadof = dpmt:
which allows you to do this:
$ gbp clone dpmt:srcpkg.git
Creating new repositories
To set up a new repository here, you should follow these instructions. They require ssh access to Alioth, but if you have push permissions, you should have ssh permissions. Ask a team member for help if you are unable to do this. These instructions set up the hooks for commit notifications and such:
$ ssh git.debian.org $ cd /git/python-modules $ ./setup-repository <srcpkgname> "<srcpkgname> packaging" $ exit
Of course, use your own source package name for <srcpkgname>.
A:We will still require all team packages to be available on git.debian.org. Please do not host your team maintained packaging branches anywhere else; e.g. ?GitHub is not appropriate for the packaging branches because it is not free software. GitLab does have a free software Community Edition and someday it might be interesting to run one of those for DPMT or Debian-at-large, but for now please do not submit merge requests on gitlab.com.
Use these Vcs-* fields in the header of your debian/control file:
Vcs-Git: https://anonscm.debian.org/git/python-modules/packages/<srcpkgname>.git Vcs-Browser: https://anonscm.debian.org/git/python-modules/packages/<srcpkgname>.git
There seems to be several tag styles in common use, and git-dpm by default uses a '-' as separator between the branch and version number. However, we recommend using '/' as a separator, yielding tags like debian/0.2.0-1. To set the git-dpm tag style, edit the debian/.git-dpm file and put this at the end of the file:
debianTag="debian/%e%v" patchedTag="patched/%e%v" upstreamTag="upstream/%e%u"
The migration script automatically added this when the repo was converted from subversion, but please be sure to set it this way for any repository that existed before the migration.
Here are some common scenarios and use cases. There may be other ways to accomplish these tasks so if you find a more efficient way of doing something, please update these pages and discuss with the team via the debian-python mailing list. Feel free to add more use cases here too! Your experience is valuable.
Creating a new package
First, Initialize the package directory on git.debian.org to set up the bare repo (with hooks installed) that you will push your branch.
$ uscan # Download your package's upstream original tarball $ tar -xvf srcpkgname_1.0.orig.tar.gz $ cd srcpkgname_1.0 $ git init $ git add . $ git commit -m "import srcpkgname_1.0.orig.tar.gz" $ git checkout -b upstream $ pristine-tar commit ../srcpkgname_1.0.orig.tar.gz upstream $ git-dpm init ../srcpkgname_1.0.orig.tar.gz
You'll now be left on the master branch, i.e. the packaging branch. You'll also have upstream (raw tarball contents) and pristine-tar branch.
Fill in the rest of your debian/ directory, then do this:
$ git add debian/* $ debcommit # assuming you have a d/changelog entry, otherwise git commit $ git-dpm prepare $ git-dpm status
And then you still need to add by hand the tag format in debian/.git-dpm. Now you can build your package using whatever tools you like, including debuild, dpkg-buildpackage, and git-buildpackage.
Push your repo to git.debian.org:
$ git push --set-upstream ssh://git.debian.org/git/python-modules/packages/<srcpkgname>.git --all
At this point, I like to cd to a different directory and do a gbp clone ssh://git.debian.org/git/python-modules/packages/<srcpkgname>.git and then continue working from the <srcpkgname> directory. To prove that all the branches got pushed correctly, in this fresh clone, checkout the master, pristine-tar, and upstream branches.
Building and tagging
Once you've built and uploaded your package, you should tag the release.
$ git-dpm tag $ git push --tags
Q: should we mandate git-dpm style tag names or git-bp style tag names? Let's work with those projects to make them more consistent, or at least provide options.
New upstream release
$ git-dpm import-new-upstream --ptc --rebase-patched path-to-orig-tar-gz
Don't forget to update debian/changelog! git-dpm will not do it for you.
If you have merge conflicts in your patched branch, you will have to resolve them manually, then continue the rebasing until everything applies cleanly. However be aware that if this happens, your pristine-tar branch will not have been updated, despite the --ptc flag. Before you try to build your package, you'll need to do pristine-tar commit <path-to-orig.tar.gz> manually. It will also fail to tag the new upstream release, so another step to do by hand.
What if the new upstream introduces quilt patch conflicts? In that case, git-dpm will leave you in the patched branch, and you'll have to do the normal git conflict resolution dance. Once you've completed the git rebase operation, you will need to call git-dpm update-patches.
You will need to push the master branch and also the upstream and pristine-tar branches.
$ git push --all
Patching (i.e. adding quilt patches) is easy. You start by entering git-dpm's patch branch, make your changes to upstream code, commit, and then switch back to the master branch. git-dpm automatically converts every commit to a separate quilt patch file. If you want to squash several commits into single quilt patch file, see below for how to rebase. If you want to control the quilt patch file names instead of letting git-dpm automatically name your patch files, be sure to add Patch-Name: <whatever> to the quilt patch files (while in the master branch), and then do a git-dpm checkout-patched followed by a git-dpm update-patches.
In general, to patch your package do this:
# git-dpm c-p is an alias $ git-dpm checkout-patched <edit & commit> # git-dpm u-p is an alias $ git-dpm update-patches
Updating your quilt patch stack
Sometimes you want to re-order patches, squash patches, remove them, etc. You do this by entering git-dpm's patch branch and doing a rebase against upstream.
$ git-dpm checkout-patched $ git rebase -i upstream <edit> $ git-dpm update-patches
if you simply need to update your patch to the new upstream (sometimes you dont have conflicts for this) what you really want is:
$ git-dpm checkout-patched <edit + commit> $ git rebase -i upstream # so that you can squash your last commit $ git-dpm update-patches
Cherry picking upstream commits
Sometimes you need to cherry pick upstream commits, e.g. to fix a bug in the Debian version when upstream hasn't yet released a new version. A convenient way to do that, if upstream is developed in git, is to add a remote and cherry pick from that. For example:
$ git remote add upstream_repo git://example.com/foo.git $ git fetch upstream_repo $ git-dpm checkout-patched $ git cherry-pick any_upstream_commit $ git-dpm update-patches
Sponsoring, mentoring, reviewing
More information and resources
Post-migration clean up
The migration script from subversion to git could not automatically convert all packages. Also, some packages may have already been maintained in git. Here are some things you can do to help clean things up.
Review the whiteboard list of packages that failed to convert automatically, and try to do a manual conversion for the packages you care about. Remember, you can't write to the subversion repos any more. The whiteboard has further instructions for manual conversion. Please cross off the packages you successfully convert manually.
- If your package was already in git, the previous repo may have been moved aside. To move your original repo back, do the following:
$ ssh alioth.debian.org $ cd /git/python-modules $ ls pre-migration-packages <if your package is listed> $ mv packages/<your-package>.git post-migration-packages $ mv pre-migration-packages/<your-package>.git packages
Note that it's possible there will be no packages/<your-package>.git directory if your package was never in subversion. You still need to move it from the pre-migrations-packages directory to the packages directory!
Recommended/required configuration settings
Both git itself and git-bp provide ways to configure defaults. Settings that are personal to you should live only in $HOME/.gitconfig or $HOME/.gbp.conf. Settings which affect how the team interoperates should live in <repo>/debian/gbp.conf so that anyone who checks out the branch will see the same settings.
Converting from svn to git
Crazy Use Cases
BarryWarsaw is working on an update to pip 7 and had some unrelease commits in the aloith svn. Because pip 7 is blocked on NEW dependencies, he wanted to fix some bugs in 1.5.6 so decided to create a 1.5.6 branch. However, he also wanted to preserve the unreleased svn commits. Here's how he did it, using Stefano's converted git repository.
$ gbp clone ssh://git.debian.org/git/python-modules/svn-migration/python-pip.git gitsvn $ cd gitsvn $ git co debian/1.5.6-5 $ git checkout -b release-1.5.6 $ git-dpm init ../src/python-pip_1.5.6.orig.tar.gz $ git-dpm prepare $ git co master $ git merge origin/svn $ git checkout release-1.5.6
Now he can work on python-pip 1.5.6-6. Then later, when he wants to continue working on pip 7, he'll need to switch to the master branch and do another git-dpm init with the pip 7 orig.tar.gz. Since this will result in conflicts (due to changes in upstream pip), the conflicts need to be resolved and then master should be properly prepared to use git-dpm for a new upstream release.
Packages No Longer Maintained By DPMT
For obsolete repositories, you should follow these instructions. They require ssh access to Alioth, but if you have push permissions, you should have ssh permissions. Ask a team member for help if you are unable to do this. This will preserve a historical copy of the repository for a package that has been either removed or is no longer maintained in DPMT (in case it comes back):
$ ssh git.debian.org $ cd /git/python-modules $ mv packages/<srcpkgname>.git/ attic $ exit