Differences between revisions 45 and 91 (spanning 46 versions)
⇤ ← Revision 45 as of 2015-10-09 14:28:31 →
Size: 14072
Editor: BarryWarsaw
Comment:
← Revision 91 as of 2022-04-27 03:59:36 → ⇥
Size: 15418
Editor: ?vimer
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
#language en
Line 5: Line 6:
{{{#!wiki caution
'''This page is OFFICIAL'''

On 2015-10-09, the DPMT transition to Git for version controlling all of our packages has been complete. See below for details, workflow, policies, etc. Write access to the subversion repos has been TURNED OFF! You can still access them in read-only mode. We welcome back all Python developers who have moved their packages away from the team because of our previous use of subversion!
}}}

There are many advantages to team-based package development. The DPMT and PAPT have very excellent, knowledgeable, and experienced Python and Debian developers and our goal is to bring the very best Python experience to Debian users and developers. We're here to help each other and provide consensus (if not 100% agreement) on best practices. Collaboration has many benefits, and one of the things that aids effective collaboration is common tools and workflows surrounding packaging, while still providing individual maintainers the freedom to adapt those practices to their preferences, and the peculiarities of the packages.

One of the perks of being in a team is that all of our packages are managed in a similar way, using similar tools and workflows. This is great because any team member can immediately get the source of a package to do an update or a bug fix, and they don't have to first try to figure out how the package is version controlled. Plus, we can have nice wiki documentation which help new team members how to do things, and almost anyone on IRC can immediately give you advice. We use a common set of vcs procedures to ensure consistency among all of our packages.  Both the [[Teams/PythonModulesTeam|DPMT]] and [[Teams/PythonAppsPackagingTeam|PAPT]] teams use the same version control system, layout, and workflows.		 There are many advantages to team-based package development. The DPT has very excellent, knowledgeable, and experienced Python and Debian developers and our goal is to bring the very best Python experience to Debian users and developers. We're here to help each other and provide consensus (if not 100% agreement) on best practices. Collaboration has many benefits, and one of the things that aids effective collaboration is common tools and workflows surrounding packaging, while still providing individual maintainers the freedom to adapt those practices to their preferences, and the peculiarities of the packages.

One of the perks of being in a team is that all of our packages are managed in a similar way, using similar tools and workflows. This is great because any team member can immediately get the source of a package to do an update or a bug fix, and they don't have to first try to figure out how the package is version controlled. Plus, we can have nice wiki documentation which help new team members how to do things, and almost anyone on IRC can immediately give you advice. We use a common set of vcs procedures to ensure consistency among all of our packages.
Line 17: Line 12:
== git workflows ==

Most of our git-based workflow is provided by [[https://honk.sigxcpu.org/piki/projects/git-buildpackage/|gbp]] (formerly `git-buildpackage`). On top of `gbp` there's the *patch management regime*. Most Debian developers are familiar with `quilt` for including Debian-specific patches to upstream pristine tarballs. While it's still possible to use `quilt` directly of course, there is a better way to integrate the best of the git world with quilt. '''Of the two predominant git-based patch regimes, the Debian Python team has chosen [[PackagingWithGit/GitDpm|git-dpm]].'''

`git-dpm` imposes some restrictions and/or defaults on branch names, but you have some leeway. However, as a team we are '''strongly recommending''' you use the default branch names, specifically:

 * {{{master}}} - The Debianized upstream source directory. IOW, this contains upstream source and a {{{debian/}}} packaging directory. It is a *source-full* checkout.		 == Policies ==

=== Git Repository ===

[[https://salsa.debian.org]], under the [[https://salsa.debian.org/python-team|python-team]]

=== Git Workflows ===

Most of our git-based workflow is provided by [[https://honk.sigxcpu.org/piki/projects/git-buildpackage/|gbp]] (formerly `git-buildpackage`). On top of `gbp` there's the ''patch management regime''. Most Debian developers are familiar with `quilt` for including Debian-specific patches to upstream pristine tarballs. While it's still possible to use `quilt` directly of course, there is a better way to integrate the best of the git world with quilt. '''Of the two predominant git-based patch regimes, the Debian Python team has chosen gbp pq.'''

=== Git Branch Names ===

As a team we are '''strongly recommending''' you use [[http://dep.debian.net/deps/dep14/|DEP-14]] style branch names, specifically:

 * {{{debian/master}}} - The Debianized upstream source directory. In other words, this contains upstream source and a {{{debian/}}} packaging directory. It is a ''source-full'', ''patches-unapplied'' checkout.
Line 25: Line 28:
 * {{{upstream}}} - The un-Debianized upstream source. This is what you get when you unpack the upstream tarball.

If you use other branch names, please have a good reason (not just personal preference, remember - take one for the team!), and you must document the differences in {{{debian/README.source}}}. The default branch names will help the repo interoperate better with other tools.

`git-dpm` creates a ''patch branch'' when you want to directly edit upstream source files for export to quilt patches, however `git-dpm` deletes this patch branch when you export back to the master branch.

== Q/A ==

'''Q: What if I don't want to use git-dpm?''' '''A: Please be nice to your fellow teammates, and reconsider!''' We want team members to easily know what workflow to use, by common convention. In rare cases, if some deviation from the team default is required, you MUST have a good rationale, and both the reason and workflow MUST be documented in the `README.Debian` file.

'''Q: Source-full or source-less branches?''' '''A: Source-full branches please!''' There are lots of good reasons for this, including the ability to easily diff between upstream versions, say to double check that nothing untoward or undocumented has been added in a new upstream version. E.g. `git checkout upstream` and `git diff upstream/0.11.3` would for example give you an overview of what's new in upstream's 0.11.4.

We require team maintained packages to contain the full upstream source, and we don't believe it will be an overwhelming performance or disk-space burden to do so. It is much easier to hack on packages, including generating the quilt patches, when a checkout of the repo gives you the full source. Source-less branches defeat much of the benefit of managing packages in git.

== Where do the team's git branches live? ==

[[http://anonscm.debian.org/cgit/|Alioth]], under the [[http://anonscm.debian.org/cgit/python-modules|python-modules]] directory.

They can be checked out individually, or managed as a group, with `mr`. It is configured to not checkout all the team packages by default, but easily manage the ones that are checkout out, collectively.

{{{
$ sudo apt install mr
$ git clone git+ssh://git.debian.org/git/python-modules/tools/python-modules.git
$ cd python-modules
$ echo $PWD/.mrconfig >> ~/.mrtrust
$ ./checkout PACKAGENAME # For each package you care about
}}}

To update the `.mr-config` file, run

{{{
$ git pull
$ ./make-mrconfig
}}}

And if there are any, commit and push changes.

To update all your checked out packages, run

{{{
$ ./quick-update
}}}		  * {{{upstream}}} - The un-Debianized upstream source. This is what you get when you unpack the upstream tarball. This could also be {{{upstream/<version>}}} if required.

If you use other branch names, please have a good reason (not just personal preference, remember - take one for the team!), and you '''MUST''' document the differences in {{{debian/README.source}}}. The default branch names will help the repo interoperate better with other tools.

`gbp pq` creates a ''patch branch'' when you want to directly edit upstream source files for export to quilt patches, however this is for local use only and should not be pushed to remote servers.

=== Git Tag Names ===

There seems to be several tag styles in common use. However, we recommend using '/' as a separator, yielding tags like `debian/0.2.0-1`.

=== Recommended/required configuration settings ===

Both git itself and git-bp provide ways to configure defaults. Settings that are personal to you should live only in {{{$HOME/.gitconfig}}} or {{{$HOME/.gbp.conf}}}. Settings which affect how the team interoperates should live in {{{<repo>/debian/gbp.conf}}} so that anyone who checks out the branch will see the same settings.

== Procedures ==

Here are some common scenarios and use cases. There may be other ways to accomplish these tasks so if you find a more efficient way of doing something, please update these pages and discuss with the team via the debian-python mailing list. Feel free to add more use cases here too! Your experience is valuable.
Line 69: Line 47:
To set up a new repository here, you should follow these instructions. They require ssh access to Alioth, but if you have push permissions, you should have ssh permissions. Ask a team member for help if you are unable to do this. These instructions set up the hooks for commit notifications and such:
Line 72: Line 49:
{{{
$ ssh git.debian.org
$ cd /git/python-modules
$ ./setup-repository <srcpkgname> "<srcpkgname> packaging"
$ exit
}}}

Of course, use your own source package name for ''<srcpkgname>''.

'''Q: should we honor pull requests and allow for mirrors on Github or Gitlab?'''

'''A:'''We will still require all team packages to be available on git.debian.org. Please do not host your team maintained packaging branches anywhere else; e.g. GitHub is '''not''' appropriate for the packaging branches because it is not free software. Gitlab does have a free software Community Edition and someday it might be interesting to run one of those for DPMT or Debian-at-large, but for now please do not submit merge requests on gitlab.com.		 To set up a new repository visit https://salsa.debian.org/python-team/packages/ and follow the 'new project' link (the link will not be accessible unless you have joined the [[https://salsa.debian.org/python-team/tools/python-modules/blob/master/policy.rst#joining-the-team|team]]). Enter the source package name as the project name and leave the visibility set to public.
Line 88: Line 54:
Vcs-Git: git://anonscm.debian.org/python-modules/packages/<srcpkgname>.git
Vcs-Browser: http://anonscm.debian.org/cgit/python-modules/packages/<srcpkgname>.git
}}}

== Tag style ==

There seems to be several tag styles in common use, and `git-dpm` by default uses a '-' as separator between the branch and version number. However, we recommend using '/' as a separator, yielding tags like `debian/0.2.0-1`. To set the `git-dpm` tag style, edit the `debian/.git-dpm` file and put this at the '''end''' of the file:

{{{
debianTag="debian/%e%v"
patchedTag="patched/%e%v"
upstreamTag="upstream/%e%u"
}}}

Any migration scripts we implement will select this tag style automatically, but please be sure to set it this way for any repository that existed before the migration.


== Use cases ==

Here are some common scenarios and use cases. There may be other ways to accomplish these tasks so if you find a more efficient way of doing something, please update these pages and discuss with the team via the debian-python mailing list. Feel free to add more use cases here too! Your experience is valuable.		 Vcs-Git: https://salsa.debian.org/python-team/packages/<srcpkgname>.git
Vcs-Browser: https://salsa.debian.org/python-team/packages/<srcpkgname>
}}}

Replace <srcpkgname> with your source package name.

You can also create the repository using the salsa tool:

{{{
salsa --group python-team/packages create_repo <srcpkgname>
}}}
Line 111: Line 68:
First, [[#initialize|Initialize the package directory on git.debian.org]] to set up the bare repo (with hooks installed) that you will push your branch.

Next, you can either [[PackagingWithGit/GitDpm/ImportDsc|import an existing .dsc file]] or [[PackagingWithGit/GitDpm/Initialize|start from scratch]]. Here's are the steps from the manpage that seem to work well:		 First, [[#initialize|initialize the project on salsa.debian.org]] to set up the bare repo that you will push your branch.

Next, you can either [[PackagingWithGit/GitDpm/ImportDsc|import an existing .dsc file]] or [[PackagingWithGit/GitDpm/Initialize|start from scratch(But be careful ，debian-python team does not support git-gdm workflow)]]. Here's are the steps from the manpage that seem to work well:
Line 120: Line 77:
$ git checkout -b upstream
Line 122: Line 80:
$ git checkout -b upstream $ git tag -s upstream/1.0
Line 124: Line 82:
$ git-dpm init ../srcpkgname_1.0.orig.tar.gz
}}}

You'll now be left on the {{{master}}} branch, i.e. the packaging branch. You'll also have {{{upstream}}} (raw tarball contents) and {{{pristine-tar}}} branch.		 $ git checkout -b debian/master
}}}

You'll now be left on the {{{debian/master}}} branch, i.e. the packaging branch. You'll also have {{{upstream}}} (raw tarball contents) and {{{pristine-tar}}} branch.
Line 134: Line 92:
$ git-dpm prepare
$ git-dpm status
}}}

You can now build your package using whatever tools you like, including {{{debuild}}}, {{{dpkg-buildpackage}}}, and {{{git-buildpackage}}}.

Push your repo to git.debian.org:

{{{
$ git push --set-upstream ssh://git.debian.org/git/python-modules/packages/<srcpkgname>.git --all
}}}

''At this point, I like to cd to a different directory and do a {{{git clone ssh://git.debian.org/git/python-modules/packages/<srcpkgname>.git}}} and then continue working from the <srcpkgname> directory. To prove that all the branches got pushed correctly, in this fresh clone, checkout the {{{master}}}, {{{pristine-tar}}}, and {{{upstream}}} branches.''

=== Building and tagging ===		 }}}

=== Checking out an existing package ===

Packages can be checked out individually, or managed as a group, with `mr`. It is configured to not checkout all the team packages by default, but easily manage the ones that are checkout out, collectively.

{{{
$ sudo apt install mr
$ git clone git@salsa.debian.org:python-team/tools/python-modules
$ cd python-modules
$ echo $PWD/.mrconfig >> ~/.mrtrust
$ ./checkout PACKAGENAME # For each package you care about
}}}

if your username differs between your local machine and Salsa, then you can add this snippet to ~/.ssh/config:

{{{
Host salsa.debian.org
User SALSA_USERNAME
}}}

To update the `.mr-config` file, run

{{{
$ git pull
$ ./make-mrconfig
}}}

And if there are any, commit and push changes.

To update all your checked out packages, run

{{{
$ ./quick-update
}}}

To check out packages individually, you might be able to just

{{{
$ debcheckout -a <srcpkgname>
}}}

but while the `Vcs-*` fields have been updated in the repository, new uploads with those fields still need to be done. In the meantime, you can:

{{{
$ gbp clone git@salsa.debian.org:python-team/packages/<srcpkgname>.git
}}}

`gbp clone` is preferable to `git clone` because `gbp clone` ensures that you'll have all the remote tracking branches set up properly. You can still use `git clone` but you'll have to check out the {{{pristine-tar}}} and {{{upstream}}} branches yourself.

You might also want to add this handy little convenience to your {{{~/.gitconfig}}} file:

{{{
[url "git@salsa.debian.org:python-team/packages/"]
    insteadof = dpt:
}}}

which allows you to do this:

{{{
$ gbp clone dpt:srcpkgname.git
}}}

make sure you cd into the source directory.

=== Ensure up-to-date ===

Before doing anything on an existing repository, always always always ensure you have pulled in any changes:

{{{
$ gbp pull --redo-pq
}}}

If you always remember to do this, you reduce the risk of causing accidental git conflicts.

Note that the {{{--redo-pq}}} option will discard any local changes to your local patch queue, and replace with the patches from {{{debian/patches}}}.

=== Building ===

Now you can build your package using whatever tools you like, including {{{debuild}}}, {{{dpkg-buildpackage}}}, and {{{git-buildpackage}}}.

=== Uploading ===
Push your repo to salsa.debian.org:

{{{
$ git push --set-upstream git@salsa.debian.org:python-team/packages/<srcpkgname>.git : --tags
}}}

''At this point, I like to cd to a different directory and do a {{{gbp clone git@salsa.debian.org:python-team/packages/<srcpkgname>.git}}} and then continue working from the <srcpkgname> directory. To prove that all the branches got pushed correctly, in this fresh clone, checkout the {{{debian/master}}}, {{{pristine-tar}}}, and {{{upstream}}} branches.''

=== Tagging ===
Line 153: Line 187:
$ git-dpm tag $ gbp buildpackage --git-tag-only
Line 157: Line 191:
'''Q: should we mandate git-dpm style tag names or git-bp style tag names?''' Let's work with those projects to make them more consistent, or at least provide options. Alternately, if you want to sign your tags:

{{{
$ gbp buildpackage --git-tag-only --git-sign-tags
$ git push --tags
}}}
Line 161: Line 200:
{{{
$ git-dpm import-new-upstream --ptc --rebase-patched path-to-orig-tar-gz
}}}		 You should import the patch queue first. For this to work correctly, `debian/gbp.conf` should already have the correct value debian-branch, otherwise `gbp import-orig` will merge to the wrong branch.

Using a debian/watch file (recommended):

{{{
$ gbp pq import
$ git checkout debian/master
$ gbp import-orig --pristine-tar --uscan
}}}

Note "gbp pq import" will switch to the wrong branch, and if not fixed then import-orig can do the wrong thing.

Using a tarball file:

{{{
$ gbp pq import
$ gbp import-orig --pristine-tar /path/to/new-upstream.tar.gz -u 0.2
}}}

where 0.2 is the new upstream version number.

{i} If the upstream tarball is already in the form ''packagename''_''version''.orig.tar.gz (E.g. {{{package_0.2.orig.tar.gz}}}), then the {{{-u}}} option is not required.
Line 167: Line 225:
 * Don't forget to update `debian/changelog`! `git-dpm` will not do it for you.
 * If you have merge conflicts in your patched branch, you will have to resolve them manually, then continue the rebasing until everything applies cleanly. ''However'' be aware that if this happens, your `pristine-tar` branch will [[DebianBug:787558|not have been updated]], despite the `--ptc` flag. Before you try to build your package, you'll need to do `pristine-tar commit <path-to-orig.tar.gz>` manually.

What if the new upstream introduces quilt patch conflicts? In that case, git-dpm will leave you in the `patched` branch, and you'll have to do the normal git conflict resolution dance. Once you've completed the `git rebase` operation, you will need to call `git-dpm update-patches`.		  * Don't forget to update `debian/changelog`!

Rebase the patches:

{{{
$ gbp pq rebase
$ gbp pq export
}}}

You will need to push the `debian/master` branch and also the `upstream` and `pristine-tar` branches.
{{{
$ git push origin : --tags
}}}
Line 174: Line 241:
Patching (i.e. adding quilt patches) is easy. You start by entering git-dpm's patch branch, make your changes to upstream code, commit, and then switch back to the master branch. git-dpm automatically converts every commit to a separate quilt patch file. If you want to squash several commits into single quilt patch file, see below for how to rebase. If you want to control the quilt patch file names instead of letting git-dpm automatically name your patch files, be sure to add `Patch-Name: <whatever>` to the quilt patch files (while in the master branch), and then do a `git-dpm checkout-patched` followed by a `git-dpm update-patches`. Patching (i.e. adding quilt patches) is easy. You import the patches to a git patch queue, edit the patch queue, then export back again.
Line 179: Line 246:
# git-dpm c-p is an alias
$ git-dpm checkout-patched
<edit & commit>
# git-dpm u-p is an alias
$ git-dpm update-patches
}}}

=== Updating your quilt patch stack ===

Sometimes you want to re-order patches, squash patches, remove them, etc. You do this by entering git-dpm's patch branch and doing a rebase against upstream.

{{{
$ git-dpm checkout-patched
$ git rebase -i upstream
<edit>
$ git-dpm update-patches		 $ gbp pq import
Edit patches
$ gbp pq export
$ vim debian/changelog
$ git add debian/changelog
$ git add debian/patches/*
$ git commit
}}}

=== Cherry picking upstream commits ===

Sometimes you need to cherry pick upstream commits, e.g. to fix a bug in the Debian version when upstream hasn't yet released a new version. A convenient way to do that, if upstream is developed in git, is to add a remote and cherry pick from that. For example:

{{{
$ git remote add upstream_repo git://example.com/foo.git
$ git fetch upstream_repo
$ gbp pq import
$ git cherry-pick any_upstream_commit
$ gbp pq export
$ vim debian/changelog
$ git add debian/changelog
$ git add debian/patches/*
$ git commit
Line 205: Line 279:
=== Converting git-dpm to gbp pq ===

Previously DPT used git-dpm at the workflow. This section documents how to convert from git-dpm to gbp pq.

 0. Ensure all branches up to date.
 1. Unapply all patches
 2. Delete {{{debian/.git-dpm}}} config file.
 3. Set {{{debian-branch}}} value to {{{debian/master}}} in {{{debian/gbp.conf}}}
 4. Commit to {{{debian/master}}} branch (use {{{git branch -m master debian/master}}} to rename the branch).
 5. Update .git/config branch debian/master to refer to ''merge = refs/heads/debian/master''.
 6. Refresh the patches.
 7. On Salsa, set the default branch to the new {{{debian/master}}} branch.

Sample {{{debian/gbp.conf}}} file:

{{{
[DEFAULT]
debian-branch=debian/master
}}}

Sample steps to do the above (upstream branch will not be automatically checkout out locally):

{{{
gbp pull
git read-tree --reset -u upstream
git reset -- debian
git checkout debian
git rm debian/.git-dpm
cat <<EOF > debian/gbp.conf
[DEFAULT]
debian-branch=debian/master
EOF
git add debian/gbp.conf
git commit -m "Convert from git-dpm to patches unapplied format"
git branch -m master debian/master
git push origin -u debian/master
}}}

To refresh the patches:

{{{
gbp pq import
gbp pq export
dch -m "Refresh patches after git-dpm to gbp pq conversion"
git add debian/patches/
git add debian/changelog
debcommit
}}}

To make the new branch the default and delete the old branch:

 1. go to https://salsa.debian.org/python-team/packages/$1/edit
 1. expand "General settings"
 1. select the new default branch
 1. go to https://salsa.debian.org/python-team/packages/$1/settings/repository
 1. expand "Protected branches"
 1. fix those settings
 1. go to https://salsa.debian.org/python-team/packages/$1/branches
 1. delete the old master branch

=== Packages no longer maintained by DPT ===

Packages repositories can be archived on [[Salsa]] by owners of the Python team namespace. If you want an obsolete repositories to be archived you will need to [[https://salsa.debian.org/groups/python-team/packages/-/group_members?sort=access_level_desc|contact one of the owners]] and ask them to do it.

== FAQ ==

'''Q: What if I don't want to use gbp pq?'''<<BR>>
'''A: Please be nice to your fellow teammates, and reconsider!''' We want team members to easily know what workflow to use, by common convention. In rare cases, if some deviation from the team default is required, you MUST have a good rationale, and both the reason and workflow MUST be documented in the `README.source` file.

'''Q: Source-full or source-less branches?'''<<BR>>
'''A: Source-full branches please!''' There are lots of good reasons for this, including the ability to easily diff between upstream versions, say to double check that nothing untoward or undocumented has been added in a new upstream version. E.g. `git checkout upstream` and `git diff upstream/0.11.3` would for example give you an overview of what's new in upstream's 0.11.4.

We require team maintained packages to contain the full upstream source, and we don't believe it will be an overwhelming performance or disk-space burden to do so. It is much easier to hack on packages, including generating the quilt patches, when a checkout of the repo gives you the full source. Source-less branches defeat much of the benefit of managing packages in git.

'''Q: Should we honor pull requests and allow for mirrors on GitHub or GitLab?'''<<BR>>
'''A:'''We will still require all team packages to be available on salsa.debian.org. Please do not host your team maintained packaging branches anywhere else; e.g. GitHub is '''not''' appropriate for the packaging branches because it is not free software. GitLab does have a free software Community Edition and someday it might be interesting to run one of those for DPT or Debian-at-large, but for now please do not submit merge requests on gitlab.com.
Line 206: Line 357:

=== Recommended/required configuration settings ===

Both git itself and git-bp provide ways to configure defaults. Settings that are personal to you should live only in {{{$HOME/.gitconfig}}} or {{{$HOME/.gbp.conf}}}. Settings which affect how the team interoperates should live in {{{<repo>/debian/gbp.conf}}} so that anyone who checks out the branch will see the same settings.

=== Converting from svn to git ===

 * For [[https://wiki.debian.org/Alioth/Git#Convert_a_SVN_Alioth_repository_to_Git|git-buildpackage]]
 * For git-dpm: [[http://anonscm.debian.org/cgit/users/barry/import-dscs.git/tree/import-dscs.py|perhaps this script will help you]]

== Crazy Use Cases ==

=== python-pip ===

BarryWarsaw is working on an update to pip 7 and had some unrelease commits in the aloith svn. Because pip 7 is blocked on NEW dependencies, he wanted to fix some bugs in 1.5.6 so decided to create a 1.5.6 branch. However, he also wanted to preserve the unreleased svn commits. Here's how he did it, using Stefano's converted git repository.

{{{
$ gbp clone ssh://git.debian.org/git/python-modules/svn-migration/python-pip.git gitsvn
$ cd gitsvn
$ git co debian/1.5.6-5
$ git checkout -b release-1.5.6
$ git-dpm init ../src/python-pip_1.5.6.orig.tar.gz
$ git-dpm prepare
$ git co master
$ git merge origin/svn
$ git checkout release-1.5.6
}}}

Now he can work on python-pip 1.5.6-6. Then later, when he wants to continue working on pip 7, he'll need to switch to the `master` branch and do another `git-dpm init` with the pip 7 orig.tar.gz. Since this will result in conflicts (due to changes in upstream pip), the conflicts need to be resolved and then master should be properly prepared to use git-dpm for a new upstream release.

Contents

  1. Using git for team packages
    1. Policies
      1. Git Repository
      2. Git Workflows
      3. Git Branch Names
      4. Git Tag Names
      5. Recommended/required configuration settings
    2. Procedures
      1. Creating new repositories
      2. Creating a new package
      3. Checking out an existing package
      4. Ensure up-to-date
      5. Building
      6. Uploading
      7. Tagging
      8. New upstream release
      9. Patching
      10. Cherry picking upstream commits
      11. Sponsoring, mentoring, reviewing
      12. Pull requests
      13. Converting git-dpm to gbp pq
      14. Packages no longer maintained by DPT
    3. FAQ
    4. More information and resources

Using git for team packages

There are many advantages to team-based package development. The DPT has very excellent, knowledgeable, and experienced Python and Debian developers and our goal is to bring the very best Python experience to Debian users and developers. We're here to help each other and provide consensus (if not 100% agreement) on best practices. Collaboration has many benefits, and one of the things that aids effective collaboration is common tools and workflows surrounding packaging, while still providing individual maintainers the freedom to adapt those practices to their preferences, and the peculiarities of the packages.

One of the perks of being in a team is that all of our packages are managed in a similar way, using similar tools and workflows. This is great because any team member can immediately get the source of a package to do an update or a bug fix, and they don't have to first try to figure out how the package is version controlled. Plus, we can have nice wiki documentation which help new team members how to do things, and almost anyone on IRC can immediately give you advice. We use a common set of vcs procedures to ensure consistency among all of our packages.

It is a team requirement that all packages be managed using the same version control system.

Policies

Git Repository

https://salsa.debian.org, under the python-team

Git Workflows

Most of our git-based workflow is provided by gbp (formerly git-buildpackage). On top of gbp there's the patch management regime. Most Debian developers are familiar with quilt for including Debian-specific patches to upstream pristine tarballs. While it's still possible to use quilt directly of course, there is a better way to integrate the best of the git world with quilt. Of the two predominant git-based patch regimes, the Debian Python team has chosen gbp pq.

Git Branch Names

As a team we are strongly recommending you use DEP-14 style branch names, specifically:

  • debian/master - The Debianized upstream source directory. In other words, this contains upstream source and a debian/ packaging directory. It is a source-full, patches-unapplied checkout.

  • pristine-tar - Contains the standard pristine-tar deltas.

  • upstream - The un-Debianized upstream source. This is what you get when you unpack the upstream tarball. This could also be upstream/<version> if required.

If you use other branch names, please have a good reason (not just personal preference, remember - take one for the team!), and you MUST document the differences in debian/README.source. The default branch names will help the repo interoperate better with other tools.

gbp pq creates a patch branch when you want to directly edit upstream source files for export to quilt patches, however this is for local use only and should not be pushed to remote servers.

Git Tag Names

There seems to be several tag styles in common use. However, we recommend using '/' as a separator, yielding tags like debian/0.2.0-1.

Recommended/required configuration settings

Both git itself and git-bp provide ways to configure defaults. Settings that are personal to you should live only in $HOME/.gitconfig or $HOME/.gbp.conf. Settings which affect how the team interoperates should live in <repo>/debian/gbp.conf so that anyone who checks out the branch will see the same settings.

Procedures

Here are some common scenarios and use cases. There may be other ways to accomplish these tasks so if you find a more efficient way of doing something, please update these pages and discuss with the team via the debian-python mailing list. Feel free to add more use cases here too! Your experience is valuable.

Creating new repositories

To set up a new repository visit https://salsa.debian.org/python-team/packages/ and follow the 'new project' link (the link will not be accessible unless you have joined the team). Enter the source package name as the project name and leave the visibility set to public.

Use these Vcs-* fields in the header of your debian/control file: 

Vcs-Git: https://salsa.debian.org/python-team/packages/<srcpkgname>.git
Vcs-Browser: https://salsa.debian.org/python-team/packages/<srcpkgname>

Replace <srcpkgname> with your source package name.

You can also create the repository using the salsa tool: 

salsa --group python-team/packages create_repo <srcpkgname>

Creating a new package

First, initialize the project on salsa.debian.org to set up the bare repo that you will push your branch.

Next, you can either import an existing .dsc file or start from scratch(But be careful ，debian-python team does not support git-gdm workflow). Here's are the steps from the manpage that seem to work well: 

$ uscan   # Download your package's upstream original tarball
$ tar -xvf srcpkgname_1.0.orig.tar.gz
$ cd srcpkgname_1.0
$ git init
$ git checkout -b upstream
$ git add .
$ git commit -m "import srcpkgname_1.0.orig.tar.gz"
$ git tag -s upstream/1.0
$ pristine-tar commit ../srcpkgname_1.0.orig.tar.gz upstream
$ git checkout -b debian/master

You'll now be left on the debian/master branch, i.e. the packaging branch. You'll also have upstream (raw tarball contents) and pristine-tar branch.

Fill in the rest of your debian/ directory, then do this: 

$ git add debian/*
$ debcommit  # assuming you have a d/changelog entry, otherwise git commit

Checking out an existing package

Packages can be checked out individually, or managed as a group, with mr. It is configured to not checkout all the team packages by default, but easily manage the ones that are checkout out, collectively. 

$ sudo apt install mr
$ git clone git@salsa.debian.org:python-team/tools/python-modules
$ cd python-modules
$ echo $PWD/.mrconfig >> ~/.mrtrust
$ ./checkout PACKAGENAME  # For each package you care about

if your username differs between your local machine and Salsa, then you can add this snippet to ~/.ssh/config: 

Host salsa.debian.org
User SALSA_USERNAME

To update the .mr-config file, run 

$ git pull
$ ./make-mrconfig

And if there are any, commit and push changes.

To update all your checked out packages, run 

$ ./quick-update

To check out packages individually, you might be able to just 

$ debcheckout -a <srcpkgname>

but while the Vcs-* fields have been updated in the repository, new uploads with those fields still need to be done. In the meantime, you can: 

$ gbp clone git@salsa.debian.org:python-team/packages/<srcpkgname>.git

gbp clone is preferable to git clone because gbp clone ensures that you'll have all the remote tracking branches set up properly. You can still use git clone but you'll have to check out the pristine-tar and upstream branches yourself.

You might also want to add this handy little convenience to your ~/.gitconfig file: 

[url "git@salsa.debian.org:python-team/packages/"]
    insteadof = dpt:

which allows you to do this: 

$ gbp clone dpt:srcpkgname.git

make sure you cd into the source directory.

Ensure up-to-date

Before doing anything on an existing repository, always always always ensure you have pulled in any changes: 

$ gbp pull --redo-pq

If you always remember to do this, you reduce the risk of causing accidental git conflicts.

Note that the --redo-pq option will discard any local changes to your local patch queue, and replace with the patches from debian/patches.

Building

Now you can build your package using whatever tools you like, including debuild, dpkg-buildpackage, and git-buildpackage.

Uploading

Push your repo to salsa.debian.org: 

$ git push --set-upstream git@salsa.debian.org:python-team/packages/<srcpkgname>.git : --tags

At this point, I like to cd to a different directory and do a gbp clone git@salsa.debian.org:python-team/packages/<srcpkgname>.git and then continue working from the <srcpkgname> directory. To prove that all the branches got pushed correctly, in this fresh clone, checkout the debian/master, pristine-tar, and upstream branches.

Tagging

Once you've built and uploaded your package, you should tag the release. 

$ gbp buildpackage --git-tag-only
$ git push --tags

Alternately, if you want to sign your tags: 

$ gbp buildpackage --git-tag-only --git-sign-tags
$ git push --tags

New upstream release

You should import the patch queue first. For this to work correctly, debian/gbp.conf should already have the correct value debian-branch, otherwise gbp import-orig will merge to the wrong branch.

Using a debian/watch file (recommended): 

$ gbp pq import
$ git checkout debian/master
$ gbp import-orig --pristine-tar --uscan

Note "gbp pq import" will switch to the wrong branch, and if not fixed then import-orig can do the wrong thing.

Using a tarball file: 

$ gbp pq import
$ gbp import-orig --pristine-tar /path/to/new-upstream.tar.gz -u 0.2

where 0.2 is the new upstream version number.

{i} If the upstream tarball is already in the form packagename_version.orig.tar.gz (E.g. package_0.2.orig.tar.gz), then the -u option is not required.

Important notes:

  • Don't forget to update debian/changelog!

Rebase the patches: 

$ gbp pq rebase
$ gbp pq export

You will need to push the debian/master branch and also the upstream and pristine-tar branches. 

$ git push origin : --tags

Patching

Patching (i.e. adding quilt patches) is easy. You import the patches to a git patch queue, edit the patch queue, then export back again.

In general, to patch your package do this: 

$ gbp pq import
Edit patches
$ gbp pq export
$ vim debian/changelog
$ git add debian/changelog
$ git add debian/patches/*
$ git commit

Cherry picking upstream commits

Sometimes you need to cherry pick upstream commits, e.g. to fix a bug in the Debian version when upstream hasn't yet released a new version. A convenient way to do that, if upstream is developed in git, is to add a remote and cherry pick from that. For example: 

$ git remote add upstream_repo git://example.com/foo.git
$ git fetch upstream_repo
$ gbp pq import
$ git cherry-pick any_upstream_commit
$ gbp pq export
$ vim debian/changelog
$ git add debian/changelog
$ git add debian/patches/*
$ git commit

Sponsoring, mentoring, reviewing

TBD

Pull requests

TBD

Converting git-dpm to gbp pq

Previously DPT used git-dpm at the workflow. This section documents how to convert from git-dpm to gbp pq.

  1. Ensure all branches up to date.
  2. Unapply all patches

  3. Delete debian/.git-dpm config file.

  4. Set debian-branch value to debian/master in debian/gbp.conf

  5. Commit to debian/master branch (use git branch -m master debian/master to rename the branch).

  6. Update .git/config branch debian/master to refer to merge = refs/heads/debian/master.

  7. Refresh the patches.

  8. On Salsa, set the default branch to the new debian/master branch.

Sample debian/gbp.conf file: 

[DEFAULT]
debian-branch=debian/master

Sample steps to do the above (upstream branch will not be automatically checkout out locally): 

gbp pull
git read-tree --reset -u upstream
git reset -- debian
git checkout debian
git rm debian/.git-dpm
cat <<EOF > debian/gbp.conf
[DEFAULT]
debian-branch=debian/master
EOF
git add debian/gbp.conf
git commit -m "Convert from git-dpm to patches unapplied format"
git branch -m master debian/master
git push origin -u debian/master

To refresh the patches: 

gbp pq import
gbp pq export
dch -m "Refresh patches after git-dpm to gbp pq conversion"
git add debian/patches/
git add debian/changelog
debcommit

To make the new branch the default and delete the old branch:

  1. go to https://salsa.debian.org/python-team/packages/$1/edit

  2. expand "General settings"
  3. select the new default branch

  4. go to https://salsa.debian.org/python-team/packages/$1/settings/repository

  5. expand "Protected branches"
  6. fix those settings

  7. go to https://salsa.debian.org/python-team/packages/$1/branches

  8. delete the old master branch

Packages no longer maintained by DPT

Packages repositories can be archived on Salsa by owners of the Python team namespace. If you want an obsolete repositories to be archived you will need to contact one of the owners and ask them to do it.

FAQ

Q: What if I don't want to use gbp pq?
A: Please be nice to your fellow teammates, and reconsider! We want team members to easily know what workflow to use, by common convention. In rare cases, if some deviation from the team default is required, you MUST have a good rationale, and both the reason and workflow MUST be documented in the README.source file.

Q: Source-full or source-less branches?
A: Source-full branches please! There are lots of good reasons for this, including the ability to easily diff between upstream versions, say to double check that nothing untoward or undocumented has been added in a new upstream version. E.g. git checkout upstream and git diff upstream/0.11.3 would for example give you an overview of what's new in upstream's 0.11.4.

We require team maintained packages to contain the full upstream source, and we don't believe it will be an overwhelming performance or disk-space burden to do so. It is much easier to hack on packages, including generating the quilt patches, when a checkout of the repo gives you the full source. Source-less branches defeat much of the benefit of managing packages in git.

Q: Should we honor pull requests and allow for mirrors on ?GitHub or GitLab?
A:We will still require all team packages to be available on salsa.debian.org. Please do not host your team maintained packaging branches anywhere else; e.g. ?GitHub is not appropriate for the packaging branches because it is not free software. GitLab does have a free software Community Edition and someday it might be interesting to run one of those for DPT or Debian-at-large, but for now please do not submit merge requests on gitlab.com.

More information and resources

CategoryGit