Translation(s): none


Packaging BOINC with git

Whenever something on this page does not work as expected, or you had difficulties to understand the one or other bit, tell us. Such external stimuli are so very helpful, they should not be missed.

1. Overview

The pkg-boinc community has several repositories. The examples below are for the the one name 'boinc', which amongst others especially provide the boinc-client and the boinc-manager packages. An overview of all packages on alioth can be seen on http://anonscm.debian.org/gitweb/ (long), among which the most active are

Repository

Description

Status

pkg-boinc/boinc.git

BOINC packaging

fine

pkg-boinc/boinc-app-seti.git

SETI@home packaging

fine

pkg-boinc/scripts.git

pkg-boinc scripts

fine

pkg-boinc/boinc-server-autodock.git

BOINC Server for AutoDock

early stage, pending completion of boinc-server-maker package

The general workflow for using any particular repository is to first have all upstream code in the branch 'upstream'. That 'upstream' branch and the 'master' branch, which is upstream plus the debian folder, are the two main branches. They are kept on a particular server, anonscm.debian.org, and everyone contributing to the pkg-boinc project can update them. The text below presents an overview on branches and how to invoke the tool 'git-buildpackage' to build the Debian packages from those branches alone.

2. Branches

When packaging with git, both the Debian packaging sources and the upstream sources are stored in the archive. And when, like we are, one is using git-buildpackage, then some default names for the different components should be respected to keep the argument list short.

There are many more branches, which are somewhat historic, representing various attempts to be very current or very feature-rich. The most imminent urge to work on are the branches for past releases of Debian, i.e. the 'stable' release and also 'old-stable'. As a Debian developer one commonly works with a newer release of Debian, like 'testing' or 'unstable', over which one forgets to feed the current official and older distributions with the latest end user software.

Formerly, we used this special branch here

which helps with provisioning the source tarballs for the building of Debian packages. These are not really required since one gets those either directly from the Debian distribution or, if not yet uploaded, can also create them from the upstream branch.

After a first checkout of the repository (cloning in git terminology, described below) a list of branches available on the server is made visible with

git branch -r

and they will then appear prefixed with the term 'origin'

$ git branch -r
  origin/HEAD -> origin/master
  origin/debian/backports/etch
  origin/debian/backports/hardy
  origin/debian/backports/jaunty
  origin/debian/backports/sarge
  origin/master
  origin/pristine-tar
  origin/upstream

3. Preparation

The programs that are referred to in the following paragraphs are contained in the Debian packages: git git-buildpackage pristine-tar. Please run

apt-get install git git-buildpackage pristine-tar

to perform the installation.

And to build the package, also install the following build-dependencies

apt-get install dh-buildinfo docbook2x automake1.9 autoconf libssl-dev libcurl4-openssl-dev freeglut3-dev libxmu-dev libxi-dev libwxgtk2.8-dev libsqlite3-dev libtool libnotify-dev libmysqlclient-dev libfcgi-dev libjpeg-dev libxss-dev
libxml2-utils libgtk2.0-dev 

This list is not unlikely to be outdated, since this page is not expected to be kept in complete sync. But don't worry, you will receive respective instructions when running git-buildpackage should anything be missing.

4. Accessing repositories

4.0.1. The traditional (still working) way to do it

Check out the source of package <package> with:

git clone ssh://<alioth-login>@git.debian.org/git/pkg-boinc/<package>.git
cd <package>

where package is typically one of

The repositories of pkg-boinc are inspectable via http://git.debian.org (long).

If you do not have an Alioth account, you can check out the repositories read-only:

git clone git://git.debian.org/git/pkg-boinc/<package>.git
cd <package>

But there is hope that we can get you actively involved.

That directory includes all the source code already, and it even shows a debian directory, all information seems available:

$ ls                                 
api                     checkin_notes_samples  db  
apps                    client                 debian
_autosetup              clientctrl             depcomp
bolt_checkin_notes.txt  clientgui              doc
[...]

For the build process we need the branches upstream and master. The master branch is checked out with the initial cloning of the repository. The upstream branch we need to add to it:

git clone git://git.debian.org/git/pkg-boinc/boinc.git
cd boinc
git checkout -b upstream origin/upstream

4.0.2. git-buildpackage comes with helping macros

We will (optionally) be using git-buildpackage to build packages. That tool also helps with the initial setup of it all:

gbp-clone ssh://<alioth-login>@git.debian.org/git/pkg-boinc/<package>.git

is a one line version of the prior manual creation of the Debian-associated packages. (Many thanks go to Guido for pointing this out.) Just, we have not been using the pristine-tar branch anymore for a couple of releases. Hence, feel free to 'git branch -d pristine-tar' if that pops in the 'git branch' list.

5. Typical workflow in git development

The branch is the central theme in git development. They are even created for some very quick local idea, just in case that that development may be in parallel to another idea. In short, a branch is some piece of development that one may accept or reject as a whole. It should be treated a bit like as if those changes done to a branch would be meant to get together as a single patch file.

There are local branches (your ideas) and remote branches (someone else's ideas or collaborative branches like the remote upstream or the remote master branch.

5.1. Working with local branches

Every new idea worked at under git is prepared as a separate branch. This may for instance be a bug that is fixed. When implementing something locally you want to

  1. create a new branch

    git branch new_unique_branchname
  2. check out that branch

    git checkout new_unique_branchname
  3. implement your change

    echo "new stuff" >> "some_file"
  4. commit change to git

    git commit "some_file"
  5. when completed the work or tired of it, check out master branch again

    git checkout master
  6. if you have a new idea, go to 1.
  7. if you want to continue with your development work, go to 2.
  8. combine your local development with your local mirror of a collaborative branch

    git merge new_unique_branchname
  9. (optional) delete that branch you have developed in

    git branch -d new_unique_branchname

The local master branch is now "ahead of 1 commit" of the origin/master branch, i.e. the branch on the remote server that the master branch was cloned from.

If the idea of yours that is represented by the branch would be generally applicable, then merge it from multiple branches, i.e. from debian/etch and debian/sarge, prior to deleting it.

5.2. Uploading local changes to the servers

If one is truly happy about the local master branch, then those local changes should become available to the others:

git push

If you do not have upload permissions then you can alternatively say

git format-patch

to have patch files created with your name as an author. For a more anonymous kind of contribution, just show how your debian folder differs from the version of that branch "master" from the same branch on the server referred to as origin (the alioth git repo), try

git diff origin debian

and send it to the list. The

git diff

command without the notion of the origin would ask for changes compared to the last committed version, i.e. the latest version locally known to git, not to the latest version on the server.

5.3. New remote branches

The branches currently existing on the servers seem sufficient. If you are a not a regular contributor, and even if you are, please consider to discuss the preparation of new branches with Rene and/or possibly Thomas. The idea previously prepared is then merged with the local representation of the server's branches. To have your branch on the remote site, just be a bit more explicit to push:

git push origin new_unique_branchname

To remove that remote one, use the following (not truly intuitive) syntax:

git push origin :new_unique_branchname
git branch -d new_unique_branchname

So, one needs to perform the removal of a remote branch both on the remote and the local site.

5.4. Committed everything changed but there are still changes

This is mightily annoying, but the build process at times changes files that are part of the upstream source tree but nonetheless changed by the build process. Those problems are to be helped by improving the source distribution (e.g. by removing the config.* files) but does this all work without version.h being at least existing?

Changes to those collaterally modified files should not be committed. When git does not allow you to do anything (like uploading a new upstream) then consider running "git reset --hard" or (more intuitively) changes those files back via

git diff | patch -p1 -R

The latter won't work for changed file permissions (hey, who opts at patching patch for that?) but it is not so bad and the rest can be helped by some "git diff | grep ^diff | cut -f3 -d" " | xargs -r chmod " magic.

6. Get updates in

For me to understand how git works I keep thinking about it as a collection of patches to some common seed version. That seed can be empty. Every commit of mine produces a new such patch. When another repository wants to profit from my work, then it needs to take my commits/patches and curate those into their latest version. When there is a change required to do so, then this creates another patch on top of the curated-in (aka merged) patches.

So, the inclusion of remotely created work takes two steps / commands:

  1. git fetch - get all that you can from some other repository (or the one that was cloned from by default). The fetched patches are then locally available in the branches like 'origin/master' or 'origin/upstream' but not yet seen when checking out the local 'master' or 'upstream' branch. And one cannot change into the 'origin/*' branches directly, or more correctly, you can, but this does not fit here.

  2. git merge origin - take all the patches that some branch has but the local branch has not and apply them. The default is to merge from the branch with the same name of that other repository.

The presumably typical learning curve goes like: "git fetch; git merge" is too tedious, I should be using "git pull" instead, which does the same thing but fetches and merges everything with the same name. And then one has somewhen merged bits that one did not intend to merge as of yet and then goes back to "git fetch; git merge" again.

7. Upstream and the Upstream branch

"Upstream" is what was available before Debian packaging started. This is typically a tarball from the software developers' web pages. With BOINC, it is slightly different.

We are interested in the differences between that "everything we need locally" master and the original upstream version. git is very good at determining such differences since differences between source trees is what source code management systems are all about. We just need another branch, upstream to keep the original. And that upstream branch remains untouched until there is a new upstream version.

What cannot be stressed enough: The only branch that is edited is the master branch and/or a branch that was derived from it. When there is a new upstream release, then we overwrite the upstream branch with that new version completely. The changes to upstream we then merge with the master branch. The only differences between the master branch and the upstream branch should be in the debian folder. If you want to edit what upstream has done, then use the quilt system to prepare a patch. That patch should be communicated to upstream.

For an initial run of the packaging, nothing needs to be done. If this is your first contact with this page and/or git, then just proceed directly with the section 'Build Package'.

7.1. Update the 'upstream' branch

When starting, just check out the version that is in the upstream branch already. The updating of that branch should be performed once you are familiar with the regular build process.

It is fairly likely that the latest branch in the upstream branch is older than the one you would like to work with. The upstream branch should then be updated. There are several ways to achieve that. Here is the traditional way, which is the most intuitive (for the old folks here).

  1. check out the source code with the right tag of the latest version. You can read about it in debian/README.source, which will point you to a script to perform the checkout and remove some non-DFSG compatible material by the side. But effectively, all you need to do is

    ./debian/rules get-orig-source
    which retrieves and executes that script.
  2. use the git-import-orig tool to perform the actual import

    git-import-orig ../boinc_7.0.23+dfsg.orig.tar.gz
    That tool already performs the merging of the the upstream branch with the master branch.
  3. $EDITOR debian/changelog and adjust it for the version now downloaded.
  4. commit your changes

    git commit -m "Bumped to new upstream version" debian/changelog
  5. bring the version to the main servers for everyone to use

    git push
  6. also upload the tags for that version (auto-provided by git-import-orig)

    git push --tags

To inspect the upstream branch, perform

git checkout upstream

To switch back to the master branch, perform analogously

git checkout master

The upstream branch does not have the debian folder but should otherwise be identical - in a newly checkout out repository. Once there are not files still looming around from past builds, there may be additional files in the upstream branch that do not belong there. The status of those files can be inspected by

git status

The general concept is that git shows only those files that are assigned to the current branch, i.e. that have been already present when the branch was created, or that are assigned to no branch at all. Thus, once you compile the code, all the object files belong to no branch and will stay even when you go to the 'upstream' branch again. This is somewhat annoying at times.

Files that should be ignored by git, e.g. because they are generated by the build process and changes to them are hence mostly neglectable are added to the file .gitignore.

8. Build Packages

Make sure you are in the master branch, if unsure, run 'git checkout master'. You will notice the directory named 'debian'. Building packages with git-buildpackage is now as easy as running:

git-buildpackage -rfakeroot

One could improve that just a bit by not requesting signatures

git-buildpackage -rfakeroot -uc -us

And if you are still evaluating bits here or there without having them checked in or the source tree was not properly cleaned from side-effects of the build, then run

git-buildpackage -rfakeroot -uc -us --git-ignore-new

git-buildpackage will learn about the version to pack from the debian/changelog file. If no .orig.tar.gz file is present, it will be recreated from the upstream branch. Once this was done, the remainder works as with dpkg-buildpackage, with no further involvement of the package maintainer.

The annoying thing with git- or dpkg-buildpackage is that those start the packaging from scratch since they call "fakeroot ./debian/rules clean" by default. If that is not desired then consider calling "fakeroot ./debian/rules binary", instead. It may also help to add the "-nc" (no clean) option to the *-buildpackage tools. Once the packages are brought to pristine shape, i.e. the packaging works reliably, run git-buildpackage again to be certain everything is prepared properly.

8.1. Variants for the invocation of git buildpackage

The following saves the generation of the orig.tar.gz from upstream. But this is not supported for the latest versions while this is written:

git-buildpackage --git-pristine-tar

These days, with the pristine-tar branch not in sync but available as a local branch irritating git-buildpackage, run as

git-buildpackage --git-upstream-branch=upstream --git-no-pristine-tar

which should do the job if (yes, 'if') the individual previously uploading the upstream branch has indeed uploaded the --tags. If not, find the uploader via 'git log' or use the gitk tool.

If the tarball is not in ".." but elsewhere, set it as in

git-buildpackage --git-tarball-dir=<path/to/tarballs/>

9. Don't do this but interesting to know

This tutorial is still evolving, and I am afraid it stops here for now. The following tidbits still need to find their place.

9.1. Tracking upstream development (out of date)

With git-svn upstream development can be directly tracked in your Git repository. For example if you are in the boinc repository, run the following command to clone BOINC's trunk from upstream's Subversion repository into your Git repository:

git svn clone --trunk=trunk/boinc --prefix=git-svn/upstream/ http://boinc.berkeley.edu/svn/ .

This creates a new remote branch git-svn/upstream/trunk which contains BOINC's complete trunk.

9.2. Tracking upstream development with git

The upstream just switched from svn to git, and it's a lot easier to track changes.

git clone git://boinc.berkeley.edu/boinc.git
# or
git clone http://boinc.berkeley.edu/git/boinc.git

Info from upstream can be found here.

10. Misc

It is at times confusing with all those many branches. To show that info directly on the shell, consider to add the following to the .profile or .bashrc (found at http://techblog.floorplanner.com/post/20528500331/working-with-git-branches):

parse_git_branch() {
  git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ \[\1\]/'
}
export PS1="\[\033[01;37m\]${debian_chroot:+($debian_chroot)}\u@\h:\w\[\033[00;35m\]$(parse_git_branch)\[\033[00m\] \$"

11. See Also

12. External References

Removed from main text

0.1. Regenerating upstream tarballs

If you are only interested in the latest version that is in Debian, then you can easily skip this section. Just use the tool git-buildpackage and the orig.tar.gz will be created from your upstream branch.

Upstream .orig.tar.gz tarballs can be regenerated with the pristine-tar tool directly from the repositories. To do so create a local pristine-tar branch from the remote origin/pristine-tar branch first:

git checkout -b pristine-tar origin/pristine-tar

As a clarification to those new to git, when cloning, the source of our clone is referred to as "origin". The "checkout" when given the "-b" option creates a new branch, which happens to have the same name as the branch on the remote git server. Normally, one would branch from the local master, i.e. one adds or changes a series of files relative to what is already available. The "origin/" prefix above informs git to branch from the remote side, i.e. a completely new environment is established that is independent from the previously checked out code:

$ ls
boinc_6.2.14.orig.tar.gz.delta  boinc_6.2.18.orig.tar.gz.id    boinc_6.6.10.orig.tar.gz.delta
boinc_6.2.14.orig.tar.gz.id     boinc_6.4.5.orig.tar.gz.delta  boinc_6.6.10.orig.tar.gz.id
boinc_6.2.18.orig.tar.gz.delta  boinc_6.4.5.orig.tar.gz.id

All other files disappear.

The new branch is now listed next to the previously cloned master branch:

git branch

Now to regenerate for example the boinc_6.2.18.orig.tar.gz tarball from the boinc repository run:

pristine-tar checkout boinc_6.2.18.orig.tar.gz

This pristine-tar is some magic on its own. The idea is to share upstream's resources and have them nicely integrated with the git and git-buildpackage functionalities. See this link http://www.eyrie.org/~eagle/notes/debian/git.html for a nice summary on what is going on.

To find out which tarballs can be regenerated with the tool pristine-tar, check out what files there are in the branch with the same name:

git ls-tree pristine-tar


CategoryGit