Differences between revisions 74 and 75
Revision 74 as of 2009-09-06 22:25:42
Size: 29920
Editor: jmtd
Comment: editor's note about a step
Revision 75 as of 2009-09-12 00:53:09
Size: 29923
Editor: ?Beuc
Comment: update mrconfig location reference
Deletions are marked like this. Additions are marked like this.
Line 133: Line 133:
echo) >> /path/to/the/SVN/trunk/checkout/.mrconfig echo) >> /path/to/the/SVN/mrconfig/checkout/.mrconfig

The Debian Games Team currently uses either git or svn to help in maintaining packages. This page will help you get started in using either git or svn repositories.

Scheme

Most of the packages are independent from each other. In svn, a [trunk,tags,branches}/<package> scheme is used with only the debian directory stored. The .orig.tar.gz files in such a case are stored in /var/lib/gforge/chroot/home/groups/pkg-games/htdocs/tarballs in the alioth servers. They can be accessed online at http://pkg-games.alioth.debian.org/tarballs. In git, each package has it's own repository. The contents of the orig tarball are stored directly within each package's repository, inside the 'upstream' branch. The pristine-tar program can be used to regenerate an orig tarball from the upstream branch.

Setting up

The first thing to do is get an account on alioth.debian.org. Visit the following link.

The next step is to join the team. Visit the Alioth page for the team and click on "request to join".

Next, you will need an ssh key for each computer you plan on working on packages with. If you don't have a key on a particular system, create one.

$ ssh-keygen

This should have created an RSA key for SSH protocol 2 by default. Upload the contents of your public keys (~/.ssh/id_rsa.pub) to your account in Alioth. To do this, copy the contents of each public key for your machines and paste them at the following link.

Next, create an SSH configuration file (~/.ssh/config) for your user profile to work with the alioth servers. Here's how to do the following in a terminal session.

$ cat >>~/.ssh/config <<EOF
Host svn.debian.org
        User <username>

Host git.debian.org
        User <username>

Host alioth.debian.org
        User <username>
EOF

Now you should be able to logon from each of your machines by doing the following.

$ ssh svn.debian.org
$ ssh git.debian.org
$ ssh alioth.debian.org

Note: There's usually a one hour delay when uploading your ssh public keys. During this delay, you can still login with your password.

Git

This section describes how to create a git repository for a package as well as convert from the svn layout.

Global git Configuration

First, you'll need to setup your global git configiration file (~/.gitconfig) with your name and email address. This is so commit messages show up with your name and email address automatically.

$ git config --global user.name "<your name>"
$ git config --global user.email "<your email address>"

Creating a git repository

These steps involve creating a brand new git repository for your package. If you are converting from svn, please see the "Converting from svn repository" section below.

Here's a simple way to create a git repository using the supplied setup-repository script in the /git/pkg-games directory on the alioth servers.

$ ssh <username>@git.debian.org
$ cd /git/pkg-games
$ ./setup-repository <package> '<package description>'

If you want to manually create the repository, the rest of this section describes how to do so.

$ ssh <username>@git.debian.org
$ cd /git/pkg-games
$ mkdir <package>.git
$ cd <package>.git
$ git --bare init --shared

Now edit the 'description' file in the directory you just created to one that is appropriate for the package.

$ echo "Packaging for <package>" >description

Let's also enable the use of 'hooks/post-update'.

$ chmod a+x hooks/post-update

The following will enable the games team to receive commit mails and irc notifications about commits. Please do the following.

$ git config --add hooks.mailinglist "pkg-games-commits@lists.alioth.debian.org"
$ git config --add hooks.cia-project debian-pkg-games
$ cat >hooks/post-receive <<END
#!/bin/sh
exec /usr/local/bin/git-commit-notice
END
$ chmod a+x hooks/post-receive

Please see the documentation about mr below, and add the new repository to list of games team repositories in the .mrconfig file. /

Uploading a package to the repository

Now let's import the package into a git repository. This is easy using 'git import-dsc' from the 'git-buildpackage' package. This is done from your local machine of course.

$ git import-dsc --pristine-tar <package>.dsc

(The --pristine-tar option is optional; if you have pristine-tar installed this will store enough information in the git repository to reproduce the pristine upstream tarball later.)

A directory with the name of the source package should have been made. Change into that directory.

$ cd <package>

Ensure the tags that are created reflect if a package has been released. git buildpackage creates tags for the debian branch and the upstream branch. If the tag created for the debian branch reflects an unreleased version, you'll have to delete that "debian" tag.

$ git tag -l
$ git tag -d 'debian/<unreleased version>'

Push the repository up to the repository on alioth:

$ git remote add alioth git+ssh://<username>@git.debian.org/git/pkg-games/<package>.git
$ git push alioth master
$ git push alioth upstream
$ git push alioth pristine-tar
$ git push alioth --tags

(Again the pristine-tar step is optional.)

And, finally, add a new section for the new repo in the .mrconfig file from from our SVN trunk and commit it:

$ (echo "[<package>]" ; \
echo "checkout = git clone git://git.debian.org/git/pkg-games/<package>.git"; \
echo) >> /path/to/the/SVN/mrconfig/checkout/.mrconfig
$ svn ci -m "Add <package>'s git repo to the .mrconfig file" /path/to/the/SVN/trunk/checkout/.mrconfig

Accessing a repository

To get a repository, do the following.

$ git clone git+ssh://<username>@git.debian.org/git/pkg-games/<package>.git

You can also enjoy read-only access by using the following.

$ git clone git://git.debian.org/git/pkg-games/<package>.git
$ git clone http://git.debian.org/git/pkg-games/<package>.git

It is generally a good idea to track certain branches that have been pushed (namely 'upstream' and 'pristine-tar'). To track them, do the following.

$ git checkout -b upstream origin/upstream
$ git checkout -b pristine-tar origin/pristine-tar #Do this if a pristine-tar branch was pushed

If you want to track more, see what branches are available.

$ git branch -r

You might see branches labelled 'tags/<some_version>'. These are really the "remotes" objects of the tags that were imported into the git repository when it was converted from svn. Although you can track these as well, it's better if you do a git checkout of the specific tags instead as follows.

$ git checkout remotes/tags/<some_tag>

See the section "Converting from svn repository" below for more details.

To view a summary of the repository on your favorite web browser, go to the following address.

http://git.debian.org/?p=pkg-games/<package>.git

Committing changes

To commit a change, first commit into your local repository.

$ git commit -m "<some comment>"

You could also do a commit of all files that were changed using the '-a' option.

$ git commit -a -m "<some comment>"

Push your changes.

$ git push #If you cloned the repository
$ git push alioth #If you created the repository using the above example

Updating

To keep updated simply do the following

$ git pull #If you cloned the repository
$ git pull alioth #If you created the repository using the above exmple

Tagging

To tag a release, you could build it using 'git-buildpackage' with the '--git-tag' option.

$ git buildpackage --git-tag --git-pristine-tar

You can also tag a release without rebuilding by supplying a different build command using the --git-builder option. In this case, we'll just perform a cleanup.

$ git buildpackage --git-tag --git-pristine-tar --git-builder='fakeroot debian/rules clean'

(As usual, the --git-pristine-tar bit is optional, but it will allow git-buildpackage to extract the pristine .orig.tar.gz.)

You could also manually tag the package. Ensure that you are in the correct branch (should be 'master').

$ git checkout master
$ git tag debian/<revision> -m "Debian release <revision>" #Please use this format

To tag an earlier revision, you'll need to find out it's commit object.

$ git rev-list --pretty --since="1 month" --all #'1 month' is just an example of course

Make note of the commit object and switch to that commit using git checkout. Then tag it and switch back to master.

$ git checkout <commit object>
$ git buildpackage --git-tag --git-pristine-tar --git-builder='fakeroot debian/rules clean' --git-ignore-new
$ git checkout master

Finally, push the tags

$ git push --tags

Using pristine-tar

You can manually use the pristine-tar command to create a pristine-tar branch for generating the orig tarball later. This is useful in case you ommitted using the --git-pristine-tar option when running the git buildpackage tools.

To create a pristine-tar branch, simply do the following.

pristine-tar commit <path_to_orig_tarball> upstream/<package_version>

This assumes you already imported the orig tarball to the git repository. upstream/<package_version> is the tag that was created when the orig tarball was imported using git import-orig. Make sure to specify the tag object.

Then push the new 'pristine-tar' branch.

git push origin pristine-tar #If you cloned the repository
git push alioth pristine-tar #If you created the repository

Converting from svn repository

To convert from the svn repository, first setup the git repository on the alioth servers as described above. Afterwards, do the following on your local machine.

git svn clone --no-metadata \
    svn://svn.debian.org/svn/pkg-games/packages \
    -T /trunk/<package> \
    --branches /branches/<package> \
    --tags /tags/<package>
mv packages <package>

Notes:

After fetching from the svn repository, it's now time to import the contents of the orig source.

$ git symbolic-ref HEAD refs/heads/upstream
$ git rm --cached -r .
$ git commit --allow-empty -m 'initial upstream branch'
$ git checkout -f master
$ git merge upstream
$ git-import-orig --pristine-tar --no-dch <path_to_orig_tarball>

Note: Using the --pristine-tar option is optional.

Now merge the branches and tags from svn into the master branch.

$ git merge \
      $(find .git/refs/remotes -type f ! -regex '.*trunk$' ! -regex '.*trunk@[0-9]*$' \
      ! -regex '.*\/alioth\/.*' -print0 | \
      xargs -0 -r echo | sed 's|\.git/refs/||g')

After this step, push the contents of your local repository.

$ git remote add alioth git+ssh://<username>@git.debian.org/git/pkg-games/<package>.git
$ git push alioth master
$ git push alioth upstream
$ git push alioth pristine-tar
$ git push alioth --tags

Note: Using a 'pristine-tar' branch is created only if you used the --pristine-tar option above.

This step here is a way to push the tags and branches that were imported from svn (an explanation about this is below). Do this to push all tags and branches that were imported.

$ for REMOTES in `find .git/refs/remotes -type f ! -regex '.*trunk$' \
      ! -regex '.*trunk@[0-9]*$' ! -regex '.*\/alioth\/.*'`; \
      do git push alioth $(echo $REMOTES | sed 's|\.git/refs/||'); done

Editing note : Why perform this last step? git-svn will have created branches for every SVN branch, even deleted ones. Should deleted/abandoned branches always be merged?

Once you have pushed all changes to the shared repository, move the packaging directory from trunk in svn so that no one continues to work on it there.

First create a directory under branches for you package if one doesn't exist already.

$ svn mkdir svn+ssh://svn.debian.org/svn/pkg-games/packages/branches/<package> -m "Creating branches section for <package>"

Move the packaging directory in trunk to the branches section and leave an appropriate message.

$ svn mv svn+ssh://svn.debian.org/svn/pkg-games/packages/trunk/<package> \
      svn+ssh://svn.debian.org/svn/pkg-games/packages/branches/<package>/<package>-svn-save \
      -m "Moving <package> from trunk to branch. Please use git repository instead."

And, finally, add a new section for the new repo in the .mrconfig file from from our SVN trunk:

$ (echo "[<package>]" ; \
echo "checkout = git clone git://git.debian.org/git/pkg-games/<package>.git"; \
echo) >> /path/to/the/SVN/trunk/checkout/.mrconfig
$ svn ci -m "Add <package>'s git repo to the .mrconfig file" /path/to/the/SVN/trunk/checkout/.mrconfig

Cloning and pulling

Anyone pulling from a converted repository will have two steps to perform. The first step is the usual cloning.

$ git clone git+ssh://<username>@git.debian.org/git/pkg-games/<package>.git

And then the next step involves pulling the "remotes" branches and tags that were pushed.

$ cd <package>
$ git pull origin +refs/remotes/*:refs/remotes/*

Note about tags and branches

Tags and branches are preserved using what's called "git remotes". This means that there are no real git tags and branches visible, but that the SVN trunk, tags and branches are tracked as separate repositories. Downside is that you need to switch to to the remotes individually before you can update them.

All tags and branches are listed under .git/refs/remotes/ and can be checked out, but it is somewhat confusing as all old schemes for using tags and branches are there too...

So to checkout a branch do the following.

$ git checkout remotes/<some_branch>

And for tags.

$ git checkout remotes/tags/<some_tag>

Generating patches

Here's one way to work with a package to generate patches.

First, create a local branch dedicated for fixes from the master branch and switch to that branch.

$ git checkout master
$ git checkout -b my-fixes

Next, do whatever changes that need to be done and commit them to your "my-fixes" branches. Once you're done with whatever fixes go back to the master branch and generate your patches using git diff.

$ git diff master my-fixes >debian/patches/<some_patch>.patch

Alternatively, you can specify which files to create patches for.

$ git diff master my-fixes -- <path ...> >debian/patches/<some_patch.patch>

If you're using quilt, the entry in the series file should look something like this.

<some_patch>.patch -p1

git format-patch

If you're making major changes to a package, you might want to consider using git format-patch to generate patches. Basically, instead of using git diff to generate the patches, use git format-patch.

$ git format-patch -o debian/patches master my-fixes

Reverting changes while building

There are cases when files have been changed when a package has been built and performing a "clean" operation does not revert all changes back. Here's a command that can be done to revert changes made in such cases. Of course, make sure you are in the master branch or whatever branch you use to commit changes before running this command.

$ git checkout .

It is also possible to revert an individual file by specifying a path argument to git checkout

$ git checkout path/to/file/to/revert

In this case no branch switching occurs and the specified file is reverted to its committed state. Say you have made a commit to your local repository but made a mistake with the commit message, or your email was not set correctly, or you just wish you had not done that commit, git reset is again your friend and you can roll back to be in sync with the last git pull with :-

Reverting commits and files scheduled for commit

If you are working on a package mistakes may be made. If you have files scheduled for commit that were added with git add or git rm and you do not want to commit these files anymore (or want to split the commit in to smaller commits etc), this action can be reverted with :-

$ git reset HEAD <file>

This just removes the specified file from the staging list to be committed on the next commit. If you do not specify a file all files added with git add or git rm will be unstaged.

If you have just committed some changes to your local git tree but want to undo the commit (but not loose changes to your files):-

$ git reset --soft HEAD^

Will remove your commit, leave your files unchanged and back in the state added/removed but not committed.

If you wish to roll back your commit and changes then do :-

$ git reset --hard HEAD~1

This rolls back the head pointer by 1 commit hence the ~1 and permanently looses the changes, DO NOT do this if you have done a git push or the repository will be out of sync and breakage may occur.

Adding a new upstream release

If you are working with an upstream and/or pristine-tar branch then if upstream makes a new release you will want to import. Firstly ensure your tree is clean and all committed and pushed/pulled. Ensure you are on the master branch and that you have the upstream (and if required) the pristine-tar branches created (if not create them with git branch).

$ git checkout master

Next import the upstream tarball, if you are using pristine-tar then specify that option, and if you need to specify the version use the -u option.

$ git import-orig [--pristine-tar] [-u version] /path/to/upstream-version.tar.gz

This will do the following actions :-

  • Upload the contents of the upstream tarball to the upstream branch.
  • (If you are using pristine-tar), uploads the delta between upstream and master to a gziped tarball on the pristine-tar branch.
  • Tags the upstream branch upstream/version
  • Merges the upstream branch into the master branch
  • runs a dch on the master branch to bump the debian version

mr

The mr command can be used to checkout, update, and commit to multiple repositories. In the pkg-games svn repository, is a packages/trunk/.mrconfig file that configures mr for all the repositories used by the games team.

New git repositories should be added to this .mrconfig file.

To use it, install the mr package, and create a ~/.mrconfig file letting mr know about the pkg-games svn repository. In this example, it will be checked out to ~/src/packages/games, but you can modify to suit. If you already have a pkg-games svn checkout, you can change the directory to point to that.

[src/packages/games]
chain = true
checkout = svn co svn+ssh://svn.debian.org/svn/pkg-games/packages/trunk games

Then run "mr checkout", which will checkout the svn repository into ~/src/packages/games.

A second "mr checkout" (necessary because of bug #447553) will checkout the additional git repositories, into subdirectories of ~/src/packages/games/.

Now you can use mr to act on all configured repositories at once.

Update all repositories:

% mr update

Get status of all checkouts:

% mr status

List all repositories that mr knows about:

% mr list

Or even commit changes in each repository:

% mr commit

If you've added a new git repository in ~/src/packages/games/<foo> and are in that directory, you can have mr automatically add it to the pkg-games .mrconfig file by running "mr register". (Don't forget to commit the file.)

SVN

The SVN repository may be browsed online using a browser via the following link:

http://svn.debian.org/wsvn/pkg-games/packages/trunk/?rev=0&sc=0.

The base URI of the repository for authorised SVN access is

svn+ssh://svn.debian.org/svn/pkg-games/

To import your package to SVN

    apt-get install svn-buildpackage

    # Import only the files that are touched
    # by the .diff.gz
    svn-inject -o -l 2 <package_dsc_file>.dsc \
    svn+ssh://svn.debian.org/svn/pkg-games/packages/

NOTE: The -l 2 option for svn-inject specifies layout two. You must have svn-buildpackage (>= 0.6.16) to use this option. Another thing to note is bug 411666 - /usr/bin/svn-inject: initial checkout fails. If your package doesn't exist in the SVN, this bug may affect you. The best thing to do is to use svn-buildpackage (>= 0.6.22).

In very seldom cases (like dead upstream) you would probably want the full source in svn instead just of the contents of the .diff.gz. In that case, remove the -o parameter from the command.

Uploading the Source Tarball

scp <package_name_and_version>.orig.tar.gz \
alioth.debian.org:/var/lib/gforge/chroot/home/groups/pkg-games/htdocs/tarballs/
  • Next, set the property svn-bp:origUrl of the debian folder of the package to point to the corresponding orig tarball

svn co \
svn+ssh://svn.debian.org/svn/pkg-games/packages/trunk/<package_name>
cd <package_name>
svn propset svn-bp:origUrl \
"http://pkg-games.alioth.debian.org/tarballs/<package_name_and_version>.orig.tar.gz" debian
svn ci -N debian -m "origUrl property set for <package_name_and_version>"

To Checkout A Project

    svn co \
    svn+ssh://svn.debian.org/svn/pkg-games/packages/trunk/"packageName"

To build

    # Place the .orig.tar.gz in ../tarballs/

    # Build the package
    svn-buildpackage -rfakeroot

    # or, if you want to make it the build before uploading, and
    # have svn-buildpackage make a svn tag for you
    svn-buildpackage --svn-tag -rfakeroot

    # if you have local changes in your working copy and want to
    # run a test-build, use the `--svn-ignore' switch like so:
    svn-buildpackage --svn-ignore -rfakeroot

    # example for freecol:
    $ svn proplist -v debian
    Properties on 'debian':
      svn-bp:origUrl : http://pkg-games.alioth.debian.org/tarballs/freecol_0.7.2.orig.tar.gz
      mergeWithUpstream : 1

To build using pbuilder

pbuilder is a great tool to check that your package will build in a basic Debian system with the proper build-dependencies. It's also useful in avoiding having to install a package's build-dependencies on your system.

First, you will have to setup pbuilder on your system. The HOWTO at https://wiki.ubuntu.com/PbuilderHowto provides a good guide and some good tips in setting up pbuilder.

Afterwards, it should be just a matter of using the --svn-builder option. For example, to use pdebuild and ignore all .svn directories, use:

svn-buildpackage --svn-dont-clean --svn-builder='pdebuild --debbuildopts "-i\.svn/ -I.svn"'

Alternatively, you can use a simple script as svn-builder. Here's a script taken from the examples from the git-buildpackage package that's been modified to work in the case of svn-buildpackage.

#!/bin/sh
# 
# pbuilder helper from git-buildpackage
# use this as "svn-builder" in ~/.svn-buildpackage.conf
set -e

# pass all options to dpkg-buildpackage:
pdebuild $PBUILDER_OPTS --debbuildopts "-i\.svn/ -I.svn $*"

Create a file called svn-pdebuild under /usr/local/bin and add the contents of the above script into the file. Then you can just run:

svn-buildpackage --svn-builder=svn-pdebuild --svn-dont-clean

You can also specify svn-builder using a configuration file at ~/.svn-buildpackage.conf. Create a file at ~/.svn-buildpackage.conf and add this snippet.

svn-builder=svn-pdebuild
svn-dont-clean

Then just run:

svn-buildpackage

Finally, if you need to specify more options for pdebuild's --debbuildopts option, just add them in like you would normally do for --debbuildopts. Here's an example that works using the method that uses ~/.svn-buildpackage.conf.

svn-buildpackage -sa -us -uc

svn-buildpackage will pass any options it doesn't recognize onto whatever command was specified as svn-builder. If you need to specify options for pbuilder, assign each option to PBUILDER_OPTS.

NOTE: The svn-dont-clean option is used since the Debian Games team SVN won't contain upstream source files.

To release

In order to have a lower threshold for someone to step in to help in for a game, some guide lines should be followed when working with the games maintained in our repository.

Thus, here are a few simple rules that should make our lives a little better.

  • when a package version it is not released, the topmost entry in the debian/changelog file should contain the string "UNRELEASED" instead of unstable or other valid suite name

    # example: boson-base changelog between the released version 0.11-2 and the to be relased 0.11-3

    $ head -n1 boson-base/trunk/debian/changelog
boson-base (0.11-3) UNRELEASED; urgency=low

This can be acomplished if svn-buildpackage is used when handling releases.

Before releasing, please check that your package is lintian clean. Automatic checks are done on our SVN tree by the Games/Autobuilder.

Due to bug 433536 in svn-buildpackage (which should be fixed in release 0.6.22), the "tags" directory was not created for packages injected using layout 2, the layout used by our team. If you are using an older version of svn-buildpackage or not at all, please make sure that the tags directory for your package exists under the "tags" directory in SVN:

svn mkdir svn+ssh://svn.debian.org/svn/pkg-games/packages/tags/<package>

Then you will be able to tag a release.

    # make a final build for a package
    $ svn-buildpackage --svn-tag

If your package is accepted into the Debian archive already, you can save some time by just tagging the release.

svn-buildpackage --svn-tag-only --svn-noautodch

The --svn-noautodch will prevent the changelog from being tagged UNRELEASED.

Possible problems

git import-orig dch error

When using git import-orig, there is a chance you'll receive an error similar to this one.

dch: fatal error at line 963:
New version specified (1.4.8.dfsg1-1) is less than
the current version number (1.4.8.dfsg1-1)!  Use -b to force.
dch returned 25
Dch failed.
Import of ../../build-area/ogre_1.4.8.dfsg1.orig.tar.gz failed

This will happen if you omit the --no-dch option while running git import-orig and the version in the changelog is at the version of the orig tarball that's being imported.

git import-orig pristine-tar error

It's possible you may receive an error such as this.

pristine-tar: more than one ref matches "upstream":
5f4736c83ea06e9b479a35f25d997b73713306b9 refs/heads/upstream
a8c20d823bc28bb8f6d127e301390bed06e49169 refs/remotes/origin/upstream
/usr/bin/pristine-tar returned 255
Couldn't run '/usr/bin/pristine-tar'

This is caused by git import-orig (or any of the git-buildpackage tools) supplying the parameter "upstream" to use when running pristine-tar commit instead of using the default refs/heads/upstream or supplying a tag object instead and you cloned a git repository. You will have to omit the --pristine-tar option from the git-buildpackage tools and use pristine-tar manually. See 481806.

General Rules

  • If a package you're working on is for a particular version that has not been uploaded to the archive, you should either append ~unrelease0 at the end of the version, or use 'UNRELEASED' as the suite in the changelog.

  • packages/ in the svn repo is for free games only. For games which should go to non-free, please use non-free/package/.
  • the SVN properties of the debian/ directory should point to the location where the orig.tar.gz file(s) can be fetched.

See Also