debian/watch

The watch file located in the debian directory serves the purpose of monitoring for updates of upstream software and, if available, initiating its download. This downloading process is facilitated by the uscan program, which is part of the devscripts package. uscan can be invoked with the path to the debian directory containing the watch file as an argument, or it can traverse through directories below the current working directory to locate the necessary files.

Please see also the Debian Policy Manual about debian/watch.

Basically a watch file will have this format:

  version=4
  http://somesite.com/dir/filenamewithversion.tar.gz

To allow the version to remain unspecified, it is expressed as a wildcard using regular expression in Perl format:

  version=4
  http://somesite.com/dir/filename_(.+).tar.gz

The uscan program will then execute a "dir" command and check all the files in that directory for the highest version number.

uscan also supports the FTP protocol. Unfortunately, when working with HTTP, HTML pages do not always contain directory listing with files. The complete (or relative) path of the .tar.gz file to be downloaded will appear as a hyperlink within the web page. We hence need two types of information, the path to the page announcing the file and a regular expression to "grep" for the right link:

  version=4
  http://somesite.com/path link_from_href-1.0.tar.gz

"http://somesite.com/path" is the site from where you are downloading the source and "link_from_href.1.0.tar.gz" is obtained from the HTML source code (from the "<a href=" tags).

If http://somesite.com/path HTML code has "<a href=foo/program-1.0.tar.gz>" inside it, for example, you will use

  http://somesite.com/dir foo/program-(.+)\.tar\.gz

Watch files with errors are generally wrong on this second part.

Generally you want a slightly more flexible regex for the tarball name so that if upstream switches tarball naming schemes or compression formats you are covered:

  http://somesite.com/dir foo/program-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

For FTP sites it changes a little, but it's basically the same thing (man uscan will help you).

But the nice part is to test it:

  uscan --no-download --verbose

If it's not working as expected, you can use debug to see what it's fetching and what it's (not) matching:

  uscan --no-download --verbose --debug
Nice write-up by João Eriberto

http://eriberto.pro.br/blog/?p=1459

Common mistakes

Version number gotchas

For upstreams that release alpha, beta, release-candidate versions it's common to use versions like 1.0-alpha, 1.0-rc1, ... before 1.0 Without proper mangling this will break dpkg version comparisons:

dpkg --compare-versions '1.0-1' gt '1.0-alpha-1'   # false
dpkg --compare-versions '1.0-1' gt '1.0~alpha-1'   # true (notice the ~)

The solution is to use uversionmangle like this opts=uversionmangle=s/(\d)[_\.\-\+]?((RC|rc|pre|dev|beta|alpha)\d*)$/$1~$2/ before importing new releases.

If you already uploaded a "broken" version number you can sometimes use the ds<n> repack suffix e.g. '1.0-ds1-1' or '1.0.ds1-1'

See https://codesearch.debian.net/search?q=repacksuffix%3D%2Bds1&literal=1

Cryptographic signature verification

If your upstream provides cryptographic signatures for their packages in the same place that the packages are available for download, and you know what OpenPGP key or keys will be used to sign these packages, uscan can verify these signatures for you if you give it the pgpsigurlmangle option.

For example, for OpenSSH (which uses a .asc suffix for the package signatures), you'd place Damien Miller's public key ascii-armored in debian/upstream/signing-key.asc:

gpg --armor --export-options export-minimal --export '59C2 118E D206 D927 E667  EBE3 D3E5 F56B 6D92 0D30' >> debian/upstream/signing-key.asc

and then update debian/watch to say:

version=4
opts=pgpsigurlmangle=s/$/.asc/ ftp://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-(.+)\.tar\.gz 

This lets you ensure that the package was not tampered with in transit, and that it came from the developer you expected it to come from. A valid signature does not mean that the contents of the package are somehow magically perfect and DFSG-free and policy compliant, of course!

Often, when upstream uses git, it doesn't sign its releases, but signs its tags. Here is an example of a debian/watch file using mode=git to retrieve releases and checking tags signatures:

version=4
opts="mode=git,pgpmode=gittag" \
https://mygitlabinstance.tld/myusername/myproject.git refs/tags/v([\d\.]+)

Upstream source sites

GitHub

If the project using version tags:

version=4
https://github.com/<REPONAME>/tags .*/v?(\d.*)@ARCHIVE_EXT@

If the project offer a Release section:

version=4
https://github.com/<REPONAME>/releases/download/v(\d+).(\d+).(\d+).tar.gz

If upstream releases beta/alpha tarballs, you will need to make use of uversionmangle.

version=4
opts="searchmode=plain,\
filenamemangle=s%v?@ANY_VERSION@%@PACKAGE@-$1.tar.gz%,\
uversionmangle=s/(\d)[_\.\-\+]?((RC|rc|pre|dev|beta|alpha|a|b)\d*)$/$1~$2/" \
https://api.github.com/repos/<user>/<project>/releases?per_page=50 \
https://github.com/<user>/<project>/releases/download/@ANY_VERSION@/@PACKAGE@-@ANY_VERSION@.tar.gz"

If your d/watch is failing to find the tarballs, you can analyze the page at:

https://api.github.com/repos/<user>/<project>/releases?per_page=50

And identify a regex that can be used to fetch the right tarball.

As of 2022-10-02, ?GitHub change the way its release/tags page is rendered. The links are dynamically loaded in ?JavaScript instead of static text. See thread for the discussions: https://lists.debian.org/debian-devel/2022/09/msg00224.html

Example using the Github API:

Although it is possible to obtain a tarball it another way, the API also works and it is up to the maintainer's discretion which of the options to use.

version=4
opts="searchmode=plain,\
filenamemangle=s%v?@ANY_VERSION@%@PACKAGE@-$1.tar.xz%" \
https://api.github.com/repos/<user>/<project>/releases?per_page=100 \
https://api.github.com/repos/[^/]+/[^/]+/tarball/v?@ANY_VERSION@

Manipulation using v1.0.0 (example) and signed tarballs (for security reasons as integrity check, it is highly recommended and see how can I verify tor source code?):

version=4
opts=downloadurlmangle=s/archive\/refs\/tags\/v(.*)\.tar\.gz/releases\/download\/v$1\/@PACKAGE@-$1\.tar\.xz/,\
    pgpsigurlmangle=s/$/.asc/ \
    https://github.com/<user>/@PACKAGE@/tags \
    (?:.*?/)?v?@ANY_VERSION@@ARCHIVE_EXT@

GitLab

From 2024/03 the packages that receive tarballs from GitLab need to be changed (showing the error debian/watch nomatching files for watch), the solution below has been tested and other previous solutions will not work. GitLab changed the tag order and debian/watch will have to be updated and suggestions below:

version=4
opts="searchmode=plain" \
 https://gitlab.com/<user>/@PACKAGE@/tags?sort=updated_desc -/archive/v?\d[\d.]+/@PACKAGE@-@ANY_VERSION@@ARCHIVE_EXT@ debian uupdate

Or

version=4
opts="searchmode=plain" \
 https://gitlab.com/<user>/@PACKAGE@/tags?sort=updated_desc -/archive/v?\d[\d.]+/@PACKAGE@-@ANY_VERSION@@ARCHIVE_EXT@

You can also use filenamemangle following the Salsa example. See https://lists.debian.org/debian-mentors/2024/03/msg00259.html

Fay Stegerman figured out how to get tarball using upload URL instead of archive URL and she explains about it here: https://lists.debian.org/debian-mentors/2024/04/msg00013.html and example Switch to new version scheme. See also uscan (wget) no longer presented the same HTML as a normal browser for releases.

Salsa (Gitlab)

Please ensure that the tags are strictly in numeric format. For example, the version tag should be written as 1.4.5.

version=4
opts=filenamemangle=s/.*?\/(\d\S+)\/archive\.tar\.gz/<project>-$1\.tar\.gz/ \
 https://salsa.debian.org/<user>/<project>/tags .*?(\d\S+)/archive\.tar\.gz

Gitea / Forgejo / Codeberg.org

If the project use release tags like v1.2.3.

version=4
https://codeberg.org/<user>/<project>/tags .*/v(\d+).(\d+).(\d+).tar.gz

Bitbucket

version=4
https://bitbucket.org/<user>/<project>/downloads?tab=tags .*/(\d\S+)\.tar\.gz

The 2nd regexp matches most often used tags like '0.1.2' or 'v0.1.2'. It may need to be adapted if upstream uses other version tags.

Launchpad

version=4
opts=pgpsigurlmangle=s/$/.asc/ \
  https://launchpad.net/<project>/ \ 
  https://launchpad.net/.*download/<project>-([.\d]+)\.tar\.xz

PyPI

The PyPI site is hard to work with directly as it includes md5sums for the downloads in the URLs which makes for truly horrible regexps. The pypi.debian.net redirector makes it much easier to work with PyPI:

version=4
opts=pgpsigurlmangle=s/$/.asc/ \
https://pypi.debian.net/<project>/<project>-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

There's also (a bit more advanced) autogenerated watch file that you can download from http://pypi.debian.net/<project>/watch

If you do want to work directly with PyPI, then make sure you use the new Simple API:

https://pypi.python.org/simple/<project>/ 

Note that direct use of the PyPI listing pages for packages (such as https://pypi.python.org/pypi/<package>/) is discouraged by the PyPI administrators. Automated use of package listing pages is not supported and is liable to break; the Simple API is provided for precisely that reason.

SourceForge

  version=4
  # qa.debian.org runs a redirector which allows a simpler form of URL
  # for SourceForge based projects. The format below will automatically
  # be rewritten to use the redirector.
  http://sf.net/audacity/audacity-src-(\d\S+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

See also: https://salsa.debian.org/qa/qa/tree/master/wml/watch/

KDE

  version=4
  opts=pgpsigurlmangle=s/$/.sig/ https://download.kde.org/stable/release-service/([\d.]+)/src/@PACKAGE@-([\d.]+)\.tar\.xz

Uncommon sites

If you have a site that has version numbers in some form but doesn't have hrefs containing them and the URL mangling capabilities of uscan are not enough, you can create a redirector. The Debian QA folks run one called fakeupstream.cgi (git) for lots of different upstream sites. If you want to add a new one, please submit a wishlist bug report assigned to qa.debian.org with or without a patch.

Files hosted on various big project hosting sites can be specified with the URLs below. SourceForge has a special "redirector" URL (see more details in 'man uscan'). This allows the upstream URLs to change without the need to adapt watch files of affected packages.

Autogenerated

A scripted debian/watch generator is available

Further information

Other distros

Fedora also has an upstream release monitoring project (more info).


CategoryDebianDirectory | CategoryPackaging