Differences between revisions 61 and 101 (spanning 40 versions)
Revision 61 as of 2014-07-26 11:43:42
Size: 7629
Editor: GuillemJover
Comment: Add hint about using pgp-clean from signing-party
Revision 101 as of 2020-11-28 23:31:50
Size: 12067
Editor: ?FabienGivors
Comment:
Deletions are marked like this. Additions are marked like this.
Line 6: Line 6:
The file named 'watch' in the debian directory is used to check for newer versions of upstream software is available and to download it if necessary. The download itself will be performed with the [[DebianMan:1/uscan|uscan]] program from the DebianPkg:devscripts package. It takes the path to the {{{debian}}} directory that uses the {{{watch}}} file as an argument or searches the directories underneath the current working directory. The file named {{{watch}}} in the {{{debian}}} directory is used to check for newer versions of upstream software is available and to download it if necessary. The download itself will be performed with the [[DebianMan:1/uscan|uscan]] program from the DebianPkg:devscripts package. It takes the path to the {{{debian}}} directory that uses the {{{watch}}} file as an argument or searches the directories underneath the current working directory.
Line 10: Line 10:
  version=3   version=4
Line 16: Line 16:
  version=3   version=4
Line 21: Line 21:
{{{uscan}}} also supports ftp protocol. Unfortunately, when working with http, HTML pages do not always contain directory listing with files. The complete (or relative) path  of the .tar.gz file to be downloaded will appear as a hyperlink within the web page. We hence need two types of information, the path to the page announcing the file and a regular expression to grep for the right link:
{{{
  version=3
{{{uscan}}} also supports the FTP protocol. Unfortunately, when working with HTTP, HTML pages do not always contain directory listing with files. The complete (or relative) path of the .tar.gz file to be downloaded will appear as a hyperlink within the web page. We hence need two types of information, the path to the page announcing the file and a regular expression to "grep" for the right link:
{{{
  version=4
Line 56: Line 56:
 Nice write-up by João Eriberto:: http://eriberto.pro.br/blog/?p=1459
Line 60: Line 62:
 * Not anchoring the version group at the right place. The solution is to include something before {{{(\d\S*)}}} like {{{fooproj-(\d\S*)\.tar\.gz}}}  * Not anchoring the version group at the right place. The solution is to include something before {{{(\d\S+)}}} like {{{fooproj-(\d\S+)\.tar\.gz}}}
Line 62: Line 64:
 * Not being flexible enough in the path to the file. The solution is to use {{{http://example.com/someproject/ .*/program-(\d\S*)\.tar\.gz}}} instead of {{{http://example.com/someproject/ /path/to/program/downloads/program-(\d\S*)\.tar\.gz}}}  * Not being flexible enough in the path to the file. The solution is to use {{{http://example.com/someproject/ .*/program-(\d\S+)\.tar\.gz}}} instead of {{{http://example.com/someproject/ /path/to/program/downloads/program-(\d\S+)\.tar\.gz}}}
Line 64: Line 66:
 * Not mangling Debian versions to remove {{{+dfsg.1}}} suffix. The solution is to use dversionmangle like this {{{opts=dversionmangle=s/\+(debian|dfsg|ds|deb)(\.\d+)?$//}}}  * Not mangling Debian versions to remove the {{{+dfsg.1}}} or {{{+dfsg1}}} suffix. The solution is to use dversionmangle like this {{{opts=dversionmangle=s/\+(debian|dfsg|ds|deb)(\.?\d+)?$//}}}
Line 75: Line 77:
gpg --armor --export '59C2 118E D206 D927 E667 EBE3 D3E5 F56B 6D92 0D30' >> debian/upstream/signing-key.asc
}}}
or by using pgp-clean from DebianPkg:signing-party to get the smallest output:
{{{
pgp-clean D3E5F56B6D920D30 >> debian/upstream/signing-key.asc
gpg --armor --export-options export-minimal --export '59C2 118E D206 D927 E667 EBE3 D3E5 F56B 6D92 0D30' >> debian/upstream/signing-key.asc
Line 85: Line 83:
version=3 version=4
Line 91: Line 89:
== Uncommon sites ==

If you have a site that has version numbers in some form but doesn't have hrefs containing them and the URL mangling capabilities of uscan are not enough, you can create a redirector. The Debian QA folks run one called [[http://qa.debian.org/cgi-bin/fakeupstream.cgi|fakeupstream.cgi]] ([[http://anonscm.debian.org/viewvc/qa/trunk/cgi-bin/fakeupstream.cgi?view=markup|SVN]]) for lots of different upstream sites. If you want to add a new one,
please submit a wishlist [[http://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=qa.debian.org|bug report assigned to qa.debian.org]]
Often, when upstream uses git, it doesn't sign its releases, but signs its tags. Here is an example of a {{{debian/watch}}} file using {{{mode=git}}} to retrieve releases and checking tags signatures:
{{{
version=4
opts="mode=git,pgpmode=gittag" \
https://mygitlabinstance.tld/myusername/myproject.git refs/tags/v([\d\.]+)
}}}

== Uncommon upstream source sites ==

If you have a site that has version numbers in some form but doesn't have hrefs containing them and the URL mangling capabilities of uscan are not enough, you can create a redirector. The Debian QA folks run one called [[https://qa.debian.org/cgi-bin/fakeupstream.cgi|fakeupstream.cgi]] ([[https://salsa.debian.org/qa/qa/blob/master/cgi-bin/fakeupstream.cgi|git]]) for lots of different upstream sites. If you want to add a new one,
please submit a wishlist [[https://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=qa.debian.org|bug report assigned to qa.debian.org]]
Line 97: Line 102:
== Common sites == == Common upstream source sites ==
Line 101: Line 106:
=== Bitbucket ===

{{{
version=4
https://bitbucket.org/<user>/<project>/downloads?tab=tags .*/(\d\S+)\.tar\.gz
}}}

The 2nd regexp matches most often used tags like '0.1.2' or 'v0.1.2'. It may need to be adapted if upstream uses other version tags.

=== GitHub ===

{{{
version=4
opts=filenamemangle=s/.+\/v?(\d\S+)\.tar\.gz/<project>-$1\.tar\.gz/ \
  https://github.com/<user>/<project>/tags .*/v?(\d\S+)\.tar\.gz
}}}

The 2nd regexp matches most often used tags like '0.1.2' or 'v0.1.2'. It may need to be adapted if upstream uses other version tags. You may also want to use {{{releases}}} instead of {{{tags}}}.

Some projects using higher versions with two or more digits without a starting 'v' so you may use the following example. This matches versions like '10.0.0' up to '99.9.9'.

{{{
version=4
opts=filenamemangle=s/.+\/v?(\d\S+)\.tar\.gz/<project>-$1\.tar\.gz/ \
  https://github.com/<user>/<project>/tags .*/?(\d{2}\.\d.\d)\.tar\.gz
}}}

Your upstream may [[Creating signed GitHub releases|sign their releases with GnuPG]]. !GitHub places these detached signatures in another directory than the actual release tarball. You may verify the tarball with something like this:

{{{
version=4
opts=[...],\
pgpsigurlmangle=s/archive\/(\d\S+)\.tar\.gz/releases\/download\/$1\/$1\.tar\.gz\.asc/ \
  https://github.com/<user>/<project>/releases/latest .*/(\d\S+)\.tar\.gz
}}}

=== Gitlab ===

{{{
version=4
opts=filenamemangle=s/.*\/(\d\S+)\/archive\.tar\.gz/<project>-$1\.tar\.gz/g \
  https://gitlab.com/<user>/<project>/tags?sort=updated_desc .*(\d\S+)/archive\.tar\.gz
}}}

Gitlab seems to have changed their directory layout. If the above no longer works, try this:

{{{
version=4
opts=filenamemangle=s/.*\/archive\/(\d\S+)\/<project>.*\.tar\.gz/<project>-$1\.tar\.gz/g \
  https://gitlab.com/<user>/<project>/tags?sort=updated_desc .*/archive/(\d\S+)/.*\.tar\.gz.*
}}}

As in 2020-06, GitLab is listing archives in a new format:

{{{
# Matches https://gitlab.com/<user>/<project>/-/archive/<ver>/<proj>-<ver>.tar.bz2
version=4
https://gitlab.com/<user>/<project>/tags?sort=updated_desc archive/@ANY_VERSION@/{project}-\d\S*@ARCHIVE_EXT@
}}}


Your upstream may sign their releases with GnuPG by [[Creating signed GitHub releases|adapting the GitHub instructions]]. !GitLab puts the signature attachment into an unpredictable subdirectory of /<user>/<project>/uploads/. You may verify the tarball with something like this (assuming they only sign the .tar.gz):

{{{
version=4

opts="pgpmode=next" \
  https://gitlab.com/<user>/<project>/tags \
  (?:.*/)@PACKAGE@@ANY_VERSION@\.tar\.gz

opts="pgpmode=previous" \
  https://gitlab.com/<user>/<project>/tags \
  (?:.*/)@PACKAGE@@ANY_VERSION@@SIGNATURE_EXT@ \
  previous
}}}

=== Salsa (Gitlab) ===

{{{
version=4
opts=filenamemangle=s/.*?\/(\d\S+)\/archive\.tar\.gz/<project>-$1\.tar\.gz/ \
 https://salsa.debian.org/<user>/<project>/tags .*?(\d\S+)/archive\.tar\.gz
}}}

Make sure that the tags should be just the numeric version in salsa.d.o. Exemple version: '''1.4.5'''

{{{$ git tag 1.4.5 -m 'Release version 1.4.5'}}}

{{{$ git push origin master --tags}}}

=== Google Code ===

{{{
version=4
http://code.google.com/p/<project>/downloads/list?can=1 .*/<project>-(\d\S+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))
}}}

=== Launchpad ===

{{{
version=4
opts=pgpsigurlmangle=s/$/.asc/ \
  https://launchpad.net/<project>/ \
  https://launchpad.net/.*download/<project>-([.\d]+)\.tar\.xz
}}}

=== PyPI ===

The PyPI site is hard to work with directly as it includes md5sums for the downloads in the URLs which makes for truly horrible regexps. The pypi.debian.net redirector makes it much easier to work with PyPI:
{{{
version=4
opts=pgpsigurlmangle=s/$/.asc/ \
https://pypi.debian.net/<project>/<project>-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))
}}}

There's also (a bit more advanced) autogenerated watch file that you can download from http://pypi.debian.net/<project>/watch

If you do want to work directly with PyPI, then make sure you use the new Simple API:
{{{
https://pypi.python.org/simple/<project>/
}}}
Note that direct use of the PyPI listing pages for packages (such as {{{https://pypi.python.org/pypi/<package>/}}}) is discouraged by the PyPI administrators. Automated use of package listing pages is not supported and is liable to break; the Simple API is provided for precisely that reason.
Line 104: Line 232:
  version=4
Line 107: Line 236:
  http://sf.net/audacity/audacity-src-(\d\S*)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))
}}}
See also: http://anonscm.debian.org/viewvc/qa/trunk/wml/watch/

=== GitHub ===

{{{
opts=filenamemangle=s/.+\/v?(\d\S*)\.tar\.gz/<project>-$1\.tar\.gz/ \
  https://github.com/<user>/<project>/tags .*/v?(\d\S*)\.tar\.gz
}}}

The 2nd regexp matches most often used tags like '0.1.2' or 'v0.1.2'. It may need to be adapted if upstream uses other version tags. You may also want to use {{{releases}}} instead of {{{tags}}}.

=== Google Code ===

{{{
http://code.google.com/p/<project>/downloads/list?can=1 .*/<project>-(\d\S*)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))
}}}

=== Gitorious ===

[[https://lists.debian.org/debian-mentors/2013/11/msg00300.html|Some reports]] say this works:
{{{
version=3
opts=filenamemangle=s/\S*download=//g \
http://qa.debian.org/cgi-bin/fakeupstream.cgi?upstream=gitorious/<user>/<project> \
.*=<project>(?:[_\-]v?|)(\d[^\s/]*)\.(?:tar\.xz|txz|tar\.bz2|tbz2|tar\.gz|tgz)
}}}



=== Bitbucket ===

{{{
https://bitbucket.org/<user>/<project>/downloads .*/(\d\S*)\.tar\.gz
}}}

The 2nd regexp matches most often used tags like '0.1.2' or 'v0.1.2'. It may need to be adapted if upstream uses other version tags.
  http://sf.net/audacity/audacity-src-(\d\S+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))
}}}
See also: https://salsa.debian.org/qa/qa/tree/master/wml/watch/

=== Autogenerated ===

A scripted [[https://salsa.debian.org/debian/debian-package-init/-/blob/master/deb_create_watch.py|debian/watch generator]] is available

= Further information =
 * Explaining blog post how to write a useful regexp line: http://eriberto.pro.br/blog/2013/10/07/how-to-write-a-good-debianwatch-easily/
 * Ignore some upstream versions, such as upstream changed from v20190101 to v1.2.3: https://lists.debian.org/debian-mentors/2020/08/msg00025.html

= Other distros =

Fedora also has an [[https://release-monitoring.org/|upstream release monitoring project]] ([[https://fedoraproject.org/wiki/Upstream_release_monitoring|more info]]).
Line 148: Line 253:
CategoryDebianDirectory CategoryDebianDirectory | CategoryPackaging

debian/watch

The file named watch in the debian directory is used to check for newer versions of upstream software is available and to download it if necessary. The download itself will be performed with the uscan program from the devscripts package. It takes the path to the debian directory that uses the watch file as an argument or searches the directories underneath the current working directory.

Basically a watch file will have this format:

  version=4
  http://somesite.com/dir/filenamewithversion.tar.gz

To allow the version to remain unspecified, it is expressed as a wildcard using regular expression in Perl format:

  version=4
  http://somesite.com/dir/filename_(.+).tar.gz

The uscan program will then execute a "dir" command and check all the files in that directory for the highest version number.

uscan also supports the FTP protocol. Unfortunately, when working with HTTP, HTML pages do not always contain directory listing with files. The complete (or relative) path of the .tar.gz file to be downloaded will appear as a hyperlink within the web page. We hence need two types of information, the path to the page announcing the file and a regular expression to "grep" for the right link:

  version=4
  http://somesite.com/path link_from_href-1.0.tar.gz

"http://somesite.com/path" is the site from where you are downloading the source and "link_from_href.1.0.tar.gz" is obtained from the HTML source code (from the "<a href=" tags).

If http://somesite.com/path HTML code has "<a href=foo/program-1.0.tar.gz>" inside it, for example, you will use

  http://somesite.com/dir foo/program-(.+)\.tar\.gz

Watch files with errors are generally wrong on this second part.

Generally you want a slightly more flexible regex for the tarball name so that if upstream switches tarball naming schemes or compression formats you are covered:

  http://somesite.com/dir foo/program-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

For FTP sites it changes a little, but it's basically the same thing (man uscan will help you).

But the nice part is to test it:

  uscan --no-download --verbose

If it's not working as expected, you can use debug to see what it's fetching and what it's (not) matching:

  uscan --no-download --verbose --debug
Nice write-up by João Eriberto

http://eriberto.pro.br/blog/?p=1459

Common mistakes

  • Not escaping dots, which match any character. The solution is \. instead of . in the regex.

  • A file extension regex that is not flexible enough. The solution is \.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

  • Not anchoring the version group at the right place. The solution is to include something before (\d\S+) like fooproj-(\d\S+)\.tar\.gz

  • Not starting the version part of the regex with a digit. The solution is to use \d instead of .

  • Not being flexible enough in the path to the file. The solution is to use http://example.com/someproject/ .*/program-(\d\S+)\.tar\.gz instead of http://example.com/someproject/ /path/to/program/downloads/program-(\d\S+)\.tar\.gz

  • Not mangling upstream versions that are alphas, betas or release candidates to make them sort before the final release. The solution is to use uversionmangle like this opts=uversionmangle=s/(\d)[_\.\-\+]?((RC|rc|pre|dev|beta|alpha)\d*)$/$1~$2/

  • Not mangling Debian versions to remove the +dfsg.1 or +dfsg1 suffix. The solution is to use dversionmangle like this opts=dversionmangle=s/\+(debian|dfsg|ds|deb)(\.?\d+)?$//

  • Not enabling cryptographic signature verification when your upstream signs their releases with OpenPGP.

Cryptographic signature verification

If your upstream provides cryptographic signatures for their packages in the same place that the packages are available for download, and you know what OpenPGP key or keys will be used to sign these packages, uscan can verify these signatures for you if you give it the pgpsigurlmangle option.

For example, for OpenSSH (which uses a .asc suffix for the package signatures), you'd place Damien Miller's public key ascii-armored in debian/upstream/signing-key.asc:

gpg --armor --export-options export-minimal --export '59C2 118E D206 D927 E667  EBE3 D3E5 F56B 6D92 0D30' >> debian/upstream/signing-key.asc

and then update debian/watch to say:

version=4
opts=pgpsigurlmangle=s/$/.asc/ ftp://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-(.+)\.tar\.gz 

This lets you ensure that the package was not tampered with in transit, and that it came from the developer you expected it to come from. A valid signature does not mean that the contents of the package are somehow magically perfect and DFSG-free and policy compliant, of course!

Often, when upstream uses git, it doesn't sign its releases, but signs its tags. Here is an example of a debian/watch file using mode=git to retrieve releases and checking tags signatures:

version=4
opts="mode=git,pgpmode=gittag" \
https://mygitlabinstance.tld/myusername/myproject.git refs/tags/v([\d\.]+)

Uncommon upstream source sites

If you have a site that has version numbers in some form but doesn't have hrefs containing them and the URL mangling capabilities of uscan are not enough, you can create a redirector. The Debian QA folks run one called fakeupstream.cgi (git) for lots of different upstream sites. If you want to add a new one, please submit a wishlist bug report assigned to qa.debian.org with or without a patch.

Common upstream source sites

Files hosted on various big project hosting sites can be specified with the URLs below. SourceForge has a special "redirector" URL (see more details in 'man uscan'). This allows the upstream URLs to change without the need to adapt watch files of affected packages.

Bitbucket

version=4
https://bitbucket.org/<user>/<project>/downloads?tab=tags .*/(\d\S+)\.tar\.gz

The 2nd regexp matches most often used tags like '0.1.2' or 'v0.1.2'. It may need to be adapted if upstream uses other version tags.

GitHub

version=4
opts=filenamemangle=s/.+\/v?(\d\S+)\.tar\.gz/<project>-$1\.tar\.gz/ \
  https://github.com/<user>/<project>/tags .*/v?(\d\S+)\.tar\.gz

The 2nd regexp matches most often used tags like '0.1.2' or 'v0.1.2'. It may need to be adapted if upstream uses other version tags. You may also want to use releases instead of tags.

Some projects using higher versions with two or more digits without a starting 'v' so you may use the following example. This matches versions like '10.0.0' up to '99.9.9'.

version=4
opts=filenamemangle=s/.+\/v?(\d\S+)\.tar\.gz/<project>-$1\.tar\.gz/ \
  https://github.com/<user>/<project>/tags .*/?(\d{2}\.\d.\d)\.tar\.gz

Your upstream may sign their releases with GnuPG. GitHub places these detached signatures in another directory than the actual release tarball. You may verify the tarball with something like this:

version=4
opts=[...],\
pgpsigurlmangle=s/archive\/(\d\S+)\.tar\.gz/releases\/download\/$1\/$1\.tar\.gz\.asc/ \
  https://github.com/<user>/<project>/releases/latest .*/(\d\S+)\.tar\.gz

Gitlab

version=4
opts=filenamemangle=s/.*\/(\d\S+)\/archive\.tar\.gz/<project>-$1\.tar\.gz/g \
  https://gitlab.com/<user>/<project>/tags?sort=updated_desc .*(\d\S+)/archive\.tar\.gz

Gitlab seems to have changed their directory layout. If the above no longer works, try this:

version=4
opts=filenamemangle=s/.*\/archive\/(\d\S+)\/<project>.*\.tar\.gz/<project>-$1\.tar\.gz/g \
  https://gitlab.com/<user>/<project>/tags?sort=updated_desc .*/archive/(\d\S+)/.*\.tar\.gz.*

As in 2020-06, GitLab is listing archives in a new format:

# Matches https://gitlab.com/<user>/<project>/-/archive/<ver>/<proj>-<ver>.tar.bz2
version=4
https://gitlab.com/<user>/<project>/tags?sort=updated_desc archive/@ANY_VERSION@/{project}-\d\S*@ARCHIVE_EXT@    

Your upstream may sign their releases with GnuPG by adapting the GitHub instructions. GitLab puts the signature attachment into an unpredictable subdirectory of /<user>/<project>/uploads/. You may verify the tarball with something like this (assuming they only sign the .tar.gz):

version=4

opts="pgpmode=next" \
  https://gitlab.com/<user>/<project>/tags \
  (?:.*/)@PACKAGE@@ANY_VERSION@\.tar\.gz

opts="pgpmode=previous" \
  https://gitlab.com/<user>/<project>/tags \
  (?:.*/)@PACKAGE@@ANY_VERSION@@SIGNATURE_EXT@ \
  previous

Salsa (Gitlab)

version=4
opts=filenamemangle=s/.*?\/(\d\S+)\/archive\.tar\.gz/<project>-$1\.tar\.gz/ \
 https://salsa.debian.org/<user>/<project>/tags .*?(\d\S+)/archive\.tar\.gz

Make sure that the tags should be just the numeric version in salsa.d.o. Exemple version: 1.4.5

$ git tag 1.4.5 -m 'Release version 1.4.5'

$ git push origin master --tags

Google Code

version=4
http://code.google.com/p/<project>/downloads/list?can=1 .*/<project>-(\d\S+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

Launchpad

version=4
opts=pgpsigurlmangle=s/$/.asc/ \
  https://launchpad.net/<project>/ \ 
  https://launchpad.net/.*download/<project>-([.\d]+)\.tar\.xz

PyPI

The PyPI site is hard to work with directly as it includes md5sums for the downloads in the URLs which makes for truly horrible regexps. The pypi.debian.net redirector makes it much easier to work with PyPI:

version=4
opts=pgpsigurlmangle=s/$/.asc/ \
https://pypi.debian.net/<project>/<project>-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

There's also (a bit more advanced) autogenerated watch file that you can download from http://pypi.debian.net/<project>/watch

If you do want to work directly with PyPI, then make sure you use the new Simple API:

https://pypi.python.org/simple/<project>/ 

Note that direct use of the PyPI listing pages for packages (such as https://pypi.python.org/pypi/<package>/) is discouraged by the PyPI administrators. Automated use of package listing pages is not supported and is liable to break; the Simple API is provided for precisely that reason.

SourceForge

  version=4
  # qa.debian.org runs a redirector which allows a simpler form of URL
  # for SourceForge based projects. The format below will automatically
  # be rewritten to use the redirector.
  http://sf.net/audacity/audacity-src-(\d\S+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

See also: https://salsa.debian.org/qa/qa/tree/master/wml/watch/

Autogenerated

A scripted debian/watch generator is available

Further information

Other distros

Fedora also has an upstream release monitoring project (more info).


CategoryDebianDirectory | CategoryPackaging