Differences between revisions 32 and 33
Revision 32 as of 2013-05-01 02:10:49
Size: 4711
Editor: PaulWise
Comment: wiki pages aren't sentient
Revision 33 as of 2013-05-02 14:54:17
Size: 5448
Editor: ?StuartPrescott
Comment: Add gitorious to the list of examples
Deletions are marked like this. Additions are marked like this.
Line 93: Line 93:
=== Gitorious ===

Gitorious provides a list of tags in the tree visible at `http://gitorious.org/<project>/<repo>/trees/master`. The download link is `http://gitorious.org/<project>/<repo>/archive-tarball/<tag>` so a downloadurlmangle is required to get the download link and then a filenamemangle is required to make a sensible filename for the downloaded file, since uscan doesn't use the filename suggested by the server in the HTTP headers.

{{{
opts=downloadurlmangle=s/trees/archive-tarball/,filenamemangle=s/.*\/(.*)/<package>-$1.tar.gz/ \
  http://gitorious.org/<project>/<repo>/trees/(\d+\.\d+\.\d+)
}}}

As with other examples on this page, adapt the tag regexp to match the style of the tag the project is using.

debian/watch

The file named 'watch' in the debian directory is used to check for newer versions of upstream software is available and to download it if necessary. The download itself will be performed with the uscan program from the devscripts package. It takes the path to the debian directory that uses the watch file as an argument or searches the dirtories underneath the current working directory.

Basically a watch file will have this format:

  version=3
  http://somesite.com/dir/filenamewithversion.tar.gz

To allow the version to remain unspecified, it is expressed as a wildcard using regular expression in Perl format:

  version=3
  http://somesite.com/dir/filename_(.+).tar.gz

The uscan program will then execute a "dir" command and check all the files in that directory for the highest version number.

uscan also supports ftp protocol. Unfortunately, when working with http, HTML pages do not always contain directory listing with files. The complete (or relative) path of the .tar.gz file to be downloaded will appear as a hyperlink within the web page. We hence need two types of information, the path to the page announcing the file and a regular expression to grep for the right link:

  version=3
  http://somesite.com/path link_from_href-1.0.tar.gz

"http://somesite.com/path" is the site from where you are downloading the source and "link_from_href.1.0.tar.gz" is obtained from the HTML source code (from the "<a href=" tags).

If http://somesite.com/path HTML code has "<a href=foo/program-1.0.tar.gz>" inside it, for example, you will use

  http://somesite.com/dir foo/program-(.+)\.tar\.gz

Watch files with errors are generally wrong on this second part.

Generally you want a slightly more flexible regex for the tarball name so that if upstream switches tarball naming schemes or compression formats you are covered:

  http://somesite.com/dir foo/program-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

For FTP sites it changes a little, but it's basically the same thing (man uscan will help you).

But the nice part is to test it:

  uscan --no-download --verbose

If it's not working as expected, you can use debug to see what it's fetching and what it's (not) matching:

  uscan --no-download --verbose --debug

Common mistakes

  • Not escaping dots, which match any character. The solution is \. instead of . in the regex.

  • A file extension regex that is not flexible enough. The solution is \.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

  • Not anchoring the version group at the right place. The solution is to include something before (.*) like fooproj-(.*)\.tar\.gz

  • Not using digits in the version part of the regex. The solution is to use \d instead of .

  • Not being flexible enough in the path to the file. The solution is to use http://example.com/someproject/ .*/program-(.+)\.tar\.gz instead of http://example.com/someproject/ /path/to/program/downloads/program-(.+)\.tar\.gz

  • Not mangling upstream versions that are alphas, betas or release candidates to make them sort before the final release. The solution is to use uversionmangle like this opts=uversionmangle=s/(\d)[_\.\-\+]?((RC|rc|pre|dev|beta|alpha|b|a)\d*)$/$1~$2/

  • Not mangling Debian versions to remove dfsg. The solution is to use dversionmangle like this opts=dversionmangle=s/\+(debian|dfsg|ds|deb)\d*$//

Common sites

Files hosted on various big project hosting sites can be specified with the URLs below. SourceForge has a special "redirector" URL (see more details in 'man uscan'). This allows the upstream URLs to change without the need to adapt watch files of affected packages.

SourceForge

  # qa.debian.org runs a redirector which allows a simpler form of URL
  # for SourceForge based projects. The format below will automatically
  # be rewritten to use the redirector.
  http://sf.net/audacity/audacity-src-(.+)\.(?:zip|tgz|tbz2|txz|tar\.gz|tar\.bz2|tar\.xz)

See also: http://anonscm.debian.org/viewvc/qa/trunk/wml/watch/

GitHub

https://github.com/<user>/<project>/tags .*/v?(\d[\d\.]+)\.tar\.gz

The 2nd regexp matches most often used tags like '0.1.2' or 'v0.1.2'. It may need to be adpated if upstream uses other version tags.

Google Code

http://code.google.com/p/<project>/downloads/list?can=1 .*/<project>-(\d[\d.]*)\.(?:zip|tgz|tbz2|txz|tar\.gz|tar\.bz2|tar\.xz)

Gitorious

Gitorious provides a list of tags in the tree visible at http://gitorious.org/<project>/<repo>/trees/master. The download link is http://gitorious.org/<project>/<repo>/archive-tarball/<tag> so a downloadurlmangle is required to get the download link and then a filenamemangle is required to make a sensible filename for the downloaded file, since uscan doesn't use the filename suggested by the server in the HTTP headers.

opts=downloadurlmangle=s/trees/archive-tarball/,filenamemangle=s/.*\/(.*)/<package>-$1.tar.gz/ \
  http://gitorious.org/<project>/<repo>/trees/(\d+\.\d+\.\d+)

As with other examples on this page, adapt the tag regexp to match the style of the tag the project is using.


CategoryDebianDirectory