Differences between revisions 14 and 15
Revision 14 as of 2014-05-04 23:46:13
Size: 11016
Editor: BenFinney
Comment: Turn off line number display in code examples.
Revision 15 as of 2014-10-21 23:01:13
Size: 11038
Editor: BenFinney
Comment:
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
Sometimes this is for reasons such as changing format (Debian upstream source packages must be tarballs), or size (some upstream source tarballs contain files that are excessively large and not needed for Debian). Most often, though, the re-pack is needed because the upstream tarball contains files that do not meet [[DebianFreeSoftwareGuidelines|Debian's free software guidelines]] for the package source, and so cannot be part of Debian. Sometimes this is for reasons such as changing format (Debian upstream source packages must be tarballs), or size (some upstream source tarballs contain files that are excessively large and not needed for Debian). Most often, though, the re-pack is needed because the upstream tarball contains files that do not meet the [[DebianFreeSoftwareGuidelines|Debian free software guidelines]] for the package source, and so that source tarball cannot be part of Debian.

A common need for Debian packages is to re-pack the upstream source, to create a source package suitable for Debian.

Sometimes this is for reasons such as changing format (Debian upstream source packages must be tarballs), or size (some upstream source tarballs contain files that are excessively large and not needed for Debian). Most often, though, the re-pack is needed because the upstream tarball contains files that do not meet the Debian free software guidelines for the package source, and so that source tarball cannot be part of Debian.

Configuration for ‘uscan’

The packaging tool ‘uscan’, configured with debian/watch in a package, can run a specific command to transform the pristine upstream source to the Debian upstream source, usually with a different name to indicate it is modified. For example:

# debian/watch
# Debian uscan file for ‘libjs-zxcvbn’ package.
# Manpage: ‘uscan(1)’

# Compulsory line, this is a version 3 file.
version=3

# Current automated tarball release at GitHub project.
opts="dversionmangle=s/\+dfsg\.\d+$//,filenamemangle=s/\S+\/v?(\S+)\.tar\.gz/zxcvbn-$1\.tar\.gz/" \
    https://github.com/dropbox/zxcvbn/releases/ \
    /dropbox/zxcvbn/archive/v?(\S+)\.tar\.gz \
    debian debian/repack

Important in this example are:

  • The dversionmangle option, which transforms the declared Debian upstream version string (e.g. “1.0+dfsg.1”) into the corresponding real upstream version string (e.g. “1.0”) for matching against available upstream versions.

  • The third and fourth parameters, being the “version number” and “action” fields (of the “alternative version of the watch file syntax for HTTP URLs” from the ‘uscan(1)’ man page):
    • The “version number” field specifies the upstream version string. debian is a token specifying that ‘uscan’ should use the Debian version from the debian/changelog (e.g. “1.0+dfsg.1-3”) to determine the upstream version string (e.g. “1.0+dfsg.1”).

    • The “action” field is the name of a command to use for transforming the downloaded upstream source into the Debian upstream source tarball. debian/repack is a relative path to a program written for this package specifically.

This is a frequent pattern in Debian packages that need to transform their upstream source immediately after download.

‘uscan’ will, when downloading the tarball for a package configured as above, invoke the specified “action” command with options specifying the upstream version and the downloaded filename. For example, downloading the upstream version “1.0” of the package would result in ‘uscan’ running the command debian/repack --upstream-version 1.0 libjs-zxcvbn_1.0.orig.tar.gz.

For a package needing a re-packed upstream source tarball, the action command (e.g. the debian/repack program) needs to produce a tarball named with the Debian upstream version. For example, for version “1.0” of the above package, the action needs to unpack the tarball named libjs-zxcvbn_1.0.orig.tar.gz and re-pack it to produce a tarball named libjs-zxcvbn_1.0+dfsg.1.orig.tar.gz.

Explaining re-pack of upstream source in ‘debian/copyright’

Debian Policy §12.5 says “… the copyright file must say where the upstream sources (if any) were obtained…”. This is commonly interpreted to imply that, if you re-pack the upstream source, the reason should be explained in the debian/copyright file. The machine-readable copyright file format specifies the free-form text of the “Source” field should be used for this purpose:

Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: zxcvbn
Upstream-Contact: Dan Wheeler <dan@dropbox.com>
Source: https://github.com/dropbox/zxcvbn/
    The pristine upstream source contains obfuscated files, that are
    thereby not appropriate for use as source files. This violates DFSG §2.
    .
    The ‘debian/repack’ program removes those files to create the Debian
    upstream source tarball.

Robust ‘debian/repack’ program

Here is a robust debian/repack example for the above package. It removes non-DFSG-free files from the pristine upstream source tarball, and re-packs the remaining files with the correct filename for the Debian upstream source tarball.

#! /bin/bash
#
# debian/repack
# Part of the Debian package ‘libjs-zxcvbn’.
#
# Copyright © 2013–2014 Ben Finney <ben+debian@benfinney.id.au>
# This is free software; see the end of this file for license terms.

# Convert the pristine upstream source to the Debian upstream source.
#
# This program is designed for use with the ‘uscan(1)’ tool, as the
# “action” parameter for the ‘debian/watch’ configuration file.

set -o errexit
set -o errtrace
set -o pipefail
set -o nounset

program_dir="$(dirname "$(realpath --strip "$0")")"
source "${program_dir}"/source_package_build.bash

function usage() {
    local progname=$(basename $0)
    printf "$progname --upstream-version VERSION FILENAME\n"
}

if [ $# -ne 3 ] ; then
    usage
    exit 1
fi

upstream_version="$2"
downloaded_file="$3"

target_filename="${upstream_tarball_basename}.tar.gz"
target_working_file="${working_dir}/${target_filename}"
target_file="$(dirname "${downloaded_file}")/${target_filename}"

repack_dir="${working_dir}/${upstream_dirname}"

printf "Unpacking pristine upstream source ‘${downloaded_file}’:\n"

extract_tarball_to_working_dir "${downloaded_file}"

upstream_source_dirname=$(ls -1 "${working_dir}")
upstream_source_dir="${working_dir}/${upstream_source_dirname}"

printf "Repackaging upstream source from ‘${upstream_source_dir}’ to ‘${repack_dir}’:\n"

mv "${upstream_source_dir}" "${repack_dir}"

printf "Removing non-DFSG-free files:\n"

nonfree_fileglobs=(
        # Third-party compiler in non-source form, and related files.
        NOTICE.txt
        tools/closure.jar
        tools/COPYING.txt

        # Auto-generated files are non-source form.
        adjacency_graphs.js
        frequency_lists.js

        # Compiled files are non-source form.
        zxcvbn.js
        zxcvbn-async.js

        # Cached downloaded third-party data files.
        data/tv_and_movie_freqlist*.html
        )

for fileglob in "${nonfree_fileglobs[@]}" ; do
    rm -v "${repack_dir}"/$fileglob
done

printf "Rebuilding DFSG-free upstream source tarball:\n"

archive_working_dirname_to_tarball "${upstream_dirname}" "${target_working_file}"

printf "Moving completed upstream tarball to ‘${target_file}’:\n"

rm -v "${downloaded_file}"
mv "${target_working_file}" "${target_file}"

printf "Done.\n"


# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# “Software”), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
# 
# The above copyright notice and this permission notice shall be included
# in all copies or substantial portions of the Software.
# 
# The Software is provided “as is”, without warranty of any kind,
# express or implied, including but not limited to the warranties of
# merchantability, fitness for a particular purpose and noninfringement.
# In no event shall the authors or copyright holders be liable for any
# claim, damages or other liability, whether in an action of contract,
# tort or otherwise, arising from, out of or in connection with the
# Software or the use or other dealings in the Software.


# Local variables:
# coding: utf-8
# mode: sh
# indent-tabs-mode: nil
# End:
# vim: fileencoding=utf-8 filetype=bash :

# debian/source_package_build.bash
# Part of the Debian package ‘libjs-zxcvbn’.
#
# Copyright © 2013–2014 Ben Finney <ben+debian@benfinney.id.au>
# This is free software; see the end of this file for license terms.

# Common code for building Debian upstream source package.

working_dir="$(mktemp -d -t)"

exit_sigspecs="ERR EXIT SIGTERM SIGHUP SIGINT SIGQUIT"

function cleanup_exit() {
    exit_status=$?
    trap - $exit_sigspecs

    rm -rf "${working_dir}"
    printf "Cleaned up working directory ‘${working_dir}’\n"

    exit $exit_status
}
trap cleanup_exit $exit_sigspecs

package_name=$(dpkg-parsechangelog | sed -n -e 's/^Source: //p')
release_version=$(dpkg-parsechangelog | sed -n -e 's/^Version: //p')
upstream_version=$(printf "${release_version}" \
        | sed -e 's/^[[:digit:]]\+://' -e 's/[-][^-]\+$//')
upstream_dirname="${package_name}-${upstream_version}.orig"
upstream_tarball_basename="${package_name}_${upstream_version}.orig"

function extract_tarball_to_working_dir() {
    # Extract the specified tarball to the program's working directory.
    local tarball="$1"
    tar -xzf "${tarball}" --directory "${working_dir}"
}

function archive_working_dirname_to_tarball() {
    # Archive the specified directory, relative to the working directory,
    # to a new tarball of the specified name.
    local source_dirname="$1"
    local tarball="$2"
    GZIP="--best" tar --directory "${working_dir}" -czf "${tarball}" "${source_dirname}"
}


# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# “Software”), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
# 
# The above copyright notice and this permission notice shall be included
# in all copies or substantial portions of the Software.
# 
# The Software is provided “as is”, without warranty of any kind,
# express or implied, including but not limited to the warranties of
# merchantability, fitness for a particular purpose and noninfringement.
# In no event shall the authors or copyright holders be liable for any
# claim, damages or other liability, whether in an action of contract,
# tort or otherwise, arising from, out of or in connection with the
# Software or the use or other dealings in the Software.


# Local variables:
# coding: utf-8
# mode: sh
# End:
# vim: fileencoding=utf-8 filetype=bash :