Some node modules are rather small and sometime are dependency of only one package. To avoid having many node modules, qa.debian.org and uscan provides hooks that can be used to simplify maintenance.
Contents
When is it recommended to embed a node module?
This chapter is a guidelines, no a set of rules. If you have any doubt, write a mail to Debian JS Team mailing list.
Some acceptable reasons to embed a node module:
- module is very small (fewer than 100 lines of code)
the usable software is split in more than one module. Examples include babel, vue etc. Upstream maintains many modules in a single repo, usually each module under packages directory. Use git tags/releases page as source tarball.
- module uses some very little dependencies
module uses a useless dependency (check npmjs website) For example, if the module is useful only on Microsoft Windows but creating a patch to remove the dependency is complicated.
- module does not have any build commands in package.json under scripts.
If you have any doubt, write a mail to Debian JS Team mailing list
When is it recommended to use component version in package version?
Adding component version in package version allows uscan to follow upstream changes? This is a security point and must be examined using this point of view. Guidelines to add component version in package version:
components used only during build => not used in version
very little component => not used in version
if main upstream versions uses a package.lock file => no need to embed component versions in package version
other case => embedded
Do I have to export embedded modules or not?
As explained after, you can export components or not using Provides: field in debian/control and install it in node path (see below). Some guidelines to choose:
if component version is locked on a too old version, don't export it
if one of your component is used by many other node modules (see npmjs.com), you must not export it:
- if your package needs dependencies that are not needed by this component, build it separately
if component is a run dependency and is significant (not just few JS code), export it to help security team to follow CVEs
if component is a test dependency or a build dependency (that does not inject code) and/or if your main component has more dependencies than this component, don't export it, neither install it
By default, pkg-js-tools will install embedded modules in path/to/main/module/node_modules. You can override this if you want to make the embedded module directly accessible for other packages (required if you "provides" it).
Building a multiple npmjs source package
In this example, we are building node-test package (named test in npmjs.com) including ta and tb npmjs modules.
Step 0: verify that embedded modules don't exist already
To avoid package conflicts and embedded copies:
- check unstable packages
check NEW queue
check pending embedded modules
To check if package needs to be embedded run this commands:
apt search <package-name> to ensure package is available in apt repo
apt-file find <package-name> to show if its embedded in another package
aptitude show <package-name> to show if package is a virtual package in another package
if response from each command not positive then we can embed
If component doesn't exist there, insert its (Debian) name in pending embedded modules. Please clean also this list when package has been accepted in unstable.
Step 1: Add/del components
pkg-js-tools way
Simply launch:
$ add-node-component -i ta tb
or
$ add-node-component -c ta tb
or
$ add-node-component -gi ta tb
-g options use "group" instead of "ignore" in debian/watch, also optionally add -c for checksum option to automatically add up component version number. See pkg-js-tools doc for more.
Then update debian/changelog version. Example:
dch -v 3.1.4+~3.0.2+~3.1.3-1 'Import upstream version 3.1.4+~3.0.2+~3.1.3-1'
To delete a component, launch del-node-component with the same options than add-node-component.
Manual way
Update debian/gbp.conf and debian/watch
git-buildpackage will build embedded sources automatically if both debian/gbp.conf and debian/watch are well configured:
gbp.conf: add component = [ 'ta', 'tb' ] in [DEFAULT] section:
[DEFAULT] pristine-tar=True component=[ 'ta', 'tb' ] [import-orig] filter=[ '.gitignore', '.travis.yml', '.git*' ]
- debian/watch: this is the more complex step. You have to one entry for each component. Full example
version=4 # Prefer source repositories, even if this example uses npm registry opts="searchmode=plain,pgpmode=none" \ https://registry.npmjs.org/test \ https://registry.npmjs.org/test/-/test-(\d[\d\.]*)@ARCHIVE_EXT@ group opts="searchmode=plain,pgpmode=none,component=ta" \ https://registry.npmjs.org/ta \ https://registry.npmjs.org/ta/-/ta-(\d[\d\.]*)@ARCHIVE_EXT@ group opts="searchmode=plain,pgpmode=none,component=tb" \ https://registry.npmjs.org/tb \ https://registry.npmjs.org/tb/-/tb-(\d[\d\.]*)@ARCHIVE_EXT@ group
See uscan manpage for more. Quick explanations:
- If you don't want to track versions of components, then use "debian" for main tar and "ignore" for components instead of "group" at the end.
If you don't want lengthy version e.g 3.1.4+~3.0.2+~3.1.3-1 use checksum option for component tar and group for main tar at the end, to add up component and main package version number e.g 9.2.9-1
NB: if npmjs name contains "_", don't use "_" character in component name in opts options, but keep npmjs name in url
See node-rollup-plugin-sourcemaps commit as an example.
Download sources
Launch uscan, then gbp import-orig --pristine-tar ../<main>.orig.tar.xz will upload all components even if they are new. The source directory contains now upstream source and one directory per sub module named by their component name:
$ ls -l debian/ ta/ tb/ index.js package.json ...
Then update debian/changelog version. Example:
dch -v 3.1.4+~3.0.2+~3.1.3-1 'Import upstream version 3.1.4+~3.0.2+~3.1.3-1'
Example shell script (manual way)
This is an example script used to update gulp. Usage: add-components <component name>
if ! grep $1 debian/gbp.conf; then echo "gbp.conf should be updated manually" exit 1 fi echo "Adding module to watch file..." WATCH_TEMPLATE=" \ opts=\"searchmode=plain,pgpmode=none,component=$1\" \\ https://registry.npmjs.org/$1 \\ https://registry.npmjs.org/$1/-/$1-(1.[\d\.]*)@ARCHIVE_EXT@ ignore" echo "$WATCH_TEMPLATE" >>debian/watch echo "Adding module to install file..." echo "$1 usr/lib/nodejs/gulp/node_modules/" >>debian/install echo "Downloading source tarballs..." uscan -dd echo "Creating source package..." dpkg-source -b . echo "Removing patches applied..." quilt pop -a && rm -rf .pc echo "Committing modified files..." git commit debian/gbp.conf debian/install debian/watch -m "Add $1" echo "Importing dsc to git..." gbp import-dsc --pristine-tar ../node-gulp_4.0.2-1.dsc echo "Removing debian tag..." git tag -d debian/4.0.2-1
Step 2: configure other debian/* files
debian/control
To avoid multiple embedding, you must share sub components with other. This is done by adding a "Provides:" field in debian/control: Note what for smooth upgrade improvement main package is called node-debbundle-foo (allow simpler merge/split if needed)
...
Package: node-debbundle-foo
Depends: ${misc:Depends},
nodejs,
pkg-js-tools (>= 0.8.10)
Provides:
node-foo (=1.0.0)
node-ta (= 1.2.3),
node-tb (= 0.3.4)
...NB: you must set the exact version provided by your components.
debian/copyright
Of course, update debian/copyright with components copyrights using this fields (Files, Copyright, License and optionally the License Full description text).
debian/rules
Just to add --with nodejs here, it will add needed symlinks. If a component needs another one during build, use debian/nodejs/component_links (see pkg-js-tools doc)
%:
dh $@ --with nodejs
debian/install
pkg-js-tools will automatically install all components in /usr/share/nodejs/<foo>/node_modules (or /usr/lib/<arch>/nodejs/<foo>/node_modules). If this is not the wanted place, use debian/nodejs/<component>/install as described in pkg-js-tools documentation. So remove debian/install unless you're building a multiple binaries package (see pkg-js-tools doc).
Other files
Check if it is useful or not to export component README, changelog,...
autopkgtest
Nothing special here except that you can build a test for each component if main component test seems not enough. It is important to write tests that really require every embedded component, so autopkgtest will prove that install is good.
pkg-js-tools can do it for you: you just have to write a debian/tests/pkg-js/test file with the test to launch (see package.json: "scripts":"test"). The test is automatically launched during build and during autopkgtest. You need to add "Testsuite: autopkgtest-pkg-nodejs" in your debian/control.
The end
When build is OK, of course verify using debc that install looks good (package are really usable.
NB: if you need to build something in a component, pkg-js-tools (>= 0.9.4) provides an easy way to do it: write your build commands in debian/nodejs/<component>/build instead of writing the build in an override_dh_auto_build. You can also do the same (debian/nodejs/build) for the main package.
Using git subtree to track multiple upstream (optional)
git subtree could be used to track multiple upstream, and allow easier cherry picking. Let suppose that you want to package node-acorn with subpackage node-acorn-node.
you should first add the remote acorn and remote acorn-node
git remote add upstream-acorn https://github.com/acornjs/acorn git remote add upstream-acorn-bigint https://github.com/acornjs/acorn-bigint git remote add upstream-acorn-import-meta https://github.com/acornjs/acorn-import-meta/ git remote add upstream-acorn-node https://github.com/browserify/acorn-node
Fetch all
git fetch --all
If not done create a fake upstream branch for main package
git checkout gitidofupstreamcorn git checkout -b upstreamdebian