- Creating .deb packages from Node.js modules
- Rationale for packaging
- Comparison with Android apks/Google play store
- Debian release cycle
- Debian Policy
- Library package naming convention
- Missing tags in upstream repo
- Package Directory
- Debian source package
- Building the .deb
Creating .deb packages from Node.js modules
By Pirate Praveen (Originally delivered as an online workshop)
With additional input from second online workshop
All of the participants have been asked to setup a debian sid environment and steps for setting up an lxc or docker container for sid was provided.
Pirate Praveen: can some one explain what do we mean by a distribution?
Raju / राजु: A distribution is a collection of (Application + System) Softwares and the core OS, which ensures all of these run together well.
Pirate Praveen: cool. What is a rolling distribution?
Nidarsh | നിദര്ശ് (Telegram): that gets constantly updated
rovonovo_zoro (IRC): frequent updates to software = rolling distribution
Pirate Praveen: yes, braodly there are two types of distros based on how frequent you get new softwares. arch is a rolling distribution. debian has stable releases.
Raju / राजु: A rolling distribution is an operating system which only recieves continuos updates. It never has a new release. Any of the rolling release os can be updated to a latest release of the OS.
Rationale for packaging
Pirate Praveen: I think all of you would know the advantages of packaging over Windows style exe installers. can someone explain it?
refpga: A Standard way of installing defined by the os, instead of leaving it to the wisdom of the programmer where to place the files?
Shanavas: Security flaws are often caught and patched by packager
Pirate Praveen: one main advantage I see is by separating the installer from the software enhances security. With exe style packages, you have to run untrusted code. with packages like rpm or deb, the logic or the package manager is always trusted (though it still allows some kind of programming, for most packages it just provide data)
Pirate Praveen: another thing is the easiness of finding software and managing all of it together. for one or two softwares, manual installation shoud be fine, but when you have to work with more software, keeping them uptodate is manual with exe style packages
AP: One bigger advantage of linux-style packaging (de-coupling package and installer) is dependency management.
Pirate Praveen: yes, size of individual packages will be smaller here and fixing bugs is easier.
Pirate Praveen: if it is one big package with all dependencies bundled, it means an app needs to be updated for bug fix in any dependency and one fix in a library needs updating all applications that use that library.
Comparison with Android apks/Google play store
Pirate Praveen: an app developer usually creates the apk and upload it to play store. Why can't the same happen for desktop/web apps as well?
refpga: Because there the developer writes apps for one OS only. and one packaging system.
Pirate Praveen: yup, great point, one apk will work on all android roms. you only need to take care of the android version itself. for gnu/linux, if the app developers itself were to provide native pckages, they will have to do it for debian, fedora, gentoo, arch ...
though if they do it for debian, all derivatives usually get it too so each distribution has a community of packagers who takes software from their original developers (what we call upstream) and create packages though this session we are hoping some of you would join the team of debian packaging community and make more software available to users via official repositories
with this challenge, upstream usually resort to providing binary packages or docker images both of them are convenient for users and developers, but has this problem of updates. with docker you need to learn and extra technology and add more layers. hope the rationale for native packages are clear
Debian release cycle
Pirate Praveen: you normally get new versions of software only when new version of the distribution is released. Though there are options like backports where newer versions of some popular software are made available, but that is a small subset of the whole distribution. During the life of the release, bug fixes are made available. So current stable release of debian is stretch and its latest update is 9.1, which includes fixes to bugs found in 9.0 but no new versions of any software is added to 9.1. New versions of the software will be included in 10.0 or next stable release buster, but new versions of a limited number software is made available via stretch-backports.
For developing the next stable release, the work is already ongoing in two different branches called testing and unstable. All new software are first uploaded to debian unstable branch which is a rolling distribution. Since we are going to add new packages, we are setting up a debian unstable environment for our work. sid is the codename for unstable branch. debian already have 51000+ packages in stretch
rovonovo_zoro (IRC): that in main? or all the 3 combined? (main, nonfree, contrib)
Pirate Praveen: I think just main. But still there are many software not available in the archive. Installing those software require adding unofficial repositories or sometimes manual installation. So to make this easy for users we are adding important software to official repo. In case of android, developers itself release the apk and add to play store but since there are many different packaging formats and release cycles, developers of software cannot add it to all the distributions, so each distribution has a team of package maintainers who creates packages.
Pirate Praveen: everyone familiar with nodejs?
gsitlani: yes. I'm familiar with front end frameworks
Pirate Praveen: yes.
Pirate Praveen: we will take a node.js module to learn packaging as that is very easy to package. Once you understand the concepts, you can package software written in any language. There is a tool called npm2deb that converts a nodejs module to a deb file. Shanavas is a core contributor to npm2deb. It tries to automate the process of deb file creation but it is not perfect yet, so we need to fix some things to make a good deb package as per debian policy.
Pirate Praveen: One of the reasons for high quality of debian is its strict adherence to its policy. Another is the concept of release only when it is ready, unlike some other distros which release every 6 months or 9 months.
So all of you who have the sid environment ready can install npm2deb make sure your npm2deb version is 0.2.7-2. We will take pretty-hrtime as an example module for packaging. You can follow the steps given here and we can discuss and problems you face
Pirate Praveen: this library is a dependency of gulp, which is a task runner like make, cmake, ant, rake etc Similar to grunt, but supposed to be better than grunt, it has a different style and uses streams.
I was foreced to package gulp because one of the dependencies of diaspora used it (handlebars templating engine).
You should setup some variables for npm2deb to use. See setting up environment variables for packaging.
npm2deb create pretty-hrtime will create pretty-hrtime/node-pretty-hrtime_1.0.3-1_all.deb and many other files.
Note 1: Refer to npm2deb documentation for all the options of npm2deb.
Note 2: always look for the keyword error in the output (that is the line you want to focus) whenever you run any command.
You may get this error dpkg-checkbuilddeps: error: Unmet build dependencies: dh-buildinfo. It means you have to install dh-buildinfo package.
There are two types of dependencies, those required during build and those required to run. If you are familiar with java, then javac/jdk will be a build dependency and java/jre will be a runtime dependency. Here dh-buildinfo just collects details about the build environment and we need that installed to successfully build the package.
You'll see these messages at the end of npm2deb create pretty-hrtime command.
Remember, your new source directory is pretty-hrtime/node-pretty-hrtime-1.0.3 This is not a crystal ball, so please take a look at auto-generated files. You may want fix first these issues: pretty-hrtime/node-pretty-hrtime-1.0.3/debian/control:Description: FIX_ME write the Debian package description pretty-hrtime/node-pretty-hrtime_itp.mail:Subject: ITP: node-pretty-hrtime -- FIX_ME write the Debian package description pretty-hrtime/node-pretty-hrtime_itp.mail: Description : FIX_ME write the Debian package description pretty-hrtime/node-pretty-hrtime_itp.mail: FIX_ME: This ITP report is not ready for submission, until you are pretty-hrtime/node-pretty-hrtime_itp.mail:FIX_ME: Explain why this package is suitable for adding to Debian. Is pretty-hrtime/node-pretty-hrtime_itp.mail:FIX_ME: Explain how you intend to consistently maintain this package
Pirate Praveen: now you need to fix them, but before that, I will explain the structure of files and directories.
We call the original project that develops the software as upstream. pretty-hrtime project is upstream here for us. For every package in debian, there will be a corresponding upstream project (sometimes debian itself is upstream, for example for apt). For all nodejs modules we can find details of upstream from npmjs.com. npm is node package manager, which helps install node modules.
rovonovo_zoro (IRC): npm:node.js, pip:python
Every project will usually release their source as a tar ball (.tar.gz or .tar.bz2 usually). For projects hosted in github.com, gitlab.com etc you can go to their tags page and download the tarballs (/tags or /releases)
Basically you'll get a tar.gz file from the projet website.
Library package naming convention
In debian, we use a convention in naming the upstrem tarballs ie, package-name_version.orig.tar.gz
Since multiple languages may use the same name for libraries, we add name of the language to the debian package name. So if we have an http library, the debian package names will be node-http, python-http, ruby-http, libhttp (for C library), libhttp-perl, libhttp-java etc This is to make sure we can easily identify libraries in a particular language and also make sure we have unique package names. So node-pretty-hrtime_1.0.3.orig.tar.gz is the exact same source code released by the upstream project. You can extract this file and compare it with the tarball you download from github, it will be the same.
Some times we may have to remove some files, but most of the time, it will be the exact same file as released by upstream.
Missing tags in upstream repo
Shanavas: it will be the same, if its tagged properly
Pirate Praveen: yes many nodejs developers are too lazy to tag releases or add good descriptions. If they don't add tags for releases, we have to manually find the release from their commit logs and download the commit tarball.
Shanavas: Better to mention fake upstream ?
Pirate Praveen: fakeupstream may not have tests and can have pre built binaries so manual download is preferred. For simple packages fakeupstream may be enough.
If they forget to add tag for their releases another option to get tarball is from npmjs.com itself, what is called as dist tarballs. It may be enough for simple cases where the npmjs.com dist tarball has tests and it does not include any minified/browserified/transpiled files. Read more about downloading npm dist tarballs
Otherwise we prefer a manual download via commit log.
Pirate Praveen: okay next we are going to see about the folder node-pretty-hrtime-1.0.3. It is a combination of the orig.tar.gz extracted and debian folder created by npm2deb.
$ ls debian index.js LICENSE package.json README.md test
In case of rubygems, debian folder will be created by another tool called gem2deb. For a lot of other tarballs, there is a tool called dh-make which will give you a generic debian folder if you give it any kind of tarball. But we prefer a specialized tool like npm2deb over dh-make if it is available.
look inside node-pretty-hrtime-1.0.3 and see if you are still with me.
node-pretty-hrtime_1.0.3-1.debian.tar.xz is just debian folder compressed since xz is more efficient algorithm debian prefers .tar.xz. Earlier it also used to be tar.gz
Debian source package
Now in node-pretty-hrtime-1.0.3 we have the required files to create a debian source package.
if you run
dpkg-source -b . dpkg-source: info: using source format '3.0 (quilt)' dpkg-source: info: building node-pretty-hrtime using existing ./node-pretty-hrtime_1.0.3.orig.tar.gz dpkg-source: info: building node-pretty-hrtime in node-pretty-hrtime_1.0.3-1.debian.tar.xz dpkg-source: info: building node-pretty-hrtime in node-pretty-hrtime_1.0.3-1.dsc
you'll get node-pretty-hrtime_1.0.3-1.dsc in the parent directory you can open it and see. It is a text file that has checksums for the tar files we need.
if you looked in the output, dpkg-source used .orig.tar.gz, created debian.tar.xz and .dsc files.
dsc file summarizes information in debian folder (just picks some important fields) and adds checksum for the tar files which will use to build the .deb file.
checksum can be used to ensure integrity ie, someone has not changed in after the checksum was created. It can also make sure the download was not corrupted.
Building the .deb
now if you run
you'll get a .deb file and .changes file
.deb you already know
.changes is similar to .dsc, but has addiitonal checksums for .dsc and .deb files too
We sign the changes file with our gpg key, so we know the checksums are not modified.
gpg uses asymmetric key cryptography. So I keep the private key with me and share the public key. Anyone who has the public key can verify my signature, but only I can make the signature.
Now we need to tune our deb file to make it compliant with debian policy. Some clues were given by npm2deb itself at the end of npm2deb create. We can use a tool called lintian to find those and more issues about our package.
lintian is run against the changes file
now you need to fix the problems reported by lintian. You need to edit files inside debian directory. Main files are control, copyright, changelog. Sometimes you will need to edit more files too.
You can actually edit, dpkg-buildpackage and run lintian again to see if the problem is gone, but we will convert the package directory to a git repo for easy tracking.