Differences between revisions 8 and 9
Revision 8 as of 2017-08-03 09:22:55
Size: 13736
Editor: Praveen A
Comment:
Revision 9 as of 2017-08-30 13:15:12
Size: 15279
Editor: Praveen A
Comment: add packaging rationale
Deletions are marked like this. Additions are marked like this.
Line 6: Line 6:
With additional input from [[https://matrix.to/#/!WaYPjxOMzwGsYOUWvX:matrix.org/$1504017752262836JCUkN:matrix.org|second online workshop]]
Line 23: Line 25:

== Rationale for packaging ==
Pirate Praveen: I think all of you would know the advantages of packaging over Windows style exe installers. can someone explain it?

refpga: A Standard way of installing defined by the os, instead of leaving it to the wisdom of the programmer where to place the files?

Shanavas: Security flaws are often caught and patched by packager

Pirate Praveen: one main advantage I see is by separating the installer from the software enhances security. With exe style packages, you have to run untrusted code. with packages like rpm or deb, the logic or the package manager is always trusted (though it still allows some kind of programming, for most packages it just provide data)

Pirate Praveen: another thing is the easiness of finding software and managing all of it together. for one or two softwares, manual installation shoud be fine, but when you have to work with more software, keeping them uptodate is manual with exe style packages

AP: One bigger advantage of linux-style packaging (de-coupling package and installer) is dependency management.

Pirate Praveen: yes, size of individual packages will be smaller here and fixing bugs is easier.


== Comparison with Android apks/Google play store ==

Pirate Praveen: an app developer usually creates the apk and upload it to play store. Why can't the same happen for desktop/web apps as well?

Creating .deb packages from Node.js modules

By Pirate Praveen (Originally delivered as an online workshop)

With additional input from second online workshop

All of the participants have been asked to setup a debian sid environment and steps for setting up an lxc or docker container for sid was provided.

Distribution

Pirate Praveen: can some one explain what do we mean by a distribution?

Raju / राजु: A distribution is a collection of (Application + System) Softwares and the core OS, which ensures all of these run together well.

Pirate Praveen: cool. What is a rolling distribution?

Nidarsh | നിദര്‍ശ് (Telegram): that gets constantly updated

rovonovo_zoro (IRC): frequent updates to software = rolling distribution

Pirate Praveen: yes, braodly there are two types of distros based on how frequent you get new softwares. arch is a rolling distribution. debian has stable releases.

Raju / राजु: A rolling distribution is an operating system which only recieves continuos updates. It never has a new release. Any of the rolling release os can be updated to a latest release of the OS.

Rationale for packaging

Pirate Praveen: I think all of you would know the advantages of packaging over Windows style exe installers. can someone explain it?

refpga: A Standard way of installing defined by the os, instead of leaving it to the wisdom of the programmer where to place the files?

Shanavas: Security flaws are often caught and patched by packager

Pirate Praveen: one main advantage I see is by separating the installer from the software enhances security. With exe style packages, you have to run untrusted code. with packages like rpm or deb, the logic or the package manager is always trusted (though it still allows some kind of programming, for most packages it just provide data)

Pirate Praveen: another thing is the easiness of finding software and managing all of it together. for one or two softwares, manual installation shoud be fine, but when you have to work with more software, keeping them uptodate is manual with exe style packages

AP: One bigger advantage of linux-style packaging (de-coupling package and installer) is dependency management.

Pirate Praveen: yes, size of individual packages will be smaller here and fixing bugs is easier.

Comparison with Android apks/Google play store

Pirate Praveen: an app developer usually creates the apk and upload it to play store. Why can't the same happen for desktop/web apps as well?

Debian release cycle

Pirate Praveen: you normally get new versions of software only when new version of the distribution is released. Though there are options like backports where newer versions of some popular software are made available, but that is a small subset of the whole distribution. During the life of the release, bug fixes are made available. So current stable release of debian is stretch and its latest update is 9.1, which includes fixes to bugs found in 9.0 but no new versions of any software is added to 9.1. New versions of the software will be included in 10.0 or next stable release buster, but new versions of a limited number software is made available via stretch-backports.

For developing the next stable release, the work is already ongoing in two different branches called testing and unstable. All new software are first uploaded to debian unstable branch which is a rolling distribution. Since we are going to add new packages, we are setting up a debian unstable environment for our work. sid is the codename for unstable branch. debian already have 51000+ packages in stretch

rovonovo_zoro (IRC): that in main? or all the 3 combined? (main, nonfree, contrib)

Packaging

Pirate Praveen: I think just main. But still there are many software not available in the archive. Installing those software require adding unofficial repositories or sometimes manual installation. So to make this easy for users we are adding important software to official repo. In case of android, developers itself release the apk and add to play store but since there are many different packaging formats and release cycles, developers of software cannot add it to all the distributions, so each distribution has a team of package maintainers who creates packages.

I maintain gitlab and diaspora packages in debian, @balasankarc bhe isaagar Shanavas rark and many others are also part of this effort. I have been working mainly on packaging ruby libraries for gitlab and diaspora and any web based application needs javascript, so pulled into packaging nodejs modules as well.

Node.js

Pirate Praveen: everyone familiar with nodejs?

gsitlani: no

Pirate Praveen: you know javascript?

gsitlani: yes. I'm familiar with front end frameworks

Pirate Praveen: usually web applications use different languages for front end and backend. front end is always javascript, where as backend is perl, php, ruby, jsp, asp, python etc some people figured a way to write backend also in javascript and it is called nodejs and these days many desktop apps are also written in javascript - riot desktop, slack, atom, microsoft visual code, and growing.

rovonovo_zoro (IRC): so basically javascript now runs on servers in the form of node.js?

Pirate Praveen: yes.

npm2deb

Pirate Praveen: we will take a node.js module to learn packaging as that is very easy to package. Once you understand the concepts, you can package software written in any language. There is a tool called npm2deb that converts a nodejs module to a deb file. Shanavas is a core contributor to npm2deb. It tries to automate the process of deb file creation but it is not perfect yet, so we need to fix some things to make a good deb package as per debian policy.

Debian Policy

Pirate Praveen: One of the reasons for high quality of debian is its strict adherence to its policy. Another is the concept of release only when it is ready, unlike some other distros which release every 6 months or 9 months.

So all of you who have the sid environment ready can install npm2deb make sure your npm2deb version is 0.2.7-2. We will take pretty-hrtime as an example module for packaging. You can follow the steps given here and we can discuss and problems you face

pretty-hrtime

Pirate Praveen: this library is a dependency of gulp, which is a task runner like make, cmake, ant, rake etc Similar to grunt, but supposed to be better than grunt, it has a different style and uses streams.

I was foreced to package gulp because one of the dependencies of diaspora used it :) (handlebars templating engine).

You should setup some variables for npm2deb to use. See ?setting up environment variables for packaging.

npm2deb create pretty-hrtime will create pretty-hrtime/node-pretty-hrtime_1.0.3-1_all.deb and many other files.

Note 1: Refer to npm2deb documentation for all the options of npm2deb.

Note 2: always look for the keyword error in the output (that is the line you want to focus) whenever you run any command.

You may get this error dpkg-checkbuilddeps: error: Unmet build dependencies: dh-buildinfo. It means you have to install dh-buildinfo package.

There are two types of dependencies, those required during build and those required to run. If you are familiar with java, then javac/jdk will be a build dependency and java/jre will be a runtime dependency. Here dh-buildinfo just collects details about the build environment and we need that installed to successfully build the package.

You'll see these messages at the end of npm2deb create pretty-hrtime command.

Remember, your new source directory is pretty-hrtime/node-pretty-hrtime-1.0.3

This is not a crystal ball, so please take a look at auto-generated files.

You may want fix first these issues:

pretty-hrtime/node-pretty-hrtime-1.0.3/debian/control:Description: FIX_ME write the Debian package description
pretty-hrtime/node-pretty-hrtime_itp.mail:Subject: ITP: node-pretty-hrtime -- FIX_ME write the Debian package description
pretty-hrtime/node-pretty-hrtime_itp.mail:  Description     : FIX_ME write the Debian package description
pretty-hrtime/node-pretty-hrtime_itp.mail: FIX_ME: This ITP report is not ready for submission, until you are
pretty-hrtime/node-pretty-hrtime_itp.mail:FIX_ME: Explain why this package is suitable for adding to Debian. Is
pretty-hrtime/node-pretty-hrtime_itp.mail:FIX_ME: Explain how you intend to consistently maintain this package

Pirate Praveen: now you need to fix them, but before that, I will explain the structure of files and directories.

Upstream

We call the original project that develops the software as upstream. pretty-hrtime project is upstream here for us. For every package in debian, there will be a corresponding upstream project (sometimes debian itself is upstream, for example for apt). For all nodejs modules we can find details of upstream from npmjs.com. npm is node package manager, which helps install node modules.

rovonovo_zoro (IRC): npm:node.js, pip:python

Pirate Praveen: http://npmjs.com/pretty-hrtime shows details of our node module. On the right side, you'll see a link to its homepage https://github.com/robrich/pretty-hrtime

Every project will usually release their source as a tar ball (.tar.gz or .tar.bz2 usually). For projects hosted in github.com, gitlab.com etc you can go to their tags page and download the tarballs (/tags or /releases)

like https://github.com/robrich/pretty-hrtime/releases

Basically you'll get a tar.gz file from the projet website.

Library package naming convention

In debian, we use a convention in naming the upstrem tarballs ie, package-name_version.orig.tar.gz

Since multiple languages may use the same name for libraries, we add name of the language to the debian package name. So if we have an http library, the debian package names will be node-http, python-http, ruby-http, libhttp (for C library), libhttp-perl, libhttp-java etc This is to make sure we can easily identify libraries in a particular language and also make sure we have unique package names. So node-pretty-hrtime_1.0.3.orig.tar.gz is the exact same source code released by the upstream project. You can extract this file and compare it with the tarball you download from github, it will be the same.

Some times we may have to remove some files, but most of the time, it will be the exact same file as released by upstream.

Missing tags in upstream repo

Shanavas: it will be the same, if its tagged properly :)

Pirate Praveen: yes :) many nodejs developers are too lazy to tag releases or add good descriptions. If they don't add tags for releases, we have to manually find the release from their commit logs and download the commit tarball.

Shanavas: Better to mention fake upstream ?

Pirate Praveen: fakeupstream may not have tests and can have pre built binaries so manual download is preferred. For simple packages fakeupstream may be enough.

If they forget to add tag for their releases another option to get tarball is from npmjs.com itself, what is called as dist tarballs. It may be enough for simple cases where the npmjs.com dist tarball has tests and it does not include any minified/browserified/transpiled files. Read more about downloading npm dist tarballs

Otherwise we prefer a manual download via commit log.

Package Directory

Pirate Praveen: okay next we are going to see about the folder node-pretty-hrtime-1.0.3. It is a combination of the orig.tar.gz extracted and debian folder created by npm2deb.

$ ls
debian  index.js  LICENSE  package.json  README.md  test

In case of rubygems, debian folder will be created by another tool called gem2deb. For a lot of other tarballs, there is a tool called dh-make which will give you a generic debian folder if you give it any kind of tarball. But we prefer a specialized tool like npm2deb over dh-make if it is available.

look inside node-pretty-hrtime-1.0.3 and see if you are still with me.

node-pretty-hrtime_1.0.3-1.debian.tar.xz is just debian folder compressed since xz is more efficient algorithm debian prefers .tar.xz. Earlier it also used to be tar.gz

Debian source package

Now in node-pretty-hrtime-1.0.3 we have the required files to create a debian source package.

if you run

dpkg-source -b .
dpkg-source: info: using source format '3.0 (quilt)'
dpkg-source: info: building node-pretty-hrtime using existing ./node-pretty-hrtime_1.0.3.orig.tar.gz
dpkg-source: info: building node-pretty-hrtime in node-pretty-hrtime_1.0.3-1.debian.tar.xz
dpkg-source: info: building node-pretty-hrtime in node-pretty-hrtime_1.0.3-1.dsc

you'll get node-pretty-hrtime_1.0.3-1.dsc in the parent directory you can open it and see. It is a text file that has checksums for the tar files we need.

if you looked in the output, dpkg-source used .orig.tar.gz, created debian.tar.xz and .dsc files.

dsc file summarizes information in debian folder (just picks some important fields) and adds checksum for the tar files which will use to build the .deb file.

checksum can be used to ensure integrity ie, someone has not changed in after the checksum was created. It can also make sure the download was not corrupted.

Building the .deb

now if you run

dpkg-buildpackage

you'll get a .deb file and .changes file

.deb you already know

.changes is similar to .dsc, but has addiitonal checksums for .dsc and .deb files too

We sign the changes file with our gpg key, so we know the checksums are not modified.

gpg uses asymmetric key cryptography. So I keep the private key with me and share the public key. Anyone who has the public key can verify my signature, but only I can make the signature.

Lintian

Now we need to tune our deb file to make it compliant with debian policy. Some clues were given by npm2deb itself at the end of npm2deb create. We can use a tool called lintian to find those and more issues about our package.

lintian is run against the changes file

lintian ../node-pretty-hrtime_1.0.3-1_amd64.changes

now you need to fix the problems reported by lintian. You need to edit files inside debian directory. Main files are control, copyright, changelog. Sometimes you will need to edit more files too.

You can actually edit, dpkg-buildpackage and run lintian again to see if the problem is gone, but we will convert the package directory to a git repo for easy tracking.

Follow Converting your source package to a git repo.