Creating a yarn plugin to resolve node modules installed as debian packages via apt is a project proposal for Outreachy 2021.

Background information

Matrix group for discussions: https://matrix.to/#/#debian-js-mentors:poddery.com

Video recording of first meeting: https://meet.nixnet.services/playback/presentation/2.0/playback.html?meetingId=e9328c7be78818a0e1c14b63f85e6eb84e033166-1617367905517

Aim of Debian (or Debian packaging):

- To have every free software available in an easy to install form and easy to configure, using just the default package manager, irrespective of which language/framework it is written in (node, ruby, python, etc)

- Many packages like gitlab, diaspora, etc have node modules as front-end dependencies and also have to be available in the same way.

But, gitlab, for example, has 1600+ dependencies.

- Till all these are packaged, we have been following a hybrid approach where when someone does apt install:

* If there is already a compatible version in debian software repositories, use that
* Otherwise, use the library from npmjs.com/yarnpkg (this option is undesirable)

- The aim of this project is to create a plugin for yarn (specifically yarn version >=2, nicknamed berry) package manager such that yarn when installing dependencies from package.json looks up the debian's package if available and compatible and only if not available goes to yarn registry and downloads the library.

- Ruby, similarly, already has a bundle install --local option which does the same.

Node/javascript development background:

- When you want to use a third party module/library in your node-javascript application, you specify the name and version of these libraries in a package called package.json
- If you share this package.json file with someone else, they will be able to download those third party libraries of the exact versions that you used on to their computer as well.
- These libraries are saved in the "node_modules" folder by default and the behaviour/features that are present in those third party libraries become available to use in your own code.

Real life example: Imagine you have a third party library called "lodash" in package.json which you might have installed with

or

You can get that in your software (index.js file, say) with this:

or :

- This helps you avoid handling third party/vendor libraries manually and you can quickly upgrade libraries, etc.
- Now, what if we could use a library that is already available on the computer instead of getting it from the internet? What if the same library has been installed as a dependency for another software?

Example: https://salsa.debian.org/ruby-team/gitlab/-/blob/master/debian/patches/0740-use-packaged-modules.patch#L120

The philosophy of Debian regarding shared dependencies:

- Whatever version of the library is available in Debian should be used by any software that depends on that library. This probably involves updating all software to work with the latest version of a library.
- Now, any software that needs this library should be able to use the Debian version of the library installed to /usr/share/nodejs - apt install node-lodash will put lodash in /usr/share/nodejs
- Now, with the yarn 2 plugin which we create in this project, I should be able to do something like this:

And it should use the lodash from the /usr/share/nodejs instead of downloading from the internet

Q: Do we have a similar thing in npm/yarn1?
A: No. npm and yarn1 do not support plugins.

Debian and upstream projects

Debian is an operating system and therefore a lot of software that Debian offers its users are not built by people in the Debian community. The people who maintain software are called "upstream maintainers". Debian works closely with upstream to package and deliver these software on Debian.

The upstream developers of yarn, for example, aren't part of Outreachy. Please keep that in mind while communicating with upstream.

Technical Notes

Setting up a Debian unstable environment

See Packaging/Pre-Requisites for various options available. Use https://matrix.to/#/#fsci-support:poddery.com for installation support.

NB: When installing the Docker as pre-requisite for the Debian unstable, you will need to add "sudo" before the command if you are not a root user. For example to pull the development image, you will run this command: sudo docker pull registry.gitlab.com/fsci/resources:debian-dev. https://wiki.debian.org/Packaging/Pre-Requisites#Docker If you omit the "sudo", you might get a message that access is denied. Just add sudo in front and it will run just fine. That's one way around.

However, on production servers, you would prefer to avoid running docker as root user (as that would mean that docker gets access to the entire computer and any insecurities in docker can lead to your server being compromised).

After running the command: "sudo docker run --privileged --name "sid" -it registry.gitlab.com/fsci/resources:debian-dev bash", you will be assigned a developer identity.

Next step is to run this command: "sudo apt-get update && sudo apt-get upgrade" You will be prompted to enter a password. The password is "developer". This will update the docker if it is not the current version and also upgrade if an upgrade is available.

Debian and Nodejs

See Javascript/Nodejs.

To install nodejs,

# apt install nodejs

Installing yarn

In Debian yarn is provided by yarnpkg package (as yarn command is already taken by cmdtest package. See 913997 for details).

# apt install yarnpkg

Using yarn 2 with node_modules plugin

$ mkdir test-project
$ cd test-project
$ yarnpkg add pretty-ms
$ yarnpkg set version berry
$ if ! grep nodeLinker .yarnrc.yml >/dev/null; then echo "nodeLinker: \"node-modules\"" >>.yarnrc.yml; fi
$ yarnpkg install

End of Scripts

This will create .yarnrc.yml file in the current direcory and will append the line nodeLinker: "node-modules" to a .yarnrc.yml file which tells yarn 2 to use node_modules plugin to have the same behaviour of yarn 1 or npm. Note that yarn 2 does not create nodes_modules directory by default.

Initial Tasks

Packaging tasks

  1. Checkout npm2deb and create a package (pretty-ms, for example). Then look at the deb file which got created and see the folder structure, etc. npm2deb create pretty-ms will get you a .deb. Inspect the deb using an archive manager like file roller or install the deb using dpkg and see where it got installed, dpkg -L node-pretty-ms https://github.com/LeoIannacone/npm2deb

  2. Learn how to create a deb package for simple modules following https://wiki.abrahamraji.in/simple-packaging-tutorial/ and https://wiki.debian.org/SimplePackagingTutorial. See also https://blog.packagecloud.io/eng/2015/07/14/using-dh-make-to-prepare-debian-packages/ Note: debmake is broken for node packages, so use dh_make command from dh-make package instead.

  3. Update an existing node package to new upstream minor or patch version following https://wiki.debian.org/Javascript/Nodejs/Npm2Deb#Updating_a_package_to_new_upstream_release (Pick a node package from https://qa.debian.org/developer.php?email=praveen%40debian.org compare the versions in unstable or experimental with version in watch column. https://semver.org/ defines what is a major, minor or a patch version.)

Nodejs tasks

Documentation tasks

PS: Send a mail to wiki@debian.org explaining why you want an account created, and included in that mail should be your email address for the account creation, else the account will not be created. Account creation is only limited to this process to prevent the system from being spammed by random editors.

About Patch File

In debian, whenever we need to change a file in the upstream code, we create a patch for that file and the regarding changes are saved in that patch file. In most of the cases, the patch file is needed to be forwarded upstream and involves coordinating with other upstream developers. One of the most simple and best way is to open a pull request to the upstream repository. Keep note of the following points:

  1. In debian, we add build dependencies in control file but upstream uses package.json, which also needed to be updated.
  2. Before opening a PR always do a yarn or npm install based on the upstream preferred package manager and run test with the your changes.

Helpful resources- https://wiki.debian.org/UsingQuilt https://tools.ietf.org/doc/quilt/quilt.html

List of packages with minor/patch updates

(Before updating existing modules, always ensure that the packages are not updated in Debian archives)

Outreachy Project Status

First Meeting ( 21st May 2021 ) : Division of tasks

First meeting on 21st May 2021 at https://jitsi.debian.social/yarn2-plugin-apt

Agenda: Division of tasks, VCS repo for the project (salsa js-team native package), License for the project (Apache 2 or LGPL v3? https://licenseuse.org), Expanding scope of adding yarn plugin apt test in autopkgtest-pkg-nodejs (autodep8)

Update yarnpkg using corepack repo https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=980316

Build yarn plugin https://salsa.debian.org/js-team/yarn-plugin-apt

Blogs


CategoryDebianDevelopment