- Background information
- Technical Notes
- Initial Tasks
- Outreachy Project Status
Creating a yarn plugin to resolve node modules installed as debian packages via apt is a project proposal for Outreachy 2021.
Matrix group for discussions: https://matrix.to/#/#debian-js-mentors:poddery.com
Video recording of first meeting: https://meet.nixnet.services/playback/presentation/2.0/playback.html?meetingId=e9328c7be78818a0e1c14b63f85e6eb84e033166-1617367905517
Aim of Debian (or Debian packaging):
- To have every free software available in an easy to install form and easy to configure, using just the default package manager, irrespective of which language/framework it is written in (node, ruby, python, etc)
- Many packages like gitlab, diaspora, etc have node modules as front-end dependencies and also have to be available in the same way.
But, gitlab, for example, has 1600+ dependencies.
- Till all these are packaged, we have been following a hybrid approach where when someone does apt install:
* If there is already a compatible version in debian software repositories, use that
* Otherwise, use the library from npmjs.com/yarnpkg (this option is undesirable)
- The aim of this project is to create a plugin for yarn (specifically yarn version >=2, nicknamed berry) package manager such that yarn when installing dependencies from package.json looks up the debian's package if available and compatible and only if not available goes to yarn registry and downloads the library.
- Ruby, similarly, already has a bundle install --local option which does the same.
- If you share this package.json file with someone else, they will be able to download those third party libraries of the exact versions that you used on to their computer as well.
- These libraries are saved in the "node_modules" folder by default and the behaviour/features that are present in those third party libraries become available to use in your own code.
Real life example: Imagine you have a third party library called "lodash" in package.json which you might have installed with
yarn add lodash
npm install lodash
You can get that in your software (index.js file, say) with this:
const lodash = require("lodash");
import lodash from "lodash";
- This helps you avoid handling third party/vendor libraries manually and you can quickly upgrade libraries, etc.
- Now, what if we could use a library that is already available on the computer instead of getting it from the internet? What if the same library has been installed as a dependency for another software?
The philosophy of Debian regarding shared dependencies:
- Whatever version of the library is available in Debian should be used by any software that depends on that library. This probably involves updating all software to work with the latest version of a library.
- Now, any software that needs this library should be able to use the Debian version of the library installed to /usr/share/nodejs - apt install node-lodash will put lodash in /usr/share/nodejs
- Now, with the yarn 2 plugin which we create in this project, I should be able to do something like this:
yarn apt-add lodash (or something similar)
And it should use the lodash from the /usr/share/nodejs instead of downloading from the internet
Q: Do we have a similar thing in npm/yarn1?
A: No. npm and yarn1 do not support plugins.
Debian and upstream projects
Debian is an operating system and therefore a lot of software that Debian offers its users are not built by people in the Debian community. The people who maintain software are called "upstream maintainers". Debian works closely with upstream to package and deliver these software on Debian.
The upstream developers of yarn, for example, aren't part of Outreachy. Please keep that in mind while communicating with upstream.
Setting up a Debian unstable environment
NB: When installing the Docker as pre-requisite for the Debian unstable, you will need to add "sudo" before the command if you are not a root user. For example to pull the development image, you will run this command: sudo docker pull registry.gitlab.com/fsci/resources:debian-dev. https://wiki.debian.org/Packaging/Pre-Requisites#Docker If you omit the "sudo", you might get a message that access is denied. Just add sudo in front and it will run just fine. That's one way around.
However, on production servers, you would prefer to avoid running docker as root user (as that would mean that docker gets access to the entire computer and any insecurities in docker can lead to your server being compromised).
After running the command: "sudo docker run --privileged --name "sid" -it registry.gitlab.com/fsci/resources:debian-dev bash", you will be assigned a developer identity.
Next step is to run this command: "sudo apt-get update && sudo apt-get upgrade" You will be prompted to enter a password. The password is "developer". This will update the docker if it is not the current version and also upgrade if an upgrade is available.
Debian and Nodejs
To install nodejs,
# apt install nodejs
In Debian yarn is provided by yarnpkg package (as yarn command is already taken by cmdtest package. See 913997 for details).
# apt install yarnpkg
Using yarn 2 with node_modules plugin
$ mkdir test-project $ cd test-project $ yarnpkg add pretty-ms $ yarnpkg set version berry $ if ! grep nodeLinker .yarnrc.yml >/dev/null; then echo "nodeLinker: \"node-modules\"" >>.yarnrc.yml; fi $ yarnpkg install
End of Scripts
This will create .yarnrc.yml file in the current direcory and will append the line nodeLinker: "node-modules" to a .yarnrc.yml file which tells yarn 2 to use node_modules plugin to have the same behaviour of yarn 1 or npm. Note that yarn 2 does not create nodes_modules directory by default.
Checkout npm2deb and create a package (pretty-ms, for example). Then look at the deb file which got created and see the folder structure, etc. npm2deb create pretty-ms will get you a .deb. Inspect the deb using an archive manager like file roller or install the deb using dpkg and see where it got installed, dpkg -L node-pretty-ms https://github.com/LeoIannacone/npm2deb
Learn how to create a deb package for simple modules following https://wiki.abrahamraji.in/simple-packaging-tutorial/ and https://wiki.debian.org/SimplePackagingTutorial. See also https://blog.packagecloud.io/eng/2015/07/14/using-dh-make-to-prepare-debian-packages/ Note: debmake is broken for node packages, so use dh_make command from dh-make package instead.
Create a git repo for your project at https://salsa.debian.org and add a README file explaining how to test your code.
Fix one of the "good first issue" bugs on yarn2: https://github.com/yarnpkg/berry/labels/good%20first%20issue (Eg: https://github.com/yarnpkg/berry/issues/1621 )
Update https://www.npmjs.com/package/typanion build dependency to @rollup/plugin-typescript instead of @wessberg/rollup-plugin-ts similar to how clipanion moved.
- Create a Debian wiki account and add notes
PS: Send a mail to firstname.lastname@example.org explaining why you want an account created, and included in that mail should be your email address for the account creation, else the account will not be created. Account creation is only limited to this process to prevent the system from being spammed by random editors.
About Patch File
In debian, whenever we need to change a file in the upstream code, we create a patch for that file and the regarding changes are saved in that patch file. In most of the cases, the patch file is needed to be forwarded upstream and involves coordinating with other upstream developers. One of the most simple and best way is to open a pull request to the upstream repository. Keep note of the following points:
- In debian, we add build dependencies in control file but upstream uses package.json, which also needed to be updated.
- Before opening a PR always do a yarn or npm install based on the upstream preferred package manager and run test with the your changes.
List of packages with minor/patch updates
- node-vue-hot-reload-api - Ajayi - RFS
- node-boom - Ajayi
- node-caniuse-db - Saakshi
- node-caniuse-lite - Sonnie
(Before updating existing modules, always ensure that the packages are not updated in Debian archives)
Outreachy Project Status
First Meeting ( 21st May 2021 ) : Division of tasks
First meeting on 21st May 2021 at https://jitsi.debian.social/yarn2-plugin-apt
Agenda: Division of tasks, VCS repo for the project (salsa js-team native package), License for the project (Apache 2 or LGPL v3? https://licenseuse.org), Expanding scope of adding yarn plugin apt test in autopkgtest-pkg-nodejs (autodep8)
Update yarnpkg using corepack repo https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=980316
Build yarn plugin https://salsa.debian.org/js-team/yarn-plugin-apt