- First Meeting (2022)
- Contact details
- Background information
- Technical Notes
- Initial Tasks
- Outreachy Project Status
- Helpful Q & A
Creating a yarn plugin to resolve node modules installed as debian packages via apt was a project proposal for Outreachy 2021. Some part was implemented in Outreachy 2021 and remaining part will be completed in Outreachy 2022.
First Meeting (2022)
https://framadate.org/2oKh3SK4XbwlcdaY - choose your preferred time if you are applying for this project - meeting link https://jitsi.debian.social/outreachy-2022 and time 18:00 GMT on 30th March 2022
You can use one of the options listed below to talk to mentors and interact with other applicants (they are bridged/interconnected so you only need to use one option from below)
XMPP group for discussions: https://firstname.lastname@example.org?join
Matrix group for discussions: https://matrix.to/#/#debian-js-mentors:poddery.com
IRC channel #debian-browserify on irc.oftc.net network. If you are new to irc, you can join via browser https://webchat.oftc.net/?channels=debian-browserify
Blog posts of previous interns (this will be very useful to new interns)
Contributor's first contribution experience (helpful for new interns)
Aim of Debian (or Debian packaging):
- To have every free software available in an easy to install form and easy to configure, using just the default package manager, irrespective of which language/framework it is written in (node, ruby, python, etc)
- Many packages like gitlab, diaspora, etc have node modules as front-end dependencies and also have to be available in the same way.
But, gitlab, for example, has 1600+ dependencies.
- Till all these are packaged, we have been following a hybrid approach where when someone does apt install:
* If there is already a compatible version in debian software repositories, use that
* Otherwise, use the library from npmjs.com/yarnpkg (this option is undesirable)
- The aim of this project is to create a plugin for yarn (specifically yarn version >=2, nicknamed berry) package manager such that yarn when installing dependencies from package.json looks up the debian's package if available and compatible and only if not available goes to yarn registry and downloads the library.
- Ruby, similarly, already has a bundle install --local option which does the same.
- If you share this package.json file with someone else, they will be able to download those third party libraries of the exact versions that you used on to their computer as well.
- These libraries are saved in the "node_modules" folder by default and the behaviour/features that are present in those third party libraries become available to use in your own code.
Real life example: Imagine you have a third party library called "lodash" in package.json which you might have installed with
yarn add lodash
npm install lodash
You can get that in your software (index.js file, say) with this:
const lodash = require("lodash");
import lodash from "lodash";
- This helps you avoid handling third party/vendor libraries manually and you can quickly upgrade libraries, etc.
- Now, what if we could use a library that is already available on the computer instead of getting it from the internet? What if the same library has been installed as a dependency for another software?
The philosophy of Debian regarding shared dependencies:
- Whatever version of the library is available in Debian should be used by any software that depends on that library. This probably involves updating all software to work with the latest version of a library.
- Now, any software that needs this library should be able to use the Debian version of the library installed to /usr/share/nodejs - apt install node-lodash will put lodash in /usr/share/nodejs
- Now, with the yarn 2 plugin which we create in this project, I should be able to do something like this:
yarn apt-add lodash (or something similar)
And it should use the lodash from the /usr/share/nodejs instead of downloading from the internet
Q: Do we have a similar thing in npm/yarn1?
A: No. npm and yarn1 do not support plugins.
Debian and upstream projects
Debian is an operating system and therefore a lot of software that Debian offers its users are not built by people in the Debian community. The people who maintain software are called "upstream maintainers". Debian works closely with upstream to package and deliver these software on Debian.
The upstream developers of yarn, for example, aren't part of Outreachy. Please keep that in mind while communicating with upstream.
Setting up a Debian unstable environment
See Packaging/Pre-Requisites for various options available. Use https://email@example.com?join or https://matrix.to/#/#fsci-support:poddery.com for installation support.
NB: When installing the Docker as pre-requisite for the Debian unstable, you will need to add "sudo" before the command if you are not a root user. For example to pull the development image, you will run this command: sudo docker pull registry.gitlab.com/fsci/resources:debian-dev. https://wiki.debian.org/Packaging/Pre-Requisites#Docker If you omit the "sudo", you might get a message that access is denied. Just add sudo in front and it will run just fine. That's one way around.
However, on production servers, you would prefer to avoid running docker as root user (as that would mean that docker gets access to the entire computer and any insecurities in docker can lead to your server being compromised).
After running the command: "sudo docker run --privileged --name "sid" -it registry.gitlab.com/fsci/resources:debian-dev bash", you will be assigned a developer identity.
Next step is to run this command: "sudo apt-get update && sudo apt-get upgrade" You will be prompted to enter a password. The password is "developer". This will update the docker if it is not the current version and also upgrade if an upgrade is available.
Debian and Nodejs
To install nodejs,
# apt install nodejs
In Debian yarn is provided by yarnpkg package (as yarn command is already taken by cmdtest package. See 913997 for details).
# apt install yarnpkg
Using yarn 2 with node_modules plugin
$ mkdir test-project $ cd test-project $ yarnpkg add pretty-ms $ yarnpkg set version berry $ if ! grep nodeLinker .yarnrc.yml >/dev/null; then echo "nodeLinker: \"node-modules\"" >>.yarnrc.yml; fi $ yarnpkg install
End of Scripts
This will create .yarnrc.yml file in the current direcory and will append the line nodeLinker: "node-modules" to a .yarnrc.yml file which tells yarn 2 to use node_modules plugin to have the same behaviour of yarn 1 or npm. Note that yarn 2 does not create nodes_modules directory by default.
Checkout npm2deb and create a package (pretty-ms, for example). Then look at the deb file which got created and see the folder structure, etc. npm2deb create pretty-ms will get you a .deb. Inspect the deb using an archive manager like file roller or install the deb using dpkg and see where it got installed, dpkg -L node-pretty-ms https://github.com/LeoIannacone/npm2deb
Learn how to create a deb package for simple modules following https://wiki.abrahamraji.in/simple-packaging-tutorial/ and https://wiki.debian.org/SimplePackagingTutorial. See also https://blog.packagecloud.io/eng/2015/07/14/using-dh-make-to-prepare-debian-packages/ Note: debmake is broken for node packages, so use dh_make command from dh-make package instead.
Create a git repo for your project at https://salsa.debian.org and add a README file explaining how to test your code.
Fix one of the "good first issue" bugs on yarn2: https://github.com/yarnpkg/berry/labels/good%20first%20issue (Eg: https://github.com/yarnpkg/berry/issues/1621 )
Update https://www.npmjs.com/package/typanion build dependency to @rollup/plugin-typescript instead of @wessberg/rollup-plugin-ts similar to how clipanion moved.
- Create a Debian wiki account and add notes
PS: Send a mail to firstname.lastname@example.org explaining why you want an account created, and included in that mail should be your email address for the account creation, else the account will not be created. Account creation is only limited to this process to prevent the system from being spammed by random editors.
About Patch File
In debian, whenever we need to change a file in the upstream code, we create a patch for that file and the regarding changes are saved in that patch file. In most of the cases, the patch file is needed to be forwarded upstream and involves coordinating with other upstream developers. One of the most simple and best way is to open a pull request to the upstream repository. Keep note of the following points:
- In debian, we add build dependencies in control file but upstream uses package.json, which also needed to be updated.
- Before opening a PR always do a yarn or npm install based on the upstream preferred package manager and run test with the your changes.
List of packages with minor/patch updates
- node-vue-hot-reload-api - Ajayi - RFS
- node-boom - Ajayi
- node-caniuse-db - Saakshi
- node-caniuse-lite - Sonnie
- node-enhanced-resolve - izzygala
- node-postcss-value-parser - Michael
- node-globals - Michael
- node-watchpack - izzygala
- node-vue-style-loader - Michael
- node-sshpk - izzygala
- node-dompurify - Omama
- node-bootstrap-sass - izzygala
- node-color - izzygala
- node-accepts - izzygala
- node-body-parser - izzygala
- node-getobject - izzygala
- node-toidentifier - izzygala
- node-bytes - izzygala
- node-regenerator-runtime - izzygala
(Before updating existing modules, always ensure that the packages are not updated in Debian archives)
New modules to be packaged
- canvas-confetti - Michael
- echarts - AmbadyAnandS
- @gitlab/favicon-overlay - Michael
- bundt - Michael
- meros (after bundt is packaged)
- graphql-ws - Michael
Advanced packaging updates
If you have done 2-3 minor updates and confident about the process and tools, then try one of these tasks
- Update graphiql upstream build system to use @types/codemirror 5.60.5 (it currently uses a very old version of the type definitions). This will help in packaging graphiql in debian.
Advanced nodejs tasks
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1010981 Fallback to tags without v prefix before switching to npm registry in watch file (npm2deb) (-izzygala)
gitlab use select2 3.5 and debian has 4.0, so it'd be good to update gitlab to use select3 4.0 https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87616
gitlab uses js-cookie 2.0 and we updated node-js-cookie in debian to 3.0. This release introduces a breaking change, so we should adapt gitlab to work with js-cookie 3.0. See https://gitlab.com/gitlab-org/gitlab/-/issues/358670 - DONE
Outreachy Project Status
TODO for 2022
First Meeting ( 21st May 2021 ) : Division of tasks
First meeting on 21st May 2021 at https://jitsi.debian.social/yarn2-plugin-apt
Agenda: Division of tasks, VCS repo for the project (salsa js-team native package), License for the project (Apache 2 or LGPL v3? https://licenseuse.org), Expanding scope of adding yarn plugin apt test in autopkgtest-pkg-nodejs (autodep8)
Update yarnpkg using corepack repo https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=980316
Build yarn plugin https://salsa.debian.org/js-team/yarn-plugin-apt
Helpful Q & A
To avoid repetition, some of our questions and answers from our mentors are being documented for reference
Q: im trying to add a test from package.json to debian/tests/pkg-js/test. I added this line from package.json
npx mocha --reporter spec --require ts-node/register "./test/**/*.ts"
but im getting this error https://paste.debian.net/1236377
A: you don't need npx when building Debian packages. mocha command is available directly in Debian when mocha installed via apt.