29505
Comment: link ftp-master/release team wiki pages
|
29688
mention Linux live patching support
|
Deletions are marked like this. | Additions are marked like this. |
Line 217: | Line 217: |
* Work with the upstream and Debian teams for Linux kernel live patching to add support for your architecture, so users don't need to reboot to fix Linux kernel security issues. |
This is an aspirational document describing the most ideal procedure for creating a new port and getting it included into Debian. Since every port will have different circumstances, the procedure may differ for your port, but the best way to do it should be documented below. It is a work in progress and is based on the port template wiki page.
Please note that the people you will interact with in the Debian community are likely volunteers, so please respect their time. Please be patient if they take a long time to complete tasks you have requested. It may be appropriate to send polite followup requests to complete work, but please wait a reasonable amount of time before sending them. Please follow the Debian code of conduct while interacting with the Debian community.
If you encounter any unfamiliar jargon, please refer to the Debian glossary for more information.
If you require help at any stage, each section below may include specific teams to contact or the debian-devel mailing list may be able to provide fallback assistance.
When filing bug reports with Debian that are related to your port, please use architecture usertags to mark the bugs as relating to your port. Until your port family has a mailing list, you can use a likely mailing list name for usertags even while it doesn't exist. If the list name gets changed before it is accepted, then you can move all existing bug usertags to the new list name. You should not use the mailing list name in the X-Debbugs-CC pseudo-header until it exists, once it exists you should start using it. The documentation below uses XCC as an abbreviation of X-Debbugs-CC. Do not add the usertags and X-Debbugs-CC for your architecture to a bug report if the bug also applies to more commonly used architectures like amd64.
It is best to not skip any of the steps.
It is best to complete each item before moving to later steps, but some items incur significant delay, so working on them in parallel often is necessary.
Preparation
These steps are useful to support the work needed on the port throughout the process. Once completed, the work will go more smoothly and quickly.
- Assemble a team of people who will work on creating the port and or getting it included in Debian.
- The work on the port will require English communication, co-ordination, social and low-level technical skills. Different people can provide different skills.
- Find a financial sponsor to support the work, or otherwise ensure the team is able to do the work on an ongoing basis for the lifetime of the port.
Upstreaming
These steps are useful for adding support for the port to the core upstream projects that Debian inherits much of our architecture support from. Once completed, they will enable the creation of ports for any distro, including Debian.
Decide on the architecture ABI details and the GNU triplet. A different GNU triplet is required for each ABI, which includes bitness, endianness, soft/hard-float, calling convention, or whether object files can be inter-linked. Document the details of the ABI/psABI.
Register accounts for each team member on the bug trackers and subscribe to the mailing lists for upstream projects: GNU config, binutils, gcc, gdb, qemu, kernel (such as Linux), and libc (such as glibc). These will be needed for the other upstreaming steps.
Prepare kernel and toolchain patches supporting the port and get the necessary changes included into these upstream projects: GNU config (instructions), binutils, gcc (instructions), gdb (instructions), the kernel (such as Linux (info)) and the libc (such as glibc (instructions)) (in that order)
If hardware supporting your port is not yet available or your hardware or FPGA softcore or software model is quite slow, prepare patches for the appropriate simulator/emulator upstream project (such as qemu (instructions)). If you do not need this, it will still be very useful to complete at some point, so consider completing it later on.
Downstreaming
These steps are useful for adding support for the port to the core Debian-specific projects that define our architecture support. Once completed, the port will be well defined and it will be possible to start creating it.
- Identify related ports and ISAs, decide that the port is worth creating and write down the reasons why.
Register accounts for each team member on the Debian wiki (registration) and Salsa (register). These will be needed for documenting the port and sending changes to Debian related projects.
Register and verify accounts for each team member on the OFTC IRC chat network and join the #debian-bootstrap, #debian-ports and other Debian IRC channels. This will be needed for chatting in realtime with the Debian community.
Discuss the port with the community, including the reasons for creating the port and the intended architecture details. This is so that the community can ask questions and provide information to you, the community is aware of the work you will be doing and can contribute to the needed work. Please send an email to the debian-devel mailing list. If the port is related to some other architectures, ensure you CC the mailing list for the architecture family. Also join the #debian-bootstrap and #debian-ports IRC channels.
- Finalise the architecture details, especially including the ABI and CPU baseline requirements.
Get any necessary changes included into dpkg (instructions), to get the new Debian architecture name supported (which is the name that end-users will use for the architecture). This requires having the architecture ABI details clear; the Debian multiarch tuple is derived from the Debian architecture tuple. Ensure your team members are also on the #debian-dpkg IRC channel to help answer questions via chat if needed. If a port family mailing list exists already, ensure you CC it or when filing bugs, XCC it.
Bootstrap
These steps are useful for creating an initial package set used for building the port. Once completed, it will be possible for people to locally build enough packages to create a chroot of the port and build further packages within the chroot.
You can discuss the bootstrap stage on the debian-cross mailing list or the #debian-bootstrap IRC channel. If a port family mailing list exists already, ensure you CC it or when filing bugs, XCC it.
Prepare patches supporting the port in rebootstrap and other Debian-specific packages.
Use your modified rebootstrap, the other patches and manual bootstrapping to build build-essential and all of the packages required by it (this is known as the build-essential package set).
Notably, it will be useful to use the nodoc and nocheck profiles that avoid a lot of build cycles. Also check for package-specific build profiles, such as pkg.cmake.bootstrap
Keep working on the bootstrap process until the build-essential package set is installable with debootstrap --variant=buildd.
- Continue building as many packages as you can using the build-essential package set.
- Get any remaining changes included into the upstream projects that you have modified.
- Get any necessary changes included into rebootstrap and other projects involved in the bootstrap process.
Unofficial port
These steps are useful for expanding the initial package set for the port from the previous step up to a full unofficial port that is useful for users of Debian unstable. Once completed, it will be possible for users to install chroots or systems of various kinds for testing the port or using it casually, with updates happening daily.
- You can discuss the unofficial port stage on the specific mailing list and IRC channel for your port or the #debian-ports IRC channel.
Register the port on the Debian wiki, replace "example" throughout the page with the chosen architecture details and fill out as many details as you have. The resulting page will serve as a status summary for your port and should get updated as tasks are completed or things change.
Summarise the details of the architecture and ABI on the arch summary wiki page.
Set up communication channels for the architecture. Use the architecture family as the name of any communication channels you create, just in case additional related architectures are created later. A mailing list and IRC channel are recommended. These should be named after the architecture family name, not after the Debian port/architecture name. For future communication with Debian, always CC the port mailing list and when filing bugs, always XCC a copy to the port mailing list. Ensure your team members are also on the #debian-lists IRC channel to help answer questions via chat if needed.
Add your architecture to the bugs arch usertags page.
Contact the Debian Ports archive admins via email asking for the port to be included amongst the unofficial ports. Include in your email a summary of the status of the port, a link to the port wiki page, your plans for build servers (known as buildds) (see the buildd item below), the status of the build-essential package set for the port (see the rebootstrap item below), and the OpenPGP keys of a few people (~1–5) who will be allowed to upload packages for the port. Ensure your team members are also on the #debian-ports IRC channel to help answer questions via chat if needed. Ensure you CC the port mailing list.
Run rebootstrap locally to produce initial .deb binary packages for your port. Create a native chroot or install using the rebootstrap .deb files. Then rebuild all of the .deb binary packages within the native chroot or install. Then read the upload policy for the Debian Ports archive. Then upload (using dupload/dput/dput-ng) the rebuilt .deb binary packages to the Debian Ports archive. The initial package set needs to be enough to run a buildd chroot (sbuild can use the sudo mode rather than schroot, to simplify the build chain, and ssmtp can be used to send mail logs), which includes the packages installed by debootstrap variant=buildd and debhelper (which is used by most packages).
Set up unofficial buildds (automated setup) for the port, either in qemu or hardware. Contact the Debian buildd support team requesting that your port and buildds be added to the package build orchestration service (known as buildd.debian.org). Ensure your team members are also on the #debian-buildd IRC channel to help answer questions via chat if needed. Ensure you CC the port mailing list.
- More dependency loops probably need to be broken. Best is to break them locally using build profiles and upload a full-profile package to debian-ports. At worse, one may need to upload a cross-built package. In all cases, a binNMU must be triggered to make sure that a properly-built package gets installed.
Include the port on the ports list on the Debian website (instructions) and add the end-user name for the port to the website arches data. If you create a page on the website, create one named after the architecture family, just in case additional related architectures are created later. Ensure your team members are also on the #debian-www IRC channel to help answer questions via chat if needed. Ensure you XCC the port mailing list on any bugs you file.
Once the website page is published, send the Debian sysadmins a patch to the web server config for redirecting from the port architecture name to the website port family page
Monitor the usertagged bug reports and continually fix issues.
Optionally, donate hardware for the GCC compile farm. It is recommended to do this so that people outside Debian also have access to hardware for testing toolchains or other software on your port. Ensure you CC the port mailing list on any email based contact.
Set up porterboxen for Debian package maintainers to login to and port packages.
Optionally, discuss setting up runners for debci with the admins. It is recommended to do this so Debian package maintainers have a way to automatically run as-installed package tests. Ensure you CC the port mailing list. Ensure your team members are also on the #debci IRC channel to help answer questions via chat if needed.
Optionally, donate hardware for debomatic. It is recommended to do this so Debian package maintainers have a way to automatically trigger custom package builds. Ensure you CC the port mailing list.
Encourage the involvement of new people in the work needed for the port, especially existing Debian members and contributors and others with some intrinsic interest in the port. Some of the items from the other work section may be useful here.
Ensure that there is at least one Debian member is working on the port, either by recruiting an existing Debian member to the team, or by having an existing port team member join Debian as a member, with uploading privileges. This is likely to be needed for the official port stage. Some of the items from the other work section may be useful here.
Port the Debian installer so that it works on the hardware you have available. When filing bugs, ensure you XCC the port mailing list. Ensure your team members are also on the #debian-boot and #debian-cd IRC channels to help answer questions via chat if needed.
- Use the Debian installer to install Debian on the hardware you have available.
Write some documentation about installing the arch on the hardware you have installed.
Submit installation reports for the hardware you have installed.
Submit some hardware probes for the hardware you have installed.
Ensure that the key packages for a Debian release are built and work on your port.
Ensure that all the tasks exported by tasksel are built for your port, installable for your port and work on your port.
Official port
These steps are useful for ensuring the port is high quality and useful for users of Debian stable. Once completed, it will be possible for users have more confidence in the port and use it in production with security support.
Improve the port until it satisfies the criteria for being included in the main archive. Document (with a wiki page named after the port below ArchiveQualification) how the port satisfies the criteria, by answering the questions included in the archive criteria. If an official port can no longer satisfy these criteria, it may be converted back to an unofficial port.
Improve the port until it satisfies the criteria for being included in a Debian release, updating the qualification status as progress is made on each of the criteria. These criteria must be satisfied for each new Debian release and are recertified each time Debian makes a new release. If an official port can no longer satisfy these criteria, it may be converted back to an unofficial port.
Grow the hardware ecosystem until it satisfies the requirements for an official port. Document (with a wiki page named after the port below HardwareQualification) how the current/planned hardware for the port meets these requirements. If an official port can no longer satisfy these criteria, it may be converted back to an unofficial port.
Send a mail to the ftp-masters, the release team and the Debian sysadmins (DSA) summarising the state of the port and including links to the requested documentation about how the port meets all of the requirements and requesting approval of the port. Ensure you CC the port mailing list. Ensure your team members are also on the #debian-ftp, #debian-release and #debian-admin IRC channels to help answer questions via chat if needed.
Donate/purchase appropriate hardware for the official buildds/porterboxen and co-ordinate with DSA via the request-tracker to set up the hardware in Debian hosting locations. Ensure you CC the port mailing list. Ensure your team members are also on the #debian-admin IRC channel to help answer questions via chat if needed.
File a request against the ftp.debian.org pseudo-package for inclusion of the port amongst the official ports, linking to the HardwareQualification, ArchiveQualification, ReleaseRecertification, Ports and other pages for the port. When filing the bug, ensure you XCC the port mailing list. Ensure your team members are also on the #debian-ftp IRC channel to help answer questions via chat if needed.
- The process for switching from an unofficial port to an official port is approximately:
- The Debian hosting providers will connect the hardware to power, network and remote access.
- The Debian sysadmins will install the unofficial Debian port on the hardware.
- The Debian archive admins will import enough binary packages from the unofficial port to be able to set up a build chroot.
- These will be signed by a special key so they can be identified during the process.
- The build chroots will be created using the current packages.
- The Debian buildd support team will add the buildds to wanna-build.
- The rebuild process is started and runs continously:
- The buildds connect to wanna-build, build packages accordingly and upload them to the archive
- The packages signed with the special key will be binNMUed and uploaded to the archive
The port team will use their own local systems to break any build cycles using build profiles and other bootstrap processes (see Ports). The packages built using non-default build profiles or other bootstrap processes must not be uploaded to the archive by the porters. The bootstrap packages can be used to locally build normal packages and then the normal packages can be uploaded.
- These will be signed by the special key so they can be identified during the process.
- The build chroots will be regularly recreated using the current packages.
- The rebuild process is finished when:
- none of the packages signed with the special key remain
- there are enough packages for the next step
- The Debian sysadmins will re-install the official port on the buildd hardware.
- The build chroots will be recreated using the current packages.
- The Debian buildd support team will add the new buildds to wanna-build.
- The buildds will continue to build packages and upload them to the main archive.
- The port team will continue to use bootstrap processes as above to make it possible to build more packages.
File a request against the release.debian.org pseudo-package for inclusion of the port in Debian testing, linking to the HardwareQualification, ArchiveQualification, ReleaseRecertification, Ports and other pages for the port. When filing the bug, ensure you XCC the port mailing list. Ensure your team members are also on the #debian-release IRC channel to help answer questions via chat if needed.
Add quality assurance for the port for Reproducible Builds, Debian CI, and piuparts (doesn't yet support non-x86 ports) and other Debian QA efforts. Ensure you CC the port mailing list or when filing bugs, XCC the port mailing list. Ensure your team members are also on the #debian-reproducible, #debci and #debian-qa IRC channels to help answer questions via chat if needed.
Add support for the port to the live images (doesn't yet support non-x86 ports) and cloud images if appropriate hardware exists. Ensure you CC the port mailing list or when filing bugs, XCC the port mailing list. Ensure your team members are also on the #debian-live and #debian-cloud IRC channels to help answer questions via chat if needed.
Other work
Unlike the above sections, this one is an unordered wishlist of optional ideas to improve the experience of a port for Debian contributors and users. Ensure you CC the port mailing list or when filing bugs, XCC the port mailing list.
- Improve the port based on reports from users and QA services.
Subscribe to debian-devel-changes and filter it for the port name and keywords related to the port such as the port family name, "big-endian" or other characteristics of the port.
- Make the port more accessible by adding support for instruction and system emulation to qemu or adding other simulators/emulators to Debian.
Make the port more accessible by donating hardware to the openSUSE Build Service and or setting up a community and or commercial cloud service. This is recommended so that people outside Debian also have access to hardware for porting.
Make the port better supported by core projects by donating hardware for these compile farms: Sourceware, Exim, PostgreSQL. It is recommended to do this so that people outside Debian also have access to hardware for building and testing their software on your port. Ensure you CC the port mailing list on any email based contact.
- Make the port support more hardware by upstreaming support to linux-firmware, bootloaders, Linux, udev, mesa, flash-kernel and other projects then updating those projects in Debian.
- Make the port more complete by porting packages that require porting to every new architecture, including these source packages: lintian linuxinfo arch-test
- Make the port more complete and correct by checking if packages, that are not marked as buildable on the port or not marked as testable on the port, are actually able to be successfully built or tested.
- Make the port more complete and correct by updating packages that are classified by lintian as needing porting:
- Ideas for classification tags:
- debian-control-architecture-blocks-$arch
- debian-control-build-depends-missing-on-$arch
- debian-control-depends-missing-on-$arch
- debian-tests-control-architecture-blocks-$arch
- debian-rules-checks-architecture-$arch
- debian-rules-references-architecture-$arch
- source-code-references-defines-$arch
- FIXME: this idea is not yet submitted or implemented
- Ideas for classification tags:
If the hardware can have optional instructions that enable programs to work faster when they are available on a particular CPU, ensure that the instruction selection mechanisms work for your port. This will often be required for making the port more performant.
Make the port more performant by porting or optimising various libraries, language runtimes and signal processing algorithms and cryptography libraries, for example: openssl, gnutls, openjdk, v8, ffmpeg, golang, gccgo, ghc, ocaml, simde.
Make it possible for users to easily update the preinstalled firmware on their devices by having the common hardware vendors add their prebuilt firmware to the Linux Vendor Firmware Service and or documenting how to update the firmware manually.
Make the port more DFSG compatible by reverse engineering common proprietary firmware, rewriting the firmware, documenting that new open firmware and packaging that firmware for Debian.
- Make the port available for maintainers and users to cross-build packages to:
Add support to the binutils source package
Add libc6*-dev-$arch-cross, libc6*-$arch-cross and linux-libc-dev-$arch-cross packages to the cross-toolchain-base (for ftp-master.d.o arches) or cross-toolchain-base-ports (for ports.d.o arches) source packages
- Add support to the current gcc-N-cross (for ftp-master.d.o arches) or gcc-N-cross-ports (for ports.d.o arches) source package
Add metapackages to the gcc-defaults (for ftp-master.d.o arches) source package or gcc-defaults-ports (for ports.d.o arches) source package
Add a crossbuild-essential-$arch metapackage to the build-essential source package
- Work with the upstream and Debian teams for Linux kernel live patching to add support for your architecture, so users don't need to reboot to fix Linux kernel security issues.
Make the bootstrap process more trustworthy through Bootstrappable Builds.
Make the port used more among Debian contributors by giving away hardware via lottery at DebConf, donating hardware to contributors, providing discounts to Debian contributors for hardware etc.
Make the port, the status of it, changes to it and the usage of it more widely known through talks and BoFs at DebConf and other events, DevNews (and other publicity team avenues), by filing bugs from machines running the port, submitting reports to the package popularity contest etc.
Make the port more solidly supported through an official partnership with Debian. Support Debian itself through sponsoring DebConf, regular donations or helping Debian more generally.
Make the port professionally supported by registering as a consultant for Debian and the port.
Hire existing contributors to the port or new people who have worked on other ports. You can use the debian-jobs mailing list, FOSSjobs or other FOSS hiring resources to advertise available positions.
Discuss about gitlab-runners with Teams/SalsaCI. gitlab-runners can be used to run the Salsa CI's pipeline, that provides Debian package maintainers with a way to automatically build and test changes to packages before upload.
Discuss with the LTS team about including the port within the releases that have LTS and or eLTS support.