Differences between revisions 76 and 77
Revision 76 as of 2013-11-13 09:52:45
Size: 23977
Editor: ?JossMouette
Comment: Comment on upstart’s strive for “technical correctness”.
Revision 77 as of 2013-11-13 09:59:54
Size: 23993
Editor: ?JossMouette
Comment: Wording fix
Deletions are marked like this. Additions are marked like this.
Line 102: Line 102:
 * The statement affirms that systemd is mostly used “primarily in sysvinit compatibility mode”, which ironically is exactly how upstart is used in most Ubuntu packages, while Fedora already ships hundreds of systemd unit files.  * The statement affirms that systemd in other distributions is used “primarily in sysvinit compatibility mode”, which ironically is exactly how upstart is used in most Ubuntu packages, while Fedora already ships hundreds of systemd unit files.

This page documents the position statement of those who think that Debian should adopt systemd as the default init system. Before editing this page, please see the instructions on how debates work. If you are not the maintainer of a position statement, and have a suggestion or change to make, please contact the maintainers.

Systemd position statement

Executive summary

Systemd is becoming the de facto standard init system for Linux. It replaces the venerable SysV init with a clean and efficient design, and brings a stream of new functionality that makes the life of users, administrators and packagers easier. It is better than existing alternatives for all of Debian’s current use cases:

  • Application and infrastructure servers benefit from reliable and easy service management, cleaner dependencies, service monitoring, security features and global system integration.
  • Desktops, laptops and session servers benefit from session management, multi-seat, unified system interfaces, as well as integration with udev, D-Bus and other system services.
  • Embedded systems benefit from speed improvements, shell-less design, ability to remove optional components, and lower memory footprint.

Systemd represents a leap in terms of functionality, one that is comparable to Debian’s existing improvements over other operating systems. People are starting to expect this functionality when running Linux, and missing it could, in the long term, make Debian lose its purpose.

Why Debian should default to systemd

Fedora, OpenSuSE, Arch and Mageia have already made the choice to use systemd, and it is getting excellent upstream support for a growing number of packages.

Architecture

Systemd is well designed. It was conceived from the top, not just to fix bugs, but to be a correct implementation for the base system services.

  • Systemd makes the boot process much simpler, entirely removing the need to specify dependencies in many cases thanks to D-Bus activation, socket activation, file/inotify activation and udev integration.
  • Systemd can handle the boot process from head to toe, without needing to use any of the existing shell scripts.
  • Systemd is straightforward. The command-line interface is probably the best existing for service management. The unit file format (like .desktop or “ini” files) is completely declarative, can be parsed using standard tools, and is a breeze to maintain.
  • Systemd unit files, unlike SysV scripts, can usually be shipped by upstream, or at least shared with other distributions (already more than 1000 existing unit files in Fedora) without any changes, the Debian specifics being handled by systemd itself.
  • Systemd is incredibly fast (1 second to boot). It was not designed with speed in mind, but doing things correctly avoids all the delays currently incurred by the boot process.
  • The transition plan is easy, since existing init scripts are treated as first-class services: scripts can depend (using LSB headers) on units, units can depend on scripts. More than 99% of init scripts can be used without a modification.

Functionality

Systemd is not just init. It unifies, in fewer lines of code, everything that is related to starting services and managing session groups: user login, cron jobs, network services (inetd), virtual TTY management… Having a single system to handle all of that allows to remove a lot of cruft, and to use less memory on the system.

  • Systemd uses control groups to ensure that any service, regardless of its state, can be shut down properly.
  • Systemd extends the logging features of the system in many ways with journald, and can remain integrated with the existing rsyslog daemon. Logs are in a structured format, attributed to filename, line of code, PID and service. They include the early boot (starting from initramfs). They can be quickly filtered and programatically accessed through an efficient interface.
  • Systemd’s virtual TTY management allows real multi-seat support, and revoking access to input devices on session switch.
  • Systemd can trivially add security settings to a service without any need to patch it: user/group change, chroot, private network, private /tmp, read-only access to parts of the tree, tcp wrappers, filtering system calls, ?NoNewPrivileges (kernel feature to avoid privilege changes), limiting effective capabilities, limiting whatever the kernel provides in an integrated manner (device bandwidth, CPU usage, memory usage, OOM settings, nice levels, timer slack, ulimits…)

  • Systemd ships a growing number of useful, unified command-line interfaces for system settings and control (timedatectl, bootctl, hostnamectl, loginctl, machinectl, kernel-install, localectl). In Debian, they use the existing configuration files without breaking compatibility.
  • In addition to command-line interfaces, Systemd provides a D-Bus API to control service management and access system settings, as well as bindings to do that from Python, node.js, php, lua, Qt… GNOME is relying on these services, and it is likely for KDE and Xfce to use pieces of it in the near future. They are not specific to systemd per se, but so far it is the only implementation.

  • Systemd can run without any configuration change in a virtual machine or a container. It will automatically skip unwanted services.
  • Systemd provides full watchdog support. If configured to do so, PID 1 will enable the watchdog and will perform the watchdog notifications itself. Suitably configured services use systemd as the watchdog, and notify it regularly to be considered alive. In this way, the whole system is supervised, with the hardware supervising systemd, and systemd supervising the rest.
  • Systemd can be used used inside the initramfs (see dracut on Fedora), which means the same code serves an additional purpose, and there's less stuff to maintain. It can also hand control back to the initramfs at shutdown, making it possible to cleanly unmount the root filesystem even in very complicated, stacked setups.
  • Systemd is introspectable and easy to debug: you can use systemd-cgls to find out about running services, you can use systemctl status to learn a lot about an individual service. There is also a GUI for that which can be used by regular users.
  • Systemd is moving strongly in the direction of configuration-less system (i.e. “empty /etc/”). So /etc/fstab might be empty on a system with a GPT partition table, with partitions mounted in appropriate places based on their type id. This means that the configuration of the common one-machine-one-disk setup becomes even simpler.

Community

Systemd is a lively project with dozens of developers from various companies, including Red Hat, Samsung and Intel. It integrates contributions from even more individual contributors: to this date, 438 authors, with 63 having at least 10 commits. It can also be noted that two of the Debian maintainers have commit permissions.

  • Other distributions who switched to systemd were also confronted to a lot of controversy before the switch. After it became clear that rivers didn’t turn into blood, the complaints have only been sporadic.
  • Systemd’s upstream is very accommodating to distributors. They are taking a lot of Debian’s needs into account, even though it has not yet been decided to make it the default.
  • Systemd developers participate to a lot of conferences, they hold regular hackfests and always ask for feedback on their plans from downstreams (including Debian) before going ahead with them.
  • Systemd ships high quality documentation, including manual pages for all tools and configuration files, as well as design documents and administration tutorials.

Limitations to take into account

Portability

Systemd is not portable. This is a choice which was made, not without reasons, by the developers. If systemd is chosen for Linux (which we recommend), something else will have to be done for kFreeBSD.

  • It is theoretically possible to port Systemd to Hurd or ?OpenSolaris, provided someone volunteers to maintain it a la portable OpenSSH, but kFreeBSD has no credible replacement for control groups. (There are other portability issues, but cgroups is the most critical one.)

  • If another solution is picked for kFreeBSD (a good option being OpenRC), maintainers of packages that start services would need to maintain two files : a systemd unit, and an OpenRC runscript. However:
    • Maintaining both files is still much less work than maintaining a single SysV init script.
    • It should only concern packages useful to this architecture; most of them have not been tested for real. If we only retain daemons which matter on a file server or a firewall (kFreeBSD’s targets), it is a very small effort.

Compatibility

Although systemd has better SysV compatibility compared to upstart, there are some documented incompatibilities. They only affect a handful of packages in the whole Debian archive, but it is worth taking into account.

  • Interactive scripts (X-Interactive Debian extension) are not supported. Since they are usually used to ask passwords, systemd provides a framework to ask passwords instead.

  • Some of Debian’s cryptsetup extensions have not yet been implemented.

Comparison with other init systems

sysvinit + insserv

Sysvinit was never designed to cope with the dynamic/event-based architecture of the current Linux kernel. The only reason why we still use it today is the cost of a migration.

  • Sysvinit/insserv has very little C code, but the real code size must take shell scripts into account. These scripts come in huge numbers, contain bugs and are hard to maintain.
  • Debugging an init script is a tedious task. It is much faster to write a working systemd unit file than to debug an existing System V init script.
  • Sysvinit is insufficient for desktops, mostly because of missing features such as the D-Bus interfaces. But the real problems arise on big server setups, where Debian is losing ground because of its antiquated init system. On these systems, you need fine service management, process monitoring, reliable dependencies, complex device setups and proper event handling.

Upstart

Upstart was a huge improvement over sysvinit, and Debian should have switched to it years ago. Now that systemd is available and well-tested, however, it cannot sustain the comparison. Upstart suffers from an improper design which replaces dependencies by purely event-driven actions, and its process tracking is implemented using the wrong tool (ptrace). It would be hard work to provide all the features systemd integrates on top of upstart, and its community, limited in size by Canonical’s policy, seems neither willing nor able to develop them.

Architecture

  • Upstart turns the dependency chain upside down, by starting a service as soon as its dependencies are ready, instead of starting the dependencies of a service when the service itself should be started.

  • By treating dependencies as events (“start A if B has started” rather than the conventional “A requires B”), native upstart jobs are prone to introduce event loops which will block starting all services in the loop. This causes misunderstandings even for experienced administrators.

  • The use of ptrace is a fundamental design issue that prevents the use of upstart jobs for important daemons like avahi, apache or postfix.

  • Another consequence of upstart abusing ptrace(2) is the impossibility to debug the startup of a process started by the init system, or to run them in valgrind like systemd allows.
  • Socket activation is incorrectly implemented and only allows to pass a socket on startup, not e.g. to keep accepting connections while the daemon is restarting (what systemd allows).
  • Upstart cannot reliably track a process and this can lead to difficult to handle bugs. Some jobs may be stuck in start/killed state because of this.

  • Upstart job files are written using a specific syntax which is not purely descriptive and can include operands or shell scripts. Among other problems, this makes it impossible to parse the files with external tools.
  • Upstart’s compatibility with SysV init scripts is only partial. Jobs and SysV scripts are not treated the same by the init system (for example, SysV scripts are not monitored), which leads to obscure limitations such as the impossibility for an upstart job to depend on a script depending on a job.

Maybe as a consequence of these design limitations, and despite being used in two LTS releases, Ubuntu did little work to convert existing SysV init scripts. Most daemons are still started with the SysV init script. By switching to upstart, we would still have to write most job files, not even knowing whether we can replace all of our legacy SysV scripts.

Functionality

Most of the new functionality brought in with systemd (and described earlier) is not available when using upstart.

  • Some of these features can be emulated with upstart, but using shell scripts and/or external tools that make them less reliable and more complex to maintain.
  • Upstart does not implement the D-Bus services that exist on top of systemd. So far, the upstart community has only proposed to fork some of the services from systemd v204, but it means losing useful functionality and diverging from upstream in the long term. Even if they succeed in maintaining compatibility (presumably in a partial way and with delays), development of the APIs would still happen in systemd. Having Ubuntu as a proxy between Debian and actual development would make it hard for Debian and its users to participate in the further development and design of core Linux APIs.

Community

  • Upstart’s contributor license agreement dismisses it as software of choice for our core system. It may not be the only important piece of software in Debian with a copyright agreement, but there is no reason to add more trouble to the existing CUPS mess we already have.

  • It is often argued that Upstart’s developers are less hostile to a port to kFreeBSD. First, it excludes the idea of a port in the Debian systemd package; especially when at the same time, upstart’s proponents talk about the possibility of a fork to avoid the CLA issue. Second, the upstart port does not exist today. To be taken into consideration, it would require credible information that someone intends to do the (huge) porting work if upstart is chosen as default init system.
  • Once Red Hat has migrated to systemd, there will only be one major distribution left using upstart. Most (if not all) contributors are Ubuntu developers. The committer base is nothing comparable to that of systemd.
  • Of all other distributions who switched their default init system after systemd became available, none picked upstart. Gentoo chose OpenRC (and supports systemd as an alternative), all the other ones went on the systemd path.
  • Upstart is not very popular in Debian. Even before the recent changes in GNOME dependencies, the number of voting popcon users for systemd was approximately 30 times higher than that of upstart users.

Sysvinit + OpenRC

OpenRC brings improvements to sysvinit while remaining compatible and portable, which are its primary selling points. However, it remains merely a sysvinit on steroids. It still relies a lot on shell scripts, is not event-based, does not implement socket activation, does not integrate with D-Bus, nor implements most of the other features we are now expecting from a modern init system.

  • OpenRC does not provide the systemd D-Bus interfaces. Using it would require someone to write or adapt, and maintain, Debian-specific tools to provide these interfaces. Gentoo currently does not have any solution for that issue apart from maintaining systemd as an alternative for OpenRC.
  • One of the improvements of OpenRC over sysvinit is the (rudimentary and experimental) cgroups implementation, but the functionality will not be available on kFreeBSD.
  • OpenRC’s support for parallel boot is not production-ready, which is an important regression compared to insserv.

  • The OpenRC runscript format is not declarative. It is a mix of shell scripts and variable declarations. It makes it impossible to parse the format by external tools.

Rebuttals

From the upstart statement

  • The statement mentions that systemd is “hasty”. It is forgetting that several major releases of several distributions have already happened using systemd as default. It is no longer an experimental technology at all. Compatibility at upgrade time should not be a concern either, since the real outside interfaces (D-Bus, unit files, Debian configuration files) have always been stable (forward compatible) and will remain so. There will, most probably, be locked-in upgrades with udev from time to time, but it does not have any impact on the ability to upgrade systems.

  • The statement talks about systemd as being “greedy”, as in being incorrect to integrate all these features so closely to PID 1. First of all, more useful features are not a disease. Getting to the point: while it is true that a handful of trivial interfaces are not really related to systemd (and could be split out if needed), most of these features cannot be implemented without close integration to PID 1. It is not possible to split the system cgroups arbitrator from the process which starts services and sessions in cgroups. It is not possible to ensure the relation of a log to a service if you do not have awareness of how the service was launched. Et caetera.

  • The statement mentions Debian should “not find itself at the mercy of other parties who wish to dictate design to us.” These words sound a bit ironic, coming from proponents of a system which is solely developed and maintained by Ubuntu, and which would make Debian depend on a single company for a core package.
  • The statement affirms that systemd in other distributions is used “primarily in sysvinit compatibility mode”, which ironically is exactly how upstart is used in most Ubuntu packages, while Fedora already ships hundreds of systemd unit files.

Overall, we agree strongly with the upstart proponents on the idea that the choice for Debian’s init system should be made mostly on technical merit. It is clear to us that systemd is overwhelmingly better than any existing alternative anywhere the technical architecture is involved.

From the sysvinit statement

  • The statement mentions “traditional UNIX principles” which systemd would not adhere to. If you look at how traditional Unixes such as Solaris or BSD are architectured, they are very similar to systemd, actually: a large set of small utilities, each one with a defined purpose, but which are absolutely not interchangeable or replaceable.
  • The statement implies systemd benefits “mostly the GNOME desktop”, which is entirely untrue. While GNOME, and probably Xfce and KDE soon, use the interfaces implemented by systemd, they are not, by far, the primary targets of systemd, its development being more driven by server and embedded requirements.

Various other sources

The OpenRC statement has removed its systemd criticism, only keeping the vague “problems with Systemd have been debated a lot” statement. Since there is a lot of this criticism being spread by OpenRC or sysvinit advocates, it is still worth commenting on some incorrect, although widespread, beliefs.

  • There have been many comments on supposed lack of documentation on systemd’s internals or interfaces.
    • 10% of the source code (14 kLoC) are comments.
    • Each component has its detailed manual page.

  • Many discussions are centered on systemd-logind as in being the only problem to address by other proposals, wildly proposing seemingly easy-to-develop replacements.
    • Logind is only a small fraction of systemd’s added value. It is important for desktop environments, but systemd is for all systems.
    • Reimplementing logind is not hard because of supposedly missing documentation. It is hard because without systemd, you are missing the necessary features to implement this service.

  • Systemd is often said to be too big for the functionality of an init system.
    • However it is much more than init, and if you take into account all the functionality it can provide or replace, you’ll soon find out that it takes less lines of code than the alternatives, in a language (C) that takes less memory to execute.
    • Linux is mostly a monolithic kernel and runs an enormous amount of code in ring 0. Having ~2% of this amount in the init system, most of which is run outside PID 1, does not sound like gigantism.

  • There is a lot of criticism about systemd’s upstreams and potential lack of quality in our relationship with them. However, this collaboration already exists and is of excellent quality.
  • Many are complaining about the absence of shell scripts, quoting ease of administration or embedded systems.
    • Shell scripts are not easy to maintain or debug; systemd unit files are. For the handful of services that still need scripts, it is possible to execute them before or in place of the daemon’s startup.
    • Shell scripts should be entirely banned from embedded platforms, where they use memory and CPU for an interpreter irrelevant to the system’s purpose.

Overall, many of the arguments of those who don’t want to migrate away from sysvinit (with or without OpenRC) boil down to rumors or irrelevant political stances, while the decision to choose a default init system should be based on technical facts and the shape of the community.

Relevant documentation

Position statement maintainers

  • Josselin Mouette
  • With contributions from Zbigniew Jędrzejewski-Szmek, Ole Laursen, Vincent Bernat, Lennart Poettering, Ondrej Balaz, “uau”, Jordi Mallach, Emilio Pozuelo Monfort, Andreas Henriksson, Ansgar Burchardt, Laurent Bigonville, Nikolaus Rath, Tom Gundersen, Michael Stapelberg, Tollef Fog Heen, Martin Pitt, Moritz Mühlenhoff.