Differences between revisions 39 and 40
Revision 39 as of 2013-11-13 00:10:53
Size: 17711
Editor: PaulWise
Comment: systemd capitalization fixes
Revision 40 as of 2013-11-26 06:28:26
Size: 17617
Editor: ?SteveLangasek
Comment: drop reference to sysvinit upstream, since there is now an upstream of sorts
Deletions are marked like this. Additions are marked like this.
Line 21: Line 21:
 * Upstart is covered by a [[http://www.canonical.com/contributors|copyright license agreement]], which contributors must sign to have their contributions included upstream. This is an issue for many Debian developers, who are unwilling to sign such an agreement (usually because of the asymmetry of the agreement in question) and object to core packages being covered by such terms that they will not contribute under. The position of the Debian maintainers is that, while contributions to upstream must be covered by the CLA, no such considerations attach to the Debian package; if Debian ever decides that upstream maintenance is inadequate, we can always fork upstart, since the code itself is GPL and would be no worse off than we are today with a sysvinit package that also has no upstream.  * Upstart is covered by a [[http://www.canonical.com/contributors|copyright license agreement]], which contributors must sign to have their contributions included upstream. This is an issue for many Debian developers, who are unwilling to sign such an agreement (usually because of the asymmetry of the agreement in question) and object to core packages being covered by such terms that they will not contribute under. The position of the Debian maintainers is that, while contributions to upstream must be covered by the CLA, no such considerations attach to the Debian package; if Debian ever decides that upstream maintenance is inadequate, we can always fork upstart, since the code itself is GPL.

This page documents the position statement of individuals who suggest that Debian should adopt upstart as the default init system. Before editing this page, please see the instructions on how debates work. If you are not the maintainer of a position statement, and have a suggestion or change to make, please contact the maintainer.

Upstart Position Statement

Upstart is a modern, well-tested, event-based init system. It is used on tens or hundreds of millions of servers, desktops, cloud instances and embedded systems throughout the world. Debian should adopt it as the default init system for the jessie release.

Pros

  • Upstart brings significant new functionality to init, using a simple, declarative job configuration format (described here) that removes the need to maintain init scripts written in shell as "configuration files". Changes to job configuration can be maintained in separate .override files, or as changes directly to .conf files that are straightforwardly diffable/mergeable on upgrades.

  • Upstart supports backwards-compatibility. It is straightforward to include an /etc/default/foo file in an upstart job if needed (though this is not the recommended configuration method where sysvinit compatibility is not required). Existing init scripts will continue to run. insserv dependency relationships between init scripts are respected, providing a smooth transition to upstart from sysvinit.

  • Upstart has been tested in battle. Two consecutive long-term supported releases of Ubuntu have shipped with native upstart job support, giving Ubuntu an opportunity to shake out all the bugs so that Debian doesn't have to. The work to integrate upstart support from Ubuntu into Debian is straightforward.
  • Upstart fixes bugs. There are many longstanding race conditions in the existing sysvinit startup sequence that cause problems in complex disk/network configuration scenarios. These problems have been solved in the upstart rollout on Ubuntu, Debian will not have to reinvent the wheel.
  • Upstart welcomes ports. Lack of support for non-Linux ports has been a concern in all of the discussions about changing Debian's default init system. Upstart upstream has expressed their interest in having ports to Hurd and kFreeBSD, so that a decision to adopt Upstart does not mean maintaining compatibility with two init systems in Debian indefinitely.
  • Upstart is elegant. The code is easy to read, making it easier to debug and modify. It uses libnih, the best C utility library around (beats the pants off of glib for code clarity by leveraging current C standards and relevant gcc extensions).
  • Upstart is tested code. There is a thorough upstream test suite that is run at package build time. The test suite is twice the size of the code itself. No other init system can credibly claim to have had the same degree of testing. The code has also undergone extensive real-world testing in Ubuntu, as the sole init system for several years, and has held up quite well to user demands.
  • Upstart is actively maintained. Canonical continues to develop upstart as an active upstream project, responding to the evolving needs of its users.

Cons

  • Upstart is covered by a copyright license agreement, which contributors must sign to have their contributions included upstream. This is an issue for many Debian developers, who are unwilling to sign such an agreement (usually because of the asymmetry of the agreement in question) and object to core packages being covered by such terms that they will not contribute under. The position of the Debian maintainers is that, while contributions to upstream must be covered by the CLA, no such considerations attach to the Debian package; if Debian ever decides that upstream maintenance is inadequate, we can always fork upstart, since the code itself is GPL.

Upstart vs. SysV init

We've gotten a lot of mileage out of sysvinit in Debian, but its limits have been showing for a while - indeed, it was these limits that led to upstart being written in the first place.

  • Sysvinit lacks service supervision. While /etc/inittab provides this capability, management of /etc/inittab is quite restrictive. Upstart eliminates the need for complicated PID file handling for all services. There are bolt-ons that allow you to do service supervision on top of sysvinit, such as runit, but it's clear that these are bolt-ons.

  • Sysvinit does not track dependencies between services. Insserv/startpar provides this on top of sysvinit, but this is again very much a bolt-on, and only handles dependencies at boot/shutdown time (i.e., during runlevel changes) and can't handle any complicated service interdependencies at runtime. Upstart expresses this information in the native service configuration format, clearly and concisely.
  • Sysvinit requires complex shell scripts for each service. While some of the complexity has been abstracted out into common helpers (lsb-functions; start-stop-daemon), having to represent each service's start/stop handling as a program is a severe handicap. Upstart jobs are simple, declarative, easy to maintain, and easy to modify locally as needed. They also eliminate the longstanding nuisance of update-rc.d reactivating services behind the administrator's back on upgrade.
  • Sysvinit is linear. It stopped being a good fit for boot management on Debian the moment Debian adopted udev. There are many race conditions that persist in Debian today when booting with sysvinit, and although these may be fixable, the complexity for fixing them with sysvinit is very high. We're better off switching to an init system that's designed to work together with the event-based kernel and udev.

There is no reasonable justification for Debian continuing to inflict sysvinit's limitations on users today.

Upstart vs. systemd

In terms of overall feature uplift of the init system itself, there is really rather little to distinguish upstart from systemd. Both would represent a huge step forward for Debian over sysvinit. If Debian did not adopt upstart, systemd would certainly be my second choice.

But despite the init systems being comparable at the feature level, there are reasons that I think upstart is a better fit for Debian than systemd.

  • systemd is Linux-specific. Upstream has been very clear that they are not interested in accepting porting patches to enable systemd to be used with kernels other than the Linux kernel. If Debian wishes to continue to support non-Linux ports, this significantly reduces the benefit that maintainers would gain from having a simpler init format to support.
  • systemd is hasty. Debian is late to the party when it comes to adopting a better init system, but systemd is evolving rapidly and does not have a long track record of backwards-compatibility that it can point to. While systemd is unlikely to break any interfaces that upstream has promised to maintain, I think there is a risk that newer dependencies will be driven by the version requirements in Fedora and RHEL, leaving Debian to fend for itself with respect to sorting out rocky transitions on upgrade - not unlike the problematic udev transitions of the past. Upstart is committed to maintaining compatibility across stable releases of Debian, just as it currently does between Ubuntu LTS releases. While upstart seeks to leverage newer kernel features where this makes sense, we are committed to having sane upgrade paths and not depend on such kernel features prematurely.
  • systemd is "greedy". Most of the recent arguments about why it's dangerous to adopt upstart instead of systemd center around features that are being built into systemd in a manner that can't be separated out (e.g., cgroup management in PID 1). There is an advantage to the implementor to put these features in-process in init, because it ensures early availability with no concerns about startup ordering at boot, but it commits downstreams to a monolithic design with respect to parts of the system architecture which are not settled questions in the wider ecosystem. Debian should take a principled position regarding its future architecture, and not find itself at the mercy of other parties who wish to dictate design to us.
  • systemd is unproven. Contrary to the ongoing marketing claims that everyone is adopting systemd, relevant practical experience with systemd is not very widespread. Most distributions which have adopted it to date are focused on a particular market segment (e.g., Fedora), or have much smaller userbases, or are continuing to use it primarily in sysvinit compatibility mode. This means that many of the corner case conditions in systemd have yet to be identified. Adopting systemd as the default implies that Debian will have to do much of the work to address the long tail of integration issues. By comparison, upstart has a long track record in Ubuntu which translates directly to Debian due to the similarity of the distributions' design and packaging. While it's possible that not all corner cases have been addressed in Ubuntu either, the vast majority of use cases have had the bugs ironed out, and those that haven't been are sure to be documented in the Ubuntu bug tracker.

Rebuttals

  • The systemd position statement incorrectly identifies Ubuntu bug #406397 as a fundamental design issue due to the use of ptrace. This is a longstanding bug, but it is not fundamental to the use of ptrace. It would be a good idea to obsolete the use of ptrace in the future for its own sake, but this bug exists primarily because upstart provides more granular definitions of service readiness that are not available in systemd. In any case, this is just a bug; and the fix for this bug is in progress.

  • The systemd position statement cites a comment from the original author and former maintainer of upstart as evidence that upstart is hostile to non-Linux ports. Scott James Remnant is no longer involved in upstart development; this is self-evidently outdated information.
  • The systemd position statement asserts that you can't attach gdb to a process run by the init system. This is not true, and can only be speculation on the part of the systemd proponents: ptrace is only used during service startup, after which upstart detaches from the process (relying on ordinary waitpid() for notification of service exit). This means that for all intents and purposes, by the time you would actually be in a position to try to attach to the process with gdb, upstart will no longer be tracing it. For valgrind, it's true that this would be less straightforward to do as part of an upstart job. If there were widespread demand for solving this, it wouldn't be difficult; but this doesn't seem like something that has much practical impact. It's unlikely that someone using valgrind for debugging will need to debug via an upstart job or systemd unit, as opposed to running the service directly.
  • The systemd position statement claims that the Upstart dependency tree is "upside down" by starting services when their dependencies are ready, instead of starting the service's dependencies when the service should be started. This ignores the reality that a Linux system is fundamentally event-driven. It is meaningless to suggest that all the dependencies should be auto-started, when some of these dependencies will be tied to hardware events that the init system does not control. This merely repeats the existing flaws with sysvinit.
  • The systemd position statement accurately points out that the event model makes it possible to create a config that deadlocks the start of a particular service. However, it's wrong in suggesting that this problem is unique to upstart. Dependency loops can be written in any init system, and the solution for avoiding these is to have robust config checkers that detect these loops. The initctl check-config and initctl2dot commands provide the basis of this configuration checking for upstart.

  • The systemd position statement claims that upstart compatibility with sysvinit scripts is "only partial" because they are not treated as first-class jobs. But this is very similar to how they're treated in systemd, as well; the autogenerated systemd units for sysvinit scripts all assume that the service is a forking service, and will not respawn the service automatically on exit. For both systems, native jobs are handled better than sysvinit scripts (which is why we're discussing adopting a different init system in the first place). Also, the claim that it's impossible "for an upstart job to depend on a script depending on a job" is only half right: startpar in Debian is patched to interface with upstart, so that any sysvinit script's LSB dependency will be satisfied by the corresponding upstart job, and it's a trivial matter to integrate with /lib/lsb/init-functions to make LSB init scripts also generate upstart jobs. This isn't a design defect in upstart, it's merely an implementation detail to be sorted out in Debian if Debian decides this is needed. It was never an issue in Ubuntu because conversions to upstart were done in dependency order, but it's certainly reasonable to add support in Debian for out-of-order converting of packages to upstart.
  • Most packages in Ubuntu do still use sysvinit scripts rather than upstart jobs. But these are init scripts for standalone services with small install bases, not core elements of the system (filesystem daemons, network configuration scripts) with complex interdependencies. The packages in question have no upstart job because they are in sync with Debian and have no Ubuntu maintainer per se. The notion that these scripts will pose a problem when converting to upstart jobs is simply FUD. I can well believe that the systemd advocates don't know if it will be possible to convert all sysvinit scripts to upstart jobs, but if so, that's because they've chosen not to take the time to understand upstart.
  • The systemd position statement simultaneously claims that the D-Bus interfaces "are stable and will remain so" and that upstart is at a disadvantage if it forks the D-Bus services from systemd v204 because "development of the APIs would still happen in systemd." They can't have it both ways. Either the interfaces are stable, and a fork is low-risk; or the interfaces are not, and relying on them is high-risk whether provided by a fork or by systemd itself.
  • Finally, the systemd position statement attacks upstart on the basis of its community. How many committers do they believe an init system needs? I don't think there's any question that the upstart developer community is sufficient to make it a viable project for the foreseeable future. And if this isn't a matter of the project's viability, then these aren't technical arguments at all; in fact, they're arguments for abdicating *any* technical decision, and instead going along with the majority. "Ten thousand distributions can't be wrong"; except that none of those distributions are Debian, and Debian is about doing what is technically best, not just what is popular.

It's true that upstart is not a very popular package in Debian. But to imply that this is because the package lacks appeal with Debian users ignores the fact that the upstart packaging has given precedence to technical correctness at the expense of its popularity: it currently conflicts with an Essential: yes package, which deters people from trying it, or having tried it, to keep it installed. This Conflicts is present because we should be driving towards the goal of having one init system installed on users' machines, not two; and while in the short term this means upstart has seen fewer early adopters in Debian, this should not trick the TC into believing that upstart is unfit.

systemd upstream paints a utopian vision where upstream services all ship with systemd unit files that Just Work everywhere (despite the fact that even trivial failures to comply with Debian policy in systemd units submitted upstream by Red Hat employees result in non-portable systemd configurations). This is a vision informed by a belief that distributions are "just" packaging. It downplays the major significance of the integration work that distributions do, and justifies treating distributions as passive consumers who don't need to be consulted before making sweeping changes to their core architecture. This vision serves systemd upstream well. It serves Debian and Debian's users less well.

We are having this debate because Debian's commitment to technical excellence gives it a pivotal position in the greater Linux community. Debian should not be swayed by fearmongering arguments that if it doesn't pick systemd, it will be all alone. Others always pay attention to the decisions Debian makes, and while Debian may have sat on the sidelines of the init decision up to now, that doesn't mean other distributions won't be paying attention when we do make a decision.

Position Statement Maintainers

  • ?SteveLangasek

Comments