Linking to an process for systemd unit file debugging and fixing grammar mistakes in the document.
Fix missing closing bracket
|Deletions are marked like this.||Additions are marked like this.|
|Line 373:||Line 373:|
|* [[https://containersolutions.github.io/runbooks/posts/linux/debug-systemd-service-units/|thorough runbook for Debugging Systemd Services]||* [[https://containersolutions.github.io/runbooks/posts/linux/debug-systemd-service-units/|thorough runbook for Debugging Systemd Services]]|
systemd - system and service manager
systemd - system and service manager
- Basic usage
- Installing and Testing
- Bugs and Bug-Tracking-Systems
- Known Issues and Workarounds
- Where to get help?
- Installing without systemd
- Debian Resources
- Other Resources
systemd is a system and service manager for Linux. It is the default init system for Debian since DebianJessie. Systemd is compatible with SysV and LSB init scripts. It can work as a drop-in replacement for sysvinit. Systemd
- Provides aggressive parallelization capabilities
- Uses socket and D-Bus activation for starting services
- Offers on-demand starting of daemons
- Implements transactional dependency-based service control logic
- Tracks processes using Linux cgroups
- Supports snapshotting and restoring
- Maintains mount and automount points
Systemd runs as a daemon with PID 1.
Systemd tasks are organized as units. The most common units are services (.service), mount points (.mount), devices (.device), sockets (.socket), or timers (.timer). For instance, starting the secure shell daemon is done by the unit ssh.service.
Systemd puts every service into a dedicated control group (cgroup) named after the service. Modern kernels support process isolation and resource allocation based on cgroups.
Targets are groups of units. Targets call units to put the system together. For instance, graphical.target calls all units that are necessary to start a workstation with graphical user interface. Targets can build on top of another or depend on other targets. At boot time, systemd activates the target default.target which is an alias for another target such as graphical.target.
Systemd creates and manages the sockets used for communication between system components. For instance, it first creates the socket /dev/log and then starts the syslog daemon. This approach has two advantages: Firstly, processes communicating with syslog via /dev/log can be started in parallel. Secondly, crashed services can be restarted without processes that communicate via sockets with them losing their connection. The kernel will buffer the communication while the process restarts.
Please see the upstream systemd page for more information.
systemctl is the main tool used to introspect and control the state of the "systemd" system and service manager. You can use systemctl for instance to enable/disable services permanently or only for the current session. Refer to the systemctl(1) manpage for more details.
Getting information on system status
Show system status:
$ systemctl status
List failed units:
$ systemctl --failed
List installed unit files:
$ systemctl list-unit-files
List all running services:
Activates the service "example1" immediately:
# systemctl start example1
Deactivates the service "example1" immediately:
# systemctl stop example1
Restarts the service "example1" immediately:
# systemctl restart example1
Shows status of the service "example1":
# systemctl status example1
Enables "example1" to be started on bootup:
# systemctl enable example1
Disables "example1" to not start during bootup:
# systemctl disable example1
Creating or altering services
Units are defined by individual configuration files, called unit files. The type of the unit is recognized by the file name suffix, .mount in case of a mount point. Unit files provided by Debian are located in the /lib/systemd/system directory. If an identically named local unit file exists in the directory /etc/systemd/system, it will take precedence and systemd will ignore the file in the /lib/systemd/system directory. Some units are created by systemd without a unit file existing in the file system.
System administrators should put new or heavily-customized unit files in /etc/systemd/system.
For small tweaks to a unit file, system administrators should use the "drop-in directory" feature (documented in systemd.unit(5)).
Start by determining the canonical systemd service name (e.g. ssh.service, not an alias like sshd.service). We'll use "name.service" for this example.
- Create the directory /etc/systemd/system/name.service.d
- Create files inside this directory ending with a ".conf" suffix. For example, /etc/systemd/system/name.service.d/local.conf
- Each file should contain the section headers and section options to be overridden, using the same format as unit files.
For more information about writing your own services, see systemd/Services.
Installing and Testing
systemd was included in Debian wheezy as a technology preview. Please make sure that you are using Debian jessie or newer to get a recent version of systemd.
To install systemd run:
# apt-get update # apt-get install systemd
This will install the systemd packages but will not configure systemd as your init system.
Configuring for testing
To test systemd before switching to it by default, you can add the following boot parameter to the kernel:
This can be done in the grub menu for a single boot - press "e" in the grub menu and add this to the kernel line. For example, depending on the options required for your particular system, it might look something like:
linux /vmlinuz-3.13-1-amd64 root=/dev/mapper/root-root init=/lib/systemd/systemd ro quiet
If PID 1 is systemd then your system is running with systemd.
Configuring as default
In order to use systemd you should also install systemd-sysv which provides the symlinks links for /sbin/init. It is recommended to run this when already running under systemd, as described in the previous section.
# apt-get install systemd-sysv
In order to boot your system with the newly installed systemd, simply reboot.
If you run a self-compiled kernel, make sure you have 2.6.39 or newer and enable the following options:
* CONFIG_DEVTMPFS=y * CONFIG_CGROUPS=y * CONFIG_AUTOFS4_FS=[y|m] * CONFIG_IPV6=[y|m], optional, but highly recommended * CONFIG_FANOTIFY=y, optional, required for systemd readahead. available in Linux kernel >= 2.6.37.
For an up-to-date list, see section "REQUIREMENTS" in the upstream README file.
In some cases units enter a failed state. The statuscommand can be used to find out some details:
$ systemctl status <UNITNAME>
Failed units can be manually cleared out:
# systemctl reset-failed
systemd hangs on startup or shutdown
Sometimes it is necessary to investigate why systemd hangs on startup or on reboot/shutdown.
Solution #0: Remove "quiet" from Kernel command line (so called "cmdline" or "grub line")
Solution #1: Increase verbosity via cmdline: Add "systemd.log_target=kmsg systemd.log_level=debug"
Of course you can have a "temporary" persistent solution:
[ /etc/default/grub ] GRUB_CMDLINE_LINUX="systemd.log_target=kmsg systemd.log_level=debug" <--- Add here (by uncommenting you can easily switch to debug) # update-grub
Solution #2: Increase verbosity via /etc/systemd/system.conf
LogLevel=debug <--- Uncomment this line and use "debug" (default: commented and "info") LogTarget=syslog-or-kmsg <--- Uncomment this line (default: commented)
Solution #3: Boot an emergency shell: Add systemd.unit=rescue.target or just 1 (the number one) to the kernel command line.
Solution #4: Enable the debug shell: Run systemctl enable debug-shell.service. (You can do this in a chroot environment after booting a rescue system.) This starts a root shell on TTY 9.
HINT: "man systemd" and "man systemd-system.conf"
HINT: Extensive debugging information about systemd is on this FreeDesktop page.
HINT: How to check Kernel command line parameters/options?
# cat /proc/cmdline
"Set log level. As an argument this accepts a numerical log level or the well-known syslog(3) symbolic names (lowercase): emerg, alert, crit, err, warning, notice, info, debug."
HINT: Keep a copy of /sbin/init from sysvinit package in case of rescue (so you can use init=/sbin/init.sysvinit in cmdline)!
# cp -av /sbin/init /sbin/init.sysvinit <--- Before installing systemd-sysv package
Kernel debug without systemd debug in Jessie
Using the old "debug" kernel parameter in Jessie will turn on systemd debug logging as well as kernel debug logging. To get the old behaviour, do not use "debug", instead use the kernel parameter "loglevel=7".
Bugs and Bug-Tracking-Systems
- For known bugs please see topic "Known Issues and Workarounds"
Known Issues and Workarounds
Shared bind mounts
The default behavior of bind mounts changes under systemd. The Linux kernel makes bind mounts of anything below / PRIVATE. Systemd changes this to SHARED.
Thus, when you do this:
mount --bind / $CHROOT mount --bind /dev/ $CHROOT/dev umount $CHROOT/dev
then /dev will be unmounted in your base/parent system as well!
What you can do now instead, is to:
mount --bind --make-rslave / $CHROOT mount --bind --make-rslave /dev/ $CHROOT/dev
this will propagate mount changes (also mount options) in the base/parent system into the $CHROOT but not from the $CHROOT back to the parent.
SSH session doesn't cleanly terminate on reboot/shutdown
If you happen to reboot/shutdown remote machine over ssh you may find out that your session isn't terminated properly, leaving you with the non-reacting terminal until long a timeout has been reached. There was bug 751636 about it. A workaround to this problem was to install:
apt-get install libpam-systemd
which would terminate the ssh session before the network was dropped. Please note, that that would require PAM to be enabled in sshd.
Missing startup messages on console(tty1) after the boot
With systemd console(tty1) is handled differently and if you used to check it to see how did your boot go now you'll see only couple of non-informative lines.
To be able to get full transcript of the system boot on your console you need to perform two steps.
1. Add to the kernel options systemd.show_status=1, for example via /etc/default/grub:
and run update-grub2.
2. Create file /firstname.lastname@example.org/noclear.conf with the content:
to disable clearing of the terminal on getty invocation.
Virtual and serial console changes
Those used to change inittab to enable/disable virtual or serial consoles will notice that that file is gone from clean installs. This is all managed through systemd directly now. For example, you can enable a serial console on COM1 with:
systemctl enable serial-getty@ttyS0.service systemctl start serial-getty@ttyS0.service
However, it is generally preferable to add console=ttyS0 on the kernel commandline, since this also enables kernel output on reboots. This is done by adding the following to /etc/default/grub:
... and running update-grub. This will take effect only on the next reboot, however.
Because it manages user sessions (taking over the role of X or other components), systemd may slightly change how processes survive a logoff. By default, when X shuts down all processes should exit and systemd will clean up the session, but there are some corner cases where certain processes don't cleanup after themselves properly.
You can configure how systemd manages leftover processes with the KillUserProcesses= parameter in logind.conf. By setting this to yes, processes will be forcibly killed when the session terminates. Note that this will break tools like screen or tmux, unless they are configured to run under a distinct user@.service unit and if enable-linger is set to yes in loginctl. A simple way to do this on the fly is to run the program in a "transient scope", using systemd-run:
systemd-run --scope --user screen
Now, normally sessions should cleanup after themselves, and such issues should be fixed without having to revert to the KillUserProcesses=yes sledgehammer. A good way to list the affected processes is to group them by "control group", with the systemd-cgls command:
Some known misbehaving applications:
Where to get help?
Systemd is a young project with a strong emphasis on solving problems in a distribution agnostic manner.
mailing-list @ https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Debian specific channels include
Several other distributions are using systemd
Installing without systemd
Jessie installs systemd by default on new installs. Should one desire to install without systemd, i.e use sysvinit-core instead (old sysV5 init), it is possible to use preseed to replace systemd with sysvinit at the end of the install (This probably won't work if selecting one of the desktop environments that require systemd specific features however). If using a preseed file already, just make sure to set the preseed value
preseed/late_command="in-target apt-get install -y sysvinit-core"
If not using a preseed file, this can be added to the boot arguments instead by hitting TAB at the boot menu on the desired entry and appending the above preseed line at the end of the boot command.
There may still be a few bits of systemd installed, but at least init itself is not systemd and cleaning up any remaining pieces should not be too hard.