This page explains troubleshooting of using Linux Containers (LXC) on Debian hosts operating with systemd's Control Group v2 unified hierarchy (i.e. booted with the systemd.unified_cgroup_hierarchy=1 kernel command-line argument). Starting with Linux kernel 4.5 (Debian Bullseye or later), cgroups v2 is now the default. The content of this page is verified with lxc (1:3.1.0+really3.0.3-8) on Debian Buster and lxc (1:3.1.0+really3.0.4-2, 1:4.0.2-1~1 and, 1:4.0.6-1~1) on Debian Bullseye. Discussion on this page
Unified cgroup hierarchy is coming
Systemd has three kinds of cgroup hierarchies, namely legacy, hybrid and unified. The unified hierarchy consists only of cgroup version 2. Traditionally, Debian has used the hybrid hierarchy, but as discussed in 943981 the unified hierarchy will likely become the new default hierarchy. This requires some adjustments to LXC container configuration files on hosts booted with the unified CGroup hierarchy, which are explained here. Advantages of the unified hierarchy are explained at The current adoption status of cgroup v2 in containers.
LXC containers started by root
Removing access control by CGroup V1 device controllers
The unified CGroup hierarchy does not have CGroup V1 device controllers. LXC container config files often have access controls of device files by using CGroup V1 device controllers, by using lxc.cgroup.devices.allow = and lxc.cgroup.devices.deny =. To start an LXC container, we have to remove those access control settings by adding
lxc.cgroup.devices.allow = lxc.cgroup.devices.deny =
to the end of a container config file, e.g. /var/lib/lxc/container/config. The above change becomes unnecessary from version 4.0.2-1~1.
Start container's systemd in the unified cgroup hierarchy
Most of Linux distros use systemd in the hybrid hierarchy. If container's systemd tries to start in the hybrid hierarchy in an LXC container, it fails to start. To avoid such a failure, we have to add
lxc.init.cmd = /sbin/init systemd.unified_cgroup_hierarchy=1
to a container config file. lxc.init.cmd is necessary only when container's /sbin/init is systemd with the hybrid hierarchy.
When LXC version is 4.0.0 or later, alternatively we can also use
lxc.mount.auto = cgroup:rw:force
By the above line, LXC lets systemd in a container choose the same CGroup hierarchy in a container as the host.
LXC containers started by non-root
Assume that preparation of unprivileged containers has been done. LXC needs a CGroup directory that can be manipulated by LXC, which was traditionally prepared by libpam-cgfs. libpam-cgfs no longer works and becomes unnecessary in the unified hierarchy as 946170.
When lxc-start or lxc-execute is run for lxc (1:4.0.6-1), we can use them, for example, as
systemd-run --user --scope -p "Delegate=yes" lxc-start -n container-name
Previous versions of LXC required as
systemd-run --user --scope -p "Delegate=yes" lxc-start -F -n container-name
otherwise we get error message reported at https://github.com/lxc/lxc/issues/3221. The above call does not work reliably if you want to start the container in the background. You cannot simply omit the -F parameter. Start the container in the background with
systemd-run --user -r -p "Delegate=yes" lxc-start -F -n container-name
See: https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1716279.html
Known Debian problems with Debian package 4.0.2-1~1
lxc-checkpoint does not work 958167
AppArmor does not work for unprivileged containers started by a non-root user 958158 and needs lxc.apparmor.profile = unconfined (This is NOT a bug but a feature).
Known Debian problems of LXC on host Linux with systemd.unified_cgroup_hierarchy=1
lxc-checkpoint does not work 947335
lxc-checkconfig gives incorrect warnings 946172
libvirt-daemon-driver-lxc: internal error: Unable to find 'devices' cgroups controller mount 947984