Differences between revisions 103 and 104
Revision 103 as of 2016-01-28 20:03:17
Size: 14092
Editor: ?GrantMcLean
Comment: Remove advice to mask udev service on Jessie containers
Revision 104 as of 2016-01-31 08:08:19
Size: 14508
Editor: TpO
Comment: added reference to solution of "Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted" problem
Deletions are marked like this. Additions are marked like this.
Line 311: Line 311:
 * if you encounter a "{{{Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted}}}", then you might consider [[https://github.com/debops/ansible-lxc/issues/15#issuecomment-87253770|this advice]]. The author of these lines here has '''not''' verified the accuracy of the proposed configuration settings - be it with respect to security or to other side effects, so please use your own judgement.

Translation(s): English - Français

(!) ?Discussion

Linux Containers (LXC) provide a Free Software virtualization system for computers running GNU/Linux. This is accomplished through kernel level isolation. It allows one to run multiple virtual units simultaneously. Those units, similar to chroots, are sufficiently isolated to guarantee the required security, but utilize available resources efficiently, as they run on the same kernel.

For all related information visit : http://lxc.sourceforge.net/

Full support for LXC (including userspace tools) is available since the Debian 6.0 "Squeeze" release.

You can also read some sub pages :


  • Install required packages

aptitude install lxc
  • Install optional packages

aptitude install bridge-utils libvirt-bin debootstrap

Prepare the host

On a Debian Jessie host, there is nothing to do.

For older releases:

Add this line to /etc/fstab. This is not necessary if libvirt-bin is installed as init.d/libvirt-bin will mount /sys/fs/cgroup automatically)

cgroup  /sys/fs/cgroup  cgroup  defaults  0   0

Try to mount it (a reboot solves an eventual "resource busy problem" in any case)

mount /sys/fs/cgroup
  • Optional: Enable memory cgroup support (CONFIG_CGROUP_MEM_RES_CTLR, 534964, builtin since 2.6.39-bpo60-2 for Squeeze). It is deactivated by default. You need to activate it with a kernel parameter. When using grub2, this can be easily done by setting GRUB_CMDLINE_LINUX="cgroup_enable=memory" in /etc/default/grub and running update-grub2.
    Same goes for swap+memory (CONFIG_CGROUP_MEM_RES_CTLR_SWAP), kernel parameter "swapaccount=1" - see http://lists.debian.org/debian-kernel/2012/01/msg00804.html .

Check kernel configuration :

# lxc-checkconfig 
Kernel config /proc/config.gz not found, looking in other places...
Found kernel config file /boot/config-2.6.32-5-amd64
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup namespace: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: missing
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

Above the lxc-checkconfig program is reporting "Cgroup memory controller: missing". If you want memory control via cgroups then you need to recompile the linux kernel (or simply add cgroup_enable=memory to the kernel command line on jessie or later).

RootFS creation

Debian 8 "Jessie"

LANG=C SUITE=jessie MIRROR=http://httpredir.debian.org/debian lxc-create -n debian8 -t debian

Alternatively you can use this command line:

lxc-create -n debian8 -t debian -- -r jessie

Debian 7 "Wheezy"

LXC installs correctly on "Wheezy" (including a working Debian template since 7.4).


lxc-create -n myvm -t debian

which will prompt you on what distribution to install.

Then adapt network configuration in /var/lib/lxc/myvm/config, e.g. to plug it on libvirt's bridge:

lxc.utsname = myvm
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = virbr0
lxc.network.ipv4 =
lxc.network.hwaddr = 00:1E:62:CH:NG:ME

Other templates can be downloaded, before 7.4 we recommended the one referenced on the LXC container mailing list:

lxc-create -n myvm -t debian-wheezy
# or for a 32-bit container:
linux32 lxc-create -n myvm -t debian-wheezy

Issues in Debian 7 "Wheezy":

Debian 6.0 "Squeeze"

  • In theory you may use any tool to create a rootfs (debootstrap, multistrap, rootstock) as long as you are sure, that the result will run inside a container.
  • For the first time and as an example, use the provided creation script of the lxc package.

cp -a /usr/lib/lxc/templates/lxc-debian /usr/lib/lxc/templates/lxc-squeeze
  • (lxc-debian is designed to install lenny, which is no longer accessible from the specified archive.) Edit lxc-squeeze, replacing "lenny" with "squeeze" and "dhcp-client" with "isc-dhcp-client"

lxc-create -n myvm -t squeeze -f /usr/share/doc/lxc/examples/lxc-veth.conf
  • Here, we use lxc-veth.conf config file

Setup networked containers

Start and stop containers

Notes/warnings on starting and stopping containers:

  • When you connect to a container console, lxc will let you know how to quit it. The first time you log in however, getty will clear the screen, so you'll probably miss this bit of information:

    Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself
  • If you're using screen and also use the Ctrl+a command prefix, type <Ctrl+a a q> to exit the console.

  • <!> When you start the container in foreground mode (without -d), there's apparently no way to quit the terminal (<Ctrl+a q> doesn't work). Make sure you start the containers in background mode with -d, unless you need to debug why a container didn't start.

  • lxc-halt yields telinit: timeout opening/writing control channel /run/initctl - instead, do a 'poweroff' inside the container, (this is bad advice - it powers off the HOST) and finish it off with lxc-stop. Alternative work-around: use lxc.cap.drop = sys_admin in the container config file.

Actual commands:

  • To start a container in the background and attached to the console at any time later run (by default, login/password is root/root):

    lxc-start -n myvm -d
    lxc-console -n myvm
  • To start a container in foregroup mode and stay attached to the console run (see warning above):

    lxc-start -n myvm
  • To halt a container cleanly by the containers initv-system (see timeout note above):

    lxc-halt -n myvm
  • To stop a container without proper halt inside the container:

    lxc-stop -n myvm
  • To have containers automatically started on booting the host, link their config file in /etc/lxc/auto/:

    ln -s /var/lib/lxc/mycontainer/config /etc/lxc/auto/mycontainer

    On hosts running jessie must be defined in config file /var/lib/lxc/mycontainer/config:

    lxc.start.auto = 1

Bind mounts inside the container

By default only the container's filesystem is mounted inside the container (even if on the host, /var/lib/lxc/mycontainer/rootfs has other mount points).

To mount another filesystem in the container, add to /var/lib/lxc/mycontainer/config:

lxc.mount.entry=/path/in/host/mount_point /var/lib/lxc/mycontainer/rootfs/mount_point none bind 0 0

and restart the container. The mount point will now be visible inside the container as well.

Both paths can be identical if necessary.

As of 2015-September-30 The recent security patches to fix CVE-2015-1335 have broken the use of absolute container mount points as shown above on some Debian derived systems. The use of relative container mount points still work and provide a workaround.

SO: for the near future you can use:

lxc.mount.entry=/path/in/host/mount_point mount_point none bind 0 0

instead of the suggestion above. NOTE that it is critical to have no leading "/" in the container mount point (making it a relative mount point).

Incompatibility with systemd

* The version in Wheezy (0.8.0~rc1-8+deb7u2) is not compatible with running systemd inside the container. See 766216 * The versions in both jessie and stretch support systemd in the container just fine for Debian guests. ** YMMV for other types of guests


Upgrading container from "Wheezy" to "Jessie"

When upgrading an lxc guest running "Wheezy" to "Jessie", the lxc VM will stop working, because at the time of writing (23.11.2014) systems will automatically be migrated to systemd. See 766233. This behaviour is being reviewed in 762194.


Switch back to sysv

If the VM was migrated to systemd automatically via an upgrade then you can switch back to sysvinit:

lxc-stop -n myvm               # stop the vm
                               # or, if that doesn't work use lxc-kill

# the next step requires the VM to be mounted at /var/lib/lxc/myvm/root

chroot /var/lib/lxc/myvm/root  # chroot into the vm
apt-get install sysvinit-core # reinstall old sysvinit

Alternatively you can try to start the container in the foreground and do the same via the container's console as described in section Debian 8 "Jessie"/testing.

Not letting your system be updated to systemd during the upgrade

Before upgrade, run:

apt-get install sysvinit-core

or run the following command in place of a usual dist-upgrade:

apt-get dist-upgrade sysvinit-core

Reconfiguring updated VMs

Note that the following recipe only works on hosts running jessie. It will not work on hosts still running wheezy.

Add the following to your container config:

lxc.autodev = 1
lxc.kmsg = 0

Do the following in the guest.

Adjust getty@.service:

cp /lib/systemd/system/getty@.service /etc/systemd/system
# Comment out the line ConditionPathExists=/dev/tty0 in the copied getty@.service

The udev service (which is a hard dependency of systemd in Jessie) won't run in a container, however the systemd config will detect that sysfs is mounted read-only and will automatically skip udev startup. Apparently this was not the case with earlier versions of systemd and this page used to advise using systemctl to mask the udev and systemd-udev services - this is no longer necessary and may cause problems later (see 812932).

Creating new "Jessie" VMs

Creating new Jessie containers should work without issue.



See also :

Known bugs and "got to know issues"

  • 600466 - "Respawning too fast" messages and can't connect to console due to missing tty(1234) nodes in generated container rootfs. Workaround: remove from container's /etc/inittab or start container in interactive mode and mknod -m 660 dev/tty1 c 5 1 for each required tty.

  • Some bugs that might apply to non-official containers - read the follow-ups for solutions.

  • "telinit: /run/initctl: No such file or directory" running lxc-halt?

    mknod -m 600 /var/lib/lxc/myvm/rootfs/run/initctl p

    and add "sys_admin" to the lxc.cap.drop line in /var/lib/lxc/myvm/config? See http://wiki.deimos.fr/LXC_:_Install_and_configure_the_Linux_Containers#telinit:_.2Frun.2Finitctl:_No_such_file_or_directory

  • 761197 - "systemd-journald eats CPU in lxc jessie container"
    As noted in the bug report, setting "lxc.kmsg=0" in "/var/lib/lxc/myvm/root" and removing "/dev/kmsg" inside the container seems to fix the problem.

  • if you encounter a "Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted", then you might consider this advice. The author of these lines here has not verified the accuracy of the proposed configuration settings - be it with respect to security or to other side effects, so please use your own judgement.

See also