Translation(s): English - Français

(!) Discussion


Linux Containers (LXC) provide a Free Software virtualization system for computers running GNU/Linux. This is accomplished through kernel level isolation using cgroups (control groups) and namespaces. It allows one to run multiple virtual units simultaneously. Those units are similar to chroot but, isolated and utilize available resources efficiently, as they run on the same kernel.

Visit https://linuxcontainers.org/ for all related information.

Full support for LXC (including userspace tools) is available since the Debian 6.0 "Squeeze" release, the newer LXD tooling packaging is still a work in progress.

You can also read some sub pages :

Supported versions of LXC

There are two LTS versions of the LXC userspace tools currently supported by upstream:

LXC 1.0 is available in Jessie. LXC 2.0 is available in Stretch and Jessie Backports. LXC 3.0 is available on Buster. And LXC 4.0 is available on experimental.

There are also short term support releases

When looking for documentation, howtos and tutorials, please check which LXC version they apply to, as things might have changed. The 2.1 release, for example, changes the configuration file structure in several ways.

The rest of this page will try to distinguish between information that applies to LTS versions and that which applies to only to one of them.

Installation

Typically required packages are lxc, debootstrap and bridge-utils (the latter two are recommended by LXC package). libvirt-bin is optional.

Optionally:

If you want lxc to run unprivileged container(s), the package requirements are slightly different.

apt-get install lxc libvirt0 libpam-cgfs bridge-utils uidmap

(Note: libpam-cgfs is unnecessary if host Linux uses puer CGroup V2.

Networking

Debian's packages do not ship any default network setup for containers (/etc/lxc/default.conf contains lxc.network.type = empty).

If you want to have network in your containers (and you usually do), you will have to either change the global default or configure each individual container. You will also probably will have to setup a bridge, firewall and maybe DHCP (see below for details how to do this).

Please note that most container templates configure the first interface to use DHCP by default.

Stretch ships with a new major release of LXC which also includes a helper for easier networking setup called lxc-net. lxc-net allows you to set up a simple bridge for your containers, providing a DHCP-and-NATed IPv4 network. IPv6 support currently (in Stretch) requires manual configuration.

For complete manual setup without the convenience of lxc-net, see the networking section below.

Caveat on internet documentation:

There is much conflicting documentation due to differing versions. As a quick overview check out the Networking Essentials section down below. This wiki may also be outdated. When using lxc-net, only the following minimal changes are required to get a networked container up and running in Stretch

Essentials

Typically containers can be given their own hardware device (from the host, phys) or can have a virtual device (veth) that is either put directly on a bridged network device that it shares with the host (uses the same DHCP server and addresses as the host) or put on a bridged network device that is masqueraded to the outside world and that is given an internal subnet.

The first case (phys) requires a physical device and is therefore not often used. It gives the container an address on the host's subnet as the second case.

The second case (veth with host-shared bridge) turns the host's ethernet device into a bridge (makes it part of a bridge) and allows the container access to the external network allowing it to acquire a DHCP address on the same network that the host is on.

The third case (veth with independent bridge) is the use case of the lxc-net package (utility, service) as part of lxc version 2.0 and implies the use of a masqueraded subnet (e.g. 10.0.3.0 as is the default for both lxc-net, the above mentioned patch and this documentation) on which the host takes address 10.0.3.1 and any container can takes IP between 10.0.3.2 to 10.0.3.255.

Network setup in buster

Edit /etc/lxc/default.conf and change the default to this:

lxc.net.0.type = veth
lxc.net.0.link = virbr0
lxc.net.0.flags = up
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1

create the network bridge:

$ sudo apt-get install -qy libvirt-clients libvirt-daemon-system iptables ebtables dnsmasq-base
$ sudo virsh net-start default
$ sudo virsh net-autostart default

Destroy and existing containers and create them again.

Minimal changes networking in “stretch”

These are system-wide changes executed prior to creating your container.

  1. Create /etc/default/lxc-net with the following line:

    • USE_LXC_BRIDGE="true"

      Missing configuration is sourced from /usr/lib/x86_64-linux-gnu/lxc/lxc-net which contains the default values which means your bridge will be given a subnet of 10.0.3.0. You can change these values if you want in the /etc/default/lxc-net file. There is an Ubuntu patch to /usr/lib/x86_64-linux-gnu/lxc/lxc-net (or /usr/lib/<architecture/lxc/lxc-net) available at its packaging on Github called 0001-Allocate-new-lxcbr0-subnet-at-startup-time.patch that will automatically configure /etc/default/lxc-net with a subnet that is available on your system; typically 10.0.3.0 would just be available. It would pick any free subnet in the range of 10.0.x.0 going up with 10.0.3.0 as the starting point. This is done on system boot in case /etc/default/lxc-net is missing. So in order to use it, you must then delete this file first.

      For other purposes, see /SimpleBridge#Using_lxc-net for values you can add yourself.

  2. Edit /etc/lxc/default.conf and change the default

    • lxc.network.type = empty
      to this:
      lxc.network.type = veth
      lxc.network.link = lxcbr0
      lxc.network.flags = up
      lxc.network.hwaddr = 00:16:3e:xx:xx:xx
      This will create a template for newly created containers.
  3. Run sudo service lxc-net restart.

  4. Newly created containers now have the above configuration. This means they will be using the lxcbr0 bridge created by the lxc-net service.

This bridge is where your containers attach themselves to. Your bridge is now automatically created (at boot-up) and your newly created containers are configured to use this bridge. Existing containers can be configured by using the above configuration by editing your /var/lib/lxc/<container>/config file.

This is the same setup as which is created in the LXC/SimpleBridge page that contains some good default values for this kind of setup. The Host device as bridge section contains an alternate setup which employs a bridge created out of the main network device of the host system, as detailed below. This is not something that can be created by lxc-net but it is something you could use if you do not want Masquerading to take place and you want your containers to be on the external network.

Creating containers

In this step your container is downloaded using debootstrap and a mimimum Debian system is installed on your rootfs location (/var/lib/lxc/<container>/rootfs). After this, your container is ready to be run, it is already completed.

Rootfs location - along with many other settings - can be configured per container (after container is created) if required.

lxc-create -n <name> -t debian -- -r stretch

The -r stands for "release". You can also install other releases. The -r is a parameter that is passed to Debian's LXC script (template). It causes Stretch to be downloaded as the mimimum "debootstrap" Debian system.

<name> is the name you give your container. It can be anything, as you like.

Alternatively you can specify the language (locale) as required(?) and additionally you can specify the mirror to use for debootstrap in this way:

LANG=C SUITE=stretch MIRROR=http://httpredir.debian.org/debian lxc-create -n debian9 -t debian

This also passes "stretch" as an environment variable instead of as a parameter to the script (template). Scripts and templates are found in /usr/share/lxc/templates/.

External mounts inside the container

By default only the container's filesystem is mounted inside the container (even if on the host, /var/lib/lxc/mycontainer/rootfs has other mount points).

To mount another filesystem in the container, add to /var/lib/lxc/mycontainer/config:

lxc.mount.entry=/path/in/host/mount_point mount_point_in_container none bind 0 0

Another bind mount example:

# Exposes /dev/sde in the container
lxc.mount.entry = /dev/sde dev/sde none bind,optional,create=file

To mount in another filesystem (for example LVM) to a container mount point

lxc.mount.entry = /dev/mapper/lvmfs-home-partition home ext4 defaults 0 2

NOTE that it is critical to have no leading "/" in the container mount point (making it a relative mount point).

Mounts in unprivileged containers

When a container is unprivileged, UID or GID of a mounted device has to be root in the container. To make sure this, do chgrp 100000 /dev/nvidiactl etc. on the host (assuming GID 100000 is container's root group).

On the other hand, when host's /home is mounted in an unprivileged container by

lxc.mount.entry = /home home none bind,rw 0 0

its UID/GID cannot be altered. To enable UID 1000 in an unprivileged container to access files of UID 1000 in /home on the host, we have to adjust UID/GID mapping between the host and the container as follows:

# Container's UID/GID 0-65535 are mapped to host's 100000-165535,
# but UID/GID 1000 on the container is mapped to host's UID/GID 1000.
lxc.idmap = u 0 100000 1000
lxc.idmap = g 0 100000 1000
lxc.idmap = u 1000 1000 1
lxc.idmap = g 1000 1000 1
lxc.idmap = u 1001 101001 64535
lxc.idmap = g 1001 101001 64535

Common hints

root passwords

In LXC releases prior to 2.0.8 containers might be created with a random root password, a static password or without a password at all.

From 2.0.8 onward no root passwords are set by default.

If you need to set the password of a container (because you forgot the random one, or want to adjust the default), you can do so with lxc-attach -n <container> passwd.

Caveats

* containers not running with full device permissions (which is the default, restricted) spew out systemd errors in the container as systemd tries to set all devices as available (or even not-available); these messages can be turned off by setting /etc/systemd/journald.conf to "?MaxLevelStore=6", if that doesn't work the cgroup also needs to be auto-mounted as "ro" (instead of "mixed") (needs confirmation).

* lxc-checkconfig Multiple /dev/pts instances: missing? Don't be alarmed: https://edmondscommerce.github.io/fedora-24-lxc-multiple-/dev/pts-instances-missing/

* defaults.conf has no mechanism to substitute the hostname in to configuration files. That means while networking can have automatically assigned values for hwaddr its impossible to for default.conf to express "containers should be logged to /var/log/lxc-container-$HOSTNAME.log".

* Use of lxc on Debian hosts in the unified CGroup hierarchy (pure CGroup V2 hierarchy) is explained in CGroupV2.

Start and stop containers

Notes/warnings on starting and stopping containers:

Actual commands:

Content accuracy warning


Caveat Emptor: The rest of this page is useful but definitely outdated if you are trying to get LXC to work on Debian “stretch” or “sid” (unstable). Proceed with caution.


Unprivileged container

Note

1. You cannot use normal iso. The simplest thing to do is to use LXC templates, of "usual" distributions, provided by LXC maintainers.

2. Know that, by design, unprivileged containers cannot use mknod to create a block or character device, nor do loop mounts or mount partition. (This is a bit inaccurate. A device special file on a host can be accessed by unprivileged containers if their UID or GID of the device file is the same as container's root and the device file is bind-mounted. One can use NVIDIA GPU from unprivileged containers.)

The containers keep the same "default" right as the normal user (because of the "normal uid" shared with the user). In fact, LXC unprivileged containers fake some parts with subuids and subgids, and others, like create devices, etc… are "bypassed" during the installation process of these "tweaked" templates.

3. Keep in mind that you will have to write down your own subuids and subgids

Jessie / Debian 8

See ?LXC/JessieSpecific for information on setting up Unprivileged containers on Jessie.

How to setup LXC v2 Unprivileged container on Debian 9

Check the system

Now, try:

Everything should be stated as "enable" in green color. If not, try to reboot the system.

Configuration of the host system

Enter this in a terminal:

Then reload sysctl without rebooting the system:

Now, get the subuids and subgids of the current user:

You should get something like this, where 1258512 is your subuids and subgids,

Note

Here debian is the user name.

If the previous command do not return anything, it means that you do not have any subuids and subgids attributed.

You will have get any subuids and subgids with usermod command. Once done, add 65536 to this number, and enter these commands in the terminal.

Note

65536 uids and gids ranges are fine in most case; and enough to share this same range with "all" your containers.

Last thing before the network configuration.

In some configuration (not needed one barebone Sid install, nor in KVM and Virtualbox vm under Jessie, but needed in barebone Jessie install), lxc unprivileged container will complain that it cannot run, and will ask to add +x right to home user folder; .local and .local/share.

In that case, you will have to enter this in a terminal:

After these preparations, lxc-usernsexec should give # prompt with no error.

Configuration of the network

1. Configure a bridge in the host.

2. Configure the virtual network interfaces:

Tip

Choose the good naming here of the bridge interface. If you setup a bridge named br0, you should replace lxcbr0 by br0.

The configuration of LXC

1. First, you have to create the lxc folder:

2. Then, configure the default template for all future lxc unprivileged containers.

Warning

Bad settings with lxc.mount.auto option can lead to security risk and data loss!

Create lxc unprivileged containers

Launch lxc-create in normal user:

Note

I recommend to choose these options (Debian versions tested are too slow):

Here an example:

Now you have succeeded to create an Ubuntu unprivileged container.

Configuration of the unprivileged container

Now every changes in the configuration have to be made where the unprivileged container reside. So in this example, the container is located:

Check man lxc.container.conf to get more options.

Start of the unprivileged container

Enter the following command:

As it, if things go wrong, you will be able to track them in the log file.

Now you can connect to the container:

Unprivileged Debian container by mmdebstrap --mode=unshare

Without using lxc-create -t download, we can also create an unprivileged Debian container (named mmdebstrap-unpriv) by the following steps by a non-root user:

$ cd .local/share/lxc
$ mkdir mmdebstrap-unpriv
$ btrfs subvolume create mmdebstrap-unpriv/rootfs # If you use btrfs home directory
$ mmdebstrap --mode=unshare --components="main contrib non-free" --variant=important buster mmdebstrap-unpriv/rootfs
# --variant=important can be replaced by --variant=minbase --include=systemd-sysv
# In the container you should change /etc/hostname

lxc-start -F -n mmdebstrap-unpriv with the following container config ($HOME/.local/share/lxc/mmdebstrap-unpriv/config) seems working fine.

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)

# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.include = /usr/share/lxc/config/userns.conf
lxc.arch = linux64

# The following lines are copied from the "download" template of unprivileged Ubuntu.
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none bind,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0
lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional 0 0
lxc.mount.entry = /sys/firmware/efi/efivars sys/firmware/efi/efivars none bind,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none bind,optional 0 0


# Unprivileged container cannot use the "generated" AppArmor profile.
lxc.apparmor.profile = unconfined
lxc.apparmor.allow_nesting = 1
# The following two lines have to be compatible with /etc/subuid and /etc/subgid
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.rootfs.path = btrfs:/home/your-user-name/.local/share/lxc/mmdebstrap-unpriv/rootfs
lxc.uts.name = mmdebstrap-unpriv

# Network configuration
lxc.net.0.type = empty

We see the following error

systemd-journald-audit.socket: Failed to create listening socket (audit 1): Operation not permitted
systemd-journald-audit.socket: Failed to listen on sockets: Operation not permitted
systemd-journald-audit.socket: Failed with result 'resources'.
[FAILED] Failed to listen on Journal Audit Socket.
See 'systemctl status systemd-journald-audit.socket' for details.

The above error is reported as 959921

Preparing host system WITHOUT Systemd for running LXC

You can skip this section if you are running systemd (default). Your host(system) is already prepared.

Systems running sysv need to be primed for the running of LXC on it as it required cgroups to be mounted (among other things, perhaps).

The best solution is to install package cgroupfs-mount.

Note

At least on Debian 10 (buster), lxc-checkconfig still complains that Cgroup v1 systemd controller: missing and containers with systemd refuse to start.

According to Gentoo wiki, if you want to run containers with systemd, the host also needs a name=systemd cgroup hierarchy mounted: mkdir -p /sys/fs/cgroup/systemd; mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd.

With that, lxc-checkconfig no longer complains and containers with systemd are able to start.

This was reported to package cgroupfs-mount in bug #939435 (patch included).

If package cgroupfs-mount is not available, add this line to /etc/fstab. (This is not necessary if libvirt-bin is installed as init.d/libvirt-bin will mount /sys/fs/cgroup automatically).

cgroup  /sys/fs/cgroup  cgroup  defaults  0   0

Try to mount it (a reboot solves an eventual "resource busy problem" in any case)

mount /sys/fs/cgroup

Check kernel configuration :

# lxc-checkconfig
Kernel config /proc/config.gz not found, looking in other places...
Found kernel config file /boot/config-2.6.32-5-amd64
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup namespace: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: missing
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

Above the lxc-checkconfig program is reporting "Cgroup memory controller: missing". If you want memory control via cgroups then you need to recompile the linux kernel. In order to avoid problems when using memory limit settings during startup of a container, you must add cgroup_enable=memory to the kernel command line (Jessie or later). This applies even if Cgroup memory controller reports "enabled"

/!\ Be aware mounting cgroup from /etc/fstab has side-effects, like being unable to edit network manager connections.

Incompatibility with systemd

* The version in Wheezy (0.8.0~rc1-8+deb7u2) is not compatible with running systemd inside the container. See 766216 * The versions in both jessie and stretch support systemd in the container just fine for Debian guests. ** YMMV for other types of guests

Upgrading container from "Wheezy" to "Jessie"

When upgrading an lxc guest running "Wheezy" to "Jessie", the lxc VM will stop working, because at the time of writing (23.11.2014) systems will automatically be migrated to systemd. See 766233. This behaviour is being reviewed in 762194.

Workarounds:

Switch back to sysv

If the VM was migrated to systemd automatically via an upgrade then you can switch back to sysvinit:

lxc-stop -n myvm               # stop the vm
                               # or, if that doesn't work use lxc-kill

# the next step requires the VM to be mounted at /var/lib/lxc/myvm/root

chroot /var/lib/lxc/myvm/root  # chroot into the vm
apt-get install sysvinit-core # reinstall old sysvinit

Alternatively you can try to start the container in the foreground and do the same via the container's console.

Not letting your system be updated to systemd during the upgrade

Before upgrade, run:

apt-get install sysvinit-core

or run the following command in place of a usual dist-upgrade:

apt-get dist-upgrade sysvinit-core

Support

References

Debian-specific information

See also :


CategorySoftware | CategoryVirtualization | CategorySystemAdministration