Why don't LXC containers start on RPi1B (lxc_seccomp policy loading error)?

Question

My LXC containers do not start, any of you knowledgeable people know why?

Your insights would be greatly appreciated; also when they do not directly offer a solution, but point me in a direction instead for me to further read-up and learn from.

For now I am totally stumped.. after 2 days of online searching:

Context:

HW:        Raspberry Pi 1B (700Mhz-512MB)
OS:        Raspbian Jessie Light (Debian based)
LXC:       1:1.0.6-6+deb8u2 (from Raspbian repo)
Container: Debian jessie armhf/armel (tried both archs)

Symptoms

stdout (cli output after trying to start the container as root):

root@raspberrypi:~# lxc-start --name debian8 -d
lxc-start: The container failed to start.
lxc-start: To get more details, run the container in foreground mode.
lxc-start: Additional information can be obtained by setting the --logfile and --logpriority options.

log (excerpt):

  lxc-start 1471983006.471 ERROR    lxc_seccomp - Error loading the seccomp policy
  lxc-start 1471983006.479 ERROR    lxc_sync - invalid sequence number 1. expected 4
  lxc-start 1471983006.479 ERROR    lxc_start - failed to spawn 'debian8'

So it looks like that the root cause of the container not starting lies with lxc-start running into an error while trying to load the lxc_seccomp policy. This assumption is backed up by the workaround:

Workaround

Disable the seccomp configuration loading at startup, by commenting it out in /usr/share/lxc/config/debian.common.conf:

# Blacklist some syscalls which are not safe in privileged
# containers
#  lxc.seccomp = /usr/share/lxc/config/common.seccomp

Than the container starts.. But turning off such a basic security setting that is - moreover - so heavily tied to containerization/sandboxing, is (to say the least) kind of defeating the purpose of LXC. From a security/stability point of view I would very much like to keep blacklisting most of the system calls when running the LXC containers on my Rpi (as configured by LXC defaults in /usr/share/lxc/config/common.seccomp):

2
blacklist
[all]
kexec_load errno 1
open_by_handle_at errno 1
init_module errno 1
finit_module errno 1
delete_module errno 1

Hypotheses

• root cause of ´the container not starting´ lies with the ´lxc-start´ process running into an error while trying to load the ´lxc_seccomp policy´.
• lxc_sync - invalid sequence number 1. expected 4 is that error.

Questions

Main

1. Why doesn't the LXC container start?

Subs

1. Are the aforementioned explanation hypotheses valid?
2. Why can't the seccomp config be loaded (any deeper logging available)?
3. What does lxc_sync - invalid sequence number 1. expected 4 technically mean in this context?
4. Am I looking at a fundamental / difficult to solve problem (arm6) here (e.g. unsolvable without manually modifying kernels etc.)?
5. What are my chances when switching to Docker system (higher level virtualization build on top of LXC)?
6. What are my chances when using Raspberry Pi 3 (arm7+)?

You guys/gals are my last hope. So any insight would be greatly valued!

If any more data/local-actions is/are required to advance with this issue, please let me know, for I am more than willing to pick it up!

Answer 1

On my Raspbian Jessie installation, the Kernel (4.4.27) is compiled with CONFIG_SECCOMP but not the filter option (CONFIG_SECCOMP_FILTER and CONFIG_HAVE_ARCH_SECCOMP_FILTER). So you will not be able to activate seccomp profiles on LXC (or Docker). See issue: https://github.com/raspberrypi/linux/issues/1172

In addition, there are no user namespace support which I find even more dangerous in terms of container security. It means that a root user within a container is basically mapped to the root user outside, so there is no really unprivileged container in this context.

Furthermore, AppArmor is not activated in the Kernel either. So even if you install apparmor user space tools, it is not running.

However, capabilities are built-in and you can still deny access to a lot of system calls using them :-)

Note: I was able to compile a new Kernel adding the CONFIG_SECCOMP_FILTER, CONFIG_HAVE_ARCH_SECCOMP_FILTER, CONFIG_USER_NS, CONFIG_SECURITY_APPARMOR, CONFIG_SECURITY_APPARMOR_BOOTPARAM_VALUE, CONFIG_DEFAULT_SECURITY_APPARMOR and CONFIG_AUDIT. Just follow the official instructions https://www.raspberrypi.org/documentation/linux/kernel/building.md and modify the config file by adding the above configuration option (set them to =y) before doing the make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2709_defconfig. It works, but at least for Docker the user space seccomp tools from Raspbian Jessie are too old to apply seccomp profiles, maybe it works for LXC but I do not know.

Building your kernel

If someone wants to build his own kernel, this is not too hard (and not too long if you have a Linux x86_64 machine with a few cores, even if it is in a VM). I already pointed out to the official instructions to build a kernel, prefer the cross building on your Linux Desktop/VM, this would be faster. After cloning the source and before building them, you need to update the config file. For RPi 2 and 3, it is arch/arm/configs/bcm2709_defconfig for first generation RPi or Pi Zero, it is arch/arm/configs/bcmrpi_defconfig. Each config I have mentioned should be on its own line and you should add =y at the end of each new lines. These configurations are enabling SECCOMP filtering, user namespace and AppArmor in the Kernel. After this change, you can resume the build instructions. Here is the config file I have been using for my RPi 2: https://github.com/jcberthon/linux/blob/rpi-sec-apparmor-seccomp-userns/arch/arm/configs/bcm2709_defconfig (or you could use the patched bcmrpi_defconfig which is in the same folder for the Raspberry Pi 1 and Zero, but I haven't tested it as I do not own such device).

Take care that when you will install the new kernel, if you make a mistake, you might break your RPi. It is not going to be bricked, but might not boot. So take care to make backups before installing the new kernel, and it is better to install the new kernel in parallel to the old one and use the kernel= instructions in the /boot/config.txt file. This is explained in the official instructions.

Once you have booted on the new kernel, you can/should install the AppArmor user space tools: sudo apt-get install apparmor you can install in addition apparmor-profiles and apparmor-profiles-extra to get even more profiles for your potentially running apps.

For the seccomp tool, you could try to use the libseccomp2 file backported to Debian Jessie (https://packages.debian.org/jessie-backports/libseccomp2). I've installed it on Raspbian, it did not break anything, but in my use case the Docker binaries are not built against it, so it still did not make use of it. And it starts to be a bit more time consuming to try to fix the docker build to use the Debian backport repo and force the seccomp dependency.

But anyway, if you have this new kernel with user namespace and AppArmor, you can use both technology to run LXC unprivileged and better contained containers than with the standard Raspbian kernel.