2

My LXC containers do not start, any of you knowledgeable people know why?

Your insights would be greatly appreciated; also when they do not directly offer a solution, but point me in a direction instead for me to further read-up and learn from.

For now I am totally stumped.. after 2 days of online searching:

Context:

HW:        Raspberry Pi 1B (700Mhz-512MB)
OS:        Raspbian Jessie Light (Debian based)
LXC:       1:1.0.6-6+deb8u2 (from Raspbian repo)
Container: Debian jessie armhf/armel (tried both archs)

Symptoms

stdout (cli output after trying to start the container as root):

root@raspberrypi:~# lxc-start --name debian8 -d
lxc-start: The container failed to start.
lxc-start: To get more details, run the container in foreground mode.
lxc-start: Additional information can be obtained by setting the --logfile and --logpriority options.

log (excerpt):

  lxc-start 1471983006.471 ERROR    lxc_seccomp - Error loading the seccomp policy
  lxc-start 1471983006.479 ERROR    lxc_sync - invalid sequence number 1. expected 4
  lxc-start 1471983006.479 ERROR    lxc_start - failed to spawn 'debian8'

So it looks like that the root cause of the container not starting lies with lxc-start running into an error while trying to load the lxc_seccomp policy. This assumption is backed up by the workaround:

Workaround

Disable the seccomp configuration loading at startup, by commenting it out in /usr/share/lxc/config/debian.common.conf:

# Blacklist some syscalls which are not safe in privileged
# containers
#  lxc.seccomp = /usr/share/lxc/config/common.seccomp

Than the container starts.. But turning off such a basic security setting that is - moreover - so heavily tied to containerization/sandboxing, is (to say the least) kind of defeating the purpose of LXC. From a security/stability point of view I would very much like to keep blacklisting most of the system calls when running the LXC containers on my Rpi (as configured by LXC defaults in /usr/share/lxc/config/common.seccomp):

2
blacklist
[all]
kexec_load errno 1
open_by_handle_at errno 1
init_module errno 1
finit_module errno 1
delete_module errno 1

Hypotheses

  • root cause of ´the container not starting´ lies with the ´lxc-start´ process running into an error while trying to load the ´lxc_seccomp policy´.
  • lxc_sync - invalid sequence number 1. expected 4 is that error.

Questions

Main

  1. Why doesn't the LXC container start?

Subs

  1. Are the aforementioned explanation hypotheses valid?
  2. Why can't the seccomp config be loaded (any deeper logging available)?
  3. What does lxc_sync - invalid sequence number 1. expected 4 technically mean in this context?
  4. Am I looking at a fundamental / difficult to solve problem (arm6) here (e.g. unsolvable without manually modifying kernels etc.)?
  5. What are my chances when switching to Docker system (higher level virtualization build on top of LXC)?
  6. What are my chances when using Raspberry Pi 3 (arm7+)?

You guys/gals are my last hope. So any insight would be greatly valued!

If any more data/local-actions is/are required to advance with this issue, please let me know, for I am more than willing to pick it up!

| improve this question | |
  • CONFIG_SECCOMP is set on the stock Raspbian kernel, although I notice two other options that are set on a default configured x86-64 kernel aren't: CONFIG_HAVE_ARCH_SECCOMP_FILTER and CONFIG_SECCOMP_FILTER. The latter depends upon the former and the former is auto-selected by the architecture, so if it's required, then you are probably out of luck. – goldilocks♦ Aug 24 '16 at 15:19
  • Thanks @goldilocks (giggles), that sounds plausible! I think I (re)confirmed your point but I am unsure about: 1) whether I am looking in the right file and 2) about the kernel configuration file's mechanism. Also, although your reply did get me to grind the web again, I was unable to verify whether LXC indeed needs all three referred options. – woosting Aug 24 '16 at 20:28
  • As @goldilocks probably pinpointed the issue; I have posted a Linux/LXC/seccomp followup security question. That may be of interest of people reading this topic too (posted on the more general Unix-Linux StackExchange) . – woosting Aug 25 '16 at 8:29
  • Looks like you got answers to those. As a clarification about the "default configured x86-64 kernel", I meant this to refer to the what's created by make defconfig which is used implicitly if you make config (or menuconfig, or whatever); those actually exist per architecture and you can find them, e.g., in [src]/arch/x86/configs/. The Pi kernel has a few of its own. Not they aren't necessarily what stock distro kernels use. – goldilocks♦ Aug 25 '16 at 11:09
  • However, SECCOMP actually isn't in the x86-64 one (which is in the x86 directory, BTW; arch/x86_64 is mostly an empty placeholder) so (not that it matters much) I must have selected it at some point -- technically that's not a "default configured" kernel, its custom rolled, but since I don't recall needing that for anything I presumed it was derived from the defconfig. If I noticed it at some point there's a good chance I would have added it based on the description. – goldilocks♦ Aug 25 '16 at 11:09
2

On my Raspbian Jessie installation, the Kernel (4.4.27) is compiled with CONFIG_SECCOMP but not the filter option (CONFIG_SECCOMP_FILTER and CONFIG_HAVE_ARCH_SECCOMP_FILTER). So you will not be able to activate seccomp profiles on LXC (or Docker). See issue: https://github.com/raspberrypi/linux/issues/1172

In addition, there are no user namespace support which I find even more dangerous in terms of container security. It means that a root user within a container is basically mapped to the root user outside, so there is no really unprivileged container in this context.

Furthermore, AppArmor is not activated in the Kernel either. So even if you install apparmor user space tools, it is not running.

However, capabilities are built-in and you can still deny access to a lot of system calls using them :-)

Note: I was able to compile a new Kernel adding the CONFIG_SECCOMP_FILTER, CONFIG_HAVE_ARCH_SECCOMP_FILTER, CONFIG_USER_NS, CONFIG_SECURITY_APPARMOR, CONFIG_SECURITY_APPARMOR_BOOTPARAM_VALUE, CONFIG_DEFAULT_SECURITY_APPARMOR and CONFIG_AUDIT. Just follow the official instructions https://www.raspberrypi.org/documentation/linux/kernel/building.md and modify the config file by adding the above configuration option (set them to =y) before doing the make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2709_defconfig. It works, but at least for Docker the user space seccomp tools from Raspbian Jessie are too old to apply seccomp profiles, maybe it works for LXC but I do not know.

Building your kernel

If someone wants to build his own kernel, this is not too hard (and not too long if you have a Linux x86_64 machine with a few cores, even if it is in a VM). I already pointed out to the official instructions to build a kernel, prefer the cross building on your Linux Desktop/VM, this would be faster. After cloning the source and before building them, you need to update the config file. For RPi 2 and 3, it is arch/arm/configs/bcm2709_defconfig for first generation RPi or Pi Zero, it is arch/arm/configs/bcmrpi_defconfig. Each config I have mentioned should be on its own line and you should add =y at the end of each new lines. These configurations are enabling SECCOMP filtering, user namespace and AppArmor in the Kernel. After this change, you can resume the build instructions. Here is the config file I have been using for my RPi 2: https://github.com/jcberthon/linux/blob/rpi-sec-apparmor-seccomp-userns/arch/arm/configs/bcm2709_defconfig (or you could use the patched bcmrpi_defconfig which is in the same folder for the Raspberry Pi 1 and Zero, but I haven't tested it as I do not own such device).

Take care that when you will install the new kernel, if you make a mistake, you might break your RPi. It is not going to be bricked, but might not boot. So take care to make backups before installing the new kernel, and it is better to install the new kernel in parallel to the old one and use the kernel= instructions in the /boot/config.txt file. This is explained in the official instructions.

Once you have booted on the new kernel, you can/should install the AppArmor user space tools: sudo apt-get install apparmor you can install in addition apparmor-profiles and apparmor-profiles-extra to get even more profiles for your potentially running apps.

For the seccomp tool, you could try to use the libseccomp2 file backported to Debian Jessie (https://packages.debian.org/jessie-backports/libseccomp2). I've installed it on Raspbian, it did not break anything, but in my use case the Docker binaries are not built against it, so it still did not make use of it. And it starts to be a bit more time consuming to try to fix the docker build to use the Debian backport repo and force the seccomp dependency.

But anyway, if you have this new kernel with user namespace and AppArmor, you can use both technology to run LXC unprivileged and better contained containers than with the standard Raspbian kernel.

| improve this answer | |
  • First of all congrats on your newborn! Second thanks for your elaborate reply that I will probably mark as the answer (after some time). I much respect you even rebuilding/testing the kernel from this perspective (and read the Github thread / will keep monitoring it)! Sadly this is where I have to accept defeat (for now) as an active "player" on this "field". I'm proficient enough to use Linux/Raspbian (Debian), but don't have the experience nor time to dig-in that deep (for now). Also it's a little too experimental for the use-case I had in mind (very insightful though!). Thx – woosting Oct 13 '16 at 10:13
  • @woosting I'm going to keep on pushing to have seccomp and other kernel features in the official Raspbian kernel. If you follow the building kernel instructions, you will found that it is rather easy to do. I advise using the cross-compilation way (took less than 10min to compile, would have taken many hours on the RPi). I will improve my answer a bit more to guide you for building the kernel. But you are fully right, disabling seccomp is not great. And you probably found out that there are no user namespace as well in the Raspbian kernel, so you can't real create LXC unprivilege container. – Huygens Oct 18 '16 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.