0

I use docker for years on my raspi 3b rev 1.2 (with RaspiOS bullseye 11.5). Recently, I don't know why, the daemon dockerd failed to start. The systemd unit remains in failed status. I can reproduce the error by running dockerd manually and get some logs, but they didn't help me to understand what's going on:

➜  ~ sudo dockerd
INFO[2022-12-12T16:51:52.725265278+01:00] Starting up                                  
INFO[2022-12-12T16:51:52.731645021+01:00] parsed scheme: "unix"                         module=grpc
INFO[2022-12-12T16:51:52.731805647+01:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-12-12T16:51:52.731938199+01:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-12-12T16:51:52.732012886+01:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-12-12T16:51:52.738021067+01:00] parsed scheme: "unix"                         module=grpc
INFO[2022-12-12T16:51:52.738182057+01:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-12-12T16:51:52.738303723+01:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-12-12T16:51:52.738366380+01:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-12-12T16:51:52.893715377+01:00] [graphdriver] using prior storage driver: overlay2 
WARN[2022-12-12T16:51:52.906082155+01:00] Unable to find memory controller             
INFO[2022-12-12T16:51:52.907810541+01:00] Loading containers: start.                   
INFO[2022-12-12T16:51:53.156768500+01:00] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
INFO[2022-12-12T16:51:53.160111835+01:00] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
failed to start daemon: Error initializing network controller: Error creating default "bridge" network: failed to check bridge interface existence: no buffer space available

I think the main error is

failed to check bridge interface existence: no buffer space available

but I cannot figure out how to fix this

I tried multiple actions:

  • Deleting /var/lib/docker folder
  • Deleting the docker0 virtual interface
  • Uninstalling/reinstalling all packages (docker-ce, containerd, docker-rootless-extras, etc.)
  • Rebooting multiple times (yeah, that seems strange but multiple posts about similar error on the internet mentionned rebooting magically solved the issue)

Note: The IPV6 feature is currently disabled in /etc/docker/daemon.json. I also tried to enable it, without any success.

{
  "ipv6": false,
  "fixed-cidr-v6": "2a02:XXXX:YYYY:ZZZZ::/80"
}

(Note: I obfuscated my CIDR)

Any help is welcome !

Edit: I updated /boot/cmdline.txt to add following options:

cgroup_memory=1 cgroup_enable=memory

After a reboot, the warning relative to memory controller disappeared. The original error remains unfixed

INFO[2022-12-13T13:21:15.441652580+01:00] Starting up                                  
INFO[2022-12-13T13:21:15.447998677+01:00] parsed scheme: "unix"                         module=grpc
INFO[2022-12-13T13:21:15.448131021+01:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-12-13T13:21:15.448253886+01:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-12-13T13:21:15.448313938+01:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-12-13T13:21:15.455202014+01:00] parsed scheme: "unix"                         module=grpc
INFO[2022-12-13T13:21:15.455321441+01:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-12-13T13:21:15.455415712+01:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-12-13T13:21:15.455475035+01:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-12-13T13:21:15.574649053+01:00] [graphdriver] using prior storage driver: overlay2 
INFO[2022-12-13T13:21:15.587653226+01:00] Loading containers: start.                   
INFO[2022-12-13T13:21:15.831901578+01:00] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
INFO[2022-12-13T13:21:15.834550486+01:00] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
failed to start daemon: Error initializing network controller: Error creating default "bridge" network: failed to check bridge interface existence: no buffer space available
2
  • look at the warning in the log
    – jsotola
    Commented Dec 12, 2022 at 21:28
  • Thanks for your comment. I investigated that clue and finally enabled cgroups in cmdline.txt. This solved the warning you were referencing, but the daemon still fail to start with the exact same error message
    – Antwane
    Commented Dec 13, 2022 at 12:36

1 Answer 1

0

I finally found my own mistake (obviously). In the past, I tried to tune my raspberry config for docker, and added following sysctl settings:

From https://gist.github.com/leeuwte/413c0e8e72dc32e4df7f8c2a68a2c11c, in /etc/sysctl.conf

# Bigger buffers (to make 40Gb more practical). These are maximums, but the default is unaffected.
net.core.wmem_max=268435456
net.core.rmem_max=268435456
net.core.netdev_max_backlog=10000

# Avoids problems with multicast traffic arriving on non-default interfaces
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0

# Force IGMP v2 (required by CBF switch)
net.ipv4.conf.all.force_igmp_version=2
net.ipv4.conf.default.force_igmp_version=2

# Increase the ARP cache table
net.ipv4.neigh.default.gc_thresh3=4096
net.ipv4.neigh.default.gc_thresh2=2048
net.ipv4.neigh.default.gc_thresh1=1024

# Increase number of multicast groups permitted
net.ipv4.igmp_max_memberships=1024

From https://github.com/ndeloof/RPiDockerKillerScore/blob/master/DockerPi/conf/etc/sysctl.d/docker.conf in /etc/sysctl.d/docker.conf

vm.overcommit_memory = 1
vm.overcommit_ratio = 500

net.core.rmem_default = 4096
net.core.rps_sock_flow_entries = 0
net.core.somaxconn = 4
net.core.tstamp_allow_data = 1
net.core.warnings = 0
net.core.wmem_default = 8192

net.core.netdev_max_backlog = 1
net.ipv4.tcp_no_metrics_save = 1

net.ipv4.tcp_mem=4096 4096 4096
net.ipv4.tcp_rmem=4096 4096 4096
net.ipv4.tcp_wmem=4096 4096 4096
net.ipv4.udp_mem=4096 4096 4096

kernel.perf_event_max_sample_rate = 1
kernel.perf_cpu_time_max_percent = 1
kernel.pid_max = 32768
kernel.threads-max = 14812
kernel.randomize_va_space = 0

vm.swappiness = 100

After removing these settings, docker daemon started again and everything works well now

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.