Docker daemon does not start: "Error creating default "bridge" network: failed to check bridge interface existence: no buffer space available"

Question

I use docker for years on my raspi 3b rev 1.2 (with RaspiOS bullseye 11.5). Recently, I don't know why, the daemon dockerd failed to start. The systemd unit remains in failed status. I can reproduce the error by running dockerd manually and get some logs, but they didn't help me to understand what's going on:

➜  ~ sudo dockerd
INFO[2022-12-12T16:51:52.725265278+01:00] Starting up                                  
INFO[2022-12-12T16:51:52.731645021+01:00] parsed scheme: "unix"                         module=grpc
INFO[2022-12-12T16:51:52.731805647+01:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-12-12T16:51:52.731938199+01:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-12-12T16:51:52.732012886+01:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-12-12T16:51:52.738021067+01:00] parsed scheme: "unix"                         module=grpc
INFO[2022-12-12T16:51:52.738182057+01:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-12-12T16:51:52.738303723+01:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-12-12T16:51:52.738366380+01:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-12-12T16:51:52.893715377+01:00] [graphdriver] using prior storage driver: overlay2 
WARN[2022-12-12T16:51:52.906082155+01:00] Unable to find memory controller             
INFO[2022-12-12T16:51:52.907810541+01:00] Loading containers: start.                   
INFO[2022-12-12T16:51:53.156768500+01:00] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
INFO[2022-12-12T16:51:53.160111835+01:00] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
failed to start daemon: Error initializing network controller: Error creating default "bridge" network: failed to check bridge interface existence: no buffer space available

I think the main error is

failed to check bridge interface existence: no buffer space available

but I cannot figure out how to fix this

I tried multiple actions:

• Deleting /var/lib/docker folder
• Deleting the docker0 virtual interface
• Uninstalling/reinstalling all packages (docker-ce, containerd, docker-rootless-extras, etc.)
• Rebooting multiple times (yeah, that seems strange but multiple posts about similar error on the internet mentionned rebooting magically solved the issue)

Note: The IPV6 feature is currently disabled in /etc/docker/daemon.json. I also tried to enable it, without any success.

{
  "ipv6": false,
  "fixed-cidr-v6": "2a02:XXXX:YYYY:ZZZZ::/80"
}

(Note: I obfuscated my CIDR)

Any help is welcome !

Edit: I updated /boot/cmdline.txt to add following options:

cgroup_memory=1 cgroup_enable=memory

After a reboot, the warning relative to memory controller disappeared. The original error remains unfixed

INFO[2022-12-13T13:21:15.441652580+01:00] Starting up                                  
INFO[2022-12-13T13:21:15.447998677+01:00] parsed scheme: "unix"                         module=grpc
INFO[2022-12-13T13:21:15.448131021+01:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-12-13T13:21:15.448253886+01:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-12-13T13:21:15.448313938+01:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-12-13T13:21:15.455202014+01:00] parsed scheme: "unix"                         module=grpc
INFO[2022-12-13T13:21:15.455321441+01:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-12-13T13:21:15.455415712+01:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-12-13T13:21:15.455475035+01:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-12-13T13:21:15.574649053+01:00] [graphdriver] using prior storage driver: overlay2 
INFO[2022-12-13T13:21:15.587653226+01:00] Loading containers: start.                   
INFO[2022-12-13T13:21:15.831901578+01:00] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
INFO[2022-12-13T13:21:15.834550486+01:00] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
failed to start daemon: Error initializing network controller: Error creating default "bridge" network: failed to check bridge interface existence: no buffer space available

Answer 1

I finally found my own mistake (obviously). In the past, I tried to tune my raspberry config for docker, and added following sysctl settings:

From https://gist.github.com/leeuwte/413c0e8e72dc32e4df7f8c2a68a2c11c, in /etc/sysctl.conf

# Bigger buffers (to make 40Gb more practical). These are maximums, but the default is unaffected.
net.core.wmem_max=268435456
net.core.rmem_max=268435456
net.core.netdev_max_backlog=10000

# Avoids problems with multicast traffic arriving on non-default interfaces
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0

# Force IGMP v2 (required by CBF switch)
net.ipv4.conf.all.force_igmp_version=2
net.ipv4.conf.default.force_igmp_version=2

# Increase the ARP cache table
net.ipv4.neigh.default.gc_thresh3=4096
net.ipv4.neigh.default.gc_thresh2=2048
net.ipv4.neigh.default.gc_thresh1=1024

# Increase number of multicast groups permitted
net.ipv4.igmp_max_memberships=1024

From https://github.com/ndeloof/RPiDockerKillerScore/blob/master/DockerPi/conf/etc/sysctl.d/docker.conf in /etc/sysctl.d/docker.conf

vm.overcommit_memory = 1
vm.overcommit_ratio = 500

net.core.rmem_default = 4096
net.core.rps_sock_flow_entries = 0
net.core.somaxconn = 4
net.core.tstamp_allow_data = 1
net.core.warnings = 0
net.core.wmem_default = 8192

net.core.netdev_max_backlog = 1
net.ipv4.tcp_no_metrics_save = 1

net.ipv4.tcp_mem=4096 4096 4096
net.ipv4.tcp_rmem=4096 4096 4096
net.ipv4.tcp_wmem=4096 4096 4096
net.ipv4.udp_mem=4096 4096 4096

kernel.perf_event_max_sample_rate = 1
kernel.perf_cpu_time_max_percent = 1
kernel.pid_max = 32768
kernel.threads-max = 14812
kernel.randomize_va_space = 0

vm.swappiness = 100

After removing these settings, docker daemon started again and everything works well now