Docker Running On RpiZW does everything but actually running container, SIGSEGV

Question

I have installed a fresh raspbian buster and docker from get.docker.com on a raspberry pi zero W [Edit: Does not work on raspbian jessie or stretch either]

I have run docker on that machine before, but now docker run <image> just starts and terminates about 18 seconds later, no output. I have tried several different Dockerfiles which works on a regular Linux laptop.

This is output to daemon.log while docker run runs. I see no errors, just a warning that (...)/mnt/shm can not be found. [Edit: This warning is also shown on working docker on pc, so it's not relevant.] Note: I have now discovered that docker container ls -l shows that the containers end with error code 139, which is SIGSEGV

tail -f /var/log/daemon.log|grep docker &
root@pi:/# docker run hello-world
..8:44 pi systemd[1]: var-lib-docker-overlay2-8a2da5298e1(...)4ed0b781f\x2dinit-merged.mount: Succeeded.
..8:44 pi systemd[463]: var-lib-docker-overlay2-8a2da5298e1(...)4ed0b781f\x2dinit-merged.mount: Succeeded.
..8:44 pi systemd[1]: var-lib-docker-overlay2-8a2da5298e1(...)4ed0b781f-merged.mount: Succeeded.
..8:45 pi systemd[463]: var-lib-docker-overlay2-8a2da5298e1(...)4ed0b781f-merged.mount: Succeeded.
..8:48 pi systemd[1]: run-docker-runtime\x2drunc-moby-6c47caad0e(...)086e9ac535ba-runc.8fnFt5.mount: Succeeded.
..8:48 pi dhcpcd[277]: docker0: carrier acquired
..8:48 pi dhcpcd[277]: docker0: IAID 29:1f:af:46
..8:48 pi dhcpcd[277]: docker0: adding address fe80::f3ff:1d1a:3346:f173
..8:48 pi avahi-daemon[241]: Joining mDNS multicast group on interface docker0.IPv6 with address fe80::f3ff:1d1a:3346:f173.
..8:48 pi avahi-daemon[241]: New relevant interface docker0.IPv6 for mDNS.
..8:48 pi avahi-daemon[241]: Registering new address record for fe80::f3ff:1d1a:3346:f173 on docker0.*.
..8:48 pi dhcpcd[277]: docker0: soliciting an IPv6 router
..8:48 pi dhcpcd[277]: docker0: soliciting a DHCP lease
..8:54 pi dhcpcd[277]: docker0: probing for an IPv4LL address
..8:59 pi dhcpcd[277]: docker0: using IPv4LL address 169.254.156.109
..8:59 pi dhcpcd[277]: docker0: adding route to 169.254.0.0/16
..8:59 pi avahi-daemon[241]: Registering new address record for 169.254.156.109 on docker0.IPv4.
..9:00 pi dockerd[324]: time="2019-10-26T23:29:00.407340153+02:00" level=info msg="ignoring event"
     module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
..9:00 pi systemd[1]: run-docker-netns-1f13548fc883.mount: Succeeded.
..9:00 pi systemd[463]: run-docker-netns-1f13548fc883.mount: Succeeded.
..9:01 pi dockerd[324]: time="2019-10-26T23:29:01.039055309+02:00" level=warning msg="6c47caad0e(...)086e9ac535ba
    cleanup: failed to unmount IPC: umount /var/lib/docker/containers/6c47caad0e(...)086e9ac535ba/mounts/shm, flags: 0x2: no such file or directory"
..9:01 pi systemd[1]: var-lib-docker-overlay2-8a2da5298e1(...)4ed0b781f-merged.mount: Succeeded.
..9:01 pi systemd[463]: var-lib-docker-overlay2-8a2da5298e1(...)4ed0b781f-merged.mount: Succeeded.
..9:02 pi dhcpcd[277]: docker0: carrier lost
..9:02 pi dhcpcd[277]: docker0: deleting address fe80::f3ff:1d1a:3346:f173
..9:02 pi avahi-daemon[241]: Withdrawing address record for fe80::f3ff:1d1a:3346:f173 on docker0.
..9:02 pi avahi-daemon[241]: Leaving mDNS multicast group on interface docker0.IPv6 with address fe80::f3ff:1d1a:3346:f173.
..9:02 pi avahi-daemon[241]: Interface docker0.IPv6 no longer relevant for mDNS.
..9:02 pi avahi-daemon[241]: Withdrawing address record for 169.254.156.109 on docker0.
..9:02 pi dhcpcd[277]: docker0: deleting route to 169.254.0.0/16

Edit, to answer tflong01

tail -f /var/log/daemon.log keep printing the log from miscellaneous daemons |grep docker send output to grep to show only lines containing docker & keep running in background, but release keyboard back, so I can type the next command

docker takes a couple of disk images: a "disk" with the operating system, another with the command and mounts them on top of each other, creating a virtual machine.

at 8:48 this machine is powered up, it gets an IPv6 address and starts looking for an IPv4 address by DHCP. There is no DHCP on that network, so

at 8:54 it starts creating its own 169.254.0.0/16 adress, which it has done at 8:59

at 9:00 the machine looks ready to run the "hello-world" command, but no output is done

at 9:01 the machine powers down, the ip-adresses get released and the disk images unmounted and the routing entries cleared.

All looks normal except nothing is being run in the machine. Or it doesn't output anything.

Answer 1

One needs to run the right architecture.

The kernel on my raspberry pi zero W is armv6l

Whatever mechanism docker have to select the right arch is not solid enough.

For instance alpine:latest have images for

• linux/386
• linux/amd64
• linux/arm/v6
• linux/arm/v7
• linux/arm64/v8
• linux/ppc64le
• linux/s390x

But docker image inspect shows both arm/v6 and arm/v7 as just "arm". So docker pull alpine happens to select arm/v7, while RasPIzW needs arm/v6

I can get the right arch by looking up the digest on dockerhub, and do a docker run alpine@sha256:29a82d50bdb8dd7814009852c1773fb9bb300d2f655bd1cd9e764e7bb1412be3 but that locks me to that particular version, so I'll not automatically use alpine:latest in my Dockerfile or docker-compose

I have solved it in this case, by using arm32v6/alpine, but that only works because some user called arm32v6 makes images specially for arm32v6

This also means I can't have the same Dockerfile on my pc as on the raspberry, if they use binary images.

I'd like a way to say

if arch="armv6l" then
  FROM arm32v6/alpine
else
  FROM alpine
end if

Answer 2

For reference, and to supplement @tifong01 and @Lenne's great responses:

The implementation of the Docker Manifest V2 Schema 2 API used by the Docker CLI does not currently differentiate properly between armv6 and armv7 architectures at runtime (related Github issues: here in moby core and here in the official-images repo, where images like alpine live). This doesn't affect armv7 boards, but does impact armv6 boards.

Here's a quick reference table I put together mapping each Raspberry Pi release (from 2012 until late 2019) to the corresponding ARM instruction set:

Boards highlighted in yellow or green use the armv7 or armv8/aarchv8 instruction set, and should be able to execute images that include any arm images in their manifest just fine.

Boards highlighted in red use the older ARM11 instruction set, which corresponds to the armv6l family of microarchitectures... TL;DR, it depends on how the images were compiled, and YMMV. But for images that reliably execute on armv6l: I'd recommended sticking to the arm32v6 images — these are, in fact, official images, actively maintained under the same docker/official-images repo mentioned earlier. I've been using them as one of the base images in various projects to cross-build images in CI for a couple years now, and have never had a problem (see e.g., minio-multiarch for an example of this pattern, if you're interested).

Hope this helps!

Answer 3

Another F.Y.I.

If you want to create "a single image tag", using multi-arch, compatible to:

• AMD64/x86_64 (macOS and Windows10 Intel-compatible machines)
• ARMv7 (RaspberryPi 3B + Buster)
• and ARMv6 (RaspberryPi ZeroW + Buster)

so to have the same Dockerfile for PC, macOS, RPi3 and RPi Zero, then you need to re-write the manifest list of the base image a little.

First, create images for each architecture and its manifest. Building as <your image name>:<arc name> and pushing to Docker Hub then pulling back might be easy.

Next, create a manifest list with the latest tag(<your image name>:latest) and add all the manifest above.

NAME_IMAGE=keinos/alpine
NAME_MANIFEST_LIST="$NAME_IMAGE:latest"

LIST_MANIFESTS="$NAME_IMAGE:armv6 $NAME_IMAGE:armv7 $NAME_IMAGE:arm64 $NAME_IMAGE:amd64"

docker manifest create $NAME_MANIFEST_LIST $LIST_MANIFESTS --amend

Finally and most important, change the variant value of the manifest for armv6 from v6 to v6l like below and push the manifest list:

NAME_IMAGE=keinos/alpine
NAME_MANIFEST_LIST="$NAME_IMAGE:latest"

MANIFEST=keinos/alpine:armv6

docker manifest annotate $NAME_MANIFEST_LIST $MANIFEST --variant v6l
docker manifest push $NAME_MANIFEST_LIST --purge

As a sample, try the below image and command on RPi Zero, RPi3, macOS, and Win10. (hope it works on all architecture)

docker pull keinos/alpine
docker run --rm keinos/alpine cat /etc/os-release

Most online articles didn't create a compatible image tag that works for both ARMv7 and ARMv6 but reading @lenne, @jesse-stuart, and @tlfong01 great responses I came up with the solution above. Thanks!

Answer 4

Question

How come Docker running Hello World on RpiZW returns nothing?

---

Answer

Update 2019oct2901

The OP just found the cause of the problem. See his answer above. Let the docker newbie, ie, me, try to summarize the OP's explanation:

(1) Rpi3/4 uses arm/v7, while RasPIzW uses arm/v6

(2) Docker by default uses architecture for arm/v7 (Installation Note 1)

(3) So it is RpiZW's "architecture" not compatible to Docker, not just the display driver as I have wrongly guessed.

My apologies for the wrong and misleading guess. Many thanks to the OP for teaching me how to run and troubleshoot Docker stuff.

Installation Note 1

When installing Docker in my Rpi4B buster, I used the (1) "stretch mature" library, and not any (2) "buster non mature" library, because I think if (1) works, then try it first, and try (2) later, in case of problems. Actually I was to lazy to find if there is any other library for luster. Anyway, if stretch library can run "Hello World" without any problem. Might need to update to buster library later.

---

Update 2019oct2801

The OP just said the following:

"Note: I have now discovered that docker container ls -l shows that the containers end with error code 139, which is SIGSEGV"

I think this may mean my conclusion might be totally wrong. So I have stricken out my conclusion below, and wait for the OP to do more research work.

---

Experimentation

(1) I installed docker on Rpi4B with buster and ran Hello World OK (Refs 1, 2, 3).

(2) I cloned buster image from Rpi4B and used it to run hello world on RpiZW.

(3) Docker in RpiZW buster returns nothing. In other words, bad luck as the OP.

---

Discussion

I compared and contrasted your RpiZW log with mine. I found them did similar things and seem to end successfully in a roughly same time period.

In other words, both docker guys think they have completed the Hello World job. The only catch is that the Docker has no eyes to look at the terminal display that it does display any thing.

In short, the Rpi4B terminal display driver gets the Docker message and display happily. But the RpiZW driver does not do anything, just sleeping!

---

Conclusion

Docker is not compatible with RpiZW buster.

Errata - As mentioned in the update above, the OP has found the error message which might indicate the real cause of the problem, not at all related to my quick and dirty conclusion of a vague remark on incompatibility.

Updated conclusion

It is Docker's default Rpi architecture not compatible, not just driver incompatible, though implied.

See the OP's answer for a very detailed explanation.

---

References

(1) Docker Rpi4B buster Installation Record v0.1 - tlfong01 Sun. 10/27/2019

(2) Docker Rpi4B buster Hello World Execution Record v0.1 - tlfong01 Sun. 10/27/2019

(3) Docker Rpi4B buster Hello World Execution Log v0.2 - tlfong01 Sun. 10/27/2019

(4) Docker Learning Notes V0.3 - tlfong01 Sun. 10/27/2019

(5) Understanding Docker Container Technology for Beginner - GeekFlare

(6) Docker Architecture and its Components for Beginner - GeekFlare

---

End of answer