Kernel can't load modules on Raspberry Pi 3B+

Question

I have a Raspberry Pi 3B+ that boots from its SD card but reads its root filesystem from an external hard drive connected by USB. Everything has been working well since I set that up until I tried to reboot it after about 3 months and it was unable to boot.

I originally followed these instructions for setting up the external rootfs: move the filesystem to a USB stick

At the end of that page is this:

Note One thing you need to keep track off is the difference in kernel versions between the boot code on the SD card and the file system on the USB drive when you apt-get upgrade. The original kernel software on the SD card may get out of synchronization with the newer boot files on the DOS partition. One way to fix that is to revert back to the SD card and also update that file system with apt-get upgrade.

My original SD card appears to be corrupted (which may have been the initial problem) so I can't boot from it. I installed Raspberry Pi OS (2021-05-07-raspios-buster-armhf-full) on a new card, edited /boot/cmdline.txt, and now it boots and reads from the external HDD as before.

There is a new problem though: I see "Failed to start Load Kernel Modules" in the bootup messages. lsmod shows that no modules were loaded. journalctl shows "systemd-modules-load: could not open moddep file '/lib/modules/5.10.17-v7+/modules.dep.bin'.

uname -a shows "linux rpi1 5.10.v7+ #1414 SMP Fri Apr 30 13:18:35 BST 2021 armv7l GNU/Linux". ls /lib/modules shows the following folders: 4.19.66+, 4.19.66-v7+, 5.4.58+, 5.4.58-v7+.

uname -a on a similar Rpi shows "Linux rpi2 5.4.47-v7+ #1322 Wed Jun 17 17:52:39 BST 2020 armv7l GNU/Linux" and ls /lib/modules is the same as on rpi1.

For reference: rpi1 is the one that uses the external HDD and now has kernel module issues. rpi2 is the one that runs entirely from the SD card and is working normally.

I have recently run apt update and apt upgrade on rpi1 already. It looks to me like I need to revert to the 5.4.47 kernel that I had been using on the old SD card, or upgrade the contents of /lib/modules. I have tried copying the files from /boot.bak into /boot which then makes the system unbootable again. I haven't been able to find any similar questions on SE other than this. Would it be possible to copy whatever kernel-related files are needed from rpi2 or rpi1, or is there a better way to fix this?

Answer 1

"In normal circumstances there is NEVER a need to run rpi-update as it always gets you to the leading edge firmware and kernel and because that may be a testing version it could leave your RPi unbootable". https://www.raspberrypi.org/forums/viewtopic.php?p=916911#p916911 Even the rpi-update documentation now warns "Even on Raspbian you should only use this with a good reason. This gets you the latest bleeding edge kernel/firmware."

sudo apt update; sudo apt install --reinstall libraspberrypi0 libraspberrypi-{bin,dev,doc} raspberrypi-bootloader raspberrypi-kernel will put it back to the latest supported kernel/bootcode.

A simple rpi-update downloaded the module dependencies to match the new kernel and that fixed everything. If only everything was always this easy...