Raspbian: dm_mod missing after mkinitramfs on luks encrypted partition

Question

It appears that after a kernel update happened, and rebooted for, Raspbian will no longer boot. (RPi4) After entering the (correct) encryption key when prompted, the mount fails with error:

Cannot initialize device-mapper. Is dm_mod kernel module loaded?

If I CTRL+C enough I can drop to a BusyBox shell and attempt to cryptsetup manually, but that has the same problem, even after trying modprobe dm_mod, so I don't think that the device mapper kernel module is being included in the initramfs at all.

I can remove the SD card and mount it without issue in a different Debian PC, and am able to read/write the file system, including installing packages via apt and rebuilding the initramfs. The process I'm using to do this is:

# replace /dev/sdf with your SD card block device
# make sure you do that for the two `sed` calls for "/etc/crypttab" too!

# mount
cryptsetup -v luksOpen /dev/sdf2 sdcard
mount /dev/mapper/sdcard /mnt; mount /dev/sdf1 /mnt/boot; mount -o bind /dev /mnt/dev; mount -o bind /dev/pts /mnt/dev/pts; mount -t sysfs none /mnt/sys; mount -t proc  none /mnt/proc

# comment out ld.so.preload
sed -i 's/^/#/g' /mnt/etc/ld.so.preload
# crypttab needs to point to the current physical device during mkinitramfs or cryptsetup won't deploy
sed -i 's/mmcblk0p/sdf/g' /mnt/etc/crypttab

# copy qemu binary
cp /usr/bin/qemu-arm-static /mnt/usr/bin/

# chroot to Raspbian to update and rebuild initramfs
chroot /mnt /bin/bash
    rm -rf /var/tmp/mkinitramfs*
    apt update && apt upgrade && apt dist-upgrade && apt autoremove
    test -L /sbin/fsck.luks || ln -s /sbin/e2fsck /sbin/fsck.luks
    mkinitramfs -o /boot/initramfs.gz -k $(ls -t /lib/modules | tail -1)
    exit

# undo damage
sed -i 's/^#//g' /mnt/etc/ld.so.preload
sed -i 's/sdf/mmcblk0p/g' /mnt/etc/crypttab

# unmount
sync
umount /mnt/{dev/pts,dev,sys,proc,boot} /mnt
cryptsetup -v luksClose sdcard

These instructions have gone through revisions to try and resolve this. Most notably the "crypttab" replacement from "mmcblk0p" to "sdf", which is only there so when mkinitramfs runs within a chroot the encrypted partition is found. Otherwise the process shows "cryptsetup: ERROR: Couldn't resolve device /dev/mmcblk0p2". (Either value give the same error when really trying to boot.)

The boot command line "/boot/cmdline.txt" refers to the encrypted partition correctly:

console=serial0,115200 console=tty1 root=/dev/mapper/sdcard cryptdevice=/dev/mmcblk0p2:sdcard rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet plymouth.ignore-serial-consoles

Kernel version going on is "4.19.97-v8+" though I've tried "4.19.97+", "4.19.97-v7+", and "4.19.97-v7l+" too.

Other than a reinstall, would you have any advice on building a good initramfs to resolve this encryption issue?

Related questions:

• https://unix.stackexchange.com/questions/327914/initramfs-luks-and-dm-mod-cant-boot-after-upgrade
• Pi won't boot after initramfs update

Answer 1

Another solution for this problem (besides the comments under the question) which helped in my case on a raspberry 3b+ running on the raspbian OS:

sudo apt-get install --reinstall --yes raspberrypi-bootloader raspberrypi-kernel
sudo reboot

Taken from this thread: https://www.raspberrypi.org/forums/viewtopic.php?t=282802

my cryptsetup command worked afterwards, while having the same error like OP stated: Cannot initialize device-mapper. Is dm_mod kernel module loaded? previously.

Answer 2

This happened again today. Fortunately I had kept the script from previously, with the small addition of @ElectRocnic's apt install --reinstall --yes raspberrypi-bootloader raspberrypi-kernel.

VERY IMPORTANT: It mattered to ensure that the correct kernel module was given to mkinitramfs, because apparently this doesn't automatically happen during regular updates. Always look in "/lib/modules" yourself as $(ls -t /lib/modules | tail -1) doesn't always give you the right one. Not doing this will give you an error about "dm_mod" module not being loaded when entering your volume passphrase.

Full script is as follows:

# Replace /dev/sdg with your SD card block device.
# Use `lsblk` to help you find where this is.

# Link "mmcblk" block device so chroot later sees the expected device block names.
test -L /dev/mmcblk0p1 || ln -s /dev/sdg1 /dev/mmcblk0p1
test -L /dev/mmcblk0p2 || ln -s /dev/sdg2 /dev/mmcblk0p2

# Mount SD card.
cryptsetup -v luksOpen /dev/mmcblk0p2 sdcard
mount /dev/mapper/sdcard /mnt; mount /dev/mmcblk0p1 /mnt/boot; mount -o bind /dev /mnt/dev; mount -o bind /dev/pts /mnt/dev/pts; mount -t sysfs none /mnt/sys; mount -t proc  none /mnt/proc

# Comment out ld.so.preload.
sed -i 's/^/#/g' /mnt/etc/ld.so.preload

# Copy qemu binary.
cp /usr/bin/qemu-arm-static /mnt/usr/bin/

# chroot to Raspbian to update and rebuild initramfs.
chroot /mnt /bin/bash
    rm -rf /var/tmp/mkinitramfs*
    apt update && apt upgrade && apt dist-upgrade && apt autoremove
    apt install --reinstall raspberrypi-bootloader raspberrypi-kernel
    test -L /sbin/fsck.luks || ln -s /sbin/e2fsck /sbin/fsck.luks
    update-initramfs -u
    # IMPORTANT: Replace "5.10.52-v7l+" with the correct kernel from "/lib/modules".
    # Doing `$(ls -t /lib/modules | tail -1)` here doesn't always give you the right one!!!
    mkinitramfs -o /boot/initramfs.gz 5.10.52-v7l+
    exit

# Undo damage to our local recovery system.
sed -i 's/^#//g' /mnt/etc/ld.so.preload

# Force flushing write buffers and unmount SD card.
sync
umount /mnt/{dev/pts,dev,sys,proc,boot} /mnt
cryptsetup -v luksClose sdcard

# Unlink "mmcblk" block device.
test ! -L /dev/mmcblk0p1 || rm /dev/mmcblk0p1
test ! -L /dev/mmcblk0p2 || rm /dev/mmcblk0p2

# All done. Eject the SD card and attempt boot in your Pi.

Did this from another Linux system, worked as normal.

N.B. The step to forcibly reinstall "raspberrypi-bootloader" and "raspberrypi-kernel" takes sodding ages. :) Didn't time it exactly, but it was about half an hour.