How to verify the correctness of a kernel upgrade before performing the final reboot

Question

I remotely manage a couple of raspberry-pi's as vpn servers using ssl over openvpn.

One of these pi's is becoming problematic: it recently somehow gets misconfigured after almost every raspberrypi-kernel or -bootloader upgrade.

Since the problem until now happened using Stretch, I was hoping that a fresh install of Buster on another sd card would fix it, but unfortunately, already the first (remote) upgrade after the fresh install failed similarly.

The upgrade is performed on a Raspberry Pi 2 Model B using the normal apt-get upgrade procedure

There were no related errors/warnings reported on screen during the upgrade

Before the reboot I perform the following checks:

debsums -c raspberrypi-kernel
debsums -c raspberrypi-bootloader
check for errors in /var/log/apt/term.log

check dates and md5sums in /boot
check dates and md5sums in /boot/overlays
check dates and md5sums in /var/cache/apt/archives/raspberrypi-*

and everything is fine

Note: I get the dates and md5sums from a similarly installed pi

The upgrade was supposed to bring the kernel from 4.19.58 to 4.19.66

After the reboot however,uname -a still returns 4.19.58

This preventing the openvpn server from starting up properly, and consequently I have to go through several hoops to get access to the problem pi over ssl (the ssl server fortunately still starts up)

The /lib/modules folder contains the (new) 4.19.66 subfolders

The md5sums of some files in /boot and /boot/overlays do not match any more

and running debsums on raspberrypi-(bootloader kernel) shows problems

It happens on both on Stretch and Buster, and I have used another sd card for the Buster installation. However, as far as I know, all 'code' comes from the sd card and nothing is written to the board...

Is there more that I could check before the reboot to predict problems?

Any hints on how to proceed would be welcome.

Answer 1

Although not a direct answer to this question, this problem has been solved and the answer may help other users having problems with kernel and/or bootloader upgrades.

In this specific case, the problem was caused by an additional USB card reader inserted into one of the free USB ports of the remote Pi, containing a 'perfect' clone of the master SD card from which the Pi boots.

Being a 'perfect' clone, the UUID's of the partitions on the master SD card and the partitions on the cloned SD card were identical. This 'confused' the boot process, which, after booting from the master SD card, mounted the boot partition of the cloned SD card on /boot, while the root partition of the master SD card was mounted on '/'.

During an upgrade of raspberrypi-kernel and bootloader, the boot partition on the cloned SD card would get updated instead of the boot partition of the master SD card.

For more information see: How does the pi select the device to boot from?

In my case, I worked around the issue by not using the UUID's as partition identifiers, but instead using the partition names (/dev/mmcblk0p1 and /dev/mmcblk0p2) as identifiers in /etc/fstab and /boot/cmdline.txt

It would also have been also possible to simply change the UUID's of the cloned SD card. Note however, that the UUID's to be used as identifiers are the partition UUID's, which can be displayed using: lsblk -o name,partuuid