Failed to start Load Kernel Modules after rsync restore

Question

After restoring the whole raspbian system using rsync, I get following errors during boot.

systemd[1]: systemd-modules-load.service: Main process exited, code=exited, sta...
systemd[1]: systemd-modules-load.service: Failed with result ´exit-code´.
systemd[1]: Failed to start Load Kernel Modules.

After this also netfilter-persistent.service and hostapd.service fail to start.

What got corrupted here and how can I correct this?

Following are the steps how I do the backup.

I followed the answer to Can a Raspberry Pi be used to create a backup of itself? backup my running raspbian system. Instead of a USB drive, I use a rsync server for backup. Here is my setup and the commands I use.

Configuration of my rsync server:

$ cat /etc/rsyncd.conf
#global (globale einstellung)
log file = /var/log/rsync.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
timeout = 300

[rpi]
path = /mnt/sdb1/backups/rpi
comment = backup
max connections = 1
hosts allow = 192.168.0.2
hosts deny = *
use chroot = yes
list = true
uid = root
gid = root
read only = false
auth users = user
secrets file = /etc/rsyncd.secrets

On my Raspberry I execute following command to start the backup:

sudo rsync -aHv --numeric-ids --delete --exclude-from=/home/user/rsync-exclude.txt --password-file=/home/user/rsync-pass / rsync://[email protected]/rpi/rsync_backup/

Content of rsync-exclude.txt

$ cat rsync-exclude.txt
/proc/*
/sys/*
/dev/*
/boot/*
/tmp/*
/run/*
/mnt/*
/media/*
/var/swap
/var/log/*

To restore the content on the sdcard, I insert the Raspberrie's sdcard in the rsync server and mount the 2nd partition on this sdcard as /mnt/sdcard. Afterwards I execute following command to start the restore.

sudo rsync -av --numeric-ids --delete-during /mnt/sdb1/backups/rpi/rsync_backup/ /mnt/sdcard/

When booting from this sdcard, I get the above described errors.

Update: I now updated my procedure so that also the boot partition is backed up and restored.

Please see the following snippets for error investigation.

$ uname -r
4.19.97+

$ ls -l /lib/modules
total 24
drwxr-xr-x 3 root root 4096 Feb 20  2019 4.14.79+
drwxr-xr-x 3 root root 4096 Mar  8 11:53 4.14.98+
drwxr-xr-x 3 root root 4096 Apr 17 19:50 4.19.97+
drwxr-xr-x 3 root root 4096 Apr 17 19:50 4.19.97-v7+
drwxr-xr-x 3 root root 4096 Apr 17 19:50 4.19.97-v7l+
drwxr-xr-x 3 root root 4096 Apr 17 19:50 4.19.97-v8+

Answer 1

The error described in my question/issue above was caused by issues of kernel files located in the boot partition. Originally I only backed up the files on the "os" partition of the Raspberry. So if you update the kernel or parts of the kernel and restore the files on the "os" partition, the kernel files on the "os" partition are not compatible with the files on the boot partition (or /boot folder). The files on the "os" partition are from a newer kernel version than on the boot partition (or vice versa).

So the solution is: Do not only backup the files from "os" partition. Include also the files from the boot partition (mounted as /boot) in your backup.

So here is the necessary configuration and the necessary steps:

Configuration of my rsync server:

$ cat /etc/rsyncd.conf
#global (globale einstellung)
log file = /var/log/rsync.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
timeout = 300

[rpi]
path = /mnt/sdb1/backups/rpi
comment = backup
max connections = 1
hosts allow = 192.168.0.2
hosts deny = *
use chroot = yes
list = true
uid = root
gid = root
read only = false
auth users = user
secrets file = /etc/rsyncd.secrets

On my Raspberry I execute following command to start the backup:

sudo rsync -aHv --numeric-ids --delete --exclude-from=/home/user/rsync-exclude.txt --password-file=/home/user/rsync-pass / rsync://[email protected]/rpi/rsync_backup/
sudo rsync -aHv --numeric-ids --delete --password-file=/home/user/rsync-pass /boot/ rsync://[email protected]/rpi/rsync_backup_boot/

Content of rsync-exclude.txt

$ cat rsync-exclude.txt
/proc/*
/sys/*
/dev/*
/boot/*
/tmp/*
/run/*
/mnt/*
/media/*
/var/swap
/var/log/*

To restore the content on the sdcard, I insert the Raspberrie's sdcard in the rsync server and mount the boot partition as /mnt/rpi_boot. The 2nd partition ("os" partition) on this sdcard is mounted as /mnt/rpi. Afterwards I execute following command to start the restore.

sudo rsync -av --numeric-ids --delete-during /mnt/sdb1/backups/rpi/rsync_backup/ /mnt/rpi/
sudo rsync -av --numeric-ids --delete-during /mnt/sdb1/backups/rpi/rsync_backup_boot/ /mnt/rpi_boot/