Recover Raspberry Pi 3B+ / Raspbian Buster powered off during upgrade

Question

Edit - I checked the version and the original SD A was actually at Buster. SD B is at Buster as well.

My Raspberry Pi 3 B+ was turned off while running sudo apt-get upgrade. The SD card (SD card A) is fine, the PI is fine, but it hangs at the rainbow screen on boot.

I flashed a new card with base Raspbian Buster (SD card B) and booted the Pi using that with SD Card A in a USB reader. Can see SD Card A file systems (rootfs, boot) and open in the terminal, read files, etc.

I don't really know what I am doing with fsck but after some reading did the following based on How to recover files from pi SD card which can't boot

# have RPi 3B+ booted with SD B and original SD A card that was powered off during upgrade in USB reader slot

pi@raspberrypi:/media/pi $ mount 
/dev/sdb2 on /media/pi/rootfs type ext4 (rw,nosuid,nodev,relatime,uhelper=udisks2)
/dev/sdb1 on /media/pi/boot type vfat (rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2)


pi@raspberrypi:/media/pi $ sudo umount /dev/sdb2
pi@raspberrypi:/media/pi $ sudo umount /dev/sdb1

pi@raspberrypi:/media/pi $ sudo e2fsck /dev/sdb2;
e2fsck 1.44.5 (15-Dec-2018)
rootfs: clean, 208916/1903104 files, 4906295/7724160 blocks

pi@raspberrypi:/media/pi $ sudo fsck /dev/sdb1
fsck from util-linux 2.33.1
fsck.fat 4.1 (2017-01-24)
/dev/sdb1: 113 files, 48952/516191 clusters

To me, this implies SD A is OK card and filesystem integrity-wise, and I might be able to recover it by cleaning up or re-initializing the card with Raspbian Stretch.

I've been searching for a couple of hours but Google is just incoherent in terms of how to do this - most of the advice for searches like "repair Raspberry PI SD card after power off during upgrade" just talk about corrupted SD card or not pulling the power while it's running, which isn't this scenario.

I have a number of sensors attached to this Pi and drivers/custom code installed and if there's any way I can recover SD A I'd like to do that. I can just copy the files to SD B but I'm afraid I'll just copy the corrupted to SD B and then it won't work either.

Can I edit files on SD A to reset the in-progress upgrade, or find out what's wrong?

Answer 1

Because you booted with a fresh flashed new SD Card (SD B) and can mount the corrupted SD Card (SD A) with a card reader, it is a good condition to chroot into the root file system of SD A and do anything to repair it. I will explain how you can do it. But very first make a backup of SD A and only use the copy.

We boot with SD B and use its running kernel to mount and enter the operating system on SD A. It may work with Buster on SD B but it is better to use the same version Stretch we will later enter on SD A. Because we only use the commandline you should use the Light version, so download Raspbian Stretch Lite, flash it to SD B and boot it.

Now you are logged in and find the attached SD A as sda* with (example):

rpi ~$ lsblk
NAME                    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                       8:0    1  7.5G  0 disk
├─sda1                    8:1    1 43.2M  0 part
└─sda2                    8:2    1  7.5G  0 part
mmcblk0                 179:0    0  3.7G  0 disk
├─mmcblk0p1             179:1    0  256M  0 part /boot
└─mmcblk0p2             179:2    0  3.4G  0 part /

We will mount its partitions and chroot into it with these commands:

rpib ~$ sudo -Es   # be user root
rpib ~# mkdir /mnt/p2
rpib ~# mount /dev/sda2 /mnt/p2
rpib ~# mount /dev/sda1 /mnt/p2/boot
rpib ~# cd /mnt/p2
# provide virtual system files from the running kernel (SD B)
rpib ~# mount --bind /proc proc
rpib ~# mount --bind /sys sys
rpib ~# mount --bind /dev dev
rpib ~# mount --bind /dev/pts dev/pts
rpib ~# mkdir run/udev
rpib ~# mount --bind /run/udev run/udev

Now we chroot into the root filesystem of SD A:

rpib ~# LANG=C.UTF-8 chroot /mnt/p2 /bin/bash
rpia ~#

Here you are now in your corrupted root filesystem of SD A and you can do whatever is needed to repair it. First I would try to complete update/upgrade. Have in mind that you are still user root:

rpia ~# apt update
rpia ~# apt full-upgrade

If finished with repair, then exit to SD B and reboot:

rpia ~# exit
rpib ~# systemctl reboot

Good luck :-)

Answer 2

You should really have a plan for the event when the SD card dies completely. Typically, you want to know which files you have modified in order to run your project, and keep an up-to-date backup of them, so that in the worst case you can put them on a freshly flashed SD card and get your project going again.

Since your old SD card is OK from the file system point of view, you can mount it, make a copy of all the files, and then carry over files you have modified to a new SD card, as well as on a separate folder / archive for backup. Then you try to boot with that new SD card and get the project running. If it doesn't run, you still have files to move, and you have the old card as a reference.

This is going to take time, but in the end you will have a complete backup of your project which is much smaller than a full SD card image and which (unlike a full image) can be put under version control. Which means you will be able to freely test new changes, knowing that if you don't like them, you can always undo them later.