1

Edit - I checked the version and the original SD A was actually at Buster. SD B is at Buster as well.

My Raspberry Pi 3 B+ was turned off while running sudo apt-get upgrade. The SD card (SD card A) is fine, the PI is fine, but it hangs at the rainbow screen on boot.

I flashed a new card with base Raspbian Buster (SD card B) and booted the Pi using that with SD Card A in a USB reader. Can see SD Card A file systems (rootfs, boot) and open in the terminal, read files, etc.

I don't really know what I am doing with fsck but after some reading did the following based on How to recover files from pi SD card which can't boot

# have RPi 3B+ booted with SD B and original SD A card that was powered off during upgrade in USB reader slot

pi@raspberrypi:/media/pi $ mount 
/dev/sdb2 on /media/pi/rootfs type ext4 (rw,nosuid,nodev,relatime,uhelper=udisks2)
/dev/sdb1 on /media/pi/boot type vfat (rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2)


pi@raspberrypi:/media/pi $ sudo umount /dev/sdb2
pi@raspberrypi:/media/pi $ sudo umount /dev/sdb1

pi@raspberrypi:/media/pi $ sudo e2fsck /dev/sdb2;
e2fsck 1.44.5 (15-Dec-2018)
rootfs: clean, 208916/1903104 files, 4906295/7724160 blocks

pi@raspberrypi:/media/pi $ sudo fsck /dev/sdb1
fsck from util-linux 2.33.1
fsck.fat 4.1 (2017-01-24)
/dev/sdb1: 113 files, 48952/516191 clusters

To me, this implies SD A is OK card and filesystem integrity-wise, and I might be able to recover it by cleaning up or re-initializing the card with Raspbian Stretch.

I've been searching for a couple of hours but Google is just incoherent in terms of how to do this - most of the advice for searches like "repair Raspberry PI SD card after power off during upgrade" just talk about corrupted SD card or not pulling the power while it's running, which isn't this scenario.

I have a number of sensors attached to this Pi and drivers/custom code installed and if there's any way I can recover SD A I'd like to do that. I can just copy the files to SD B but I'm afraid I'll just copy the corrupted to SD B and then it won't work either.

Can I edit files on SD A to reset the in-progress upgrade, or find out what's wrong?

2
  • 1
    The normal fix for a failed upgrade is to restore from backup. If you can't boot you are unlikely to be able to repair, but if determined you should examine the boot logs. – Milliways Mar 25 '20 at 5:55
  • I don't know how to examine the boot logs from SD Card A when it's mounted as an external file system. – lonstar Mar 25 '20 at 20:28
2

Because you booted with a fresh flashed new SD Card (SD B) and can mount the corrupted SD Card (SD A) with a card reader, it is a good condition to chroot into the root file system of SD A and do anything to repair it. I will explain how you can do it. But very first make a backup of SD A and only use the copy.

We boot with SD B and use its running kernel to mount and enter the operating system on SD A. It may work with Buster on SD B but it is better to use the same version Stretch we will later enter on SD A. Because we only use the commandline you should use the Light version, so download Raspbian Stretch Lite, flash it to SD B and boot it.

Now you are logged in and find the attached SD A as sda* with (example):

rpi ~$ lsblk
NAME                    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                       8:0    1  7.5G  0 disk
├─sda1                    8:1    1 43.2M  0 part
└─sda2                    8:2    1  7.5G  0 part
mmcblk0                 179:0    0  3.7G  0 disk
├─mmcblk0p1             179:1    0  256M  0 part /boot
└─mmcblk0p2             179:2    0  3.4G  0 part /

We will mount its partitions and chroot into it with these commands:

rpib ~$ sudo -Es   # be user root
rpib ~# mkdir /mnt/p2
rpib ~# mount /dev/sda2 /mnt/p2
rpib ~# mount /dev/sda1 /mnt/p2/boot
rpib ~# cd /mnt/p2
# provide virtual system files from the running kernel (SD B)
rpib ~# mount --bind /proc proc
rpib ~# mount --bind /sys sys
rpib ~# mount --bind /dev dev
rpib ~# mount --bind /dev/pts dev/pts
rpib ~# mkdir run/udev
rpib ~# mount --bind /run/udev run/udev

Now we chroot into the root filesystem of SD A:

rpib ~# LANG=C.UTF-8 chroot /mnt/p2 /bin/bash
rpia ~#

Here you are now in your corrupted root filesystem of SD A and you can do whatever is needed to repair it. First I would try to complete update/upgrade. Have in mind that you are still user root:

rpia ~# apt update
rpia ~# apt full-upgrade

If finished with repair, then exit to SD B and reboot:

rpia ~# exit
rpib ~# systemctl reboot

Good luck :-)

2
  • Thanks! I will try this procedure this weekend and report back. – lonstar Mar 25 '20 at 20:29
  • I tried this - all of the steps went fine (SD A was already mounted at /media/pi) but SD A will still not boot - just hangs at rainbow screen. I tried SD A in another Pi and same result - hang at rainbow screen. I will need to go forward with copying SD A's project files and rebuilding the libraries, etc on SD B. Thanks for your help, appreciate the solution and it seemed like it would work. – lonstar Apr 10 '20 at 12:25
1

You should really have a plan for the event when the SD card dies completely. Typically, you want to know which files you have modified in order to run your project, and keep an up-to-date backup of them, so that in the worst case you can put them on a freshly flashed SD card and get your project going again.

Since your old SD card is OK from the file system point of view, you can mount it, make a copy of all the files, and then carry over files you have modified to a new SD card, as well as on a separate folder / archive for backup. Then you try to boot with that new SD card and get the project running. If it doesn't run, you still have files to move, and you have the old card as a reference.

This is going to take time, but in the end you will have a complete backup of your project which is much smaller than a full SD card image and which (unlike a full image) can be put under version control. Which means you will be able to freely test new changes, knowing that if you don't like them, you can always undo them later.

3
  • Thanks for the overview - I am not too experienced with Raspbian and I understand the flow of what you're describing but I am worried that just copying SD A -> SD B will wind up with SD B unbootable as well. – lonstar Mar 25 '20 at 20:31
  • @lonstar My idea is to install fresh Raspbian image on SD B (will be bootable), then carry over your project-specific files from SD A to SD B (and to a backup). – Dmitry Grigoryev Mar 26 '20 at 6:17
  • I have libraries and configurations installed that I kind of "got to work" and don't have the steps to recreate, and since the Pi is OK (boots fine with SD B) and SD A is ok (can read the filesystem and fsck is OK), I want to restore the card to bootable condition over rebuilding the driver libraries and config on a new card. – lonstar Apr 10 '20 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.