6

I have a Pi running Raspbian in a headless configuration that has crashed a few times, once for unknown reasons, and once due to a power outage. In both cases, I had to connect a monitor and keyboard and "help" it boot up (once it required an extra fsck execution, and once it wouldn't boot because the clock was off and the timestamp in the superblock was in the future; setting the correct date/time fixed it).

I'd like to make this system as robust as possible so that it is less likely to require intervention. What steps can help me achive this?

  • Sounds like ext2 and not ext3. A journaling file system is much more robust against unexpected shutdowns. – Thorbjørn Ravn Andersen Jun 8 '13 at 8:26
  • Parted reports it as ext4; when I ran fsck one of the times it mentioned that it was applying journaled changes (don't remember exact language). – TomG Jun 8 '13 at 11:51
  • The errors you mention sounds like it should be fixed upstream. I suggest you report it as bugs against the raspbian distribution. – Thorbjørn Ravn Andersen Jun 8 '13 at 15:37
  • I've opened a bug with Raspbian for this issue: bugs.launchpad.net/raspbian/+bug/1212020 – Florin Andrei Aug 13 '13 at 23:06
4

you might think about separating your system to the read-only partition and your data files somewhere else. read-only system is unlikely to get corrupted and after you have it up and running, you may start recovery scripts and/or login over SSH to fix the possible problems.

  • Although not mounted ro, it's pretty close to read only. As far as I can determine, there's only one file being written to on my data partition, and it has a log entry added once or twice a day; I'll be removing that soon, so mounting ro would certainly be a possibility. Not sure if I can get away with that for the root partition, though. – TomG Jun 8 '13 at 11:53
  • Accepting, as I think that a RO system partition is the only sure solution, any other solution will have some scenario where it won't boot. – TomG Mar 4 '14 at 2:36
14

What happens is that fsck gets run after so many reboots or when it suspects the disk is in trouble. The number of reboots can be changed with tune2fs, but that won't fix your problem. Working backwards to the solution, here's the correct option for fsck:

   -y     For some filesystem-specific checkers, the -y option will cause the fs-specific fsck  to
          always  attempt  to  fix any detected filesystem corruption automatically.  Sometimes an
          expert may be able to do better driving the fsck manually.  Note that not  all  filesys‐
          tem-specific   checkers   implement   this  option.   In  particular  fsck.minix(8)  and
          fsck.cramfs(8) do not support the -y option as of this writing.

What causes the fsck to happen? Well, there's an init file at /etc/init.d/checkfs.sh. Look at line 17:

[ "$FSCKFIX" ] || FSCKFIX=no

Later, line 67:

            if [ "$FSCKFIX" = yes ]
            then
                    fix="-y"
            else
                    fix="-a"

So FSCKFIX controls if fsck is called with -y. We want -y so you can have it check automagically while headless. The easiest way is to modify /etc/default/rcS around line 24 and enable FSCKFIX, like this:

Original: #FSCKFIX-no

Modify to: FSCKFIX=yes

After that, you're done!

  • +1, edit made. Waiting to see if any other suggestions come along before accepting. – TomG Jun 8 '13 at 2:17
  • checkfs.sh is not involved if using systemd, at least in jessie. Analog for systemd-fsck is to add parameter to kernel cmdline: fsck.repair=yes – Justinas Jaronis Mar 15 '17 at 11:18
3

If you feel like file corruption is one of your major problems, consider moving the entire OS away from the SD-card. On a Raspberry Pi 1 / 2 / Zero (W) you need at least some boot files on the SD-card but you can run the rest from a USB device. With a freshly downloaded image:

  • Copy the entire Raspbian image to a USB-device.
  • Copy (only!) the /boot/ partition to an SD-card.

Attention for the next 2 steps / edits: these paths are followed by a number. Don't edit or remove the number! (mmcbkl0p1 becomes sda1, mmcbkl0p2 becomes sda2, etc. ...).

  • Edit cmdline.txt in the /boot/ partition on the SD-card and change /dev/mmcblk0p to /dev/sda. Consider editing the same file in the /boot/ partition on the USB device so you have a ready-to-use backup of your /boot/ partition.
  • Edit /etc/fstab and change /dev/mmcblk0p to /dev/sda on the USB device.

Done! You can make the /boot/ partition your SD-card read-only (but there wouldn't be any writes anyway). Also, if the SD card would ever fail again, you have a backup of the /boot/ partition on your USB device.

Optional

If you have a Raspberry Pi 3 B, perform these extra steps (on top of the previous steps) to not need an SD-card at all.

  • Boot Raspbian (with the SD-card)
  • Update/upgrade your Raspbian OS
  • Edit /boot/config.txt on the SD-card, add a new line at the end with: program_usb_boot_mode=1
  • Reboot (Raspbian will enable USB boot mode during this boot cycle)

Done! Shut down, remove the SD-card from the Raspberry Pi 3 B and boot again. This will take longer than usual. Mine did a strange sequence on it's own: I put the power in, then it showed a color image, it rebooted on it's own, showed the color image again, rebooted on it's own again, showed the color image again and suddenly booted into Raspbian. The next reboots were much faster.

Make sure to remove the program_usb_boot_mode=1 line in /boot/config.txt on the SD-card if you plan on reusing the SD-card in other Raspberry Pi's that don't need USB boot mode enabled.

1

Power-related crashes are better handled with some sort of UPS solution like juice4halt.

For cases where unexpected shutdown cannot be prevented, I suggest you try data=journal option for your root partition. It will make writes slower (but as you said in comments your root is virtually rea-only so there will be little impact), and fsck will be able to fix most issues automatically upon restart.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.