7

I'm developing a Buildroot OS for the Raspberry Pi, and my workflow requires very frequent re-flashing of the SD card to test new iterations. The process of removing the SD card from the Pi, flashing it on a Windows PC and then re-inserting it takes a lot of time. I would like to write a script that uses the currently running OS on the Rpi (accessed over SSH) to

  1. Download the new SD image
  2. Flash it to the SD, overwriting all existing OS files
  3. Reboot into the new OS

Step 2 is where I'm stuck. Is it possible for an operating system to overwrite itself?

Improve this question
4
  • 5
    You may consider to use net booting for development.
    – Ingo
    Commented Jun 25, 2019 at 15:01
  • This is a comment because I don't know if it would work, but can you have two partitions on the SD card, write to the second partition and then boot from that new partition next time? As you iterate through changes, you're just booting from one partition or the other?
    – JPhi1618
    Commented Jun 25, 2019 at 16:17
  • @JPhi1618 Yes, that's possible, except you would not want to write a (normative) Pi OS image to the other partition -- you'd want to write the contents of the root filesystem, which you can extract from the image. That requires a bit of manual (or scriptable) tinkering. Then you'd want to update the boot partition, and set root= in cmdline.txt.
    – goldilocks ♦
    Commented Jul 24, 2019 at 14:05
  • The easiest way is to use a usb sd card reader/writer.
    – NomadMaker
    Commented Feb 12, 2020 at 21:59

6 Answers 6

Reset to default
11

To overwrite itself, your operating system must run fully in RAM and neither read nor write from SD card after booting. piCore/TinyCore is probably one of the few Linux OS which can do that.

Improve this answer
13
  • Is is necessary that the OS never read/write to SD, or merely that is doesn't require the SD during the dd operation?
    – Jeremiah Rose
    Commented Jun 25, 2019 at 7:11
  • 2
    @JeremiahRose If anything else writes to a part of the SD card dd is done with, it's gonna screw up the image.
    – UTF-8
    Commented Jun 25, 2019 at 16:23
  • 3
    @JeremiahRose Strictly speaking, as long as you can completely certainly guarantee that absolutely no modification of the SD card will happen while dd is running, then it's fine if it modifies the SD card outside of that. Practically speaking, just go with an OS that never modifies the SD card; it'll be simpler to reason about.
    – anon
    Commented Jun 25, 2019 at 16:53
  • 2
    "while dd is running" -- not just while it's running, after it's finished. The OS is likely to cache various filesystem metadata, such as where files live on the SD card. If, after dd is done running, it decides to write to one of these files (which are no longer where the OS thinks they are), it'll overwrite some other arbitrary data instead.
    – Roger Lipscombe
    Commented Jun 25, 2019 at 17:35
  • 1
    An OS running purely in RAM is a straightforward and general purpose solution, but in some way I'm a bit disappointed if there doesn't exist a program that would just lock itself and the image file to memory, prevent other processes from running (setting itself to run with real-time scheduling or something) and then write the image and reboot the system.
    – ilkkachu
    Commented Jun 25, 2019 at 19:41
6

As pointed out elsewhere, dd-ing over a running OS is never a good idea. An alternative (and the usual way you'd develop embedded systems) is to have your OS image served from a network drive and the target board perform a network boot. This speeds up the initial development and you only need to write out the SD cards as you get closer to completion and need to test on actual hardware.

There's a write up on setting the PI3B and PI3B+ to boot from a server on the Foundation pages: https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net_tutorial.md

Improve this answer
3
  • 1
    In my case, dd-ing over the running OS has caused absolutely no problems at all
    – Jeremiah Rose
    Commented Jun 25, 2019 at 9:19
  • 8
    Beware this is a bit like saying you drive drunk often and have no problems at all, or take the stairs in the dark all the time and have no problems at all, etc. It is a bit different than that those things, though, in that when it goes wrong you won't have any opportunity to react to it. OTOH, you're not going to get injured or killed, so "it works until it doesn't" maybe fine with your workflow.
    – goldilocks ♦
    Commented Jun 25, 2019 at 13:39
  • This is similar to the technique people use to run utilities like nwipe for securely erasing an entire hard disk: github.com/martijnvanbrummelen/nwipe
    – user105131
    Commented Jun 25, 2019 at 17:54
3

Use an SD card large enough to hold multiple partitions

The SD card should hold a boot partition (0) and three OS partitions (1-3).

The boot loader on the boot partition would examine the partition labels to determine what to load. Suppose the labels are:

0: BOOT
1: OS-0004!
2: OS-0002
3: OS-0003

The loader knows to load the OS from partition 1, as it's the last label when they're sorted. And the first OS label is partition 2, to which it writes the update

2: OS-0005_

This is now the last label, but the underscore flag tells the loader it's the first attempt to boot from it. It changes the label to indicate it's doing a test boot right before it chains to it:

2: OS-0005? If the OS successfully boots and passes its own sanity checks, it updates the label one more time, as well as the label of the previous OS partition: 1: OS-0004 2: OS-0005!

If it fails, the next time it tries to boot, it will see the ? flag and revert to load from partition 1, whereupon the update will start over.

Improve this answer
2
  • This seems to be an interesting outline, but do you have any specifics how to do this?
    – RalfFriedl
    Commented Jun 26, 2019 at 5:20
  • I haven't done it myself, but I know that boot loaders are capable of this sort of thing. The Google Chromebook system is similar, in that it has a boot partition, at least two OS partitions, and a user-space partition, so that updates are always applied to the OS partition the system did not boot from.
    – Monty Harder
    Commented Jun 26, 2019 at 17:44
2

Here a rough approach:

  1. Replace /sbin/init with upgrade tool.
  2. Tell init to re-exec itself.
  3. Kill all other remaining processes (for example with kill -9 -1)
  4. Use pivot_root and chroot to replace the root filesystem.
  5. If necessary exec the next stage to drop references the old root.
  6. Recursively unmount the old root.
  7. Write the new image.
  8. Reboot.
Improve this answer
2

Use with caution

This works for me because I am using a read-only root filesystem with a custom Buildroot OS. This script hasn't been tested on Raspbian yet, but will probably work.

#!/bin/bash
set -e
echo "Copying image to RPi..."
sshpass -p PASSWORD scp /PATH/TO/SDCARD.img USER@RPI:/tmp/sdcard.img
echo "Flashing image to /dev/mmvblk0 on RPi..."
sshpass -p PASSWORD ssh USER@RPI 'nice -20 sudo sh -c "sync && dd if=/tmp/sdcard.img of=/dev/mmcblk0 conv=fdatasync && reboot -f"'

This should be run on the host computer (not the RPi) and does the following:

  1. Copies the SD image to the RPi via scp
  2. Logs into the RPi via ssh
  3. Syncs all pending OS write operations to SD card
  4. Sets priority to -20 to prevent other processes from running
  5. Flashes the image to the SD card followed by a fdatasync
  6. Performs a hard reset

Limitations:

  • If data is written to the SD card by another process while the script is running, the SD may become corrupted. In a few days of active use, I haven't encountered this yet - but I am not using Raspbian.
  • Make sure you only use this script for development/testing, not for upgrading your OS or any other data-sensitive applications.
  • Could use some improvements to further prevent other processes from accessing the SD card - suggestions welcome.

To use:

  1. Copy the above script into a flash.sh file on your host PC
  2. Fill in your settings for password, username, SD image etc
  3. Make it executable with chmod a+x flash.sh
  4. Install sshpass and ssh on host: sudo apt install sshpass openssh
  5. Make sure ssh login is working on the RPi
  6. Run ./flash.sh
Improve this answer
2
  • 1
    There are cases where the image is heavier than the memory of the raspy, what could we do in those cases?
    – user111682
    Commented Nov 28, 2019 at 12:34
  • Yeah, those cases sound impossible to work around
    – Jeremiah Rose
    Commented Feb 25, 2020 at 0:46
0

I've been thinking about something similar for testing development/deployment from a fresh install of Raspbian. My current thinking is a dual sd-card, usb-drive set up. So on each boot, the pi first boots into the sd-card, flashes a fresh install to the usb drive, and then chroots into the usb-drive to start the os.

Though as this is only text, i've only thought about it conceptually while manually flashing sd card's with other devices. So I guess I should start looking into implementation..

Improve this answer
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.