3

I am trying to back up a 16GB SD card as it is proposed here: How do I backup my Raspberry Pi?

I have started with 2017-01-11-raspbian-jessie-lite.img and then did some work with FFmpeg (cloned & compiled the source code). The compilation takes quite a long time so I need to create a backup image to go back if needed.

Both dd and win32 Disk Imager generate a ~16GB .img file.

Question: Is there a way to generate a smaller file and not something that reaches the size of the SD card?

1
  • Use the RaspiOS built-in SD Card Copier that only copies the used space to make a backup.
    – Dougie
    Jan 31, 2021 at 22:51

9 Answers 9

5

Surprise! creating an image of a 16GiB SD Card creates a 16GiB image.

There are 2 options;

only backup files can-a-raspberry-pi-be-used-to-create-a-backup-of-itself

or compress the image https://raspberrypi.stackexchange.com/a/29085/8697 - this is a macOS script, but you can develop similar for other OS.

1
3

It's worse than you think == chances are your '16Gb' .img WONT WRITE BACK to another '16Gb' SD card !

1) The stupid Pi system autoexpands to every last block of the SD card on FIRST BOOT !! This is stupid because :-
2) No two SD cards have the same 'block count'. All cards have 'bad blocks' that are 'mapped out' during manufacturing. As compitition drives prices down, SD cards with more and more bad blocks are being 'pssssed' - so 'todays' '16Gb' card will be 15.9 or 15.8 or 15.7 or even less 'tomorrow' ...
3) Win32 disk imager WONT WRITE a 15.90 Gb .img to a 15.8999999 Gb card, full stop. Even if it did, the partition would 'run off the end of the card' and the Pi will abort with 'Corrupted file system' errors

a) You have to reduce the partition to 15.5 or so to 'guarantee' it will write back (BEST way is to MODIFY the .img on your PC BEFORE burning to SD so it doesn't 'autoexpand' on first boot-up (remove ' init= ..' from end of cmdline.txt on the FAT32 partition). Then, manually change /usr/lib/raspi-config/init_resize.sh so it ewxpands to 95% of the card, not 100% [find 'TARGET_END=$((ROOT_DEV_SIZE - 1))', change to
'TARGET_END=$((ROOT_DEV_SIZE / 100 \* 95 ))' ] then add ' init=/usr/lib/raspi-config/init_resize.sh' back to cmdline.txt and reboot .
b) Finally, DONT USE win32 imager to make your back-up .img (it will grab every block, you only want to the end partition, not the whole SD card). Instead (on the PC) use (free) DiskInternals Linux Reader .. (after which Win32 Imager will 'burn' the .img just fine to any card witb at least 95% of the space of the original)

Please note == This is what I ACTUALLY DO (with my PoE headless Pi Zero projects), and I can assure I have written my .img's to many different SD cards and it ACTUALLY WORKS (it seems that most 'guides' on 'How to back-up the Pi' are written by people who have never actually tried to 'restore' their .img to a new SD card :-) )

PS - or, if you are using a Pi B3, by all means start by 'shrinking' the end partition and then add a USB SD card reader/writer and play with 'dd' ... I don't do this because all my projects are based on the Pi Zero (which I usually run 'headless' with a USB/Ethernet 'dongle' on the end of a PoE cable)

1
  • 2
    Please could you edit your answer so its more clear, your answer is quite unreadable at the moment
    – Darth Vader
    Nov 11, 2017 at 15:01
2

Both DD and the disk imager will 'naively' just create an exact replica of the SD card, empty space and all hence why they are 16GB. These backups you can simply image directly onto SD card and its back where you left off.

If you want to snapshot your compilation process outcomes you might want to look at alternatives to entire backups. Using Docker containers would let you incrementally build on what you have done by committing the containers at each step and only pay for (in storage terms) the differences between each version. There is a recent Raspbian base image on the docker hub for the Pi which should give you an accurate enough base system, you can even copy your existing work directly into it to avoid starting from scratch. If you want to backup your work now, you only have to export your docker containers which will be Base Image size + changes, probably under 1gb. This way you can test anyway and if it goes south just bin the image and return to a known good state.

2
  • Thanks @tobyd! Yeah, I know about Docker and I have created some ffmpeg images, I just wanted to also test it with a pure installation on the RPi.
    – tgogos
    Jan 20, 2017 at 11:00
  • Ah, pehaps give the configure script the --prefix=PREFIX option and keep them in your home folder to avoid polluting the /usr ones and separate them that way? Should keep the purity but without having to maintain entire SD card images?
    – user29019
    Jan 20, 2017 at 11:10
2

Since nobody mentioned it yet, here is the tool I use to shrink images obtained by dd: https://github.com/Drewsif/PiShrink

The shrunk images are autoexpanded at first boot to cover the SD card they are dd'ed onto by default, which can be disabled.

0
1

I recommend using Partimage to create a backup of the partitions on your your SD card. From the documentation:

Partimage will only copy data from the used portions of the partition.

Partimage can also compress its output on-the-fly, to make the resulting output even smaller.

If you are making the copy on an x86 system, you can use Clonezilla instead. It will use a variety of tools to make an image of your entire SD card.

2
  • Hi David, thanks for the reply. I noticed that Partimage does not support ext4 and the main partition of the SD card is ext4. So this might not work.
    – tgogos
    Jan 20, 2017 at 15:47
  • Try Clonezilla. It can handle a longer list of partition types.
    – user46953
    Jan 20, 2017 at 15:56
1

An easy approach if you have less than about 7GB in use on your 16GB card is to use the "Copy SD Card" function to copy to an 8 GB SD card. That will give you an 8 GB image instead of 16. Other answers will give you a smaller image file, but this is the easiest.

0

ApplePi-Baker (Mac only) now supports shrinking .img files during SD card backup (and expansion during restore): ApplePi-Baker v2 – Backup & Restore SD cards, USB drives, etc. - Shrink IMG Process

Unfortunately the .img is expanded prior to writing back to the memory card, so it actually takes longer overall - but at least the .img backup file is smaller for storage and transmission.

0

I use a three step process:

  1. Shrink the root partition of the SD card to the size of its contents using parted (gparted) on a linux machine (such as a rPi)
  2. Copy it to an image
  3. Flash the image to a SD card using Raspberry Pi Imager or similar.

For example, in a fit of insanity I used a 256G SD card and discovered it was effectively impossible to back up. Using parted I shrank it to about 20G - leaving the remainder as unallocated space ( later you can expand into it or make it another partition, at your discretion).

Now you have a manageable size SD card that you can clone using dd or Win32DiskImager or equivalent.

  • If you use dd (linux) then clone enough of the disk to contain your partition. For example, if your partitions (boot plus root) amount to 20GB then you might clone the first 21GB of the SD card as follows (21GB = bs x count = 1M x 21000)

    sudo dd if=/dev/sda bs=1M count=21000 of=/mnt/sdcard.img status=progress

  • If you use WIn32DiskImager then be sure to check the "Read Only Allocated Partitions" box. At time of writing GoogleDrive was not playing nicely with Win32DiskImager - if W32DI won't open you have to quit GD first.

0

I do the following to shrink the img file. At the end I get two img files. First one with boot and rootfs partitions. Second with the third partition.

I flash the first img, resize the filesystem and partition of the second partition to my desired size, create third partition and write the second img to third partition

# to setup img
sudo kpartx -a -v your_image.img

# add map loop15p1 (252:0): 0 524288 linear 7:16 8192
# add map loop15p2 (252:1): 0 16384000 linear 7:16 532480
# add map loop15p3 (252:2): 0 8388608 linear 7:16 16916480

# identify the loop number

loop_device="loop15"
dev_loop_device="/dev/loop15"
dev_loop_device_part2="/dev/loop15p2"
dev_mapper_loop_device_part2="/dev/mapper/loop15p2"
dev_loop_device_part3="/dev/loop15p3"
dev_mapper_loop_device_part3="/dev/mapper/loop15p3"

sudo fdisk -l /dev/loop15

# Disk /dev/loop15: 12.07 GiB, 12956205056 bytes, 25305088 sectors
# Units: sectors of 1 * 512 = 512 bytes
# Sector size (logical/physical): 512 bytes / 512 bytes
# I/O size (minimum/optimal): 512 bytes / 512 bytes
# Disklabel type: dos
# Disk identifier: 0xd3753906

# Device        Boot    Start      End  Sectors  Size Id Type
# /dev/loop15p1          8192   532479   524288  256M  c W95 FAT32 (LBA)
# /dev/loop15p2        532480  16916479  16384000  7.8G 83 Linux
# /dev/loop15p3      16916480 25305087  8388608    4G 83 Linux

part2_start_sector=532480
part3_start_sector=16916480

sudo e2fsck -y -f "$dev_mapper_loop_device_part2"
sudo e2fsck -y -f "$dev_mapper_loop_device_part3"

sudo resize2fs -M "$dev_mapper_loop_device_part2"

# resize2fs 1.46.5 (30-Dec-2021)
# Resizing the filesystem on /dev/mapper/loop15p2 to 723587 (4k) blocks.
# The filesystem on /dev/mapper/loop15p2 is now 723587 (4k) blocks long.

# size_in_bytes = 723587 * 4096 = 2963812352
# size_in_sectors_part2 = 2963812352 / 512 = 5788696

sudo resize2fs -M "$dev_mapper_loop_device_part3"

# resize2fs 1.46.5 (30-Dec-2021)
# Resizing the filesystem on /dev/mapper/loop15p3 to 102706 (4k) blocks.
# The filesystem on /dev/mapper/loop15p3 is now 102706 (4k) blocks long.

# size_in_bytes = 102706 * 4096 = 420683776
# size_in_sectors_part3 = 420683776 / 512 = 821648

part2_end_sector=6321176s # "$((part2_start_sector + size_in_sectors_part2))s"
part3_end_sector=17738128s # "$((part3_start_sector + size_in_sectors_part3))s"

sudo parted $dev_loop_device resizepart 2 $part2_end_sector
sudo parted $dev_loop_device resizepart 3 $part3_end_sector

sudo dd if="$dev_loop_device_part3" of=your_image_part3.img
sudo chmod 777 your_image_part3.img

sudo fdisk -l $dev_loop_device # notice the new sizes. 

# Disk /dev/loop15: 12.07 GiB, 12956205056 bytes, 25305088 sectors
# Units: sectors of 1 * 512 = 512 bytes
# Sector size (logical/physical): 512 bytes / 512 bytes
# I/O size (minimum/optimal): 512 bytes / 512 bytes
# Disklabel type: dos
# Disk identifier: 0xd3753906
# 
# Device        Boot    Start      End Sectors   Size Id Type
# /dev/loop15p1          8192   532479  524288   256M  c W95 FAT32 (LBA)
# /dev/loop15p2        532480  6321176 5788697   2.8G 83 Linux
# /dev/loop15p3      16916480 17738128  821649 401.2M 83 Linux

# remove part3 as it is exported now
sudo parted $dev_loop_device rm 3

sudo kpartx -d -v your_image.img

# Since we got part3 export using dd command, we have to now truncate the whole image to 6321176 sectors

img_file_truncate_to=3236442112 # 6321176 * 512 or $((part2_end_sector*512))

sudo truncate -s $img_file_truncate_to your_image.img

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.