I am trying to back up a 16GB SD card as it is proposed here: How do I backup my Raspberry Pi?
I have started with 2017-01-11-raspbian-jessie-lite.img and then did some work with FFmpeg (cloned & compiled the source code). The compilation takes quite a long time so I need to create a backup image to go back if needed.
Both dd and win32 Disk Imager generate a ~16GB .img file.
Question: Is there a way to generate a smaller file and not something that reaches the size of the SD card?
Surprise! creating an image of a 16GiB SD Card creates a 16GiB image.
There are 2 options;
only backup files can-a-raspberry-pi-be-used-to-create-a-backup-of-itself
or compress the image https://raspberrypi.stackexchange.com/a/29085/8697 - this is a macOS script, but you can develop similar for other OS.
It's worse than you think == chances are your '16Gb' .img WONT WRITE BACK to another '16Gb' SD card !
1) The stupid Pi system autoexpands to every last block of the SD card on FIRST BOOT !! This is stupid because :-
2) No two SD cards have the same 'block count'. All cards have 'bad blocks' that are 'mapped out' during manufacturing. As compitition drives prices down, SD cards with more and more bad blocks are being 'pssssed' - so 'todays' '16Gb' card will be 15.9 or 15.8 or 15.7 or even less 'tomorrow' ...
3) Win32 disk imager WONT WRITE a 15.90 Gb .img to a 15.8999999 Gb card, full stop. Even if it did, the partition would 'run off the end of the card' and the Pi will abort with 'Corrupted file system' errors
a) You have to reduce the partition to 15.5 or so to 'guarantee' it will write back (BEST way is to MODIFY the .img on your PC BEFORE burning to SD so it doesn't 'autoexpand' on first boot-up (remove ' init= ..' from end of cmdline.txt on the FAT32 partition). Then, manually change /usr/lib/raspi-config/init_resize.sh so it ewxpands to 95% of the card, not 100%
[find 'TARGET_END=$((ROOT_DEV_SIZE - 1))', change to
'TARGET_END=$((ROOT_DEV_SIZE / 100 \* 95 ))' ]
then add ' init=/usr/lib/raspi-config/init_resize.sh' back to cmdline.txt and reboot .
b) Finally, DONT USE win32 imager to make your back-up .img (it will grab every block, you only want to the end partition, not the whole SD card).
Instead (on the PC) use (free) DiskInternals Linux Reader .. (after which Win32 Imager will 'burn' the .img just fine to any card witb at least 95% of the space of the original)
Please note == This is what I ACTUALLY DO (with my PoE headless Pi Zero projects), and I can assure I have written my .img's to many different SD cards and it ACTUALLY WORKS (it seems that most 'guides' on 'How to back-up the Pi' are written by people who have never actually tried to 'restore' their .img to a new SD card :-) )
PS - or, if you are using a Pi B3, by all means start by 'shrinking' the end partition and then add a USB SD card reader/writer and play with 'dd' ... I don't do this because all my projects are based on the Pi Zero (which I usually run 'headless' with a USB/Ethernet 'dongle' on the end of a PoE cable)
Both DD and the disk imager will 'naively' just create an exact replica of the SD card, empty space and all hence why they are 16GB. These backups you can simply image directly onto SD card and its back where you left off.
If you want to snapshot your compilation process outcomes you might want to look at alternatives to entire backups. Using Docker containers would let you incrementally build on what you have done by committing the containers at each step and only pay for (in storage terms) the differences between each version. There is a recent Raspbian base image on the docker hub for the Pi which should give you an accurate enough base system, you can even copy your existing work directly into it to avoid starting from scratch. If you want to backup your work now, you only have to export your docker containers which will be Base Image size + changes, probably under 1gb. This way you can test anyway and if it goes south just bin the image and return to a known good state.
--prefix=PREFIX option and keep them in your home folder to avoid polluting the /usr ones and separate them that way? Should keep the purity but without having to maintain entire SD card images?
Since nobody mentioned it yet, here is the tool I use to shrink images obtained by dd: https://github.com/Drewsif/PiShrink
The shrunk images are autoexpanded at first boot to cover the SD card they are dd'ed onto by default, which can be disabled.
I recommend using Partimage to create a backup of the partitions on your your SD card. From the documentation:
Partimage will only copy data from the used portions of the partition.
Partimage can also compress its output on-the-fly, to make the resulting output even smaller.
If you are making the copy on an x86 system, you can use Clonezilla instead. It will use a variety of tools to make an image of your entire SD card.
ext4 and the main partition of the SD card is ext4. So this might not work.
An easy approach if you have less than about 7GB in use on your 16GB card is to use the "Copy SD Card" function to copy to an 8 GB SD card. That will give you an 8 GB image instead of 16. Other answers will give you a smaller image file, but this is the easiest.
ApplePi-Baker (Mac only) now supports shrinking .img files during SD card backup (and expansion during restore): ApplePi-Baker v2 – Backup & Restore SD cards, USB drives, etc. - Shrink IMG Process
Unfortunately the .img is expanded prior to writing back to the memory card, so it actually takes longer overall - but at least the .img backup file is smaller for storage and transmission.
I use a three step process:
For example, in a fit of insanity I used a 256G SD card and discovered it was effectively impossible to back up. Using parted I shrank it to about 20G - leaving the remainder as unallocated space ( later you can expand into it or make it another partition, at your discretion).
Now you have a manageable size SD card that you can clone using dd or Win32DiskImager or equivalent.
If you use dd (linux) then clone enough of the disk to contain your partition. For example, if your partitions (boot plus root) amount to 20GB then you might clone the first 21GB of the SD card as follows (21GB = bs x count = 1M x 21000)
sudo dd if=/dev/sda bs=1M count=21000 of=/mnt/sdcard.img status=progress
If you use WIn32DiskImager then be sure to check the "Read Only Allocated Partitions" box. At time of writing GoogleDrive was not playing nicely with Win32DiskImager - if W32DI won't open you have to quit GD first.
I do the following to shrink the img file. At the end I get two img files. First one with boot and rootfs partitions. Second with the third partition.
I flash the first img, resize the filesystem and partition of the second partition to my desired size, create third partition and write the second img to third partition
# to setup img
sudo kpartx -a -v your_image.img
# add map loop15p1 (252:0): 0 524288 linear 7:16 8192
# add map loop15p2 (252:1): 0 16384000 linear 7:16 532480
# add map loop15p3 (252:2): 0 8388608 linear 7:16 16916480
# identify the loop number
loop_device="loop15"
dev_loop_device="/dev/loop15"
dev_loop_device_part2="/dev/loop15p2"
dev_mapper_loop_device_part2="/dev/mapper/loop15p2"
dev_loop_device_part3="/dev/loop15p3"
dev_mapper_loop_device_part3="/dev/mapper/loop15p3"
sudo fdisk -l /dev/loop15
# Disk /dev/loop15: 12.07 GiB, 12956205056 bytes, 25305088 sectors
# Units: sectors of 1 * 512 = 512 bytes
# Sector size (logical/physical): 512 bytes / 512 bytes
# I/O size (minimum/optimal): 512 bytes / 512 bytes
# Disklabel type: dos
# Disk identifier: 0xd3753906
# Device Boot Start End Sectors Size Id Type
# /dev/loop15p1 8192 532479 524288 256M c W95 FAT32 (LBA)
# /dev/loop15p2 532480 16916479 16384000 7.8G 83 Linux
# /dev/loop15p3 16916480 25305087 8388608 4G 83 Linux
part2_start_sector=532480
part3_start_sector=16916480
sudo e2fsck -y -f "$dev_mapper_loop_device_part2"
sudo e2fsck -y -f "$dev_mapper_loop_device_part3"
sudo resize2fs -M "$dev_mapper_loop_device_part2"
# resize2fs 1.46.5 (30-Dec-2021)
# Resizing the filesystem on /dev/mapper/loop15p2 to 723587 (4k) blocks.
# The filesystem on /dev/mapper/loop15p2 is now 723587 (4k) blocks long.
# size_in_bytes = 723587 * 4096 = 2963812352
# size_in_sectors_part2 = 2963812352 / 512 = 5788696
sudo resize2fs -M "$dev_mapper_loop_device_part3"
# resize2fs 1.46.5 (30-Dec-2021)
# Resizing the filesystem on /dev/mapper/loop15p3 to 102706 (4k) blocks.
# The filesystem on /dev/mapper/loop15p3 is now 102706 (4k) blocks long.
# size_in_bytes = 102706 * 4096 = 420683776
# size_in_sectors_part3 = 420683776 / 512 = 821648
part2_end_sector=6321176s # "$((part2_start_sector + size_in_sectors_part2))s"
part3_end_sector=17738128s # "$((part3_start_sector + size_in_sectors_part3))s"
sudo parted $dev_loop_device resizepart 2 $part2_end_sector
sudo parted $dev_loop_device resizepart 3 $part3_end_sector
sudo dd if="$dev_loop_device_part3" of=your_image_part3.img
sudo chmod 777 your_image_part3.img
sudo fdisk -l $dev_loop_device # notice the new sizes.
# Disk /dev/loop15: 12.07 GiB, 12956205056 bytes, 25305088 sectors
# Units: sectors of 1 * 512 = 512 bytes
# Sector size (logical/physical): 512 bytes / 512 bytes
# I/O size (minimum/optimal): 512 bytes / 512 bytes
# Disklabel type: dos
# Disk identifier: 0xd3753906
#
# Device Boot Start End Sectors Size Id Type
# /dev/loop15p1 8192 532479 524288 256M c W95 FAT32 (LBA)
# /dev/loop15p2 532480 6321176 5788697 2.8G 83 Linux
# /dev/loop15p3 16916480 17738128 821649 401.2M 83 Linux
# remove part3 as it is exported now
sudo parted $dev_loop_device rm 3
sudo kpartx -d -v your_image.img
# Since we got part3 export using dd command, we have to now truncate the whole image to 6321176 sectors
img_file_truncate_to=3236442112 # 6321176 * 512 or $((part2_end_sector*512))
sudo truncate -s $img_file_truncate_to your_image.img
SD Card Copierthat only copies the used space to make a backup.