Determining correct image size for Raspbian backup to fit only real data on rootfs

Question

I regularly backup my Raspi using a script that rsyncs rootfs and /boot onto a image created with losetup. This is the current branch I am working on:

https://github.com/ephestione/RaspiBackup/blob/autosize-root-plus-free-space/RaspiBackup.sh

The improvement over the master branch is that, instead of determining the image size by using the rootfs partition size (still usually better than using the whole device size), I want to directly create a shrinked image, where the rootfs partition size is pre-calculated to be just enough to fit the extant data, plus a forfait 100MB free space (I can expand filesystem after restore if needed). In the first version I used:

df / --output=used | tail -n1

to determine the target partition size, then adding 250MB for boot and 100MB of free space, but it was incorrect, as I wasn't taking into account the reserved blocks. The resulting image, even by adding 100MB of space to the determined size, resulted as 100% in use, and I suspect was still smaller than the real data.

So, now, I use the difference between:

df / --output=size | tail -n1

and

df / --output=avail | tail -n1

like so:

ROOTPU=`expr "$(grep '/ ' /etc/fstab)" : 'PARTUUID=\(.[a-z0-9\-]*\)'`
ROOT=/dev/disk/by-partuuid/$ROOTPU
BOOTSIZE=250
ROOTFREE=100
SIZE=$(blockdev --getsz $ROOT)
# calc kilobytes size of data on rootfs
AVAILROOT=$(df / --output=avail | tail -n1)
SIZEROOT=$(df / --output=size | tail -n1)
((USEDROOT=SIZEROOT-AVAILROOT))
# bytes size of backup rootfs including free space after data
((USEDROOT=USEDROOT*1024 + ROOTFREE*1024*1024))

BLOCKSIZE=$(blockdev --getss $ROOT)
# blocks count of rootfs round up division result:
((SIZE=(USEDROOT+BLOCKSIZE-1)/BLOCKSIZE + BOOTSIZE*1024*1024/BLOCKSIZE))
#[...]
dd if=/dev/zero of="${IMAGE}" bs=${BLOCKSIZE} count=0 seek=${SIZE}

which is still WAY better than just imaging an expanded filesystem on a 32GB SD card, but it will yield an image which is 1GB bigger than it needs to be (1.1GB of resulting free space using df -h on the restored system).

EDIT: I did some further research, taking as an example my own filesystem. Right now I have a 4.3GB partition as reported by df. So let's say I want to determine the total size of the root partition on Raspbian:

sudo blockdev --getsz /dev/mmcblk0p2 multiplied by sudo blockdev --getsz /dev/mmcblk0p2 returns (512*9142272):

4 680 843 264 bytes

default df with no other parameters (1k blocks of 1024bytes each, 4433716*1024) reports:

4 540 125 184 bytes

sudo tune2fs -l /dev/mmcblk0p2 with block count * block size (1142784*4096):

4 680 843 264‬ bytes

Two out of three methods, tune2fs and blockdev, report the same total size of the partition, while df reports a smaller size. The difference though, which is 134MB, doesn't seem to really account for the "missing" space I found... I'm at a loss.

EDIT 2: I just tested by using the values reported from tune2fs -l $ROOT, namely doing ((Block count)-(Free blocks))*(Block size), and then added 100MB. This yielded an image 200somethingMB bigger than the df method, still (And I already knew it even before trying) on restore, df reported 0 available blocks... at this point I know that a "0 free space" image with 250MB boot should be about 3.7GB, but with this latest method, even adding 100MB to the plate, I get a 3.4GB image size...

EDIT 3: now this is getting funny. So I started doing calculations by using df / --output=used as base, and increasing it by 5%, and adding 200MB (I actually went further than that, and used tune2fs to determine current percent of reserved blocks... which ended up being 5% anyway). I started getting images with just around 60MB free space instead of 200. WTH? I tried then just using "used space"*1.05 as image size, and I just now booted a such restored backup, this is the output of df /:

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        3081144 3030756         0 100% /

and these are the relevant info from tune2fs -l /dev/mmcblk0p2:

Block count:              799232
Reserved block count:     39961
Free blocks:              9524

According to tune2fs I should have 9524*4096/1024/1024 megabytes free, which amounts to 37MB, surely no 0 anyway. What gives? Who's the scumbag here? df, tune2fs or the filesystem?

Question: is there a way to predictably determine the image size needed to have a resulting backup image which is more or less ${ROOTFREE}MBs larger than the extant data on rootfs plus 250MB boot partition?

Answer 1

On EXT file systems, by default 5% of the space is reserved to root user, so the amount df reports in the "Available" column is reduced by those 5%. As a result, the sum of "Used" and "Available" blocks does not correspond to "Size". The amount of reserved space can be seen (and changed) using tune2fs -l/-m/-r.

Note that if the partition is mostly empty, shrinking it e.g. to 10% of its original size will increase the space reserved for the root user from 5% to 50%, so in your case (4.3GB used on a 32GB is a factor of 7.5) you should expect to have an extra 5%*4.3GB*(7.5-1) = 1.2GB of free space, which is roughly what you see.

The difference between partition size (reported by blockdev) and the "Size" in filesystem blocks reported by df is explained by additional space being reserved for internal filesystem structures like inodes or journal. It's important to preserve this overhead while resizing, because, while it may shrink together with the filesystem, some parts of it may not shrink.

I would calculate the minimum size by subtracting the amount of "Available" blocks reported by df from a total partition size as reported by blockdev, plus a fixed quota you want to have free, and minus the bloated 5% reserve I described above. So if you have the initial partition size of 32768 MB and df reports that 28468 MB are "Avaliable" and and 32276 MB total, you should get

Used + root-reserved space =               32768 - 28468 = 4300 MB  # size from `blockdev`
5% of the initial size =                   32276 * 5% = 1613 MB     # size from `df`
Used space without root reserve =          4300 - 1613 = 2687 MB
5% of the used space =                     2687 * 5% = 134 MB
Used space including 5% reserve =          2687 + 134 = 2821 MB
Image size incl. boot and 100 MB quota =   2821 + 250 + 100 = 3171 MB

This will never be precise down to a single MB (because as file system grows or shrinks its internal structures may resize too), but it should get you at least the amount of free space you were aiming for.

Answer 2

This is what -for now- works, I used Dmitry's answer to rework it some (curiously if I first calculated 5% and then subtracted it from original size, I got 192MB free setting ROOTFREE to 10MB):

BTOT=$(tune2fs -l $ROOT | grep "Block count:" | grep -o -E [0-9]*)
BLOCKSIZE=$(blockdev --getss $ROOT)
((TOTPART=BTOT*4096))
SIZEROOT=$(df / --output=size | tail -n1)
((SIZEROOT=SIZEROOT*1024))
AVAILROOT=$(df / --output=avail | tail -n1)
((AVAILROOT=AVAILROOT*1024))
((USEDROOT=((TOTPART-AVAILROOT)*96-1)/100))
((USEDROOT=((USEDROOT+ROOTFREE*1024*1024)*106-1)/100)) #round up as above, and additional free space n$
((SIZE=(USEDROOT+BLOCKSIZE-1)/BLOCKSIZE + BOOTSIZE*1024*1024/BLOCKSIZE))

With ROOTFREE set to 10, I get 79MB of free space after restore... I can deal with it B-)