2

I regularly backup my Raspi using a script that rsyncs rootfs and /boot onto a image created with losetup. This is the current branch I am working on:

https://github.com/ephestione/RaspiBackup/blob/autosize-root-plus-free-space/RaspiBackup.sh

The improvement over the master branch is that, instead of determining the image size by using the rootfs partition size (still usually better than using the whole device size), I want to directly create a shrinked image, where the rootfs partition size is pre-calculated to be just enough to fit the extant data, plus a forfait 100MB free space (I can expand filesystem after restore if needed). In the first version I used:

df / --output=used | tail -n1

to determine the target partition size, then adding 250MB for boot and 100MB of free space, but it was incorrect, as I wasn't taking into account the reserved blocks. The resulting image, even by adding 100MB of space to the determined size, resulted as 100% in use, and I suspect was still smaller than the real data.

So, now, I use the difference between:

df / --output=size | tail -n1

and

df / --output=avail | tail -n1

like so:

ROOTPU=`expr "$(grep '/ ' /etc/fstab)" : 'PARTUUID=\(.[a-z0-9\-]*\)'`
ROOT=/dev/disk/by-partuuid/$ROOTPU
BOOTSIZE=250
ROOTFREE=100
SIZE=$(blockdev --getsz $ROOT)
# calc kilobytes size of data on rootfs
AVAILROOT=$(df / --output=avail | tail -n1)
SIZEROOT=$(df / --output=size | tail -n1)
((USEDROOT=SIZEROOT-AVAILROOT))
# bytes size of backup rootfs including free space after data
((USEDROOT=USEDROOT*1024 + ROOTFREE*1024*1024))

BLOCKSIZE=$(blockdev --getss $ROOT)
# blocks count of rootfs round up division result:
((SIZE=(USEDROOT+BLOCKSIZE-1)/BLOCKSIZE + BOOTSIZE*1024*1024/BLOCKSIZE))
#[...]
dd if=/dev/zero of="${IMAGE}" bs=${BLOCKSIZE} count=0 seek=${SIZE}

which is still WAY better than just imaging an expanded filesystem on a 32GB SD card, but it will yield an image which is 1GB bigger than it needs to be (1.1GB of resulting free space using df -h on the restored system).

EDIT: I did some further research, taking as an example my own filesystem. Right now I have a 4.3GB partition as reported by df. So let's say I want to determine the total size of the root partition on Raspbian:

sudo blockdev --getsz /dev/mmcblk0p2 multiplied by sudo blockdev --getsz /dev/mmcblk0p2 returns (512*9142272):

4 680 843 264 bytes

default df with no other parameters (1k blocks of 1024bytes each, 4433716*1024) reports:

4 540 125 184 bytes

sudo tune2fs -l /dev/mmcblk0p2 with block count * block size (1142784*4096):

4 680 843 264‬ bytes

Two out of three methods, tune2fs and blockdev, report the same total size of the partition, while df reports a smaller size. The difference though, which is 134MB, doesn't seem to really account for the "missing" space I found... I'm at a loss.

EDIT 2: I just tested by using the values reported from tune2fs -l $ROOT, namely doing ((Block count)-(Free blocks))*(Block size), and then added 100MB. This yielded an image 200somethingMB bigger than the df method, still (And I already knew it even before trying) on restore, df reported 0 available blocks... at this point I know that a "0 free space" image with 250MB boot should be about 3.7GB, but with this latest method, even adding 100MB to the plate, I get a 3.4GB image size...

EDIT 3: now this is getting funny. So I started doing calculations by using df / --output=used as base, and increasing it by 5%, and adding 200MB (I actually went further than that, and used tune2fs to determine current percent of reserved blocks... which ended up being 5% anyway). I started getting images with just around 60MB free space instead of 200. WTH? I tried then just using "used space"*1.05 as image size, and I just now booted a such restored backup, this is the output of df /:

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        3081144 3030756         0 100% /

and these are the relevant info from tune2fs -l /dev/mmcblk0p2:

Block count:              799232
Reserved block count:     39961
Free blocks:              9524

According to tune2fs I should have 9524*4096/1024/1024 megabytes free, which amounts to 37MB, surely no 0 anyway. What gives? Who's the scumbag here? df, tune2fs or the filesystem?

Question: is there a way to predictably determine the image size needed to have a resulting backup image which is more or less ${ROOTFREE}MBs larger than the extant data on rootfs plus 250MB boot partition?

  • 1
    An observation that won't affect the outcome but may trivially simplify some of your calculations: The figure df reports in bytes is always an even multiple of the filesystem block size -- not, as per df output in blocks, a default 1K. So, eg., if you have a directory with 10 1 byte files on a 4096 byte/block fs, df will report the size as 44K (not 11K), 1 block for the directory and 1 block for each (non-zero size) file. – goldilocks Oct 18 at 11:58
  • @goldilocks yes, thank you, you confirmed what I started thinking when I noticed df reported 1K blocks while instead tune2fs reported 4K blocks – ephestione Oct 18 at 12:03
  • 1
    "I tried then just using "used space"*1.05 as image size" ->The reserved blocks percentage is of the total filesystem size, not the amount of used space. The purpose is so that you can't fill the filesystem over 95%. – goldilocks Oct 18 at 13:04
  • Please post the output of df and blockdev before you shrink the file system. – Dmitry Grigoryev Oct 18 at 13:15
  • @goldilocks I had some kind of "aha!" moment after reading your comment, then again I realized I am still correct. I want the final root partition size on the backup image to be exactly the size of the data (+100 or any MB if desired), then the 5% must be calculated on that value, hence (used+100MB)*105/100, isn't that correct? – ephestione Oct 18 at 15:58
3

On EXT file systems, by default 5% of the space is reserved to root user, so the amount df reports in the "Available" column is reduced by those 5%. As a result, the sum of "Used" and "Available" blocks does not correspond to "Size". The amount of reserved space can be seen (and changed) using tune2fs -l/-m/-r.

Note that if the partition is mostly empty, shrinking it e.g. to 10% of its original size will increase the space reserved for the root user from 5% to 50%, so in your case (4.3GB used on a 32GB is a factor of 7.5) you should expect to have an extra 5%*4.3GB*(7.5-1) = 1.2GB of free space, which is roughly what you see.

The difference between partition size (reported by blockdev) and the "Size" in filesystem blocks reported by df is explained by additional space being reserved for internal filesystem structures like inodes or journal. It's important to preserve this overhead while resizing, because, while it may shrink together with the filesystem, some parts of it may not shrink.

I would calculate the minimum size by subtracting the amount of "Available" blocks reported by df from a total partition size as reported by blockdev, plus a fixed quota you want to have free, and minus the bloated 5% reserve I described above. So if you have the initial partition size of 32768 MB and df reports that 28468 MB are "Avaliable" and and 32276 MB total, you should get

Used + root-reserved space =               32768 - 28468 = 4300 MB  # size from `blockdev`
5% of the initial size =                   32276 * 5% = 1613 MB     # size from `df`
Used space without root reserve =          4300 - 1613 = 2687 MB
5% of the used space =                     2687 * 5% = 134 MB
Used space including 5% reserve =          2687 + 134 = 2821 MB
Image size incl. boot and 100 MB quota =   2821 + 250 + 100 = 3171 MB

This will never be precise down to a single MB (because as file system grows or shrinks its internal structures may resize too), but it should get you at least the amount of free space you were aiming for.

  • Yes, I found this explanation during my reseach, yet, if this was the only reason, then calculating the target image size as "df size" - "df available" would yield an image which is an exact fit for the data on rootfs, with theoretically no free space left. Adding 100MB to that image size would then yield an image with roughly 100MB free space when restored. In my case though, this produced an image with 1.1GB free space when restored... – ephestione Oct 18 at 9:56
  • please consider my EDIT3 in the original question, I did that already, the total size of the device you get from blockdev --getsize64 /dev/mmcblk0p2 is the same value you get from tune2fs -l /dev/mmcblk0p2 albeit easier to extract with a script instead of matching with grep a multiline output. So I used that method, and adding 200MB to that image size only yields 59MB free after restore. – ephestione Oct 18 at 12:55
  • After much rework, I came up with the code that I published in my own answer, yet yours has put me on the right track so it gets the answered tag – ephestione Oct 18 at 17:50
1

This is what -for now- works, I used Dmitry's answer to rework it some (curiously if I first calculated 5% and then subtracted it from original size, I got 192MB free setting ROOTFREE to 10MB):

BTOT=$(tune2fs -l $ROOT | grep "Block count:" | grep -o -E [0-9]*)
BLOCKSIZE=$(blockdev --getss $ROOT)
((TOTPART=BTOT*4096))
SIZEROOT=$(df / --output=size | tail -n1)
((SIZEROOT=SIZEROOT*1024))
AVAILROOT=$(df / --output=avail | tail -n1)
((AVAILROOT=AVAILROOT*1024))
((USEDROOT=((TOTPART-AVAILROOT)*96-1)/100))
((USEDROOT=((USEDROOT+ROOTFREE*1024*1024)*106-1)/100)) #round up as above, and additional free space n$
((SIZE=(USEDROOT+BLOCKSIZE-1)/BLOCKSIZE + BOOTSIZE*1024*1024/BLOCKSIZE))

With ROOTFREE set to 10, I get 79MB of free space after restore... I can deal with it B-)

  • As I said, you should get at least the amount of free space you asked for, so getting 79 MB instead of 10 is expected. Getting rid of the extra 69 MB will require you to dig rather deeply in EXT4 internals to predict the inode table and journal area sizes on the shrunk partition, and they will not be a function of partition size alone. – Dmitry Grigoryev Oct 24 at 9:00
  • Yes, I understood the reason behind that ;) – ephestione Oct 25 at 9:44
  • @DmitryGrigoryev I tried with a 10GB partition and around 3GB used (Same system on another device with a larger partition), setting 10MBs free I get 576MB available on the destination image. Something must be off anyway – ephestione Oct 25 at 16:28
  • Perhaps you should print values you get in each calculation step. – Dmitry Grigoryev Oct 25 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.