1

I have two disk images, compressed with the xz algorithm and shrunk using the PiShrink tool, that are 669.5MB and 1.03GB respectively. The thing is, the smaller image was copied from the larger one and all I did to modify it was delete a few bash scripts/python scripts and add a couple new ones with less lines. I need both to be as small as possible for use in a production setting, but I can't for the life of me figure out what could be causing this discrepancy despite all of my sleuthing/comparison of the two.

Perhaps someone with a deeper knowledge of how the Pi handles storage/compression could shed some light on possible reasons for this? Or things I can try to pin down contributing factors?

I just looked at the size of all the folders on each disk image, using the same exact SD card, by running the following from '/':

du -s * | sort -nr

Comparing the outputs from both of the mounted images only shows a 1066 byte difference in size, across all the folders, between the larger and smaller disk images....

Where could this extra data be lurking? At the very least I need the second image to be under 1GB but even after deleting every excess script or tar I could find (that also exist on the smaller image?!?) it hasn't gotten below 1.02GB. Thank you for any and all information that might help me solve this mystery or easily reduce the larger image.

Context: I've been using Buster with my Rpi 3B+, copying and cloning mostly with the dd utility on MacOS or the SD Clone software from twocanoes. The micro SD card I've been using is a SanDisk Ultra 16GB, 10 A1.

Improve this question
2
  • the order of data in a datastream that is being shrunk can make a difference - also the content of the "unused space" within an allocated blocks of a file can make a difference too
    – Jaromanda X
    May 13, 2021 at 23:21
  • Hmm, at the point I cloned the larger image initially to make small edits what processes potentially reordered data before I shrunk it? If I understand correctly, PiShrink only picked out blocks that were actually allocated, but within those blocks is "unused space" that could actually carry a lot of information? I guess I would like to find away to clear that out but I am unsure why the cloning process wouldn't of preserved that information in the first place.
    – cnrcbr
    May 14, 2021 at 18:14

1 Answer 1

Reset to default
3
+50

I would be surprised if modifying a couple of text files confused the compression algorithm to the point it produced an extra 300MB of output. So I would start by comparing the sizes of uncompressed images: if the two images are not the same size, it's no surprise they compress differently, even if the amount of data on filesystem level is the same. Images contain some amount of free sectors, and those sectors are not necessarily trimmed (erased), so they tend to contain random garbage data.

Getting a different uncompressed image size is not that surprising. The size depends on the relocation algorithm that PiShrink uses, and I doubt that algorithm is as optimized as the compression algorithm used by xz. In fact, I would expect it to leave some amount of free space in the image on purpose, because an image with zero free space is unbootable. It could be that free space is padded to the next GB, or that non-continuous free space is not removed.

A simple trick that wipes most of the free space on most filesystems could be:

sudo cp /dev/zero /path/to/filesystem/tmpfile; sudo sync; sudo rm /path/to/filesystem/tmpfile

Or, if you prefer to use the right tool for the job, sfill -llz /path/to/filesystem/

This consumes all empty space on the filesystem by filling it with zeroes, then frees that space again. Running cp as root allows you to get a hold of the space which is outside of regular users' quota.

Your images may still have different size after the process, but at least compressing extra blocks which are zeroed will contribute very little to the compressed size.

Improve this answer
3
  • They are indeed different sizes and starting out I had the educational experience of trying to boot an image without enough free space, but I guess I am trying to pinpoint where exactly in the process of cloning an image this difference is created and how I might remediate it. It is important to note even before the application of PiShrink the difference existed. How is random garbage data that is introduced usually ironed out?
    – cnrcbr
    May 20, 2021 at 17:51
  • @cnrcbr Try zeroing free space.
    – Dmitry Grigoryev
    May 21, 2021 at 7:56
  • Using the simple sudo cp /dev/zero approach worked well. The smaller image was only reduced about 40MB, but the larger one dropped all the way down to within 3.5MB of that after compression. It is quite surprising to me that this kind of step is not built into standard processes as it can have such a substantial effect on image file size.
    – cnrcbr
    May 26, 2021 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.