2

I am surprised by the large amount of data I'm writing to the root partition in a Raspberry Pi. My applications and system logs are not writing that much. I use the usual tricks to reduce it. What's going on ? The question is... how much are modifications of pointers in the file system burning out my SD card ?

Raspberry Pi's, due to their small size, boot from a micro-SD card, which is a flash drive. The root file system is also on that flash drive unless a second memory device is used for it. There are hundreds of posts about which file system to choose for a USB flash drive used with a Raspberry Pi. The limited life of the SD card is discussed in countless posts, usually of the form, make it larger, write less to it, and use noatime.

My question is different. Is the Raspberry Pi OS, since it presumably assumes the use of a flash drive for at least the boot file system and usually also the root partition, doing anything different to use a flash drive ? Since the binary bits in a block of flash memory are much more limited in their ability to be inverted than the bits in a magnetic disk, I cannot find any posts that discuss why the file system works AT ALL, * more than once *, from the point of view of the need to change pointers in the file system to point to a changed data file.

I'm trying to understand how a file system REALLY works on a Raspberry Pi on a flash drive to help find a way to prolong the life of the flash drive, regardless of all the other tricks mentioned above. ext4 is recommended for the root file system on a Raspberry Pi. How many files in the ext4 file system are modified when a data file is modified, however slightly ? Since flash memory has a finite number of program-erase cycles, and it's not possible to flip bits like in the old magnetic disk days (with flash, a single erased bit can be flipped but a single set bit cannot, so it's not possible to rewrite pointers, only to clobber them), I'm trying to see how many file system files and how many bytes in total get changed when a data file is updated. I must be missing something because it seems to me that every change in a data file ripples all the way up to the top level of the file system when flash memory is used, and that's a lot to change to change a little, thus wearing out the flash memory. And how is the top-most file system pointer changed ?

Improve this question
3
  • 4
    Your question, as I understand it, does not seem to be specific to the Raspberry Pi. I guess that Superuser is a more appropriate stack. Since there are however 500 results on this search I have decided to not migrate this question over there but have you check them first for relevance and post your question if you do not find an answer.
    – Ghanima ♦
    Aug 28, 2018 at 22:45
  • when a data file is updated - or even when it is read
    – Jaromanda X
    Aug 29, 2018 at 1:00
  • Ghanima: please see my edited posting above.
    – NewtownGuy
    Aug 30, 2018 at 0:32

1 Answer 1

Reset to default
1

Is the Raspberry Pi OS, since it presumably assumes the use of a flash drive for at least the boot file system and usually also the root partition, doing anything different to use a flash drive ?

It's not clear what you are after here, but in general flash media including SD cards are a black box to the OS; they have their own microcontroller inside. The OS cannot optimize anything in that regard.

the need to change pointers in the file system to point to a changed data file

If you mean the relationship between a filesystem and physical storage, with SD cards the latter is virtual from the perspective of the OS, meaning addresses used in transactions with the card do not correspond to physical blocks. The SD card's microcontroller translates them. This is because it may end up moving data around opaquely as part of its function (e.g., to implement wear levelling). So if the OS is using address 1001, and the card MC moves that data physically, it will still be accessed via the same address -- hence the OS does not know where anything in the black box really is.

The limited life of the SD card is discussed in countless posts, usually of the form, make it larger, write less to it, and use noatime.

A great many of them somewhat paranoid and/or obsessive compulsive. This is not to say they are all out to lunch, but you may find this contra perspective interesting.

I'm trying to see how many file system files and how many bytes in total get changed when a data file is updated.

Linux, like any contemporary general purpose operating system, caches frequently accessed files in free memory -- this is why if you look at memory stats it may appear to be all used despite also seeming to have a good bit free. That free bit is the file cache, parts of which the OS will discard when more memory is needed by user applications.

The purpose is actually to improve performance (since RAM is much faster than secondary storage) but it will also reduce wear and tear on the storage medium. Hence, leaving a good chunk of memory (say a hundred MB or so) free is a good idea.

SD cards might also do some caching, although I'd guess this more a characteristic of higher end flash media. Again, the purpose would primarily be to improve performance.

Improve this answer
4
  • I think people are unaware how flash memory works and that is causing it to wear out far faster than expected. The contra perspective focuses on how much data is written, claiming it is insignificant in their example, but ignoring how much the file system files are changing to write that data. If I add one character to a file, is that character appended to the block of memory storing the file or is a new block required ? How does the pointer in the file system to that block get updated ? Flash can't modify pointer, a new block is needed, and so on up the file system, causing far more writing.
    – NewtownGuy
    Aug 30, 2018 at 20:24
  • It's arbitrary and indeterminate. Arbitrary because the SD card's microcontroller is not operating in terms of files or filesystems but just virtualized addresses and chunks of data; indeterminate because the OS has no knowledge of the actual physical layout in the card. SD cards typically use an erase block of 4 MB; whenever anything in that block needs changing, the entire block is erased and re-written. What's in that block in relation to file and filesystem layouts is (again) arbitrary and indeterminate.
    – goldilocks ♦
    Aug 30, 2018 at 22:51
  • ...The SD MC is not trying to keep files together. It's just shuffling data in and out. Changing 4 bytes in a file may require changing one block or (an extreme case), four (or more, if you include relevant filesystem metadata). Filesystem metadeta (pointers, inodes, etc. etc.) is just more data. It is treated the same way, and will end up randomly distributed because (again), the card doesn't care what the data is, and the OS cannot truly see into or govern the card.
    – goldilocks ♦
    Aug 30, 2018 at 22:52
  • 1
    So it doesn't matter whether or not people understand how SD cards work except in one important point: They're electronic. If you pull the plug on a computer such as a Pi, you create two opportunities for file system corruption to take place. One, because the OS is not synced with the storage. Two, because the card may be doing anything at that point, meaning stuff can get lost. These are the major causes of data loss/storage failure on the Pi, and they happen because people get into a habit of not shutting the system down properly, not because their SD card suddenly wore out.
    – goldilocks ♦
    Aug 30, 2018 at 22:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.