0

I have a circular buffer of files in ram disk. A new file is created every second. I want to always refer to the latest file using the same file name for access by a browser and apache. So, each time I update the file, I update a symlink from FIXED_FILE_NAME to VARIABLE_FILE_NAME. My concern is that such a frequent update of the file system wears out the flash memory that stores the system. How can I avoid this wear ? Thank you in advance for your help.

  • 2
    Can't the symlink be on the RAM disk as well? – joan Aug 16 '15 at 14:22
4

You can't, but it does not matter because it will take more than a decade to wear a card out that way. I've pointed out previously that there is much paranoia about this amongst pi users, perhaps because of reports of filesystem corruption which litter the internet, although these have nothing to do with wearing out cards. I don't think there is the same degree of hysteria amongst camera and smartphone users.

Anyway, let's unsuperstitiously go through the math. Say you have a cheap 4 GB card. It doesn't matter whether you allocate the whole thing or not, because SD cards use virtual addressing and wear leveling. Physically, the whole card is in play no matter what, even if you are only using a 1 GB filesystem.

Let's also say that cheap card has a very crappy lifespan -- 1000 write cycles. Then let's say it only takes half that time before enough failed blocks have been excluded by the card's built-in controller to make using it silly and splurging another $10, as painful as it seems, becomes worthwhile.

So in total, you can write 2 terabytes to the card. Now let's say that changing an inode is 1 kB. It certainly won't really be that much from a filesystem perspective, but we might as well increase it to 4 kB in case there's something about the SD card which requires changing entire blocks of a certain minimal size, such as 4 kB (this is probably the case). If all you use the SD card for is small writes, this means you could do this on average:

2000000000000 / 4000 = 500000000

500 million times. There are 86400 seconds in a day; at 365 days per year that's

500000000 / 31536000 seconds = 15 years

But of course that's not all that is going on with the card. Add on average another 100 MB per day writing to the card. Spread out, that's another ~1157 bytes per second; we don't need to compensate for the minimal block size here presuming the average write is more than 4 kB. Now we have:

2000000000000 / 5117 / 31536000 = 12 years

To be on the safe side, I suggest you replace the card every 2-3 years if you can afford the expense. If you are really strapped, try pushing it to 5-6. You should keep a backup of the system and, if it is crucial, have a spare card at hand anyway.

There is an additional factor in your favour, which is that the OS uses a page cache in free memory. Frequent changes will not all take place on disk, they will take place in the page cache and occasionally get synced to disk -- this is one reason filesystem corruption is possible in the first place, because if you suddenly kill the power when the system is busy, the cache is not completely synced with the disk at that moment (however, the risk of this really causing a problem is low). See also man sync.

Of course, if you reduce the amount of free memory by using a giant ram disk, you reduce the efficacy of the page cache. That may or may not be wise depending on context, but it likely it is not. Try to leave at least 50-100 MB free unless you do not care about the overall performance of the system.1

Note that the pi, or some pis, seem to be prone to causing corruption even when using a read-only filesystem -- at least, we've had a few separate reports of that here. You cannot guard against this unknown problem, which may have to do with an old kernel bug (if you have a kernel built in the past year, don't worry), or it may be an electrical flaw in some pis.


1. Note there are two metrics common in reporting memory usage, the total amount of RAM used by userspace processes, and the total amount of RAM used. The difference between the two is mostly because of the page cache; if you look at the output of free, this is indicated on the second line, +/- buffers/cache. If the system has been running for long enough, particularly on a relatively low memory device like the pi, chances are the kernel will have filled all free RAM (as in, not used by userspace processes) with the page cache. This is why the metric that's relevant for most purposes is the first one, i.e., the memory used by userspace process, because the amount of "free" memory available for use by userspace processes (that is, all the programs you run) is the total amount minus the amount they are already using, and not including the page cache. Remember, stuff in the page cache isn't really in use, it's stuff that was in use and is cached in RAM in case something wants to use it again. If instead something needs more memory for some other purpose, chunks of the page cache are discarded to provide it (possibly being synced to disk first).

So for user purposes, you can consider the page cache free RAM. Hence the second line of free. Here's a headless B+ that's been up for a few days:

>free
             total       used       free     shared    buffers     cached
Mem:        448376     278064     170312       8416      45968     194780
-/+ buffers/cache:      37316     411060

On the first line, it looks like I've used 278 MB. But that includes the page cache. If I added up all the memory used by all my running processes, it wouldn't be anywhere near that amount. That's the number on the second line -- I've really only used 37.3 MB. However, some tools, including top, only report the number from the first line (however, top does report numbers for buffers and cached, which you can subtract from that).

  • Based on your reply, I am concerned about three things: (1) does wear leveling work as well as you say on micro-SD cards and USB flash drives, (2) do 2-bits/cell and 3-bits/cell flash drives have as many write cycles as you estimate, and (3) how do I maintain at least 50 to 100 MB of free memory ? I usually see only about 15 to 20 MB of free memory in 'top' and most of the memory usages reported by 'top' are 0% so the memory usage numbers don't add up and I don't know what functions to massage to free up more memory. – NewtownGuy Aug 17 '15 at 12:28
  • I've added a lengthy footnote that may explain what you are seeing in top. Looks like Ronny Nilsson has a clever solution to your symlink question, although I still feel tackling this as an XY problem was the way to go. WRT wear leveling and various things, you'd have to do your own research, but I believe pretty much all the flash memory in consumer storage devices use it.. – goldilocks Aug 17 '15 at 13:09
  • ..My general point is there's no point in bending over backward this way. Say the card only lasts 18 months? So what? It is easy to keep a backup of and restore a 4 GB file filesystem; not doing so is just lazy and that's what you should put an effort into. Why not find out for yourself here -- if the card fails in a year, then maybe you want to take some kind of action. But right now you have no good reason to believe that will happen. – goldilocks Aug 17 '15 at 13:09
1

Create symlinks in two levels. When creating a new file, also create a symlink to it and put that symlink in the same directory as the temp file. Then create a second symlink for the webserver which points to the first symlink. The second symlink should never have to be rewritten, only recreate the first symlink.

  • This is interesting. Does this mean the file system is clever enough not to update inode tables in flash memory about ANY files or folders that are stored in ram disk, which has been defined in /etc/fstab ? – NewtownGuy Aug 17 '15 at 13:05
  • @NewtownGuy A symlink does't point to an inode; it contains an actual path. So if you link /ramdisk/whatever to /ramdisk/foo, and /ramdisk/foo to /fs/foo, you can change the target of /ramdisk/foo as much as you like; even when its own inode changes, as long as its path remains the same, it can be accessed via /fs/foo, and nothing about /fs/foo needs to change. – goldilocks Aug 17 '15 at 13:20
  • It's correct as goldilocks say. Something like this should solve the problem: ln -sf /ramdisk/one_tempfile /ramdisk/latest_tempfile. ln -s /ramdisk/lastes_tempfile /fs/www/tempfile.txt. – Ronny Nilsson Aug 17 '15 at 14:27
  • I had a problem linking a filesystem file to a tempfilesystem file because the tempfilesystem file does not exist at boot time because tempfilesystem is flushed at boot. So if the system is up and running, and I made a symlink from filesystem to tempfilesystem, that link disappears when the system reboots. I had to put startup code in /etc/rc.local and make sure the file exists, so I had to wait for my application to start, before I make the symlink again. Am I doing something wrong ? – NewtownGuy Aug 17 '15 at 16:53
  • I appreciate that I need to add a link to a link. My concern is that when I tried to make a static link to a temp file, that link disappeared at boot because the file did not exist yet. Won't I have the same problem trying to make a permanent link to a temp link ? – NewtownGuy Aug 17 '15 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.