1

It seems that the RPi does not necessarily write data to the SD very promptly and data loss can occur if the power is pulled after a write operation; even after a few minutes or longer. this can be easily demonstrated with something simple like ls > DEMO and then pulling the power within a second or so. On power up, DEMO will not be there. ls > DEMO && sync and power-cycle and the file will be there. If the fs is mounted 'sync' then the file is also intact, but this doesn't seem to be an ideal solution. Is there a way to not have to mount 'sync', but reduce the time taken to (auto)sync?

The above demo is harsh, but a delayed sync by only a few seconds would be acceptable, the seemingly quite long wait at the moment is frustrating. (and I realise that 'don't pull the power, use a proper shutdown' is a solution, but this cannot be enforced in the environment these are used!)

Thanks,

2
1

You want to have your cake and eat it too:

If the fs is mounted 'sync' then the file is also intact, but this doesn't seem to be an ideal solution.

[...]

I realise that 'don't pull the power, use a proper shutdown' is a solution, but this cannot be enforced in the environment these are used!

It may be worth noting that there are no operating systems which can satisfy these criteria, period because it is a logical impossibility.

Either:

  1. You do your best to ensure a proper shutdown. If the environment poses a problem, this may mean you need need to add circuitry and a small battery. Or...

  2. You use a read-only filesystem. Or...

  3. You keep the filesystem synced to hardware, which is a potential performance hit.

That's it. This is literally physics. There is no possibility for a magic bullet. There are no devices in existence which are a magical exception to this reality.

I'm not sure if using sync affects reads, but I would doubt it. The OS caches things like shared libraries in memory largely read-only to start with, so you should not suffer too much that way. It is only applications that have to write data to storage that will be affected.

However...

a delayed sync by only a few seconds would be acceptable

This is plausible using an ext3/4 filesystem. Instead of using sync in the mount options, use commit=N, where N is a number of seconds. This is documented in man filesystems:

commit=nrsec

Start a journal commit every nrsec seconds. The default value is 5 seconds. Zero means default.

If you look under "ext4" in that manual page, you will see other options such as max_batch_time; if you are doing a lot of writing, fine tuning these may be helpful.

Of course, since the default commit time is 5 seconds, this does not explain the behaviour you are seeing on the scale of minutes. That being the case, simply changing the time will probably not solve the problem. A potential complication is the use of "erase blocks" by SD media: These are commonly 4 MB or more. When a change is made to a file which doesn't occupy an entire block, the entire block where it resides is re-written, which could include pretty much anything from whatever filesystem (including metadata). If you suddenly kill the power to the card (which has its own microcontroller inside) during this re-write process, you stand to lose data in a random way. Unfortunately the cards are a black box to the OS, so there is no possible strategy here; even keeping the filesystem synced does not remove the risk. The only solution in this case is to either use a read-only system, or add power loss prevention circuitry.

3
  • Hi, thanks for the reply; yes, cake and eat it please :) but seriously, I realise the example I gave was unrealistic, it was just a handy way to illustrate the conditions. commit=N would seem to be the thing, although I'm surprised that the default 5s is not enough in the situation that is being described to me. I will have to investigate further. Thanks again. Aug 17 '18 at 0:55
  • I added a final paragraph about this. Good luck!
    – goldilocks
    Aug 17 '18 at 12:24
  • Hi, your insights appreciated. thanks. One other thread I was reading was talking about the same 4MB block and how that may be an issue. The symptom as described to me is a file being opened, written and closed (in Python) but then after a power cycle the file being present, but empty. (0-length). mounting 'sync' seems to have solved this, even for 'instantaneous' switch-offs. I want to try some experiments not on a production system to try and capture the apparent total-loss. Aug 17 '18 at 21:38
0

One simple solution would be to run sync from a script or a cron job every X seconds. Such a script can be activated and deactivated on demand, so that you can avoid the performance hit if you stop it e.g. before installing software updates.

Not the answer you're looking for? Browse other questions tagged or ask your own question.