I have to ensure that every minute I open, write, and close a log file. It must be closed as we trigger power loss randomly on the Rpi3. I known it is bad and I also know that you want to use something like this for a nice shutdown: https://www.raspberrypi.org/forums/viewtopic.php?t=39059

But, well I have no time to implement the hardware and on top of that the behavior is abnormal from my point of view as if I close(myfile) in Python then the system should release all handles, am I correct ? Hence any damage on power-loss should be randomly happening in the file system, if any, but not specifically to my file.... - which after a power cut ends up usually with "^@^@^@^@^@^@^@^@" - and needs to be cleaned...which triggers all sort os issues.

While testing my loop I cut power at 20 seconds, meaning 20 seconds AFTER the close(myfile) and 40 seconds before the next open(myfile,'a') & write(mytring) and close(myfile). - all this in a separate function from main which is called every minute. But the file EOF is anyway corrupted with many non UTF-8 characters.... The side effect for my application which needs to read the last data in the log file is that of course when I restart my program and read this file I have an issue - either to clean or re-initialise the log, which is bad for my work as the log can be huge and I have near RT data to collect....

My question is how to ensure Python (or the system?) release ALL handles on files it as used after a close() ?

For info, I have tried 3 different methods to open write and close on a log (text) and all end-up the same way of having bad EOF on power loss....- so suddenly I wonder if the issue is not somewhere else... as reference, I tried with (a)python logging function and (b) command-line push from python and (c) standard python open()/write()/close() and the very same issue of non UTF-8 characters/ corrupted EOF appears if I trigger a power-cut during my loop when the file should firmly closed and my program not writing to the log file.... Weird, Am I missing something?

  • 3
    independently of Python you might also want to "sync" so that you get a consistent file system on disk. – OyaMist Jun 8 '18 at 20:27
  • 3
    Nothing to do with Python or the Pi. If you pull the plug on any OS you risk file system corruption. Most systems prevent you from doing this – Milliways Jun 8 '18 at 22:41
  • it is quite possible that the SD card file system is not updated until some time after close() .... raspberrypi.stackexchange.com/questions/7424/… – jsotola Jun 9 '18 at 0:07
  • Well done @OyaMistAeroponics ! That was it the 'sync'. I force the sync commandin my function after the write(myfile) and the it forces the system to close! Thanks! – Robert VERGNES Jun 9 '18 at 13:23
  • Thank you for verifying. I've added an answer so others may benefit. Is that acceptable? – OyaMist Jun 9 '18 at 14:29

Applications such as Python write disk files. Behind the scenes, the operating system is caching your changes in memory, not on disk. The operating system caches your changes so that it can efficiently write them out at a later time. Disk writes are slow, therefore infrequent.

Given that your changes are cached in memory, power loss is catastrophic. It never got written to disk. Unix provides a single command called "sync" that writes out all operating system caches to disk and preserves the sanity of your system. This is why best practice for shutting down a Unix system is to call "shutdown". The power switch is deadly without sync.

Trying sync before powering off or simply "sudo shutdown now"

| improve this answer | |

My question is how to ensure Python (or the system?) release ALL handles on files it as used after a close() ?

If you are running a system where it is normal to arbitrarily kill the power, you probably want the filesystem to stay synced to storage. This is not the default since it can be a performance hit and will increase wear and tear on the storage, but it is a better choice in this context.

Edit /etc/fstab and find the entry for the root filesystem. This is the one with just / in the second field. It will also be the only one with a 1 as the value in the last column.

To the fourth column, which reads defaults,noatime add sync, i.e., so you now have defaults,noatime,sync. Beware not to include any whitespace around the comma.

All writes to the filesystem will now be done immediately and not cached by the OS. However, buffering on an application level will still exist -- there is only an immediate write to the filesystem if you flush-on-write (I'm not a python user so can't tell you how to do that, but it should be simple) or, as you are doing now, close the filehandle.

Note that if the power kills are completely random there is still the chance of corruption occurring because of bad timing (e.g., if a hardware transfer is in progress), but it should be significantly reduced.

| improve this answer | |

The python command you miss is:
This command forces python to sync all buffers to disk. https://www.geeksforgeeks.org/python-os-sync-method/

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.