Safely Logging Data on Raspberry Pi

Question

Currently I am reading data from a GPS receiver through USB on a raspberry pi and I would want to log this data on the SD card. I am collecting the data through Python by using

file = open("file.txt", "w")
while True:

    ###PULL DATA THROUGH USB###

    file.write( DATA )

Currently I am SSHing ( using Putty) into the RPi to initiate this program. However there is no constant network connection to the RPi and thus SSH connection loses communication at times and sometimes the data doesn't isn't written and saved. I have also tried running the program at start-up by changing the rc.local but when I pull the power and extract the SD Card, the data seems to be lost.

Is the program still running if the SSH disconnects (goes out of receiver range)?

I was wondering what the best way to log data when there is no way to send a stop command (like a keyboard interrupt to end python data collection program) is?

Answer 1

to avoid losing data, you have to open your file in append mode:

f = open( 'file.txt', 'a')

then write your data and then close it:

f.write(DATA)
f.close()

when you close the file, your data fill be saved to SD and all file tables properly updated.

if you want to keep your file open, you may try to flush/fsync the contents:

f.flush()
os.fsync(f.fileno())

however in the flush/fsync case some data still might be lost, so i'd recommend using append/close approach above.

Answer 2

Along with the append mode for file I/O (as answered by others), you can also try using screen command. This how, you are able to put any task into separate "screen" and don't need to worry about connectivity problems when doing stuff remotely.

I would also recommend using some other logging method, like SQL database (e.g. MySQL with Python), if there is a lot of logging going on. SQL data can easily get exported (e.g. to CSV file) and also any big data management is faster, like (conditional) searching, updating and deleting the data.

Answer 3

I think there are a few questions here- first, dropped ssh means processes in the shell will go away. Look into nohup.

Second, rc.local should work, ensure you have given proper paths since your environment variables won't be provided to the script. (look at /var/log/messages and know it's similar to the same problem happening in cron). Otherwise you might have buffering issues that keep it from writing.

My suggestion is to put a test program in rc.local that simply opens/writes/closes a file and make sure it runs properly, which will ensure your paths are correct. You can start it with /full/path/to/python /full/path/to/script.py. You can also add it to your crontab to ensure it works rather than starting the pi each time.