4

Currently I am reading data from a GPS receiver through USB on a raspberry pi and I would want to log this data on the SD card. I am collecting the data through Python by using

file = open("file.txt", "w")
while True:

    ###PULL DATA THROUGH USB###

    file.write( DATA )

Currently I am SSHing ( using Putty) into the RPi to initiate this program. However there is no constant network connection to the RPi and thus SSH connection loses communication at times and sometimes the data doesn't isn't written and saved. I have also tried running the program at start-up by changing the rc.local but when I pull the power and extract the SD Card, the data seems to be lost.

Is the program still running if the SSH disconnects (goes out of receiver range)?

I was wondering what the best way to log data when there is no way to send a stop command (like a keyboard interrupt to end python data collection program) is?

  • If you use RDP (sudo apt-get install xrdp) then you know for sure that when you disconnect the session will continue to run. You can access it with Windows Remote Desktop or remmina. – SDsolar Aug 14 '17 at 15:08
2

to avoid losing data, you have to open your file in append mode:

f = open( 'file.txt', 'a')

then write your data and then close it:

f.write(DATA)
f.close()

when you close the file, your data fill be saved to SD and all file tables properly updated.

if you want to keep your file open, you may try to flush/fsync the contents:

f.flush()
os.fsync(f.fileno())

however in the flush/fsync case some data still might be lost, so i'd recommend using append/close approach above.

  • This is the best answer. If the file doesn't exist when this is executed, it will be created. For my data loggers I take your advice in the first part. I open the file, write to it then close it right away. – SDsolar Aug 14 '17 at 15:05
2

Along with the append mode for file I/O (as answered by others), you can also try using screen command. This how, you are able to put any task into separate "screen" and don't need to worry about connectivity problems when doing stuff remotely.

I would also recommend using some other logging method, like SQL database (e.g. MySQL with Python), if there is a lot of logging going on. SQL data can easily get exported (e.g. to CSV file) and also any big data management is faster, like (conditional) searching, updating and deleting the data.

  • +1 I was just about to post this very thing. I wouldn't recommend MySQL on the pi. It is usable, but it's a bit overkill for most needs. Sqlite3 would probably be a better bet. – Jacobm001 Jul 30 '13 at 21:14
  • Yes, I agree with you. I just put example of possible data logging facility. :) – TomiL Oct 16 '13 at 16:33
0

I think there are a few questions here- first, dropped ssh means processes in the shell will go away. Look into nohup.

Second, rc.local should work, ensure you have given proper paths since your environment variables won't be provided to the script. (look at /var/log/messages and know it's similar to the same problem happening in cron). Otherwise you might have buffering issues that keep it from writing.

My suggestion is to put a test program in rc.local that simply opens/writes/closes a file and make sure it runs properly, which will ensure your paths are correct. You can start it with /full/path/to/python /full/path/to/script.py. You can also add it to your crontab to ensure it works rather than starting the pi each time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.