1

I am currently building offline logger - every 10 seconds the application will save GPS location + temperature and humidity.

And now comes the question: what is the best way to store these data on RPi?

I'm thinking about storing these data in .json file or sqlite database, but I suppose there can be better way to do this (something quick and what won't break my SD card by many write operations).

Thanks!

Improve this question

4 Answers 4

Reset to default
3

Just use CSV. That way you can just append your data to the existing file. That way only the last block will change, resulting in fewer writes to the SD. Once the block is full, it will never be rewritten.

Maybe do some logrotate to create a separate file for every day. That way any data-corruption will only mess up one day of data.

Improve this answer
5
  • 1
    +1 For recommending a flat file. Something to watch out for if a persistent process is doing the logging and you rotate the logs it writes to: if the process is holding a file handle open to the log, that handle is to an inode not a path, even though you use a path to create it. This means if you move whatever.log to whatever.log.1 and create a new whatever.log, the process's filehandle will not be for either of those, and subsequent logging will be lost probably without causing any error. So if there's a persistent process with an open file handle involved...
    – goldilocks ♦
    Feb 16, 2014 at 16:14
  • ... you must restart the process AFTER rotating the log. I wouldn't use CSV, BTW, I'd just use space separated columns (e.g. timestamp temp humidity) --in fact I log temperature and barometric pressure with the pi exactly that way.
    – goldilocks ♦
    Feb 16, 2014 at 16:15
  • I always thought the file handle would move with the file (so whatever.log.1). Only if you delete the old file, or replace it with a new file, will the old file handle point to a "non-existing" file.
    – Gerben
    Feb 16, 2014 at 20:54
  • 1
    If the logrotate includes compressing the old file, it would be replaced.
    – TomG
    Feb 17, 2014 at 3:13
  • 1
    @Gerben : Hmmm, yeah, you're right. Still not desirable though. It does get lost if you then change the moved file (as TomG points out), but this will happen if you change it without moving it too...anyway, something to be aware of. If you are only writing to the file every ten seconds, opening and closing it each time might be a better option.
    – goldilocks ♦
    Feb 17, 2014 at 14:00
0

any plain old text file would do. still there are two things you should watch out for:

don't forget to use your file in open/write/close sequence. people often tend to forget about the "closing" part, they open a file and keep writing until something happens (crash, reboot?) leaving incomplete file. if you use python, this should take care of it:

with open( 'log.txt', 'a' ) as log :
    print >>log, date_time, temperature, humidity, latitude, longitude, etc...

"with" will automatically close the file, so you don't have to worry about all this "you must restart after rotating the log" nonsense.

also, you don't really need to store temperature and humidity every 10 seconds, unless in a very harsh environment. it might be a good idea to write three separate log files and store the values only when changed -- this might considerably reduce your file size and wear of your SD card.

Improve this answer
0

I might have suggested a flat file in the past, but SQLite has compelling advantages:

  • it handles queries without having to hand-roll code; would you like to query a CSV file for the average temperature for last Wednesday?
  • files and transactions are managed sensibly. Sure, it might not cope very well if the power to the Raspberry Pi were yanked in mid-write, but most filesystems don't, either.
  • SQLite is already quietly managing much more of your data than you know. Your phone and your browser rely on it. To paraphrase the authors, it won't replace Oracle but it might replace fopen().
  • A SQLite file is readable on any system. CSV … uh, are the newlines CRs, LFs, or CRLFs? Commas are quoted as …? Strings are quoted how? Embedded newlines: disallowed, handled gracefully, or fail insideously?
  • If you are recording GPS locations, consider the SQLite extension SpatiaLite (in the Raspbian repos: sudo apt-get install python-pyspatialite libspatialite3 proj-bin). It turns SQLite into a reasonably fully-featured geodatabase, so your data points have geometry. Want to know how far your logger has travelled? Want to see its track as a KML file for Google Earth? Both are single queries in SpatiaLite.

Yes, SQLite files can get a lot bigger than flat files, especially when they are writing large journals. But this won't likely happen for a simple logging application.

Improve this answer
0

I'm doing something similar with Node-RED. Basically, my phone sends GPS coordinates, battery temperature and charge level to Node-RED running on my Raspberry Pi, which then stores it in a database or CSV file or json file. You can get started Node-RED by following this tutorial.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.