Setting up the Raspberry as a data logger

Question

As the title states I'm working on a simple embedded application which reads some information from the serial port and logs it to file, in addition to displaying a bit of status on the display.

I'm now at the stage the logging seems to be running and I'm trying to configure the system and I've got a couple of questions.

1. The system is intended to log data to a FAT32 USB stick, which may be unplugged, replugged or replaced at any time. What is the preferred method of auto-mounting any inserted USB mass storage into a fixed directory? The usbmount package seems to nearly do this but the lack of maintenance makes bit wary.

2. I cannot guarantee the power supply and perform clean shut-downs. Thankfully the only files to be written, aside from internal OS information, should be the logs. Will a journalling file system for the OS save me from myself or should I be mounting the system read-only or tinkering with RAM disks?

3. Will the FAT file system on the USB stick survive and recover from power outages and sudden ejections? Should I create a new log file each time it gets mounted?

4. I've set up the X server to start automatically and fiddled with the LXDE autostart scripts to start the status display and disable the window manager/screen saver. Are there any other gotchas to maintaining a continuous display?

5. The logger will not have Ethernet access and rather needs an accurate clock. Is the RasClock module a suitable option?

Sorry about the long post and multiple semi-related questions.

I'm used to dealing with simple embedded devices for which I have full control over the software, and the task of correctly configuring a real operating system is making me a bit anxious. On the other hand I rather suspect that I'm not the only one (ab-)using the Pi for this type of application so hopefully the questions are not too specific to my particular situation.

Answer 1

The system is intended to log data to a FAT32 USB stick, which may be unplugged, replugged or replaced at any time.

The unplugged part of that is problematic. AFAIK, no computer system anywhere promises you the right to yank a USB stick out unannounced at any point without potential problems. So you will have to think about how to get around that.

I cannot guarantee the power supply and perform clean shut-downs.

Again, suddenly cutting the power on (just about) any computer system is risky. It usually ends up okay (who hasn't done this a bunch of times, by accident, due to power failure, etc?), but there seems to be an unusual amount of people reporting SD card corruption with the pi. So if you mean, "Mostly the system will not be shut down cleanly" you need to rethink, because that will be a dead end strategy. Note that it is possible to trigger a shutdown without a keyboard or network connection. You could use a similiar method to trigger a clean unmounting of the usb stick too.

You can add the sync option in /etc/fstab to restrict caching, but man mount notes:

In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening.

This includes SD cards, although as long as you remember to intermittently test (and replace, when appropriate) your media it may be worth while. This will impact general performance slightly, I think.

So, try to avoid yanking the plug or usb stick suddenly as best you can, and certainly don't design a setup that depends on this. Eg, don't tell yourself or other users, "It is okay to unplug the USB stick whenever you want", because it isn't and you can't make it so.

Making appropriate sync calls in your application (eg, when idle, or after a significant write) is a good idea.

Once you have everything you want installed and configured, you can potentially mount the root filesystem read-only, if you create separate partitions for /var, /run, and /tmp and mount those rw; you could also create one such mounted partition and symlink the rw directories into it.

On a raspbian pi, you don't have to worry about /run, as it is tmpfs (RAM) already; /sys and /proc are also in RAM (but don't use them for anything except what they are intended). If the system is for a single purpose and all your data goes to the USB stick, /var is the only problematic one but not, I think, critically so.

Answer 2

In a former life I worked on device drivers that read/wrote memory cards. They could be yanked in the middle of any operation, so even battery or capacitor backed up power wouldn't help. Based on the formatting specs of the chips, I inferred the behavior required to have minimum damage done by "hot" removal and insertion.

Basically, what you need to do is ensure that if you are cut off mid-write of a data block (or the contacts are squirrely) you can recognize the fact. basically prepend the data with a block number (and length if not fixed size), and append a crc and (the same) block number at the end.

The first write you do is at the end, with a value that is NOT the next block number. This way, if your write doesn't complete, there will be a block number mismatch and that data block is known to be garbage. Then write your data block with the real block number, followed by the length, followed by your CRC generated by your favorite algorithm, and the real block number again. (The CRC is to protect against squirrely contacts that corrupt your data).

When power is restored or the memory card re-inserted, only accept sequential blocks to the point the block begin and end sequence numbers don't match or the CRC is incorrect. Remove that block and resume logging.

This is low level stuff though. Doing it through a file system is another story.

What you really need is redundant copies of your FAT, and do a similar thing around the FAT table. The sequential number will tell you which is most recent, and the front/back match and CRC will ensure the integrity.

It's past my bedtime so I hope this is clear enough.

Answer 3

As for the accurate clock, there are lots of cheap GPS modules available. Hard to find a more accurate clock than that.