persistent canbus data logging during power loss

Question

I am trying to setup a robust raspberry canbus data logger. The pi will be installed on a mobile vehicle where the power to the pi is disconnected every time the vehicle is switched off, which will happen several times during the day.

The pi will have a readonly mounted SD card for the OS and an external SSD (probably ext4) where the logfiles are saved. The datarate of the canbus is 125kbps.

It's crucial that even in case of powerfail the logfile or worse the SSD does not get corrupted and no data is lost.

Primary I want to explore software based solution that will minimize the risk of data loss.

Secondary I also will consider hardware based solutions like adding a battery or capacitor, so the pi can be shutdown properly.

Is there any kind of best practice or are there any existing linux tools that will minimize the risk of data loss for IO operations where power can suddenly fail?

Answer 1

Outside of adding hardware to clean shutdown the system when the vehicle is turned off. One option is to mount the usb using the sync option of the mount command. From the manpage for mount:

sync: All I/O to the filesystem should be done synchronously. In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening.

N.B. The comment on shortening the life of the drive.

using the mount command where sdb1 is the usb partition to mount:

sudo mount -o sync /dev/sdb1

for automounting the usb the fstab should be edited to include the drive, something like e.g.

/dev/sdb1 /mnt/usb vfat auto,users,sync,noatime,uid=1000,gid=1000  0  0

Answer 2

Barring motorbikes, bicycles, and antique cars with manual cranks (of which the last two types don't have CAN buses), every "vehicle" has a battery, either as a primary source of energy, or as a source for combustion engine starter. All you have to do is find a way to connect your data logger directly to the battery, not to a secondary circuit which is unpowered when the vehicle stops.

A second option would be to attach some sort of UPS or independent power supply to the RPi. Juice4halt would be one option, but many more can be found with casual googling.

Finally, there's an option of letting the RPi to power-down suddently and deal with the consequences. In my experience, enabling the journalling on EXT4 file system is usually enough to deal with filesystem corruption. It will not prevent the file corruption though: the journalling roughly guarantees that the file contents will match its size as reported by the file system, but it has no means of enforcing "correctness" of a file, since it has no idea what it means for a specific file type (e.g. for XML closing tags would be needed, for a ZIP file the archive directory must be fully written, and so on). Simple text logs can usually be parsed as long as a file is readable, and journalling helps with that.

If the performance is not a concern, adding sync to mount options in addition to journalling will improve robustness: without sync more write operations immediately preceding the power loss will be incomplete and will have to be rolled back, so your log will have less data. Note that sync by itself will not guarantee that the system is safe to power down anytime, it will just result in more frequent disk writes by essentially ignoring the Linux write cache.