Writing data (dynamically) to an external drive

Question

I am attempting to gather a day's worth of thermal data from a thermal camera. This obviously will need more data than the sd card can handle so I have a 1 TB hard drive I have setup using ntfs-3g (with exec option).

When I run my script and either fprintf redirect (./script > file.txt) the output to a location on the HD the file is created, but it is always empty.

Running ./script sends the expected thermal data to the terminal. I have looked all over the internet for similar problems, but I'm not finding them. What have I messed up here?

Added Information:

My data gathering program does not end, I kill it with CTRL-C.

If I wait long enough (a minute or so) files with multiples of 4096 bytes (the block size of the disc, confirmed by "blockdev --getbsz /dev/sda1") will begin to appear.

Some of the initial tests only waited ~5 to 10 seconds with the purpose of making sure the process was working. From what I can tell, when I kill the process with control C, any pending writes are also killed.

The "disc queue" does not write out to disk until it hits a block. So when my script runs for a day, I will probably lose the last minute of data, but that is alright.

I could look into reformatting the disc to a smaller block size, but honestly losing one minute out of around 10 hours of data does not concern me.

Answer 1

Register Signal Handlers in your program in order to catch the ctrl-c and flush all the buffers/files opened by your program and script.

Edit Following Additional Information

Killing a program with ctrl-c will likely cause data truncation. This is actually implementation dependent and not always consistent between linux systems (some will flush all buffers on exit). You need to change your design to shutdown gracefully.

Some Background:: There are actually Two Buffers

1. Userspace Buffer, when a file is opened by a program you get a kernel provided buffer, this is where data from fprintf goes.
2. Kernel Disk Cache, kernel caches file in memory before writing to disk.

sync (or fsync) is used to for #2 , flush is used for #1.

Note: close is guaranteed to issue flush, but exit is not guaranteed to close all open files.

When you interrupt the program, you lose unwritten data in the Userspace buffer, data already written to the kernel (flushed) may be cached but won't be lost.

However, if pull the power from your raspberry PI before performing an sync some data may be lost on reboot.

Install a Signal handler

If this is a program or script you have written yourself, you should install a signal handler , in order to "react" to the ctrl-c you have issued.

You can install signal handler to catch SIGINT (ctrl-c) and issue flush, fsync and sync commands

Signal Handlers can be used in all major programming languages and shell scripts.

In the context of an application, your signal handler would issue the command flush or close to any open streams/files followed by an fsync per open file for good measure.

In the context of a shell script, your signal handler should gracefully close the running process, and maybe a sync to make sure no data is lost on reboot.

Other things to try

1. In order to better diagnose your issue you run

   ./script &> file.txt

   This will cover the unusual case that your script is writing to stderr and not stdout, just to be sure its not something so trivial

2. To verify FS and Driver are functioning

   echo test > file.txt

3. Sometimes kernel buffering and driver options will result in extended delays for writes to disk drives. Use sync to force kernel to flush filesystem caches

   ./script > file.txt ; sync