2

I am attempting to gather a day's worth of thermal data from a thermal camera. This obviously will need more data than the sd card can handle so I have a 1 TB hard drive I have setup using ntfs-3g (with exec option).

When I run my script and either fprintf redirect (./script > file.txt) the output to a location on the HD the file is created, but it is always empty.

Running ./script sends the expected thermal data to the terminal. I have looked all over the internet for similar problems, but I'm not finding them. What have I messed up here?

Added Information:

My data gathering program does not end, I kill it with CTRL-C.

If I wait long enough (a minute or so) files with multiples of 4096 bytes (the block size of the disc, confirmed by "blockdev --getbsz /dev/sda1") will begin to appear.

Some of the initial tests only waited ~5 to 10 seconds with the purpose of making sure the process was working. From what I can tell, when I kill the process with control C, any pending writes are also killed.

The "disc queue" does not write out to disk until it hits a block. So when my script runs for a day, I will probably lose the last minute of data, but that is alright.

I could look into reformatting the disc to a smaller block size, but honestly losing one minute out of around 10 hours of data does not concern me.

  • run dmesg ... any errors regarding the drive? – Jaromanda X Nov 4 '17 at 1:53
  • 1
    Also use >> to append to the file rather than > which overwrites (unless you are actually planning to write a massive file each time). – joan Nov 4 '17 at 1:56
  • Why would you use NTFS rather than exFAT? What do you mean by ”thermal data”? Are they individual JPEGs or video? Do you ever flush or sync in your program? – Mark Setchell Nov 5 '17 at 21:47
3

Register Signal Handlers in your program in order to catch the ctrl-c and flush all the buffers/files opened by your program and script.

Edit Following Additional Information

Killing a program with ctrl-c will likely cause data truncation. This is actually implementation dependent and not always consistent between linux systems (some will flush all buffers on exit). You need to change your design to shutdown gracefully.

Some Background:: There are actually Two Buffers

  1. Userspace Buffer, when a file is opened by a program you get a kernel provided buffer, this is where data from fprintf goes.
  2. Kernel Disk Cache, kernel caches file in memory before writing to disk.

sync (or fsync) is used to for #2 , flush is used for #1.

Note: close is guaranteed to issue flush, but exit is not guaranteed to close all open files.

When you interrupt the program, you lose unwritten data in the Userspace buffer, data already written to the kernel (flushed) may be cached but won't be lost.

However, if pull the power from your raspberry PI before performing an sync some data may be lost on reboot.

Install a Signal handler

If this is a program or script you have written yourself, you should install a signal handler , in order to "react" to the ctrl-c you have issued.

You can install signal handler to catch SIGINT (ctrl-c) and issue flush, fsync and sync commands

Signal Handlers can be used in all major programming languages and shell scripts.

In the context of an application, your signal handler would issue the command flush or close to any open streams/files followed by an fsync per open file for good measure.

In the context of a shell script, your signal handler should gracefully close the running process, and maybe a sync to make sure no data is lost on reboot.

Other things to try

  1. In order to better diagnose your issue you run

    ./script &> file.txt

    This will cover the unusual case that your script is writing to stderr and not stdout, just to be sure its not something so trivial

  2. To verify FS and Driver are functioning

    echo test > file.txt

  3. Sometimes kernel buffering and driver options will result in extended delays for writes to disk drives. Use sync to force kernel to flush filesystem caches

    ./script > file.txt ; sync

  • Thanks for your reply! a few interesting things: the first bullet point worked, I inserted an error, and it made the appropriate file. The second bullet point worked as well. However the third still results in an empty file. I know my script generates output, as I sanity check multiple times that it is, but I still cannot write it to the disk for some reason. – slightlyconfused Nov 4 '17 at 16:29
  • Correction: it works occasionally, but not often. (1 in 10 times or so) – slightlyconfused Nov 4 '17 at 16:34
  • It seems like short tests fail more often - is it possible these small writes simply get removed from the "write queue" when the process requesting the write is killed before they go out to disk? – slightlyconfused Nov 4 '17 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.