4

Guys I'm having a problem when I write/save many files in Raspberry Pi.

I'm using the RaspiCam to record videos in HD and at the same time I'm collecting alot of data and writing them in XML files every 1 second. The data is collected at 10 Hz.

I don't know why, but sometimes when I'm saving the XML file, it takes 3-4 seconds to save the file, that's too much. It happens once within 30-40 seconds.

It occurs very often and I'm losing data because of that. I tried to debug and I noticed that when it occurs is where I call the function fopen.

Does somebody know, what is happening?

I'm using the tinyxml2 library to write and save the XML files.

"Edited"

Here is a log file that I did to see what times it's freezing. 

[INFO]: [OBD Buffer]: 0 [GPS Buffer]: 26    [IMU Buffer]: 3  [XML Time]: 0.988022(s)

[INFO]: [OBD Buffer]: 2 [GPS Buffer]: 45    [IMU Buffer]: 22 [XML Time]: 2.89335(s)

As you can see, the XML files were saving normally, but suddenly it took 3 seconds to save. Because of that, as my other threads are still running and collecting data, my OBD/GPS/IMU buffers gets full and I start to lose data because the XML is freezing when saving.

Solution: I had to increase the time of saving the XML's files. RaspbianOS isn't a good choice for perfomance in real time. I'm saving the XML's files every minute now and I'm not losing information anymore. So, to sum up I cannot save the XML's files every second. I think the nsilent22' answer can describe it better.

Thank you all for your support.

Improve this question
5
  • 1
    IIRC fopen() ("standard buffered file open") opens the file in "blocking" mode - it does not return until either it has succeeded or failed - if the file-system is busy servicing other requests your program will "block" until the file-system is free to use. You may need to look at open() with a O_NONBLOCK option, in either case I suggest that you explore your options with a perusal of the manual for fopen and open.
    – SlySven
    Commented Jan 19, 2016 at 4:35
  • I tried some time ago write only the XML files, so I stopped to record videos and I started to write/save only the XML files to the SD Card. This didn't solve and the problem still happening. So, I don't know what can be blocking the I/O operation. Isn't too dangerous opening with a O_NONBLOCK option? I need to be sure that the XML was saved successfully. Perhaps I should change the OS to a RTOS? Thank you for your help.
    – Lucas Casagrande
    Commented Jan 21, 2016 at 13:17
  • @LucasCasagrande: I don't think a RTOS would help you any. Are you using an RPi 2? What language C, Java, or Python?
    – Jacobm001
    Commented Jan 21, 2016 at 16:09
  • You need to think about the program flow a bit if you use 0_NONBLOCK - although the open will return straight away- it does not mean that you can write to the file so opened - you may get an EAGAIN error if you try and your request cannot be honoured or EINTR if it was interrupted before any bytes were written or a byteWritten count less than you expected if any were written. Indeed the man page for read(2) points out: A successful return from write() does not make any guarantee that data has been committed to disk! You are likely to have to maintain your own cache in memory.
    – SlySven
    Commented Jan 21, 2016 at 18:16
  • I'm using the RPi 1 and programming in C++.
    – Lucas Casagrande
    Commented Feb 8, 2016 at 14:22

1 Answer 1

Reset to default
6

Linux kernel is caching disk operations. That means data is written not in real time, but when it is "time to do it".
There could be two reasons of behaviour you described:
1. Your SD card (or disk) is not fast enough to receive all the data you are producing.
2. Once in 30-40 seconds kernel is 'flushing' it's disk buffers, which unluckily freezes all disk operations - and your process.
There's no 'cure' for the first reason other than buying a faster medium.
For the second reason you can try following solution:

  1. Mount tmpfs (ramdisk) system in some directory, e.g. /tmp/mydata

    sudo mount tmpfs /tmp/mydata -t tmpfs

  2. Grant yourself write permissions to that directory:

    sudo chown youruserid:yourgroupid /tmp/mydata

  3. Make your program write to this directory. It will be lightning fast, as it's ramdisk. After writing each somename.XML file you should also 'touch' somename.XML.ready file, to signal that somename.XML is complete.

  4. Run second process that will monitor /tmp/mydata for existence of somename.XML.ready files and then will move somename.XML file to destination disk device (and remove somename.XML.ready).
  5. The second process could get stuck for some seconds because of disk operations, but later it should catch up (if your disk is fast enough), and your first shall operate normally, as it won't be affected.
Improve this answer
3
  • I did a little benchmark and here is what I got. My SD Card is a Sandisk class 10 and the write speed was 10mb/s. I think the SD Card is fast enough. While I'm writing the XML he is in memory and the freezes occurs only when I'm saving the XML to the SD, so I Don't know if the second solution would solve the entire problem. I think the second process will still freezing and my buffer will overflow. Perhaps I should change the OS to a RTOS or change to a non-buffered file system. What do you think? I need a way to be sure that the XML files will save in time. 1 XML per second.
    – Lucas Casagrande
    Commented Jan 21, 2016 at 13:43
  • @LucasCasagrande: Your methodology is kind of absurd. Why do you need to save files like this?
    – Jacobm001
    Commented Jan 21, 2016 at 16:06
  • @LucasCasagrande: Just try it.
    – nsilent22
    Commented Jan 21, 2016 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.