Skip to main content
spelling
Source Link
nsilent22
  • 421
  • 4
  • 6

Linux kernel is caching disk operations. That means data is written not in real time, but when it is "time to do it".
There could be two reasons of behaviour you described:

  1. Your SD card (or disk) is not fast enough to receive all the data you are producing.

  2. Once in 30-40 seconds kernel is 'flushing' it's disk buffers, whatwhich unluckily freezes all disk operations - and your process.
    There's no 'cure' for the first reason other than buying a faster medium.
    For the second reason you can try following solution:

  3. Mount tmpfs (ramdisk) system in some directory, e.g. /tmp/mydata

    sudo mount tmpfs /tmp/mydata -t tmpfs

  4. Grant yourself write permissions to that directory:

    sudo chown youruserid:yourgroupid /tmp/mydata

  5. Make your program write to this directory. It will be lightning fast, as it's ramdisk. After writing each somename.XML file you should also 'touch' somename.XML.ready file, to signal that somename.XML is complete.

  6. Run second process that will monitor /tmp/mydata for existence of somename.XML.ready files and then will move somename.XML file to destination disk device (and remove somename.XML.ready).

  7. The second process could get stuck for some seconds because of disk operations, but later it should catch up (if your disk is fast enough), and your first shall operate normally, as it won't be affected.

Linux kernel is caching disk operations. That means data is written not in real time, but when it is "time to do it".
There could be two reasons of behaviour you described:

  1. Your SD card (or disk) is not fast enough to receive all the data you are producing.

  2. Once in 30-40 seconds kernel is 'flushing' it's disk buffers, what unluckily freezes all disk operations - and your process.
    There's no 'cure' for the first reason other than buying a faster medium.
    For the second reason you can try following solution:

  3. Mount tmpfs (ramdisk) system in some directory, e.g. /tmp/mydata

    sudo mount tmpfs /tmp/mydata -t tmpfs

  4. Grant yourself write permissions to that directory:

    sudo chown youruserid:yourgroupid /tmp/mydata

  5. Make your program write to this directory. It will be lightning fast, as it's ramdisk. After writing each somename.XML file you should also 'touch' somename.XML.ready file, to signal that somename.XML is complete.

  6. Run second process that will monitor /tmp/mydata for existence of somename.XML.ready files and then will move somename.XML file to destination disk device (and remove somename.XML.ready).

  7. The second process could get stuck for some seconds because of disk operations, but later it should catch up (if your disk is fast enough), and your first shall operate normally, as it won't be affected.

Linux kernel is caching disk operations. That means data is written not in real time, but when it is "time to do it".
There could be two reasons of behaviour you described:

  1. Your SD card (or disk) is not fast enough to receive all the data you are producing.

  2. Once in 30-40 seconds kernel is 'flushing' it's disk buffers, which unluckily freezes all disk operations - and your process.
    There's no 'cure' for the first reason other than buying a faster medium.
    For the second reason you can try following solution:

  3. Mount tmpfs (ramdisk) system in some directory, e.g. /tmp/mydata

    sudo mount tmpfs /tmp/mydata -t tmpfs

  4. Grant yourself write permissions to that directory:

    sudo chown youruserid:yourgroupid /tmp/mydata

  5. Make your program write to this directory. It will be lightning fast, as it's ramdisk. After writing each somename.XML file you should also 'touch' somename.XML.ready file, to signal that somename.XML is complete.

  6. Run second process that will monitor /tmp/mydata for existence of somename.XML.ready files and then will move somename.XML file to destination disk device (and remove somename.XML.ready).

  7. The second process could get stuck for some seconds because of disk operations, but later it should catch up (if your disk is fast enough), and your first shall operate normally, as it won't be affected.

Source Link
nsilent22
  • 421
  • 4
  • 6

Linux kernel is caching disk operations. That means data is written not in real time, but when it is "time to do it".
There could be two reasons of behaviour you described:

  1. Your SD card (or disk) is not fast enough to receive all the data you are producing.

  2. Once in 30-40 seconds kernel is 'flushing' it's disk buffers, what unluckily freezes all disk operations - and your process.
    There's no 'cure' for the first reason other than buying a faster medium.
    For the second reason you can try following solution:

  3. Mount tmpfs (ramdisk) system in some directory, e.g. /tmp/mydata

    sudo mount tmpfs /tmp/mydata -t tmpfs

  4. Grant yourself write permissions to that directory:

    sudo chown youruserid:yourgroupid /tmp/mydata

  5. Make your program write to this directory. It will be lightning fast, as it's ramdisk. After writing each somename.XML file you should also 'touch' somename.XML.ready file, to signal that somename.XML is complete.

  6. Run second process that will monitor /tmp/mydata for existence of somename.XML.ready files and then will move somename.XML file to destination disk device (and remove somename.XML.ready).

  7. The second process could get stuck for some seconds because of disk operations, but later it should catch up (if your disk is fast enough), and your first shall operate normally, as it won't be affected.