0

I have a small script which continuously logs data to my Raspi3 (kinda a data logger).I am writing all the collected data in to a plain txt file. My Pi logs data for a month or two continuously. So the file size increases (like 2 gb ). Now I am trying to plug in a USB in to a port of the Pi so that it copies the txt file automatically to the USB. How can i achieve this (also i want the data to get copied soon ) ?

I have tried automount, and I have added a line to the USB config file

KERNEL=="sd*", SUBSYSTEMS=="usb", ACTION=="add", RUN+="/home/pi/Testscript.sh"

This triggers the script when a USB is plugged and copies the data if the USB is empty as per the program in the script.But the copying process for huge file is slow.

Any suggestions or help ?

  • Some beermat maths suggests that, if your Pi's logging 2GB in 30 days, your Pi is logging something in the region of 830 characters per second. That sounds like quite a lot of characters. Have you considered attempting to reduce the file size, and the copying time required, by increasing your recording interval or reducing your overheads (identifiers, separators, etc.)? – goobering Sep 19 '16 at 14:34
2

What's "slow"?

A transfer from the SD card can't exceed the speed of the SD card reader, which with a decent card will be 15-20 MB/s and has an absolute theoretical maximum of 25 MB/s.

At ~20 MB/s it should take 1.5 - 2 minutes to transfer.

If the system is busy with other I/O to the card, e.g., you are still logging, I would expect that to have a noticeable impact.

The USB and ethernet jack share one bus limited to 280 Mbps total (35 MB/s), which should be enough to allow for any network connection and the transfer to occur simultaneously at high speed, but that's it. So this could be a factor as well.

1

copying is pretty much only IO bound, not a lot you can do to speed this up. What you can and probably should do is gzip -c >/mnt/usb/target.gz the text file, which will reduce its size considerably and thus speed the operation.

  • 2
    This won't help with a normal transfer to USB. Compressing the data is much slower than transferring it. Logically, it cannot happen faster than the data is being read, and it is the read speed from the SD card that is probably the bottleneck. Unless writing to the USB device is the bottleneck, but it is unusual for them to be slower than the SD card and it would have to be really bad. Gzip works as a single thread; I just tried it on a Pi 3 with a 60 MB file and it took 15+ seconds. – goldilocks Sep 19 '16 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.