I am using this Python program to log data from a serial device, and want to add a second document that logs hourly averages of the data. Anyone have an idea on how to add hourly averages?

#!/usr/bin/env python
# Log data from serial port

import argparse
import serial
import datetime
import time
import os

parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument("-d", "--device", help="device to read from", default="/dev/ttyUSB0")
parser.add_argument("-s", "--speed", help="speed in bps", default=9600, type=int)
args = parser.parse_args()

outputFilePath = os.path.join(os.path.dirname(__file__),
                 datetime.datetime.now().strftime("%Y-%m-%d") + ".csv")

with serial.Serial(args.device, args.speed) as ser, open(outputFilePath,'w') as outputFile:
    print("Logging started. Ctrl-C to stop.") 
        while True:
            x = (ser.read(ser.inWaiting())) 
            data = x.decode('UTF-8')
            if data !="":
                outputFile.write(time.strftime("%Y/%m/%d %H:%M ") + " " + data  )

    except KeyboardInterrupt:
        print("Logging stopped")

Here is a sample of the output:

2019/02/09 00:00  22.2,32.0
2019/02/09 00:01  22.2,31.7
2019/02/09 00:02  20.3,29.5
2019/02/09 00:03  17.3,24.6
2019/02/09 00:04  18.9,27.7
2019/02/09 00:05  18.2,27.0
2019/02/09 00:06  16.9,24.6
2019/02/09 00:07  16.7,24.9
2019/02/09 00:08  15.8,23.1
2019/02/09 00:09  15.5,23.5
2019/02/09 00:10  14.8,21.8
2019/02/09 00:11  14.6,21.3
2019/02/09 00:12  14.2,20.7
2019/02/09 00:13  14.0,20.8
2019/02/09 00:14  14.2,21.3
2019/02/09 00:15  14.2,20.4
2019/02/09 00:16  14.2,20.6
2019/02/09 00:17  15.4,23.3
2019/02/09 00:18  15.8,24.1
2019/02/09 00:19  15.8,23.7
2019/02/09 00:20  16.2,24.2
2019/02/09 00:21  15.6,22.6
  • Do you need the averages to be calculated "live", or would be it be OK to process the log files you're generating above and output the averages? If you need them live, is it a requirement that the process pick up where it left off if you stop and restart it, or is it OK, for example, to just print an average 60 minutes after you start running it? Finally, what format is your data in? All we can see from that code is that some data is read and logged to file with a timestamp - we can't see what the data is. – Mark Smith Mar 4 '19 at 16:29
  • A live calculation is preferred. – Michael H. Mar 4 '19 at 23:30

If you want to do it in the same script:

  • When your script starts, record the current time in a variable.
  • Every time you loop, as well as writing out the data, parse it (split on the comma, convert to values) and store in lists.
  • When the current time is one hour or more after the start time:
    • Average the values in the list and write it out
    • Empty the list
    • Set the start time to the current time

IMO a slightly nicer way, though more work, would be to do it in a separate script: it keeps each script clear and simple and with one purpose:

  • Make existing script flush the output after every write
  • Second script reads the log file, parses each line. When it has read an hour's worth it calculates the average and logs it.

By the way, your code has a latent bug: you make the assumption that you will either have a complete "data" ready to read, or nothing. This is risky - you might be in the middle of it and read half of it, and then you will store, for example

2019/02/09 00:00  22.2,3
2019/02/09 00:01  2.0

You could create a separate script that would access the file made by the first script and then calculate and write the averages to another file.

  • I am looking to have this done as the data is collected not after. – Michael H. Mar 4 '19 at 23:30
  • You would run the scripts simultaneously – ThisRandomGuy Mar 5 '19 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.