1

We have a student project where part of it requires us to have an Arduino Nano sending data to a Pi 2 Model B via USB.

The Pi's have been setup using PySerial. My question is regarding the absence of the Pi collecting the data sent.

Will the data stay in an internal buffer on the Pi until serial.readline() is called, and the data is collected? What happens after? Will the act of collecting, (copying/moving) the data delete it from the USB channel? What happens to the data if it's never collected? Does it stay available for collecting a certain time, and then deleted?

Any help is greatly appreciated.

1

Most Arduinos reset (that is, restart the sketch, clearing all data) when the serial connection is opened. So if there's no connection open, any data present before the connection is opened will be lost. There are ways around this, but they interfere with your ability to upload sketches.

The serial output buffer in an Arduino is very small: 64 bytes or fewer. If there were no system buffers, your Arduino sketch would hang, pausing all readings until the serial buffer is read.

Collecting data from the channel deletes it. If you want to see if data is there but not read it, the in_waiting() method returns the number of bytes in the receive buffer.

Update: so my previous answer was based on lots of ‘in theory’ assumptions. I thought I'd test what happens to unread data with a sketch that:

  • wrote an incrementing counter at 115200 bps;

  • indicated while the main loop was running by flashing an LED. If the LED wasn't flashing, the loop() subroutine in the sketch was hung up; and

  • indicated if the Arduino's serial output buffer was nearly full by lighting the internal LED. (This turned out to be less useful than I thought, because it's almost always nearly full.)

I then wrote a Python 3 program to open the serial port, read a few lines, pause for a few seconds, read a few more lines, pause for a longer period, then read some more lines. The results surprised me, especially with the differences between two types of (official) Arduino boards.

Arduino sketch

#define WARN_PIN 13   // built in LED
#define WARN_COUNT 4  // warn at this many bytes left
#define WORK_PIN 7    // alternate this to simulate “work” …
long count = 0;
boolean state = true;

void setup() {
  pinMode(WARN_PIN, OUTPUT);
  pinMode(WORK_PIN, OUTPUT);
  Serial.begin(115200);
  Serial.println("# Buffer Test Started");
}

void loop() {
  Serial.println(count);
  if (Serial.availableForWrite() < WARN_COUNT) {
    digitalWrite(WARN_PIN, HIGH); // set warning LED if buffer nearly full
  }
  else {
    digitalWrite(WARN_PIN, LOW);
  }
  count++;
  if (count % 1000 == 0) {
    state = !state; // every 1000 cycles change work led
    digitalWrite(WORK_PIN, state);
  }
}

The expected output from this sketch would be:

# Buffer Test Started
0
1
2
…

As with all Arduino sketches, each line is terminated by CRLF (\r\n). At a very rough estimate based on watching the LED on port 7, the sketch generated 1000 lines per second.

This sketch was installed on four Arduino boards: Uno R3, Mega 2560, Leonardo and also a cheap Nano clone.

Python Serial Reader

I tested several variants of the following code, testing a pause of ten seconds, a minute, ten minutes and finally an hour between reading from the serial port:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import serial
import time
ser = serial.Serial('/dev/ttyACM0', baudrate = 115200, timeout = 1.0)

# read the first three lines
print(ser.readline())
print(ser.readline())
print(ser.readline())

print("## Sleeping for 5 seconds …")
time.sleep(5.0)

# read the next three lines
print(ser.readline())
print(ser.readline())
print(ser.readline())

print("## Sleeping for 3600 seconds … zzz…")
time.sleep(3600.0)

# read three more lines
print(ser.readline())
print(ser.readline())
print(ser.readline())

If the Raspberry Pi is buffering serial data correctly, the output lines should have consecutive numbers.

Results

The Uno R3, Mega 2560 and Nano clone all produced similar results. Here's the output from an hour pause on the Uno R3 connected to a Raspberry Pi 3, complete with initial line noise burst (and Python's weird obstinacy against do what I mean data type conversion):

b'\t\x86\x1a\xb4\x94\xa7\x96\xa7\x96\xa7\x96\x87\x96$\xb1\xc9\x1fb\x1b\xb3\x0b\x13\x13\x93\x13\xad\x16\x12\xa1\x12\xa1\xb6&\xb6\x1a\xb6\x96\xa7\x96\xa7\x96\x87\x964t\xb2\x0b\xb3\x133\x13\x93\x03\xa1\xd3\xf2}\xcdi\xcd\x0f`&`\x13-\x12\x88\x89\xa20\xb4\x94\xa6\x96\xa7\x96\xa7\x96\xa7\x96\x87\x96\x02\xe3a<@\xb0H\xb0@\xb1J\x1b-\x12\xc9\x89$\xa8\xa6\x94\xa6\x96\xa7\x96\xa7\x96\x87\x96\x07\x96\x03\xeb\xc9=@T\xb3\x1b\x93\x1b-\x16\x98\x8b$\xa2\xb51\xb4\x94\xa7\x96\xa7\x96\xa7\x96\x87\x96$\xb1\xc9\x1fb\x1b\xb3\x0b\x13\x13\x93\x13\xad\x16\x12\xa1\x12\xa1\xb6&\xb6\x1a\xb6\x96\xa7\x96\xa7\x96\x87\x964t\xb2\x0b\xb3\x133\x13\x93\x03\xa1\xd3\xf2}\xcdi\xcd\x0f`&`\x13-\x12\x88\x89\xa20\xb4\x94\xa6\x96\xa7\x96\xa7\x96\xa7\x96\x87\x96\x02\xe3a<@\xb0H\xb0@\xb1J\x1b-\x12\xc9\x89$\xa8\xa6\x94\xa6\x96\xa7\x96\xa7\x96\x87\x96\x07\x96\x03\xeb\xc9=@T\xb3\x1b\x93\x1b-\x16\x98\x8b$\xa2\xb51\xb4\x94\xa7\x96\xa7\x96\xa7\x96\x87\x96$\xb1\xc9\x1fb\x1b\xb3\x0b\x13\x13\x93\x13\xad\x16\x12\xa1\x12\xa1\xb6&\xb6\x1a\xb6\x96\xa7\x96\xa7\x96\x87\x964t\xb2\x0b\xb3\x133\x13\x93\x03\xa1\xd3\xf2'
b'# Buffer Test Started\r\n'
b'0\r\n'
## Sleeping for 5 seconds …
b'1\r\n'
b'2\r\n'
b'3\r\n'
## Sleeping for 3600 seconds … zzz…
b'4\r\n'
b'5\r\n'
b'6\r\n'

The loop() progress indicator LED kept flashing throughout the hour-long pause. The Raspberry Pi successfully buffered the data and output the first lines without losing any. There wasn't any obvious trend in the size of system buffers (reported periodically using free) during the run, despite the test program buffering thousands of characters per second. As expected, the line count reset to zero if the serial test restarted, losing all previous data.

(As the OP is using a Nano, the clone I'm using isn't based on the official FTDI serial chip. It uses the much cheaper QinHeng Electronics HL-340 interface. If yours is a real one, there's a very slim chance your results will be different.)

The Leonardo behaved quite differently:

  • It did not restart on re-running the test program. Subsequent runs showed the numbers increasing monotonically.

  • The loop() progress indicator LED stopped flashing after a time, indicating that the main Arduino program did not keep running if output was blocked.

  • Despite these differences, the initial data lines were preserved by the Raspberry Pi's serial input buffer. What's not clear from this test is how much of the later data is lost through the Leonardo ‘pausing’.

Conclusion

  • For an Arduino Nano (and Uno/Mega, and very likely older Duemilanove too) the serial data appears to stay in the Raspberry Pi's serial buffer indefinitely until read.

  • Upon reading the serial data, it is no longer available to the user.

  • If the physical connection is disturbed, the serial connection program is restarted, or the Raspberry Pi crashes, the buffered serial data will be lost. One should read and store these data to somewhere less volatile as frequently as practical.

  • While the Arduino Leonardo (and very likely the Micro, which is also based on the same µc) does not reset on serial reconnection, it also seems to lock up unhelpfully if serial reading is not maintained.

  • Other types of Arduino boards (such as the newer ARM-based ones) may exhibit behaviour different even from the two sets of results here.

  • Note that the Pi (or rather, the OS, presuming linux) has a much bigger, 64 kB buffer, and as long as the file handle (or the equivalent in pyserial) is open, data will be read into there by the kernel. When you do a serial read from userspace, you are actually reading from the kernel buffer, not the wire. So if you are running at 115200 baud and the Arduino is continuously transmitting, you could have about 4.5 seconds between reads before loosing data -- probably not a good practice, but the point is you may not quite need a dedicated thread running a poll/read loop either. – goldilocks May 29 '17 at 14:14
  • Would multithreading get around this issue, if, for example, we have two arduinos sending data? – CupinaCoffee May 29 '17 at 20:14
  • Two Arduinos would likely confuse the issue: how would you keep them in sync, for instance? They'd have separated buffers, equally fragile. – scruss May 31 '17 at 15:15
  • @goldilocks under Raspbian, the serial buffer is much larger than 64 kB and seems to grow as resources allow. You're absolutely correct that one does not need a dedicated polling thread, but - given the fragility of a typical Arduino serial session - it's not the best idea leaving this buffer hanging around unread. – scruss Mar 9 at 14:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.