0

TL;DR : The PI was not the issue. The sender was. Take a look into my answer.

long story

I am facing a performance issue right now on a test setup i use to investigate limitations for my upcoming project. AFAIK the rpi supports USB 2.0. And with this it should roughly match speeds up to 400 MiBit/s. A friend of mine told me that he already saw data rates of 50 MiBit/s using a USB WiFi stick. So this would be my minimum expectation. However, my current test setup shows that the pi is not getting even close to these speeds.

Question

The main question is: #1 Does anybody have an idea where the low throughput comes from, or have an idea on how to narrow down the problem a bit further? Please focus on this question :)

This problem bugs me for already 3 days now, so there is already a lot of stuff that i have tested, but there are also things that are still unclear to me. I will mark the side questions in bold with a hash and an identifying number.

Setup

I connect an arduino mkrzero over its usb to the raspberry pi's usb. The arduino sends sensor data and the rpi reads it as fast as possible. In a comparison setup i use a mac with osx to read the data from the arduino as a reference system. If the either the mac or the rpi is not fast enough the buffers on the arduino will overflow. Then the retrieved data will contain gaps.

How the systems are setup:

  • Arduino: The arduino mkrzero runs a near to real world application similar to that which i am going to use in my project later on. The mkrzero application sends sensor data continuously at a fixed sample rate. Each sample has a fixed set of columns.
  • Raspberry PI: The pi runs a raspbian. On the pi i tested small applications to monitor the incoming data. One application just forwards the incoming data to the stdout and another keeps track of the incoming data to mainly print statistical data like the throughput.
  • Mac OSX: The mac uses the same programs as the rpi to measure the same set of metrics. The mac is capable to retrieve the data fast enough.

Programs

Both programs running on the either the mac and the pi or the arduino mkrzero are documented and uploaded to github.

Reciever

The programs running on the rpi and the mac are variations of rpi-uart-reciever.

Printer

This is the version of main.c when compiled WITHOUT #define USE_BPS. It prints out all incoming bytes as is to STDOUT using printf.

Output

$ run /dev/cu.usbmodem1411 20
overwrite default port /dev/ttyACM1 with /dev/cu.usbmodem1411
overwrite default #bytes-to-read -1 with 20
try opening serial port /dev/cu.usbmodem1411...
now reading 20 bytes...
0 229 1382  1365  1385
0 230 1382  1365  1385
0 231 1382  1366  1365
0 232 1382  1369  1385
0 233 1382  1366  1365
0 234
closing port /dev/cu.usbmodem1411...

The first column indicates the buffer size on the arduino and should roughly keep at a constant value below 2047 which is the buffer limit on the arduino. Otherwise the data retrieval is not fast enough. In this case it was 0 for the 5 complete data sets being retrieved.

Code

#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <unistd.h>     //Used for UART
#include <fcntl.h>      //Used for UART
#include <termios.h>    //Used for UART
#include <time.h>

#include <sys/time.h>
#include <sys/resource.h>

#define BUFFER_SIZE 2048
#define RUN_INFINITELY -1

static volatile int running = 1;
void process_step(int);
void int_signal_handler(int);

void process_step(int fd) {
  static unsigned char rx_buffer[BUFFER_SIZE];
  if (fd != -1) {
    int rx_length = read(fd, (void*)rx_buffer, BUFFER_SIZE);
    if (rx_length < 0) {
      printf("no bytes to read (error)\n");
    } else if (rx_length == 0) {
      printf("zero bytes to read (empty)\n");
    } else {
      rx_buffer[rx_length] = '\0';
      printf("%s", rx_buffer);
    }
  } else {
    printf("nothing to read\n");
  }
}

int main(int argc, char *argv[]) {
  int fd = -1;
  long read_bytes = RUN_INFINITELY;
  char *port = "/dev/ttyACM1";
  unsigned char rx_buffer[BUFFER_SIZE];
  setpriority(PRIO_PROCESS, 0, -20);

  if (argc > 1) {
    printf("overwrite default port %s with %s\n", port, argv[1]);
    port = argv[1];
  }

  if (argc > 2) {
    char *end;
    long b2r = strtol(argv[2], &end, 10);
    if (errno == ERANGE) errno = 0;
    else if(b2r < 0) {
      printf("overwrite default #bytes-to-read %ld with RUN_INFINITELY\n", read_bytes);
      read_bytes = RUN_INFINITELY;
    }
    else {
      printf("overwrite default #bytes-to-read %ld with %ld\n", read_bytes, b2r);
      read_bytes = b2r;
    }
  }

  printf("try opening serial port %s...\n", port);
  fd = open(port, O_RDWR | O_NOCTTY);    // Open in non blocking read/write mode
  if (fd == -1) {
    printf("Error - Unable to open UART.  Ensure it is not in use by another application\n");
  }

  struct termios options;
  tcgetattr(fd, &options);
  options.c_cflag = B230400 | CS8 | CLOCAL | CREAD;   // Set baud rate
  options.c_iflag = IGNPAR;
  options.c_oflag = 0;
  options.c_lflag = 0;
  tcflush(fd, TCIFLUSH);
  tcsetattr(fd, TCSANOW, &options);

  signal(SIGINT, int_signal_handler);
  if(read_bytes == RUN_INFINITELY) {
    printf("now reading bytes...\n");
    while (running) process_step(fd);
  } else {
    printf("now reading %ld bytes...\n", read_bytes);
    for (long i = 0; i < read_bytes && running; i++) process_step(fd);
  }
  printf("\nclosing port %s...\n", port);
  close(fd);

  return 0;
}

void int_signal_handler(int signal) {
  running = 0;
}

Stats

This is the version of main.c when compiled WITH #define USE_BPS. It counts all incoming bytes and prints statistics onto STDOUT like this:

Output (arduino @ 4000sps => 83.5kB/s over ~30s Mac)

$ bps /dev/cu.usbmodem1411
overwrite default port /dev/ttyACM1 with /dev/cu.usbmodem1411
try opening serial port /dev/cu.usbmodem1411...
now reading bytes...
stats:    rate = 83.5kB/s    time = 29s    bytes = 2422kB    zeros=0    cpurate = 1432.5kB/s    cputime =  2s
closing port /dev/cu.usbmodem1411...

Here the output is constantly updating the stats line with the bytes read over the time the program is running to calculate the rate of 83.5kB/s on the mac.

Output (arduino @ 4000sps => 18.9kB/s over ~30s RPI)

$ bps /dev/ttyACM0
overwrite default port /dev/ttyACM1 with /dev/ttyACM0
try opening serial port /dev/ttyACM0...
now reading bytes...
stats:    rate = 18.9kB/s    time = 29s    bytes =  549kB    zeros=0    cpurate =  132.6kB/s    cputime =  4s
closing port /dev/ttyACM0...

Code (just the deltas)

// ...
void int_signal_handler(int);

struct timeval rt1;
clock_t t1;
void setup_time() {
  t1 = clock();
  gettimeofday(&rt1, NULL);
}

void process_step(int fd) {
  static struct timeval rt2;
  static clock_t t2;
  static long bytes_read = 0;
  static long zeros_read = 0;
  static unsigned char rx_buffer[BUFFER_SIZE];
  if (fd != -1) {
    int rx_length = read(fd, (void*)rx_buffer, BUFFER_SIZE);
    if (rx_length < 0) {
      printf("ERROR<\r");
    } else if (rx_length == 0) {
      zeros_read++;
    } else {
      t2 = clock();
      gettimeofday(&rt2, NULL);
      float cpu_elapsed = ((float)(t2 - t1) / CLOCKS_PER_SEC );
      long elapsed = (long int)rt2.tv_sec - (long int)rt1.tv_sec;
      bytes_read += rx_length;
      float rate = (float) bytes_read / elapsed;
      float cpu_rate = (float) bytes_read / cpu_elapsed;
      rx_buffer[rx_length] = '\0';
      printf("stats:    rate = %2.1fkB/s    time = %2lds    bytes = %ldkB    zeros=%ld    cpurate = %2.1fkB/s    cputime = %2.0fs    \r",
        rate / 1000,
        elapsed,
        bytes_read / 1000,
        zeros_read,
        cpu_rate / 1000,
        cpu_elapsed
      );
    }
  } else {
    printf("nothing to read\r");
  }
}

int main(int argc, char *argv[]) {
  // ...
  tcsetattr(fd, TCSANOW, &options);

  #ifdef USE_BPS
  setup_time();
  #endif

  signal(SIGINT, int_signal_handler);
  // ...
  return 0;
}
// ...

Comparison

Data from above is measured over 30s with an arduino serving data @ 4000sps. Below first two lines are the two datasets measured above. Notice that 4000sps is too fast for both the mac and the rpi. With a sample rate of 3000sps the mac is able to process data fast enough, but the rpi not. With a sample rate of 800sps both are fast enough.

 sps  sys  rate    time  bytes  zeros   cpurate  cputime
4000  Mac 83.5kB/s  29s  2422kB     0 1432.5kB/s      2s
4000  RPI 18.9kB/s  29s   549kB     0  132.6kB/s      4s
3000  Mac 60.7kB/s  29s  1758kB     0 1166.0kB/s      2s
3000  RPI 18.7kB/s  29s   542kB     0   59.8kB/s      9s
 800  Mac 16.5kB/s  29s   479kB     0  430.2kB/s      1s
 800  RPI 16.6kB/s  29s   480kB     0   73.9kB/s      7s

Now comparing the amount of read bytes with the rates show that both perform linearly to each other. That is because this equation roughly holds: 2422kB / 549kB = 83,38kB/s / 18,9kB/s. Thus the rpi performs 2422kB / 549kB = 4,4 times slower than the mac on this benchmark. Now comparing the rates on a benchmark where the arduino served the data at 800sps shows that both processed the incoming data at the same byte rates ~16.6kB/s.

However, it seems that the rpi seems to do a lot more cycles on the cpu since the cputime is 7 times higher than on the mac. CPU usage via top revealed that there is still head room with about 12-15% overall usage during runtime. #2 I don't expect the CPU usage to influence the throughput according to theese numbers, or am i wrong here?

Sample Size

For the samples being sent to the pi take the Mac data sets as a reference. E.g. with a specified rate of 4000sps the Mac processed these with 83.5kB/s. Regarding the quotient of kB/s divided by Samples/s gives the size Size in kB per Sample: 83,500Bytes/s / 4000Samples/s is roughly ~20Bytes/Sample. This holds for the other reference measures at 3000sps and 800sps on the mac.

Here is the code which spits out a sample on the arduino See Here:

Serial.print(rbuf.size());
for (int i = 0; i < NWORDS; i++) {
  Serial.print('\t');
  Serial.print(m.words[i], DEC);
}
Serial.write('\n');

The rbuf.size() is printed just for debugging purposes. NWORDS evaluates to 4, so that in total 5 words of data are sent. Each word is 16 Bits long. But note that these routine sends data as DEC text. And we need to take the '\t' into account as well.

Baud Rates

On the arduino the baud rate is set to See Here:

const long serialRate = 2000000;

The baud rate on the rpi program is set as follows See Here:

options.c_cflag = B230400 | CS8 | CLOCAL | CREAD;   // Set baud rate

However, changing this parameter did not influence the throughput as far as i could see. I will re evaluate this parameter. The mac termios implementation exposes 230400 as the maximum, but on the rpi higher rates are available. The original piece of code used a matching value of 2000000, thous the same as in the arduino.

Sender

The programs running on the arduino mkrzero are variations of the mkrzero-timer-adc. Since i suppose the bug not to be in the sender i will not include the code into the question here. However, the sample rate on the arduino is compiled into the program via this parameter in mkrzero-timer-adc.ino:

const int sampleRate = 800;

Excluded sources of errors

Now follows a list of possible sources of errors that i already checked and can hopefully be excluded. I have added a small description to illustrate my analysis, to keep track what and how i did it.

Please correct me if something is wrong:

  • Data source: I have tested the arduino test application with the mac osx version of the programs and the Arduino IDE's serial monitor. So i am pretty confident, that my data is served correctly.
  • General Malfunction: I have turned down the sample rate on the arduino to verify if a general problem exists. At lower speeds the Arduino IDE and the small test applications were capable to retrieve the information fast enough.
  • Arduino IDE on the PI: The arduino IDE on the rpi did blow up very fast with the target sample rates, but always worked long enough to tell that it wasn't retrieving information fast enough.
  • Text vs Binary: However, since it is sent as text the size per sample varies slightly. Over the time of 60s these variations smooth up. In the real application later on, data is going to be sent as binary to achieve higher sampling throughput, but this is not an issue ATM.
  • Power Supply: I have connected the pi to a 2.5 A power supply and connected the mkrzero only to the usb port. The mkrzero drained only 20-40 mA during my tests but i have powered it separately too, just to be on the safe side.
  • Different Ports: I have tested all ports of the rpi to see wether this might influence the throughput but without any noticeable effect. Namely: /dev/ttyACM0 and /dev/ttyACM1.
  • Slow devices may slow down other devices: I had a keyboard connected to the another port but disconnecting it didn't influence the throughput.
  • CPU Usage: The program to read the data from the rpi is not using all available CPU, so i am pretty sure that it is not a performance problem on the code. CPU usage is measured with top command on both the mac and the rpi.
  • Process priority: I used setpriority to raise the process priority and possibly gain a performance benefit over other processes. #3 Not sure if i did this correctly though.
  • Baudrate: Since the /dev/ttyACM* serial ports are just virtual ones, the Baudrate should not affect the throughput. In fact they didn't. I tested all available in termios for the mac and for the rpi.
  • C/scala/java: I have created a similar application for the rpi in scala using PI4J's wrapper around the wiringPi library but this turns out to be CPU hungry and doesn't perform any better regarding the throughput.
  • STDOUT may slow down: I have piped the output to /dev/null and changed the statistical program to not even print the incoming data anymore. So this doesn't influence the throughput.

Possible sources of errors

There are still several things i can check. However, i don't really know if these are worth the effort:

  • #4 Can the speed of the used SD card or the underlying filesystem influence the throughput when everything resides in ram?

  • #5 Can the usb driver cause these issues and if so are there alternatives do i have to change the distribution for this change?

  • #6 Can different FTDI chips influence the throughput for instance when comparing the one on the mkrzero board to the one on the sparkfun breakout mini board? To verify test this i could port the mkrzero program to the sparkfun mini breakout.

Motive

I am trying to build a modular high performance data acquisition platform for my upcoming project. Multiple arduinos should send their high frequency analog/digital signal input to the raspberry pi. Data aggregation and a processing should run on the pi as well as the HTTP based frontend to control the platform. In order to reuse existing standards, i would like to plugin several arduino modules into the system using USB if possible. That's the basic idea.

  • 1
    Could you add a summary line saying how many samples per second you send to the Pi, how big in bytes is each sample, and what baud rate you are using? – joan Dec 30 '17 at 16:16
  • 1
    This is really many questions lumped together. Try to cut it down? – Brick Dec 30 '17 at 18:15
  • Thanks so far :) Please focus on question #1. I have added the facts i could gather according the baud rates and sample size. SPS stands for Samples per Second. In the Comparison there it is the first column. It describes how the arduino is setup. It doesn't automatically mean that this SPS arrive at the target device. However the data seems to match regarding the measurements. I have added 2 sections "Baud Rates" and "Sample Size" to clarify this. – isaias-b Dec 30 '17 at 19:52
0

I later figured out that it was not the PI causing the performance drawbacks. The problem was caused by the senders compiled code. My write calls were compiled into blocking code for the µc. This blocking stuff is going on on both setups, the mac and the pi. But the effect just appears in magnitudes differently on both systems. Probably due to more advanced hardware on the mac. However, this was very misleading because i doubted this could be caused by the sender, which actually was wrong.

There is at least one open Ticket to this on the arduino community out there. Especially on the SAMD21 Sparkfun Github Issue Tracker. https://github.com/sparkfun/Arduino_Boards/issues/39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.