TL;DR : The PI was not the issue. The sender was. Take a look into my answer.
long story
I am facing a performance issue right now on a test setup i use to investigate limitations for my upcoming project. AFAIK the rpi supports USB 2.0. And with this it should roughly match speeds up to 400 MiBit/s. A friend of mine told me that he already saw data rates of 50 MiBit/s using a USB WiFi stick. So this would be my minimum expectation. However, my current test setup shows that the pi is not getting even close to these speeds.
Question
The main question is: #1 Does anybody have an idea where the low throughput comes from, or have an idea on how to narrow down the problem a bit further? Please focus on this question :)
This problem bugs me for already 3 days now, so there is already a lot of stuff that i have tested, but there are also things that are still unclear to me. I will mark the side questions in bold with a hash and an identifying number.
Setup
I connect an arduino mkrzero over its usb to the raspberry pi's usb. The arduino sends sensor data and the rpi reads it as fast as possible. In a comparison setup i use a mac with osx to read the data from the arduino as a reference system. If the either the mac or the rpi is not fast enough the buffers on the arduino will overflow. Then the retrieved data will contain gaps.
How the systems are setup:
- Arduino: The arduino mkrzero runs a near to real world application similar to that which i am going to use in my project later on. The mkrzero application sends sensor data continuously at a fixed sample rate. Each sample has a fixed set of columns.
- Raspberry PI: The pi runs a raspbian. On the pi i tested small applications to monitor the incoming data. One application just forwards the incoming data to the stdout and another keeps track of the incoming data to mainly print statistical data like the throughput.
- Mac OSX: The mac uses the same programs as the rpi to measure the same set of metrics. The mac is capable to retrieve the data fast enough.
Programs
Both programs running on the either the mac and the pi or the arduino mkrzero are documented and uploaded to github.
Reciever
The programs running on the rpi and the mac are variations of rpi-uart-reciever.
Printer
This is the version of main.c when compiled WITHOUT #define USE_BPS. It prints out all incoming bytes as is to STDOUT using printf.
Output
$ run /dev/cu.usbmodem1411 20
overwrite default port /dev/ttyACM1 with /dev/cu.usbmodem1411
overwrite default #bytes-to-read -1 with 20
try opening serial port /dev/cu.usbmodem1411...
now reading 20 bytes...
0 229 1382 1365 1385
0 230 1382 1365 1385
0 231 1382 1366 1365
0 232 1382 1369 1385
0 233 1382 1366 1365
0 234
closing port /dev/cu.usbmodem1411...
The first column indicates the buffer size on the arduino and should roughly keep at a constant value below 2047 which is the buffer limit on the arduino. Otherwise the data retrieval is not fast enough. In this case it was 0 for the 5 complete data sets being retrieved.
Code
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <unistd.h> //Used for UART
#include <fcntl.h> //Used for UART
#include <termios.h> //Used for UART
#include <time.h>
#include <sys/time.h>
#include <sys/resource.h>
#define BUFFER_SIZE 2048
#define RUN_INFINITELY -1
static volatile int running = 1;
void process_step(int);
void int_signal_handler(int);
void process_step(int fd) {
static unsigned char rx_buffer[BUFFER_SIZE];
if (fd != -1) {
int rx_length = read(fd, (void*)rx_buffer, BUFFER_SIZE);
if (rx_length < 0) {
printf("no bytes to read (error)\n");
} else if (rx_length == 0) {
printf("zero bytes to read (empty)\n");
} else {
rx_buffer[rx_length] = '\0';
printf("%s", rx_buffer);
}
} else {
printf("nothing to read\n");
}
}
int main(int argc, char *argv[]) {
int fd = -1;
long read_bytes = RUN_INFINITELY;
char *port = "/dev/ttyACM1";
unsigned char rx_buffer[BUFFER_SIZE];
setpriority(PRIO_PROCESS, 0, -20);
if (argc > 1) {
printf("overwrite default port %s with %s\n", port, argv[1]);
port = argv[1];
}
if (argc > 2) {
char *end;
long b2r = strtol(argv[2], &end, 10);
if (errno == ERANGE) errno = 0;
else if(b2r < 0) {
printf("overwrite default #bytes-to-read %ld with RUN_INFINITELY\n", read_bytes);
read_bytes = RUN_INFINITELY;
}
else {
printf("overwrite default #bytes-to-read %ld with %ld\n", read_bytes, b2r);
read_bytes = b2r;
}
}
printf("try opening serial port %s...\n", port);
fd = open(port, O_RDWR | O_NOCTTY); // Open in non blocking read/write mode
if (fd == -1) {
printf("Error - Unable to open UART. Ensure it is not in use by another application\n");
}
struct termios options;
tcgetattr(fd, &options);
options.c_cflag = B230400 | CS8 | CLOCAL | CREAD; // Set baud rate
options.c_iflag = IGNPAR;
options.c_oflag = 0;
options.c_lflag = 0;
tcflush(fd, TCIFLUSH);
tcsetattr(fd, TCSANOW, &options);
signal(SIGINT, int_signal_handler);
if(read_bytes == RUN_INFINITELY) {
printf("now reading bytes...\n");
while (running) process_step(fd);
} else {
printf("now reading %ld bytes...\n", read_bytes);
for (long i = 0; i < read_bytes && running; i++) process_step(fd);
}
printf("\nclosing port %s...\n", port);
close(fd);
return 0;
}
void int_signal_handler(int signal) {
running = 0;
}
Stats
This is the version of main.c when compiled WITH #define USE_BPS. It counts all incoming bytes and prints statistics onto STDOUT like this:
Output (arduino @ 4000sps => 83.5kB/s over ~30s Mac)
$ bps /dev/cu.usbmodem1411
overwrite default port /dev/ttyACM1 with /dev/cu.usbmodem1411
try opening serial port /dev/cu.usbmodem1411...
now reading bytes...
stats: rate = 83.5kB/s time = 29s bytes = 2422kB zeros=0 cpurate = 1432.5kB/s cputime = 2s
closing port /dev/cu.usbmodem1411...
Here the output is constantly updating the stats line with the bytes read over the time the program is running to calculate the rate of 83.5kB/s on the mac.
Output (arduino @ 4000sps => 18.9kB/s over ~30s RPI)
$ bps /dev/ttyACM0
overwrite default port /dev/ttyACM1 with /dev/ttyACM0
try opening serial port /dev/ttyACM0...
now reading bytes...
stats: rate = 18.9kB/s time = 29s bytes = 549kB zeros=0 cpurate = 132.6kB/s cputime = 4s
closing port /dev/ttyACM0...
Code (just the deltas)
// ...
void int_signal_handler(int);
struct timeval rt1;
clock_t t1;
void setup_time() {
t1 = clock();
gettimeofday(&rt1, NULL);
}
void process_step(int fd) {
static struct timeval rt2;
static clock_t t2;
static long bytes_read = 0;
static long zeros_read = 0;
static unsigned char rx_buffer[BUFFER_SIZE];
if (fd != -1) {
int rx_length = read(fd, (void*)rx_buffer, BUFFER_SIZE);
if (rx_length < 0) {
printf("ERROR<\r");
} else if (rx_length == 0) {
zeros_read++;
} else {
t2 = clock();
gettimeofday(&rt2, NULL);
float cpu_elapsed = ((float)(t2 - t1) / CLOCKS_PER_SEC );
long elapsed = (long int)rt2.tv_sec - (long int)rt1.tv_sec;
bytes_read += rx_length;
float rate = (float) bytes_read / elapsed;
float cpu_rate = (float) bytes_read / cpu_elapsed;
rx_buffer[rx_length] = '\0';
printf("stats: rate = %2.1fkB/s time = %2lds bytes = %ldkB zeros=%ld cpurate = %2.1fkB/s cputime = %2.0fs \r",
rate / 1000,
elapsed,
bytes_read / 1000,
zeros_read,
cpu_rate / 1000,
cpu_elapsed
);
}
} else {
printf("nothing to read\r");
}
}
int main(int argc, char *argv[]) {
// ...
tcsetattr(fd, TCSANOW, &options);
#ifdef USE_BPS
setup_time();
#endif
signal(SIGINT, int_signal_handler);
// ...
return 0;
}
// ...
Comparison
Data from above is measured over 30s with an arduino serving data @ 4000sps. Below first two lines are the two datasets measured above. Notice that 4000sps is too fast for both the mac and the rpi. With a sample rate of 3000sps the mac is able to process data fast enough, but the rpi not. With a sample rate of 800sps both are fast enough.
sps sys rate time bytes zeros cpurate cputime
4000 Mac 83.5kB/s 29s 2422kB 0 1432.5kB/s 2s
4000 RPI 18.9kB/s 29s 549kB 0 132.6kB/s 4s
3000 Mac 60.7kB/s 29s 1758kB 0 1166.0kB/s 2s
3000 RPI 18.7kB/s 29s 542kB 0 59.8kB/s 9s
800 Mac 16.5kB/s 29s 479kB 0 430.2kB/s 1s
800 RPI 16.6kB/s 29s 480kB 0 73.9kB/s 7s
Now comparing the amount of read bytes with the rates show that both perform linearly to each other.
That is because this equation roughly holds: 2422kB / 549kB = 83,38kB/s / 18,9kB/s.
Thus the rpi performs 2422kB / 549kB = 4,4 times slower than the mac on this benchmark.
Now comparing the rates on a benchmark where the arduino served the data at 800sps shows that both processed the incoming data at the same byte rates ~16.6kB/s.
However, it seems that the rpi seems to do a lot more cycles on the cpu since the cputime is 7 times higher than on the mac. CPU usage via top revealed that there is still head room with about 12-15% overall usage during runtime. #2 I don't expect the CPU usage to influence the throughput according to theese numbers, or am i wrong here?
Sample Size
For the samples being sent to the pi take the Mac data sets as a reference. E.g. with a specified rate of 4000sps the Mac processed these with 83.5kB/s. Regarding the quotient of kB/s divided by Samples/s gives the size Size in kB per Sample: 83,500Bytes/s / 4000Samples/s is roughly ~20Bytes/Sample. This holds for the other reference measures at 3000sps and 800sps on the mac.
Here is the code which spits out a sample on the arduino See Here:
Serial.print(rbuf.size());
for (int i = 0; i < NWORDS; i++) {
Serial.print('\t');
Serial.print(m.words[i], DEC);
}
Serial.write('\n');
The rbuf.size() is printed just for debugging purposes. NWORDS evaluates to 4, so that in total 5 words of data are sent. Each word is 16 Bits long. But note that these routine sends data as DEC text. And we need to take the '\t' into account as well.
Baud Rates
On the arduino the baud rate is set to See Here:
const long serialRate = 2000000;
The baud rate on the rpi program is set as follows See Here:
options.c_cflag = B230400 | CS8 | CLOCAL | CREAD; // Set baud rate
However, changing this parameter did not influence the throughput as far as i could see. I will re evaluate this parameter. The mac termios implementation exposes 230400 as the maximum, but on the rpi higher rates are available. The original piece of code used a matching value of 2000000, thous the same as in the arduino.
Sender
The programs running on the arduino mkrzero are variations of the mkrzero-timer-adc. Since i suppose the bug not to be in the sender i will not include the code into the question here. However, the sample rate on the arduino is compiled into the program via this parameter in mkrzero-timer-adc.ino:
const int sampleRate = 800;
Excluded sources of errors
Now follows a list of possible sources of errors that i already checked and can hopefully be excluded. I have added a small description to illustrate my analysis, to keep track what and how i did it.
Please correct me if something is wrong:
- Data source: I have tested the arduino test application with the mac osx version of the programs and the Arduino IDE's serial monitor. So i am pretty confident, that my data is served correctly.
- General Malfunction: I have turned down the sample rate on the arduino to verify if a general problem exists. At lower speeds the Arduino IDE and the small test applications were capable to retrieve the information fast enough.
- Arduino IDE on the PI: The arduino IDE on the rpi did blow up very fast with the target sample rates, but always worked long enough to tell that it wasn't retrieving information fast enough.
- Text vs Binary: However, since it is sent as text the size per sample varies slightly. Over the time of 60s these variations smooth up. In the real application later on, data is going to be sent as binary to achieve higher sampling throughput, but this is not an issue ATM.
- Power Supply: I have connected the pi to a 2.5 A power supply and connected the mkrzero only to the usb port. The mkrzero drained only 20-40 mA during my tests but i have powered it separately too, just to be on the safe side.
- Different Ports: I have tested all ports of the rpi to see wether this might influence the throughput but without any noticeable effect. Namely:
/dev/ttyACM0and/dev/ttyACM1. - Slow devices may slow down other devices: I had a keyboard connected to the another port but disconnecting it didn't influence the throughput.
- CPU Usage: The program to read the data from the rpi is not using all available CPU, so i am pretty sure that it is not a performance problem on the code. CPU usage is measured with
topcommand on both themacand therpi. - Process priority: I used
setpriorityto raise the process priority and possibly gain a performance benefit over other processes. #3 Not sure if i did this correctly though. - Baudrate: Since the
/dev/ttyACM*serial ports are just virtual ones, the Baudrate should not affect the throughput. In fact they didn't. I tested all available intermiosfor the mac and for therpi. - C/scala/java: I have created a similar application for the rpi in scala using PI4J's wrapper around the wiringPi library but this turns out to be CPU hungry and doesn't perform any better regarding the throughput.
- STDOUT may slow down: I have piped the output to
/dev/nulland changed the statistical program to not even print the incoming data anymore. So this doesn't influence the throughput.
Possible sources of errors
There are still several things i can check. However, i don't really know if these are worth the effort:
#4 Can the speed of the used SD card or the underlying filesystem influence the throughput when everything resides in ram?
#5 Can the usb driver cause these issues and if so are there alternatives do i have to change the distribution for this change?
#6 Can different FTDI chips influence the throughput for instance when comparing the one on the mkrzero board to the one on the sparkfun breakout mini board? To verify test this i could port the mkrzero program to the sparkfun mini breakout.
Motive
I am trying to build a modular high performance data acquisition platform for my upcoming project. Multiple arduinos should send their high frequency analog/digital signal input to the raspberry pi. Data aggregation and a processing should run on the pi as well as the HTTP based frontend to control the platform. In order to reuse existing standards, i would like to plugin several arduino modules into the system using USB if possible. That's the basic idea.
