2

I'm using PySerial to communicate with an Arduino Uno from a Pi. A Python script writes a short message to "ping" the Arduino once a second, and the Arduino writes a short message to respond. This seems to work fine for several hours, and then PySerial starts reporting write timeouts. However, I know the Arduino is still running fine because a routine that blinks an LED once a second is still running.

What's really odd is that some pings do get responded to, but very infrequently. It's almost like the serial connection degrades or becomes out of sync over the course of a few hours.

If I terminate my Python script and re-run, it immediately works again. However, my application needs to eventually control a motor, so that kind of downtime is unacceptable.

How can I diagnose this problem?

Due to the slow speed of the read and writes, I've ruled out the Arduino's serial buffers getting overloaded. I'm also using a short and properly shielded USB cable, so I've ruled out interference.

I'm trying to initialize my Serial instance using different options, to see if it's a configuration issue with PySerial. Here's what I'm currently using:

import serial
serial.Serial(
    port='/dev/ACM0',
    baudrate=115200,
    timeout=1,
    dsrdtr=True,
)

Should this be changed?

Are there any known clock skew issues with serial communication between a Raspberry Pi and an Arduino?

2

Are there any known clock skew issues with serial communication between a Raspberry Pi and an Arduino?

It's not something that is going to show up like this. Where it might show up is occasional misreads; if you try to push the Pi over 1 Mbps you may notice this (although I also think the implementation could catch the potential and throw an error, I don't know if it does).

UART timing is done with hardware clocks that may have some miniscule skew relative to one another. However, that potential, miniscule skew is not something that could have a cumulative effect over the course of hours. The clocks run faster than the actual transfer rate; a rule of thumb you may find with regard to implementing a UART interface is that the clock should be at least eight times faster than the bitrate -- it's probably still pretty feasible at four, but less than that and it won't work. Or not well and not for long.

To explain: If there are (at least) 8 clock ticks per bit, and one bit is a continuous pulse either high or low, it isn't hard for a patiently waiting receiver to sync when it arbitrarily receives the start bit of a transmission. Further, even with 8N1 there's two extra bits per byte which always follow the same pattern for this purpose (last bit high, first bit low). I've done this with an FPGA and so has this person; have a look at the diagram at the top there, which is much like this one from Wikipedia:

enter image description here

Those are the serial data bits, so again, there's a clock running at least 8 times as fast for timing, meaning all the receiver has to do is count. The furthest this count is going to be off is one tick, and that's not enough to misread an entire bit. For a mismatch in the clocks to problematize this, it would have to be whopping -- not like the equivalent of a second difference every few hours, but something closer to 10 seconds per minute. Unless they are damaged beyond any usability, as far as I'm aware crystal oscillator clocks used for hardware timing can't possibly go that far wrong.

So, rule out the clock skew.

If I terminate my Python script and re-run, it immediately works again. However, my application needs to eventually control a motor, so that kind of downtime is unacceptable.

Ideally the logic of the relationship should be such that the Arduino can be easily reset by the Pi in the event of some condition such as this. As in, via a connection between a GPIO on the Pi and the reset pin on the Arduino.

By "ideally the logic..." I mean something that is probably pretty simple, since if the Pi can't be sure of what's going on with the Arduino, then likely the only useful thing that can be done is reset it anyway. And a hard reset is bulletproof.

You don't have to restart the python program on the Pi, so it can maintain state, know what the Arduino is supposed to do next even if the Arduino doesn't, and that division of responsibility should be clear; whatever the Arduino is doing, it should be functioning as something with little or no need for long term accumulation of local data (especially since it doesn't have any potential for such) or long term independent changes of state, meaning having it stop and start again should't matter much (especially at the point where what it's actually doing in relation to the Pi has become ambiguous, perhaps very wrong, etc). Where it restarts from should, by definition, be a known state, and whatever states it's capable of should be ones it can then immediately be put into by the Pi.

Looking at your problem, if it were me, I'd start by blaming myself, i.e., assume I've done something wrong (likely on the Arduino side) that I've missed. The arbitrary reset is a ham fisted and not very satisfying solution, but:

  • If this is a long running system you have to have it anyway. Even if you are 100% positive you will never need to use it, you should have it anyway. Like a seatbelt. If it is too much trouble, then we might as well consider "doing things properly" in general too much trouble, in which case being 100% positive there could never be a need for a reset is overly optimistic. "It work good until it broken", as they say.

  • If you do need to make regular use of the reset, but not often enough that it's seriously impairing functionality, then it may allow you to keep going and working on things until you find your mistake (or even if already know where it is, but have some reason to defer correcting it until some other things are done).

  • This is a good writeup, and I agree that a reset function is necessary. I actually already have one, but my concern is I can't control when the link to the Arduino fails, meaning if an emergency condition arises (e.g. stop motors now to avoid injury!) and the link is unresponsive, I have to wait about 10 seconds for the reset procedure to complete. In that time, a lot of damage can occur. By my estimates, the link seems to fail after about 5 hours. I'll see if a "preventative reset" in the 4 hour mark during a known "safe period" will prevent the need for a reactive reset later. – Cerin Jan 10 '17 at 14:44
  • I'm not sure what you mean by "10 seconds for the reset procedure to complete" but if you pull the RST line on an Arduino low it will stop whatever it is doing instantaneously, and won't start doing anything again until you pull it high. If the motor can continue running independent of any input (e.g., if you took a hammer to the Arduino, it might not stop), then perhaps you want a similar hardware control over the power line, so it only has juice when the Arduino is actually running normally. – goldilocks Jan 10 '17 at 14:51
  • My motor controller is a separate device controlled via I2C through the Arduino, so resetting the Arduino doesn't automatically stop it. I have used the RST feature to trigger a hard reset of the Arduino. By "10 seconds", I mean that's how long it takes to re-establish a functioning serial connection. – Cerin Jan 11 '17 at 11:57
  • If you want to do the control thing for it (I think you probably know all this but I'll say it anyway) I think the reset state for the Arduino pins is floating, but they may actually be pulled down (they could be pulled down with an external resistor too). If you use a transistor with a flyback diode you can control the power to power to the motor board (or it may have a reset pin too, which would make things easier). – goldilocks Jan 11 '17 at 13:54
1

Due to the slow speed of the read and writes, I've ruled out the Arduino's serial buffers getting overloaded

Slow speeds doesn't rule out overflow, it only takes more time before the overflow happens (seem like your case, right?) Restarting your python script flushes the IO buffers on RPi side, and you say it helps, so I'm inclined to believe your problem is exactly that.

You should make sure you empty the buffer every time you read and before you send a reply, both on RPi side and on Arduino side. Either flush the buffers after a read, or keep reading until there's nothing left to read (not just the sequence you expect), and only then send a reply.

You may add relevant parts of your code to the question (both RPi and Arduino) if you need help programming that logic.

  • I completely agree, and I believe I'm doing this. On the Pi side, I have a thread reading from serial input, which only pauses when nothing's read. On the Arduino side, there's a similar loop that continually reads as long as there's data available. – Cerin Jan 10 '17 at 14:58
  • @Cerin You seem to access to self._acks from two different threads with no synchronization in place, is that right? – Dmitry Grigoryev Jan 10 '17 at 15:18
  • 1
    @ Dmitry Grigoryev, You're right, that should use be using a lock. I'll add that and see if it has any effect. – Cerin Jan 11 '17 at 11:38
  • @Cerin I hope so, because I haven't seen any other problems with your code. – Dmitry Grigoryev Jan 11 '17 at 12:16
  • Unfortunately, that did not resolve the problem. – Cerin Jan 12 '17 at 0:05
0

Gradual or intermittent problems are difficult. But here are some tips:

  • Check this with a different cable. In my systems that is usually the first thing to fail in serial communications.

  • Second thing to fail is the connector, which means replacing the device with another running the same program and see if the same thing happens.

Then after you have eliminated hardware issues you can start looking at the software.

  • Not to say you're wrong, but I literally never experienced an intermittent issue with UART cables. Provided the OP did a better job than gluing wires with duct tape and sticks to reasonable cable lengths, I wouldn't expect cables to cause any problems. – Dmitry Grigoryev Jan 10 '17 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.