I2C detectable but stopped working - using RasPi 3B, RaspbianOS (and MatLab/Simulink)

Question

Dear community, I'm almost at my wit's end and would be so happy to receive help. It would be a great Christmas present to me if some of you take the time to read and provide inquiry, knowledge or even a solution. I tried to structure it as neatly as possible. Happy about any help!

I'll update the information to my problem if I missed something or upon your request.

My Setup:
I'm using a Simulink model created with MatLab 2017b.
It's being transferred to and built on a Raspberry Pi 3B running Raspbian Buster
via SSH wirelessly over local network.
The second to fourth and otherwise identical setups are running Raspbian Jessie.
Connected to the RasPi are two circuit boards using TWI / I2C protocol. One controls two motors and the voltage (from a battery pack) for the RasPi and the other board. This second one provides data from two light sensors.
(The boards are controlled by an STM32 microcontroller, the firmware for which was written by a colleague.)

My Problem:
The whole setup (using Jessie) was working just fine. Some random day it stopped working.
At first it just hung-up presumably when there was more data transfer than possible with the set bus frequency.
Then it stopped seconds after begin of successful transmission.
Now it doesn't respond whatsoever. The Simulink model still builds but there's no action happening.

Edit: using another fresh OS-build I got it to work for now.

Debug attempts:
Using MatLab's Log signal data and reading from the log tells me: EXT_I2C_write/write: Remote I/O error

Trying to debug using some python code with the smbus package gives the folloving error message:

Traceback (most recent call last):
  File "Downloads/read_sensor.py", line 14, in <module>
    value1 = read_line_value(i2cbus,i2caddress,0x01)
  File "Downloads/read_sensor.py", line 6, in read_line_value
    return bus.read_word_data(dev, addr)
IOError: [Errno 121] Remote I/O error

(As the script doesn't contain a lot more than what's seen in the error message, I didn't include it here. dev, addr, i2cbus and i2caddress are simply storing address values.)

Falling back to linux command line:
i2cdetect is working just fine, showing devices (on 0x11 and 0x19)
i2cset seemed to work (or at least didn't output an error message).
i2cget returns Error: Read failed

Steps I tried:

• reinstalling i2c-tools
• setting up the whole system from zero
  • reinstalling RaspbianOS (updated to Buster) + packages necessary for MatLab = MathWorks' Raspbian version 1
  • configuring network and the like, enabling i2c
• taking a first look at it with an oscilloscope. Not being used to using that I can, as of now, only tell:
  • i2cdetect shows a signal - this behaviour matches with the successful result
  • i2cset and i2cget don't seem to cause a signal, but it appears to cause a short lag / delay in the signal

Edit: Checking up on the firmware implementation...

Further...
I found a couple somewhat similar questions/threads but nothing that helped me. Some of these include:

• problems with the wiring
  • as I have four setups all showing similar problems I don't think that's the reason. Also it doesn't explain why some commands should work while others don't
  • though I don't quite understand / nowhere read how that's supposed to cause the problem
  • Edit: using another OS-build I got it to work for now and checked the cables and sensor-boards one by one. All are working. So there's no general problem with the hardware.
• in another seemingly similar case [2] it's recommended to activate the repeated start condition
  • summary: create file /etc/modprobe.d/i2c.conf and write options i2c_bcm2708 combined=1 to it
  • I tried that, rebooted, does not work in my case, i2cdump still shows XX everywhere, so I removed it again.
  • indicating to use this option is a missing STOP signal. I'll check for that with the oscilloscope soon.
  • I'm not too fond of specifying some option I don't understand* and don't know how it interacts with other configurations, especially when "simply creating file XY" in some directory*, without afore mentioned knowledge... So if someone has detailed information or links for further reading regarding how the different config-files in different directories interact, I'd be happy about that, too. (* see links [2a-c])

Generally I didn't come across a lot of posts explaining exactly why and how the proposed method causes the failure or interacts with other configurations etc.

Sources:
1 https://github.com/mathworks/Raspbian_OS_Setup
[2a] https://stackoverflow.com/questions/44550692/i2c-tools-i2cdetect-and-i2cset-works-perfectly-but-i2cget-returns-read-failed
[2b] https://rabbit-note.com/2015/02/15/raspberry-pi-i2c-repeated-start/
[2c] https://rabbit--note-com.translate.goog/2015/02/15/raspberry-pi-i2c-repeated-start/?_x_tr_sl=ja&_x_tr_tl=de&_x_tr_hl=en&_x_tr_pto=sc

Images:
I hope this picture makes something clearer. On specified request I can take more detailed pictures.

Answer 1

You've asked a good and detailed question. Unfortunately, it seems unlikely that anyone here is going to divine a silver bullet solution that will resolve your problems. That should be no cause for despondence, but rather to take a different tack to resolve your issue(s) - and please believe me when I say that there is a solution.

The different tack is to adopt a logical and systematic troubleshooting effort; i.e. you must consider all potential sources of failure, and go about eliminating them one-by-one through a sequence of tests and measurements.

Reading your question, it sounds like you have devoted most of your effort to software-related causes. Without trying to be rude, I feel obligated to ask, "Why do you suspect software?" Your question states, "Some random day it stopped working" - which sounds like there were no software changes that would have triggered the malfunction. Am I missing something?

Based on the information provided in your question, I would lean very much toward an initial diagnosis of a hardware or wiring issue. I say this even if the hardware had never changed either. Why do I say this? Because a change in external environment (e.g. a new source of EMI) can create interference; because wiring and connections (crimp joints, solder, etc) can be moved in subtle ways that will create issues, and of course hardware can fail.

Raspberry Pi was designed to serve hobbyists and students with an inexpensive platform for learning and experimentation. In my experience, low cost and interconnect integrity are not generally both residing in the same hardware. Ultimately, one pays the price for this through fussy hardware - especially in a case such as yours with what sounds to be a fairly elaborate set of interconnects.

As far as a suggestion on where to start, I wonder if reducing your I2C clock frequency might be worthwhile? If so, I'd also suggest reducing the clock to the lowest frequency supported by your hardware. I say this because in troubleshooting, it can be productive to make large changes for maximum effect as they may expose issues that smaller changes won't reveal.

I think you also should make an effort to overcome your inexperience with basic instrumentation - specifically the oscilloscope. They are invaluable for situations like the one you've described.