I2c errors: serial-over-i2c protocol

Hi,

I am getting inconsistent i2c error messages. My setup will work for hours (anywhere between 2-15 hours) and the errors start randomly every time, between multiple devices (4, to be exact) with the same exact firmware and hardware setup.

My sensors operate well and produce accurate readings with all notecard stuff removed from the code only (kept in hardware setup) and without initializing the Notecard.

Example of my error messages:

10:06:01.570 -> [INFO] {"req":"card.time","crc":"0002:1110F247"}
10:06:02.661 -> [ERROR] serial-over-i2c: no response to read request {io}
10:06:02.661 -> [WARN] serial-over-i2c: reattempting to read Notecard response
10:06:03.783 -> [ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
10:06:03.783 -> [WARN] serial-over-i2c: reattempting to read Notecard response
10:06:04.875 -> [ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
10:06:04.875 -> [WARN] serial-over-i2c: reattempting to read Notecard response
10:06:05.981 -> [ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
10:06:05.981 -> [WARN] serial-over-i2c: reattempting to read Notecard response
10:06:05.981 -> [ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
10:06:05.981 -> [ERROR] error occured during receive
10:06:06.202 -> [ERROR] i2c: received NACK on transmit of data {io}
10:06:06.202 -> [ERROR] error encountered during I2C transmit hook execution
10:06:07.956 -> [WARN] retrying I/O error detected by host
10:06:08.503 -> [INFO] {"req":"card.time","crc":"0002:1110F247"}
10:06:09.502 -> [ERROR] serial-over-i2c: no response to read request {io}
10:06:09.502 -> [WARN] serial-over-i2c: reattempting to read Notecard response
10:06:10.522 -> [ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
10:06:10.522 -> [WARN] serial-over-i2c: reattempting to read Notecard response
10:06:11.605 -> [ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
10:06:11.605 -> [WARN] serial-over-i2c: reattempting to read Notecard response
10:06:12.693 -> [ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
10:06:12.693 -> [WARN] serial-over-i2c: reattempting to read Notecard response
10:06:12.693 -> [ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
10:06:12.693 -> [ERROR] failed to query Notecard
10:06:14.354 -> [ERROR] serial-over-i2c: no response to read request {io}
10:06:14.354 -> [WARN] serial-over-i2c: reattempting to read Notecard response
10:06:15.431 -> [ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
10:06:15.431 -> [WARN] serial-over-i2c: reattempting to read Notecard response
10:06:16.537 -> [ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
10:06:16.537 -> [WARN] serial-over-i2c: reattempting to read Notecard response
10:06:17.628 -> [ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
10:06:17.628 -> [WARN] serial-over-i2c: reattempting to read Notecard response
10:06:17.628 -> [ERROR] i2c: unknown error on TwoWire::endTransmission() {io}


More info:

  • I have one device using serial communication ( STEVAL-MKI208V1K), and 3 devices using i2c. Two of these i2c devices are ADS1015 from Adafruit and one MLX90614 from DFRobot. I cannot remove the pull-up resistors from these.

  • I am using the Notecard Cell+WiFi and Notecarrier F V1.3, and the Swan mcu. I tried the same setup on the Notecarrier v1.0 and Notecard Cellular and same issue occurs.

  • I am using Arduino IDE v 2.3.0

  • I looked at this forum post: Notecard not responding + I2C error messages - #7 by mat But adding an I2C multiplexer to my existing hardware setup is not ideal, so I’d like to really find the cause of the errors.

Things I’ve tried:

  • I have tried old versions of my code and old versions of Notecard.h (currently using 1.5.4)

  • Slowing the data rate down of ADS1015 devices

  • I tried switching Notecard communications to Serial and if I do that I just get “notecard not responding”

I am not a well-seasoned programmer so please bear with me. Since these errors are inconsistent and started randomly, it is difficult to provide accurately what may have led to this. I am happy to provide any more information needed to help me work through this.

Thank you! :slight_smile:

Hi,
If I understand your setup correctly than you have four devices on I2C bus including notecard, right?
And your setup works well for around 2 hours.

Something like this happened with me with Zephyr LVGL library and my display stops working after 10 mins or so. I realized that it was due to stack overflow. I am still looking for way to overcome this situation as Zephyr tasks can’t be deleted. Or to be honest, I do not know how.

Are you sure stack overflow is not in your case?
What happens if you reset your devices? does it start working again? if so, than stack overflow is likely.

Thanks

1 Like

Yes, 4 total devices on i2c bus.
It works perfectly for several hours, even up to 24 hours. It’s random and inconsistent.

Yes, if I reset my devices it starts working again. I will look into stack overflow.

Thank you! :slight_smile:

tsuzenski,

Just wondering how much is the value of the termination resistors of the I2C bus?

Regards,
Rob Oudendijk

I had some problems with the I2C communication as well when using the Blues Notecard library. I wrote my own library that supports only I2C, no Serial communication and is using static memory allocation instead of dynamic. Blues-Minimal-I2C. That solved my problems.

1 Like

Bernd,

Nice. Can you share the library you made (best by GitHub?)?
So we all can benefit from the nice work.

Warm regards,
ob Oudendijk

It’s installable through the ArduinoIDE library manager. Search for Blues-Minimal

Code is here ==> GitHub - beegee-tokyo/Blues-Minimal-I2C: This library covers the communication between a RAKwireless WisCore module and a Blues.IO Notecard over I2C.

It has some examples, but they are focusing on the RAKwireless WisBlock modules and our carrier board RAK13102

1 Like

Hi @robouden ,

I am still learning about pullup and termination resistors as a result of having these problems. Could you recommend the best way to determine this value?

By observation, my first I2C device (MLX90614-DCC) has a pullup resistance of 4.7kohm on the SDA/SCL lines and my next 2 devices (both ADS1015) have 10kohm pullup resistors.

I know you asked for termination resistance, but I am more unfamiliar with this concept.

Update:

As a workaround, I am going to use Serial communication for the Notecard instead of I2C. The only issue is that I will not be able to use OTA updates.

For now, I am content disabling OTA DFU firmware updates until I can figure out how to utilize Serial2.

On an I2C bus should be only one pullup and not multiple.
10kOhm + 10kOhm + 4.7kOhm in parallel results in a very low resistance pullup.
Remove the 10kOhm resistors and use only the 4.7kOhm as a single pullup.

1 Like

Thank you, unfortunately it is not ideal to remove any of these as they are a part of the purchased boards I am using and I do not want to damage them in any way. Maybe I can investigate new hardware without any pull-up resistors.

1 Like

I would not worry about multiple resistors. They are connected in parallel so your 10K+10K+4.7K will give you about 2.4K. The rule of thumb is anything between 1K and 10K but as a rule of thumb this might or might not work in your situation. There are many factors that you should take into account.

Usually, you have a speed vs. power tradeoff. Weak pullups (i.e. high resistor values) will leak less current. If your pullups are strong, you have faster transitions, if they are too strong, your signal might not reach low, causing problems.

You might want to have a read here: https://www.ti.com/lit/an/slva689/slva689.pdf?ts=1708627352179
BTW: if you look at the figure-1, you will also see multiple pullup resistors.

And from what you posted in the beginning, this really looks more like a memory issue than an I2C-bus issue. If the I2C bus works for hours and then fails, it is hard to believe that suddenly the pullups are a problem. Unless you have thermal problems.

1 Like

Thank you. Yes, it is odd because for some setups/devices, it will work for hours and then start having the errors, but with other devices they happen immediately… I would also think that it is some sort of memory issue.

This does not pertain to the i2c errors specifically, but I feel it may be relevant:

After switching the notecard to use serial communication, I seem to have much greater success, but notefiles with my sensor data have stopped uploading after 12+ hours on rare occasions. Because of this, it is very difficult to determine what the error is. Once I plug my device into my computer via usb to observe the serial monitor/debug output after sensor data notes have stopped uploading, it does not show anything (I’ve waited for them to show up for 10 minutes… perhaps I should wait longer to really see - my sensors are to sample every 2 minutes). I leave the devices plugged in at my desk, so unless the notes stop uploading during work hours (which has not happened yet) I cannot plug it in to see the errors at the time it actually happens.

I do not think it is a connectivity issue since inbound syncs still happen and system/session notefiles still show up in my events.