I’ve been working with the Swan v1.7 plus Notecard-F over the past few months and am nearing putting my project into the wild but just ran into a problem I’ve not seen before. I’ve begun getting the following error messages:
notecard not responding and
19:59:56.351 → i2c transmit: i2c: unknown error on TwoWire::endTransmission() {io} 19:59:56.351 → {“err”:“i2c: unknown error on TwoWire::endTransmission() {io}”}
Up until today I’ve had no issues sending sensor data and seeing it show up on Notehub.
I went back and walked through the Notecard QuickStart (connecting directly to the Notecard and sending/receiving via a Chrome browser). I get the expected results and all seems normal.
I then walked back through the Sensor tutorial, recreated and flashed the the bare sketch to the Swan. I believe the only thing that version of the sketch does (at that point in the tutorial) is a hub.set – and it’s there I got the “notecard not responding” message (via the serial monitor).
I also tested a version of my code that only does the sensor polling – no Notecard comms – on the same build as I tested above (that returned the Notecard error messages), and the code there seems to run fine and I get the expected sensor readings.
Also ran an I2C scanner sketch and I get back a list of 5 addresses, all of them expected (they are 3 sensor boards + RTC) and the same as seen in earlier testing, to include 0x17, which I believe is the Notecard address.
I have seven devices in the works and have so far only seen this happen on these two, but I’ve not checked the other five yet, as I wanted to see what you all might recommend first. Appreciate any / all suggestions.
Update: Identified a fix action but not the cause.
As mentioned, I have several of these identically built (called BOBs), so I swapped out the Notecards between one exhibiting the issue (BOB-07) and one with no issue (BOB-02).
Notecard from BOB-07 shows no issue when installed in BOB-02; However, the reverse is not true. When the notecard from BOB-02 is installed into BOB-07 – same issue. So the issue doesn’t follow the Notecard. I’m assuming both Notecards are good since they both work fine in BOB-02.
The next thing I was going to do was to swap out the Swans, but I also have an Adalogger board+RTC sandwiched between the Swan and Notecarrier. So I thought perhaps the RTC battery was somehow a contributing factor. Removing the Adalogger cleared the problem on BOB-07 (still with the BOB-02 notecard). There is also an Adalogger board on BOB-02 that hasn’t shown a problem. I replaced the Adalogger board into BOB-07 – problem reappears - so apparently that board has gone bad – and simply removing the board is the fix action.
However, the question still remains: what caused the problem in the first place? One thing that is different about BOB-07 is that I recently installed a battery (3.7v LiPO) and a solar panel (6v). (BOB-02 also has them installed - but I have not yet connected them up to the Notecarrier). The initial test seemed fine and both the battery and solar panel have been disconnected since last weekend (when I first tested them in complete package). Once I realized the difference I began to think that maybe the battery and/or solar panel were wired incorrectly. I tested both before, during and after wiring them up – but I tested them again for good measure and they seem to be wired correctly and putting out the correct voltages and have correct polarities.
Is there any particular order in which the battery or solar panel need or should be connected in? The Notecarrier documentation simply says that if you use a solar panel, you must have a LiPO battery as well (that’s how I read it anyway). But I don’t see any other caveats. I connected the battery first, then the solar panel. Any other safety recommendations or best practices?
Thanks in advance for any advice / next steps!!
…Norm…
Update #2 – Continued down the path of checking the hardware via parts swapping and moved the Adalogger board from BOB-07 into BOB-02. No error messages seen!
Then installed the BOB-02 Adalogger board into BOB-07 – and I get the error messages.
This points at the Notecarrier. The presence of the Adalogger board does trigger the error messages, but the board itself appears to be fine, and both boards worked without generating error messages while in BOB-02.
Went on to test BOB-05 where I did see the I2C error messages, but unlike BOB-07, removing the Adalogger board does not alleviate the cause of the problem. But it is random. About every 7 Swan board resets will result in a successful hub.set. Tested the battery and solar panel voltages and measured 3.6v and 6.2v respectively, and the polarities are correct.
Haven’t swapped the MCUs but will try that next after I continue testing the rest of the builds.
Hi @noforan, sorry to hear you’re experiencing this and thanks for reaching out.
It sounds to me like it could be related to the I2C pullup resistors, which is the “typical” reason I2C communications fail. Did the problem start happening after a change in the hardware setup?
Please check which I2C devices include their own I2C pullup resistors. If there is more than one device with I2C pullups, try disabling them on all devices bar one.
This thread describes this in more detail and some possible solutions:
With respect to the I2C bus, I’m tracking 5 devices using the bus:
0x17 - the Notecard
0x61 - Atlas Scientific EZO DO circuit
0x63 - Atlas EZO pH circuit
0x64 - Atlas EZO EC circuit
0x68 - Adafruit Adalogger (SD card reader + RTC – only the RTC uses I2C
The Atlas breakout boards don’t have integrated pull-ups – you’re expected to provide your own per their documentation. In my correspondence with them they recommended adding 4.7k resistors. However, the Adalogger already has 10k pull-ups, so I never added any additional resistance thinking I’d wait to see if the resistors provided by the Adalogger were sufficient for the bus. When I removed the Adalogger from the one build, and thus the only pull-up resistors, the I2C error disappeared. Would you expect that result since the bus would have no pull-up resistors at that point?
I’ve been testing this build for about 4 months and this is the first I’ve seen these I2C messages, which is what makes me think I’ve done something to compromise the Notecarrier.
In one of the updates I posted, I described that there was a recent hardware change: I just added a LiPO battery and solar panel. I’m aware of the warnings about connecting the LiPOs and how the connectors are often swapped, but I made up my own connectors and tested them frequently as I was building them. So, now I’m wondering if the solar panel is mis-wired. I’ve successfully used this battery / solar panel on an earlier build using a MKR1310 and the Adafruit solar charger, so I’m not unfamiliar with the wiring, but maybe I’m in denial! In any case I’m rechecking all that.
I did read the post you attached and the one thing that did stand out was the suggestion to use TX/RX for the Notecard instead of I2C. I will look for the documentation on how that’s done, but if you can point me to it and/or a tutorial that would be great.
Hi there Norm! Thanks for the detailed response. I’d be surprised if the Notecarrier has been damaged in any way, but if you have a second one to try then swapping them out would help confirm or eliminate that as a root cause.
Even though I2C is a digital protocol, it ultimately comes down to the behavior of the signals, which can become decidedly “analog” when there is too little resistance, or too much capacitance on the bus. If you have access to an analog scope, that would give you a more detailed view of what is happening on the I2C bus regarding signal timing and levels.
One other thing to add is that the Notecard itself includes 10k pull up resistors.
The datasheet for the EZO DO shows it also includes 4.7k pullups, and similarly for the EZO PH and the EZO EC boards. This is in addition to that the 10k pullups on the Adafruit logger.
This calculator shows the effective resistance of the 5 devices connected in parallel on the bus. When you plug in the numbers, you’ll see an effective resistance of ~1.2k Ohms (1.19 to be exact). This is much too strong a pullup (that is, the resistance is too low) which will mean the individual I2C devices will not be able to pull the signal low.
If you were to use the TX/RX serial interface with the Notecard, that would give an effective resistance of 1.35k Ohms (10k, 4.7k, 4.7k, 4.7k), which is still too low of a resistance, creating a strong pull up.
I suggest you remove the pullups from the EZO boards, leaving just the two 10k pullups from the Notecard and the Logger. That gives an effective resistance of 5k, which is much more in the ballpark of what’s needed.
I hope that works out for you - please let me know how it goes!
Thanks,
mat.
I appreciate the detailed response and especially that you have looked at the EZO data sheets. Running down the responses sequentially…
I’m glad to hear that you don’t think the hardware has been damaged.
I don’t have a scope but your other responses have given me something to pursue.
I read through the Notecard interfaces link and see how to implement a serial alternative.
I didn’t realize the Notecard also had 10k pull-ups – that’s good to know.
Thanks for the link to the resistor calculator.
My workaround was going to be to try using TX/RX rather than I2C, but you pointing out that it won’t be a valid workaround has saved me some effort!
I mention in one of my replies that I had specifically asked Atlas Scientific about the need for pull-ups because in that same data sheet you looked at, there’s a page showing the user potentially adding their own 4.7k resistors. So, I’m now puzzled by their response and that’s also why I had assumed the EZO boards couldn’t be the problem. I’ve hunted down my earlier correspondence with them and followed up with some additional questions spurred by your response.
I will follow-up on this with a solution, if I do identify one, so that it’s documented for the next person.
Update #3 (and solution?):@mat – following your suggestions, I started removing the EZO circuits (breakout boards) one by one. With the removal of the first EZO circuit the error messages remained intermittent on one build and disappeared on the second build. With the removal of the second board the error messages disappeared on both builds. This supported your analysis.
Unfortunately, the EZO circuit boards are coated in resin (potted), so there’s no way to remove the pull-up resistors, that I can see, without damaging the board. But I have been working on another related project that uses an I2C mux, a TCA9548A. I am using it with some light sensors that all have the same I2C address, but I was wondering if I could use it with the EZO boards even though they have unique addresses, because I thought it might isolate their pull-ups from the rest of the bus.
I found a post on a TI website that talks about this (https://e2e.ti.com/support/interface-group/interface/f/interface-forum/746917/tca9548a-pull-up) and says “The main SDA/SCL lines are seperated from the other channels with a pass FET so the other sides do not see the main SDA/SCL pull up resistors. You will need to have a pair of pull up resistors on each channel even if you are turning them on one at a time.” Wiring it this way would seem to isolate the built-in resistors on the EZO boards from the rest of the bus, effectively removing them.
I rewired the one build to use the TCA9548A, rewrote my sampling code, and I couldn’t produce any of the I2C errors I was seeing before! I wired up a second build the same way and got the same result. I even reinstalled the Adaloggers into both builds and I’m still not seeing errors. Most of the TCA9548A boards I see on AliE or Amazon also have their own pull-up resistors, but the ones I have are from Adafruit. Their breakout board has traces you can cut to remove the pull-ups. I’ve left the traces intact so far, but I could remove them if needed. I thought I would wait to see some further results.
I hesitate to call this a solution just yet, and it feels a bit kluge, but it does seem to be working.
Again, thanks for your suggestions as they helped me get past the initial denial about those being the potential source of the problem. I’ve got some follow-on correspondence with Atlas and will recommend they update their documentation to be less confusing about the potential need for additional pull-up resistors. (Lesson learned: Trust but verify – and re-verify).
I’m very happy to hear you have things working. Using the I2C MUX is also a viable solution, so you’re essentially removing the Notecard’s 10k pullups from the set of pullup resistors on the bus, as you noted.
Even though the EZO boards and the logger still have their 4.7k pullups present, and the combined pullup is fairly strong, it is not so strong that the devices cannot pull the line low, or that the resistance of the pullups combined with the inherent capacitance doesn’t skew the signal too much.
Here’s a great article explaining how I2C hardware works and how adding more devices to the bus can limit the maximum clock rate.
A logic analyzer/scope is useful here, so you can see the voltages on the SCL/SDA lines and how clean the signal is.
It’s great to hear you have something working for you, which is the result we wanted.
Nice article on I2C. One of these days I’ll pony up for the scope – maybe sooner than later given this issue. I’ve got this working on 6 devices now and sort of working on a 7th – which on the 7th one I think must be a wiring problem, but declaring victory and marking it as a solution.
