Notecard - Random shutdowns

Troubleshooting
1

Greetings!

I am evaluating Host Firmware Upgrade process with a cellular notecard, output of card.version is below:

{
        "version":      "notecard-3.4.1.15128",
        "device":       "dev:864475040539180",
        "name": "Blues Wireless Notecard",
        "sku":  "NOTE-NBGL-500",
        "board":        "1.11",
        "api":  3,
        "body": {
                "org":  "Blues Wireless",
                "product":      "Notecard",
                "version":      "notecard-3.4.1",
                "ver_major":    3,
                "ver_minor":    4,
                "ver_patch":    1,
                "ver_build":    15128,
                "built":        "Aug 11 2022 18:38:27"
        }
}

Before investing more time and building a robust DFU setup on the host side, I have built a shell interface with some Notecard API commands, to get a better feeling how the DFU process works and what will be needed in my firmware.

I have logic analyzer attached to the UART lines. I can see the logical level as well has the analog level of the lines.

I am experiencing a weird behavior where Notecard randomly shuts down and I am not able to communicate with it. When this happens the TX line of the Notecard goes low.

You can see this on the bottom screenshot.

image
image2235×838 125 KB

This happens in different cases:

  • I am communicating with the Notecard (TX line goes down immediately, or after a few seconds)
  • Or I am just waiting and not talking to the Notecard.

I am powering my entire board (and thus also Notecard) from a power supply set to 3.8V and 3A.

As you can see there are some voltage drops on the RX from MCU line (TX line from Notecard). They reach down to 3.3V. When this is happening the power supply shows around 400mA draw (I know shown current draw is averaged and the peaks are probably higher).

We were using thinner wires at some point, and voltage drops went down to 2.2V. We replace the wires with bulkier ones, and the current state is seen on the above screenshot.

I want to emphasize that shutdowns do not follow any repeatable pattern. I used the same setup for the whole day, and for the most of it I could download the firmware image, see the payload string and switch back to continuous mode. I did few shutdons, but since last two hours this now happens regularly, about a minute after I power reset the Notecard.

Only after the power reset of the Notecard, I can communicate again.
The only thing between the power supply and the power pins of the Notecard is a power load switch (SIP32431DR3-T1GE3) and two fuses (0ZCG0110BF2B, one for VIO_P, one for V_MODEM_P).

I hope that you can help with this somehow, if you need anymore information ask.

2

I am continuing to debug the random shutdown behavior.

To isolate the problem and figure out if the problem is in my hardware, firmware or the Notecard itself I have done following setup:

  • I removed the Notecard from my board and plugged it in the Notecarrier-AF.
  • I am powering the Notecarrier-AF through Battery JST connector with power analyzer (Otii, set to 3.75V and 5A).
  • I am talking to the Notecard via AUX uart lines. I made connection from my board test points to the AUXTX and AUXRX header pins. AUXEN pin is connected to the 3V3 pin.
  • My board has its own power supply.

I tried to do the DFU procedure from the start.

While the Notecard was downloading the image I could see very small voltage spikes on the UART lines.

The DFU procedure finished until the end. After I verified that dfu mode was ready (as response to the {"req":"dfu.status"} command), I sent the {"req":"hub.set", "mode":"dfu"} command.

After this Notecard went a completly beserk and started spamming its TX line:

image
image2231×890 131 KB

All spikes look the same, a double pull down to zero.

I could not communicate with the Notecard, after this, only thing I could do is power cycle it.

This issue happened once again, in that case I got to the point where I was getting the payload for some 20ish second, but then the Notecard died in the same way as described above.

One thing that I noticed while try to get DFU procedure to work: after the Notecard fails in some way (UART lines go down, UART line goes crazy, etc.), after the power cylce is done, the response JSON of the {"req":"dfu.status"} is often plain wrong.

example 1: Response JSON is {"status":"successfully downloaded","mode":"ready","on":true}, however there is no body key with the payload information. This status is also shown in the Notehub, giving false impression that everything is okay.

example 2: Response JSON is {"status":"cannot update host: can't connect to stm32") downloaded","mode":"error","on":true}. This message is also shown on the NoteHub.

3

I used the same setup as in previous post, except that now the Notecarrier is powered directly from two Li-Ion cells, connected in parallel, measured at 4.00V.

Same behavior with spikes happens. Here is one of the spike pairs from close:

image
image1415×590 41.5 KB

Pairs are spaced 109,7 ms apart from each other.

4

Hello @MarkoSagadin and welcome to the Blues forum. You’ve come to the right place!

Thank you for showing all the traces and the output of card.version; that’s a great starting point. :+1:

However, before we really start debugging this, I need to know a little bit more about your hardware.

Can you tell me:

  • which MCU you are using
  • every connection between the host MCU and the Notecard (table preferred)

Can you share a picture of your hardware, in case it were to illuminate any problems?

The next question is whether you are trying to perform an Inboard (IAP) or Outboard (ISP) DFU.

Once we have a little more information surrounding the hardware and the path you’re taking, then we can offer better help.

Cheers,
Zak

5

Hello Zak, thank you for the quick response.

Can you tell me:

  • which MCU you are using
  • every connection between the host MCU and the Notecard (table preferred)

I am using Nordic’s nRF52840 SOC, packaged in the NINA-B301 module. We are using Nordic’s nRF Connect SDK (NCS) and Zephyr to develop this.

Connections between the host MCU and Notecard are:

host MCU Notecard (on the Notecarrier)
P0.31 (TX) AUXRX
P0.28 (RX) AUXTX
GND GND

Can you share a picture of your hardware, in case it were to illuminate any problems?

Sadly no, It is proprietary client work.

The next question is whether you are trying to perform an Inboard (IAP) or Outboard (ISP) DFU.

I am trying to do Host Firmware Update procedure, as described here: Host Firmware Update Requests - Blues Developers, so that would be inboard DFU (I think).

I have made some new advancements from yesterday:

  • I have replicated the issue just by using nRF52840DK and the Notecarrier (using same shell interface as one the client board).
  • I have replicated the issue just by using a Uart/USB bridge dongle and Notecarrier.

IMG_20230215_123216
IMG_20230215_1232161440×1080 82.8 KB

Below is an example log of the UART transactions, after which Notecard died. If you know some good tool that can send Uart messages in chunks, (like Cutecom), and knows how to log into file both TX and RX messages, (Cutecom logs only RX), I would be happy, as I had to snip the bellow log together by hand.

Transactions

  1. Turn on the power and check dfu status, I clear it, so Notecard can fetch new firmware payload.
{"req":"dfu.status"}
{"status":"successfully downloaded","mode":"ready","on":true}
{"req":"dfu.status", "stop":true}
{"status":"successfully downloaded","mode":"ready","on":true}
{"req":"dfu.status"}
{"status":"successfully downloaded","mode":"completed","on":true}
  1. Upload new firmware payload on Notehub and set it to update a host. Force a sync on the Notecard.
{"req":"hub.sync"}
{}
{"req":"hub.sync.status"}
{"status":"read byte range: off:0 len:4096 app_update$20230213094834.bin","requested":5}
  1. Notecard starts downloading the image
{"req":"dfu.status"}
{"status":"downloaded 3% (12288/383471)","mode":"downloading","on":true}
{"req":"dfu.status"}
{"status":"downloaded 91% (352256/383471)","mode":"downloading","on":true}
{"req":"dfu.status"}
{"status":"downloaded 100% (383471/383471)","mode":"downloading","on":true}
{"req":"dfu.status"}
{"status":"successfully downloaded","mode":"ready","on":true,"body":{"crc32":1438301845,"created":1676281714,"info":{"fourth":"try"},"length":383471,"md5":"bd4db612d15b6e19456dec2a5b12ebf5","modified":1676281714,"name":"app_update$20230213094834.bin","source":"app_update.bin","type":"firmware"}}
  1. Enter the dfu mode, Notecard dies on this point
{"req":"hub.set", "mode":"dfu}

I should mention that I have also seen Notecard die, while doing nothing, no communication. It tied around 60 seconds after power cycle.

6

I have updated Notecard to the 4.1.1 version to see if this is is resolved there.

Notecard correctly starts downloading the firmware, I can track the progress with dfu.status. After it downloads it I get following status message:

{
        "status":       "cannot update host: outboard DFU using AUX pins requires using a supported card.aux mode such as \"dfu\"",
        "mode": "error",
        "on":   true
}
7

Sorry for the delayed response, this is a curious problem, and I was doing some research…

If possible, we need you to try and capture a complete trace log of the DFU process. It would help us see several things that may be influencing your device.

In the meanwhile, can you run the command {"req":"card.dfu","off":true}, and try again. It will disable “Outboard DFU” (ISP), so that it will get out of the way of the (nboard) DFU you are trying to accomplish.