Notecard has communication problems after running for hours

I have the LTE notecard running with a I2C interface to a ESP32. My program posts hourly data to notehub which seems to work fine. Occasionally when some important input change, it posts an immediate data packet to notehub. When this happens, the notecard seems to refuse to connect.

I’m using the latest notecard python library. My configuration is basically:

init ():
    req = {"req": "hub.set"}
    req["product"] = productUID
    req["mode"] = "minimum"
                           
    rsp = card.Transaction(req)
    print (req, rsp)
                  
    req = {"req": "hub.sync"}                   # force an initial synchronization
    rsp = card.Transaction(req)
    
post_data():
    req = {"req": "note.update"}
    req["note"] = "data_readings"
    req["file"] = "sensors.db"
    req["body"] = data.get_data()          # get latest data
    req["sync"] = True
    rsp = card.Transaction(req)
                  
    req = {"req": "hub.sync"}                   # force an send to the hub
    rsp = card.Transaction(req)
    
    # We're waiting for transaction to complete so that we can synchronize our
    # clock with the cell tower clock
    while (1):
        req = {"req": "hub.sync.status"}        # wait for completion
        rsp = card.Transaction(req)
        print (req, rsp)
        if "time" in rsp:
            now = rsp.get("time")
            update_our_clock (now)              # keep our clock synced with cell tower time
            return                              # get outta here
        time.sleep (2)                          # give the card time to work

These are the error messages I get:

{'req': 'hub.sync.status'} {'requested': 19682, 'status': "modem now OFF {modem-off} (network: can't connect (31.2 min remaining) {registration-failure}{network}{extended-network-failure})", 'seconds': 7385, 'sync': True}
{'req': 'hub.sync.status'} {'requested': 19684, 'status': "disconnected (was {connected}) via close of notebox {notehub-disconnected} (network: can't connect (31.2 min remaining) {registration-failure}{network}{extended-network-failure})", 'seconds': 7385, 'sync': True}
{'req': 'hub.sync.status'} {'status': 'starting communications {wait-module} {connecting} {registration-failure} {network}{extended-network-failure}', 'sync': True, 'requested': 19686}
{'req': 'hub.sync.status'} {'status': 'starting communications {wait-module} {connecting} {registration-failure} {network}{extended-network-failure}', 'sync': True, 'requested': 19689}
{'req': 'hub.sync.status'} {'status': 'starting communications {wait-module} {connecting} {registration-failure} {network}{extended-network-failure}', 'sync': True, 'requested': 19691}

The init() function is called once on power up. The post_data() function is called hourly or when some important input changes. This error only happens when the input changes. I’ve had the system running for days without input changing and everything seems to work OK.

I don’t think it is a antenna/communications problem because, when it is in its “locked up” state, I can cycle power and everything reconnects within a minute.

This is the response from a card version request:
{‘req’: ‘card.version’} {‘sku’: ‘NOTE-NBGL-500’, ‘board’: ‘1.11’, ‘name’: ‘Blues Wireless Notecard’, ‘body’: {‘ver_major’: 4, ‘org’: ‘Blues Wireless’, ‘ver_build’: 4015681, ‘built’: ‘Dec 5 2022 12:54:58’, ‘ver_minor’: 1, ‘ver_patch’: 1, ‘product’: ‘Notecard’, ‘version’: ‘notecard-4.1.1’}, ‘device’: ‘dev:xxxxxxxxxxxx’, ‘api’: 4, ‘version’: ‘notecard-4.1.1.4015681’}

Any ideas?

Hi @ktowers,

When you say “This error only happens when the input changes.”, does that mean the post_data() method is what is executed when the input changes? The only slight issue I see is that I don’t believe the sync argument is supported on a note.update request (but that shouldn’t be impacting this).

Can you post the results of a call to card.wireless as well?

Thanks,
Rob

Hi Rob,

post_data() is called hourly or when an input changes. The hourly posts seem work. The posts on change-of-state of the input seem to lock up. I’ve restricted the change-of-state input to make sure they don’t happen too often - eg. less than 5 minutes.

I’m running the card in ‘minimum’ mode in order to save power. I’m wondering if this has something to do with the problem as the card goes into power off modem mode.

Here is the response from the card.wireless call:

{‘req’: ‘card.wireless’} {‘net’: {‘mnc’: 610, ‘rat’: ‘emtc’, ‘imsi’: ‘310170852057685’, ‘cid’: 53897996, ‘mcc’: 302, ‘bars’: 1, ‘lac’: 11025, ‘rssi’: -74, ‘rssir’: -73, ‘modem’: ‘BG95M3LAR02A03_01.006.01.006’, ‘rsrp’: -109, ‘rsrq’: -20, ‘sinr’: -6, ‘iccid’: ‘89011703278520576854’, ‘imei’: ‘867730051247553’, ‘internal’: True, ‘updated’: 1681079093, ‘band’: ‘LTE BAND 12’}, ‘status’: ‘{modem-on}’, ‘mode’: ‘auto’, ‘count’: 1}

Hi @ktowers,

minimum mode shouldn’t be an issue (it’s effectively the same as periodic mode, but without the timed sync cadence). My concern is with the quality of your cellular connection. I’m looking at the following outputs from your card.wireless call and comparing them to the cellular connection quality table we provide:

‘bars’: 1, ‘rssi’: -74, ‘rsrp’: -109, ‘rsrq’: -20, ‘sinr’: -6,

It’s a weak connection (fair at best). The problem is if the Notecard repeatedly can’t make a connection, it will be placed in a “penalty box” to prevent battery drain.

My next steps would be to optimize antenna placement and follow our guide on diagnosing connectivity issues.

Rob

So I’ve tried all of the things on the diagnosing page. Still no luck.

I’ve tried different voltages on VMODEM (3.9 - 5.0V).
I’ve tried different power supplies in VMODEM.
I’ve confirmed good voltage regulation on VMODEM.
The antenna is separated from the PCB by 10 feet.
I’ve turned off the gps (I dont’ have an antenna)
I’ve set the card for LTE-M mode only.

I’ve moved the antenna and now getting better numbers:
{‘req’: ‘card.wireless’} {‘net’: {‘internal’: True, ‘mcc’: 302, ‘rsrp’: -65, ‘band’: ‘LTE BAND 12’, ‘rsrq’: -9, ‘sinr’: 27, ‘mnc’: 720, ‘rat’: ‘emtc’, ‘imsi’: ‘310170852057685’, ‘bars’: 4, ‘lac’: 35155, ‘modem’: ‘BG95M3LAR02A03_01.006.01.006’, ‘iccid’: ‘89011703278520576854’, ‘rssi’: -39, ‘imei’: ‘867730051247553’, ‘cid’: 102513163}, ‘status’: ‘{network-up}’, ‘mode’: ‘m1’, ‘count’: 4}

I’ve turned on the tracing. Here is the tail end of the latest trace. I’m not sure what is going on here. This is immediately after power up. It connected with a tower and did the first hub.sync. Then at time 1:35.15 it shut down. I would have expected it to stop here until the next command from the host. Instead, it immediately starts up the modem again. It then “hangs” at time 1:44.98.

The hub.sync command that my change-of-state input causes executes at time 2:38.95. The modem does not respond because it off somewhere… If I wait long enough (135 seconds) the modem does come back in a penalty box.

Ideas?

S01:24.05 sync: request: get notefile changes: _env.dbs
S01:24.52 16 req + 1 rsp = 17 byte network transaction
S01:24.54 sync: work: download _env.dbs (1 changes) {sync-get-remote-changes}
S01:24.56 sync: request: update change tracker: _env.dbs
S01:25.45 20 req + 1 rsp = 21 byte network transaction
S01:25.55 sync: request: get notefile changes: sensors.db
S01:25.93 18 req + 1 rsp = 19 byte network transaction
S01:25.96 sync: iteration 2 of 50 for this sync session
S01:26.34 checkpointAllNoteboxes {checkpointed} _env.dbs
S01:26.57 sync: request: get notefile changes: _env.dbs
S01:26.99 16 req + 1 rsp = 17 byte network transaction
S01:27.09 sync: request: get notefile changes: sensors.db
S01:27.51 18 req + 1 rsp = 19 byte network transaction
S01:27.53 sync: work: completed {sync-end}
S01:27.55 sync: sync triggered by first sync; explicit sync request
S01:27.57 sync: work: begin (anything pending) {sync-begin}
S01:27.59 sync: request: get notebox change summary
S01:28.02 18 req + 1 rsp = 19 byte network transaction
S01:28.07 sync: work: completed (nothing was pending) {sync-end}
S01:28.09 sync: advancing last sync time from (uninitialized) to 22:28:11Z
S01:28.58 sync: network: request socket shutdown {socket-disconnected}
S01:28.60 lwip: sending RST to remote host packet:11945 ackno:185550474
S01:28.73 lwip: closed
S01:28.75 sync: network: IP terminated (sent:45 rcvd:45) {socket-ip-term}
S01:28.77 sync: network: session closed {socket-closed-session}
S01:28.79 sync: notehub: closed notehub {notehub-disconnected}
S01:28.81 sync: wireless: modem disconnect requested {cell-disconnect-begin}
S01:28.83 sync: wireless: idle {disconnected}
S01:28.85 sync: wireless: idle {disconnected}
S01:28.87 lwip: tcp task deactivated (msgs:0 queues:1 sems:0 mutexes:1)
M01:29.38 S01:29.39 modem: wireless: idle {disconnected}
modem: waiting for shutdown
M01:29.80 _log.qo: modem: powered down
M01:29.87 usage: adding 8934/12117 to sent/rcvd bytes = 1148699/2526422
M01:29.89 usage: adding 2/2 to TCP/TLS = 550/208
M01:29.91 usage: adding 11/0 to sent/rcvd notes = 638/8
M01:29.93 modem: wireless: modem now OFF {modem-off}
M01:29.95 modem: **********************************
M01:29.97 modem: active for 84 sec
M01:29.99 modem: user tx+rx=total: 4396+3583=7979 (35.2% of total)
M01:30.01 modem: tls tx+rx=total: 6580+9413=15993 (overhead of 8014, 35.3% of total)
M01:30.03 modem: ppp tx+rx=total: 9716+12983=22699 (overhead of 6706, 29.5% of total)
M01:30.05 modem: ota tx+rx=total: 8934+12117=21051 (to/from tower, not to/from modem)
M01:30.07 modem: **********************************
M01:35.15 sync: wakeup: modem shutdown
M01:35.27 modem: wireless: idle {disconnected}
S01:35.34 sync: wireless: modem disconnected {cell-disconnected}
S01:35.45 checkpointAllNoteboxes {checkpointed} _notefiles.json
S01:36.59 sync: connect requested by (DATA)
S01:36.61 apn: for a.notefile.net using apn a-notehub.com.attz
S01:36.63 sync: wireless: domain a.notefile.net using APN a-notehub.com.attz
S01:36.91 modem: 4.91V 22.94C at power-on (2023-04-10T22:28:19Z UTC)
S01:36.93 sync: wireless: starting communications {wait-module} {connecting}
T01:38.88 lwip: IP deactivated to conserve energy
S01:41.26 modem: *************** ON ***************
S01:41.28 sync: wireless: modem now ON using external SIM {modem-on}
M01:42.71 modem: 4.91V 22.94C at power-on (2023-04-10T22:28:25Z UTC)
M01:42.73 modem: wireless: starting communications {wait-module} {connecting}
M01:43.95 _log.qo: modem: powered up in data mode
M01:44.98 modem: will recompute access technology: SIM insertion or removal
R02:38.95 sync: wakeup: hub.sync

@ktowers A few comments from a user:

We are running a Notecarrier F with ESP32, running the notecard in “minimum” mode. We take readings hourly using card.attn to put notecard in sleep mode until the next reading. Sleep mode powers down the ESP32. We can also wake up the notecard (and ESP32) with a pushbutton.

  1. I saw similar issues (sometimes data won’t transmit) until I added allow:true to hub.sync. This seems to override most penalty box states.
  2. If you input changes occur too often (or in my case multiple button pushes), I’ve seen issues with data syncing.
  3. The notecard operates independently of the ESP32. In our case, if the notecard doesn’t connect within 3 minutes, we use card.attn to put notecard in sleep mode (and ESP32 powers down), but the LEDs on the notecard can continue to flash, and after a minute or two the notecard may connect and sync, then go to sleep. I assume this relates to #2 above.

Anyway, my observations FWIW.

Karl

Thanks for the feedback Karl. Unfortunately, that didn’t work for me.

I just change hub.set mode to continuous. Everything works as expected. When some input changes a note.update and hub.sync commands are sent to the notecard and I get immediate responses. I haven’t measured the current, but I assume the notecard is not going to sleep in this state.

If I use ‘minimum’ for hub.set mode, the notecard only seems to communicate the first time after power up. After that, any note.updates and hub.sync have no effect. The trace output says the notecard is in data mode, but nothing happens.

Without changing any antenna positions or code, I can cycle power and the card will initially link to the hub and then after that nothing.

Any ideas?

Kevin

Kevin,

I’m sorry that it didn’t work for you.

I’m running the same version of notecard firmware that you are so it shouldn’t be a firmware version difference.

The main differences I can see are:

  • I use card.attn to put the notecard to sleep and power down my ESP32 after every reading.
  • I use note.add not note.update to add a new readings to the note file.
  • In note.update you have a sync:true argument, when I looked at the notecard api on the web I don’t see a sync argument. (note Requests - API Reference - Blues Developers)
  • My code in is C++ not python.

I have no idea if any of these contribute to the issue you are having.

I wish I had better news.

Karl

I changed my code to use note.add. This fixes the problem! Updates are now sent as expected. This also keeps the notecard powered down when there are no updates being sent. Using note.update and having a .db file stored, the modem would immediately turn back on after an initial transmission.

So there seems to be a problem with the combination of note.update and running in minimum mode…

Kevin

1 Like