Modem not responding

Looking for advice on a unit reporting “Modem not responding” after working the previous day with no issues. Unit is in the same location as others being built. Signal strength is low but decent enough. After failure, downgraded the firmware from 7.2 → 6.2.2 OTA with no issues. Seems like a hardware fault as this is only unit out of many that has this problem but would appreciate advice on how to proceed with diagnosing the issue. Have included the notehub error messages as well as device info and trace.

Thu 07:53:33 AM: boot (modem not responding [16425] [5.0] G:Gp:W~ M:Mpw U:W0 R:Rp:W0 S:W0 T:W~ I:W~ W X Y)
Thu 07:42:26 AM: boot (modem not responding [16425] [5.0] G:Gp:W~ M:Mpw U:W0 R:Rp:W0 S:W0 T:W~ I:W~ W X Y)
Thu 07:31:39 AM: boot (modem not responding [16425] [5.0] G:Gp:W~ M:Mpw U:W0 R:Rp:W0 S:W0 T:W~ I:W~ W X Y)
Thu 07:20:52 AM: boot (modem not responding [16425] [5.0] G:Gp:W~ M:Mpw U:W0 R:Rp:W0 S:W0 T:W~ I:W~ W X Y)
Thu 07:10:05 AM: boot (modem not responding [16425] [5.0] G:Gp:W~ M:Mpw U:W0 R:Rp:W0 S:W0 T:W~ I:W~ W X Y)

Blues Wireless Notecard
              ProductUID: ****:****
               DeviceUID: dev:*****
           Serial Number: ltered-S3-12
            Notehub Host: a.notefile.net
        Firmware Version: notecard-6.2.2.16520
                     SKU: NOTE-WBGLW
                   Modem: LE910C1-WWXD / 25.30.406-P0F.405200 / M0F.403006 / P0F.405200 / A0F.403006
                   ICCID: 89011704274021228078
                    IMSI: 310170402122807
                    IMEI: 351077454527410
             Provisioned: 2024-04-24T15:07:02Z (2024-04-24 3:07:02 PM MDT)
       Used Over-the-Air: 1136858 bytes
               Sync Mode: continuous
    Sync Outbound Period: 1 minutes
          Inbound Period: 10 minutes
          Notehub Status: idle {disconnecting} {network}{extended-network-failure}
             Last Synced: 2024-04-25T14:51:16Z (2024-04-25 2:51:16 PM MDT)
                 Voltage: 5.33V
             Temperature: 30.69C
                GPS Mode: periodic
                Location:
            Current Time: 2024-04-25T14:52:55Z (2024-04-25 2:52:55 PM MDT)
               Boot Time: 2024-04-25T14:50:42Z (2024-04-25 2:50:42 PM MDT)
               Notefiles: pistatus.db, data.qo, pipower.db, config.db
   Notefile Storage Used: 2%
                     Env: {
                            "_fwc": "notecard-6.2.2.16520$20240410160742.bin",
                            "_fwc_retry": "1714053544",
                            "_restart_host": "1691104612",
                            "_restart_host_use_attn": "true",
                            "_sn": "ltered-S3-12",
                            "_tri_mins": "1440",
                            "update_rate": "900"
                          }
trace
none,+comm,+mem,+sync,+gps
sync
R09:13.56 sync: wakeup: diag sync

{"req":"card.trace","mode":"on"}
{"stop":true}
M10:14.45 * MODEM TRANSACTION TOO LONG disc ATH (125 sec)
M10:14.48 _log.qo: * modem unexpectedly unresponsive during orderly powerdown
M10:14.52 modem: wireless: modem now OFF {modem-off}
M10:14.52 modem: **
M10:14.52 modem: active for 131 sec
M10:14.53 modem: **
S10:15.01 sync: network: LWIP terminated (sent:0 rcvd:0) {socket-ip-term}
S10:15.53 sync: network: LWIP terminated (sent:0 rcvd:0) {socket-ip-term}
S10:16.05 sync: network: LWIP terminated (sent:0 rcvd:0) {socket-ip-term}
S10:16.56 sync: network: LWIP terminated (sent:0 rcvd:0) {socket-ip-term}
S10:17.08 sync: network: LWIP terminated (sent:0 rcvd:0) {socket-ip-term}
S10:17.60 sync: network: LWIP terminated (sent:0 rcvd:0) {socket-ip-term}
S10:18.11 sync: network: LWIP terminated (sent:0 rcvd:0) {socket-ip-term}
S10:18.63 sync: network: LWIP terminated (sent:0 rcvd:0) {socket-ip-term}
S10:19.14 sync: network: LWIP terminated (sent:0 rcvd:0) {socket-ip-term}
M10:19.57 sync: wakeup: modem shutdown
S10:19.66 sync: network: LWIP terminated (sent:0 rcvd:0) {socket-ip-term}
M10:19.67 modem: wireless: idle {disconnected} {network}{extended-network-failure} {transport}
S10:20.17 sync: network: LWIP terminated (sent:0 rcvd:0) {socket-ip-term}
S10:20.18 sync: connect requested by explicit sync request; GPS (GPS)
S10:20.19 connect: 5.35V 31.44C before comms power-on (2024-04-25T21:00:04Z UTC)
S10:20.20 transport: avoiding WiFi because its SSID is not configured
S10:20.20 transport: using celluar transport because GPS is needed
S10:20.20 apn: for a.notefile.net using apn a-notehub.com.attz
S10:20.21 sync: wireless: domain a.notefile.net using APN a-notehub.com.attz
S10:20.21 sync: wireless: starting communications {wait-module} {connecting} {network}{extended-network-failure} {transport}
S10:20.22 modem: beginning power-on sequence
S10:24.93 modem: *** ON ***
S10:24.94 sync: wireless: modem now ON {modem-on}
M10:39.35 *
*usart2: transmit TIMEOUT ***

M10:53.61 *
*usart2: transmit TIMEOUT ***

M10:54.34 * MODEM TRANSACTION TOO LONG aftr  (29 sec)
M10:54.38 _log.qo: * MODEM HARD RESET aftr  (29 sec)
M10:54.42 _log.qo: * NO MODEM RESPONSE FOR TOO LONG - RESTARTING MODULE *

Hi @szygmunt,

Sorry about the troubles - I just want to clarify though, is the Notecard functioning under one version of firmware (6.2.2), but not another (7.2.2)? If it’s not responding with either firmware version, I would recommend removing the Notecard from the Notecarrier, clean out the M.2 connector, re-insert it, and try it again.

Rob

No, same error regardless of firmware version. Will clean and reseat the card and see if it improves.

No change with clean and reseat card. Swapped the card out for another and the unit works perfectly. Think we have a bad notecard. From the stack trace it will connect and communicate just fine with notehub, then disconnects as expected, but then it is unable to ever connect to the modem again.

S01:24.72 modem: beginning power-on sequence
S01:24.76 modem: waiting for shutdown
M01:24.79 modem: wireless: idle {disconnecting} {transport}
M01:25.12 modem: powering down
M01:25.17 modem: waiting 5 secs for modem soft-powerdown quiescence
R01:27.83 journey: unless we start moving, the journey will end in 217 secs
M01:30.18 usage: adding 13793+13982=27775 to sent+rcvd bytes [79498/1060229]
M01:30.18 usage: adding 3+2=5 to TCP/TLS sessions [10/8]
M01:30.19 usage: adding 13+1=14 to sent+rcvd notes [72/10]
M01:30.19 modem: wireless: modem now OFF {modem-off}
M01:30.19 modem: **********************************
M01:30.20 modem: active for 73 sec
M01:30.20 modem: user tx+rx=total: 8097+4516=12613 (43.3% of total)
M01:30.20 modem:  tls tx+rx=total: 9692+10611=20303 (overhead of 7690, 26.4% of total)
M01:30.21 modem:  ppp tx+rx=total: 14473+14682=29155 (overhead of 8852, 30.4% of total)
M01:30.21 modem:  ota tx+rx=total: 13793+13982=27775 (to/from tower, not to/from modem)
M01:30.22 modem: **********************************
M01:30.28 checkpointAllNoteboxes {checkpointed} pistatus.db
T01:34.52 lwip: IP deactivated to conserve energy
M01:35.38 sync: wakeup: modem shutdown
M01:35.49 modem: wireless: idle {disconnected} {transport}
S01:35.57 sync: wireless: modem disconnected {cell-disconnected}
S01:36.19 sync: network: LWIP terminated (sent:0 rcvd:0) {socket-ip-term}
S01:36.21 sync: connect requested by GPS (GPS)
S01:36.22 connect: 5.32V 27.94C before comms power-on (2024-04-29T17:34:04Z UTC)
S01:36.23 transport: using celluar transport because GPS is needed
S01:36.23 apn: for a.notefile.net using apn a-notehub.com.attz
S01:36.23 sync: wireless: domain a.notefile.net using APN a-notehub.com.attz
S01:36.24 sync: wireless: starting communications {wait-module} {connecting} {transport}
S01:36.24 modem: beginning power-on sequence
S01:40.95 modem: *************** ON ***************
S01:40.96 sync: wireless: modem now ON {modem-on}
S01:42.32 journey: unless we start moving, the journey will end in 202 secs
M01:55.22 ***
***usart2: transmit TIMEOUT ***
***
G01:57.36 journey: unless we start moving, the journey will end in 187 secs
S02:03.47 sync: wakeup: gps poll
M02:09.47 ***
***usart2: transmit TIMEOUT ***
***
M02:10.18 *** MODEM TRANSACTION TOO LONG aftr  (28 sec)
M02:10.19 template set for _log.qo bodyLen:53 payloadLen:1
M02:10.19 _log.qo: *** MODEM HARD RESET aftr  (28 sec)
M02:10.19 *** HARD RESET ***
M02:10.20 modem: wireless: modem now OFF {modem-off}
M02:10.21 modem: **********************************
M02:10.21 modem: active for 33 sec
M02:10.21 modem: **********************************
M02:10.29 checkpointAllNoteboxes {checkpointed} pistatus.db
S02:10.73 sync: network: LWIP terminated (sent:0 rcvd:0) {socket-ip-term}
S02:10.74 sync: next wake-up due in 24h 0m 0s for housekeeping
G02:12.79 journey: unless we start moving, the journey will end in 172 secs
M02:15.41 sync: wakeup: modem shutdown
M02:15.53 modem: wireless: idle {disconnected} {network}{extended-network-failure} {transport}
M02:15.54 modem: wireless: starting communications {wait-module} {connecting} {network}{extended-network-failure} {transport}
M02:15.54 modem: beginning power-on sequence
S02:15.93 sync: network: LWIP terminated (sent:0 rcvd:0) {socket-ip-term}
S02:15.93 sync: next wake-up due in 24h 0m 0s for housekeeping
M02:20.26 modem: *************** ON ***************
M02:20.26 modem: wireless: modem now ON {modem-on}
M02:21.31 modem: wireless: idle {disconnecting} {network}{extended-network-failure} {transport}
G02:28.14 journey: unless we start moving, the journey will end in 156 secs
G02:43.17 journey: unless we start moving, the journey will end in 141 secs
R02:57.52 journey: unless we start moving, the journey will end in 127 secs
G03:13.22 journey: unless we start moving, the journey will end in 111 secs
G03:28.24 journey: unless we start moving, the journey will end in 96 secs
G03:42.33 journey: unless we start moving, the journey will end in 82 secs
G03:57.37 journey: unless we start moving, the journey will end in 67 secs
G04:12.40 journey: unless we start moving, the journey will end in 52 secs
M04:26.33 *** MODEM TRANSACTION TOO LONG disc ATH (125 sec)
M04:26.34 _log.qo: *** modem unexpectedly unresponsive during orderly powerdown
M04:26.34 modem: wireless: modem now OFF {modem-off}
M04:26.35 modem: **********************************