Some time after UTC 2026-06-09 12:00, all of our devices in one project stopped sending data to Notehub. One device that happens to be on a debugger says:
SyncStatus { status: Some("connect delayed (1261 min remaining): can't open session to notehub: connect: not connected {service}{extended-service-failure}")
This looks sort of like the penalty box messages as mentioned here in the docs or in this forum post but there is no explicit mention of the penalty box as in the docs.
Did all the devices at once have some issue connecting to the Notehub or have some cellular service failure?
It’s worth noting that this happened to the whole fleet which are all in USA, but in different parts of the USA.
Was there some sort of AT&T outage this morning that could have caused this?
Hey @daterdots,
I know other members of the Blues team have been in touch, but I also wanted to provide an update for you and anyone else who stumbles upon this thread.
First, the message you see is indeed a penalty box message. It indicates how much time your device has left in the penalty box—1261 minutes in the log above. When a Notecard has connection issues, it gets placed in a penalty box for half of your outbound sync interval, or 60 minutes, whichever is greater.
Second, we have confirmed the devices were placed in a penalty box after a brief period where they couldn’t reach Notehub, which isn’t really the intent of the penalty box behavior. We’ve updated the Notecard firmware to better handle this scenario, and that improvement will be included in the next Notecard developer firmware release.
Now that 24 hours have passed, we can see that your devices are now out of their penalty boxes and connecting to Notehub as expected.
Sorry again for the inconvenience here, and we really appreciate you taking the time to report this and helping us get to the bottom of it. If you notice anything else odd on your project let us know
Thanks,
TJ