We have a few hundred Notecard devices happily uploading data hourly via Notehub, which then routes the data to our endpoint, which adds them to a Firebase DB. We’ve found that if cellular communications is marginal, the uploads might be delayed (i.e. the notes are buffered on the Notecard), and recently added code to monitor and report any delays by comparing the time sent vs time received.
This works as expected, highlighting the few sensors with marginal communications. But this morning at 7:55 am Pacific, all 200 sensors received a second routing of Notecard messages that were sent at 11:03 am 4 days ago! Note that routing has been, and is still, working normally. It’s just a batch of notes that were delayed and sent a second time. I am attaching a screen grab for one device showing the duplicate routing of the original event. Time warp? How might this happen? Help appreciated.
Thanks for reporting this.
We are currently in the process of migrating our routing backend to be more scalable. While we were hoping for this transition to be completely transparent, we encountered an bug in the code on Friday which caused us to initiate a rollback. Unfortunately, this means that approximately 600 events (across about 30 projects) got stuck in a queue waiting to be processed which we were unaware of. This morning when we deployed a fix, those lingering events got processed. Since we noticed this issue around the same time you did, so we don’t have a complete accurate count yet.
It seems the extent of the damage is that some events got delayed or double routed between Friday and today. Though it’s possible some events may not have been routed.
It seems like your project has already been fully migrated, so I expect that this issue will not happen again. I apologize for the confusion and inconvenience that this transition might have caused.
If you need me to compile a list of affected events or anything else, let me know and I’d be happy to help
Scott
Scott, thanks for the prompt reply. Your explanation matches our symptoms exactly. I will clean up our database. Sounds like it will not be a recurring event. Thanks again!
1 Like
