Duplicate events

1

I have been receiving the same messages again and again via the Notehub. Each event is received at a different time by the Notehub (part of a different syncing session from the Notecard). But the events contain the exact same data (other than the event_id and the time when the notehub received the note).

For example, I have received this exact body created at this when 28 times in the Notehub (and paying credits to forward it every time to my application!).

"body": {"samples": [{"ledi": 7.30412, "sipm": 0.3162501, "vbias": 35.45036, "lediset": 10}, {"ledi": 47.0663, "sipm": 2.139923, "vbias": 38.09255, "lediset": 50}, {"ledi": 96.41521, "sipm": 1.2924395, "vbias": 35.347477, "lediset": 100}, {"ledi": 186.73506, "sipm": 1.0897858, "vbias": 33.698624, "lediset": 200}], "version": 1}, "file": "samples.qo", "when": 1721937841

what could be causing this?

2

Hi Danny! Thanks for reaching out. I have a hypothesis, but would need to confirm. Could you DM me the device UID? (dev:xxxxxxxxxxxxxx)?

3

Hi, I wanted to report I’m seeing the same thing. An additional data-point is that by default I’m setting all the outbound notes as “sync”:true.

I can provide Device UID to get further assistance.

4

We believe this issue of duplicate events was occurring under the following conditions.

  1. The Notefile has an applied template via note.template
  2. A note.add request was made to the Notecard for the templated Notefile while the Note was syncing to the Notehub.

This was most likely to occur if note.add statements are occurring frequently, and while the Notecard is in continuous connection mode with Notehub.

We submitted a fix to our most recent Developer version 7.4.1 (you may see a later version depending on when you find this message)

For more information about updating Notecard firmware, please see:

1 Like
5

I am facing the same issue and using 7.4.2 version.

6

There are cases where duplicate Events on Notehub are going to occur by the design of the Notecard/Notehub interaction model. This interaction model is designed to be robust to data loss by default.

If a Notefile sync is interrupted by a note.add it can still result in duplicate events on Notehub.

Common Causes

We most often see this when either:

  1. The device tries to “stream” data up to the cloud, that is push data at short intervals.

  2. The device batches note.add statements, and it includes the "sync":true option.

Resolutions

“Streaming” Data to Notehub

Chances are, if the application is designed to upload data at a relatively high frequency, then the Notecard is connected to Notehub with {"req":"hub.set","mode":"continuous"} or some variation that enables a continuous connection connection to Notehub.

If that’s the case, we recommend either using

{"req":"note.add","live":true,"file":"mydata.qo","body":{...}}

or using proxied web requests

{"req":"web.post","body":{...}}

The second option does require a Notehub web request proxy route to be configured.

For more information, check out Web Transactions - Blues Developers

Batching Data to Notehub

In this case, the host MCU stores a set of data, and then only provides it to Notecard when it’s ready to upload the data to the cloud. This could be triggered by time, some local event or condition, or a command initiated from Notehub.

Often the host MCU firmware function that adds the data to Notecard has a loop that performs a note.add request on each loop to append new data sets to a Notefile. Something like

while more_data {
   ...
  {"req":"note.add","file":"mydata.qo","sync":true,"body":{...}}
  more_data = dataSent < totalDataAvailable
}

where a note.add occurs on each pass of the loop.

If the request contains "sync":true this could cause any following syncs to stomp on the previous sync depending on the timing of the loop. A variation of this is a {"req":"hub.sync"} is made for each pass through the loop.

Our recommendation here would be to remove the "sync":true from the note.add request and put a {"req":"hub.sync"} request upon loop completion.

(If you want to be tricky, you could set "sync":true on the last pass through the loop, but that’s a little more complicated)

Continuous Mode Options

In addition to the hub.sync option above, if the device is batching data to Notehub, and the Notecard connection to Notehub is in a continuous connection, you have the option to use

{"req":"note.add","live":true,...}

Similar to the “streaming” option above, it will push the data to Notehub.

OR

You could also use card.binary in the loop to append the data to the binary storage on the Notecard, and then do a single note.add.

{"req":"note.add","file":"mydata.qo","binary":true,"live":true}

See Sending and Receiving Large Binary Objects - Blues Developers for more info.

1 Like