NTN notes send ok, but don't arrive in Notehub

[rberkelm]

For the past month or so, I don’t seem to be able to get any more than 2 NTN notes to arrive in Notehub, even though 3 or 4 are sent. In my setup, I queue an NTN note every 4 hours and after 12 hours I sync 3 notes. Of these, only the first 2 arrive in Notehub. Here is an example 12 hour log which shows that 3 notes were successfully sent.
notecard_2025-07-06T07_50_12.log (18.5 KB)
This is what I see in Notehub:

If I manually add 4 NTN notes and then sync, only the first two arrive. See this log:
notecard_2025-07-06T23_03_00.log (6.9 KB)
Is there a filter to limit the number of notes at the server?

[hroche]

Hi @rberkelm. I’m sorry to hear about this problem. I just attempted to reproduce the issue using an NBGLN and the same firmware version you’re running, but I wasn’t successful; the 4 Notes showed up on Notehub. We just added some additional debug logging on the Notehub side to try to get to the bottom of this. Could you please run your experiment where you manually add 4 Notes again, and once again provide us with the trace log? Thanks very much for your patience!

[rberkelm]

Hi @hroche
I’ve manually added 3 Notes and had 2 Notes previously queued - total 5 Notes successfully sent according to the log, but still only the first 2 arrive in Notehub.
notecard_2025-07-08T04_41_14.log (7.2 KB)

The rest must be caught and rejected somewhere in the server? I don’t think this issue is isolated to this particular Notecard + Starnote combo. I have another Notecard + Starnote set up and the same is happening to its data.

[rberkelm]

There is another issue as well which may or may not be related. The timestamp on the Notecard is currently around 7 minutes behind the actual time. The Notecard reports a timestamp of 1751952148 ( Tuesday, 8 July 2025 15:22:28 [GMT+10:00]) but the actual time is 15:29.

I reported a previous experience of this issue here: Card.time request returns incorrect time
The Notecard has remained powered for the past 5 weeks, only syncing via NTN, and other than the issue of only receiving 2 Notes, it has run without any problems.
The Notecard Time issue was supposed to be addressed in the next Dev release and LTS8. Since LTS8 is out and I am running the latest Dev release (Version: 9.1.1.17181), I assumed it was fixed, but it appears not to be.

[hroche]

Thanks for the additional info!

With the additional debug logging we added on the Notehub side, we are starting to get a better picture of what’s going on, but we are still searching for the root cause. Unfortunately, we haven’t been able to reproduce the issue, still.

You mentioned that you have two setups that both exhibit this same problem. Are both Notecards using the same project?

Something that may be revealing is if you are able to reproduce the issue using NTN mode WITHOUT a paired Starnote. That is, run the experiment using NTN transport and just a Notecard (no Starnote) as we describe here. Under the hood, this will cause the Notecard to send the packets in NTN format but ultimately over UDP+cell instead of a satellite connection.

Finally, would you be ok with inviting Blues support to join your project? We are wondering if the issue is somehow dependent on the project, since we are unable to reproduce using our own projects. Perhaps if we link our setup to your project, we’ll be able to reproduce the issue.

Thanks again for your patience. We are keen to resolve this issue ASAP and apologize for the headache!

On the issue with card.time, the bug fix for that one is coming in the next developer release (and LTS patch), which we are aiming to release this week. A release with that fix has not been made public, yet.

[rberkelm]

Hi @hroche
My second setup is a different project, different firmware etc. Just going through the history… Both projects were sending 3 NTN Notes every 12 hours. Up until (and incl) June 10, all three Notes were received fine in Notehub. As of June 11, only 2 Notes were received with each sync. So, something changed on June 11. I don’t think it was on my side because I was travelling on June 9 and didn’t touch anything until I got back this week.

As requested, I removed the Starnote (without powering down) and sent 4 NTN notes. The Notecard did not sync over cellular. I let it go for ~90 min and the Notecard just kept reporting “sync: connect error: ntn: satellite module not responding”. I then power-cycled the Notecard and let it run for another ~40 min. Still no sync. I then did a ntn.reset, followed by a power cycle and added an extra Note. After that, it was able to sync via UDP. All 5 Notes arrived in Notehub.
notecard_2025-07-09T02_18_11.log (256.2 KB)

I was curious to see if this process might have solved my issue, so I reconnected the Starnote, power cycled the Notecard, let it do a cellular sync. I then changed to ntn mode, queued 4 Notes and sync’d via NTN. Unfortunately, only the first 2 Notes arrived in Notehub, as before.

I’ve invited Blues to my project. Hope you can find some further clues there.

[hroche]

Thanks for the additional debug work. We really appreciate it, and I’m happy to report that I think we’ve fixed the issue. Of course, you should validate that we’ve actually fixed it on your end.

We were ultimately able to reproduce your problem and traced the problem back to a timeout issue with Routes. We believe the Route you’re using is particularly slow (more than 10 seconds) and was causing us to hit a timeout path specific to NTN that we hadn’t exercised before. There was a Notehub bug in that path that we’ve fixed.

Thanks again for your patience. We’re lucky to have customers like you to help us make our products better. Please let me know if the issue persists, but I am cautiously optimistic that it should be resolved.

[rberkelm]

Hey @hroche , that is great news! Thanks so much! Just so I understand things a little better, does this mean the process of routing is done before the data arrive in Notehub?
The route I’m using is a dodgy temporary one to overcome an issue with rate-limits in the Blues routing on NTN Notes, as detailed here. Is there still a fix on the way for this bug? It’s been over 5 months since I reported it.
Thanks again for your great work!

[scottfrazer]

Hello @rberkelm, the issue you mentioned about the rate limiting not working has been improved. It is not live yet but it will be deployed during our next deployment. I’ll mention it here when it’s deployed

[rberkelm]

Great to hear there is progress!! Thanks @scottfrazer!

[rberkelm]

Hi @scottfrazer , do you have an update on the rate-limit routing of NTN notes? I used to be able to get at least the first route to ThingSpeak to work, but now none of my NTN routes to ThingSpeak work. All return a response of “0”.

[scottfrazer]

I took a quick glance at the aggregate results of your routes to ThingSpeak are working from our perspective. I see we get back valid HTTP responses a vast majority of the time. It does appear like one of your routes is consistently timing out. Could you give me more information about which route specifically you’re referring to and I can dig in further?

[rberkelm]

For now, all my NTN routes are being done via special route TJ cobbled together for me ( Only one of multiple Notes is successfully routed - #25 by rberkelm ). It works but it is only a temporary solution. Sending my routes direct to ThingSpeak does not work. Do you want me to share my project with you so you can have a look?

[scottfrazer]

Yeah, can you go to Settings → Members then click the “Invite Support” button? Then tell me the name of the route that isn’t working?

Thanks!

[rberkelm]

hey @scottfrazer The route to look at is “Thingspeak_Sat”

[scottfrazer]

As far as I can tell, the Thingspeak_Sat route is working from Notehub’s perspective. The HTTP request is being issued and a response is being returned and it’s only taking 10s of milliseconds for each request. I don’t see anything wrong from Notehub’s perspective.

However, it seems like the response it’s getting back is a “400 Bad Request” and the body is just a 0. This leads me to believe that the HTTP request is somehow malformed, which leads me to believe that the route is not configured correctly. It’s hard to tell why api.thingspeak.com is rejecting this request though.

Should this route work the same as the route for rate-limit-route.onrender.com? Maybe we can look at how that proxy service is issuing the HTTP request to gain more insight

[scottfrazer]

If I’m not mistaken, the documentation for that endpoint is here: https://www.mathworks.com/help/thingspeak/writedata.html.

One thing that stands out is that there’s no API key presented with the request. Maybe that’s the issue. The documentation says:

You can also send the Write API Key by using a THINGSPEAKAPIKEY HTTP header.

perhaps you can try adding that header and trying it again?

[scottfrazer]

I also noticed that the Thinkspeak_rate-limit-route route has "api_key": $key in the JSONata expression but Thingspeak_Sat has "api_key": body.key, in the JSONata expression. It looks like the former is actually resolving to something that looks like an API key but the latter is empty.

[rberkelm]

Well spotted! It should be “api_key": $key”. I did a bit too much copying and pasting trying to get to the bottom of this issue. However, even after correcting this and manually routing to the Thingspeak_Sat route the Status shows as 200, but the Response body is still 0.

The Thingspeak_Sat and Thinkspeak_rate-limit-route route are identical as far as I can see, barring the URL.

Similarly, the ThingSpeak_LoRa_sat and ThingSpeak_LoRa_cell routes are identical, except the former is directed to https://rate-limit-route.onrender.com and the latter to https://api.thingspeak.com/update.json. If I manually reroute from sat to cell I get

The last time I had a Status of 200 and a Response of “0”, it turned out to be the rate limit issue with Notes coming from NTN. See the start of this thread: Only one of multiple Notes is successfully routed

So, is there still an outstanding issue with NTN notes and how they get routed?

[gwolff]

Hi @rberkelm !

Looking at the thread, and working with @scottfrazer, I think we understand what’s going on. This doesn’t appear to be a Notehub bug.

Recommended Actions:

1. Enable only 1 Thingspeak route in Notehub per Thingspeak API key
2. Set the Route Requests per second configuration to 0.066

Why only 1 Thingspeak Route?
Our understanding is Thingspeak rate limits incoming requests per API key. Rate limiting of requests performed by Notehub routes is done per route. This means if you have more than one Notehub route pointing to Thingspeak, if they both end up executing with the same 15 second window, one of them will succeed and one will fail. To prevent this, use a single Notehub route for the same Thingspeak API key.

Why do we think it’s a rate limit issue?

The Thingspeak endpoint will return a 0 for requests that were not processed. This could be for any number of reasons including violating the rate limiting, or providing an incorrect API key. The unfortunate thing from Notehub’s point of view is Thingspeak returns an HTTP status code of 200, which indicates a successful transaction. So Notehub thinks it was successful, but the data won’t appear in Thingspeak channel.

Normally this type of error would return a 429 for rate limit violations or 401 unauthorized …or literally anything other than a code that is in the 200’s.

In addition, since we see there are multiple routes pointing to the same Thingspeak endpoint, and one will succeed, and the other can fail. This indicates that the data can get through, but may be denied if because one route has already executed in the 15 second time window.

Since the routes have the same rate configuration, why don’t they work the same?
The 2 routes are fundamentally different.

The one going directly to Thingspeak needs to abide by the Thingspeak API rate limit of 1 request every 15 seconds. In the Notehub route this is 0.066 requests per second (this just over 15 seconds between requests, but it will get saturated to 15 seconds by Notehub under the hood)

The one going to the leaky bucket algorithm doesn’t require any rate limiting. That’s by design. The purpose of the leaky bucket is to convert burst data transmissions into a series of requests occurring at a uniform rate. The leaky bucket imposes the rate limit, not Notehub.

Wasn’t there a bug in the leaky bucket?

Yes, the leaky bucket implementation we provided did have a bug in it. Though that may not have actually prevented data from reaching Thingspeak.

The primary issue is the API rate limit was being exceeded because the Thingspeak API had already been called via the same API key within the 15 second window by a different Notehub route. This conflict of course could appear somewhat random, and may be exacerbated by the fact that multiple messages from the Notecard may have been uploaded to Notehub in the same sync from Notecard, which would have resulted in more Thingspeak API calls.

Hope this helps!