When there are multiple Notes that need routing, only the first Note is successfully routed; subsequent Notes are considered successful by Blues (green tick, Status 200), but ThingSpeak returns a “0” - nothing received.
For example this route log shows 2 Notes successfully routed.
But only, the first route is received and visible in ThingSpeak. The second route returns “0” from Thingspeak and no data are visible.
I don’t think it is related to the ThingSpeak rate limit as I have rate limiting enabled in the Route at 0.07/s (1 per 14.3s) and I have a paid ThingSpeak account which allows for 1 upload per sec. My Route looks like this:
{
"created_at": $fromMillis(when*1000),
"api_key": $key,
"field1": body.weight,
"field2": body.temp,
"field4": body.soc,
"latitude":best_lat,
"longitude":best_lon,
"status": best_location
}
Where key is obtained from an env.set variable. This phenomenon is perfectly repeatable.
Any ideas?
Hey @rberkelm,
Yeah this is weird.
Unfortunately I’m not sure I can offer much help though, as everything in Notehub seems to be working correctly. I was able to recreate your problem, but in my testing Notehub was sending the correct data and also rate limiting appropriately. ThingSpeak doesn’t appear to say what a response of 0 means in their documentation either.
I think you may need to reach out to their support to see why they’re rejecting your data. It kind of feels like they’re rate limiting you (even if they’re not supposed to be).
Thanks,
TJ
Thanks @tjvantoll for looking into this from your end and recreating the problem. I agree, it does still feel like a rate limit issue. I can certainly re-rout the Notes manually and they get processed fine. Before I reach out to ThingSpeak, can you please double-check that the rate limit works ok on your end? I tried to set it to a lower value (0.05 or 0.06) but these were not accepted.
Hey @rberkelm,
I just verified that our rate limiting does work as expected in my project.
We don’t allow for very small values for Requests per Second. Long delays can cause our outbound queues to fill up quick, and to potentially stay full for a long period of time. I was actually surprised we allow 0.07 requests/second, but I tested with that value and it appears to work fine.
That being said we do a pretty awful job communicating that restriction, especially in the Notehub UI. I’m going to pass this along to our cloud team and see what we can do to improve our UI/docs.
Thanks,
TJ
Thanks @tjvantoll ! I’ll let you know what transpires.
I hear what you saying about outbound queues filling up, but bear in mind that 0.07 requests/sec is 14.3 sec and the free tier in ThingSpeak is 15 sec per update?
Hi @tjvantoll ,
I fear there might still be something we’ve missed. ThingSpeak want me to do some more experimenting. So, I did a little experiment: I sent 3 Notecard-formatted Notes to my notefile data.qo, then sync’d via the cellular network and all three were routed fine to ThingSpeak (rate limit=0.07). Then I sent 3 NTN-formatted notes to my notefile sat.qo, then sync’d via NTN and the first note routed fine, but not the other 2 (rate limit=0.07). I then changed the rate limit to 0.5 and did the same and the result was the same. Then I changed the rate limit to 1 (which should still work for my paid plan) and sent another 3 NTN notes and the result remained the same. So is there something in the way I have formatted my routes? I’ve stared at them both a 100 times and for the life of me, I cannot see what I’m doing wrong.
This is my cellular data.qo route setup
and the JSONata:
{
“created_at”: $fromMillis(when
1000),
“api_key”: body.key,
“field1”: body.weight,
“field2”: body.temp,
“field3”: body.bat,
“field4”: body.soc,
“field5”: body.netrawwt,
“latitude”:best_lat,
“longitude”:best_lon,
“status”: best_location
}
This is my NTN sat.qo route setup:
And the JSONata:
{
“created_at”: $fromMillis(when1000),
“api_key”: $key,
“field1”: body.weight,
“field2”: body.temp,
“field4”: body.soc,
“latitude”:best_lat,
“longitude”:best_lon,
“status”: best_location
}
Any ideas?
I’ll pass this feedback along, but our cloud team chose that limit as a compromise that allows some degree of delaying your events, while still protecting our infrastructure from filling up with unrouted events.
So, I did a little experiment: I sent 3 Notecard-formatted Notes to my notefile data.qo, then sync’d via the cellular network and all three were routed fine to ThingSpeak (rate limit=0.07). Then I sent 3 NTN-formatted notes to my notefile sat.qo, then sync’d via NTN and the first note routed fine, but not the other 2 (rate limit=0.07). I then changed the rate limit to 0.5 and did the same and the result was the same. Then I changed the rate limit to 1 (which should still work for my paid plan) and sent another 3 NTN notes and the result remained the same. So is there something in the way I have formatted my routes?
The outlier here seems to be that first sync over cellular. In your initial test (and mine), sending multiple events simultaneously never seemed to work. Did you do anything different on that run that may explain that?
I looked over your config and JSONata and nothing seems off, for better or worse 
Another thing you can try when testing is setting up a second route to a testing site like https://webhook.site that uses the exact same config. That can help prove that Notehub is indeed forwarding your data correctly and rate limiting appropriately.
TJ
Hi @tjvantoll
Great suggestion re the webhook.site! So I set up a second route with them. Rate limit was set to 0.07. I then sent 4 x NTN-formatted notes and sync’d them over NTN. All four notes arrived at the Webhook Site at the same time!
.
So these notes are NOT being rate-limited for some reason.
Just to check again with my Notecard-formatted route, I sent 3 x notes and sync’d over cellular and they arrived ~15sec apart.
So, my data.qo route works fine. I thought maybe it had something to do with whether the api key was sent in the body of the note or obtained from an Environment variable, but that made no difference.
I wonder if the rate limit code you have in your route settings doesn’t get implemented with compact format notes?
Hey @rberkelm,
Oh wow interesting. I was able to recreate the issue. If you send data over NTN route rate limiting doesn’t apply at all. (Compact data works fine over cellular, so it’s only NTN mode that’s problematic.)
I passed this along to our cloud team. Thanks for reporting this!
TJ
Glad we managed to get to the cause! Any chance you could report here when they have implemented a fix?
Thanks again for your help!
Hey @tjvantoll
Can your cloud guys give a rough time for they they might have a fix? I have some customers waiting but I can’t ship until the routing works.
Hey @rberkelm,
Unfortunately I don’t have an estimate for you. I added a note to our issue tracker that this is blocking you, which should help give it a bit more priority.
TJ
Hi @tjvantoll ,
Time is ticking along and my customers are still waiting. Is there any chance you could set up a temporary leaky-bucket route until this bug is fixed? Something like the one @gwolff set up for me some years back? Maybe Greg still has it and could re-activate it? It would help me out a lot.
Hey @rberkelm,
Just talked with @gwolff. Unfortunately we can’t find the code he gave you a few years ago, but he let me know the general approach of the route. I’ll see if I can recreate something that’ll work for you on Monday.
Sorry for the delay on this. Our cloud team is aware this issue blocking you but it’s not a trivial fix. I’ll see what I can do to help you out in the meantime.
TJ
@tjvantoll ,Thank you, that would be fantastic!
I wrote a quick Express server that seems to do the trick when I test it locally index.js · GitHub. I’ll put it on some hosting provider tomorrow but I ran out of time today.
TJ
2 Likes
Ok I put it up on some hosting and in my testing it seems to work. Here’s the source code: GitHub - tjvantoll/rate-limit-route. To use it you’ll want to update the URL in index.js here rate-limit-route/index.js at main · tjvantoll/rate-limit-route · GitHub
Then you’ll need to host this somewhere. I used Render because they have a free Node hosting tier that’s pretty easy to use. I imported the repo from GitHub, gave it a Build Command of npm install, and a Start Command of node index.js. After it deploys it gives you a URL you can use for your Notehub route (for example https://rate-limit-route.onrender.com). You can try my URL to see the rate limiting working, but it’ll be sending data to my webhook and not your ThingSpeak.
You can use any Node hosting service to try it, but I find Render pretty easy to use. If you have any questions let me know.
TJ
1 Like
Beautiful work @tjvantoll ! I managed to set up the code on Render with a little help from ChatGPT (gotta love it!) 
I’ll know this evening if it works.
Thanks for your time on this, I really appreciate it!
Cheers
Ray
1 Like
Hey @tjvantoll
I feel like I’m close but its not quite working. I don’t have enough experience with this to work out what’s going on. The error in Notehub is:
error performing request: error on HTTP request to https://rate-limit-route-o6eh.onrender.com: Post "https://rate-limit-route-o6eh.onrender.com": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Sorry to keep hogging your time, but can you please point me in the right direction? My setup in Notehub is:
In Render it looks to be deployed ok:
In Github:
https://github.com/rberkelm/rate-limit-route/blob/main/index.js
Ugh.
So, based off that error message I’m guessing the problem is related to timeouts of some sort. I’d guess that either:
- Render is shutting down the server when it’s inactive with a long cold start, and the request from Notehub times out waiting for Render to answer.
- Too much data is building up on the server and the requests time out. Now that I think about it, my server design probably needs to be improved. I probably should send back a response to Notehub immediately, and forward requests along to webhook/ThingSpeak/whatever in a way that isn’t tied to the incoming request.
I can take a stab at that as it shouldn’t be too hard. In the meantime though, could you tell me how often you’re sending data through? That might help narrow down what the problem is.
TJ
