Problems with web.* requests and proxy route

Notehub.io
1

I have created a proxy route to forward web.get and web.post requests to the Notehub API, so that we can control firmware updates from firmware (say, from provisioning/testing to production firmware.)

The route URL is set to https://api.notefile.net/v1/projects/[project], and the Authorization header set to Bearer <PAT>

I started with this request to fetch a list of firmware:

{"req":"web.get","route":"firmware-api","name":"/firmware"}

Which gives this error:

{
 "result": 503,
 "body": {
  "err": "hub request error: response length (18849) is larger than max (8192) {too-big}"
 }
}

I tried using pageNum and pageSize query parameters to cursor through the list, but these don’t seem to be supported for that firmware API.

Next, added "binary”:true :

{"req":"web.get","route":"firmware-api","name":"/firmware", "binary":true}

Which gives this error :

{
 "result": 503,
 "body": {
  "err": "hub request error: json: cannot unmarshal array into Go value of type map[string]interface {}"
 }
}

It seems the JSON deserialization doesn’t like the firmware API response format. But I didn’t want Notehub to deserialize the data, so I changed the content type:

{"req":"web.get","route":"firmware-api","name":"/firmware", "binary":true, "content":"application/octet-stream"}

Which surprisingly gives the same unmarshal error.

I tried various other content types, including application/x-binary, application/pdf, image/jpeg, application/x-gzip but all gave the same error.

The route doesn’t have any JSONata at present.

Looking at the event in _web.qo, there’s no mention of the content type.

{
    "method": "GET",
    "name": "/firmware",
    "route": "firmware-api"
}

It’s as if Notehub is always trying to parse the response even when the MIME type override in the request is given. Even if Notehub has to parse the JSON (which I imagine it will when using JSONata), it would be helpful if unsupported formats in JSON were either directly supported, or transformed in some way to make them compatible, or simply not output by your own APIs

For example, the /firmware API returns a response that is an array of objects, with each object carrying firmware details:

[
{ … firmware 1 … },
{ … firmware 2 … },
… etc
]

Perhaps Notehub could transform top level array formats automatically to something like:

{ “array”: [
   { … firmware 1 … },
   { … firmware 2 … },
]}

before JSONata processing begins. Alternatively, perhaps add a parameter to the firmware list API that changes the output format to match that above, with pagination. Just my thoughts, no doubt there are other options too.

Thanks!

1 Like
3

I’ve implemented a lambda/hosted function which performs the transformation of top-level arrays in API responses, and adds pagination and ordering. With these changes, Notehub correctly processes the JSON response from the firmware API.

4

Hey @devElert,

The Notecard’s web.* requests only work with JSON RPC, which requires the JSON response to be an object and not an array of objects.

We don’t have this detail in our documentation, and we should. I’ll be putting it up there later today.

You can work around this pretty quickly with an inbound JSONata expression though. I was able to get /firmware results down to my Notecard by configuring my route’s Inbound Response Transform to { "items": $ }.

Screenshot 2025-11-20 at 11.45.24 AM
Screenshot 2025-11-20 at 11.45.24 AM777Ă—248 24.2 KB

You can also use JSONata to do a bit of filtering. For example the following JSONata returns only the first three entries in the array, which helps reduce the amount of data you’re sending to your device each time: { "items": $filter($, function($v, $i, $a) { $i < 3 }) }.

As a side tip: AI is super good at writing / understanding JSONata. I find it hard to write JSONata by hand, but Claude makes me feel like an expert :slight_smile:

TJ

2 Likes
5

I’m also going to add an item to our backlog to add pagination to /firmware. What ordering are you currently doing in your lambda? I want to make sure we account for that too.

TJ

1 Like
6

Thanks for looking into this so quickly @tjvantoll!

Great to see you got it working with JSONata on the inbound response - I did try that but got the same unmarshal error. I’ll got back to that and see what was different. (It was failing when the jsonata returned an object that contained just the device uid and pageNum/pageSize values.)

I’ve added JSON:API sorting and filtering support to the lambda, so it’s pretty open-ended. In practice, I’ll be reverse sorting by date/version, and filtering by firmware name (particularly bin vs binpack as well on as the firmware name prefix, since there are multiple types of host firmware). Using the filename to encode details is effective for now, but we’ll probably move to using the JSON metadata registered with the firmware (although I just checked and don’t see that in the docs for the Get Available Firmware API) so we’ll stick with filtering the filename with a regex for now.

7

I tried the { "items": $ } jsonata on the inbound response, but still get the same unmarshal error, both with running our provisioning firmware on Notecard, and in the in-browser terminal.

{"req":"web.get","route":"firmware-api?pageCount=5","name":"/firmware", "binary":true}

{
 "result": 503,
 "body": {
  "err": "hub request error: notehub route with alias 'firmware-api?pageCount=5' cannot be found"
 }
}

But I just noticed my mistake - pageCount is on the wrong parameter. It should be:

{“req”:“web.get”,“route”:“firmware-api”,“name”:“/firmware?pageCount=5”, “binary”:true}

:person_facepalming:

I only noticed it when removing binary:true from the request:

{"req":"web.get","route":"firmware-api?pageCount=5","name":"/firmware"}

{
 "result": 503,
 "body": {
  "err": "hub request error: notehub route with alias 'firmware-api?pageCount=5' cannot be found"
 }
}

But this turns into the unmarshal error when binary:true is present (even with the JSONata transform in place.)

Glad to have gotten to the bottom of this. Thanks for your help!

8

I can understand the JSON restriction to RPC format, but is this true even when the response is not JSON? For example, could I use web.get to retrieve some arbitrary resources?

9

For example, could I use web.get to retrieve some arbitrary resources?

Yep. There might be some restrictions I’m not thinking of, but I just set up a simple example and it worked fine.

I ran {"req":"web.get","route":"test","content":"text/plain"} on my Notecard, and I had a proxy route point to a server that returned “test” and a 200 response. I got this back. (the payload is “test” but base64 encoded).

{
 "result": 200,
 "body": {
  "Content-Length": [
   "4"
  ],
  "Content-Type": [
   "text/plain"
  ],
  "Date": [
   "Thu, 20 Nov 2025 19:56:29 GMT"
  ]
 },
 "payload": "dGVzdA=="
}

If you hit any issues trying to GET stuff from a proxy route let me know. And appreciate your feedback on the firmware APIs!

TJ

1 Like
10

Ok so I actually have an update on the /firmware API. We have a few additional filters that for some reason aren’t on the API docs today (I’m working on adding them now): filename, md5, and version.

filename and version do partial matching, so version=1.2 will find a binary with a version of 1.2.3, but there’s no regex support.

And by default the binaries are sorted by created descending.

TJ

1 Like