Trouble Sending Large Binary Objects

Troubleshooting
1

Hello!

My team and I have been working on a project that requires sending large binary objects from the Notecard to Notehub and then to an endpoint. We’ve followed the relevant tutorial and have been testing our set up using the Notehub API and a USB connected Notecard. I currently have a test route set up as a proxy route for web requests pointed to a webhook site. My Notecard is also set to be in continuous mode.

However, I’ve been running into a bit of trouble while testing.

When I run this JSON in the terminal:

{
“req”: “web.post”,
“route”: “test2”,
“binary”: true,
“content”: “application/octet-stream”
}

I receive this data on the webhook:

image
image692×301 6.99 KB

Now based on the documentation, I can verify that web.post sent all the binary data by calling card.binary, however no matter how many times I send the POST request, card.binary always returns as if there is still data in the binary buffer:

> {"req":"card.binary"}
{
 "connected": true,
 "max": 130554,
 "status": "a651409a3cf0a3e4a0b9ea19038cbe54",
 "length": 145,
 "cobs": 146
}

My questions are as follows:

  • Is the data I am receiving on the Webhook supposed to appear like that? How should I be decoding it in order to be able to use it?
  • Why is it that it seems like the binary buffer is never cleared after a POST request, as demonstrated by my call to card.binary?
  • Is a proxy route the correct set up for this kind of project?
2

For example, this is the data printed in Teraterm that was stored in Notecard’s binary storage as a payload - 010000000066C50377C4810006B61021080000200000000000000020.

Then, my webhook received this data - f?wā?!. This resulting string of gibberish ASCII when converted to binary is far too short to represent the original payload. Is Notehub applying some kind of compression/encoding method to the data before it is posted to the webhook?

3

Hello @AnnaStevens, we’re glad to have you here on the forums.

It sounds like you’ve got the basics right!

Is the data I am receiving on the Webhook supposed to appear like that? How should I be decoding it in order to be able to use it?

It’s binary data, it is very difficult to validate without a hex editor.

Can you tell us a little bit more about the data you’re trying to send?

Why is it that it seems like the binary buffer is never cleared after a POST request, as demonstrated by my call to card.binary?

This is by design. This allows you to retry, or send the binary payload to an alternate endpoint.

When you are finished, call NoteBinaryReset(); to ready the binary store for the next payload.

It is shown as step #4 in this part of the guide, but it is necessary for all transmissions.

Is a proxy route the correct set up for this kind of project?

Yes, you’ve got it.

4

Thank you for the response!

The step mentioning NoteBinaryReset() is under the instructions for Arduino SDK, but we’ve been using the card.binary APIs. Do we still need to call NoteBinaryReset()?

The binary data we are sending is a payload of signals gathered from nearby sensors. Here is a breakdown of the data we are sending and receiving at each step of our current process:

Bytes obtained from nearby sensors: 010000000066C3AA4FC8550022F611E16C001024009801000001

  • Payload buffer: 010000000066C3AA4FC8550022F611E16C1024010098010000010000Y
  • Original string: 010000000066C3AA4FC8550022F611E16C001024009801000001
  • Payload after bin store: 080B0B0B0B0D6CC9A045C25F0328FC1BEB661A2E0B09920B0B080B0AY
  • Data received at webhook: fêO?U"??l$?
  • Convert to binary/hex : 01 66 EA 4F 3F 55 22 3F 11 3F 6C 10 24 01 3F 01

So apart from being different values, it’s not enough bytes to represent the original payload. Is some kind of data compression occurring?

Also, does the Notehub handle COBS decoding, or is this something we need to perform on the data once our endpoint receives it?

5

So, currently, I believe I figured out what the issue was. The webhook.site was automatically UTF-8 encoding our payload. Any bytes UTF-8 did not recognize were getting replaced causing the data we received to be garbage. Unfortunately, this meant we couldn’t use the webhook.site to test our route. Instead, I whipped up a ASP .Net API that could read binary data from the HTTPS request body and logged the resulting data as a hex string.

3 Likes
6

Hi @AnnaStevens,

I’m happy to hear you’ve worked it out! I apologize for the delayed responses, but I was traveling and haven’t had an opportunity to sit down and draft thoughtful responses.

The only outstanding question I see:

Do we still need to call NoteBinaryReset()?

Yes. It’s not specific to Arduino, and is a shortcoming in the documentation. We’ll get it fixed. :+1:

Best,
Zak