Notecard Outboard DFU support for large (>1.5MB) binaries

Firmware
1

Our project uses an ESP32-S3 with 16MB flash. At present, the firmware fits within the 1.5MB outboard DFU limit, so this is not an immediate concern.

Thinking ahead, the host firmware may expand beyond that limit - the Wi-Fi and Bluetooth stacks on ESP32 add a lot of overhead to the binary, and it’s pretty close to the limit when those features are compiled in.

Has anyone managed or have any thoughts on how to overcome the Notecard’s host firmware size limit?

Thinking aloud, perhaps splitting the binary into 1MB chunks and putting these in a series of binpack files, each with the correct memory offset?

This might be workable, so long as the ESP32 doesn’t attempt to boot/apply the firmware until all binpack files have been downloaded, which I believe is handled by the ESPs secondary bootloader.

Perhaps Blues already have this on their roadmap so that Notecard can scale with larger memory devices?

When the time comes, I’ll experiment a little. Who knows, it might just work as is!

2

Hi @devElert,

Thanks for reaching out.

I’ve been doing a bunch of work on the ODFU code for ESP32-based hosts lately, specifically the S3. I’m going to run some experiments to better understand the size limitations and get back to you.

Best,

Hayden Roche
Blues staff firmware engineer

3

After some research and discussion with the team, I’ve found 1.5 MB is the limit. The Notecard’s DFU design downloads the entire image, verifies it, and then begins programming. This requires buffering the image somewhere, and 1.5 MB is that buffer size. For certain Notecard SKUs, you may find that you can push beyond 1.5 MB, but this is out of spec, and proper behavior is not guaranteed.

I can imagine a different design where we stay connected to Notehub and buffer image chunks instead. While technically feasible, this approach is incompatible with the Notecard’s current design.

4

Thanks for the response.

I understand the hard limit of 1.5MB, and understand this won’t change anytime soon. (By the way, is that limit for the uncompressed or compressed size of the binpack? I see we get 39% compression at present, which is great and gives more room should the decompression be done while the image is being sent to the host.)

What I’m thinking is that large (>1.5MB) firmware is split up into chunks, each one a separate binpack under the 1.5MB limit. Each binpack would correspond to a different address on the host, appropriately offset to not step on the binpacks that come before it.

The binpacks would be be flashed one by one to a separate area from the running application (e.g. OTA_1 partition on ESP32.) The ESP32, and other MCUs, have a secondary bootloader which performs integrity checks, so it would not boot this second application until all the parts are fully downloaded. For the EPS32, there is an OTA data region that instructs the bootloader which partition to boot from, so this data would be flashed last, allowing the OTA update to be cancelled while still leaving the device in a working state.

The one thing I’m not sure of yet is whether the bootloader will take care of copying the image from OTA_1 to OTA_0, so that OTA_1 is free for the next update.

Ideally, Notehub would orchestrate updating all of the binpacks, in sequence, from a single file, such a zipfile. In the absence of that, flashing the individual parts could be co-ordinated via Notehub’s new Firmware Management API.

One positive aspect of this is that the separate parts could be sent out over time, allowing the app to run in between each binpack that is flashed.

MCUs that have only enough flash for one copy of the firmware would not use this double buffering approach, and instead would flash to the primary app region, with the caveat that the device won’t run the app until all binpacks have been laid down in memory, which is similar to how ODFU works at present, namely, that the app doesn’t start until the update is complete.

I hope this outline shows the approach I’m thinking - feedback very welcome!