Outboard DFU failure with larger binaries

[calcut]

I’m hoping to get some input from Blues on an ongoing issue with outboard DFU.

My previous topic hasn’t had any response,

Outboard DFU issue - error: esp: erasing flash failed

Hi I’ve been testing outboard DFU, and have had success with my hardware setup. However my firmware image now includes more libraries and is larger, and I’m seeing a failure. (The binpack file is 622KB rather than 379KB, so I don’t think I’m hitting the 1.5MB limit) Using Notecard FW 4.4.1.4015700 and CLI version 1.2.1 I’ve pasted the notecard trace output below at the end The offending line seems to be S52:54.66 loader: erasing flash for firmware.bin from 0x00010000 len 0x000e3690 As a …

but having encountered it with different hardware combinations I’m convinced it is a real issue.

I’ll try to break it down to a minimal reproducible example.

Hardware:
Host- ESP32-S3, Unexpected Maker FeatherS3 board
Notecard ESP1.4 (aka Wifi Version 2), latest firmware 6.1.1.16332
Notecarrier-F (wired as per ESP32-S2 Outboard DFU)

Host Firmware:
This is a super simple arduino firmware that just includes variable length audio data so I can control the size.
2024-02-12-102331_audio_490kB.binpack.txt (478.6 KB)
2024-02-12-101251_audio_578kB.binpack.txt (565.3 KB)

I’m attaching 2 versions (remove the .txt extension to use), the smaller one works, the larger one fails with:

S47:20.51 loader: loaded partitions.bin 0x00008116 off 278 len 00000278 (0/7914/557524 remaining)
S47:20.55 loader: flash verified for partitions.bin
S47:20.55 loader: end-to-end bin verified for partitions.bin
S47:20.55 loader: programming completed for partitions.bin
S47:20.55 loader: resetting source offset to 0x00003459 to align image to page boundary
S47:20.55 loader: reading source offset 0x00003459 len 0x00002000 leaving 0x000880be remaining
S47:20.56 loader: processing firmware.bin off 0 (565438/8192/557246 remaining)
S47:20.56 loader: erasing flash for firmware.bin from 0x00010000 len 0x000ac650
S47:23.92 loader: 2971ms timeout waiting for reply
S47:23.92 *** loader program_begin err esp: erasing flash failed (reason #2) *** (err 0)
S47:23.93 loader: programming error: esp: erasing flash failed (reason #2)
S47:24.63 AUX deactivated
S47:26.14 _health.qo: outboard DFU of esp32 firmware: esp: erasing flash failed (reason #2)
S47:26.25 dfu: disabled DFU mode
S47:26.25 sync: wakeup: dfu disabled
S47:26.25 dfu: cannot update host: esp: erasing flash failed (reason #2)
S47:26.25 dfu: retry #10

My best guess at the root cause is the 2971ms timeout waiting for reply might be too short.
Although I’m noticing messages like
U42:25.28 *** WARNING: LARGE ALLOCATION size:72734 remaining:593560 Which might be relevant

Also attaching logs
success_audio_490kB.log (75.6 KB)
fail_audio_578kB.log (68.3 KB)

I’ve seen some inconsistency when reproducing this error that I can’t explain.
i.e. it does occasionally work with larger firmware.
(As per my last comment on my original topic)

[calcut]

Out of curiosity, I tried this with a Blues Swan host too.
It works OK with a ~920kB firmware, but I did see it fail at ~1344kB
This looks like a completely different error, but I did expect this to work given the DFU limit of 1.5MB and the flash size of 2MB

S04:19.73 sync: advancing last sync time from 11:50:10Z to 11:50:21Z
S04:20.52 dfu user: begin 2024-02-12-114905_audio_1344kB_STM32$20240212114959.binpack (1344407 bytes)
S06:10.38 _health.qo: DFU of host firmware aborted: error appending to local file: error writing file
S06:10.38 _health.qo: can't add note: can't exceed 80% of filesystem (currently 100%) {file-storage-full}
S06:10.70 sync: wakeup: dfu SetStateToError
S06:10.70 env: environment modified
S06:10.73 env: updated _state_vars_dfu
S06:10.73 env: dfu updated

[calcut]

I think I have a reliable workaround. Only a little bit ugly.
Should have thought of this earlier!

I can split the firmware.bin into parts and add them individually to binpack so they get flashed sequentially.

e.g. using the “split” command on macOS/linux to keep sizes below 300kB

split -b 300K .pio/build/um_feathers3/firmware.bin ./split_firmware_

~/git/Notecard-DFU/note-cli_1.2.1_macOS_arm64/notecard -binpack esp32 \
0x0:.pio/build/um_feathers3/bootloader.bin \
0x8000:.pio/build/um_feathers3/partitions.bin \
0x10000:split_firmware_aa \
0x5B000:split_firmware_ab \
0xA6000:split_firmware_ac

[RobLauer]

Hi @calcut,

I’m looking into this now and will get back to you when we have some updates. I just wanted to confirm you’re on the latest LTS release (6.1.1) and using only the Notecard WiFi v2?

Thanks,
Rob

[RobLauer]

Also @calcut can you please post a picture of your wiring when using the ESP32-S3?

[calcut]

Yes. on the latest LTS 6.1.1.16332, but also observed with older FW.
I’ve seen this on Wifi v2 and v1 and 99% sure with WBEX cellular ones too.

I’ve seen it with ESP32, ESP32-S2 and ESP32-S3
Notecarrier B and Notecarrier F, and a custom board.

Photos below, which is the same setup as discussed in ESP32-S2 Outboard DFU

[calcut]

Any success recreating the issue?