Problem flashing firmware outside Arduino IDE

[thiagogba]

Board

ESP32 Dev Module

Device Description

Plain Module

Hardware Configuration

Default

Version

v1.0.6

IDE Name

Arduino IDE

Operating System

Windows 10

Flash frequency

80Mhz

PSRAM enabled

no

Upload speed

921600

Description

I was using an ESP32 with 4MB flash and now I am migrating to modules with 16MB.
With 4MB flash module, I can program using the flash_download_tool or esptool outside the Arduino IDE, without any problem, by coping the files and using the addresses show in log from Arduino IDE or even by merging the bin files with esptool and programming from address 0x0.
But with the 16MB flash module, it only works using the Arduino IDE, if I try the same process with esptool and the files or flash_download_tool, the module does not boot anymore, showing the message in the debug section.

Sketch

You can use any example skecth.

Debug Message

rst:0x3 (SW_RESET),boot:0x16 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:1216
ho 0 tail 12 room 4
load:0x40078000,len:10944
load:0x40080400,len:6388
entry 0x400806b4
ets Jul 29 2019 12:21:46

Other Steps to Reproduce

Arduino IDE command to write the modules:
esptool.exe --chip esp32 --port COM3 --baud 921600 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size detect 0xe000 boot_app0.bin 0x1000 bootloader_dio_80m.bin 0x10000 ino.bin 0x8000 ino.partitions.bin

The difference between each module is the partition file, Default "4MB with ffat" and "16MB flash 3MB APP/9MB FATFS", and the flash size in Arduino IDE, the app bin is at 1.01MB at the moment.

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[SuGlider]

@thiagogba - Could you please check the version of the esptool.exe you are using?
You can do it by executing esptool.exe --version and then reporting the output to us.

[lbernstone]

Just a general comment here: Unless you are using the quad pin or have flash that you know is only rated for double speed, you should use qio flash mode.

[thiagogba]

@SuGlider, I was using version v4.1, today I updated to version v4.2.1 and the result is the same. And the flash_download_tool version was 3.9.2 and 3.9.3.
With esptool version v3.0-dev it works, but I cannot create a single bin for flashing.

@lbernstone, thanks, will do that.

[lbernstone]

You can use esptool to get a single binary. After you flash from Arduino, call esptool read_flash 0 0x400000 my_firmware.img (16M is 0x1000000). This will read the whole flash and write it into a monolithic file on your pc. IIRC, there is a tool in esp-idf that can build a single firmware, but using esptool is easy and includes a help listing.

[Jason2866]

Not using Arduino IDE but idk if version 1.0.6 ever worked with flash size 16MB.
Try latest release 2.0.4.
At @lbernstone what makes you think mode qio is not used? The --flash_mode dio needs to be used for mode qio too. The final wanted flash mode is in magic byte and read from 2nd stage bootloader which tries to activate the mode.
And this is maybe the problem here. The magic bytes in firmware needs to be set from esptool.py (firmware header patching). If this is not done or not correctly done boot will fail.
Using esptool v4.21 can be problematic in this case. This version does check if a SHA diggest is there, if this is the case header patching is NOT!! done.
So if in doubt if the SHA diggest is there better use latest v3.x version from esptool.py!

[thiagogba]

Thanks @lbernstone, I tried this once, but the writing is not so efficient, I believe it wastes time writing a lot of unnecessary nulls.
@Jason2866, I will try the new release, thanks. I already wanted to make this change, but the others in the project had vetoed it, as it would extend require time for testing and approval.
In fact what I really wanted was to use pure IDF instead, one day I will get it.

In the meantime I will instruct production to use esptool v3 with all the separate files.

[Jason2866]

@thiagogba We create verified working combined firmwares (the are needed for the webflasher). It is a python script which is integrated in platformio setup of our project. Maybe it helps to understand what needs to be done.This is the script

[jameszah]

There is a similar issue here -- esphome/esp-web-tools#288 -- that produces these messages when programmed using esp-web-tools

23:28:30.382 -> rst:0x1 (POWERON_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT)
23:28:30.382 -> configsip: 0, SPIWP:0xee
23:28:30.382 -> clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
23:28:30.382 -> mode:QIO, clock div:2
23:28:30.382 -> load:0x3fff0030,len:1344
23:28:30.382 -> load:0xf01d020c,len:-2118480896
23:28:30.430 -> 1162 mmu set 00010000, pos 00010000 
23:28:30.430 -> 1162 mmu set 00020000, pos 00020000
23:28:30.430 -> 1162 mmu set 00030000, pos 00030000
23:28:30.477 -> 1162 mmu set 00040000, pos 00040000
....

Arduino 1.8.19 with arduino-esp32 2.04 (and 2.0.3) will both program the esp32 correctly, but using the ".bin" files for esp-web-tools, then the files produced by 2.0.4 do not work. Arduino produces the same esptool programming string, and they both work with the different files.

[Jason2866]

@jameszah Yes, it is the same since in both cases the flashed firmware does not have valid values for flash mode, flash size and flash frequency in the magic bytes of the app firmware. Since core 2.0.4 the precompiled bootloader does not have default values set (finally, which is correct!), so there is nowhere the correct info for the 1st stage and the 2nd stage bootloader what is needed! The flashing tool has to be feed with this infos and has to write it in the magic bytes of the firmware (so called "header patching").
In short this is not an bug, the tools have to take care of the correct valuses.
Before there where assumptions made, which where wrong often, and was resulting very often in issues with the ESP32-S3.

[Jason2866]

23:28:30.382 -> mode:QIO, clock div:2

Clearly is a sign for a invalid or wrong flashed firmware. A device can not start in mode qio. Boot start flash mode has to be dio or dout. Only the 2nd stage bootloader can and will activate mode qio (or qout).

For esptool.py v3.31 this is a valid correct flash command (80M flash freq, flash size detected and mode dio) The final flash mode is the magic header of ino.bin.

python3 esptool.py --chip esp32 --baud 921600 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size detect 0xe000 boot_app0.bin 0x1000 bootloader_dio_80m.bin 0x10000 ino.bin 0x8000 ino.partitions.bin