Change the partition table and bootloader by OTA (IDFGH-14686)

[fernandoeqc]

Hello @KonstantinKondrashov, I have many devices in the field running ESP-IDF v4.0.1, and I would like to change the bootloader and partition table in these devices. I am aware that changing the partition table or bootloader via OTA is not safe.

Setup specifications

• IDF Version: ESP-IDF v4.0.1
• Operating system used: Ubuntu

Goal

Increase the 'nvs' partition add the 'log' partition at the end of the flash and change bootloader to make field devices compatible with a new design structure

Old partition table:

Name	Type	Subtype	Offset	Length	Flags
nvs	data	nvs	0x9000	64K
otadata	data	ota		8K
phy_init	data	phy		4K
ota_0	0	ota_0		1536K
ota_1	0	ota_1		1536K
coredump	data	coredump		128K

New partition table:

Name	Type	Subtype	Offset	Length	Flags
nvs	data	nvs	0x9000	128K
otadata	data	ota		8K
phy_init	data	phy		4K
ota_0	app	ota_0		1536K
ota_1	app	ota_1		1536K
log	data	nvs		800K

Steps

I think of two main workflows:

1:

1. Compile the Project with the new partition table.
2. Attach the compiled partition-table.bin hard-coded in the project Repartitioner.
3. Device starts OTA binary Repartitioner. Restart the device.
4. Make sure the app is running in ota_0 to avoid corruption in the next OTA.
5. Device starts OTA binary Project compiled with the new partition table.
   1. Instead of writing the received binary in the current ota_1, save it to address ota_1 in the new partition table (I'm not sure if it's possible in v4.0).
6. Erase partition table memory.
7. Re-write the new partition table.
8. Restart the device.

2:

1. ...
2. ...
3. ...
4. Erase partition table memory and bootloader.
5. Re-write the new partition table and bootloader.
6. Device starts OTA binary project compiled with the new partition table. (Here we have a critical point. During download, any reset will brick the device.)
7. Restart the device.

Possible partition re-writing methods:

• Adapt ESP-IDF partitions_ota to change the partition table and bootloader. In my opinion, the biggest challenge is adapting functions, structs and enums from ESP-IDF v5.4 to ESP-IDF v4.0. For example, the struct esp_https_ota_config_t changed a lot and is used by esp_https_ota() to specify the range to write the new OTA. The old structure does not have the data of where to start writing.

• Another method is using esp-idf-repartitioner as a reference to make the change. It uses the function spi_flash_erase_range() to erase the partition table and spi_flash_write() to write the new partition table, which are functions that exist in IDF v4.0.

In summary, I have some questions:

• Would it be realistic for me to adapt the code in the partitions_ota to make this change?
• If I use method 2 with spi_flash_write(), would I have a problem with the hash of each partition? Or maybe with the hash of the flash itself? Is there a way I can get around this?

[KonstantinKondrashov]

Hi @fernandoeqc!

Yes, the OTA update bootloader and partition table are not safe operations. You can do it if only necessary. If the goal is to update the partition table then it is necessary to update the bootloader. The old bootloader should be able to load the new app and partition table. If you still think that the bootloader should be updated too then you can split it into two tasks if the partition table is left at the same address.

Would it be realistic for me to adapt the code in the partitions_ota to make this change?

Supporting it in the 4.x version will require some effort to backport these commits. It is not a simple task. I think it would be difficult to achieve that.

• 02d61c1
• 8c4f576
• 97d150d
  Won't your app work on the newest ESP-IDF? You can try it and leave the old bootloader on the devices.

I also want to note that the partition table has an MD5 checksum but early versions do not have it. Please be aware of it, it has to be compatible with both versions. https://docs.espressif.com/projects/esp-idf/en/v4.2/esp32/api-guides/partition-tables.html#md5-checksum. The partition table offset is hardcoded in the bootloader and its address defines the max size of bootloader, so the part table can not be moved and have something in between. (Your image does not show the position of the partition table).

The bootloader with secure version V1 can not be updated (the SB digest is allocated in front of the bootloader image). The commits above do not care about this case.

If I use method 2 with spi_flash_write(), would I have a problem with the hash of each partition? Or maybe with the hash of the flash itself? Is there a way I can get around this?

We do not have a hash for each partition. If the secure boot feature is enabled the app has an appended signature block at the end image. It is part of app images only. I do not see in issue with that.

This approach can be also considered, use the intermediate parttition table, at least applies to your case.

• OTA update app that will be capable of copying ota_0 to ota_1.
• OTA update the old part table to the intermediate part table.
• Copy the ota_0 to ota_1 (ota_1 has changed offset corresponding to the intermediate part table.)
• Now the new part table can be finally downloaded.
• ota_1 is running.
  Due to the partition table being left at the same address, the bootloader will be happy with that, no need for it to be updated.

[KaeLL]

The bootloader with secure version V1 can not be updated (the SB digest is allocated in front of the bootloader image)

It's still possible to overwrite the SB digest + Bootloader in case of host-based Secure Boot, right?

Also, are there any other hard requirements similar to the ones about partition table offset update implicitly demanding bootloader updates (which, in turn, if I'm right in the question above, also implicitly requires updating the SB digest)? Or it's only a matter of shuffling them around (other than the risk of bricking the device)?

[fernandoeqc]

@KonstantinKondrashov,
Thanks for your reply! I used your suggestion with some adjustments.
I used functions available in IDF v4.0 Here are the steps in summary, in case anyone has the same goal:

Stage 0

• Perform OTA of the Migration project.
• If app running in OTA_1, perform OTA again to change to OTA_0.
• Restart

Stage 1

• Copy binary from OTA_0 to the new OTA_1.
• Running in OTA_0, change the partition table to the intermediate table.
• Change the bootloader
• Set the boot to OTA_1
• Reboot

Stage 2

• Before continuing, check if the operation bellow has already been performed
• Running in OTA_1, Move the otadata and phy_init partitions to the space corresponding to the new partition table
• Clear the space of the NVS partition corresponding to the old partition table
• Change the partition table to the new table.
• Restart

Stage 3

• Perform OTA of the Final project in OTA_0
• Restart

I needed to change the 2º stage bootloader because some settings in my project changed from the old version to the new one.