improvements to esp32 build configuration

[UncleGrumpy]

These changes include several enhancements to the ESP32 build configuration and deployment of custom and test builds. The language flavor is determined by the offset of the main.avm partition in the partition table, simplifying Elixir supported builds and making the configuration of the image less error prone. The configuration of an Elixir build may be done without altering git tracked files by using idf.py -DAVM_ELIXIR_SUPPORTT=on set-target ${CHIP}. This is the equivalent of setting the custom partition file name to partitions-elixir.csv using idf.py menuconfig. Other existing AtomVM configuration options have been added to main/Kconfig.projbuild and are exposed in idf.py menuconfig under the AtomVM menu to make the configuration process more natural for users already familiar with ESP-IDF and its build and configuration system.

The auto discovered flavor of the esp32boot (boot.avm partition) libraries are now included in the idf.py flash task. This makes use of the build/mkimage.sh script only necessary for building release images in the CI or to create a complete image for future use, and makes the experience like most other ESP-IDF based projects. To flash only the factory (AtomVM) partition idf.py app-flash may be used or idf.py bootloader-flash app-flash if changes made affect the bootloader partition as well.

Changes the build configuration for reproducible builds, this strips leading path information (which is replaced by placeholders), and allows locally building from a release tag and generating a binary with the same sha256 checksum as official release binaries (as long as ESP-IDF and toolchain versions match those used to build the release). See:
https://docs.espressif.com/projects/esp-idf/en/v5.5/esp32/api-guides/reproducible-builds.html

The nvs partition may now be generated and included in the build/mkimage.sh script (or flashed manually using the idf.py nvs-flash command) if the file nvs_partition.csv is present in the top level esp32 platform directory. An example is provided and nvs_partition.csv is added to .gitignore to prevent accidental pushes of sensitive passwords.

Closes atomvm/exatomvm#34

These changes are made under both the "Apache 2.0" and the "GNU Lesser General
Public License 2.1 or later" license terms (dual license).

SPDX-License-Identifier: Apache-2.0 OR LGPL-2.1-or-later

[petermm]

Not sure I like this bundling of commands:

1. Imho it's a coupling, implicit behaviour change - which might hinder future things, like point 2..

2. In a simulator flow I use: idf.py build && touch ~/path_to_project/atomvm_wokwi_bins/flasher_args.json (as to build and then reboot sim) - and having mkimage in there would slow things down.

But I'm also a poweruser of idf.py build && ./build/mkimage.sh - so I get the drift, but not sure I understand how it simplifies deploying custom images..

[bettio]

I'm not sure if it is an addition that fills some kind of huge gap in v0.6 (or not), so regardless to the outcome of this discussion I would target main branch anyway.

[UncleGrumpy]

@petermm, I don’t want to disrupt anyone’s workflows. My goal was just to streamline the process for newer users who need to build a custom image to include additional components, like atomvm_mqtt. It makes more sense for me to add an additional cmake job called “deploy” (or similar) that will build the complete image and flash it to the target device using the generated ./build/flasjimage.sh script, leaving the default “build” task unchanged, this will better my goal and should have no affect on any pre-existing workflows.

[pguyot]

Should we keep a way to not create the image? (it's not much longer anyway). When working on the VM, I usually only reflash with idf.py flash which is faster than flashing the whole image.

Also, one of my workflows (la machine, obviously) does not include esp32boot.avm at all because it seems to me that esp32boot.avm doesn't bring anything to production images. I'll definitely work around this improvement for newcomers, though.

Likewise, qemu-based tests only use artifacts created by idf.py build (the VM, the partition map and the bootloader) and not the full image.

[UncleGrumpy]

Should we keep a way to not create the image?

I converted this back to draft, because I realize after @petermm’s comments that could be intrusive on existing workflows, so instead I will create and document a new build task that can be used as an alternative to idf.py build that will create the ready to deploy image, or even flash the image after building so that flashing a new custom image can be achieved with one command.

[pguyot]

Should we keep a way to not create the image?

I converted this back to draft, because I realize after @petermm’s comments that could be intrusive on existing workflows, so instead I will create and document a new build task that can be used as an alternative to idf.py build that will create the ready to deploy image, or even flash the image after building so that flashing a new custom image can be achieved with one command.

This sounds like a better approach indeed and would be very welcome. I also find it frustrating to have to call the bash script to build the image when I need it.

[UncleGrumpy]

Should we keep a way to not create the image? (it's not much longer anyway). When working on the VM, I usually only reflash with idf.py flash which is faster than flashing the whole image.

I backed out of assembling the complete image as part of the build process, and it is no longer needed for most users. Instead I added flashing the correct flavor of esp32boot.avm to the idf.py flash task. Images can still be created for releases, or if users want to generate a portable image for future use.

In the case of reflashing during development work idf.py app-flash may be used to reflash only the AtomVM binary to the factory partiton. If flashing an updated bootloader is required, this may be accomplished using idf.py bootloader-flash (or combined with app-flash if necessary). This should actually save you a little more time since this does not include the partition map (which is unlikely to be changed often) that is also included when using idf.py flash.

[petermm]

I'm confused by the naming of app-flash, and what it actually does..
avoid re-flashing the entire image if no changes were made to the Erlang or Elixir libraries

[UncleGrumpy]

The app-flash is an IDF included task, I did not have a choice there. It only flashed the esp-idf application - in this case that is the AtomVM binary. Perhaps I should explain the name of this task in the documentation, and that it is a native IDF task.

[petermm]

gotcha, but most unfortunate name in our context, almost feels like we should add this to cmakelists:

add_custom_target(libatomvm-flash)
add_dependencies(libatomvm-flash app-flash)

and maybe add eavmlib/exavmlib to this sentence..
avoid re-flashing the entire image if no changes were made to the Erlang(eavmlib) or Elixir (exavmlib) libraries

I'm assuming app-flash flashes libatomvm..

[UncleGrumpy]

It refers to the esp-idf application, app-flash means flash build/atomvm-{target-chip}.bin, which is the AtomVM executable with libatomvm statically linked in, not the atomvmlib.avm packed BEAM files. To flash those the documentation already describes using build/flash.sh -l ../../../build/libs/esp32boot/esp32boot.avm.

[UncleGrumpy]

To put it another way, it is the contents of the factory partition.

I think when you think of it in terms of ESP-IDF app-flash is a very natural name for it. It just doesn't match an application in terms of AtomVM, but I don't think anyone is expecting to flash their beam application with esp-idf tools.

[UncleGrumpy]

I updated this section, hopefully it is more clear now.

[petermm]

This is great.

1. I'm confused by the app-flash naming and meaning, is it flashing the runtime/vm? should it be named runtime-flash or vm-flash?

2. I believe, that doing a set-target, automatically includes the reconfigure, eg. idf.py set-target esp32 reconfigure doesn't need a reconfigure, and it wastes time/cpu - in general the reconfigure is kinda confusing as it removes your changes in menuconfig.. for beginners this is not obvious, my point being not having a single mention of reconfigure might be beneficial.

[petermm]

also the simtest/wokwi ci will need some minor adjustment..

[UncleGrumpy]

also the simtest/wokwi ci will need some minor adjustment..

We don’t test Elixir in the simtest, and it has its own sdkconfig.defaults, so I don’t see any chances that need to be made… what did I miss?

Also note that the old method of copying the partitions-elixir.csv to partitions.csv still works, so if Elixir tests are added to simtest that is a viable option for the CI to use.

[UncleGrumpy]

2. I believe, that doing a set-target, automatically includes the reconfigure, eg. idf.py set-target esp32 reconfigure doesn't need a reconfigure, and it wastes time/cpu - in general the reconfigure is kinda confusing as it removes your changes in menuconfig.. for beginners this is not obvious, my point being not having a single mention of reconfigure might be beneficial.

You are absolutely correct. Thank you! Not sure why I was keeping the reconfigure task. Simply set-target and include the Elixir support option is much cleaner and faster.

[petermm]

This is awesome, and smooths out a LOT of friction..

The idf.py app-flash is still confusing to me, the name it self, as well as the description of it. eg. AtomVM binary (which doesn't seem to align with docs, which include bootloader, nvs, boot.avm in that term

AtomVM/doc/src/build-instructions.md

Line 432 in 21671de

| boot.avm | 256-512KB v

)

I suggest aliasing the app-flash name, and aligning the usage of AtomVM binary, to whatever is deemed correct. Maybe call it platform binary, or factory binary, esp32 factory binary etc.

add_custom_target(platform-flash)
add_dependencies(platform-flash app-flash)

(or factory-flash? instead of platform-flash)

thinking of it maybe alias it to factory-flash and call it factory binary

But let's also ship this asap - it's great!

[UncleGrumpy]

The app-flash command is an esp-idf built in that will only flash the project binary. I was only pointing out its existence for anyone who still wanted to use something closer to the original behavior of only flashing the project app (AtomVM binary) and bootloader. If you need both (typically the bootloader would only need to be reflashed if a few specific Kconfig options were changed) this would now be a 2 step process.

[UncleGrumpy]

I don't think most users will be needing to use these fine grained flash commands. The majority are just wanting to use main branch, or build a custom image with a downstream driver included. For developers already familiar with ESP-IDF there is no point in shadowing a standard command with a custom alias, that seems like it could potentially add more confusion. If this was a custom command I added I certainly would have chosen a different name 😉

[petermm]

suing -> using

[petermm]

missing space: theidf.py flash

[petermm]

reproducable -> reproducible