This project provides an STM32 microcontroller embedded real-time environment to execute STEdgeAI generated models targeting instance segmentation application. The code developement is lead by its understandability rather than its performance, and is intended to be used as a starting point for further development.
Detected classes are displayed on the bounding boxes. The instance segmentation is displayed in overlay.
This is a standalone project that can be used as-is to deploy the application on the hardware. However, it is also part of the ST ModelZoo repo and is needed to deploy the instance segmentation use case. The ModelZoo allows you to train, evaluate and deploy automatically any supported model. If you wish to use it as part of the ModelZoo, please refer to the Quickstart using stm32ai-modelzoo-services section on how to use it.
This top readme gives an overview of the app. Additional documentation is available in the Doc folder.
This repository is structured as follows:
| Directory | Content |
|---|---|
| Application/<board_name>/Inc | Application include files |
| Application/<board_name>/Src | Application source files |
| Application/Model/* | Placeholder for AI C-model; files generated by cubeAI |
| STM32Cube_FW_N6/Drivers/CMSIS | CMSIS Drivers |
| STM32Cube_FW_N6/Drivers/BSP | Board Support Package and Drivers |
| STM32Cube_FW_N6/Drivers/STM32XXxx_HAL_Driver | Hardware Abstraction Layer for STM32XXxx family products |
| Middlewares/AI_Runtime | Placeholder for AI runtime library |
| Middlewares/Camera_Middleware | Middleware to ease camera configuration |
| Middlewares/lib_vision_models_pp | Computer vision models post processing |
README Content
- Features demonstrated in this example
- Hardware Support
- Tools version
- Boot modes
- Quickstart using stm32ai-modelzoo-services
- Quickstart using prebuilt binaries
- Quickstart using source code
- Known Issues and Limitations
Doc Folder Content
- Application overview
- Boot Overview
- Camera build options
- Camera Orientation
- Aspect Ratio mode
- Deploy your tflite model on STM32N6570-DK
- Deploy your tflite model on NUCLEO-N657X0-Q
- Program Hex Files STM32CubeProgrammer
- Sequential application flow
- NPU accelerated quantized AI model inference
- Dual DCMIPP pipes
- DCMIPP crop, decimation, downscale
- DCMIPP ISP usage
- LTDC dual-layer implementation
- Dev mode
- Boot from External Flash
One of the following development platforms:
- STM32N6570-DK discovery board
- The board should be connected to the onboard ST-LINK debug adapter CN6 with a USB-C to USB-C cable to ensure sufficient power
- Set the boot mode in dev mode (Both switches to the right side, (cf Boot modes of STM32N6 reference manual))
- OTP fuses are set in this example for xSPI IOs in order to get the maximum speed (200MHz) on xSPI interfaces
- NUCLEO-N657X0-Q Nucleo board
- The board should be connected to the onboard ST-LINK debug adapter CN9 with a USB-C to USB-C cable to ensure sufficient power
- Set the boot mode in dev mode (BOOT0 JP1 on position 1 and BOOT1 JP2 on position 2, (cf Boot modes of STM32N6 reference manual))
- OTP fuses are set in this example for xSPI IOs in order to get the maximum speed (200MHz) on xSPI interfaces
And one of the following camera modules:
- Provided IMX335 camera module
- STEVAL-55G1MBI
- STEVAL-66GYMAI1
For the Nucleo board, you will need one of the following display:
- An USB host to which data will be transmitted via USB/UVC (using the USB OTG port)
- An X-NUCLEO-GFX01M2 SPI display
STM32N6570-DK board with MB1854B IMX335.
- STM32CubeIDE (STM32CubeIDE 1.17.0)
- STM32CubeProgrammer (v2.18.0)
- STEdgeAI (v2.1.0)
The STM32N6 does not have any internal flash. To retain your firmware after a reboot, you must program it in the external flash. Alternatively, you can load your firmware directly from SRAM (dev mode). However, in dev mode if you turn off the board, your program will be lost.
Boot modes:
- Dev mode (STM32N6570-DK: both boot switches to the right, NUCLEO-N657X0-Q: BOOT0 JP1 in position 1, BOOT1 JP2 in position 2): used to load the firmware from debug session in RAM, or program firmware in external flash
- Boot from flash (STM32N6570-DK: both boot switches to the left, NUCLEO-N657X0-Q: BOOT0 JP1 in position 1, BOOT1 JP2 in position 1): used to boot the firmware in external flash
This application is a C-based project needed by the deployment service in the ModelZoo ModelZoo. The ModelZoo enables you to train, evaluate, and automatically deploy any supported model.
To deploy your model using the ModelZoo, refer to the Deployment README for STM32N6 for detailed instructions on deploying to either the STM32N6570-DK or the NUCLEO-N657X0-Q.
Note: This C-Based application is already in the ModelZoo repo in the STM32N6 instance segmentation application code folder.
Three binaries must be programmed in the board external flash using the following procedure:
- Set your board in dev mode
- Program
Binary/ai_fsbl.hex(First Stage Boot Loader) - Program
Binary/STM32N6570-DK_network_data.hexorBinary/NUCLEO-N657X0-Q_network_data.hex(params of the networks; To be changed only when the network is changed) - Program
Binary/STM32N6570-DK_GettingStarted_InstanceSegmentation.hexorBinary/NUCLEO-N657X0-Q_GettingStarted_InstanceSegmentation.hexdepending on the board you use (firmware application) - Set your board in boot from flash mode
- Power cycle the board
Note: The Binary/STM32N6570-DK_GettingStarted_InstanceSegmentation.hex firmware is built for MB1939 STM32N6570-DK REV C02 with any of the listed required camera module.
Note: The Binary/NUCLEO-N657X0-Q_GettingStarted_InstanceSegmentation.hex firmware is built for MB1940 NUCLEO-N657X0-Q REV C01 with USB/UVC host display and any of the listed required camera module.
See How to program hex files STM32CubeProgrammer
Make sure to have the STM32CubeProgrammer bin folder added in your path.
export DKEL="<STM32CubeProgrammer_N6 Install Folder>/bin/ExternalLoader/MX66UW1G45G_STM32N6570-DK.stldr"
# First Stage Boot loader
STM32_Programmer_CLI -c port=SWD mode=HOTPLUG -el $DKEL -hardRst -w Binary/ai_fsbl.hex
# Network parameters and biases
STM32_Programmer_CLI -c port=SWD mode=HOTPLUG -el $DKEL -hardRst -w Binary/STM32N6570-DK_network_data.hex
# Application Firmware
STM32_Programmer_CLI -c port=SWD mode=HOTPLUG -el $DKEL -hardRst -w Binary/STM32N6570-DK_GettingStarted_InstanceSegmentation.hexMake sure to have the STM32CubeProgrammer bin folder added in your path.
export NUEL="<STM32CubeProgrammer_N6 Install Folder>/bin/ExternalLoader/MX25UM51245G_STM32N6570-NUCLEO.stldr"
# First Stage Boot loader
STM32_Programmer_CLI -c port=SWD mode=HOTPLUG -el $NUEL -hardRst -w Binary/ai_fsbl.hex
# Network parameters and biases
STM32_Programmer_CLI -c port=SWD mode=HOTPLUG -el $NUEL -hardRst -w Binary/NUCLEO-N657X0-Q_network_data.hex
# Application Firmware
STM32_Programmer_CLI -c port=SWD mode=HOTPLUG -el $NUEL -hardRst -w Binary/NUCLEO-N657X0-Q_GettingStarted_InstanceSegmentation.hexBefore building and running the application you have to program <board_name>_network_data.hex (model weights and biases).
This step only has to be done once unless you change AI model.
See Quickstart using prebuilt binaries for details.
More information about boot modes is available at Boot Overview
Go to the ST Ultralytics fork and download Yolo v8 seg 256x256, or clic on this link to download directly the model. Put the model in the Model/ folder and run the following command:
cd Model
generate-n6-model.sh
cd ..The Model/network.c and Model/network_data.hex files are being generated.
Set your board in dev mode.
Double click on Application/<board_name>/STM32CubeIDE/.project to open project in STM32CubeIDE. Build and run.
Go to Application/<board_name>/, and run the following commands (be sure to add them in your PATH before).
- Build project using the provided
Makefile:
make -j8- Open a gdb server connected to the STM32 target:
ST-LINK_gdbserver -p 61234 -l 1 -d -s -cp <path-to-stm32cubeprogramer-bin-dir> -m 1 -g- In a separate terminal launch a GDB session to load the firmware into the device memory:
$ arm-none-eabi-gdb build/Application/<board_name>/Project.elf
(gdb) target remote :61234
(gdb) monitor reset
(gdb) load
(gdb) continueSet your board in dev mode.
Double click on Application/<board_name>/STM32CubeIDE/.project to open project in STM32CubeIDE. Build and run.
Before running commands below be sure to have commands in your PATH.
Build project using the provided Makefile:
make -j8Once your app is built with Makefile, STM32CubeIDE, you can add a signature to the bin file:
STM32_SigningTool_CLI -bin build/Application/<board_name>/Project.bin -nk -t ssbl -hv 2.3 -o build/Application/<board_name>/Project_sign.binYou can program the signed bin file at the address 0x70100000.
On the STM32N6570-DK:
export DKEL="<STM32CubeProgrammer_N6 Install Folder>/bin/ExternalLoader/MX66UW1G45G_STM32N6570-DK.stldr"
# Adapt build path to your IDE
STM32_Programmer_CLI -c port=SWD mode=HOTPLUG -el $DKEL -hardRst -w build/Application/<board_name>/Project_sign.bin 0x70100000On the NUCLEO-N657X0-Q:
export NUEL="<STM32CubeProgrammer_N6 Install Folder>/bin/ExternalLoader/MX25UM51245G_STM32N6570-NUCLEO.stldr"
# Adapt build path to your IDE
STM32_Programmer_CLI -c port=SWD mode=HOTPLUG -el $NUEL -hardRst -w build/Application/<board_name>/Project_sign.bin 0x70100000Note: Only the app binary needs to be programmed if the fsbl and network_data.hex was previously programmed.
Set your board in boot from flash mode.
Do a power cycle to boot from the external flash.
- Only RGB888 format for nn input has been tested.
- Only UINT8 format for nn input is supported.