video: introduction of STM32 VENC driver for H264 hardware video compression #92884
Conversation
|
The following west manifest projects have changed revision in this Pull Request:
⛔ DNM label due to: 1 project with PR revision Note: This message is automatically posted and updated by the Manifest GitHub Action. |
|
There was a problem hiding this comment.
Thank you for preparing VENC!
This will really make video over network on Zephyr interesting.
It seems like Zephyr lacks a few APIs from Linux (no H.264 format definition, no planar API for NV12...), so if anything feels like missing, let us know so that the missing API can be provided in parallel.
drivers/video/video_stm32_venc.c
Outdated
| input = k_fifo_get(&data->fifo_input, K_NO_WAIT); | ||
| if (!input) | ||
| return 0; | ||
| LOG_DBG("input=%p\n", input); |
There was a problem hiding this comment.
In the meantime that a better buffering model is worked on, the way to video_buffer_free() a buffer is by giving it back to the application which video_dequeue() it and reuse it in the next cycle, for instance via data->fifo_input_out (in addition to a data->fifo_input_in).
If encountering errors where running out of buffers hopefully this can help.
There was a problem hiding this comment.
I don't get the point sorry. If no input buffer is there we silently exit and we will retry at next encoding, do you see a problem around that ?
drivers/video/video_stm32_venc.c
Outdated
| } | ||
|
|
||
| data->pixelformat = fmt->pixelformat; | ||
| data->pitch = fmt->pitch; |
There was a problem hiding this comment.
Following these changes, when the application sets the format, the driver sets the ->pitch field.
But for H.264 the pitch might as well be 0 as a convention for variable data size...
zephyr/include/zephyr/drivers/video.h
Lines 1638 to 1647 in 9219c81
There was a problem hiding this comment.
pitch is used by tcpserversink application to allocate buffers, see
No specific processing is necessary currently as part of the stm32_dcmipp_isp_start function hence remove the assert(0) currently preventing stop of the isp. Signed-off-by: Alain Volmat <[email protected]>
Allow to call HAL_DCMIPP_PIPE_SetConfig when the pipe state is READY. Currently it is only possible to use this function if the pipe state is RESET and ERROR states. Indeed, it is possible to reconfigure the PIPE as soon as the pipe is not currently being used, aka in the READY state. With that done, it becomes possible to change the PIPE configuration between two use-cases without having to DeInit / Init the HAL_DCMIPP or use the unitary pixel packer functions. Signed-off-by: Alain Volmat <[email protected]>
There was a problem hiding this comment.
First round of comment, driver aside.
Please split the PR and do a specific PR for the first commits that are not directly linked to the VENC driver. These can be reviewed and integrated faster.
For the changes on the sample, I let video team decide if this should be kept within the tcpserver sample or split into a different sample
hfruchet-st
force-pushed
the
iso_stm32n6_1080p_h264_streaming
branch
from
a32b1da to
deac1fb
Compare
zephyrbot
requested review from
djiatsaf-st,
etienne-lms,
FRASTM,
gautierg-st,
GeorgeCGV,
loicpoulain and
marwaiehm-st
github-actions
bot
added
manifest-hal_stm32
DNM (manifest)
hfruchet-st
force-pushed
the
iso_stm32n6_1080p_h264_streaming
branch
2 times, most recently
from
99e9938 to
2f05a7c
Compare
hfruchet-st
force-pushed
the
iso_stm32n6_1080p_h264_streaming
branch
from
2f05a7c to
4422502
Compare
hfruchet-st
force-pushed
the
iso_stm32n6_1080p_h264_streaming
branch
from
4422502 to
a0f000d
Compare
There was a problem hiding this comment.
SonarQube issues that need attention
https://sonarcloud.io/project/issues?id=zephyrproject-rtos_zephyr&pullRequest=92884&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true
The first two are incorrect because zephyr/drivers/video/video_stm32_venc.c Lines 137 to 151 in a0f000d The third one is the opposite: API takes an opaque |
| k_sem_init(&ewl_instance.complete, 0, 1); | ||
| k_sem_reset(&ewl_instance.complete); |
There was a problem hiding this comment.
Is the k_sem_init plus k_sem_reset required because the instance may get recycled or something? Usually, a single k_sem_init is enough.
|
|
||
| uint32_t mem_counter; | ||
|
|
||
| typedef struct { |
There was a problem hiding this comment.
Is a typedef necessary? struct venc_ewl should work and would better suit the snake case coding guideline.
There was a problem hiding this comment.
It's a type declared in the Encoder Wrapping Layer and needs to be defined.
There was a problem hiding this comment.
Do you refer to ewl_impl.h. I don't see the same structure from that header file. It seems to me that this current structure is internal to this driver and passed back and forth between this driver and the EWL as an opaque structure reference/pointer.
| uint32_t mem_counter; | ||
|
|
||
| typedef struct { | ||
| uint32_t clientType; |
There was a problem hiding this comment.
Ditto about snake case for structure fields, prefer:
uint32_t client_type;
uint32_t *chunks[MEM_CHUNKS];
uint32_t *aligned_chunks[MEM_CHUNKS];
uint32_t total_chunks;There was a problem hiding this comment.
field name is defined by Encoder Wrapping Layer
| static int encoder_end(struct video_stm32_venc_data *data); | ||
| static int encoder_start(struct video_stm32_venc_data *data); | ||
|
|
||
| static inline H264EncPictureType to_h264pixfmt(uint32_t pixelformat) |
There was a problem hiding this comment.
You can can drop the inline keyword.
| #include "ewl.h" | ||
| #include "h264encapi.h" | ||
| #include "reg_offset_v7.h" |
There was a problem hiding this comment.
These include file are found from the header path, not locally.
Prefer #include <...>.
(edited) May be addressed in a later P-R.
| return -ENODATA; | ||
| } | ||
| output->bytesused = encOut.streamSize; | ||
| break; |
There was a problem hiding this comment.
Indentation
| break; | |
| break; |
Ditto below.
| data->resync = true; | ||
|
|
||
| return -EIO; | ||
| break; |
There was a problem hiding this comment.
Can remove the break instruction.
| /* Stop VENC */ | ||
| encoder_end(data); | ||
|
|
||
| return 0; |
There was a problem hiding this comment.
Remove this instruction?
| inst->irq_status = irq_status; | ||
| inst->irq_cnt++; | ||
|
|
||
| if (!hw_handshake_status && (irq_status & ASIC_STATUS_FUSE)) { |
There was a problem hiding this comment.
| if (!hw_handshake_status && (irq_status & ASIC_STATUS_FUSE)) { | |
| if (hw_handshake_status == 0 && (irq_status & ASIC_STATUS_FUSE) != 0) { |
(edited) May be addressed in a later P-R.
| static void video_stm32_venc_irq_config_func(const struct device *dev) | ||
| { | ||
| IRQ_DIRECT_CONNECT(DT_INST_IRQN(0), DT_INST_IRQ(0, priority), | ||
| stm32_venc_isr, 0); |
hfruchet-st
force-pushed
the
iso_stm32n6_1080p_h264_streaming
branch
from
a0f000d to
107c233
Compare
zephyrbot
added
area: Tests
hfruchet-st
force-pushed
the
iso_stm32n6_1080p_h264_streaming
branch
from
107c233 to
6a03a6d
Compare
Pull hal_stm32 vc8000nanoe library. Signed-off-by: Hugues Fruchet <[email protected]>
Addition of description for the STM32 Video encoder (VENC). Signed-off-by: Hugues Fruchet <[email protected]>
hfruchet-st
force-pushed
the
iso_stm32n6_1080p_h264_streaming
branch
from
6a03a6d to
ac3fb6b
Compare
The STM32 video encoder (VENC) peripheral is a hardware accelerator allowing to compress RGB/YUV frames into H264 video bitstream chunks. Signed-off-by: Hugues Fruchet <[email protected]>
Add node describing the venc in stm32n6.dtsi Signed-off-by: Hugues Fruchet <[email protected]>
hfruchet-st
force-pushed
the
iso_stm32n6_1080p_h264_streaming
branch
from
ac3fb6b to
ae105ff
Compare
| #define VIDEO_PIX_FMT_H264_NO_SC VIDEO_FOURCC('A', 'V', 'C', '1') | ||
|
|
||
| /** | ||
| * H264 without start code |
There was a problem hiding this comment.
| * H264 without start code | |
| * H264 MVC video elementary stream. |
|
This PR introduces H264 hardware video compression thanks to VENC peripheral of STM32N6.
A new stm32 video encoder driver has been introduced which relies on vc8000nanoe software stack [1].
This is a first basic porting to enable H264 encoding use-case.
RAM memory resources required by VENC encoding process are allocated in external PSRAM, camera frames
captured by DCMIPP and encoded by VENC are also allocated in external PSRAM.
The video encoder driver currently lack of controls to configure the encoding process, default configuration
targets H264 1080p 2Mb/s constant bitrate.
In order to test, the tcpserversink video sample has been enhanced to support video compression in order to stream small H264 compressed video chunks instead of big raw uncompressed images [2].
[1] zephyrproject-rtos/hal_stm32#295
[2] #95862