Support Video Pipeline

[szw0407]

Is it possible to create a pipeline for models like SmolVLM2-256M-Video-Instruct which allows video input? Thanks for your reply.

[szw0407]

#103

I am working. It should be open in a few days.

[netdur]

Thanks @szw0407 for putting this together, and for surfacing SmolVLM2-Video it's a model worth supporting

I dug into it against our pinned llama.cpp (b9360), and unfortunately the central assumption here doesn't hold: libmtmd has no video decoder. mtmd_helper_bitmap_init_from_buf only does two things, decode audio (miniaudio) or decode a still image (stbi_load_from_memory, i.e. PNG/JPG/BMP/GIF). There's no mp4/mov/webm path, and the source even marks it as future work:

// clip.cpp:1651
// TODO @ngxson : support both audio and video in the future
Concretely, that means:

Passing raw video bytes hits the image decoder, which returns null → the existing evalChunks throws.
mtmd_context_params only has image_min/max_tokens, there's no field for the new videoMin/MaxTokens, so those params are inert
The good news: these models consume video as a sequence of image frames, and that path already works in the binding today (multiple <media> markers ↔ multiple image bitmaps) I added a small helper + example and verified it end-to-end against SmolVLM2-256M-Video-Instruct on a real clip:

LlamaMedia.videoFrames(frames) — wraps pre-extracted frame bytes as one image media item each.
example/probes/video_describe.dart — extracts frames with ffmpeg, then feeds them through the normal multimodal chat path (EngineChat.addUser adds one marker per frame automatically).

final media = LlamaMedia.videoFrames(frames);
chat.addUser('Describe this video.', media: media);

Frame extraction has to live outside libmtmd (ffmpeg or a platform decoder, a real native-dependency call, especially on mobile), so I kept the library decoder-free and put the ffmpeg step in the example

[szw0407]

I understand and thanks for your reply!