Skip to content

Support multiple box prompts in video inference - #615

Open
oValach wants to merge 1 commit into
facebookresearch:mainfrom
oValach:support-multi-box-video-prompts
Open

Support multiple box prompts in video inference#615
oValach wants to merge 1 commit into
facebookresearch:mainfrom
oValach:support-multi-box-video-prompts

Conversation

@oValach

@oValach oValach commented Aug 17, 2026

Copy link
Copy Markdown

Summary

Sam3VideoInference.add_prompt() accepts box inputs with shape N x 4, but the current video inference path requires an initial visual prompt to contain exactly one box.

This change allows multiple initial box prompts to be packed into a single Prompt, while preserving the existing single-box and refinement behavior.

The multi-box representation uses the existing Prompt.append_boxes() path also used by the multiplex video implementation.

Motivation

Detector-driven video pipelines commonly produce multiple boxes for the same frame. With the current API, these boxes need to be submitted individually, which results in one single-frame inference call per box.

Supporting multiple boxes in one prompt allows all box prompts from a frame to be submitted together and processed in a single inference call.

Changes

  • Support multiple initial box prompts in Sam3VideoInference.
  • Preserve the existing single-box path unchanged.
  • Preserve existing behavior when a visual prompt already exists.
  • Add unit tests covering:
    • single-box visual prompts,
    • multiple boxes packed into one prompt,
    • existing visual-prompt/refinement behavior.

Validation

Tested with:

python test/test_video_inference.py -v
python test/test_video_predictor.py -v

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant