`build_sam3_image_model(checkpoint_path=...)` can silently load nothing for fine-tuned checkpoints

[vayset]

Context

After fine-tuning SAM3 with the training pipeline described in README_TRAIN.md, the natural way to load the resulting checkpoint for inference is:

model = build_sam3_image_model(checkpoint_path="my_finetuned.pt", load_from_HF=False)

But _load_checkpoint (sam3/model_builder.py) only handles the official release key layout, and its failure mode is silent.

Problem 1: keys without a detector. prefix are all dropped

_load_checkpoint filters the state dict with if "detector" in k. A checkpoint saved directly from the image model (keys like backbone.…, transformer.…) is filtered down to an empty dict, and the subsequent load_state_dict(..., strict=False) succeeds without loading anything. The model silently keeps its random initialization and produces garbage predictions — there is no exception, only a print of missing keys that is easy to miss.

Problem 2: DDP module. prefixes are not stripped

Checkpoints saved from a DDP-wrapped model have keys like module.detector.backbone.…. They pass the "detector" in k filter, but k.replace("detector.", "") produces module.backbone.…, which matches nothing in the model. unexpected_keys is discarded (missing_keys, _ = …), so these dropped keys are never reported.

Minimal repro

import torch
from sam3.model_builder import build_sam3_image_model

# any checkpoint whose keys have no "detector." prefix, e.g. saved from the image model itself:
ckpt = {k: v for k, v in torch.load("sam3_finetuned.pt")["model"].items()}
torch.save(ckpt, "/tmp/plain.pt")
model = build_sam3_image_model(checkpoint_path="/tmp/plain.pt", load_from_HF=False)
# -> no error, but every weight is randomly initialized

Proposal

• Strip a leading module. prefix before filtering.
• If no detector.* key exists, fall back to using the keys as-is (image-model layout).
• Raise a clear ValueError when no checkpoint key matches the model state dict, instead of returning a randomly-initialized model.
• Report unexpected_keys alongside missing_keys.

This keeps the official checkpoints loading exactly as before. I have a small PR ready with these changes plus unit tests, will link it here.