Skip to content

feat(models): add Phi-4 multimodal audio support - #525

Open
zitai-wang wants to merge 10 commits into
inclusionAI:mainfrom
zitai-wang:feat/phi4-audio-adapter
Open

feat(models): add Phi-4 multimodal audio support#525
zitai-wang wants to merge 10 commits into
inclusionAI:mainfrom
zitai-wang:feat/phi4-audio-adapter

Conversation

@zitai-wang

Copy link
Copy Markdown

Summary

  • add native Phi-4-Multimodal audio processing and placeholder expansion
  • implement the official causal Conformer audio encoder, Nemo Conv subsampling, T5 relative attention bias, and speech/vision projector branches
  • support Speech LoRA activation and independent recurrent state across chunked prefill and KV-cache decode
  • load and save the complete official audio tower, projector, buffers, and TP-aware Speech LoRA weights
  • preserve PR1 text and PR2 vision behavior, including mixed text/image/audio batches

This is a stacked PR based on feat/phi4-vision-adapter (PR2). After PR2 is merged, this branch should be rebased and the PR target adjusted to upstream/main.

Validation

  • official Phi-4-Multimodal checkpoint: 2047/2047 tensors consumed; missing=0, unknown=0, skipped=0
  • audio projector and merged embeddings: exact HF parity (max_abs=0)
  • last-token logits: cosine 0.9999907, matching top-1 and 5/5 top-5 overlap
  • greedy KV-cache generation: 8/8 token ids match HF; audio encoder runs once during prefill
  • real checkpoint TP1/TP2/TP4: matching 8-token greedy generation
  • multi-audio: different-length segments produce 44 + 22 matching placeholder slots
  • full save/reload: 2047/2047 tensors consumed, logits max_abs=0, matching generation
  • Phi/multimodal focused tests: 82 passed
  • repository suite excluding three confirmed upstream baseline failures: 686 passed, 3 skipped
  • available CUDA tests: 12 passed
  • pre-commit, Ruff lint/format, compileall, pip check, and git diff check passed

The three excluded repository failures reproduce on clean upstream/main@4cc3d38: one MLX log-capture assertion and two Gloo/NCCL tests that select duplicate CUDA devices in this environment.

Known limitations

  • inference support only; training and rollout reward/logprob integration remain out of scope
  • cached generation crossing the inherited Phi LongRoPE 4096-token boundary continues to fail fast

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant