Multi-Frame Anchor Support for WanImageToVideoSVIPro

[diodiogod]

Pull Request: Multi-Frame Anchor Support for WanImageToVideoSVIPro

Background & Exploration

I started exploring whether the anchor_samples in WanImageToVideoSVIPro could accept multiple images as references, similar to how IP-Adapter works for image generation. Through testing with Claude Code, I discovered that:

1. SVI Pro LoRAs appear to use only the first frame of the anchor latent as the reference (at least with anchor_frame_count=1)
2. However, feeding a multi-frame anchor latent works - the model accepts it without errors
3. Using 1+N image anchors (1 reference image + N starting frames) creates an IP-Adapter-like effect where the reference influences the generated video's appearance

While experimenting with the anchor_frame_count parameter to mask multiple frames, I found that masking only the first frame (default behavior) works best. Setting anchor_frame_count > 1 causes blending artifacts where multiple anchor frames visually merge in the output.

The sweet spot: Using a multi-frame anchor with anchor_frame_count=1 (mask only first frame) allows the reference image to influence generation without artifacts.

What This PR Fixes

Critical Bug: Hardcoded Batch Size

The original WanImageToVideoSVIPro had hardcoded batch_size=1 assumptions that caused dimension mismatch errors when trying to use multi-frame anchors:

Before:

padding = torch.zeros(1, C, padding_size, H, W, dtype=dtype, device=device)  # ❌ Hardcoded
mask = torch.ones((1, 1, empty_latent.shape[2], H, W), device=device, dtype=dtype)  # ❌ Hardcoded

After:

padding = torch.zeros(B, C, padding_size, H, W, dtype=dtype, device=device)  # ✅ Dynamic
mask = torch.ones((B, 1, image_cond_latent.shape[2], H, W), device=device, dtype=dtype)  # ✅ Dynamic

New Feature: anchor_frame_count Parameter

• Default: 1 (recommended - standard SVI Pro behavior)
• Purpose: Controls how many frames from anchor_samples are marked as anchor in the mask
• Remaining frames become starting frames when prev_samples is not connected
• Values > 1 are experimental and may cause blending artifacts

New Utility: BatchLatentToWanVideoLatent

The second issue was no way to properly format multi-image latents for use as anchors.

The Problem:

2 images → Wan VAE Encode → [2, C, T=1, H, W]  (2 separate single-frame latents)
                              ↓
              WanImageToVideoSVIPro ❌ ERROR: Creates 2 videos, not multi-frame anchor

The Solution:

[ref, start, start, start, start] → Batch (5 images)
                                        ↓
                            Standard VAE Encode → [5, C, H, W]
                                        ↓
                        BatchLatentToWanVideoLatent  ← Converts batch→temporal
                                        ↓
                                 [1, C, 5, H, W]
                                        ↓
                      WanImageToVideoSVIPro ✓ Works! 1+4 anchor

Node Features:

• Simple batch→temporal dimension conversion: [B, C, H, W] → [1, C, B, H, W]
• No complex parameters - just does one thing well
• Works with any VAE (SD, SDXL, Flux, etc.)
• Users control frame count by repeating images in the batch before encoding

Use Cases & Testing

Reference-Based Video Generation (IP-Adapter-like)

Using a 1+N configuration (1 reference + N repeated starting frames):

Step 1: Batch images - [ref_img, start_img, start_img, start_img, start_img]
                                         ↓
Step 2: Standard VAE Encode → [5, C, H, W]
                                         ↓
Step 3: BatchLatentToWanVideoLatent → [1, C, 5, H, W]
                                         ↓
Step 4: WanImageToVideoSVIPro (anchor_samples)
                                         ↓
                Video with reference characteristics applied

I will post video examples demonstrating:

• 1+4 frames (reference + 4 start frames)
• 1+8 frames
• 1+12 frames
• 1+20 frames
• 1+40 frames

Each configuration shows different levels of reference influence on the generated video.

Needs more testing but could be good for:

• ✅ Clothing transfer from reference image
• ✅ Style influence from reference
• ✅ Face/character consistency

Known Limitations & Workarounds:

⚠️ First frame color mismatch: The generated video's first frame doesn't perfectly match the starting image's colors due to reference influence. The reference image's colors "bleed through" affecting the overall color palette.

• The amount of color mismatch MAY depend on the 1+N ratio (needs more testing) - Lower N (e.g., 1+4) appears to cause more noticeable color bleeding than higher N (e.g., 1+20, 1+40)
• Impact varies based on reference-to-start color difference - Similar colored images have less mismatch; drastically different colors show more bleeding
• Overall the effect is subtle in most cases
• ⚠️ High N values (1+40 and above) may hinder movement - Very high ratios seem to result in more static videos with less motion (requires more experimentation to confirm)

Workarounds:

1. Use ColorMatch node - Apply color matching between reference and starting image before encoding

2. Crude montage technique - Manually composite elements from reference onto the starting frame:

   • Paste clothing from reference onto starting image
   • Paste face from reference onto starting image
   • This gives more control over what gets transferred
   • Examples will be posted later

   Example:

⚠️ Color contamination: Some color bleeding from reference to generated frames is expected - this is the trade-off for reference control.

Multi-Frame Anchor Experiments

Tried using anchor_frame_count > 1 to mask multiple frames as anchors:

• Result: Causes visible blending artifacts where anchor frames merge
• Recommendation: Keep anchor_frame_count=1 for best results

Relation to PR #474

This PR shares one fix with PR #474 (by @siraxe):

• Both fix: The hardcoded batch_size=1 bug in padding/mask creation
• PR WanImageToVideoSVIPro with end_samples #474 adds: End-frame blending feature (end_frame_latent, end_frame_fill, end_frame_max_strength)
• This PR adds:
  • anchor_frame_count parameter for controllable multi-frame masking
  • BatchLatentToWanVideoLatent utility node
  • Negative padding edge case handling

The fixes are compatible and could potentially be merged together.

Technical Details

Batch-to-Temporal Conversion

Wan VAE batch encoding creates: [B, C, T=1, H, W] where each item is a separate single-frame video.

For multi-frame anchors, we need: [1, C, T=B, H, W] where frames are in temporal dimension.

The BatchLatentToWanVideoLatent node:

1. Takes batched latents [B, C, H, W] from standard VAE
2. Converts batch dimension to temporal dimension
3. Outputs [1, C, B, H, W] video format

Simple and straightforward - users control frame distribution by arranging images in the batch before encoding.

Files Changed

1. nodes/nodes.py:

   • Fixed WanImageToVideoSVIPro batch size hardcoding
   • Added anchor_frame_count parameter
   • Added negative padding handling
   • Added BatchLatentToWanVideoLatent node (new)

2. __init__.py:

   • Registered BatchLatentToWanVideoLatent

Backwards Compatibility

✅ Fully backwards compatible

• Original single-frame anchor usage unchanged (default anchor_frame_count=1)
• Fixes only affect edge cases that were previously broken
• All new functionality is additive

Example Workflow

Before (Broken):

2 Images → Batch → Wan VAE Encode → [2, C, 1, H, W]
                                          ↓
                            WanImageToVideoSVIPro ❌ Dimension mismatch error

After (Fixed):

[ref, start, start, start, start] → Batch → Standard VAE Encode → [5, C, H, W]
                                                                        ↓
                                                    BatchLatentToWanVideoLatent
                                                                        ↓
                                                                 [1, C, 5, H, W]
                                                                        ↓
                                                      WanImageToVideoSVIPro ✓ Works!
                                                                        ↓
                                                  Video with reference influence (1+4)

Checklist

• Fixes actual bugs (hardcoded batch size)
• Adds missing functionality (multi-frame anchor support)
• Backwards compatible
• Input validation and error messages
• Console debug output
• Comprehensive node descriptions
• Tested with multiple configurations
• Will post video examples of different frame configurations

Notes for Reviewers

The core bug fix is simple (changing 1 to B), but it enables a previously impossible workflow. The BatchLatentToWanVideoLatent node is the proper way to format multi-image latents for this use case.

The reference-based generation technique creates an IP-Adapter-like effect for video generation, which is particularly useful since Wan models don't have native IP-Adapter support (? I think, we have wan fun vace, but it's a full other model, while this is a lora).

Note on anchor_frame_count: While the parameter allows experimental multi-frame masking, testing shows that keeping it at 1 (default) produces the best results. Values > 1 cause blending artifacts.

---

cc: @kijai (original author of WanImageToVideoSVIPro)
cc: @siraxe (PR #474 - related batch size fix)

[diodiogod]

AnimateDiff_00070.mp4

[diodiogod]

Workflow for reference, its a big workflow, it has many custom nodes, many nodes that if from forks I did... anyway. Just for reference.
wan2.2-continuos-svi-latent-2ksampler-fixed diog v0.2.json