[Looking for community contribution] support DiffSynth Controlnet in diffusers

[yiyixuxu]

Model/Pipeline/Scheduler description

Hi!
We want to add first party support for DiffSynth controlnet in diffusers, and we are looking for some help from the community!

Let me know if you're interested!

Open source status

• The model implementation is available.
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

https://huggingface.co/SahilCarterr/Qwen-Image-Blockwise-ControlNet-Canny
https://huggingface.co/SahilCarterr/Qwen-Image-Blockwise-ControlNet-Depth

[tanuj-rai]

Hey @yiyixuxu! I am interested in this issue.

[yiyixuxu]

@tanuj-rai look forward to your PR!

[Aishwarya0811]

Hi @yiyixuxu! 👋 I'm very interested in contributing the DiffSynth ControlNet support for Qwen-Image.

What I understand from the existing models:

• DiffSynth-Studio has already trained Qwen-Image ControlNet models (Canny, Depth, Inpaint) using a "Blockwise-ControlNet" architecture
• These models use blockwise_controlnet_inputs=[ControlNetInput(image=controlnet_image)] in their pipeline
• The current implementation requires DiffSynth-Studio's custom QwenImagePipeline

What needs to be implemented in diffusers:

1. QwenImageControlNet model class - following the pattern of FluxControlNet, SD3ControlNet
2. QwenImageControlNetPipeline - similar to existing ControlNet pipelines in diffusers
3. Support for the "blockwise" architecture used by DiffSynth models

Questions before I start:

• Should this follow the same API pattern as FluxControlNetPipeline and StableDiffusion3ControlNetPipeline?
• Are there any specific requirements for the blockwise ControlNet implementation?
• Should I start with one model type (e.g., Canny) and expand from there?

I'd be happy to take this on!

[MengAiDev]

LGTM! I love to work on this.

[akiseakusa]

Hi @ALL,

I’ve opened a draft PR for this: #12317.
I wasn’t able to run full GPU tests yet due to resource limitations, but I’ve aligned the pipeline changes with the DiffSynth Blockwise ControlNet implementation.

Any feedback, suggestions, or improvements would be greatly appreciated!