[LoRA] improve LoRA fusion tests

[sayakpaul]

What does this PR do?

This PR improves our LoRA fusion tests by:

• Additionally checking the pipe.num_fused_loras argument.
• Correctly determines num_fused_loras within lora_base.py.
• Adding a test test_lora_scale_kwargs_match_fusion that checks LoRA + scale outputs match with LoRA fusion with the same scale.

@BenjaminBossan if I could pick your brain, the following tests are failing on a GPU but passing on a CPU:

FAILED tests/lora/test_lora_layers_sdxl.py::StableDiffusionXLLoRATests::test_simple_inference_with_text_denoiser_lora_unfused - AssertionError: False is not true : Fused lora should not change the output
FAILED tests/lora/test_lora_layers_sdxl.py::StableDiffusionXLLoRATests::test_simple_inference_with_text_lora_denoiser_fused_multi - AssertionError: False is not true : Fused lora should not change the output
FAILED tests/lora/test_lora_layers_sdxl.py::StableDiffusionXLLoRATests::test_lora_scale_kwargs_match_fusion_0 - AssertionError: False is not true : Fused lora should not change the output
FAILED tests/lora/test_lora_layers_sdxl.py::StableDiffusionXLLoRATests::test_lora_scale_kwargs_match_fusion_1 - AssertionError: False is not true : Fused lora should not change the output

Would you have any pointers? I have tested this with both peft main and latest.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

if I could pick your brain, the following tests are failing on a GPU but passing on a CPU:

I ran those 4 tests locally on a 4090 and they all passed. Same with the CPU. How did you run the tests?

My first instinct would be that it's a precision problem. Assuming the issue is that some tensors are not torch.allclose, could you jump in and check the values? Are they hugely different or just barely?

[sayakpaul]

@BenjaminBossan could you share your diffusers-cli env?

Also, feel free to review the rest of PR for the changes in lora_base.py and tests

[BenjaminBossan]

could you share your diffusers-cli env?

- 🤗 Diffusers version: 0.33.0.dev0 [221726b82df3a71be5870e842aa3677a9a1f80f6]
- Platform: Linux-6.11.0-21-generic-x86_64-with-glibc2.39
- Running on Google Colab?: No
- Python version: 3.12.9
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.30.1
- Transformers version: 4.52.0.dev0 [0ddad2d6553b42fe21d64c18ad6617360cbe8af8]
- Accelerate version: 1.6.0
- PEFT version: 0.15.2.dev0 [5e9ee26e79fe1d6f5fe74c6aa1d0054278f6ed15]
- Bitsandbytes version: 0.45.3
- Safetensors version: 0.5.3
- xFormers version: not installed
- Accelerator: NVIDIA GeForce RTX 4090, 24564 MiB
NVIDIA GeForce RTX 4090, 24564 MiB
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

Also, feel free to review the rest of PR for the changes in lora_base.py and tests

I'll take a look.

[BenjaminBossan]

The PR generally LGTM. I have one comment, but it's probably an edge case.

Regarding the failing tests, I don't see why they would start failing now (the ones that were already there), since the changes shouldn't really affect them. Are you sure they pass without the changes, all else being equal? As mentioned, I think tweaking the tolerances might be a solution (albeit not a nice one).

[sayakpaul]

Are you sure they pass without the changes, all else being equal? As mentioned, I think tweaking the tolerances might be a solution (albeit not a nice one).

@BenjaminBossan I should have been clear. The first two tests fail without the PR changes (the latter two being added in this PR).

[BenjaminBossan]

I should have been clear. The first two tests fail without the PR changes (the latter two being added in this PR).

I see. Did you check by how much the values differ?

[sayakpaul]

@BenjaminBossan I decided to just relax the tolerance for GPU. I requested for a clarification here: #11274 (comment)

[BenjaminBossan]

I decided to just relax the tolerance for GPU

Makes sense.

I requested for a clarification here: #11274 (comment)

I already replied 🤔

[BenjaminBossan]

Overall LGTM, thanks. I made one small suggestion for a possible edge case, up to you if you want to fix it.

[sayakpaul]

@DN6 could you give the changes under lora_base.py a look too?