Inquiry regarding the T-LoRA implementation in the FLUX training framework

[EllieWang023]

I noticed that the current LoRA implementation in flux/model/lora.py uses standard initialization (down.weight is random, up.weight is zero). However, the T-LoRA paper suggests a weight revision trick ($\tilde{W} = W - BSA + BSA$) to enable orthogonal initialization. Could you clarify if the current version is intended to be a Vanilla T-LoRA (focusing primarily on timestep-dependent rank control) rather than the full framework including orthogonal initialization? Thanks!

[jialima-sg]

@EllieWang023 I am also experimenting FLUX with T-LORA. If you check the paper, you will find explanation as below:
In Section: 4.6 FLUX-1.dev Experiments

We found that, unlike SD-XL, LoRA adapters trained on MM-DiT are full-rank (see Figure 9), making the additional orthogonal initialization unnecessary. Therefore, we used the Vanilla variant of T-LoRA for all FLUX-1.dev experiments.

Hope that answers you questions.