Not a bug, but some questions.

[Kevin-naticl]

Great work! About why it works i have some ideas, but IDK if i am right, So here i am.
My perspective:

Thought: Why is it that even though SVF has fewer trainable parts, it is still more effective?

From the perspective of linear transformation, among the three matrices derived from Singular Value Decomposition (SVD), the most crucial one is the middle matrix. The first and third matrices are essentially just rotations in the linear transformation. The diagonal matrix in the middle actually determines the scaling of weights across various dimensions. SVF can directly alter min(m,n) dimensions.

Secondly, during LoRA fine-tuning, it appears that there are (m+n)*r' parameters. However, if these two matrices are also subjected to SVD, in reality, when adjusting the most critical scaling dimensions, only r'+r' dimensions are actually available for use.

THX so much.

[floatingsun]

Hi @Kevin-naticl , Thank you for your interset and sharing your thought.

I think you are right. The information that the LoRA finetuning process can bring is essentially upper bounded by the number of RANK. Meanwhile, the SVF tunes the singular value matrix, it actually introduces full-rank information. Which we believe is where the effectiveness root from.

[SuperSonnix71]

I recon that by adjusting only the singular values, SVF directly influences the most significant factors affecting the model's performance which’s more efficient than LoRA, that adjust a larger number of parameters for faster training and reduced overfitting risks but I guess tasks which require nuanced adjustments beyond scaling might not benefit from SVF.

[SuperSonnix71]

I was playing with SVD but SVF seems to be more efficient since SVD decomposes a matrix into two orthogonal matrices and one diagonal matrix of singular values where the middle matrix in SVD contains the singular values. Good one! Awesome work!
https://www.cs.ucr.edu/~epapalex/papers/22-ICDMW-OEDM-SVLearn.pdf

[BradKML]

@floatingbigcat a bit of a side question: do you think this method would also apply to BitNet-like models, or MoE models?