How to load a PEFT adapter with per-layer rank variation via from_pretrained?

[yuxuan-z19]

Hi all,

I'm working on a PEFT variant where each adapter layer has its own rank, instead of using a globally uniform rank (as in standard LoRA). For example, q_proj might use rank=8 while v_proj uses rank=32, etc.

However, when I call PeftModel.from_pretrained(...), it seems that:

• The framework expects a single rank=r value for all layers.
• It pre-allocates all adapter parameters based on this fixed rank before loading the actual adapter weights.
• As a result, any pre-trained adapter with per-layer variable ranks causes shape mismatch errors or wastes memory.

What I'm looking for:
A way to dynamically allocate LoRA parameters after inspecting the actual pretrained adapter weights, so that I can support different ranks per layer during loading.

Is there any recommended way to delay or override adapter allocation inside from_pretrained(), so that it can dynamically allocate based on the actual adapter checkpoint (e.g., the lora_A shapes)?

Or perhaps, is there a hook or extension point that allows me to customize how LoRA layers are created during from_pretrained?

Any guidance would be appreciated! Thanks!

[BenjaminBossan]

When creating the LoraConfig, you can pass rank_pattern with specific r values for specific layers.

[yuxuan-z19]

Thanks @BenjaminBossan! That’s exactly what I needed!

[yuxuan-z19]

Duplicates of #2601