FEAT support any causal LM in GCG embedding helpers

[devangpratap]

Closes #2489 (embedding adapters portion).

Problem

GCG loads models with AutoModelForCausalLM, but get_embedding_layer, get_embedding_matrix and get_embeddings each dispatched through an isinstance chain over six hard-coded architectures and ended in raise ValueError(f"Unknown model type: {type(model)}"). A newly supported Hugging Face causal model therefore loads successfully and then fails later, at the gradient step.

Reproduced on main with tiny random configs, no weights or GPU:

model_type     has get_input_embeddings   GCG accessors
gpt2           True                       layer=ok          matrix=ok          embed=ok
llama          True                       layer=ok          matrix=ok          embed=ok
qwen3          True                       layer=ValueError  matrix=ValueError  embed=ValueError
gemma3_text    True                       layer=ValueError  matrix=ValueError  embed=ValueError
olmo2          True                       layer=ValueError  matrix=ValueError  embed=ValueError
starcoder2     True                       layer=ValueError  matrix=ValueError  embed=ValueError
mpt            True                       layer=ValueError  matrix=ValueError  embed=ValueError

Every one of them implements PreTrainedModel.get_input_embeddings(), so what GCG needs was already reachable through the public interface.

Change

All three helpers now resolve through model.get_input_embeddings().

Before collapsing the branches I checked that they were actually equivalent rather than assuming it. For each previously hard-coded architecture, the old expression returns the identical object to the universal accessor:

gpt2       layer_identical=True   matrix_identical=True
gptj       layer_identical=True   matrix_identical=True
gpt_neox   layer_identical=True   matrix_identical=True
llama      layer_identical=True   matrix_identical=True
mistral    layer_identical=True   matrix_identical=True
mixtral    layer_identical=True   matrix_identical=True
phi3       layer_identical=True   matrix_identical=True

So this is not a behaviour change for models that already worked.

On the .half() asymmetry. get_embeddings called .half() for GPT-2, GPT-J and GPT-NeoX but not for Llama, Mistral, Mixtral or Phi-3. I raised this on the issue and have deliberately preserved it exactly rather than folding a dtype change into a compatibility fix. If that asymmetry is incidental rather than intended, I am happy to remove it in a separate PR where it can be reviewed on its own merits.

The now-unused imports for LlamaForCausalLM, MistralForCausalLM, MixtralForCausalLM and Phi3ForCausalLM are dropped. The GPT-2, GPT-J and GPT-NeoX imports stay, since the dtype branch still needs them.

Tests

The three test_*_raises_for_unknown_model tests asserted the behaviour this PR intentionally removes, so they are replaced rather than left to fail. New parametrized coverage over nine architectures builds tiny random models on CPU and asserts:

• get_embedding_layer(model) is model.get_input_embeddings()
• get_embedding_matrix(model) is model.get_input_embeddings().weight
• get_embeddings returns the model's hidden size
• float16 for GPT-2, GPT-J and GPT-NeoX
• the embedding dtype, unchanged, for the rest

$ pytest tests/unit/executor/promptgen/gcg/ -q
242 passed

Pre-commit clean on both changed files, including ruff format, ruff check and ty.

Not in this PR

The issue also asks for a tokenizer preflight check and a maintained compatibility matrix. I have left those out so this stays reviewable on its own, and I am happy to follow up with them once the direction here is agreed.

[devangpratap]

@microsoft-github-policy-service agree