[P1] Confusion over what generation currently does

[leogagnon]

Hi, I'm a bit confused by the comment that means, we intervene on every generated tokens! here https://github.com/stanfordnlp/pyvene/blob/31d3986e5b0b3d812eef8fdd0c39759fa1c1b58e/pyvene/models/intervenable_base.py#L2020C1-L2022C41. From what I understand setting unit_locations = {"base": 0} leads to intervening only on the first token, not on every generated token. Am I understanding right?

Also am I right in saying that intervening on the last generated token is not currently supported? This is how I interpreted this comment https://github.com/stanfordnlp/pyvene/blob/31d3986e5b0b3d812eef8fdd0c39759fa1c1b58e/pyvene/models/intervenable_base.py#L1989C1-L1991C29.

Thanks,

[frankaging]

hey @leogagnon thanks for your question!

there are other questions on how to use pyvene with model generation. i created a new example and added into our tutorial to show details in https://github.com/stanfordnlp/pyvene/blob/main/pyvene_101.ipynb:

1.2.15. Advanced Intervention on LMs Generation (Model Steering)