Using a VLM inside a reward function

[korbinian-hoermann]

Hi,

I would like to fine-tune a model on web trajectories.

One component of the reward should be a low-level action execution evaluation,
which makes use of a VLM to determine if a generated command, e.g. click(x, y) is suitable to execute a high-level action,
e.g. Click the yellow button.

Therefore, I would like to annotate the original input image with a circle at x,y , and ask the evaluator VLM
if the click action at the annotated coordinates is suitable to execute the high-level action.

How can i access the image inside my reward function ?