Social Decision Modeling Evaluation of LLMs

This repository contains the official implementation of our ACL 2026 Findings Paper on evaluation of motive modeling capabilities of LLMs using frameworks from cognitive science and game theory, providing code and evaluation scripts.

Update as of July 2: The code is currently under development for readability and modularity.

📄 Paper Information

Title: Can Large Language Models Infer Human Actions and Motives? Evaluation in Social Prediction and Inspection Games

Authors: Kaleen Shrestha, Abhinav Gupta, Harish Dukkipati, Zhonghao Shi, and Maja Matarić

[Conference] ACL 2026 Findings

Running the main.py Script

To run the main.py script with specific parameters, use the following command:

python3 main.py --n 16 --motive "risk-averse" --game_model gpt-4o --repetition 5 --history_window "full" --experiment_identifier "no-system-prompt"

Parameters

• --n: Specifies the number of trials or experiments to run. In this example, n is set to 16. Can also be set to 100.
• --motive: Sets the experiment strategy. Options include "greedy", "risk-averse", "inverse greedy", and "inverse risk-averse". In this example, it’s set to "greedy".
• --game_model: Specifies the model to use for the game, e.g., "gpt-4o".
• --repetition: Defines the number of times each experiment is repeated. Here, it is set to 5.
• --history_window: Determines the length of the history window. Use "full" for a complete history window.
• --experiment_identifier: A descriptive name for the experiment such as describing the prompt for use in the folder/file names