LLM Reasoning Fidelity Evaluation

This project evaluates the reasoning consistency of Large Language Models (LLMs) on mathematical word problems. It specifically focuses on how models handle perturbations in questions—like changing numerical values—to measure if their logic remains consistent or if they rely on memorized patterns.

The current setup uses the GSM8K dataset and tests against a local Llama 3.2 model running via Ollama.

Project Structure

• dataset.py: Handles data ingestion from the GSM8K dataset, samples 60 questions, and applies perturbations (randomly incrementing numbers) to create a control set.
• min.py & hil.py: These scripts interface with the local Ollama API to run experiments. They send both the original and perturbed questions to the model and parse the reasoning/answers into structured results.
• comput.py: A utility script to calculate final metrics from the experiment results, including:
  • Accuracy: Overall correctness on the original questions.
  • SOR (Step Omission Rate): Frequency of missing reasoning steps.
  • PCS (Perturbation Consistency Score): How often the model adapts correctly to changed numbers.
  • FDS (Fidelity Degradation Score): A combined metric of inconsistency and reasoning gaps.

Getting Started

Prerequisites

• Python 3.8+
• Ollama installed and running locally with the llama3.2 model pulled.

Installation

1. Clone the repository.
2. Install the required dependencies:

   pip install -r requirements.txt

Running Experiments

1. Prepare Data:

   python dataset.py

2. Execute Tests: Run min.py or hil.py to start the model evaluations. These will generate/update results in CSV files.

   python min.py

3. Analyze Results: Calculate the final performance metrics:

   python comput.py

Results

The experiment results are saved to minimal_reasoning_results.csv and llama32_full_results.csv for further inspection or visualization.