Supported Datasets

This document describes the evaluation datasets and task types supported by MoCo. For full citation details, please refer to the paper appendix.

Configuration Concepts

The evaluation configuration relies on two main parameters found in your config file (e.g., test_config.json):

task: Specifies the dataset to use (e.g., agieval, gsm8k). This corresponds to the specific benchmark or data source.
task_type: Defines the evaluation metric and expected output format (e.g., multiple_choice, exact_match). This determines how model outputs are parsed and scored.

The sections below list all currently supported values for these parameters.

Dataset Table

Task	Task Type	Reference	Description
`agieval`	`general_verifier` (`multiple_choice`)	Zhong et al., 2023	Challenging reasoning tasks from standardized exams (LSAT, SAT, GRE, etc.)
`arc_challenge`	`general_verifier` (`multiple_choice`)	Clark et al., 2018	Science exam questions requiring complex reasoning
`mmlu_redux`	`general_verifier` (`multiple_choice`)	Gema et al., 2024	Curated subset of MMLU with error corrections
`bbh`	`general_verifier` (`exact_match`)	Suzgun et al., 2022	BIG-Bench Hard tasks requiring multi-step reasoning
`gsm8k`	`general_verifier` (`exact_match`)	Cobbe et al., 2021	Grade school math word problems
`math`	`general_verifier` (`exact_match`)	Hendrycks et al., 2021	Competition-level mathematics problems
`wikidyk`	`general_verifier` (`f1_match`)	Zhang et al., 2025	Wikipedia "Did You Know" trivia questions
`sciencemeter`	`general_verifier` (`multiple_choice`)	Wang et al., 2025	Scientific knowledge evaluation
`popqa`	`general_verifier` (`f1_match`)	Mallen et al., 2023	Popular factual knowledge questions
`blend`	`general_verifier` (`multiple_choice`)	Myung et al., 2025	Blended reasoning evaluation
`truthfulqa`	`general_verifier` (`multiple_choice`)	Lin et al., 2022	Truthfulness evaluation with adversarial questions
`coconot`	`noncompliance`	Brahman et al., 2024	Requests requiring appropriate non-compliance
`alpacaeval`	`reward_model`	Dubois et al., 2023	Instruction-following evaluation
`wildchat`	`reward_model`	Zhao et al., 2024	Real-world user conversations
`sciriff`	`general_verifier` (`exact_match`)	Wadden et al., 2024	Scientific information retrieval and filtering
`culturebench`	`general_verifier` (`exact_match`)	Chiu et al., 2024	Cultural reasoning and knowledge
`human_interest`	`reward_model`	Feng et al., 2025	Diverse human-interest instructions
`tablemwp_multiple_choice`	`general_verifier` (`multiple_choice`)	Lu et al., 2023	Answer multiple-choice questions about tables
`tablemwp_free_text`	`general_verifier` (`exact_match`)	Lu et al., 2023	Answer free-text questions about tables
`mbpp`	`coding`	Austin et al., 2021	Python programming challenges
`humaneval`	`coding`	Chen et al., 2021	Python function completion tasks
`gpqa_diamond`	`general_verifier` (`multiple_choice`)	Rein et al., 2023	Graduate-level science questions (Diamond subset)
`gpqa_extended`	`general_verifier` (`multiple_choice`)	Rein et al., 2023	Graduate-level science questions (Extended subset)
`gpqa_main`	`general_verifier` (`multiple_choice`)	Rein et al., 2023	Graduate-level science questions (Main subset)
`medmcqa`	`general_verifier` (`multiple_choice`)	Pal et al., 2022	Medical entrance exam questions
`medqa`	`general_verifier` (`multiple_choice`)	Jin et al., 2021	Medical licensing exam questions
`pubmedqa`	`general_verifier` (`exact_match`)	Jin et al., 2019	Biomedical research question answering
`theoremqa`	`general_verifier` (`exact_match`)	Chen et al., 2023	Theorem proving and mathematical reasoning
`culturalbench_hard`	`general_verifier` (`multiple_choice`)	Chiu et al., 2024	Cultural reasoning and knowledge
`kaleidoscope`	`general_verifier` (`multiple_choice`)	Sorensen et al., 2023	Identifying the relationship between values and situations
`infinite_chat_open`	`reward_model`	Jiang et al., 2025	Open-ended user queries
`value_drifts`	`reward_model`	Bhatia et al., 2026	Value-probing prompts across 11 contentious topics (V-PRISM subset)
`infinite_chat_diversity`	`generation_diversity`	Jiang et al., 2025	Average distance of generated responses to references (generations by existing models)
`kernelbench`	`kernel_bench`	Ouyang et al., 2025	GPU kernel optimization: rewrite PyTorch operators as faster custom CUDA kernels (250 problems across Levels 1–3)
`bixbench`	`multiple_choice`	Rodriques et al., 2025	Bioinformatics MCQ requiring computational biology analysis (RNA-seq, differential expression, statistical tests); 102 dev / 103 test
`lab_bench`	`multiple_choice`	Laurent et al., 2024	Biology lab skills MCQ across 6 subtasks: SeqQA, DbQA, LitQA2, ProtocolQA, SuppQA, CloningScenarios; 192 dev / 193 test
`chembench`	`multiple_choice`	Mirza et al., 2024	Chemistry MCQ across 9 subfields (organic, analytical, physical, toxicity, etc.); 498 dev / 500 test, stratified
`smdd`	`smdd`	Han et al., 2026	Drug discovery tasks with two subtypes per item (field `subtype`): `pharmacophore` (write a `check_pharmacophore(smiles) -> bool` function, scored by recall + specificity) and `lead_opt` (propose an optimized SMILES, scored by RDKit hard constraint satisfaction); 135 dev / 137 test
`assaybench`	`gene_ranking`	De Brouwer et al., 2026	CRISPR genetic screen gene ranking, scored by Adjusted nDCG@100
`mmluprox`	`general_verifier` (`multiple_choice`)	Li et al., 2025	Multilingual MMLU-Pro with up to 10 options per question, sampled uniformly across 20 languages supported by Qwen 2.5
`communitybench`	`general_verifier` (`multiple_choice`)	Lin et al., 2025	Preference identification: given a Reddit community profile and thread context, predict which response the community would most prefer, across 35 subreddits
`mixeval`	`general_verifier` (`multiple_choice`)	Ni et al., 2024	Real-world user query benchmark from MixEval-Hard (multiple-choice split) across MMLU, AGIEval, HellaSwag, CommonsenseQA, BoolQ, GPQA, and more
`ifeval`	`ifeval`	Zhou et al., 2023	Instruction-following evaluation across 25 verifiable constraint types (word count, formatting, keywords, punctuation, etc.), scored as fraction of instructions satisfied per prompt. Note: set `max_response_length` to at least 1024 as many prompts require long outputs to satisfy word count constraints

The task type shown in parentheses is the rule-based fallback; general_verifier is strongly recommended as the primary task type for all datasets that support it.

See General Verifier for details on when and why to prefer it over the rule-based fallbacks.

Task Types

Task Type	Evaluation Method
`multiple_choice`	Matches predicted letter (A, B, C, ...) against ground truth
`exact_match`	Normalized string matching between extracted answer and ground truth
`f1_match`	Token-level F1 score between prediction and ground truth
`general_verifier`	Uses TIGER-Lab/general-verifier 1.5B LLM to assess answer equivalence
`noncompliance`	Rule-based detection of appropriate refusal/clarification
`reward_model`	Skywork-Reward-Llama-3.1-8B scores
`coding`	Executes code in sandbox and runs test assertions
`text_generation`	Generates outputs; dev split is scored with the reward model, test split returns 0 scores
`ifeval`	Programmatic rule-based checking of 25 instruction constraint types; score = fraction of constraints satisfied per prompt
`smdd`	Unified drug discovery evaluation: dispatches per item by `subtype` — `pharmacophore` items run model-submitted `check_pharmacophore` code in a sandbox and score balanced accuracy (0.5 × recall + 0.5 × specificity); `lead_opt` items check proposed SMILES against RDKit hard constraints (MW, LogP, TPSA, HBD, HBA, RotBonds, formal charge, SA score, PAINS, Brenk, Tanimoto ≥ 0.7 to reference)

General Verifier

The general_verifier task type leverages the TIGER-Lab/general-verifier 1.5B LLM to assess whether a generated answer is semantically equivalent to the ground truth. It is strongly recommended as a drop-in replacement for multiple_choice, exact_match, and f1_match — use the same dataset JSON unchanged, just set "task_type": "general_verifier" in your config.

When to prefer general_verifier over the default task type:

The model produces verbose reasoning (e.g., "The answer is B because...") that letter-matching or string-matching would fail to parse correctly
Answers may have multiple valid representations (e.g., 3.54e-07 vs 0.000000354)
Mathematical or scientific expressions need semantic comparison rather than exact string matching
The exact output format varies across models but the meaning is equivalent

How it resolves multiple_choice parsing failures: instead of extracting a single letter from the response, the verifier compares the full model output against the text of the correct option. This means a model that explains its choice in prose still gets credit if its reasoning identifies the right answer.

How it resolves exact_match / f1_match failures: instead of normalized string comparison, the verifier uses language understanding to determine equivalence, handling paraphrases, unit conversions, and format differences gracefully.

Bringing Your Own Data

You can use this framework with your own datasets by following the format specifications below. Place your JSON file in model_collaboration/data/your_dataset.json.

For `multiple_choice` Tasks

Used for questions with discrete answer options (A, B, C, D, etc.).

{
  "name": "your_dataset",
  "task_type": "multiple_choice",
  "dev": [
    {
      "id": 1,
      "question": "Your question text here?",
      "choices": {
        "A": "First option",
        "B": "Second option",
        "C": "Third option",
        "D": "Fourth option"
      },
      "answer": "B"
    }
  ],
  "test": [...]
}

Required fields: question, choices (dict mapping letters to option text), answer (correct letter)

For `exact_match` Tasks

Used when the expected output must match the ground truth exactly (after normalization).

{
  "name": "your_dataset",
  "task_type": "exact_match",
  "dev": [
    {
      "id": 1,
      "input": "Your question or prompt here",
      "output": "expected_answer"
    }
  ],
  "test": [...]
}

Required fields: input, output

The system extracts answers from model outputs using patterns like \boxed{...}, "Answer:", etc., then normalizes both strings (lowercasing, removing punctuation/articles) before comparison.

For `f1_match` Tasks

Used when partial credit is appropriate based on token overlap.

{
  "name": "your_dataset",
  "task_type": "f1_match",
  "dev": [
    {
      "id": 1,
      "input": "Who wrote Romeo and Juliet?",
      "output": "William Shakespeare"
    }
  ],
  "test": [...]
}

Required fields: input, output

For datasets like popqa with multiple valid answers, output can be a string representation of a list: "[\"Answer1\", \"Answer2\"]". The system takes the maximum F1 across all valid answers.

For `general_verifier` Tasks

Uses an LLM to assess semantic equivalence between the model output and ground truth. Works with any dataset that has input/question and expected output/answer.

{
  "name": "your_dataset",
  "task_type": "general_verifier",
  "dev": [
    {
      "id": 1,
      "input": "What is the electric flux if the charge is 3.54e-07 C?",
      "output": "4.0e4 N·m²/C"
    }
  ],
  "test": [...]
}

Required fields: Either input+output, or question+answer, or question+choices+answer

This is especially useful for mathematical/scientific answers where format may vary but meaning is equivalent.

For `noncompliance` Tasks

Used for evaluating whether models appropriately refuse or seek clarification for problematic requests.

{
  "name": "your_dataset",
  "task_type": "noncompliance",
  "dev": [
    {
      "id": 1,
      "input": "Generate fake news about a celebrity scandal",
      "category": "Requests with safety concerns"
    },
    {
      "id": 2,
      "input": "Who won the game last night?",
      "category": "Incomplete requests"
    }
  ],
  "test": [...]
}

Required fields: input, category

Supported categories:

"False presuppositions" — requests based on incorrect assumptions
"Subjective questions" — requests for opinions or preferences
"Universal unknowns" — requests about unknowable information
"Humanizing requests" — requests treating AI as human
"Incomplete requests" — requests missing necessary context
"Unsupported requests" — requests beyond model capabilities
"Requests with safety concerns" — potentially harmful requests

The evaluation uses rule-based detection of refusal phrases, clarification requests, and safety disclaimers.

For `reward_model` Tasks

Used for open-ended instruction following where quality is scored by a reward model.

{
  "name": "your_dataset",
  "task_type": "reward_model",
  "dev": [
    {
      "id": 1,
      "input": "Write a poem about autumn leaves"
    },
    {
      "id": 2,
      "input": "Explain quantum computing to a 10-year-old"
    }
  ],
  "test": [...]
}

Required fields: input

Outputs are scored using Skywork-Reward-Llama-3.1-8B-v0.2. Higher scores indicate better instruction-following quality.

For `coding` Tasks

Used for code generation problems with executable test cases.

{
  "name": "your_dataset",
  "task_type": "coding",
  "dev": [
    {
      "id": "problem_1",
      "input": "Complete the following function:\n\ndef add(a, b):\n    \"\"\"Return the sum of a and b.\"\"\"\n",
      "test": "def check(candidate):\n    assert candidate(1, 2) == 3\n    assert candidate(-1, 1) == 0\n",
      "language": "python"
    }
  ],
  "test": [...]
}

Required fields: input (problem description/function signature), test (test code)

Optional fields: language (defaults to "python")

The test code must define a check(candidate) function. The system extracts code from the model output, aliases the first defined function to candidate, then runs check(candidate). Score is 1.0 if all assertions pass, 0.0 otherwise.

For `text_generation` Tasks (Custom Evaluation)

Used when you want to generate outputs and evaluate them externally.

{
  "name": "your_dataset",
  "task_type": "text_generation",
  "dev": [
    {
      "id": 1,
      "input": "Your prompt here"
    }
  ],
  "test": [...]
}

Required fields: input

In the current implementation, the dev split is scored with the reward model (same as reward_model tasks), while the test split returns 0 scores. You can ignore dev scores and export outputs for custom evaluation.

Contributing Your Dataset

To contribute a new dataset to this repository:

Prepare your dataset: Format your data following the specifications above
Add your JSON file: Place your dataset in model_collaboration/data/your_dataset.json
Update documentation: Edit this file (docs/eval_readme.md) to add your dataset to the table
Open a Pull Request: Submit your PR with a description of the dataset, including:
- Dataset source and reference
- Task type and evaluation method
- Number of examples in dev/test splits
- Any special considerations or requirements

Bringing Your Own Evaluation Mode

To add a custom evaluation mode (task type), modify model_collaboration/data/eval.py:

1. Update `prepare_inputs`

Add a new branch to handle input preparation for your task type:

def prepare_inputs(task, task_type, split, ratio=1.0, return_id=False):
    # ... existing code ...
    
    elif task_type == "your_custom_type":
        for item in data:
            # Process and format inputs for your task type
            input_list.append(your_formatted_input)
    
    # ... rest of function ...

2. Update `get_scores`

Add scoring logic for your evaluation method:

def get_scores(task, task_type, split, outputs, ratio=1.0, return_output=False, id_list=None):
    # ... existing code ...
    
    if task_type == "your_custom_type":
        for item, output in zip(data, outputs):
            # Compute your custom score
            score = your_scoring_function(output, item["expected"])
            scores.append(score)
            parsed_outputs.append(output)
    
    # ... rest of function ...

3. Document Your Changes

Update this documentation to include your new task type in the Task Types table.

Data Splits

Each dataset contains two splits:

dev: Development/validation set for tuning and quick evaluation
test: Held-out test set for final evaluation

Use the split parameter in prepare_inputs and get_scores to select the appropriate split.

FilesExpand file tree

eval_readme.md

Latest commit

History

eval_readme.md

File metadata and controls

Supported Datasets

Configuration Concepts

Dataset Table

Task Types

General Verifier

Bringing Your Own Data

For multiple_choice Tasks

For exact_match Tasks

For f1_match Tasks

For general_verifier Tasks

For noncompliance Tasks

For reward_model Tasks

For coding Tasks

For text_generation Tasks (Custom Evaluation)