The Kotlin Benchmark

A Kotlin software-engineering benchmark for evaluating coding agents.

The benchmark is built on the Multi-SWE-bench methodology, extended with Kotlin support. It is packaged in the Harbor task format — every task is a self-contained directory with its own reproducible Docker environment, instruction, gold solution, and test verifier.

Dataset

106 tasks across nine open-source Kotlin repositories:

Repository	License	Tasks
pinterest/ktlint	MIT	43
detekt/detekt	Apache-2.0	28
oss-review-toolkit/ort	Apache-2.0	12
Hannah-Sten/TeXiFy-IDEA	MIT	8
ankidroid/Anki-Android	GPL-3.0	6
Kotlin/dataframe	Apache-2.0	5
square/okhttp	Apache-2.0	2
GradleUp/shadow	Apache-2.0	1
linreal/cascade-editor	MIT	1
Total		106

For every task the dataset captures:

• the base commit (repository state before the change),
• the human-written gold solution patch,
• the regression tests that define the expected behavior,
• the natural-language issue description.

Task structure (Harbor format)

Each task lives under tasks/<owner>_<repo>-<pr_number>/ and is fully self-contained:

tasks/ankidroid_Anki-Android-18903/
├── task.toml                     # Task metadata + run config (timeouts, resources, network policy)
├── instruction.md               # The issue/PR description handed to the agent
├── environment/                 # Docker build context for the task image
│   ├── Dockerfile               #   FROM a shared local base (kotlin-bench/<repo>:<base-tag>);
│   │                            #   runs prepare.sh + agent tooling, then seals Git history
│   ├── prepare.sh               #   checks out the base commit, warms the build cache
│   ├── check_git_changes.sh     #   asserts a clean working tree
│   ├── seal_git_history.sh      #   leaves only the base commit reachable (removes the upstream fix)
│   └── exclude-flaky-tests.*    #   (some tasks) Gradle init script that skips flaky tests
├── solution/
│   ├── fix.patch                # The gold (reference) solution
│   └── solve.sh                 # Applies fix.patch — the "oracle" run
└── tests/
    ├── test.patch               # Adds/updates the hidden regression tests
    ├── test.sh                  # Applies test.patch, runs the suite, writes the reward
    ├── expected_tests.json      # Expected test transitions (f2p / p2p / s2p / n2p)
    ├── kotlin_logs_collector.sh # Collects JUnit XML from Gradle build output
    └── junit_compare.py         # Parses JUnit XML and computes the reward

task.toml

[metadata.source]      # provenance: repo, PR/issue URLs, upstream SPDX license, base info
[verifier]             # verification timeout;      network_mode = "public"
[agent]                # agent solving timeout;     network_mode = "allowlist" + allowed_hosts
[environment]          # build timeout, cpus, memory_mb; network_mode = "public"

[metadata.source] records a license marker holding the SPDX identifier of the upstream repository the task is derived from (e.g. license = "Apache-2.0"), so each task carries the provenance of its source project's license.

Network policy. The agent runs under network_mode = "allowlist". Every task uses the same build-host list: Maven Central, the Gradle plugin and distribution services, Google Maven, JetBrains Maven and download services, and the Dart package registries. The verifier and environment phases use network_mode = "public" so dependency resolution during build/verification is unrestricted.

Scoring

tests/test.sh runs inside the task container after the agent's patch is applied. It is fail-closed: it writes reward 0 to /logs/verifier/reward.txt before doing anything, and a reward of 1 is only ever produced by junit_compare.py validating freshly generated reports. Concretely it:

1. writes the default reward 0 and clears any previous verifier logs,
2. applies test.patch to inject the regression tests — aborting with reward 0 if it does not apply,
3. deletes any stale */build/test-results/*/TEST*.xml so only this run's reports can count,
4. runs the Gradle test suite (a non-zero Gradle exit does not by itself change the reward),
5. collects JUnit XML via kotlin_logs_collector.sh, and
6. compares the results against expected_tests.json with junit_compare.py, which writes the final reward (1 = resolved, 0 = not resolved).

Tests are classified by their status transition between the unpatched and patched runs:

• f2p — fail → pass (the bug being fixed)
• p2p — pass → pass (no regressions in already-passing tests)
• s2p / n2p — skip/new → pass

A task passes only when every expected test lands in its correct category.

Base images

Task images are not self-contained from JDK up — they build FROM a shared base image that bundles the JDK, any SDKs, and a clone of the upstream repo. A repository can have several base variants on different JDKs (e.g. kotlin-bench/pinterest_ktlint:base on JDK 21 and kotlin-bench/pinterest_ktlint:base-JDK-17 on JDK 17); the variants differ in more than the JDK line, so each has its own recipe. Base images need to be built once before running any task:

scripts/build_bases.sh            # build every base listed in bases/manifest (skips existing)
scripts/build_bases.sh --rebuild  # force a rebuild

Each base recipe lives in bases/<repo>/Dockerfile.<base-tag> and is built as kotlin-bench/<repo>:<base-tag>; bases/manifest lists all of them. Each task's environment/Dockerfile selects the variant it needs via its FROM line.

Running the benchmark

Tasks are run with the Harbor CLI, which builds each task's container from environment/, runs the agent against instruction.md, then executes tests/test.sh. Install it with uv tool install harbor. See harbor run --help for available agents and flags.

# Build the shared base images once (see "Base images" above)
scripts/build_bases.sh

# Sanity-check a task with the Oracle agent (applies the gold patch; reward should be 1)
harbor run -p tasks/ankidroid_Anki-Android-18903 -a oracle

# Evaluate an agent on a single task
harbor run -p tasks/ankidroid_Anki-Android-18903 -a "<agent>" -m "<model>"

# Run the full suite (a dataset is just a directory of tasks)
harbor run -p tasks -a "<agent>" -m "<model>"

License

See LICENSE. Tasks are derived from open-source repositories; each upstream project retains its own license. The SPDX identifier of the source project's license is recorded in the license field under [metadata.source] in every task's task.toml (see the table above for the per-repository breakdown).