Add Gradle project support for Kotlin/Java tasks and expand agent allowed hosts

- Require Gradle project structure for Kotlin/Java tasks with offline-compatible dependencies.
- Extend default agent allowed hosts to include Maven, Gradle, Kotlin, and related build-tool documentation.
- Adjust default retry policy: set `n_attempts=3` and `max_retries=2`.
- Update templates and Dockerfile to reflect Gradle integration.
- Introduce new dependencies (`ipykernel`, `matplotlib`, `pandas`) in `pyproject.toml`.
- Expand test coverage for Gradle enforcement, allowed hosts, and policy adjustments.

Author: Frame17

pyproject.toml

@@ -17,6 +17,9 @@ dependencies = [
     "tqdm>=4",
     "rich>=13",
     "huggingface_hub>=0.20",
+    "ipykernel>=7.3.0",
+    "matplotlib>=3.10.9",
+    "pandas>=2.3.3",
 ]
 
 [project.scripts]

scripts/language_comparison/config.yaml

@@ -10,10 +10,14 @@ agent:
   name: codex
   model_name: openai/gpt-5.6-luna
 
+#agent:
+#  name: claude-code
+#  model_name: claude-sonnet-5
+#  kwargs:
+#    max_budget_usd: "20"
+
 tasks:
   task_names:
-#    - ammarabouzor__tui-journal.2b4540d
-#    - jesseduffield__lazygit.1d0db51
       - nikolassv__bartib.6b9b5ce
       - svenstaro__genact.16f96e3
       - svenstaro__miniserve.8449e8b
@@ -30,4 +34,6 @@ tasks:
 # Optional settings copied into the generated Harbor job config. Job name,
 # output paths, concurrency, agents, and datasets are generated by the harness.
 harbor:
-  n_attempts: 1
+  n_attempts: 3
+  retry:
+    max_retries: 2

src/kotlinai/README.md

@@ -52,10 +52,12 @@ uv run programbench harbor export ./out <id> --agent-user root
 The agent phase runs as the cleanroom image's non-root `agent` user, so the
 mode-111 reference binary at `/workspace/executable` can be executed but not
 read, copied, or statically inspected; the generated agent image also drops the
-image's NOPASSWD sudo grant, which would otherwise hand root back. The agent's
-network allowlist covers the model API only — Harbor installs the agent CLI
-during *setup*, which runs before the agent-phase policy applies, so no
-package-registry or source-hosting host belongs on that list.
+image's NOPASSWD sudo grant, which would otherwise hand root back. The default
+agent network allowlist covers model APIs plus Maven, Gradle, Kotlin, Android,
+and related build-tool hosts so dependencies can be stored in `/workspace` for
+the offline verifier build. Source-hosting hosts remain blocked. Harbor installs
+the agent CLI during *setup*, before the agent-phase policy applies, so its
+installer hosts do not belong on the task allowlist.
 
 ## Language comparison experiment
 
@@ -162,4 +164,4 @@ harbor trials start --path ./out/<instance_id> --agent oracle
 
 A literal 100% oracle pass is not reachable for every task in a single-container
 Harbor run — some tests fail on the true reference binary itself in the sandbox
-(missing OS facilities, TUI hangs, snapshot date-drift).
+(missing OS facilities, TUI hangs, snapshot date-drift).

src/kotlinai/data/templates/harbor_instruction.md.j2

@@ -10,6 +10,12 @@ reproduces its observable behavior.
 {% if target_language %}
 You MUST implement the deliverable in {{ target_language }}. Write all source code in {{ target_language }}. Do not reimplement the program in another language.
 {% endif %}
+{% if use_gradle %}
+You MUST structure the implementation as a Gradle project. `compile.sh` MUST invoke
+Gradle to build the program. Include the Gradle wrapper and vendor the Gradle
+distribution and all project dependencies needed for `compile.sh` to run without
+network access.
+{% endif %}
 ## What to deliver
 
 Work inside `/workspace`. When you are done, that directory must contain:

src/kotlinai/data/templates/harbor_task.toml.j2

@@ -31,10 +31,11 @@ build_timeout_sec = {{ build_timeout }}
 cpus = {{ env_cpus }}
 network_mode = "public"
 
-# No network during inference beyond the model API: the agent must
-# reverse-engineer the binary, not fetch upstream source. Overriding the baseline
-# per phase requires a dynamic_network_policy environment backend (e.g.
-# Docker/OrbStack on Linux with nftables).
+# During inference, the agent can reach only the configured model, package,
+# build-tool, and toolchain documentation hosts. Source hosting stays blocked:
+# the agent must reverse-engineer the binary, not fetch upstream source.
+# Overriding the baseline per phase requires a dynamic_network_policy
+# environment backend (e.g. Docker/OrbStack on Linux with nftables).
 #
 # The agent runs as {{ agent_user }}. The default is a non-root user, so the
 # mode-111 reference binary at /workspace/executable can be executed but not

src/kotlinai/harbor.py

@@ -44,16 +44,40 @@
 AGENT_TIMEOUT_SEC = 10800.0
 KOTLIN_VERSION = "2.4.10"
 KOTLIN_COMPILER_SHA256 = "473dd66c7a3ef4b182065b3da670466c1bf2773a9dbb0ed8b33a39fe9d4f876d"
-# The agent phase reaches the model API and nothing else: fetching upstream
-# source would defeat the black-box task. The npm/nvm hosts Harbor's Codex
-# installer needs are deliberately absent — agent setup runs outside the
-# agent-phase policy (Harbor's `Trial._setup_agent` is not wrapped by
-# `_phase_network_policy`), so it installs under the `[environment]` public
-# baseline. Language toolchains are baked in while Harbor builds the task images.
-# Parameterized per runner (override with `harbor export --allowed-host`).
+# The agent phase can reach model APIs and the package, build-tool, and
+# toolchain documentation hosts needed to assemble an offline JVM submission.
+# Source-hosting and agent-installer hosts remain absent: fetching upstream
+# source would defeat the black-box task, while agent setup runs outside the
+# agent-phase policy under the `[environment]` public baseline. Parameterized
+# per runner (override with `harbor export --allowed-host`).
 DEFAULT_AGENT_ALLOWED_HOSTS = [
     "api.openai.com",
     "api.anthropic.com",
+    "repo.maven.apache.org",
+    "repo1.maven.org",
+    "plugins.gradle.org",
+    "services.gradle.org",
+    "downloads.gradle.org",
+    "maven.google.com",
+    "maven.pkg.jetbrains.space",
+    "maven.reposilite.com",
+    "oss.sonatype.org",
+    "kotlinlang.org",
+    "*.kotlinlang.org",
+    "kotl.in",
+    "*.jetbrains.com",
+    "docs.gradle.org",
+    "docs.gradle.com",
+    "gradle.com",
+    "gradle.org",
+    "scans.gradle.com",
+    "help.gradle.org",
+    "developer.android.com",
+    "schemas.android.com",
+    "pub.dartlang.org",
+    "pub.dev",
+    "pnpm.js.org",
+    "ant.apache.org",
 ]
 # The cleanroom image's uid-1000 user. It owns /workspace but, unlike root,
 # cannot read the execute-only reference binary at /workspace/executable.
@@ -136,7 +160,10 @@ def convert_instance(
         )
     )
     (out / "instruction.md").write_text(
-        _env.get_template("harbor_instruction.md.j2").render(target_language=target_language)
+        _env.get_template("harbor_instruction.md.j2").render(
+            target_language=target_language,
+            use_gradle=install_jdk,
+        )
     )
     (out / "environment" / "Dockerfile").write_text(
         _env.get_template("harbor_environment.Dockerfile.j2").render(

tests/test_harbor.py

@@ -77,6 +77,18 @@ def test_convert_instance_instructs_agent_to_use_target_language(tmp_path):
     assert "You MUST implement the deliverable in Kotlin." in instruction
     assert "Write all source code in Kotlin." in instruction
     assert "Do not reimplement the program in another language." in instruction
+    assert "You MUST structure the implementation as a Gradle project." in instruction
+    assert "`compile.sh` MUST invoke\nGradle to build the program." in instruction
+    assert "vendor the Gradle\ndistribution and all project dependencies" in instruction
+    assert "run without\nnetwork access" in instruction
+
+
+def test_convert_instance_only_requires_gradle_for_jvm_targets(tmp_path):
+    java_out = convert_instance(_calc_instance(), tmp_path / "java", target_language="Java")
+    rust_out = convert_instance(_calc_instance(), tmp_path / "rust", target_language="Rust")
+
+    assert "Gradle project" in (java_out / "instruction.md").read_text()
+    assert "Gradle project" not in (rust_out / "instruction.md").read_text()
 
 
 def test_convert_instance_adds_kotlin_toolchain_to_agent_and_verifier(tmp_path):
@@ -113,15 +125,52 @@ def test_convert_instance_does_not_add_jvm_toolchain_for_other_targets(tmp_path)
         assert "kotlin-compiler" not in contents
 
 
-def test_default_agent_allowlist_does_not_expose_toolchain_or_source_hosts(tmp_path):
+def test_default_agent_allowlist_exposes_build_hosts_but_not_source_hosts(tmp_path):
     out = convert_instance(_calc_instance(), tmp_path, target_language="Kotlin")
     allowed_hosts = tomllib.loads((out / "task.toml").read_text())["agent"]["allowed_hosts"]
 
+    assert allowed_hosts == harbor.DEFAULT_AGENT_ALLOWED_HOSTS == [
+        "api.openai.com",
+        "api.anthropic.com",
+        "repo.maven.apache.org",
+        "repo1.maven.org",
+        "plugins.gradle.org",
+        "services.gradle.org",
+        "downloads.gradle.org",
+        "maven.google.com",
+        "maven.pkg.jetbrains.space",
+        "maven.reposilite.com",
+        "oss.sonatype.org",
+        "kotlinlang.org",
+        "*.kotlinlang.org",
+        "kotl.in",
+        "*.jetbrains.com",
+        "docs.gradle.org",
+        "docs.gradle.com",
+        "gradle.com",
+        "gradle.org",
+        "scans.gradle.com",
+        "help.gradle.org",
+        "developer.android.com",
+        "schemas.android.com",
+        "pub.dartlang.org",
+        "pub.dev",
+        "pnpm.js.org",
+        "ant.apache.org",
+    ]
     assert "archive.ubuntu.com" not in allowed_hosts
-    assert "repo.maven.apache.org" not in allowed_hosts
-    assert "services.gradle.org" not in allowed_hosts
+    assert "repo.maven.apache.org" in allowed_hosts
+    assert "repo1.maven.org" in allowed_hosts
+    assert "plugins.gradle.org" in allowed_hosts
+    assert "services.gradle.org" in allowed_hosts
+    assert "downloads.gradle.org" in allowed_hosts
+    assert "maven.google.com" in allowed_hosts
+    assert "maven.pkg.jetbrains.space" in allowed_hosts
+    assert "maven.reposilite.com" in allowed_hosts
+    assert "oss.sonatype.org" in allowed_hosts
     # Agent setup installs the CLI before the agent-phase policy applies, so
-    # these are not needed here and would let the agent fetch upstream source.
+    # its package and source hosts are not needed here. Source hosting stays
+    # blocked so the agent cannot fetch the upstream implementation.
     assert "raw.githubusercontent.com" not in allowed_hosts
     assert "nodejs.org" not in allowed_hosts
     assert "registry.npmjs.org" not in allowed_hosts
@@ -131,7 +180,7 @@ def test_convert_instance_task_toml_enforces_programbench_policy(tmp_path):
     out = convert_instance(_calc_instance(), tmp_path, allowed_hosts=["api.example.com"])
     cfg = tomllib.loads((out / "task.toml").read_text())
 
-    # Inference is offline; only the model API host is reachable in the agent phase.
+    # An explicit allowlist replaces the defaults for the agent phase.
     assert cfg["environment"]["network_mode"] == "public"
     assert cfg["agent"]["network_mode"] == "allowlist"
     assert cfg["agent"]["allowed_hosts"] == ["api.example.com"]