Reframe AgentKernelArena for A/B experiments and improve agent setup

.gitignore

@@ -37,9 +37,12 @@ traj.json
 **/baseline_metrics.json
 **/profile.json
 
-# Generated held-out test data (methodology is in held_out/, data is private)
+# Generated held-out test data (methodology is in src/held_out/, data is private)
 held_out_tests/
 
 # Documentation build environment and output
 .docvenv/
 docs/_build/
+
+# Generated visualization data and manually collected report bundles
+.visualization/

CONTRIBUTING.md

@@ -4,31 +4,38 @@ Thanks for your interest in AgentKernelArena! This guide explains how to contrib
 
 ## Before You Start
 
-- Read `README.md` to understand the project scope — a standardized arena for evaluating LLM coding agents on GPU kernel optimization tasks.
-- Skim `config.yaml` to understand how agents, tasks, and LLM parameters are configured.
-- Ensure you have a supported GPU environment (AMD GPU with ROCm 6.4+ / 7.0+ / 7.1+) and Docker.
-- Confirm you have access to at least one supported agent (Cursor Agent, Claude Code, or Codex) and any required API keys.
+- Read `README.md` to understand the project scope: controlled A/B experiments and RL-ready feedback for GPU kernel agents.
+- Skim the files under `example_configs/` for run-level agent/task/GPU selection and the relevant `agents/<name>/agent_config.yaml` for agent-specific model and runtime settings.
+- Ensure you have an AMD GPU with ROCm-compatible Docker access; the supported workflow uses the pinned ROCm/SGLang images documented in the compatibility matrix.
+- Confirm that the selected agent integration and its authentication/dependencies are available.
 
 ## Development Setup
 
 Docker is the only supported path. All runs happen inside the pinned ROCm/SGLang
 container; see `docs/install/install.md`.
 
 ```bash
-# Verify the container can see Python, ROCm tools, GPU, and your agent logins
+# Verify the container can see Python, ROCm tools, and the GPU
 make docker-smoke
-make docker-check-agents
+
+# Select a run config and verify only its agent
+CONFIG_PATH=example_configs/quickstart_claude_mi300.yaml
+make docker-check-agents CONFIG="$CONFIG_PATH"
+
+# Optional strict check of all three first-class CLIs
+make docker-check-agents AGENTS=all
 
 # Optional: install the Cursor Agent CLI on the host (so it can be mounted)
 make install-cursor-agent
 
-# Optional: install FlyDSL into the container (only for flydsl2flydsl tasks)
+# Optional: install FlyDSL when the image lacks it (for all three FlyDSL task types)
 make docker-setup-flydsl
 
 # Optional: install local commit hooks
 pre-commit install
 
-# Optional: start a local vLLM server for self-hosted models
+# Optional: start a local OpenAI-compatible vLLM endpoint. Connecting an agent
+# to it is integration-specific; the endpoint does not reconfigure agents.
 make vllm
 ```
 
@@ -39,7 +46,7 @@ make vllm
 3. Run a smoke test against at least one task before submitting:
 
 ```bash
-make docker-run CONFIG=config.yaml
+make docker-run CONFIG=example_configs/quickstart_claude_mi300.yaml
 ```
 
 4. Open a Pull Request with motivation, impact, and verification steps.
@@ -48,7 +55,7 @@ make docker-run CONFIG=config.yaml
 
 - Follow PEP 8 for Python code.
 - Keep agent integrations isolated under `agents/<agent_name>/` — don't leak agent-specific logic into `src/`.
-- Update `config.yaml`, `README.md`, and the agent registry (`agents/__init__.py`) when adding a new agent.
+- Update the relevant example run configurations, docs, `AgentType`, and the launcher/handler branches in `src/module_registration.py` when adding a new agent. `agents/__init__.py` only provides the shared decorator registry.
 - Add documentation or comments when intent is non-obvious.
 - Performance timing helpers are generated into run workspaces from
   `src/tools/perf/`.
@@ -65,12 +72,12 @@ This project depends on GPU hardware/drivers and orchestrates external LLM agent
 
 - Test environment (GPU model, ROCm version, Docker image, OS)
 - Agent(s) used and their versions
-- Task category exercised (rocm-examples, rocprim, customer_hip, triton, torch2hip)
+- Task selector exercised (for example `hip2hip`, `triton2triton`, `instruction2triton`, `torch2hip`, a FlyDSL task type, or `repository`)
 - Key commands and output summary, e.g.:
 
 ```bash
-make docker-run CONFIG=config.yaml
-python compare_runs.py --runs <run-id-1> <run-id-2>
+make docker-run CONFIG=example_configs/quickstart_claude_mi300.yaml
+python3 compare_runs.py <run-directory-1> <run-directory-2>
 ```
 
 - For changes to scoring or evaluation logic, attach before/after results on at least one task category.
@@ -80,26 +87,26 @@ python compare_runs.py --runs <run-id-1> <run-id-2>
 
 Please include:
 
-- Reproduction steps (exact `config.yaml` snippet or command flags)
+- Reproduction steps (exact run-configuration snippet or command flags)
 - Expected vs actual behavior
 - Environment (OS, GPU, ROCm version, Python version, agent CLI version)
-- Relevant logs from the workspace under `works/` or a minimal repro
+- Relevant files from `logs/` and `workspace_<gpu>_<agent>/run_<timestamp>/`, or a minimal repro
 
 ## Security
 
 If you discover a security issue, do not open a public issue. Contact maintainers through a private channel.
 
-This project executes third-party AI agents in isolated workspaces — flag any sandbox-escape, credential-leakage, or supply-chain concerns privately.
+This project executes third-party AI agents permissively inside privileged Docker containers. Per-task workspaces are a reproducibility boundary, not a security sandbox; report unexpected access to mounted credentials, repository files, or host resources privately.
 
 ## Suggested Contributions
 
 - Add new agent integrations under `agents/`
-- Extend task coverage (new HIP, Triton, or Torch2HIP tasks under `tasks/`)
+- Extend task coverage across HIP, Triton, FlyDSL, PyTorch conversion, instruction-generated, or repository-level tasks
 - Improve scoring or fairness logic in `src/score.py`
-- Improve the leaderboard or visualization (`visualization/`)
+- Improve A/B comparison, experiment tracking, or visualization (`src/visualization/`)
 - Add support for new models / providers (OpenAI, Anthropic, OpenRouter, vLLM)
 - Improve docs, examples, and tests
 
 ## License
 
-By contributing, you agree that your contributions are licensed under the repository `LICENSE` (MIT).
+By contributing, you agree that your contributions are licensed under the repository `LICENSE` (Apache License 2.0).

Makefile

@@ -1,52 +1,64 @@
 # Makefile for AgentKernelArena — Docker-first workflow
 #
-# All benchmarking runs inside the pinned ROCm/SGLang container. Docker is the only
+# All experiments run inside the pinned ROCm/SGLang container. Docker is the only
 # supported path; the legacy host venv / `python main.py` workflow has been removed.
 # See src/scripts/docker_benchmark.sh and docs/install/install.md.
 
 SHELL := /bin/bash
 
 .PHONY: help docker-shell docker-check-agents docker-smoke docker-run docker-parallel-run docker-setup-flydsl \
-        check-docker-runner check-evaluator \
+        check-docker-runner check-evaluator check-held-out check-visualization \
+        visualization-build visualization-serve visualization-run \
         sync-perf-helpers check-perf-helpers materialize-perf-workspace \
         materialize-perf-task cleanup-works install-cursor-agent vllm
 
 help:
-	@echo "AgentKernelArena Evaluation Framework - Makefile Commands"
+	@echo "AgentKernelArena Experimentation Platform - Makefile Commands"
 	@echo "======================================================"
 	@echo "Docker-first workflow (the only supported path):"
-	@echo "make docker-shell        - Enter the benchmark Docker image with repo and agent auth mounted"
-	@echo "make docker-check-agents - Verify Codex, Claude Code, and Cursor Agent login reuse in Docker"
+	@echo "make docker-shell        - Enter the runtime image with repo and agent auth mounted"
+	@echo "make docker-check-agents - Verify the first-class host CLI selected by CONFIG"
+	@echo "                         Use CONFIG=... for another config; AGENTS=... overrides it"
+	@echo "                         AGENTS=all explicitly checks all three first-class CLIs"
 	@echo "make docker-smoke        - Verify Docker Python, ROCm tools, imports, and GPU access"
-	@echo "make docker-run CONFIG=config.yaml RUN_ARGS=\"--run-suffix test\" - Run benchmark in Docker"
-	@echo "make docker-parallel-run CONFIG=config.yaml GPU_IDS=0,1 - Run benchmark across one worker container per GPU"
+	@echo "make docker-run CONFIG=example_configs/quickstart_claude_mi300.yaml RUN_ARGS=\"--run-suffix test\" - Run an experiment in Docker"
+	@echo "make docker-parallel-run CONFIG=example_configs/benchmark_cursor_mi355x.yaml GPU_IDS=0,1 - Run an experiment across one worker container per GPU"
+	@echo "                         Default CONFIG is the MI300/MI300X Claude quickstart"
+	@echo "                         On other GPUs, pass a matching CONFIG explicitly"
 	@echo "                         Images: gfx942->mi30x, gfx950->mi35x; override with AKA_DOCKER_IMAGE=..."
-	@echo "make docker-setup-flydsl - Install FlyDSL into the container (needed for flydsl2flydsl tasks)"
+	@echo "make docker-setup-flydsl - Install FlyDSL when absent (for flydsl2flydsl, torch2flydsl, and triton2flydsl)"
 	@echo "make check-docker-runner - Check Docker runner syntax and runtime-specific arguments"
 	@echo "make check-evaluator     - Run centralized evaluator unit tests"
+	@echo "make check-held-out      - Run held-out module unit tests"
+	@echo "make visualization-run   - Build and serve the local comparison dashboard"
+	@echo "make check-visualization - Run visualization module unit tests"
 	@echo ""
 	@echo "Maintenance:"
 	@echo "make sync-perf-helpers   - Refresh committed perf-helper stubs in task sources"
 	@echo "make check-perf-helpers  - Verify task perf-helper stubs and markers are valid"
 	@echo "make materialize-perf-workspace WORKSPACE=workspace_x - Inject canonical perf helpers into workspace(s)"
 	@echo "make materialize-perf-task TASK=tasks/... OUT=/tmp/aka-task - Copy task(s) and inject canonical perf helpers"
 	@echo "make cleanup-works       - Remove workspace_* directories and logs"
-	@echo "make install-cursor-agent- Install the Cursor Agent CLI on the host"
+	@echo "make install-cursor-agent - Install the Cursor Agent CLI on the host"
 
 DOCKER_RUNNER := src/scripts/docker_benchmark.sh
-CONFIG ?= config.yaml
+CONFIG ?= example_configs/quickstart_claude_mi300.yaml
 RUN_ARGS ?=
+AGENTS ?=
 WORKSPACES ?= $(WORKSPACE)
 TASKS ?= $(TASK)
 OUT ?= /tmp/aka-materialized-tasks
 FORCE ?= 0
+VISUALIZATION_ARGS ?= --include-workspace-runs
+VISUALIZATION_HOST ?= 127.0.0.1
+VISUALIZATION_PORT ?= 8080
 MATERIALIZE_FORCE_ARG := $(if $(filter 1 true yes,$(FORCE)),--force,)
 
 docker-shell:
 	@$(DOCKER_RUNNER) shell
 
 docker-check-agents:
-	@$(DOCKER_RUNNER) check-agents
+	@AKA_AGENTS="$(AGENTS)" $(DOCKER_RUNNER) check-agents --config_name $(CONFIG)
 
 docker-smoke:
 	@$(DOCKER_RUNNER) smoke
@@ -57,8 +69,8 @@ docker-run:
 docker-parallel-run:
 	@GPU_IDS="$(GPU_IDS)" $(DOCKER_RUNNER) parallel-run --config_name $(CONFIG) $(RUN_ARGS)
 
-# Install FlyDSL into the container's persistent pip user-base (the base image does
-# not ship it). Run once per machine/image; needed only for flydsl2flydsl tasks.
+# Install FlyDSL into the container's persistent pip user-base when the selected
+# image does not ship it. Needed by all three FlyDSL task types.
 docker-setup-flydsl:
 	@$(DOCKER_RUNNER) setup-flydsl
 
@@ -68,6 +80,22 @@ check-docker-runner:
 check-evaluator:
 	@python3 -m unittest discover -s tests -p 'test_evaluator_*.py'
 
+check-held-out:
+	@python3 -m unittest discover -s tests -p 'test_held_out.py'
+
+check-visualization:
+	@python3 -m unittest discover -s tests -p 'test_visualization.py'
+
+visualization-build:
+	@python3 -m src.visualization build $(VISUALIZATION_ARGS)
+
+visualization-serve:
+	@python3 -m src.visualization serve --host "$(VISUALIZATION_HOST)" --port "$(VISUALIZATION_PORT)"
+
+visualization-run:
+	@python3 -m src.visualization run $(VISUALIZATION_ARGS) \
+		--host "$(VISUALIZATION_HOST)" --port "$(VISUALIZATION_PORT)"
+
 # Refresh committed perf-helper stubs/markers in task sources. Runtime workspaces
 # are materialized from src/tools/perf/ by setup_workspace().
 sync-perf-helpers:
@@ -97,7 +125,7 @@ install-cursor-agent:
 	@echo "Installing Cursor agent..."
 	@curl https://cursor.com/install -fsSL | bash
 
-# Run vLLM server with latest ROCm 6.4.1 and vLLM 0.10.1
+# Run the pinned local vLLM endpoint. Agent/provider wiring is integration-specific.
 vllm:
 	@if ss -ltn | grep ':30001 ' > /dev/null; then \
 		echo "vLLM server is already running on port 30001."; \
@@ -122,7 +150,7 @@ vllm:
 			--enable-auto-tool-choice \
 			--tool-call-parser hermes \
 			--trust-remote-code; \
-		echo "Don't forget to set local_llm_enabled: true in configs/config.yml"; \
+		echo "Configure a compatible agent integration to use the OpenAI-compatible endpoint at port 30001."; \
 		echo "vLLM server will be running on port 30001, please wait 3 minutes for it to start..."; \
 		echo "You can use docker logs -f container_id to check the server status"; \
 	fi