Eval servers have no auth — anyone can run code and read held-out private scores; weather eval executes on the bare host

[EvolveAegis]

Summary

The recursive-self-improvement loop's eval infrastructure has three problems, all code-conclusive:

1. Zero-auth eval servers. benches/ale/eval_server.py (and benches/kernel/eval_server.py) accept unauthenticated HTTP: do_POST (eval_server.py:142-160) has no token/header check, and POST /session (any problem_id) creates a session for anyone. Any network peer can then POST /public (and /private) with arbitrary code, consuming the Docker session pool (unbounded sessions = DoS) and getting real evaluation results.

2. The held-out private scores are readable. POST /private (eval_server.py:149-150) returns {score, rank, performance} — the private/held-out split is the paper's anti-cheat mechanism, and this endpoint hands it to anyone who can reach the server (docs: eval_server.py:16 "POST /private ... -> {score, rank, performance}").

3. Weather eval runs agent-authored code on the bare host. src/weather/evalClient.ts:44-51: writeFileSync(join(dir, "model.py"), code) then execFile(PY, [RUN, dir, ...]) with only a SIGKILL timeout — no Docker, no sandbox. The inner agent's LLM output (which the loop writes and improves across generations) executes with host privileges.

Details

• benches/ale/eval_server.py:142-160 — do_POST routes /session, /public, /private, /problem, /usage, /close with no auth; session ids are uuids but not secrets (GET /health even lists active session ids, :137-139, making the no-auth endpoints trivially enumerable).
• benches/ale/eval_server.py:149-150 — /private → s.private_eval(...) → rank/performance returned to the caller.
• benches/kernel/eval_server.py:94-112 — same no-auth pattern for the GPU kernel eval (/eval compiles+runs arbitrary CUDA/host code on the GPU pod).
• src/weather/evalClient.ts:43-51 — evalModel: writes model.py + execFile on the host; the SIGKILL timeout is a compute-budget control, not a security control — no sandbox; benches/weather/run_model.py:22-23 then spec.loader.exec_module(m).
• The RSI loop writes these artifacts from the inner agent's LLM output (src/rsiLoop.ts:144-242 propose→critique→eval→verify), so attacker-influenced task content reaches the eval paths.

Threat model: the expected deployment is a localhost bench harness, but the surface still breaks it — same-host processes reach the ports, port-forwarding exposes them, and the evolving agent itself can query /private to read its own held-out score and game the selection loop.

How to reproduce

curl -X POST http://<host>:8137/session -H 'Content-Type: application/json' \
  -d '{"problem_id": "<any>", "lite": true}'            # -> session_id (no credentials)
curl -X POST http://<host>:8137/private -H 'Content-Type: application/json' \
  -d '{"session_id": "<above>", "code": "..."}'         # -> {score, rank, performance} (held-out data)
# GET /health lists active session ids (enumerability)

Impact

• Anti-cheat integrity: anyone who can reach the server reads the held-out private scores (the paper's core evaluation guarantee) and can game/subvert the loop's fitness signal.
• Resource abuse / DoS via unbounded Docker sessions.
• Code execution on the host via the weather eval path (agent-authored model.py runs without a sandbox; the timeout is budget, not isolation).

Note: code-level findings; no live deployed instance was verified (the ale_bench dependency is not installable here).

Suggested change

• Require a token on all eval server routes (and bind loopback by default).
• Gate /private behind the experiment owner's credential (or remove the endpoint from the public surface).
• Run weather eval in Docker like the ALE path, with a network-isolated container.