eval_runner: a re-evaluation that produces no results.json is scored from the previous run's file

[kai392]

Summary

evaluate_submission never clears the submission workspace between runs, so a re-evaluation that produces no results.json silently parses the previous run's file and reports the previous run's score as completed.

Why the existing guards don't catch it

Submission workspaces are keyed by submission id:

local_workspace   = _local_workspace(settings, submission.id)   # .../submissions/<submission_id>
local_results_path = local_workspace / "results.json"

and submission rows are reused, not recreated — create_pr_submission() updates the existing row when a miner pushes a new commit to the same PR (status is reset to queued and the job is re-enqueued). So the second, third, ... evaluation of a submission all write into the same directory, and nothing removes the previous results.json.

Both "did the run produce anything?" guards are therefore satisfied by the old file:

if completed != 0 and not local_results_path.exists():   # False - old file is still there
    raise subprocess.CalledProcessError(...)
...
metrics, score = _prepare_results(local_results_path, settings=settings)   # parses the stale payload

and _is_missing_results_payload() (added for #12) never sees results_missing, because the results were not missing — they were just from a different run.

The remote path has the same hole one level up: _remote_attempt() rsyncs the remote workspace back without --delete, and the remote workspace is likewise keyed by submission id, so a stale remote results.json is copied back even if this run wrote nothing.

cost_ledger.jsonl is affected the same way — the previous run's tokens are billed to the new evaluation whenever the eval command never starts.

Reproduction

1. Evaluate a submission successfully -> results.json contains {"results": {"TRINITY": {"accuracy": 0.91}}}, latest_score = 0.91.
2. Re-evaluate the same submission with an eval command that exits non-zero without writing anything (bad checkpoint, provider outage, remote box down after ssh succeeded, ...).
3. Expected: run failed, score = None.
   Actual: run completed, score = 0.91, cost_usd copied from run 1.

Impact

The wrong score is not confined to the run row — it propagates to submission.latest_score, best_eval_id, the public leaderboard, update_king_score(), and the worker's should_promote_submission() merge/close decision. A miner can bank one passing score and have any later broken checkpoint inherit it, which is both a data-integrity bug and a scoring-gaming vector. It also masks genuine evaluation failures, since a broken run looks like a clean pass.

Suggested fix

Clear the run outputs (results.json, cost_ledger.jsonl) at the start of evaluate_submission, and rm -f the remote results.json in the remote command prologue next to the ledger truncation that is already there. Both existing guards then behave as written.

Provider-route evaluations are keyed by a unique EvaluationRun.id, so their workspace is never reused and they are not affected.