Live evaluation progress is never visible: the worker holds one uncommitted transaction for the whole run

[kai392]

Summary

eval_runner parses the harness's [submission] item i/N ... output and records it on the run (phase, message, progress_current, progress_total). /api/submissions/{id} exposes those as current_phase / current_message / current_progress_current / current_progress_total, and the public submission page renders them.

None of it ever reaches a reader while the evaluation is running. The progress fields stay empty for the whole run and only appear once it has already finished, and /api/jobs reports a claimed job as queued for hours.

Root cause

Both writers flush instead of committing:

# services/eval_runner.py::_touch_progress
    submission.updated_at = _utcnow()
    session.flush()

# worker.py::process_once
    job.status = "running"
    job.claimed_by = "worker"
    ...
    session.flush()

A flush writes into the worker's own transaction. process_once() opens exactly one session and does not commit until after the job body returns:

result = evaluate_submission(session, submission, runtime_settings, ...)
job.status = "completed" if result.run.status == "completed" else "failed"
...
session.commit()          # <- first commit, up to EVAL_TIMEOUT_SECONDS later

EVAL_TIMEOUT_SECONDS defaults to 7200. Under Postgres READ COMMITTED the API process cannot see uncommitted rows, so for up to two hours:

• every _touch_progress() call is written and then discarded from the reader's point of view;
• job.status, claimed_by, claimed_at, heartbeat_at stay at their pre-claim values in /api/jobs;
• submission.status = "running" is invisible, so the submission page shows the previous state.

The whole _PROGRESS_START_RE / _PROGRESS_DONE_RE / _consume_progress_line path — which exists only to drive live progress — is dead in production as a result.

A second, non-user-visible consequence is that a single evaluation keeps one Postgres transaction open for hours. That is the contention ensure_schema()'s lock_timeout and its _add_column_if_missing catalog pre-check were added to work around.

Reproduction

1. Queue a submission and start the worker.
2. Poll GET /api/submissions/{id} (or open /submission/{id}) while the evaluation runs.
3. Expected: current_phase moves through remote_gpu -> evaluation_running and current_progress_current climbs toward current_progress_total.
   Actual: all four fields stay null until the run ends, then jump straight to their final values. GET /api/jobs shows the job as queued throughout.

Suggested fix

Commit rather than flush in _touch_progress() and at the worker's claim point. Committing the claim also makes it durable and releases the SELECT ... FOR UPDATE row lock early; the committed queued -> running transition is what actually keeps other workers off the row, so skip_locked claiming still behaves correctly.

Related, not duplicate

• Validator worker re-queues submissions forever after uncaught evaluation errors #50 covers the error path (a rollback restoring status="queued" and retrying forever) and asks for dead-lettering. This issue is about the normal path being unobservable. Committing the claim removes one of Validator worker re-queues submissions forever after uncaught evaluation errors #50's symptoms but does not implement the dead-letter behaviour it asks for.
• Separately, _local_attempt() never forwards an on_line callback to _run_bash_stream(), so local and local-fallback runs parse no progress lines at all, even once the updates are durable. Happy to send that as its own change.