Skip to content

Post-fire verify loses run-scoped acceptance commands #67

Description

@samfoy

Bug description

autoloop verify <project> <run-id> forgets run-scoped acceptance.verify_cmds supplied with autoloop run --set ... and can report an acceptance-verified run as unverifiable.

Steps to reproduce

  1. Launch a run with a run-scoped override:
autoloop run autocode "task" \
  --set 'acceptance.verify_cmds=python3 -m compileall -q .,python3 -m pytest -q'
  1. Allow it to complete. The registry can show:
{"status":"completed","acceptance_verified":true,"outcome":"verified"}
  1. Run independent post-fire verification:
autoloop verify /path/to/project <run-id> --json

Actual:

{"reconcile":"unverifiable","ranChecks":0,"failures":[]}

Root cause

In autoloop-cli/dist/commands/verify.js, post-fire verify does:

const cfg = config.loadProject(record.project_dir || projectDir);
const cmds = postFireCheckCommands(config.getList(cfg, "acceptance.verify_cmds"), ...)

The effective run-scoped overrides are not persisted in the run record, so a later process reloads only project/preset config and loses the commands used by the run.

The compatibility path is confirmed: setting AUTOLOOP_CONFIG to a temporary TOML containing the same acceptance.verify_cmds makes post-fire verify return confirmed.

Expected behavior

A later autoloop verify should reuse the exact effective acceptance contract from that run without requiring callers to resupply it.

Suggested solution

Persist the effective post-fire acceptance commands and timeout in run metadata/journal at launch, then have verify prefer the recorded immutable contract. Avoid reinterpreting mutable project config for historical runs.

Also consider preserving commands as a true array: AutoLoop 0.10.1's string-list parser splits commas inside command strings, so CSV serialization is lossy.

Environment

  • AutoLoop 0.10.1
  • Node 22.23.1
  • Linux

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions