Skip to content

[Golden Test] Modularize the golden-verify agent into export-kit so it can be installed on other projects #1196

Description

@PeterJhongLinksys

Goal

Make the golden-verify daily triage agent installable elsewhere — another repository, another maintainer, another machine — by folding it into the existing export-kit module pattern instead of leaving it as a one-off wired to one laptop.

Where it stands

export-kit already solves this for the other pipelines. Its stated design is that adding a project means filling in a yaml and the engine never changes, and it ships the supporting scaffolding: SETUP.md, identity.example.yaml, .env.example, example.yaml, prompts/ with neutralized agent prompts, state-schema/ with seeds and an init script, plus PROMPT-CUSTOMIZATION.md and SKILLS-GENERALIZATION.md.

golden_verify is the one pipeline that never got folded in. Its collector sits under pipelines/golden_verify/ — so the code is in the right place — but everything around it is single-instance:

  • The playbook prompt is not in the kit and is not neutralized. It hardcodes absolute paths under one user's home directory, one repo pair, one Telegram destination, and one clone location. export-kit/prompts/ has no golden-verify equivalent.
  • No config template. example.yaml has no golden_verify block, so there is nothing to copy from; the only working example is one project's live config.
  • No setup path. SETUP.md walks through triage / design / dispatch / pr_review and does not mention this pipeline, so a new installer has no way to discover it exists, let alone enable it.
  • Defaults are baked into the collector. Repo names, the pages base URL, the report subpath and the workflow name are module-level constants that a config file happens to override. Anyone running it without a config silently targets someone else's repository.
  • No skill layer. The knowledge lives in a local skill outside the kit, while the kit's own skills/ carries generalized guides for the other pipelines.

Proposed work

  1. Neutralize the playbook into export-kit/prompts/golden-daily.md using the same placeholder conventions as the existing prompts: no absolute home paths, no hardcoded repo pair, no baked-in notification target. Keep the live copy as the reference implementation.
  2. Add a golden_verify block to example.yaml with every knob and a one-line comment each — including the two systemic-breaker thresholds and why both exist, since a newcomer will otherwise reach for the volume knob alone.
  3. Remove the collector's baked-in defaults, or make them fail loudly when no config is supplied. Silently defaulting to another organization's repository is the worst possible fallback.
  4. Extend SETUP.md with a golden-verify section: prerequisites (an orchestration repo publishing a verify report to Pages, a baseline branch, the label and project used for tracking issues), how to enable the feature switch, and how to do the two-phase validation before going live.
  5. Document the cron shape, since it is easy to get wrong: a thin shim job whose first action is to read the playbook, a pinned vision-capable model, and deliver: local rather than a chat target — because the playbook pushes its own notification, so pointing deliver at the chat produces two messages per day.
  6. Add a skill under the kit's skills/ covering operation and the load-bearing rules, generalized in the style of the existing pr-review-automation-ops.
  7. State the portability constraint the module depends on: config files use relative paths, resolved against the automation root. Absolute paths in a config are a defect — align the consumer instead.

Non-goals

Rewriting the judgment logic. This is packaging: same behaviour, new install target.

Acceptance

A second project can be brought up by filling in a yaml plus the identity file, with no edits to the collector and no edits to the playbook beyond its documented placeholders — and the two-phase validation can be run against it before it is allowed to write anything.

Metadata

Metadata

Labels

2.xLabeled for 2.x version

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions