Skip to content

/harness update may not absorb new mandatory AGENTS guidance into target repos #36

Description

@baskduf

Summary

When an existing target repository is refreshed with /harness update, new starter-kit AGENTS.md rules may only be partially absorbed into the target AGENTS.md. This is usually correct because the update workflow avoids wholesale overwrites, but it can cause newly mandatory reporting contracts to be missed.

Observed during dogfood after commit 416409d (Add task outcome evidence gate):

  • The agent recorded task outcome evidence and first-pass verification details.
  • The final report did not include the exact required line:
    • Task outcome evidence: recorded in <path>
    • or Task outcome evidence: skipped because <reason>
  • This suggests the target repo had enough guidance to produce evidence, but not enough of the new final-report contract.

Why this matters

The current safe-update behavior is valuable and should stay. The issue is that some starter-kit guidance is effectively mandatory for consistent harness behavior. If /harness update treats all AGENTS.md changes as ordinary reference material, existing target repos can drift from the kit's expected reporting and evidence rules.

Desired behavior

/harness update should preserve target-specific guidance, but it should also detect and surface missing mandatory guidance blocks from the starter kit.

Possible approach:

  • Mark or document a small set of starter-kit guidance blocks as mandatory update candidates.
  • During /harness update, compare target AGENTS.md against those mandatory blocks.
  • If a block is missing, either patch it safely or report it as a manual-review item.
  • Avoid wholesale replacement of target AGENTS.md.

Candidate mandatory block

The task outcome evidence final-report contract should be present in target guidance:

Before the final report, decide whether task outcome evidence is needed.

Record task outcome evidence for product tasks, substantial harness-maintenance work, harness gate changes, benchmark/effectiveness changes, failed first-pass verification, known failure paths, or cross-environment mismatches.

Skip only trivial docs-only wording, typo, link-label, or formatting changes.

The final report must include exactly one of:
- Task outcome evidence: recorded in <path>
- Task outcome evidence: skipped because <reason>

Acceptance criteria

  • /harness update guidance distinguishes target-specific adaptation from missing mandatory starter-kit rule absorption.
  • Existing target AGENTS.md files are still not overwritten wholesale.
  • The update report calls out missing mandatory guidance as applied, deferred, or manual review.
  • A test or repository-hygiene check covers the presence of the task outcome evidence final-report contract in starter-kit guidance and/or update workflow docs.

Priority

P2. This does not block product work and does not indicate data loss, but it weakens evidence consistency across dogfood runs and updated target repos.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions