Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/architecture/operating-loop.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,8 @@ Do not run full ceremony for every observation. Promote progressively:

The ratchet is what keeps `.agents/` from becoming a landfill. Compounding only happens when capture meets pruning.

**R3 self-enforcement (no learning without a constraint).** The "Must never regress → add a validation gate" rung used to be prose only — a learning could be promoted to a durable maturity tier without ever compiling into a gate/test/rule. `scripts/check-ratchet-r3-constraint.sh` enforces it against the live (gitignored) `.agents/learnings/` corpus: any durable-tier learning (`candidate`/`established`/`canonical`/`stable`/`promoted`) that cites no constraint — a `scripts/`/`.github/workflows/` gate, a `_test.go`/`tests/` reference, a `skills/**/SKILL.md` step, or a `constraint:`/`enforced_by:` frontmatter field — is flagged. Warn-only by default; `--strict` (or `RATCHET_R3_BLOCKING=true`) makes it blocking, mirroring the same warn-then-fail ladder. A CI path-filter gate is intentionally *not* used because the learnings corpus is gitignored (dead-by-design, like the retired learning-coherence job); the script's own correctness is gated by `tests/scripts/check-ratchet-r3-constraint.bats`.

## Skill → loop-move map

| Loop move | Primary skills | Produces |
Expand Down
168 changes: 168 additions & 0 deletions scripts/check-ratchet-r3-constraint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
#!/usr/bin/env bash
set -euo pipefail

# check-ratchet-r3-constraint.sh — enforce Ratchet rule R3:
# "no learning without a constraint."
#
# Doctrine (docs/3.0.md, docs/architecture/operating-loop.md): a learning is
# durable only when it COMPILES into a gate/test/rule. The ratchet mechanism
# (cli/internal/ratchet/) already models maturity promotion, but R3 itself was
# never enforced — promotion to a durable tier was a manual operator decision
# with no check that the learning actually produced a constraint. This script
# closes that gap: a durable-tier learning that cites NO constraint is flagged.
#
# Why a script (not a CI path-filter gate): the operator's learnings corpus at
# .agents/learnings/** is gitignored (see .gitignore; the retired
# learning-coherence CI job is dead-by-design for the same reason). R3 therefore
# enforces against the LIVE local corpus at ratchet/flywheel time, and its own
# correctness is gated in CI by tests/scripts/check-ratchet-r3-constraint.bats.
#
# Promotion ladder (mirrors the ratchet's own warn-then-fail pattern, itself a
# canonical learning: .agents/learnings/pattern/users-agentops-warn-then-fail-ratchet.md):
# - Default: WARN-only. Flags durable learnings missing a constraint, exit 0.
# - Strict: --strict OR RATCHET_R3_BLOCKING=true -> the same gaps FAIL, exit 1.
#
# A learning is "durable-tier" when maturity is one of: candidate, established,
# canonical, stable, promoted (i.e. past provisional). Provisional learnings are
# still forming and are exempt — R3 binds only once a learning claims durability.
#
# A learning "cites a constraint" when EITHER:
# 1. Frontmatter carries a constraint-link field:
# constraint / enforced_by / gate / ratchet_gate / compiled_to / enforces
# 2. The body references a concrete constraint surface:
# - a path under scripts/ (a *.sh gate)
# - a path under .github/workflows/ (a CI gate)
# - a *_test.go / tests/ reference (a test)
# - a skills/**/SKILL.md reference (an encoded skill step / rule)
# - an explicit "Constraint:" or "Enforced-by:" line
#
# Usage:
# scripts/check-ratchet-r3-constraint.sh [LEARNINGS_DIR]
# RATCHET_R3_BLOCKING=true scripts/check-ratchet-r3-constraint.sh
# scripts/check-ratchet-r3-constraint.sh --strict [LEARNINGS_DIR]
#
# Exit codes: 0 = pass (or warn-only), 1 = blocking failures (strict mode).

STRICT="${RATCHET_R3_BLOCKING:-false}"
VERBOSE="${VERBOSE:-false}"

POSITIONAL=()
for arg in "$@"; do
case "$arg" in
--strict) STRICT=true ;;
--verbose) VERBOSE=true ;;
*) POSITIONAL+=("$arg") ;;
esac
done

LEARNINGS_DIR="${POSITIONAL[0]:-.agents/learnings}"

FLAGGED=0
CHECKED=0
DURABLE=0

log() { [[ "$VERBOSE" == "true" ]] && echo "$@" || true; }

is_learning_artifact() {
local basename
basename=$(basename "$1")
[[ "$basename" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}-.*\.md$ ]] || \
[[ "$basename" =~ ^(learn|learning|pattern|users-).*\.md$ ]]
}

# Extract frontmatter (between the first and second '---' fences).
frontmatter_of() {
awk 'NR==1 && $0!="---"{exit} NR==1{next} /^---$/{exit} {print}' "$1"
}

# Extract body (everything after the second '---').
body_of() {
awk 'BEGIN{n=0} /^---$/{n++; if(n==2){found=1; next}} found{print}' "$1"
}

is_durable() {
# $1 = frontmatter text
echo "$1" | grep -qiE '^maturity:[[:space:]]*(candidate|established|canonical|stable|promoted)\b'
}

cites_constraint() {
local frontmatter="$1" body="$2"
# 1. Frontmatter constraint-link field (must have a non-empty value).
if echo "$frontmatter" | grep -qiE '^(constraint|enforced_by|gate|ratchet_gate|compiled_to|enforces):[[:space:]]*[^[:space:]]'; then
return 0
fi
# 2. Body references a concrete constraint surface.
if echo "$body" | grep -qE 'scripts/[A-Za-z0-9_./-]+\.sh|\.github/workflows/|[A-Za-z0-9_./-]+_test\.go|(^|[[:space:]])tests/|skills/[A-Za-z0-9_./-]+/SKILL\.md'; then
return 0
fi
# 3. Explicit constraint declaration line in the body.
if echo "$body" | grep -qiE '^[[:space:]]*(Constraint|Enforced-by):[[:space:]]*[^[:space:]]'; then
return 0
fi
return 1
}

check_file() {
local file="$1" basename
basename=$(basename "$file")
[[ "$file" == *.md ]] || return 0
is_learning_artifact "$file" || return 0

CHECKED=$((CHECKED + 1))
local frontmatter body
frontmatter=$(frontmatter_of "$file")
body=$(body_of "$file")

# No frontmatter => no maturity claim => not durable-tier; exempt.
[[ -n "$frontmatter" ]] || { log "skip(no-frontmatter): $basename"; return 0; }

if ! is_durable "$frontmatter"; then
log "skip(provisional): $basename"
return 0
fi
DURABLE=$((DURABLE + 1))

if cites_constraint "$frontmatter" "$body"; then
log "PASS: $basename (durable + constraint cited)"
return 0
fi

local rel="$file"
if [[ "$STRICT" == "true" ]]; then
echo "FAIL R3: $rel — durable-tier learning cites NO constraint (gate/test/SKILL/rule)"
else
echo "WARN R3: $rel — durable-tier learning cites NO constraint (gate/test/SKILL/rule)"
fi
FLAGGED=$((FLAGGED + 1))
}

main() {
if [[ ! -d "$LEARNINGS_DIR" ]]; then
echo "R3: no learnings directory at $LEARNINGS_DIR — nothing to check"
exit 0
fi

while IFS= read -r -d '' file; do
check_file "$file"
done < <(find "$LEARNINGS_DIR" -type f -name '*.md' -print0 2>/dev/null)

echo ""
echo "R3 constraint check: $CHECKED learnings scanned, $DURABLE durable-tier, $FLAGGED missing a constraint"

if [[ "$FLAGGED" -eq 0 ]]; then
echo "R3 gate passed — every durable-tier learning compiles into a constraint"
exit 0
fi

if [[ "$STRICT" == "true" ]]; then
echo "R3 gate FAILED (strict): $FLAGGED durable-tier learning(s) without a constraint"
echo " Fix: add a constraint (gate/test/SKILL step) and cite it, or demote to maturity: provisional."
exit 1
fi

echo "R3 gate WARN-ONLY: $FLAGGED durable-tier learning(s) without a constraint"
echo " Set RATCHET_R3_BLOCKING=true (or pass --strict) to make this blocking."
exit 0
}

main
147 changes: 147 additions & 0 deletions tests/scripts/check-ratchet-r3-constraint.bats
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
#!/usr/bin/env bats
#
# Tests for scripts/check-ratchet-r3-constraint.sh — the Ratchet R3 gate
# ("no learning without a constraint"). Runs against a synthetic learnings
# fixture (NOT the operator's gitignored .agents/learnings/ corpus), so the
# test is hermetic and committed-and-gated in CI even though the real corpus
# is local-only.

setup() {
REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../.." && pwd)"
SCRIPT="$REPO_ROOT/scripts/check-ratchet-r3-constraint.sh"

TMP_DIR="$(mktemp -d)"
LEARN="$TMP_DIR/learnings"
mkdir -p "$LEARN"
}

teardown() {
rm -rf "$TMP_DIR"
}

# Helper: write a learning file with given maturity + body.
write_learning() {
local name="$1" maturity="$2" body="$3"
cat > "$LEARN/$name" <<EOF
---
type: learning
maturity: $maturity
confidence: high
---

$body
EOF
}

@test "durable learning that cites a script gate passes" {
write_learning "2026-06-06-durable-with-gate.md" "established" \
"Encoded as a gate in scripts/check-ratchet-r3-constraint.sh so it can't regress."

run bash "$SCRIPT" "$LEARN"
[ "$status" -eq 0 ]
[[ "$output" == *"R3 gate passed"* ]]
}

@test "durable learning that cites a frontmatter constraint field passes" {
cat > "$LEARN/2026-06-06-fm-constraint.md" <<'EOF'
---
type: learning
maturity: canonical
enforced_by: .github/workflows/validate.yml::process-hygiene
---

The doctrine rule is now a CI gate.
EOF

run bash "$SCRIPT" "$LEARN"
[ "$status" -eq 0 ]
[[ "$output" == *"R3 gate passed"* ]]
}

@test "durable learning with NO constraint warns but exits 0 (warn-first)" {
write_learning "2026-06-06-durable-no-constraint.md" "established" \
"We learned that warm caches matter, but never compiled it into anything."

run bash "$SCRIPT" "$LEARN"
[ "$status" -eq 0 ]
[[ "$output" == *"WARN R3"* ]]
[[ "$output" == *"WARN-ONLY"* ]]
}

@test "durable learning with NO constraint FAILS under --strict" {
write_learning "2026-06-06-durable-no-constraint.md" "established" \
"We learned that warm caches matter, but never compiled it into anything."

run bash "$SCRIPT" --strict "$LEARN"
[ "$status" -eq 1 ]
[[ "$output" == *"FAIL R3"* ]]
[[ "$output" == *"R3 gate FAILED (strict)"* ]]
}

@test "RATCHET_R3_BLOCKING=true is equivalent to --strict" {
write_learning "2026-06-06-durable-no-constraint.md" "candidate" \
"Durable claim with no gate, test, or SKILL behind it."

RATCHET_R3_BLOCKING=true run bash "$SCRIPT" "$LEARN"
[ "$status" -eq 1 ]
[[ "$output" == *"R3 gate FAILED (strict)"* ]]
}

@test "provisional learning without a constraint is exempt" {
write_learning "2026-06-06-provisional.md" "provisional" \
"Early observation, not yet durable, no constraint expected."

run bash "$SCRIPT" --strict "$LEARN"
[ "$status" -eq 0 ]
[[ "$output" == *"0 missing a constraint"* ]]
}

@test "durable learning citing a test surface passes" {
write_learning "2026-06-06-durable-test.md" "stable" \
"Regression-guarded in cli/internal/ratchet/gate_test.go::TestR3."

run bash "$SCRIPT" --strict "$LEARN"
[ "$status" -eq 0 ]
[[ "$output" == *"R3 gate passed"* ]]
}

@test "durable learning citing a SKILL step passes" {
write_learning "2026-06-06-durable-skill.md" "established" \
"Encoded as a step in skills/evolve/SKILL.md."

run bash "$SCRIPT" --strict "$LEARN"
[ "$status" -eq 0 ]
[[ "$output" == *"R3 gate passed"* ]]
}

@test "empty frontmatter field does NOT count as a constraint citation" {
cat > "$LEARN/2026-06-06-empty-field.md" <<'EOF'
---
type: learning
maturity: established
constraint:
---

Has a constraint key but no value — must still be flagged.
EOF

run bash "$SCRIPT" --strict "$LEARN"
[ "$status" -eq 1 ]
[[ "$output" == *"FAIL R3"* ]]
}

@test "missing learnings directory is a clean no-op" {
run bash "$SCRIPT" "$TMP_DIR/does-not-exist"
[ "$status" -eq 0 ]
[[ "$output" == *"nothing to check"* ]]
}

@test "mixed corpus reports correct durable + flagged counts" {
write_learning "2026-06-06-a-good.md" "established" "Gated via scripts/foo.sh."
write_learning "2026-06-06-b-bad.md" "established" "No constraint here."
write_learning "2026-06-06-c-prov.md" "provisional" "Still forming."

run bash "$SCRIPT" "$LEARN"
[ "$status" -eq 0 ]
[[ "$output" == *"2 durable-tier, 1 missing a constraint"* ]]
}
Loading