Fix setup skill frontmatter validation and add testing milestone

reference/testing-milestones.md

@@ -88,6 +88,46 @@ All 15 primitives validated successfully.
 
 ---
 
+## Milestone 1b: Skill Frontmatter Validation
+
+**What it tests:** Generated skills in `.claude/skills/` retain YAML frontmatter format (not markdown tables or malformed blocks) and include required setup fields.
+
+**Prerequisites:**
+- A generated vault from /setup with skills in `.claude/skills/`
+- `validate-setup.sh` accessible at `./reference/validate-setup.sh`
+
+**Pass criteria:** `validate-setup.sh` returns zero FAILs for the generated vault.
+
+**Verification steps:**
+
+```bash
+# Run setup-frontmatter validation against the generated vault
+./reference/validate-setup.sh /path/to/generated-vault
+
+# Or validate a single skill
+./reference/validate-setup.sh /path/to/generated-vault reduce
+```
+
+**Expected output on success:**
+
+```
+=== Skill Frontmatter Validation: /path/to/generated-vault/.claude/skills ===
+  PASS ...
+=== Skill Frontmatter Summary ===
+  FAIL: 0
+All N skill(s) have valid frontmatter.
+```
+
+**Common failure modes and remediation:**
+
+| Failure | Cause | Fix |
+|---------|-------|-----|
+| First line is a table row (for example `\| name \| ...`) | Frontmatter was emitted as markdown table instead of YAML | Re-read source template frontmatter from `skill-sources/[name]/SKILL.md`, vocabulary-transform values, replace only frontmatter |
+| Missing required field (`name`, `description`, `user-invocable`, `allowed-tools`, `context`, `model`) | Field dropped during generation | Regenerate frontmatter from source template; preserve the skill body |
+| Invalid line inside frontmatter block | Frontmatter closing delimiter or YAML shape was mangled | Regenerate frontmatter block only |
+
+---
+
 ## Milestone 2: Context File Composition
 
 **What it tests:** The generated context file (CLAUDE.md) contains all required sections with domain-appropriate content. No placeholder variables remain. No orphaned references to features that were not enabled.
@@ -867,6 +907,7 @@ Each preset should produce:
 
 # 2. Run milestones in order
 echo "=== Milestone 1: Kernel ===" && ./reference/validate-kernel.sh /tmp/test-research
+echo "=== Milestone 1b: Skill Frontmatter ===" && ./reference/validate-setup.sh /tmp/test-research
 echo "=== Milestone 2: Context ==="  # (run section checks manually or via script above)
 echo "=== Milestone 3: Vocabulary ===" # (run against therapy vault)
 echo "=== Milestone 4: Pipeline ===" # (requires active agent session)
@@ -880,6 +921,7 @@ echo "=== Milestone 6: Presets ===" # (run for all 3 presets)
 
 ```
 M1 (Kernel) ← no dependencies
+M1b (Skill Frontmatter) ← no dependencies
 M2 (Context) ← M1 (kernel must pass first)
 M3 (Vocabulary) ← M2 (context must exist)
 M4 (Pipeline) ← M1 + M2 (kernel + context)
@@ -889,4 +931,4 @@ M5c (Condition-Based) ← M1 + M2 (kernel + context)
 M6 (Presets) ← M1 + M2 + M3 (validates all three for each preset)
 ```
 
-Milestones 1-3 can be automated as a CI check. Milestones 4-5c require an active agent session. Milestone 6 requires generating multiple vaults and is best run as a manual test suite.
+Milestones 1, 1b, 2, and 3 can be automated as a CI check. Milestones 4-5c require an active agent session. Milestone 6 requires generating multiple vaults and is best run as a manual test suite.

reference/validate-setup.sh

@@ -0,0 +1,190 @@
+#!/usr/bin/env bash
+# validate-setup.sh — Validate YAML frontmatter on generated skills
+# Usage:
+#   ./validate-setup.sh                          # Validate all skills in .claude/skills/
+#   ./validate-setup.sh reduce                   # Validate a single skill in current vault
+#   ./validate-setup.sh /path/to/vault           # Validate all skills in target vault
+#   ./validate-setup.sh /path/to/vault reduce    # Validate a single skill in target vault
+#   VAULT=/path/to/vault ./validate-setup.sh     # Equivalent via env var
+
+ARG1="${1:-}"
+ARG2="${2:-}"
+VAULT="${VAULT:-.}"
+SKILL_NAME=""
+
+# Backward compatible argument parsing:
+# - One arg -> skill name (unless it looks like a vault path)
+# - Two args -> [vault-path] [skill-name]
+if [ -n "$ARG1" ]; then
+    if [ -d "$ARG1" ] || [[ "$ARG1" == */* ]]; then
+        VAULT="$ARG1"
+        SKILL_NAME="$ARG2"
+    else
+        SKILL_NAME="$ARG1"
+    fi
+fi
+
+SKILLS_DIR="$VAULT/.claude/skills"
+
+PASS=0
+WARN=0
+FAIL=0
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m'
+
+pass() { echo -e "  ${GREEN}PASS${NC} $1"; PASS=$((PASS + 1)); }
+warn() { echo -e "  ${YELLOW}WARN${NC} $1"; WARN=$((WARN + 1)); }
+fail() { echo -e "  ${RED}FAIL${NC} $1"; FAIL=$((FAIL + 1)); }
+
+# Validate a single skill's SKILL.md frontmatter
+# Returns 0 on pass, 1 on failure
+validate_skill() {
+    local skill_dir="$1"
+    local skill_file="$skill_dir/SKILL.md"
+    local name
+    name=$(basename "$skill_dir")
+
+    if [ ! -f "$skill_file" ]; then
+        fail "$name: SKILL.md not found in $skill_dir"
+        return 1
+    fi
+
+    local failed=0
+
+    # Check 1: First line must be '---'
+    local first_line
+    first_line=$(head -1 "$skill_file")
+    if [ "$first_line" != "---" ]; then
+        fail "$name: first line is '$(echo "$first_line" | head -c 40)' — expected '---'. Frontmatter needs regeneration."
+        failed=1
+    else
+        pass "$name: opens with '---'"
+    fi
+
+    # Check 2: Closing '---' delimiter (frontmatter is complete)
+    # Find the first delimiter after line 1 in absolute line numbers.
+    local closing_line
+    closing_line=$(awk 'NR > 1 && $0 == "---" { print NR; exit }' "$skill_file")
+    if [ -z "$closing_line" ]; then
+        fail "$name: no closing '---' delimiter found. Frontmatter is incomplete — needs regeneration."
+        failed=1
+    else
+        pass "$name: closing '---' delimiter present"
+    fi
+
+    # Check 3: Required fields and structure in frontmatter
+    # Extract frontmatter (between first and second '---')
+    if [ -n "$closing_line" ]; then
+        if [ "$closing_line" -le 2 ]; then
+            fail "$name: frontmatter block is empty."
+            failed=1
+            return $failed
+        fi
+
+        local frontmatter
+        frontmatter=$(sed -n "2,$((closing_line - 1))p" "$skill_file")
+
+        # Structural sanity check:
+        # frontmatter must stay as YAML key:value lines, not markdown table/body text.
+        local invalid_line
+        invalid_line=$(echo "$frontmatter" | awk 'NF && $0 !~ /^[A-Za-z0-9_-]+:[[:space:]]*.*$/ { print; exit }')
+        if [ -n "$invalid_line" ]; then
+            fail "$name: invalid frontmatter line '$(echo "$invalid_line" | head -c 60)'. Expected YAML key:value format."
+            failed=1
+        else
+            pass "$name: frontmatter structure is YAML-like"
+        fi
+
+        # Required fields shared by all source skills.
+        for field in "name" "description" "user-invocable" "allowed-tools" "context" "model"; do
+            if echo "$frontmatter" | grep -Eq "^${field}:[[:space:]]*.+$"; then
+                pass "$name: has '${field}:' field"
+            else
+                fail "$name: missing or empty '${field}:' in frontmatter. Frontmatter needs regeneration."
+                failed=1
+            fi
+        done
+
+        # Value checks that /setup explicitly enforces.
+        local context_value
+        context_value=$(echo "$frontmatter" | sed -n 's/^context:[[:space:]]*//p' | head -1 | sed -E 's/^[[:space:]]+//; s/[[:space:]]+$//; s/^["'\'']//; s/["'\'']$//')
+        if [ -n "$context_value" ] && [ "$context_value" != "fork" ]; then
+            fail "$name: context is '$context_value' (expected 'fork')."
+            failed=1
+        fi
+
+        local invocable_value
+        invocable_value=$(echo "$frontmatter" | sed -n 's/^user-invocable:[[:space:]]*//p' | head -1 | sed -E 's/^[[:space:]]+//; s/[[:space:]]+$//; s/^["'\'']//; s/["'\'']$//')
+        if [ -n "$invocable_value" ] && [ "$invocable_value" != "true" ]; then
+            fail "$name: user-invocable is '$invocable_value' (expected 'true')."
+            failed=1
+        fi
+    fi
+
+    return $failed
+}
+
+# --- Single-skill mode ---
+if [ -n "$SKILL_NAME" ]; then
+    # Find the skill directory (may have a domain prefix)
+    skill_dir=""
+    for candidate in "$SKILLS_DIR/$SKILL_NAME" "$SKILLS_DIR"/*"$SKILL_NAME"; do
+        [ -d "$candidate" ] && skill_dir="$candidate" && break
+    done
+
+    if [ -z "$skill_dir" ]; then
+        echo -e "${RED}FAIL${NC} Skill directory not found for '$SKILL_NAME' in $SKILLS_DIR"
+        exit 1
+    fi
+
+    echo "=== Skill Frontmatter Validation: $(basename "$skill_dir") ==="
+    echo ""
+    validate_skill "$skill_dir"
+    result=$?
+    echo ""
+    if [ $result -eq 0 ]; then
+        echo -e "${GREEN}Frontmatter valid.${NC}"
+        exit 0
+    else
+        echo -e "${RED}Frontmatter invalid — needs regeneration. Preserve the skill body, replace only the frontmatter.${NC}"
+        exit 1
+    fi
+fi
+
+# --- Whole-vault mode ---
+echo "=== Skill Frontmatter Validation: $SKILLS_DIR ==="
+echo ""
+
+skill_count=0
+fail_count=0
+
+for skill_dir in "$SKILLS_DIR"/*/; do
+    [ ! -d "$skill_dir" ] && continue
+    [ ! -f "$skill_dir/SKILL.md" ] && continue
+    skill_count=$((skill_count + 1))
+    validate_skill "$skill_dir" || fail_count=$((fail_count + 1))
+done
+
+if [ "$skill_count" -eq 0 ]; then
+    echo -e "${RED}No skills found in $SKILLS_DIR${NC}"
+    exit 1
+fi
+
+echo ""
+echo "=== Skill Frontmatter Summary ==="
+echo -e "  ${GREEN}PASS:${NC} $PASS"
+echo -e "  ${RED}FAIL:${NC} $FAIL"
+echo "  Skills checked: $skill_count"
+echo ""
+
+if [ "$fail_count" -eq 0 ]; then
+    echo -e "${GREEN}All $skill_count skill(s) have valid frontmatter.${NC}"
+    exit 0
+else
+    echo -e "${RED}$fail_count skill(s) have invalid frontmatter — needs regeneration. Preserve skill bodies, replace only the frontmatter.${NC}"
+    exit 1
+fi

skills/setup/SKILL.md

@@ -1288,6 +1288,9 @@ For each skill:
 2. Apply vocabulary transformation — rename and update ALL internal references using the vocabulary mapping from `ops/derivation.md`
 3. Adjust skill metadata (set `context: fork` for fresh context per invocation)
 4. Write the transformed SKILL.md to the user's skills directory
+5. Validate generated skill frontmatter from the generated vault root:
+   - Single skill: `${CLAUDE_PLUGIN_ROOT}/reference/validate-setup.sh [skill-name]`
+   - If validation fails: re-read the source template's frontmatter from `${CLAUDE_PLUGIN_ROOT}/skill-sources/[name]/SKILL.md`, vocabulary-transform the field values, and replace only the frontmatter section of the generated skill. Do NOT regenerate the skill body.
 
 **For Claude Code:** Write to `.claude/skills/[domain-skill-name]/SKILL.md`
 
@@ -1539,6 +1542,12 @@ Run all 15 primitive checks against the generated system. Use `${CLAUDE_PLUGIN_R
 
 Report results: pass/fail per primitive with specific failures listed.
 
+### Skill Frontmatter Validation
+
+Run `${CLAUDE_PLUGIN_ROOT}/reference/validate-setup.sh` from the generated vault root (whole-vault mode). This catches any skill frontmatter that was mangled during generation.
+
+If any skill fails: re-read that skill's source template frontmatter from `${CLAUDE_PLUGIN_ROOT}/skill-sources/[name]/SKILL.md`, vocabulary-transform the field values, and replace only the frontmatter of the generated skill. Preserve the skill body — only the frontmatter needs regeneration.
+
 ### Pipeline Smoke Test
 
 After kernel validation, run a functional test: