[Schema Consistency] 🔍 Schema Consistency Check - 2025-11-17: 83% Consistency Rate (CLI-Only Feature Gap)

[github-actions[bot]]

🔍 Schema Consistency Check - 2025-11-17

This analysis traced 6 key features end-to-end through schema → parser → compiler → documentation → workflows to identify implementation drift and consistency gaps. Results show 83% consistency with 1 schema documentation gap for a CLI-only feature.

Summary

• Inconsistencies Found: 1 (CLI-only feature documentation gap)
• Categories Analyzed: Schema, Parser, Compiler, Documentation, Workflows
• Strategy Used: Cross-File Feature Consistency & Implementation Gap Detection (Strategy 009)
• New Strategy: No (reused proven strategy from 2025-10-27)
• Features Traced: 6 (strict-mode, stop-after, github-token, post-steps, web-search, manual-approval)
• Consistency Rate: 83% (5/6 features fully consistent)

Full Analysis Report

Features Analyzed

✅ Fully Consistent Features (5/6)

1. stop-after

End-to-end validation: Schema ✓ Parser ✓ Compiler ✓ Docs ✓ Workflows ✓

• Schema Location: on.oneOf[1].properties["stop-after"]
• Implementation: pkg/workflow/compiler.go:1173 (processStopAfterConfiguration)
• Documentation: docs/src/content/docs/reference/triggers.md:63-78
• Usage: 2 workflows (ci-doctor.md, daily-team-status.md)
• Validation: Supports absolute dates and relative time deltas (+25h, +3d, etc.)

2. github-token

End-to-end validation: Schema ✓ Parser ✓ Compiler ✓ Docs ✓ Workflows ✓

• Schema Location: Top-level property with $ref: "#/$defs/github_token"
• Implementation:
  • Parser: pkg/parser/github.go:10 (GetGitHubToken)
  • Compiler: pkg/workflow/compiler.go:1006, extensive token precedence logic
• Documentation:
  • docs/src/content/docs/reference/frontmatter.md:140-165 (comprehensive)
  • docs/src/content/docs/reference/safe-outputs.md:245-260 (token precedence)
• Usage: 2 workflows with proper secret expressions
• Token Precedence: Individual safe-output > safe-outputs global > top-level > default

3. post-steps

End-to-end validation: Schema ✓ Compiler ✓ Docs ✓ Workflows ✓

• Schema Location: Top-level property with oneOf (object | array)
• Implementation: pkg/workflow/compiler.go:1099-1115 (extractTopLevelYAMLSection with action pinning)
• Documentation: docs/src/content/docs/reference/frontmatter.md:295-305
• Usage: 1 workflow (test-post-steps.md)
• Features: Supports action pinning, runs after AI execution

4. web-search

End-to-end validation: Schema ✓ Parser ✓ Compiler ✓ Docs ✓ Workflows ✓

• Schema Location: tools.properties["web-search"]
• Implementation:
  • Parser: pkg/parser/frontmatter.go:982 (neutral tool merging)
  • Compiler: pkg/workflow/agent_validation.go:161-173 (validateWebSearchSupport)
• Documentation:
  • docs/src/content/docs/reference/tools.md:118-122
  • docs/src/content/docs/reference/engines.md:89-92 (engine compatibility note)
• Usage: 2 workflows (craft.md, ci-doctor.md)
• Engine Support: Claude ✓ Copilot ✗ (requires MCP server)

5. manual-approval

End-to-end validation: Schema ✓ Compiler ✓ Docs ✓ Workflows ✓

• Schema Location: on.oneOf[1].properties["manual-approval"]
• Implementation:
  • Compiler: pkg/workflow/manual_approval.go:11-50 (extractManualApprovalFromOn)
  • Job generation: pkg/workflow/compiler_jobs.go:651-654 (environment setting)
• Documentation:
  • docs/src/content/docs/reference/triggers.md:80-92 (comprehensive)
  • docs/src/content/docs/guides/security.md:95 (security context)
  • docs/src/content/docs/troubleshooting/errors.md:457-467 (error handling)
• Usage: 1 workflow (test-manual-approval.md)
• Integration: Sets GitHub Actions environment for approval gates

❌ Schema Documentation Gap (1/6)

6. strict-mode

Finding: CLI-only feature with extensive implementation but no schema representation

• Schema Location: NOT in main_workflow_schema.json
• Parser: Not handled (CLI-only via compiler.SetStrictMode())
• Compiler Implementation (Extensive):
  • pkg/workflow/compiler.go:111 (SetStrictMode)
  • pkg/workflow/compiler.go:684 (validateStrictMode)
  • pkg/workflow/action_pins.go:110,121 (action pinning enforcement)
  • pkg/workflow/strict_mode_test.go (comprehensive test coverage)
  • 15+ test files reference strict mode functionality
• Documentation:
  • docs/src/content/docs/guides/security.md:124-126 (mentioned)
  • NOT in docs/src/content/docs/reference/frontmatter.md
• Usage: 0 workflows (CLI-only flag, not frontmatter-configurable)
• Enforcement Areas:
  • Action pinning (requires SHA pins)
  • Network permissions (requires explicit configuration)
  • Write permissions (requires safe-outputs)
  • Bash tool restrictions

Impact: Users and IDE tooling have no schema-level indication that strict mode exists as a CLI feature. Schema should include a $comment field explaining this is a CLI-only option.

Implementation Quality Observations

Positive Patterns

1. Comprehensive Documentation: All 5 frontmatter-configurable features have excellent documentation across reference docs, guides, and error messages

2. Consistent Implementation: Schema definitions accurately reflect compiler implementation for all frontmatter-configurable features

3. Engine Compatibility Validation: web-search includes runtime validation with helpful error messages for unsupported engines

4. Token Precedence: github-token implements clear precedence hierarchy (individual > global > top-level > default) consistently across safe-outputs

5. Error Message Documentation: manual-approval and other features have dedicated error documentation in troubleshooting/errors.md

Architecture Insights

1. Two-Phase Processing: Features in on: section (stop-after, manual-approval) are extracted separately then "compiled away" or transformed into GitHub Actions constructs

2. Action Pinning Integration: post-steps properly integrates with action pinning system

3. CLI vs Frontmatter Separation: strict-mode demonstrates clear separation between CLI-only features and frontmatter-configurable features, but lacks schema documentation of this distinction

Detailed Findings

Critical Issues

None - The single gap is a documentation enhancement rather than a functional bug.

Documentation Gaps

1. CLI-Only Features: Schema should document CLI-only features like strict-mode with $comment fields explaining they cannot be set in frontmatter but affect compilation behavior

Schema Improvements Needed

1. Add schema metadata for CLI-only features:

{
  "$comment": "CLI-ONLY: strict-mode is set via gh-aw CLI flag --strict, not in frontmatter",
  "strict-mode": {
    "type": "null",
    "description": "This field is not configurable in frontmatter. Strict mode is enabled via CLI flag: gh-aw --strict. Enforces action pinning, network configuration, and safe-outputs usage."
  }
}

Parser Updates Required

None - All frontmatter-configurable features are properly parsed.

Workflow Violations

None - All real workflow usage complies with schema definitions.

Recommendations

Immediate Actions

1. Document CLI-Only Features in Schema: Add $comment fields or dedicated schema section explaining features like strict-mode that affect compilation but aren't frontmatter-configurable

Long-term Improvements

2. Schema Metadata Completeness: Consider adding all compiler flags and options to schema documentation even if not frontmatter-configurable, to provide complete reference

3. Maintain Current Quality: Continue the excellent pattern of comprehensive documentation across reference docs, guides, and error messages

Strategy Performance

• Strategy Used: Cross-File Feature Consistency & Implementation Gap Detection (Strategy 009)
• Findings: 1 documentation gap (strict-mode schema documentation)
• Effectiveness: Very High - Successfully identified CLI-only feature documentation gap through end-to-end tracing
• Should Reuse: Yes - Excellent for validating feature consistency across codebase boundaries
• Last Used: 2025-10-27 (21 days ago)
• Reuse Recommendation: Use every 3-4 weeks to validate new features and detect implementation drift

Strategy Insights

This strategy excels at:

• Finding features implemented in code but missing from schema
• Validating documentation accuracy against implementation
• Tracing feature integration across multiple subsystems
• Identifying architectural patterns and design decisions

Complements other strategies by:

• Providing end-to-end validation (vs point-in-time field checks)
• Revealing intentional design choices (CLI-only features)
• Validating documentation completeness
• Confirming real-world usage patterns

Next Steps

• Traced 6 key features end-to-end
• Identified 1 schema documentation gap
• Validated 5 features as fully consistent
• Documented CLI-only feature pattern
• Consider adding CLI-only features to schema with $comment documentation
• Review other CLI-only features for schema documentation needs

---

Analysis Date: 2025-11-17
Strategy: 009 (Cross-File Feature Consistency)
Features Analyzed: strict-mode, stop-after, github-token, post-steps, web-search, manual-approval
Consistency Rate: 83% (5/6)

AI generated by Schema Consistency Checker

[pelikhan]

/plan

[github-actions[bot]]

✅ Agentic Plan Command completed successfully.

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.