[Schema Consistency] 🔍 Schema Consistency Check - 2025-10-28 - 8 Issues Found (2 Critical)

[github-actions[bot]]

🔍 Schema Consistency Check - 2025-10-28

Summary

• Inconsistencies Found: 8 issues
• Categories Analyzed: Schema, Parser, Documentation, Workflows
• Strategy Used: Hybrid Strategy Remove ai-inference, opencode, genaiscript agentic engines for now #10 (Recent Changes + Enhanced Enum Validation)
• New Strategy: YES (combines git history + enum validation + Task agent exploration)
• Workflows Analyzed: 86 files
• Documentation Files: 14 reference docs

---

Critical Issues 🚨

1. ⚠️ Permission Level Unsafe Casting (HIGH PRIORITY)

Location: pkg/workflow/permissions.go:322 and pkg/workflow/permissions.go:337

Issue: Permission level values are directly cast to PermissionLevel type without validation:

// Line 322
perms.Set(scope, PermissionLevel(value))  // UNSAFE CAST

// Line 337  
permsMap[scope] = PermissionLevel(value)  // UNSAFE CAST

Schema defines: ["read", "write", "none"]
Problem: Invalid values like "admin" or "execute" would be accepted without error

Impact: 🔴 Security issue - invalid permission levels could pass through validation

Recommendation: Add explicit validation:

if value != "read" && value != "write" && value != "none" {
    return fmt.Errorf("invalid permission level '%s': must be read, write, or none", value)
}

---

2. ⚠️ Roles Validation Completely Missing (HIGH PRIORITY)

Location: pkg/workflow/role_checks.go:54-80 in extractRoles()

Issue: No validation of role values. Any arbitrary string is accepted:

// Lines 58-63: Single string accepted without validation
if s, ok := rolesRaw.(string); ok {
    return []string{s}, nil  // NO VALIDATION
}

// Lines 64-71: Array items accepted without validation  
for _, item := range arr {
    if s, ok := item.(string); ok {
        result = append(result, s)  // NO VALIDATION
    }
}

Schema defines: ["admin", "maintainer", "maintain", "write", "triage", "all"]
Problem: Invalid roles like "superuser" or "root" would be silently accepted

Impact: 🔴 Security issue - unauthorized access control could be bypassed

Recommendation: Add role validation:

validRoles := map[string]bool{
    "admin": true, "maintainer": true, "maintain": true,
    "write": true, "triage": true, "all": true,
}

if !validRoles[role] {
    return nil, fmt.Errorf("invalid role '%s': must be one of admin, maintainer, maintain, write, triage, or all", role)
}

---

Documentation Gaps 📝

3. Three Schema Fields Missing from Documentation

Fields missing documentation in frontmatter.md:

3a. githubActionsStep - DEAD CODE

• Found in schema: ✅
• Found in parser: ❌ (0 occurrences)
• Found in compiler: ❌ (0 occurrences)
• Found in workflows: ❌ (0 occurrences)
• Recommendation: 🗑️ Remove from schema entirely

3b. mcp-servers - ACTIVE but UNDOCUMENTED

• Found in schema: ✅
• Found in code: ✅ (used by parser and compiler)
• Found in workflows: ❌ (not used in production workflows)
• Documented in: docs/src/content/docs/guides/mcps.md (partial)
• Recommendation: 📝 Add to frontmatter.md reference documentation

3c. runtimes - ACTIVE but UNDOCUMENTED

• Found in schema: ✅
• Found in compiler: ✅ (19 occurrences)
• Implementation: pkg/workflow/runtime_setup.go:142 - DetectRuntimeRequirements()
• Purpose: Auto-detects and sets up required runtimes (Node, Python, etc.)
• Found in workflows: ❌ (0 explicit uses - auto-detected)
• Recommendation: 📝 Add to frontmatter.md with explanation of auto-detection

---

4. ✅ Documentation Structure Excellent

Positive Finding: Schema fields are well-distributed across documentation:

• Main reference: frontmatter.md - 27 out of 31 fields documented (87% coverage)
• Specialized docs:
  • ✅ tools.md - tools field
  • ✅ safe-outputs.md - safe-outputs field
  • ✅ network.md - network field
  • ✅ imports.md - imports field
  • ✅ engines.md - engine field
  • ✅ cache-memory.md - cache field

---

Schema Improvements Needed 🔧

5. Reaction Enum Missing from Schema

Issue: Reaction validation exists in code but NOT in schema

Code location: pkg/workflow/reactions.go:3-28

Valid reactions defined in code:

var validReactions = map[string]bool{
    "+1": true, "-1": true, "laugh": true, "confused": true,
    "heart": true, "hooray": true, "rocket": true, "eyes": true, "none": true,
}

Validation: ✅ Implemented correctly in compiler.go:1048-1050

Schema location: Should be in properties.on.oneOf[1].properties["stop-after"].properties.reaction.enum

Current schema: ❌ No enum defined (accepts any string)

Recommendation: Add enum to schema:

"reaction": {
  "type": "string",
  "enum": ["+1", "-1", "laugh", "confused", "heart", "hooray", "rocket", "eyes", "none"],
  "description": "GitHub reaction emoji to add when stop-after triggers"
}

---

6. Models Permission Special Case Not Obvious

Issue: models permission scope only supports ["read", "none"] (not "write")

Schema location: pkg/parser/schemas/main_workflow_schema.json:1106-1107

"models": {
  "type": "string",
  "enum": ["read", "none"],
  "description": "Permission for GitHub Models"
}

Problem: Not clearly documented that this is an exception
Other permissions: All support ["read", "write", "none"]

Recommendation: Enhance description:

"description": "Permission for GitHub Models. Note: write permission is not available for models scope, only read and none."

---

Parser Updates Required 🔍

7. ✅ Permission Scope Validation Works Correctly

Positive Finding: Permission scope names ARE validated

Location: pkg/workflow/permissions.go:345-380 in convertStringToPermissionScope()

Valid scopes:

• actions, attestations, checks, contents, deployments, discussions, id-token, issues
• models, packages, pages, pull-requests, repository-projects, security-events, statuses

Behavior: Returns empty string "" for invalid scopes, which are then silently ignored via if scope != "" { check

Issue: Silent failure - no error message for invalid scope

Recommendation: Consider logging warning or returning error for invalid scopes in strict mode

---

Workflow Violations 🎯

8. ✅ No Workflow Violations Found

Positive Finding: All 86 production workflows follow the schema correctly

Workflows analyzed:

• .github/workflows/*.md - 86 files
• ✅ No usage of undefined fields
• ✅ No invalid enum values detected
• ✅ No missing required fields

Sample workflows checked:

• artifacts-summary.md, audit-workflows.md, blog-auditor.md, brave.md, changeset-generator.firewall.md, ci-doctor.md, and 80 more...

---

Recommendations 📋

Immediate Actions (High Priority) 🚨

1. Add PermissionLevel validation in permissions.go:322 and permissions.go:337

   • Validate against: "read", "write", "none"
   • Return error for invalid values
   • Impact: Fixes critical security issue

2. Add Role validation in role_checks.go:54-80

   • Validate against: "admin", "maintainer", "maintain", "write", "triage", "all"
   • Return error for invalid roles
   • Impact: Fixes critical security issue

3. Remove githubActionsStep from schema

   • Confirmed dead code with 0 uses
   • Clean up schema definitions
   • Impact: Reduces schema complexity

---

Documentation Updates (Medium Priority) 📝

4. Document mcp-servers field in frontmatter.md

   • Link to mcps.md guide
   • Provide minimal example
   • Impact: Improves developer experience

5. Document runtimes field in frontmatter.md

   • Explain auto-detection behavior
   • Show when manual override is needed
   • Impact: Reduces confusion

6. Add reaction enum to schema

   • Define 9 valid reaction values
   • Match validation in reactions.go
   • Impact: Improves schema completeness

---

Schema Improvements (Low Priority) 🔧

7. Enhance models permission description

   • Clarify write permission is not available
   • Explain why it's different from other scopes
   • Impact: Prevents user confusion

8. Consider stricter scope validation

   • Log warnings for invalid scopes
   • In strict mode, return errors instead of silent ignore
   • Impact: Better debugging experience

---

Strategy Performance 📊

• Strategy Used: Hybrid Strategy Remove ai-inference, opencode, genaiscript agentic engines for now #10 (Recent Changes + Enhanced Enum Validation)
• Findings: 8 issues (2 critical security issues, 3 documentation gaps, 3 improvements)
• Effectiveness: ⭐ VERY HIGH
• Should Reuse: ✅ YES
• Innovation: Combined git analysis + Task/Explore agent + systematic checks

What Made This Strategy Successful

1. Multi-layered approach: Checked schema → parser → compiler → docs → workflows
2. Enum validation focus: Deep dive into validation logic found missing checks
3. Task agent usage: Found exact file:line code locations automatically
4. Systematic documentation check: Field-by-field coverage analysis
5. Dead code detection: Confirmed githubActionsStep has zero uses
6. Positive findings: Documented what works well, not just problems
7. Actionable recommendations: Provided concrete code fixes with snippets

---

Code Locations Reference 📍

For developers fixing these issues:

unsafe_permission_cast:      permissions.go:322, 337
missing_role_validation:     role_checks.go:54-80
permission_scope_validation: permissions.go:345-380 (WORKS)
engine_validation:           engine.go:260-274 (WORKS)
reaction_validation:         reactions.go:3-28, compiler.go:1048-1050 (WORKS)
runtime_detection:           runtime_setup.go:142 (UNDOCUMENTED)

---

Next Steps ✅

• Analyze schema field enumeration
• Check enum validation implementation
• Verify documentation coverage
• Sample workflow validation
• Create comprehensive report
• Fix critical validation issues (permissions.go, role_checks.go)
• Update documentation (add mcp-servers, runtimes; remove githubActionsStep)
• Enhance schema (add reaction enum, improve descriptions)
• Create follow-up issues for each recommendation

---

Generated by the Schema Consistency Checker using Strategy #10 on 2025-10-28

Full detailed report: Available in cache memory at /tmp/gh-aw/cache-memory/consistency-report-2025-10-28.md

AI generated by Schema Consistency Checker

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 month ago.