[Schema Consistency] 🔍 Schema Consistency Check - Oct 26, 2025: Schema Completeness Analysis

[github-actions[bot]]

🔍 Schema Consistency Check - October 26, 2025

Summary

• Inconsistencies Found: 5 major categories
• Categories Analyzed: Schema metadata quality, nested field documentation, parser implementation, documentation examples
• Strategy Used: Schema Completeness & Internal Reference Analysis (Strategy Add workflow: githubnext/agentics/weekly-research #8)
• New Strategy: YES - completely different from previous 7 strategies

Executive Summary

This analysis took a radically different approach by examining schema metadata quality rather than schema-vs-code consistency. Key discovery: while the schemas validate correctly, they lack metadata (descriptions, examples, defaults) that would make them useful as documentation tools for IDEs and developers.

Key Metrics

• Main Schema: 612 objects, 74% description coverage, grade B
• Included Schema: 110 objects, 75.5% coverage, grade C+
• MCP Schema: 30 objects, 50% coverage, grade D
• Overall Grade: C+

---

Critical Issues

1. Schema Examples Field Almost Completely Unused ⚠️

Severity: HIGH
Impact: IDE autocomplete and inline hints unavailable

Schema	Total Properties	With Examples	%
Main	31	1	3%
Included	~15	0	0%
MCP	~8	0	0%

Meanwhile: Documentation contains 119 YAML examples across 6 files!

Why This Matters:

• JSON Schema supports examples array for IDE integration
• VSCode, IntelliJ can show examples during autocomplete
• Users copy-paste from docs instead of getting inline hints
• Schema could be single source of truth for examples

Recommendation: Extract examples from documentation and add to schema examples fields.

---

2. Nested Property Description Coverage Gaps ⚠️

Severity: MEDIUM-HIGH
Impact: Users drilling into complex fields get no guidance

159 nested fields in main schema lack descriptions, including:

Area	Missing Descriptions	Impact
on triggers	54 fields	Users configuring triggers
safe-outputs	13 fields (65%!)	Security-sensitive features
jobs	11 fields	GitHub Actions job config
permissions	7 fields	Security scopes
tools items	6 fields	Tool configuration

Example: safe-outputs has 20 nested properties, but 13 lack descriptions:

safe-outputs:
  create-issue: ??? # no description in schema
  add-comment: ??? # no description
  create-pull-request: ??? # no description
  # ... 10 more without descriptions

Recommendation: Prioritize safe-outputs and on triggers for description additions.

---

3. MCP Schema Only 50% Description Coverage ⚠️

Severity: HIGH
Impact: Most complex feature is least documented in schema

The MCP schema has the worst quality of all three schemas:

• 15 out of 30 objects (50%) lack descriptions
• Critical fields undocumented: args.items, entrypointArgs.items, network.proxy-args.items
• No examples despite MCP being complex external process execution

Why This Matters:

• MCP servers execute external processes - security-sensitive
• Users need inline guidance for configuration
• Docs exist but schema doesn't leverage them

Recommendation: Bring MCP schema to 80%+ coverage, add examples for common patterns.

---

4. Top-Level Fields Missing Descriptions ⚠️

Severity: MEDIUM
Impact: User-facing fields lack basic documentation

Missing descriptions:

• runs-on - standard GitHub Actions field
• concurrency - important for parallel execution control
• engine_config (in $defs) - custom engine configuration

Recommendation: Add descriptions to all top-level fields - quick win for schema quality.

---

5. Safe-Outputs Documentation Gap ⚠️

Severity: MEDIUM-HIGH
Impact: Security features lack inline guidance

13 out of 20 safe-outputs nested properties (65%) lack descriptions:

{
  "field": "safe-outputs",
  "nested_props": 20,
  "props_without_desc": [
    "create-issue", "create-agent-task", "create-discussion",
    "add-comment", "create-pull-request", 
    "create-pull-request-review-comment",
    "create-code-scanning-alert", "add-labels", 
    "update-issue", "push-to-pull-request-branch",
    "missing-tool", "upload-assets", "threat-detection"
  ]
}

Why This Matters: These are security-sensitive output mechanisms. Users need inline schema guidance.

---

Documentation Gaps

Schema vs Documentation Examples

Documentation has excellent examples:

• safe-outputs.md: 46 examples
• frontmatter.md: 21 examples
• tools.md: 18 examples
• engines.md: 15 examples
• triggers.md: 12 examples
• Total: 119 YAML examples

But schemas have almost none:

• Main schema: 1 field with examples
• Included schema: 0 fields with examples
• MCP schema: 0 fields with examples

Gap: Documentation and schema are disconnected. Schema should be single source of truth.

---

Schema Quality Improvements Needed

Immediate Actions (HIGH Priority)

1. Add examples to all top-level schema properties

   • Extract from existing documentation
   • Use JSON Schema examples array
   • Target: 100% of top-level fields

2. Fix safe-outputs nested descriptions

   • Add descriptions to all 13 missing properties
   • Priority: security-sensitive features need docs
   • Target: 100% coverage for safe-outputs

3. Improve MCP schema coverage (50% → 80%+)

   • Add descriptions to all items fields
   • Document network configuration options
   • Add examples for common MCP patterns

4. Add descriptions to: runs-on, concurrency, engine_config

   • Quick wins - top-level fields
   • Users expect schema documentation

Medium-Term Actions (MEDIUM Priority)

5. Add descriptions to nested on trigger properties

   • 54 fields missing descriptions
   • Users configuring triggers need guidance
   • Consider: use schema description or title fields

6. Add default values where applicable

   • Currently only 5 fields have defaults defined
   • Many fields have implicit defaults in code
   • Make defaults explicit in schema

7. Consider x-examples or x-usage schema extensions

   • For fields needing more context than examples allows
   • Could include usage notes, common patterns, warnings

Long-Term Actions (LOW Priority)

8. Create examples gallery in documentation

   • Link documentation examples to schema examples
   • Show common patterns for each field
   • Consider: JSON Schema $ref to shared examples

9. Schema quality CI check

   • Fail CI if schema coverage drops below thresholds
   • Enforce: all top-level fields must have descriptions
   • Enforce: all security-sensitive fields must have examples

---

Parser Implementation Analysis

✅ Positive Findings

The parser implementation is exemplary:

1. Comprehensive validation using github.com/santhosh-tekuri/jsonschema/v6
2. Sophisticated error formatting:
   • Precise file/line/column error locations
   • Schema-based "did you mean" suggestions
   • Rewrites "additional properties not allowed" to user-friendly messages
3. Performance optimizations:
   • Schemas embedded at compile time (//go:embed)
   • Cached compilation with sync.Once
4. Excellent error messages - see pkg/parser/schema.go:
   • Finds closest field matches using similarity scoring
   • Generates example JSON from schema
   • Provides context lines around errors

Code Reference: pkg/parser/schema.go:56-871

---

Recommendations

Immediate (This Sprint)

• Add descriptions to runs-on, concurrency top-level fields
• Add description to engine_config $def
• Add descriptions to all 13 safe-outputs nested properties
• Create tracking issue for schema examples extraction project

Short-Term (Next Sprint)

• Extract 119 documentation examples → schema examples fields
• Improve MCP schema coverage to 80%+
• Add nested descriptions to top 20 most-used fields
• Add default values to fields with implicit defaults

Medium-Term (Next Quarter)

• Add descriptions to all 54 on trigger nested fields
• Implement schema quality CI checks
• Create schema-driven examples gallery
• Consider JSON Schema extensions for advanced use cases

---

Strategy Performance

• Strategy Used: Schema Completeness & Internal Reference Analysis (Strategy Add workflow: githubnext/agentics/weekly-research #8)
• Findings: 5 major categories, 159 nested fields affected
• Effectiveness: VERY HIGH
• Should Reuse: YES - every 4-5 runs
• Novelty: Completely different dimension from previous 7 strategies

What Makes This Strategy Different

Previous strategies focused on:

• Schema vs code consistency (strategies 1-3)
• Schema vs documentation field coverage (strategies 2, 7)
• Cross-schema consistency (strategy 4)
• Conditional logic and security (strategies 5-6)

This strategy focuses on: Schema as a documentation tool - metadata quality that enables IDE integration and developer experience.

Complementary Strategies

This strategy complements:

• Strategy 2: Field enumeration (found fields, this finds field docs)
• Strategy 6: Security audit (found validation gaps, this finds doc gaps)
• Strategy 7: Example validation (validated examples, this finds missing ones)

---

Schema Quality Grade Card

Schema	Objects	Descriptions	Examples	Patterns	Enums	Grade
Main Workflow	612	74% ✅	1 ❌	4 ⚠️	52 ✅	B
Included File	110	75.5% ✅	0 ❌	2 ⚠️	3 ⚠️	C+
MCP Config	30	50% ❌	0 ❌	2 ⚠️	1 ⚠️	D

Overall Schema Quality: C+

Strengths:

• ✅ Good top-level description coverage (90%+)
• ✅ Excellent parser validation implementation
• ✅ Comprehensive enum definitions

Weaknesses:

• ❌ Almost no schema examples (3% coverage)
• ❌ Poor nested field descriptions (40-60% coverage)
• ❌ MCP schema needs significant improvement

Target Grades:

• Main Workflow: B → A (add examples, improve nested docs)
• Included File: C+ → B (add examples, improve nested docs)
• MCP Config: D → B+ (50% → 80% coverage, add examples)

---

Next Steps

1. Discuss priorities - which schema improvements are most valuable?
2. Create tracking issues for HIGH priority improvements
3. Assign owners for schema documentation improvements
4. Schedule follow-up - re-run this analysis after improvements

Strategy to use next: Recommend Strategy #2 or #6 (proven high-effectiveness strategies that complement this one)

---

Appendix: Analysis Methodology

Tools Used

• jq for JSON schema parsing and analysis
• Python for recursive schema traversal
• Manual analysis of parser implementation (pkg/parser/schema.go)
• Documentation examples counting via regex

Metrics Calculated

• Description coverage: % of schema objects with description field at ALL nesting levels
• Examples coverage: Count of fields with examples array
• Nested field analysis: Recursive traversal to find undocumented nested properties
• Cross-schema comparison: Consistency check across 3 schemas

Files Analyzed

• pkg/parser/schemas/main_workflow_schema.json (122KB, 612 objects)
• pkg/parser/schemas/included_file_schema.json (19KB, 110 objects)
• pkg/parser/schemas/mcp_config_schema.json (4.5KB, 30 objects)
• pkg/parser/schema.go (871 lines)
• docs/src/content/docs/reference/*.md (all 9 reference docs)

Strategy Cache

This analysis has been recorded in /tmp/gh-aw/cache-memory/strategies.json as Strategy #8.

---

Generated by: Schema Consistency Checker (Automated Workflow)
Run ID: 18823850355
Date: October 26, 2025

AI generated by Schema Consistency Checker

[pelikhan]

/plan focus on adding examples to the schema. Mine from existing workflows in the repo. Split in chunks of max 5 elements.

[github-actions[bot]]

✅ Agentic Plan Command completed successfully.

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 month ago.