🎯 Repository Quality Improvement Report - Workflow Validation Error UX

[github-actions[bot]]

Analysis Date: 2026-01-01
Focus Area: Workflow Validation Error User Experience
Strategy Type: Custom
Custom Area: Yes - This focus area is unique to gh-aw's core functionality as a workflow compilation tool

Executive Summary

With 178 active workflow files and 58+ validation-related Go files, validation error messages are the primary interface between users and the workflow compilation system. Analysis reveals that while validation coverage is comprehensive (35 test files, 22 validation functions), the error user experience has significant improvement opportunities.

Current findings show that only 7 validation errors include documentation URLs, console formatting is inconsistently used (28 instances across validation files), and error messages lack contextual information like file names, line numbers, or actionable "next steps." With error messages averaging 200-400 characters and some exceeding 400 characters, there's substantial room to improve clarity, consistency, and user productivity when compilation fails.

This analysis identifies 5 high-impact improvements to transform validation errors from technical obstacles into helpful guidance that accelerates workflow development.

Full Analysis Report

Focus Area: Workflow Validation Error User Experience

Current State Assessment

The validation system is architecturally sound with clear separation of concerns across domain-specific files. However, the user-facing error experience reveals several consistency and usability gaps.

Metrics Collected:

Metric	Value	Status
Total validation files	58	✅ Comprehensive coverage
Validation functions	22	✅ Good organization
Error messages with URLs	7	⚠️ Only ~12% include documentation
Console formatting usage	28 instances	⚠️ Inconsistent application
Error messages with context (line/col)	21 mentions	⚠️ Limited positional info
Average error message length	200-400 chars	⚠️ Some exceed readability threshold
Validation test coverage	35 test files	✅ Excellent test coverage
fmt.Errorf usage	1,278 instances	❌ Vast majority lack console formatting
Validation errors with recovery suggestions	~5 instances	❌ Minimal actionable guidance

Findings

Strengths

• Comprehensive validation coverage: 58 validation files covering all aspects of workflow compilation
• Clear architectural separation: Domain-specific validation (strict mode, MCP, schema, runtime, etc.)
• Excellent test coverage: 35 validation test files ensure reliability
• Strong documentation: docs/src/content/docs/troubleshooting/errors.md provides 546 lines of error reference
• URL inclusion in critical errors: Strict mode and security errors include helpful documentation links
• Typo detection: Schema validation includes fuzzy matching for common field name mistakes

Areas for Improvement

• Inconsistent console formatting: Only 74 instances use console.FormatErrorMessage() vs 1,278 raw fmt.Errorf() calls
• Limited contextual information: Most errors lack file paths, line numbers, or YAML locations
• Minimal recovery guidance: Only ~5 validation errors suggest concrete next steps or alternatives
• No error categorization: Validation errors aren't marked by severity (critical, warning, info)
• Long error messages: Some exceed 400 characters, reducing readability in terminal output
• No progressive disclosure: Complex validation errors could benefit from summary + detailed explanation pattern
• Missing error codes: No standardized error codes for programmatic error handling or documentation cross-references

Detailed Analysis

Error Message Structure Patterns

Current validation errors follow several inconsistent patterns:

Pattern 1: Basic error (no context)

return fmt.Errorf("job name cannot be empty")

Issue: No file context, no recovery suggestion

Pattern 2: Error with explanation (good)

return fmt.Errorf("strict mode: write permission '%s: write' is not allowed for security reasons. Use 'safe-outputs.create-issue', 'safe-outputs.create-pull-request', 'safe-outputs.add-comment', or 'safe-outputs.update-issue' to perform write operations safely. See: (redacted), scope)

Strengths: Clear explanation, alternatives, documentation URL
Issue: Very long (422 chars), could be split into summary + details

Pattern 3: Error with console formatting

fmt.Fprintln(os.Stderr, console.FormatWarningMessage(fmt.Sprintf("Engine '%s' does not support the web-search tool...", engine.GetID())))

Strengths: Visual distinction via console colors
Issue: Only used in 74 places vs 1,278 raw errors

User Journey Pain Points

1. Compilation fails with cryptic error

   • User receives: "invalid on: section format"
   • No indication of which workflow file, which line, what's invalid
   • No suggestion of valid formats or examples

2. Multiple validation errors

   • Currently only first error is shown
   • User must fix and recompile repeatedly (slow iteration)
   • No indication of total error count or remaining issues

3. Error severity unclear

   • Some validations warn but continue (web-search support)
   • Others fail compilation (schema errors)
   • No visual distinction between critical and advisory errors

4. Documentation lookup required

   • Only 12% of errors link to documentation
   • User must manually search docs or error reference
   • Context switching reduces productivity

Comparison to Industry Best Practices

TypeScript Compiler Errors (Best in Class)

error TS2322: Type 'string' is not assignable to type 'number'.

  workflow.ts:42:5

  42     timeout: "30"
              ~~~

  Expected type 'number', found 'string'.
  Did you mean: timeout: 30

Rust Compiler Errors (Excellent UX)

error[E0308]: mismatched types
 --> src/main.rs:4:18
  |
4 |     let x: i32 = "hello";
  |            ---   ^^^^^^^ expected `i32`, found `&str`
  |            |
  |            expected due to this
  |
help: try using a conversion method
  |
4 |     let x: i32 = "hello".parse()?;
  |                         +++++++

gh-aw Current State

failed to parse frontmatter: yaml: unmarshal errors:
  line 3: cannot unmarshal !!str `30` into int

Gap Analysis: gh-aw errors lack:

• File context
• Line/column numbers
• Code snippets showing the problem
• Suggestions or auto-fix hints
• Error codes for quick lookup

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.

Task 1: Standardize Console Formatting Across All Validation Errors

Priority: High
Estimated Effort: Large
Focus Area: Consistency & Visual Clarity

Description:
Currently, only 74 validation errors use console.FormatErrorMessage() while 1,278 errors use raw fmt.Errorf(). This creates visual inconsistency and makes critical errors harder to distinguish from warnings or info messages. Standardize all validation errors to use console formatting for consistent, color-coded error presentation.

Acceptance Criteria:

• All validation errors in pkg/workflow/*validation*.go use console.FormatErrorMessage()
• Distinction between error/warning/info levels using appropriate console formatting functions
• Validation helper functions return structured errors that can be formatted at display time
• Unit tests verify error formatting consistency
• Documentation in specs/validation-architecture.md updated with formatting guidelines

Code Region: pkg/workflow/*validation*.go (all validation files)

Review all validation files in pkg/workflow/ that match *validation*.go pattern and standardize error formatting:

1. Replace raw `fmt.Errorf()` calls with structured errors that can be formatted
2. Use `console.FormatErrorMessage()` for compilation-blocking errors
3. Use `console.FormatWarningMessage()` for advisory warnings that don't block compilation
4. Ensure error messages maintain their helpful context (URLs, suggestions, etc.)
5. Create helper function `formatValidationError(category, message, details)` for consistency
6. Update validation tests to verify console formatting is applied

Priority files to start with:
- pkg/workflow/validation.go (main validation orchestrator)
- pkg/workflow/strict_mode_validation.go (security-critical errors)
- pkg/workflow/schema_validation.go (common user-facing errors)

---

Task 2: Add File Context and Line Numbers to Validation Errors

Priority: High
Estimated Effort: Medium
Focus Area: Contextual Information

Description:
Validation errors currently lack file names and line numbers, making it difficult for users to locate problems in their workflow markdown files. Enhance the validation system to track and report the source location of errors, similar to TypeScript or Rust compiler errors.

Acceptance Criteria:

• Validation errors include workflow filename (e.g., workflow-name.md)
• YAML frontmatter errors include line numbers when possible
• Error format: [filename:line] Error message
• Parser tracks line numbers during frontmatter parsing
• Integration tests verify location information is present and accurate
• Location context preserved through validation chain

Code Region: pkg/parser/ and pkg/workflow/*validation*.go

Enhance validation error messages with source file context:

1. Modify pkg/parser/ to track line numbers during YAML frontmatter parsing
2. Create a ValidationContext struct that includes:
   - filename: string
   - lineNumber: int (optional)
   - fieldPath: string (e.g., "tools.github.toolsets[0]")
3. Thread ValidationContext through validation functions
4. Update error formatting to include: `[workflow-name.md:42] Error message`
5. For schema validation errors, map YAML path to line number when possible
6. Add example: `[my-workflow.md:15] strict mode: write permission 'contents: write' is not allowed`
7. Write integration tests that verify location information in common error scenarios

---

Task 3: Implement Multi-Error Collection and Reporting

Priority: Medium
Estimated Effort: Medium
Focus Area: User Productivity

Description:
The compilation process currently stops at the first validation error, requiring users to fix and recompile repeatedly. Implement error collection to report multiple validation errors in a single compilation pass, showing users all issues at once and enabling batch fixes.

Acceptance Criteria:

• Validation system collects all errors instead of returning on first error
• Errors are categorized by severity (critical, warning, info)
• Compilation fails only on critical errors; warnings are displayed but allow continuation
• Error summary shows count: Found 3 errors and 2 warnings
• Errors are grouped by category (schema, security, MCP, etc.)
• Flag --max-errors N to limit error collection (default: 10)
• Update compilation command to display all collected errors with clear visual separation

Code Region: pkg/workflow/validation.go, pkg/cli/compile_command.go

Implement multi-error collection during validation:

1. Create ErrorCollector type in pkg/workflow/validation.go:
   - Add(error, severity, context)
   - HasCriticalErrors() bool
   - GetErrors() []ValidationError
   - GetSummary() string
2. Modify validation functions to accept *ErrorCollector parameter
3. Change validation flow: continue collecting errors instead of early return
4. Update pkg/cli/compile_command.go to:
   - Display all errors grouped by severity
   - Show summary line: "Found X errors, Y warnings in workflow-name.md"
   - Exit with code 1 only if critical errors exist
5. Add --max-errors flag to limit collection (prevent excessive output)
6. Format output with clear visual separation:

❌ ERRORS (2):
[workflow.md:15] strict mode: write permission not allowed
[workflow.md:23] invalid MCP server configuration

⚠️ WARNINGS (1):
[workflow.md:8] engine does not support web-search

---

Task 4: Create Validation Error Code System and Enhanced Documentation

Priority: Medium
Estimated Effort: Large
Focus Area: Documentation & Error Codes

Description:
Implement a standardized error code system (e.g., VAL-001, SCH-042) that enables quick documentation lookup, better error tracking, and programmatic error handling. Enhance the error reference documentation with error codes, common causes, and troubleshooting steps.

Acceptance Criteria:

• Error code system defined in pkg/constants/ (e.g., VAL-001 = schema validation)
• All validation errors assigned unique error codes
• Error messages include code: [VAL-001] job name cannot be empty
• Error codes documented in docs/src/content/docs/troubleshooting/errors.md
• Documentation includes: error code, message, cause, solution, related errors
• CLI command: gh aw error VAL-001 to look up error details
• Error codes grouped by category (VAL=validation, SCH=schema, SEC=security, MCP=MCP config)

Code Region: pkg/constants/, pkg/workflow/*validation*.go, docs/src/content/docs/troubleshooting/errors.md

Implement validation error code system:

1. Create pkg/constants/error_codes.go:
   - Define error code constants (e.g., ErrSchemaInvalidField = "SCH-001")
   - Group by category: VAL (general), SCH (schema), SEC (security), MCP (MCP), NET (network)
   - Include error code metadata (category, severity, documentation URL)
2. Update validation functions to use error codes:
   - `return fmt.Errorf("[%s] %s", ErrSchemaInvalidField, message)`
3. Enhance docs/src/content/docs/troubleshooting/errors.md:
   - Add error code to each documented error
   - Create error code index (searchable/sortable table)
   - Add "Related Errors" section linking similar issues
4. Implement `gh aw error <code>` command in pkg/cli/:
   - Look up error code in embedded documentation
   - Display: code, message, cause, solution, related errors, docs URL
5. Update validation tests to verify error codes are present and correct

---

Task 5: Add Actionable Recovery Suggestions to Common Validation Errors

Priority: Low
Estimated Effort: Medium
Focus Area: Error Actionability

Description:
Only ~5 validation errors currently include concrete recovery suggestions or "next steps." Enhance the top 20 most common validation errors with actionable guidance, code examples, and auto-fix suggestions where possible. This helps users self-serve and reduces support burden.

Acceptance Criteria:

• Top 20 validation errors include "Suggestion:" or "Try:" section
• Suggestions are specific and actionable (not generic advice)
• Include code examples showing correct usage when applicable
• For typos/misspellings, suggest correct values (leveraging existing fuzzy matching)
• Multi-step fixes are numbered for clarity
• Suggestions reference relevant documentation sections
• Error messages follow consistent suggestion format

Code Region: pkg/workflow/*validation*.go (focus on most common errors)

Enhance validation errors with actionable recovery suggestions:

1. Identify top 20 most common validation errors by:
   - Reviewing error tests for frequently tested scenarios
   - Checking GitHub issues for commonly reported errors
   - Analyzing error reference documentation usage
2. For each error, add structured suggestions:

Error: [SCH-005] unknown property 'permisions'
Suggestion: Did you mean 'permissions'? Update your frontmatter:

permissions:
contents: read
issues: write

3. Implement suggestion framework in validation_helpers.go:
- formatErrorWithSuggestion(error, suggestion string)
- formatErrorWithExample(error, codeExample string)
- formatErrorWithSteps(error, steps []string)
4. Focus on these high-impact error categories:
- Schema validation (field name typos, type mismatches)
- Strict mode (permission/network configuration)
- MCP configuration (missing required fields, type conflicts)
- Network configuration (wildcard usage, missing domains)
5. Include "Learn more:" links to relevant documentation sections
6. Add tests to verify suggestions are included and accurate

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2026-01-01	Workflow Validation Error UX	Custom	Y	First quality improvement run - established baseline and identified 5 high-impact improvements for validation error user experience

---

🎯 Recommendations

Immediate Actions (This Week)

1. Task 1: Standardize Console Formatting - Priority: High - Improves visual consistency and error clarity across all validation messages
2. Task 2: Add File Context and Line Numbers - Priority: High - Dramatically improves user's ability to locate and fix errors quickly

Short-term Actions (This Month)

1. Task 3: Multi-Error Collection - Priority: Medium - Reduces fix-recompile iteration cycles, saving significant user time
2. Task 4: Error Code System - Priority: Medium - Enables better documentation, tracking, and programmatic error handling

Long-term Actions (This Quarter)

1. Task 5: Recovery Suggestions - Priority: Low - Reduces support burden by making errors more self-service friendly
2. Consider implementing auto-fix suggestions for common errors (typos, formatting)
3. Explore validation error analytics to identify most problematic error types

---

📈 Success Metrics

Track these metrics to measure improvement in the Workflow Validation Error UX:

• Console Formatting Coverage: 74/1352 (5%) → 100% of validation errors
• Errors with File Context: ~5% → 95% of validation errors include filename
• Errors with Line Numbers: ~3% → 80% of schema/YAML errors include line numbers
• Documentation URL Inclusion: 7 errors (12%) → 40+ errors (70%+)
• Errors with Recovery Suggestions: ~5 → 30+ common errors
• Error Codes Assigned: 0% → 100% of validation errors
• Multi-Error Support: Single error per compile → Average 3-5 errors collected per failed compilation
• User Time to Fix: Baseline (measure) → 40% reduction in time from error to fix

---

Next Steps

1. Review and prioritize the tasks above
2. Assign tasks to Copilot agent via planner agent
3. Track progress on improvement items
4. Re-evaluate this focus area in 4-6 weeks to measure impact
5. Next analysis: 2026-01-02 - Focus area will be selected based on diversity algorithm (targeting 60% custom areas)

---

Generated by Repository Quality Improvement Agent
Next analysis: 2026-01-02 - New custom focus area will be selected to maintain diversity

AI generated by Repository Quality Improvement Agent