🎯 Repository Quality Improvement - Workflow Error Message Quality [2025-11-15]

[github-actions[bot]]

🎯 Repository Quality Improvement Report - Workflow Error Message Quality

Analysis Date: 2025-11-15
Focus Area: Workflow Error Message Quality and Developer Experience
Strategy Type: Custom
Custom Area: Yes - This focus area addresses a critical quality aspect unique to gh-aw as a workflow compilation tool where error messages are the primary interface for developers to understand compilation failures.

Executive Summary

The analysis reveals that gh-aw has established an excellent error message style guide (.github/instructions/error-messages.instructions.md) with clear templates and examples. However, compliance is currently at only 8% - only 20 out of 237 error messages follow the recommended format of providing "what's wrong + what's expected + example."

While the codebase demonstrates sophisticated error handling infrastructure with 824 console formatting calls, 259 error generation points, and comprehensive validation test coverage (7,682 lines), the vast majority of error messages lack the actionable examples that would significantly improve the developer experience when workflow compilation fails.

This represents a high-impact improvement opportunity: enhancing error message quality would directly reduce debugging time, improve onboarding, and increase confidence in the tool without requiring major architectural changes.

Full Analysis Report

Focus Area: Workflow Error Message Quality and Developer Experience

Current State Assessment

Metrics Collected:

Metric	Value	Status
Total error message points (fmt.Errorf)	259	ℹ️ Info
Error messages with examples	20	⚠️ Low
Compliance rate	8%	❌ Needs improvement
Console formatting usage	824 calls	✅ Good
Error handling test lines	7,682	✅ Excellent
Validation-related files	44 files	✅ Comprehensive
Error style guide existence	Yes	✅ Present

Findings

Strengths

1. Excellent Error Message Style Guide: The .github/instructions/error-messages.instructions.md file provides comprehensive guidance with clear templates, examples, and anti-patterns.

2. Strong Test Coverage: With 7,682 lines of validation and error-related tests, the codebase demonstrates commitment to quality and reliability.

3. Sophisticated Console Formatting: The console package provides 10+ formatting functions (FormatErrorMessage, FormatWarningMessage, etc.) for consistent user-facing output.

4. Good Examples Exist: Several files demonstrate excellent error messages:

   • time_delta.go: "invalid time delta format: +%s. Expected format like +25h, +3d, +1w, +1mo, +1d12h30m"
   • manual_approval.go: "manual-approval value must be a string, got %T. Example: manual-approval: "production""
   • repository_features_validation.go: "invalid repository format: %s. Expected format: owner/repo. Example: githubnext/gh-aw"

5. Dedicated Error Quality Tests: The file error_message_quality_test.go exists to validate error message quality.

Areas for Improvement

1. Low Compliance with Error Message Standards (Critical)

   • Only 8% of error messages follow the "what's wrong + what's expected + example" template
   • 92% of error messages (217 out of 237) lack actionable examples
   • Many errors provide the problem but not the solution path

2. Inconsistent Error Message Patterns (High)

   • Some validation errors are bare ("empty time delta")
   • Others wrap errors without context ("failed to parse configuration: %w")
   • No systematic enforcement of the style guide during code review

3. Missing Examples in Complex Validations (High)

   • Schema validation errors reference external schema but don't provide inline examples
   • MCP configuration errors could benefit from showing correct YAML structure
   • Engine validation errors don't always list valid options

4. CLI Error Presentation (Medium)

   • While console formatting is used extensively (25 instances), not all compilation errors flow through the formatting layer
   • Some errors are returned raw without user-friendly formatting

Detailed Analysis

Error Message Categories by Quality

Category A: Excellent (8% - 20 messages)
Follow the complete template with examples:

fmt.Errorf("invalid time delta format: +%s. Expected format like +25h, +3d, +1w, +1mo, +1d12h30m", deltaStr)
fmt.Errorf("manual-approval value must be a string, got %T. Example: manual-approval: \"production\"", val)

Category B: Good (15% - ~35 messages)
Explain what's wrong and expected but lack examples:

fmt.Errorf("time delta must start with '+', got: %s", deltaStr)
fmt.Errorf("duplicate unit '%s' in time delta: +%s", unit, deltaStr)

Category C: Needs Improvement (77% - ~182 messages)
Minimal context or wrapped errors:

fmt.Errorf("empty time delta")
fmt.Errorf("failed to parse YAML for schema validation: %w", err)

Impact on Developer Experience

Current Developer Journey with Poor Error:

1. Developer writes workflow with typo in engine name: engine: coppilot
2. Runs gh aw compile workflow.md
3. Gets error: invalid engine: coppilot
4. Must search documentation or code to find valid options
5. Tries again with corrected value

Improved Developer Journey with Good Error:

1. Developer writes workflow with typo in engine name: engine: coppilot
2. Runs gh aw compile workflow.md
3. Gets error: invalid engine: coppilot. Valid engines are: copilot, claude, codex, custom. Example: engine: copilot
4. Immediately fixes the typo
5. Success on next attempt

Time Saved: 2-5 minutes per error × frequency of errors = significant productivity gain

Components Most in Need of Improvement

1. pkg/workflow/validation.go - Core validation logic
2. pkg/workflow/engine_validation.go - Engine configuration validation
3. pkg/workflow/schema_validation.go - Schema-based validation errors
4. pkg/workflow/mcp_*.go - MCP configuration validation
5. pkg/workflow/runtime_validation.go - Runtime package validation

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

Task 1: Enhance Engine Validation Error Messages

Priority: High
Estimated Effort: Small
Focus Area: Error Message Quality

Description:
Improve error messages in pkg/workflow/engine_validation.go to follow the error message style guide. Each error should include what's wrong, what's expected, and a concrete example.

Acceptance Criteria:

• All engine validation errors include "Example:" in the message
• Invalid engine errors list all valid options (copilot, claude, codex, custom)
• Type mismatch errors show the received type using %T
• Model validation errors provide example of correct format
• All improved messages have corresponding test assertions

Code Region: pkg/workflow/engine_validation.go

Review all error messages in pkg/workflow/engine_validation.go and enhance them to follow the error message template from .github/instructions/error-messages.instructions.md.

For each error message:
1. State what's wrong clearly
2. Explain what's expected (valid values, formats, types)
3. Provide a concrete example using "Example:" prefix

Reference the excellent examples in time_delta.go and manual_approval.go for the proper format.

Ensure all error messages follow YAML syntax in examples (use quotes for strings, proper indentation).

Add test assertions to verify error messages contain "Example:" where appropriate.

---

Task 2: Improve MCP Configuration Error Messages

Priority: High
Estimated Effort: Medium
Focus Area: Error Message Quality

Description:
Enhance error messages across all MCP-related validation files to provide actionable examples of correct MCP server configurations.

Acceptance Criteria:

• MCP server configuration errors show complete YAML examples
• Type errors for MCP fields include received type and expected type
• Mode validation errors list valid modes (local, remote) with examples
• Command/container mutual exclusivity errors show both options
• HTTP MCP URL validation errors provide example URLs
• All multi-line YAML examples use proper \n formatting

Code Region: pkg/workflow/mcp_*.go (all MCP-related files)

Systematically improve error messages in all MCP configuration validation files:
- pkg/workflow/mcp_config.go
- pkg/workflow/mcp_config_validation.go
- pkg/workflow/mcp_http_*.go
- pkg/workflow/mcp_servers.go

Apply the error message template:
[what's wrong]. [what's expected]. [example of correct usage]

For complex MCP configurations, provide multi-line YAML examples:
Example:\ntools:\n  my-tool:\n    command: "npx `@my/tool`"

Ensure examples use realistic tool names and show proper YAML structure.

---

Task 3: Add Examples to Schema Validation Errors

Priority: Medium
Estimated Effort: Medium
Focus Area: Error Message Quality

Description:
Enhance schema validation error messages to provide inline examples rather than just referencing the schema. When validation fails, show the user exactly what correct YAML should look like.

Acceptance Criteria:

• Schema validation errors extract field name from validation error
• Field-specific examples are provided based on the failed field
• Common schema violations (type mismatches, missing fields) have tailored examples
• Error messages reference both the schema and provide inline examples
• Tests verify example inclusion in schema validation errors

Code Region: pkg/workflow/schema_validation.go

Enhance pkg/workflow/schema_validation.go to provide more actionable error messages when schema validation fails.

Current behavior: Returns generic "GitHub Actions schema validation failed: %w" error.

Improved behavior: Parse the schema validation error to identify which field failed, then provide a targeted example.

For example, if validation fails on "timeout-minutes" field:
"GitHub Actions schema validation failed for 'timeout-minutes': must be an integer. Example: timeout-minutes: 10"

If validation fails on "engine" field:
"GitHub Actions schema validation failed for 'engine': invalid value. Valid engines are: copilot, claude, codex, custom. Example: engine: copilot"

Add helper function extractFieldFromSchemaError() to parse JSON schema validation errors.
Create a map of common field examples for quick lookup.

---

Task 4: Enhance CLI Error Presentation

Priority: Medium
Estimated Effort: Small
Focus Area: Developer Experience

Description:
Ensure all compilation errors are formatted through the console formatting layer before being displayed to users. Add color coding and structure to make errors more scannable.

Acceptance Criteria:

• All compilation errors in pkg/cli/compile_command.go use console.FormatErrorMessage
• Error output includes context (file path, line number when available)
• Multi-error scenarios show a summary count
• Success messages use console.FormatSuccessMessage consistently
• Warning messages about missing files use console.FormatWarningMessage

Code Region: pkg/cli/compile_command.go, pkg/cli/compile*.go

Review error handling in all compile-related CLI commands and ensure consistent use of console formatting:

1. Wrap all error messages with console.FormatErrorMessage() before output
2. Add context to errors (which workflow file, which field, etc.)
3. For validation errors with multiple issues, show:
   "✗ 3 validation errors found in workflow.md:"
   Then list each error with proper formatting

4. Ensure success paths use console.FormatSuccessMessage:
   "✓ Successfully compiled workflow.md → workflow.lock.yml"

5. Add file location context to errors when available:
   "✗ Error in workflow.md (line 15): invalid engine configuration"

Reference existing good examples in pkg/cli/trial_command.go for console formatting patterns.

---

Task 5: Create Error Message Quality Linter

Priority: Low
Estimated Effort: Large
Focus Area: Long-term Quality

Description:
Create an automated linter that checks error messages for compliance with the error message style guide. This should run as part of CI to prevent regression.

Acceptance Criteria:

• Linter detects error messages without "Example:" where required
• Linter identifies error messages that don't explain what's expected
• Linter checks for proper YAML formatting in examples
• Linter can be run as make lint-errors
• CI integration added to prevent non-compliant errors from being merged
• Linter provides suggestions for improving detected issues

Code Region: scripts/lint_error_messages.go (new file)

Create a new error message quality linter in scripts/lint_error_messages.go that:

1. Parses all Go files in pkg/workflow/ and pkg/cli/
2. Extracts error messages from fmt.Errorf, errors.New, and similar patterns
3. Checks each error message against quality criteria:
   - Contains "Example:" when appropriate (validation, configuration errors)
   - Includes "Expected" or "Valid" for format/enum errors
   - Shows received value/type for type mismatches (%T, %v)
   - Uses proper YAML formatting in examples
4. Reports compliance metrics:
   - Total error messages analyzed
   - Compliant count and percentage
   - Non-compliant messages with suggestions
5. Exit with non-zero code if compliance falls below threshold (e.g., 80%)

Add to Makefile:
lint-errors:
	`@go` run scripts/lint_error_messages.go

Update CI workflow to run lint-errors and fail if quality threshold not met.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2025-11-15	Workflow Error Message Quality and Developer Experience	Custom	Y	First analysis - established baseline, identified 8% compliance rate with error message style guide

---

🎯 Recommendations

Immediate Actions (This Week)

1. Improve top 5 most-encountered validation errors - Priority: High

   • Engine validation errors
   • MCP configuration errors
   • Schema validation errors for common fields
   • Time delta format errors (already good, but ensure consistency)
   • Permission validation errors

2. Add error message quality tests - Priority: High

   • Extend error_message_quality_test.go to cover more validation functions
   • Test that critical errors include "Example:" in the message

Short-term Actions (This Month)

1. Systematic improvement of validation errors - Priority: Medium

   • Work through pkg/workflow/ files alphabetically
   • Update 10-15 error messages per file
   • Add tests for each improved message

2. CLI error presentation enhancement - Priority: Medium

   • Ensure all compile errors use console formatting
   • Add context (file, line number) to errors
   • Implement error summary for multiple validation failures

3. Documentation updates - Priority: Medium

   • Add "Common Errors and Solutions" section to documentation
   • Create troubleshooting guide with error message examples

Long-term Actions (This Quarter)

1. Error message quality linter - Priority: Low

   • Automate error message quality checks
   • Integrate into CI pipeline
   • Set target compliance threshold (80%+)

2. Developer experience improvements - Priority: Low

   • Add --explain flag to compile command for detailed error explanations
   • Consider error codes for programmatic error handling
   • Implement "did you mean?" suggestions for common typos

---

📈 Success Metrics

Track these metrics to measure improvement in Workflow Error Message Quality:

• Error Message Compliance Rate: 8% → 60% (short-term), 90% (long-term)
• Error Messages with Examples: 20 → 140 (60% of 237)
• Developer Time to Fix Compilation Error: Baseline TBD → 50% reduction target
• Error-related GitHub Issues: Monitor for reduction in "unclear error" issues
• Style Guide Adherence: 100% for new error messages (enforced via linter)

---

Next Steps

1. Review and prioritize the tasks above
2. Assign Task 1 (Engine Validation) and Task 4 (CLI Error Presentation) as quick wins
3. Begin systematic improvement of MCP configuration errors (Task 2)
4. Track progress with compliance rate metric
5. Re-evaluate this focus area in 2 weeks to measure improvement

---

Generated by Repository Quality Improvement Agent
Next analysis: 2025-11-16 - Focus area will be selected based on diversity algorithm

AI generated by Repository Quality Improvement Agent

[pelikhan]

/plan

[github-actions[bot]]

✅ Agentic Plan Command completed successfully.

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.