🎯 Repository Quality Improvement - Workflow Observability & Debugging Experience

[github-actions[bot]]

🎯 Repository Quality Improvement Report - Workflow Observability & Debugging Experience

Analysis Date: 2025-11-25
Focus Area: Workflow Observability & Debugging Experience
Strategy Type: Custom
Custom Area: Yes - This focus area addresses a critical user success factor specific to gh-aw as a workflow compilation and execution tool. Understanding why workflows fail, debugging issues, and optimizing performance directly impacts developer productivity and tool adoption.

Executive Summary

Analysis of the gh-aw repository reveals a mature debugging infrastructure with comprehensive log analysis (logs command), failure investigation (audit command), and MCP server inspection capabilities. The tooling includes engine-specific log parsers, firewall analysis, and structured reporting with both console and JSON output formats.

However, several gaps exist in the debugging experience: only 3.4% of error messages include examples, 474 instances of direct print statements bypass proper console formatting, and error recovery guidance could be more comprehensive. The custom focus on "Workflow Observability" uncovered opportunities to improve error message quality, enhance debugging documentation, and strengthen the feedback loop for users troubleshooting failed agentic workflows.

Full Analysis Report

Focus Area: Workflow Observability & Debugging Experience

Current State Assessment

The repository demonstrates strong foundations in workflow observability with dedicated commands and comprehensive log analysis capabilities. The system provides multiple layers of debugging support from low-level debug logging to high-level audit reports.

Metrics Collected:

Metric	Value	Status
Debug logger usage	168 instances	✅ Good
Console formatting calls	881 instances	✅ Excellent
Direct print statements	474 instances	⚠️ Needs cleanup
Error messages with examples	34 / 989 (3.4%)	❌ Low coverage
Actionable error messages	59 / 989 (6.0%)	⚠️ Needs improvement
Log parser implementations	3 engines	✅ Good
Observability test files	44 files	✅ Strong
Documentation coverage	29 / 46 docs (63%)	✅ Good

Findings

Strengths

• Comprehensive Debug Commands: The logs, audit, and mcp inspect commands provide thorough workflow analysis capabilities
• Structured Reporting: Dual output format (console + JSON) supports both human and programmatic consumption
• Engine-Specific Parsers: Dedicated log parsers for Copilot, Claude, and Codex engines
• Firewall Analysis: Automatic parsing and analysis of network traffic logs
• Test Coverage: 44 test files dedicated to observability features demonstrate commitment to reliability
• Console Rendering System: Sophisticated struct tag-based rendering with formatting support (number, cost)
• Debug Logger Package: Well-designed logger with namespace filtering and zero-overhead when disabled

Areas for Improvement

• Error Message Quality: Only 3.4% of error messages include examples, making it harder for users to fix issues quickly
• Logging Consistency: 474 direct print statements bypass the established console formatting system
• Missing Troubleshooting Workflows: No dedicated "debugging guide" for common workflow failure scenarios
• Limited Error Recovery: Only 6% of errors provide actionable recovery guidance
• MCP Debugging Gap: While mcp inspect exists, there's no integrated MCP server debugging workflow
• Performance Profiling: No built-in tooling for identifying slow workflow execution bottlenecks

Detailed Analysis

1. Error Message Quality Assessment

Current State:

• Total error messages: 989
• Messages with examples: 34 (3.4%)
• Actionable messages: 59 (6.0%)

Good Patterns Found:

// Excellent: Clear, with valid options and example
fmt.Errorf("invalid engine: %s. Valid engines are: copilot, claude, codex, custom. Example: engine: copilot", engineID)

// Excellent: Explains constraint, shows correct format
fmt.Errorf("manual-approval value must be a string, got %T. Example: manual-approval: \"production\"", manualApproval)

Problematic Patterns Found:

// Poor: No context, no example
fmt.Errorf("failed to resolve %s@%s: %w", repo, version, err)

// Poor: No recovery guidance
fmt.Errorf("invalid SHA format for %s@%s: %s", repo, version, sha)

Impact: Users spend more time investigating errors when messages lack examples and recovery guidance.

2. Logging Infrastructure Analysis

Console Formatting System:

• ✅ 881 uses of console.Format* functions
• ✅ Struct tag-based rendering system (console:"header:Name")
• ✅ Support for number and cost formatting
• ❌ 474 direct print statements that should use console formatting

Debug Logger System:

• ✅ 168 logger instances with namespace-based filtering
• ✅ Zero-overhead design (logs only computed when DEBUG matches)
• ✅ Time diff tracking between log calls
• ✅ Auto-colorization for terminal output

Gap: While the infrastructure is sophisticated, inconsistent adoption (474 direct prints) reduces effectiveness.

3. Debugging Tools Ecosystem

Available Commands:

gh aw logs Command:

• Downloads artifacts and logs for workflow runs
• Supports filtering by engine, date range, branch, run ID
• Generates tool usage graphs with --tool-graph
• Parses logs with --parse flag
• JSON output for programmatic analysis

gh aw audit Command:

• Analyzes individual workflow runs
• Detects errors and warnings
• Provides failure analysis with recommendations
• Generates concise markdown reports
• Supports both run IDs and GitHub URLs

gh aw mcp inspect Command:

• Lists workflows with MCP configurations
• Inspects MCP server capabilities
• Shows available tools, resources, and roots
• Validates required secrets
• Launches MCP inspector web interface

Gaps:

• No unified "debug workflow" that guides users through common failure scenarios
• Limited integration between commands (e.g., audit could suggest mcp inspect for MCP-related failures)
• No performance profiling built into the tools

4. Log Parsing and Analysis

Engine-Specific Parsers:

• parse_copilot_log.cjs - Parses GitHub Copilot CLI output
• parse_claude_log.cjs - Parses Anthropic Claude API logs
• parse_codex_log.cjs - Parses OpenAI Codex API logs

Firewall Log Analysis:

• 3 implementation files for firewall log parsing
• Classifies requests as allowed/denied
• Aggregates by domain
• Integration with logs and audit commands

Structured Reporting:

• audit_report.go - Generates audit reports with findings and recommendations
• logs_report.go - Generates log summaries with metrics and tool usage

Gap: No visibility into MCP server internal errors beyond basic connectivity checks.

5. Documentation for Debugging

Coverage:

• 29 out of 46 documentation files mention debugging/troubleshooting
• setup/cli.md includes basic command usage
• Reference docs cover individual features

Gaps:

• No dedicated "Troubleshooting Guide" or "Debugging Workflows" section
• Limited "common failure scenarios" documentation
• No visual debugging workflow diagrams
• Missing "how to read audit reports" tutorial

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.

Task 1: Improve Error Message Quality with Examples

Priority: High
Estimated Effort: Medium
Focus Area: Workflow Observability & Debugging Experience

Description:
Enhance error messages across the workflow compilation and execution system to include concrete examples and actionable guidance. Target error messages in validation, compilation, and runtime phases that currently lack examples.

Acceptance Criteria:

• At least 50% of error messages in pkg/workflow/*.go include examples
• Error messages follow the template: "[what's wrong]. [what's expected]. [example]"
• Examples use realistic YAML syntax appropriate for agentic workflows
• Tests validate error message content includes "Example:"

Code Region: pkg/workflow/*.go (excluding files with existing examples: time_delta.go, manual_approval.go, engine_validation.go)

Review error messages in the workflow package and improve them to include concrete examples following the Error Message Style Guide. Focus on validation errors in these files:
- `pkg/workflow/action_resolver.go`
- `pkg/workflow/add_comment.go`
- `pkg/workflow/add_labels.go`
- `pkg/workflow/add_reviewer.go`
- `pkg/workflow/docker_validation.go`

For each error message without an example:
1. Identify what validation failed
2. Explain what format or value is expected
3. Provide a concrete YAML example showing correct usage

Follow the pattern from `time_delta.go`:
```go
fmt.Errorf("invalid time delta format: +%s. Expected format like +25h, +3d, +1w, +1mo, +1d12h30m", deltaStr)

Add tests to verify error messages contain "Example:" where appropriate.

---

#### Task 2: Eliminate Direct Print Statements

**Priority**: Medium  
**Estimated Effort**: Large  
**Focus Area**: Workflow Observability & Debugging Experience

**Description:**
Replace 474 instances of direct `fmt.Println` and `fmt.Printf` calls with proper console formatting functions to ensure consistent, styled output across all CLI commands.

**Acceptance Criteria:**
- [ ] All user-facing output uses `fmt.Fprintf(os.Stderr, ...)` instead of `fmt.Println`
- [ ] Error messages use `console.FormatErrorMessage()`
- [ ] Success messages use `console.FormatSuccessMessage()`
- [ ] Info messages use `console.FormatInfoMessage()`
- [ ] Warning messages use `console.FormatWarningMessage()`
- [ ] No linting errors for direct print usage

**Code Region:** `pkg/**/*.go` and `cmd/**/*.go` (excluding test files)

```markdown
Refactor all direct print statements to use the console formatting system:

1. Identify all instances of `fmt.Println()` and `fmt.Printf()` that should use stderr
2. Replace with appropriate console formatting:
   - Errors: `fmt.Fprintln(os.Stderr, console.FormatErrorMessage(err.Error()))`
   - Success: `fmt.Fprintln(os.Stderr, console.FormatSuccessMessage("message"))`
   - Info: `fmt.Fprintln(os.Stderr, console.FormatInfoMessage("message"))`
   - Warnings: `fmt.Fprintln(os.Stderr, console.FormatWarningMessage("message"))`

3. For simple messages without formatting, use: `fmt.Fprintf(os.Stderr, "message\n")`

Do NOT modify:
- Test files (*_test.go)
- Debug logger output (logger.Printf)
- Code that already uses fmt.Fprintf(os.Stderr, ...)

Reference: `.github/instructions/github-agentic-workflows.instructions.md` - Console Message Formatting section

---

Task 3: Create Comprehensive Troubleshooting Guide

Priority: High
Estimated Effort: Medium
Focus Area: Workflow Observability & Debugging Experience

Description:
Create a new troubleshooting guide that walks users through diagnosing and fixing common workflow failures using the available debugging tools (logs, audit, mcp inspect).

Acceptance Criteria:

• New documentation file created in docs/src/content/docs/guides/troubleshooting.md
• Covers at least 5 common failure scenarios with step-by-step solutions
• Includes decision tree or flowchart for debugging approach
• Shows examples of using logs, audit, and mcp inspect commands
• Links to relevant reference documentation
• Follows Diátaxis "How-to Guide" format (goal-oriented)

Code Region: docs/src/content/docs/guides/ (new file)

Create a comprehensive troubleshooting guide for debugging agentic workflows:

## Structure:

1. **Introduction**: Brief overview of available debugging tools
2. **Quick Diagnosis Flowchart**: Decision tree for identifying issue category
3. **Common Scenarios**:
   - Workflow compilation errors
   - Runtime failures with error patterns
   - MCP server connection issues
   - Permission errors
   - Timeout issues
   - Network/firewall restrictions

4. **Tool Usage Patterns**:
   - When to use `gh aw logs` vs `gh aw audit`
   - How to read audit reports
   - Interpreting firewall logs
   - Using `mcp inspect` for tool debugging

5. **Advanced Debugging**:
   - Enabling debug logging with DEBUG environment variable
   - Analyzing tool usage patterns
   - Performance profiling with `--tool-graph`

Use Astro Starlight syntax with asides for notes, tips, and warnings. Include code examples showing actual command usage and expected output.

Reference existing debugging features from:
- `pkg/cli/logs.go`
- `pkg/cli/audit.go`  
- `pkg/cli/mcp_server.go`

---

Task 4: Add Integrated MCP Debugging Workflow

Priority: Medium
Estimated Effort: Medium
Focus Area: Workflow Observability & Debugging Experience

Description:
Enhance the audit command to automatically suggest MCP server inspection when MCP-related failures are detected, and add a --inspect-mcp flag that automatically runs mcp inspect for failed MCP servers.

Acceptance Criteria:

• Audit report detects MCP server failures from error patterns
• Audit recommendations suggest mcp inspect command when MCP issues found
• Optional --inspect-mcp flag triggers automatic MCP inspection
• MCP inspection results integrated into audit report
• Tests validate MCP failure detection and recommendation generation

Code Region: pkg/cli/audit*.go, pkg/cli/mcp_server.go

Enhance the audit command with MCP debugging integration:

1. In `generateRecommendations()` function in `audit_report.go`:
   - Detect MCP-related errors in the error patterns
   - When MCP failures detected, add recommendation:
     "MCP server issue detected. Run: gh aw mcp inspect (workflow-name) --server (server-name)"

2. Add new flag to audit command:
   ```go
   auditCmd.Flags().Bool("inspect-mcp", false, "Automatically inspect MCP servers if failures detected")

3. When --inspect-mcp is true and MCP failures detected:

   • Extract MCP server names from failures
   • Call MCP inspect functionality programmatically
   • Include MCP inspection results in audit report under new section "MCP Server Analysis"

4. Add tests:

   • Test MCP failure detection in error patterns
   • Test recommendation generation for MCP issues
   • Test --inspect-mcp flag integration

This creates a seamless debugging experience where users get immediate MCP insights without manually running separate commands.

---

#### Task 5: Performance Profiling and Bottleneck Detection

**Priority**: Medium  
**Estimated Effort**: Small  
**Focus Area**: Workflow Observability & Debugging Experience

**Description:**
Add performance metrics to the `logs` command output showing execution time breakdown by phase (activation, agent execution, safe outputs) and identify slow steps in workflow runs.

**Acceptance Criteria:**
- [ ] Logs summary includes average execution time by phase
- [ ] Identify and flag slow runs (> 2x median duration)
- [ ] Show execution time distribution (p50, p90, p99)
- [ ] Add `--performance` flag for detailed timing analysis
- [ ] Performance metrics included in JSON output

**Code Region:** `pkg/cli/logs_report.go`, `pkg/cli/logs.go`

```markdown
Enhance logs command with performance profiling:

1. In `LogsSummary` struct in `logs_report.go`:
   - Add fields: `AverageDuration`, `MedianDuration`, `P90Duration`, `P99Duration`
   - Add `SlowRuns []int64` for run IDs exceeding 2x median

2. In log processing:
   - Calculate duration percentiles from run data
   - Identify outliers (runs > 2x median)
   - Track time spent in each phase (if available in logs)

3. Add `--performance` flag:
   - When enabled, show detailed timing breakdown per run
   - Display phase-level timing if available
   - Highlight slow steps within runs

4. Update console rendering:
   - Add "Performance Metrics" section to summary
   - Use duration formatting (e.g., "5m 30s", "1h 15m")
   - Flag slow runs with ⚠️ indicator

5. Include in JSON output under `summary.performance`

This helps users identify performance bottlenecks and optimize workflow execution time.

---

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2025-11-25	Workflow Observability & Debugging Experience	Custom	Y	First quality improvement run - identified error message quality and logging consistency opportunities

Statistics:

• Total runs: 1
• Custom rate: 100.0% (1 custom / 1 total)
• Unique areas explored: 1

---

🎯 Recommendations

Immediate Actions (This Week)

1. Improve Top 20 Error Messages - Priority: High

   • Target validation errors in pkg/workflow/ files
   • Add examples following the error message style guide
   • Focus on user-facing errors that happen during compilation

2. Create Troubleshooting Documentation - Priority: High

   • New guide in docs/src/content/docs/guides/troubleshooting.md
   • Cover 5 most common failure scenarios
   • Include debugging tool usage examples

Short-term Actions (This Month)

1. Eliminate Direct Print Statements - Priority: Medium

   • Refactor to use console formatting system
   • Ensure all user output goes to stderr
   • Update linting rules to catch violations

2. Add MCP Debugging Integration - Priority: Medium

   • Enhance audit command with automatic MCP inspection
   • Integrate MCP insights into failure reports
   • Reduce steps needed to diagnose MCP issues

Long-term Actions (This Quarter)

1. Performance Profiling System - Priority: Low

   • Add execution time breakdown by phase
   • Identify and flag slow workflow runs
   • Provide optimization recommendations

2. Interactive Debugging Mode - Priority: Low

   • Consider adding interactive debugging session
   • Allow stepping through workflow execution
   • Real-time MCP server interaction testing

---

📈 Success Metrics

Track these metrics to measure improvement in Workflow Observability & Debugging Experience:

• Error Message Quality: 3.4% → 50%+ with examples
• Logging Consistency: 474 → 0 direct print statements
• Troubleshooting Documentation: 0 → 1 comprehensive guide
• Mean Time to Debug (MTTD): Establish baseline, target 30% reduction
• Audit Command Usage: Track adoption with telemetry (if available)

---

Next Steps

1. Review and prioritize the tasks above
2. Assign tasks to Copilot agent via planner agent
3. Track progress on improvement items
4. Re-evaluate this focus area in 3 months to measure impact

---

Generated by Repository Quality Improvement Agent
Next analysis: 2025-11-26 - Focus area will be selected based on diversity algorithm

AI generated by Repository Quality Improvement Agent

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.