📚 Documentation Noob Test Report - January 3, 2026

[github-actions[bot]]

Executive Summary

Date of Test: January 3, 2026
Testing Method: Manual HTML analysis (Playwright unavailable in environment)
Pages Analyzed:

• Homepage: /gh-aw/
• Quick Start Guide: /gh-aw/setup/quick-start/
• CLI Commands: /gh-aw/setup/cli/
• Creating Workflows: /gh-aw/setup/agentic-authoring/
• Example workflows (2): Manual trigger, IssueOps

Overall Impression as a New User:
The documentation is comprehensive and well-structured with good organization. However, there are some critical gaps in the beginner journey that could leave new users confused or blocked.

---

🔴 Critical Issues Found

1. Quick Start - Missing Prerequisites Clarity

Location: Quick Start Guide - Prerequisites section
Issue: The prerequisites section mentions "Before installing, ensure you have:" but the actual list is not immediately clear in the parsed content. For a complete beginner, it's not obvious what they need before starting.

Expected: Clear bullet list:

• GitHub account
• Git installed
• GitHub CLI (gh) installed and authenticated
• Access to a repository with Actions enabled
• Basic familiarity with terminal commands

Impact: HIGH - Users may start the tutorial without necessary tools installed, leading to failure and frustration.

2. Authentication Token Confusion ⚠️ BLOCKER

Location: Quick Start Guide - Step 3 "Add an AI secret"
Issue: The guide mentions "Create a Personal Access Token (PAT)" and "Add the token to your repository" but doesn't clearly explain:

• WHY you need this token (what API is it for?)
• What permissions/scopes are required
• The difference between GitHub PAT and AI service tokens (OpenAI, Anthropic, etc.)

Expected: Clear explanation that this is for AI services (not GitHub), with specific scopes needed for each provider.

Impact: CRITICAL - This is a blocker. Users won't know what token to create or where to get it.

3. "Engine" Concept Not Explained Early

Location: Homepage and Quick Start
Issue: The documentation mentions "AI agents" and shows workflow examples but doesn't explain what an "engine" is until later. The Quick Start sample workflow likely contains engine: copilot but beginners won't understand what this means or what other options exist.

Expected: Early explanation in Quick Start:

What is an Engine? An engine is the AI service that powers your workflow (like GitHub Copilot, Claude, or OpenAI Codex). You'll specify which engine to use in your workflow's frontmatter.

Impact: HIGH - Confusion about core concepts makes it hard to customize or troubleshoot.

4. No Clear "What Happens Next" After Quick Start

Location: Quick Start Guide - End of guide
Issue: After completing the Quick Start, there's a "What's next?" section but it's not clear what a beginner should do:

• Should they modify the sample workflow?
• Should they create their own workflow?
• What are the most common next steps?

Expected: Clear next steps:

1. ✅ Understand the workflow structure (link to reference)
2. ✅ Modify the sample to learn (link to tutorial)
3. ✅ Browse examples for inspiration (link to examples)
4. ✅ Create your first custom workflow (link to authoring guide)

Impact: MEDIUM - Users complete the Quick Start but don't know where to go next.

---

🟡 Confusing Areas

1. Two-File System (.md + .lock.yml) Not Emphasized Enough

Location: Quick Start Guide
Issue: The concept of markdown source files and compiled lock files is mentioned but not prominently displayed. This is a CORE concept that beginners need to understand immediately.

Suggestion: Add a prominent callout box early:

💡 Key Concept: You work with TWO files:

• .md file = Your workflow source (human-editable)
• .lock.yml file = Compiled GitHub Actions workflow (auto-generated, DO NOT EDIT)

Always edit the .md file, then run gh aw compile to regenerate the .lock.yml.

Impact: MEDIUM - Users might edit .lock.yml files directly and lose changes.

2. Jargon Without Definitions

Location: Throughout documentation
Issues found:

• "Frontmatter" - Not defined until you click through
• "MCP server" - Acronym not explained upfront
• "Safe outputs" - Technical term without context
• "Agentic" - Marketing term, not clear what it means technically

Suggestion: Add a glossary link in the sidebar AND inline tooltips for key terms.

Impact: MEDIUM - Slows down learning, requires multiple page visits to understand concepts.

3. CLI Commands Page Overwhelming

Location: CLI Commands page
Issue: While the page has a "🚀 Most Common Commands" section (good!), there are SO many commands listed that it's overwhelming for a beginner. The page lists ~20+ commands when 95% of users only need 5.

Suggestion:

1. Move "Most Common Commands" to the very top (before "Overview")
2. Collapse advanced commands into an expandable "Advanced Commands" section
3. Add a "You probably won't need these" note for rarely-used commands

Impact: MEDIUM - Cognitive overload for beginners.

4. Examples Section Navigation

Location: Examples section
Issue: The examples are organized by trigger type (manual, scheduled, issue-pr-events, etc.) which makes sense technically, but beginners often want to browse by USE CASE:

• "I want to auto-label issues"
• "I want to create daily reports"
• "I want to respond to PR comments"

Suggestion: Add a "Browse by Use Case" view or tags on example pages.

Impact: LOW - Examples are findable, just takes longer.

5. "Creating Workflows" vs "Agentic Authoring" Terminology

Location: Setup section
Issue: The sidebar shows "Agentic Authoring" but the page title is "Creating Workflows". For beginners, "Agentic Authoring" is jargon-heavy and doesn't clearly communicate what the page is about.

Suggestion: Rename to "Creating Workflows (Using AI Assistance)" or split into:

• "Writing Workflows Manually"
• "Using GitHub Copilot to Generate Workflows"

Impact: LOW - Minor confusion, but page content is good once you're there.

---

🟢 What Worked Well

1. Clear Installation Instructions

The installation commands for the CLI extension are clear and copy-pasteable:

gh extension install githubnext/gh-aw

This is perfect - no ambiguity.

2. Step-by-Step Quick Start Structure

The Quick Start follows a logical progression:

• Install extension
• Add sample workflow
• Add AI secret
• Trigger workflow

This flow makes sense and is easy to follow.

3. "🚀 Most Common Commands" Section

The CLI page includes a fantastic "Most Common Commands" section that highlights the 5 commands most users need. This is EXACTLY what beginners need.

4. Good Use of Callout Boxes

The documentation uses callout boxes (Caution, Note, Tip) effectively to highlight important information.

5. Workflow Examples Are Real and Practical

The example workflows (daily team status, IssueOps, etc.) are realistic use cases, not toy examples. This helps beginners see the value.

---

📊 Detailed Findings by Page

Homepage Analysis

Strengths:

• Clear tagline: "write AI-powered automation workflows in natural language"
• Immediate CTA to Quick Start
• Examples visible on homepage

Weaknesses:

• Doesn't explain what "agentic" means
• No visual diagram showing how it works
• Jump to "Key Features" doesn't define core concepts first

Beginner-Friendly Score: 7/10

Quick Start Guide Analysis

Strengths:

• Numbered steps (Step 1, Step 2, etc.)
• Uses actual commands that can be copy-pasted
• Includes verification steps

Weaknesses:

• Prerequisites not detailed enough
• Token/authentication step is confusing
• Doesn't explain what the sample workflow does before you add it
• Missing "expected output" for each step

Beginner-Friendly Score: 6/10

CLI Commands Analysis

Strengths:

• Excellent "Most Common Commands" section
• Comprehensive coverage of all commands
• Good organization with sections

Weaknesses:

• Too many commands visible at once (cognitive overload)
• No "Basic vs Advanced" distinction
• Examples could be more prominent

Beginner-Friendly Score: 7/10

Creating Workflows (Agentic Authoring) Analysis

Strengths:

• Explains the gh aw init command clearly
• Shows integration with VS Code/Copilot

Weaknesses:

• Title "Agentic Authoring" is jargon
• Assumes familiarity with GitHub Copilot
• Doesn't show what a manually-written workflow looks like first
• "Dictation" section seems out of place for beginners

Beginner-Friendly Score: 5/10

---

🎯 Prioritized Recommendations

🔥 Quick Wins (1-2 hours each)

1. Add Prerequisite Checklist to Quick Start

   • Clear bullet list of what you need BEFORE starting
   • Include links to install each prerequisite
   • Est. time: 30 minutes

2. Add "Core Concepts" Glossary Box to Homepage

   • Define: Workflow, Engine, Frontmatter, Agent, Lock file
   • Make it a sticky/pinned element or prominent callout
   • Est. time: 45 minutes

3. Clarify Token/Authentication Section ⚠️ URGENT

   • Add explicit callout: "This token is for AI services (OpenAI, Anthropic, GitHub Copilot), NOT your GitHub token"
   • Link to token creation pages for each provider
   • Est. time: 30 minutes

4. Add "What to Expect" for Each Quick Start Step

   • Show sample output after each command
   • Add "✓ Success looks like this" boxes
   • Est. time: 1 hour

Total Quick Wins Time: ~3 hours

📈 High-Impact Improvements (4-8 hours each)

5. Create Visual "How It Works" Diagram

   • Show: Write .md → Compile → .lock.yml → GitHub Actions → AI Agent
   • Add to homepage and Quick Start
   • Est. time: 4 hours

6. Reorganize CLI Commands Page

   • Keep "Most Common Commands" at top
   • Collapse advanced commands into expandable sections
   • Add visual indicators for "Beginner" vs "Advanced"
   • Est. time: 3 hours

7. Add "Your First Custom Workflow" Tutorial

   • Step-by-step guide to creating a workflow from scratch
   • Explain each frontmatter field
   • Show debugging process
   • Est. time: 6 hours

8. Improve "What's Next" Section

   • Add decision tree: "What do you want to do?"
   • Link to appropriate next steps based on goal
   • Est. time: 2 hours

Total High-Impact Time: ~15 hours

🚀 Long-Term Improvements

9. Interactive Tutorial/Playground

   • In-browser tutorial where users can try commands
   • Live preview of workflow compilation
   • Est. time: 20+ hours

10. Video Walkthrough

    • 5-minute "Getting Started" video
    • Show actual workflow creation, compilation, and execution
    • Est. time: 8-12 hours (filming + editing)

11. Use Case-Based Navigation

    • Reorganize examples by use case (auto-labeling, daily reports, etc.)
    • Add tags and filtering
    • Est. time: 10 hours

---

🧪 Testing Methodology Note

Limitation: This test was conducted via HTML analysis rather than live browser interaction due to Playwright installation restrictions in the GitHub Actions environment. This means:

• ✅ Content structure and text analysis: Complete
• ✅ Link structure and navigation: Analyzed
• ❌ Visual design and layout: Not assessed
• ❌ Interactive elements (dropdowns, search): Not tested
• ❌ Mobile responsiveness: Not tested
• ❌ Accessibility (screen readers, keyboard nav): Not tested

A follow-up test with full browser automation would be valuable for assessing:

• Visual hierarchy and information density
• Search functionality effectiveness
• Mobile user experience
• Accessibility compliance

---

📞 Conclusion

Overall Assessment: The documentation is well-structured and comprehensive, but has significant gaps in the beginner onboarding experience.

Key Takeaway: A brand-new user can probably get through the Quick Start with some confusion, but will struggle with:

1. Understanding what tokens to create and why (CRITICAL BLOCKER)
2. Grasping core concepts (engine, frontmatter, lock files)
3. Knowing what to do after Quick Start

Most Urgent Fix: The authentication/token section in Quick Start is a critical blocker. This should be addressed immediately.

Biggest Opportunity: Adding a "Core Concepts" section or prominent glossary would dramatically improve the beginner experience across all pages.

---

Report Generated: January 3, 2026
Tester Role: Brand new user (simulated)
Environment: GitHub Actions runner with local documentation server
Documentation Version: Built from current main branch

Labels: documentation, user-experience, automated-testing

AI generated by Documentation Noob Tester