📚 Documentation Noob Test Report - 2026-01-04

[github-actions[bot]]

Summary

Date of test: 2026-01-04
Tester perspective: Complete beginner with no prior knowledge of GitHub Agentic Workflows
Pages reviewed:

• Home page (/gh-aw/)
• Quick Start (/gh-aw/setup/quick-start/)
• CLI Commands (/gh-aw/setup/cli/)
• Creating Workflows (/gh-aw/setup/agentic-authoring/)
• About/Overview (/gh-aw/introduction/overview/)
• Glossary (/gh-aw/reference/glossary/)

Overall impression: The documentation is comprehensive and well-structured, but newcomers face a steep learning curve with lots of unfamiliar terminology and concepts introduced simultaneously. The Quick Start guide does a good job explaining the workflow lifecycle, but assumes significant familiarity with GitHub Actions, CLI tools, and AI concepts.

---

🔴 Critical Issues (Block Getting Started)

1. Missing Prerequisites Context

Location: /gh-aw/setup/quick-start/ - Prerequisites section
Issue: The prerequisites list technical requirements but doesn't explain WHY each is needed or WHAT they do for someone unfamiliar with GitHub tooling.

Example problems:

• "GitHub CLI (gh)" - A complete beginner might not know what this is or why it's different from Git
• "GitHub Actions enabled" - No explanation of what GitHub Actions is or how to verify it's enabled
• "Operating System: Linux, macOS, or Windows with WSL" - Why the WSL requirement for Windows isn't explained

Impact: Beginners spend time researching each prerequisite individually instead of understanding the full picture upfront.

Recommendation:

- ✅ **GitHub CLI** (`gh`) - A command-line tool for interacting with GitHub from your terminal. 
  Used to install gh-aw and manage workflows. [Install here](https://cli.github.com)
- ✅ **GitHub Actions** - GitHub's automation platform that runs your workflows in the cloud.
  Check if enabled: Go to your repo → Settings → Actions → General
- ✅ **Git** - Version control system for tracking code changes
- ✅ **Operating System:** Linux, macOS, or Windows with WSL (Windows Subsystem for Linux)
  *Note: WSL is required on Windows because gh-aw uses Unix-specific features*

2. "Agentic" Terminology Barrier

Location: Throughout documentation, especially home page and Quick Start
Issue: The term "agentic" is used extensively before being defined. While there's a glossary entry, beginners encounter the term dozens of times in the Quick Start before learning what it means.

Examples from Quick Start:

• Line 10: "Using agentic workflows" (warning box)
• Line 35: "Agentic Setup" (section heading)
• Line 95: "Agentic workflows need to authenticate"

Impact: Creates confusion and cognitive load. Beginners might think they need to understand "agentic" to proceed, when it's really just a descriptor meaning "AI-powered and autonomous."

Recommendation:

1. Add a simple definition in the Quick Start intro:

> 💡 **What does "agentic" mean?** An agentic workflow uses AI to make smart decisions 
> automatically, like a helpful assistant that understands context. Think of it as 
> automation that can reason and adapt, not just follow fixed rules.

2. Use "AI-powered" more frequently as a simpler alternative
3. Add glossary links on first use in each major section

3. Token Setup Is Confusing

Location: /gh-aw/setup/quick-start/ - Step 3: Add an AI secret
Issue: The Personal Access Token (PAT) setup instructions are detailed but confusing for beginners. Multiple concepts introduced at once: PATs, tokens, secrets, permissions, resource owners, Copilot subscriptions.

Confusing aspects:

• Why "Resource owner: Your personal account" when working with a repository?
• Why "Public repositories" is required for Copilot Requests permission
• The distinction between "Account permissions" vs "Repository permissions"
• The troubleshooting tip about Copilot subscriptions comes AFTER the setup steps

Impact: High drop-off point. Beginners get stuck trying to find the "Copilot Requests" permission or don't understand why they need to select "Public repositories" when working with a private repo.

Recommendation:

1. Add a visual flowchart showing the token creation decision tree
2. Reorganize to check prerequisites FIRST:

#### Before you begin: Check Copilot Access

You need an active GitHub Copilot subscription to create the required token.

**Verify you have Copilot:**
1. Visit https://github.com/settings/copilot
2. You should see "GitHub Copilot" with a status (Individual, Business, or Enterprise)
3. If you see "Get access to GitHub Copilot", you'll need to subscribe first

**Don't have Copilot?** You can use alternative AI engines (Claude, Codex). 
See [AI Engines Reference](/gh-aw/reference/engines/).

#### Create a Personal Access Token

3. Add a "Why this configuration?" callout explaining the seemingly odd requirements

---

🟡 Confusing Areas (Slow Down Learning)

1. Frontmatter vs Markdown vs YAML Confusion

Location: Quick Start "How Agentic Workflows Work" section
Issue: The guide introduces THREE file format concepts simultaneously (Markdown, YAML, frontmatter) without clear boundaries between them.

Quote from docs:

".md file: Human-friendly markdown with natural language instructions and simple YAML frontmatter"

Why it's confusing:

• Beginners see .md and think "Markdown file"
• Then learn it has YAML at the top (frontmatter)
• Then learn it compiles to .lock.yml (more YAML)
• The relationship between these formats isn't visually clear

Recommendation:
Add a visual diagram showing file structure:

### Workflow File Structure

Your workflow file (`.md`) has two parts:

┌─────────────────────────────┐
│ --- │ ← Frontmatter start
│ on: issues │
│ permissions: read-all │ ← YAML configuration
│ --- │ ← Frontmatter end
│ │
│ # Issue Clarifier │
│ │
│ Analyze the issue and │ ← Markdown instructions
│ ask clarifying questions │ (natural language)
│ │
└─────────────────────────────┘
workflow.md

       ↓  gh aw compile

┌─────────────────────────────┐
│ name: Issue Clarifier │
│ on: │
│ issues: │ ← Complete GitHub Actions
│ types: [opened] │ YAML (machine-ready)
│ jobs: │
│ ... │
└─────────────────────────────┘
workflow.lock.yml

### 2. "Safe Outputs" Concept Unclear

**Location:** Throughout documentation, especially Quick Start  
**Issue:** "Safe outputs" is a critical security concept but is introduced as a configuration option without explanation of WHAT it is or WHY you need it.

**Quote from Quick Start:**
```yaml
safe-outputs:
  create-issue:

Problems:

• Beginners see safe-outputs: in examples without understanding its purpose
• The concept isn't explained until much later in reference docs
• The name "safe outputs" doesn't intuitively convey "these are the GitHub operations the AI is allowed to perform"

Recommendation:

1. Add an inline explanation in the first example:

# What the AI can create without direct write permissions
# Think of this as a "pre-approved actions" list
safe-outputs:
  create-issue:  # AI can create issues

2. Add a callout in Quick Start:

:::tip[What are safe outputs?]
Safe outputs let the AI create GitHub content (issues, comments, PRs) without 
giving it direct write access to your repository. The AI generates a "request" 
that gets reviewed and sanitized before being executed. It's like giving 
someone a form to fill out instead of the keys to your database.

Learn more: [Safe Outputs Reference](/gh-aw/reference/safe-outputs/)
:::

3. Compilation Step Not Clearly Explained

Location: Quick Start and CLI Commands
Issue: The documentation mentions "compile" frequently but doesn't explain WHEN you need to compile or WHY until later in the guide.

Confusion points:

• Step 2 says the gh aw add command creates "compiled" .lock.yml - beginners don't know what "compiled" means yet
• Step 4 uses gh aw status to "verify" but doesn't mention checking if files compiled successfully
• The "Customize Your Workflow" section suddenly introduces gh aw compile without explaining when to use it

Recommendation:
Add a "When to Compile" callout early in Quick Start:

:::tip[When to compile workflows]
You need to run `gh aw compile` whenever you:
- Create a new workflow file (`.md`)
- Edit an existing workflow file
- Update the gh-aw extension

**Think of it like this:** `.md` files are your source code, `.lock.yml` files 
are the compiled version that runs. Just like you compile code before running 
it, you compile workflows before GitHub Actions can execute them.

**Good news:** The `gh aw add` command compiles automatically, so you don't need 
to compile manually after adding workflows from the collection.
:::

4. The "Add a Sample Workflow" Step Creates a PR Without Warning

Location: Quick Start Step 2
Issue: The command gh aw add githubnext/agentics/daily-team-status --create-pull-request creates a pull request, but beginners might not understand what happens next.

Quote: "This creates a pull request that adds .github/workflows/daily-team-status.md and the compiled .lock.yml. Review and merge the PR into your repo."

Why it's confusing:

• Beginners might not expect a PR to be created
• No guidance on WHAT to review in the PR
• No mention of what happens if you don't merge it

Recommendation:
Expand the explanation:

```bash
gh aw add githubnext/agentics/daily-team-status --create-pull-request

What happens next:

1. A pull request is automatically created in your repository
2. The PR adds two files:
   • .github/workflows/daily-team-status.md (the workflow you'll edit)
   • .github/workflows/daily-team-status.lock.yml (the compiled version)
3. Before merging, review the PR to see what the workflow will do
4. Merge the PR to activate the workflow in your repository

Why a PR? This lets you review what's being added before it runs automatically.
It's like reviewing code changes before deploying to production.

Alternative: Use gh aw add githubnext/agentics/daily-team-status (without the flag)
to commit directly without a PR.

### 5. Examples Use Unexplained GitHub Features

**Location:** Throughout examples, especially in workflow configuration  
**Issue:** Example workflows use GitHub features (triggers, permissions, cron syntax) without explanation for beginners who may not be familiar with GitHub Actions.

**Examples:**
```yaml
on:
  schedule:
    - cron: "0 9 * * 1-5"  # What does this mean?

Recommendation:
Add comments to ALL example code:

on:
  schedule:
    # Run at 9 AM UTC, Monday-Friday
    # Format: minute hour day month weekday
    - cron: "0 9 * * 1-5"
  workflow_dispatch:  # Also allow manual triggering

---

🟢 What Worked Well

1. Clear Navigation Structure

The sidebar organization makes it easy to find information. The progression from Setup → Introduction → Reference → Examples is logical.

2. The "How Agentic Workflows Work" Section

The three-step diagram in Quick Start is excellent:

1. You write    →  2. Compile  →  3. GitHub Actions runs

This immediately clarifies the workflow lifecycle for beginners.

3. Verify Your Setup Section

The gh aw status verification step with expected output is VERY helpful. Showing what success looks like reduces anxiety.

4. Comprehensive CLI Commands Reference

The CLI docs are thorough, well-organized by use case, and include examples. The "Most Common Commands" section at the top is perfect for beginners.

5. Glossary with Contextual Links

Having a comprehensive glossary with linked terms throughout the docs is excellent. The glossary explanations are clear and beginner-friendly.

6. Warning Boxes

The prominent warnings about research prototype status and security considerations set appropriate expectations.

7. Troubleshooting Tips

The inline troubleshooting tips in Quick Start (e.g., "If the workflow isn't listed, run...") are helpful and well-placed.

---

📊 Recommendations

Priority 1: Quick Wins (Immediate Impact)

1. Add a "Beginner's Checklist" at the top of Quick Start

   :::note[Prerequisites Checklist]
   Before starting, make sure you:
   - [ ] Understand what GitHub Actions are ([Learn here](https://docs.github.com/en/actions))
   - [ ] Have GitHub CLI installed and authenticated
   - [ ] Have an active GitHub Copilot subscription
   - [ ] Are comfortable using the terminal/command line
   
   **New to some of these?** That's okay! Each section below explains what you need to know.
   :::

2. Create a Visual Quick Start Video or GIF
   Show the entire process in 2-3 minutes: install → add workflow → configure token → run workflow

3. Add "Why am I doing this?" callouts
   For every step that seems arbitrary (like selecting "Public repositories" for token), explain WHY

4. Simplify the token setup section
   Break it into: Check Prerequisites → Create Token → Add to Repo → Verify, with clear success indicators at each stage

5. Add inline comments to ALL code examples
   Never show YAML/configuration without explaining what each line does

Priority 2: Medium-term Improvements

6. Create a "Concepts Overview" page
   A single page that explains frontmatter, compilation, safe outputs, and agents BEFORE the Quick Start. Let beginners read this first to build mental models.

7. Add troubleshooting for common failure points

   • "I can't find the Copilot Requests permission" (with screenshots)
   • "The workflow didn't create an issue" (with debug steps)
   • "Compilation failed with YAML error" (with examples)

8. Create a "Common Gotchas" section

   • Don't edit .lock.yml files directly
   • Remember to compile after editing
   • Token expiration handling
   • Repository permissions vs account permissions

9. Add a "Next Steps" learning path
   After Quick Start succeeds, provide a clear path:

   • Customize your first workflow
   • Add a second workflow type
   • Understand safe outputs and permissions
   • Explore the Agentics collection

10. Improve the Getting Started CTA on homepage
    The current "Getting Started" button is good, but could include a time estimate: "Get Started (15 minutes)"

Priority 3: Long-term Documentation Enhancements

11. Create interactive tutorials
    Use a tool like interactive code tutorials where users can practice commands in a safe environment

12. Add more real-world examples with explanations
    Show complete workflows with step-by-step breakdowns of how the AI interprets instructions

13. Create a "Migration Guide for GitHub Actions Users"
    Many users will come from GitHub Actions. Show them how to translate their existing workflows to agentic format

14. Build a troubleshooting decision tree
    "My workflow isn't working" → Answer questions → Get to root cause → See solution

15. Add video walkthroughs for complex topics
    Especially for token setup, safe outputs, and MCP configuration

---

🎯 Success Metrics to Track

To measure documentation improvement, track:

1. Drop-off rate at each Quick Start step

   • How many users complete Step 1 but not Step 2?
   • Highest drop-off is likely at token setup

2. Most viewed troubleshooting pages

   • Indicates common pain points

3. Time to first successful workflow run

   • Goal: Under 20 minutes for beginners

4. Community feedback on documentation

   • GitHub issues tagged documentation
   • Discussion questions about getting started

5. Search queries within docs

   • What are users looking for but not finding?

---

🔍 Testing Methodology

Approach: Reviewed documentation from the perspective of a complete beginner with:

• No prior knowledge of GitHub Agentic Workflows
• Basic familiarity with Git/GitHub (but not GitHub Actions)
• No experience with AI/ML concepts
• General software development knowledge

Pages reviewed: Home, Quick Start, CLI Commands, Creating Workflows, About, Glossary, and sample workflow examples

Focus areas:

• Clarity of explanations for newcomers
• Assumptions about prior knowledge
• Ease of following step-by-step instructions
• Quality and placement of examples
• Navigation and information architecture

Limitations:

• Could not test actual workflow execution (testing doc reading only)
• Could not capture screenshots (Playwright installation blocked by network restrictions)
• Did not review every single documentation page (focused on critical getting-started path)

---

📝 Conclusion

The GitHub Agentic Workflows documentation is comprehensive and well-organized, but the onboarding experience for complete beginners is challenging due to:

1. High density of new concepts (agentic, frontmatter, compilation, safe outputs, MCP)
2. Assumed familiarity with GitHub Actions, CLI tools, and AI terminology
3. Complex token setup process that's a major friction point
4. Lack of inline explanations in code examples

The good news: Most of these issues can be addressed with relatively small changes—adding callouts, inline comments, visual diagrams, and better explanations of "why" alongside "how."

Biggest opportunity: Simplifying the token setup flow and adding a "Concepts Overview" page before Quick Start would dramatically improve the beginner experience.

The documentation team has done excellent work creating comprehensive reference material. With some beginner-focused enhancements, this could become a model for onboarding users to complex technical tools.

---

Next Steps:

1. Review and prioritize recommendations
2. Assign Priority 1 items to documentation maintainers
3. Consider user testing with actual beginners to validate improvements
4. Track metrics to measure impact of changes

AI generated by Documentation Noob Tester