[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-06-03

[github-actions[bot]]

Summary

• Date of test: 2026-06-03
• Pages visited: Home (/gh-aw/), Quick Start (/gh-aw/setup/quick-start/), CLI Commands (/gh-aw/setup/cli/)
• Overall impression: The docs have a professional feel and good structure, but a brand-new user may struggle with several assumptions about prior knowledge — especially around GitHub CLI, PATs, and what "frontmatter" means.

---

🔴 Critical Issues Found

1. The tool name confusion: gh-aw vs gh aw vs "GitHub Agentic Workflows"

As a noob, the home page mentions "GitHub Agentic Workflows" but the install command is gh extension install github/gh-aw. The CLI command is gh aw. I wasn't sure if these were different things or the same. There's no clear statement like: "This tool is called gh aw — you run it as gh aw <command>" near the top of the Quick Start.

2. "Frontmatter" is never defined for beginners

The Quick Start uses the term frontmatter without any definition:

"If you changed the frontmatter (the configuration block between the --- markers...)"

For someone who has never used Jekyll, Hugo, or Astro, this term is completely opaque. A one-line tooltip or parenthetical explanation in the first occurrence would help enormously.

3. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN confusion

The Quick Start mentions needing a separate token called COPILOT_GITHUB_TOKEN — distinct from the default GITHUB_TOKEN. This is non-obvious and the note buried in a collapsible block. New users often don't realize GitHub Actions already provides a GITHUB_TOKEN; explaining why a second token is needed would reduce confusion.

4. No mention of required workflow scope for gh auth login

The prerequisites say to run gh auth login --scopes repo,workflow. A beginner won't know they need the workflow scope specifically, or that forgetting it causes a confusing permission error when the extension tries to commit .lock.yml files. This should be called out more prominently.

---

🟡 Confusing Areas

5. What does add-wizard actually do vs add?

The CLI page lists both gh aw add-wizard and gh aw add but the distinction ("with interactive guided setup" vs "non-interactive") isn't immediately clear. The Quick Start only uses add-wizard — a note explaining when to use each would help.

6. The .lock.yml concept dropped too early

Step 2 of the Quick Start adds a workflow and then immediately explains:

"The .lock.yml file is the compiled GitHub Actions workflow generated from the .md source file. Keep editing the .md file..."

This is important information but may overwhelm a beginner who just wants their first workflow running. It could be moved to a "How it works" callout or linked to as optional reading.

7. CLI Commands page is very long — no quick reference for beginners

The CLI page is dense (~47 KB). The "Most Common Commands" table at the top is helpful, but after that, the page becomes a deep reference. A dedicated "Getting Started" section at the top with the 3-4 commands most beginners need would improve discoverability.

8. Home page tagline is abstract

The tagline: "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions" is jargon-heavy for a complete beginner. "coding agents" may not be a familiar term. A concrete one-liner like "Automate your GitHub repo using plain English — no YAML required" would be more accessible.

---

🟢 What Worked Well

• ✅ Prerequisites are explicit — The Quick Start clearly lists GitHub Copilot/Claude/etc., GitHub CLI version requirements, and OS support. This is great.
• ✅ Step-by-step structure — The 4-step Quick Start (Install → Add → Wait → Customize) is logical and easy to follow.
• ✅ Inline notes for token setup — The collapsible notes for COPILOT_GITHUB_TOKEN and ANTHROPIC_API_KEY are thorough and include the exact commands to set secrets.
• ✅ "What's next?" section — Guides the user to their next learning steps after the Quick Start.
• ✅ add-wizard interactive approach — Using an interactive wizard reduces the number of manual steps a beginner needs to know.
• ✅ Security guardrails diagram — The Mermaid flow diagram on the home page explaining the security model is visually effective.
• ✅ Multiple engine support is prominent — Clearly listed on home page and Quick Start (Copilot, Claude, Codex, Gemini).

---

Recommendations

Quick wins

1. Add a "TL;DR" box at the very top of the Quick Start: "You'll install a CLI extension, add one workflow file, and watch your repo automate itself."
2. Define "frontmatter" inline at first use: "frontmatter (the YAML config block between --- lines at the top of the markdown file)"
3. Rename or explain the COPILOT_GITHUB_TOKEN in a prominent note early in the Quick Start — not just in a collapsible.
4. Add a one-sentence intro to the CLI Commands page before the table: "If you're new, you only need add-wizard, compile, and run to get started."

Longer-term improvements

1. Add a glossary page covering: frontmatter, lock file, safe outputs, engine, workflow, agentic, compile.
2. Add a "Common Errors" section to the Quick Start (e.g., missing workflow scope, wrong token type).
3. Create a "Beginner path" nav group that guides users through: Home → Quick Start → Creating Workflows → FAQ, separately from the reference docs.

---

Screenshots

📎 [home.png] — Home page overview

📎 [quick-start.png] — Quick Start guide

📎 [cli.png] — CLI Commands reference

---

References: §26865403559

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · sonnet46 333.3K · ◷

• expires on Jun 4, 2026, 5:34 AM UTC