[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-06-02

[github-actions[bot]]

Summary

• Date of test: 2026-06-02
• Pages visited: Home (/gh-aw/), Quick Start (/gh-aw/setup/quick-start/), CLI Commands (/gh-aw/setup/cli/)
• Overall impression: The docs have a professional look and a clear quick-start path, but a newcomer without GitHub Actions or AI API experience will hit multiple knowledge gaps that aren't explained inline.

---

🔴 Critical Issues Found

1. "Frontmatter" used without definition

The Quick Start guide instructs users to edit the frontmatter and even shows a code block using it, but never explains what frontmatter is. A complete beginner reading the phrase "if you changed the frontmatter (the configuration block between the --- markers)" will be confused—the term is introduced only in parentheses at Step 4, not upfront.

Suggestion: Add a one-line tooltip/callout at the first use: "Frontmatter is the YAML configuration block at the top of the .md file, between the --- markers."

2. No explanation of what the sample workflow does before installing it

The guide tells users to run gh aw add-wizard githubnext/agentics/daily-repo-status without first clearly stating what daily-repo-status actually does in plain English. There is a link to view the raw source on GitHub, but that's a 150+ line markdown file—not beginner-friendly.

Suggestion: Add 2-3 sentences describing the workflow's output before the install command.

3. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN confusion

The prerequisite says "AI Account - GitHub Copilot" but the setup note reveals a separate fine-grained PAT (COPILOT_GITHUB_TOKEN) is needed—different from the GITHUB_TOKEN that GitHub Actions automatically provides. This distinction is buried in a collapsible NOTE box in Step 2.

For a newcomer, this will cause: "Wait, I thought GitHub already provides a token to Actions? Why do I need a second one?"

Suggestion: Add a callout box in the Prerequisites section: "Note: Copilot requires a separate personal access token (not the automatic GITHUB_TOKEN). [Why?]"

---

🟡 Confusing Areas

4. .lock.yml / lock file explanation is too technical too early

Step 2 ends with a multi-paragraph explanation of what .lock.yml is and warns users not to edit it directly. For a first-time user who just wants to see the tool work, this is noise. It should be in a collapsible <details> block or moved to a separate "How it works" reference.

5. CLI Commands page is overwhelming (47KB)

The /setup/cli/ page lists every single command with all flags, subcommands, and enterprise GHES configuration. There's a "Most Common Commands" table at the top which is helpful, but the page has no clear visual hierarchy separating beginner vs advanced commands. A newcomer who lands here will feel buried.

Suggestion: Add a sidebar filter, group commands by beginner/advanced, or split into multiple focused pages.

6. Navigation: "Peli's Agent Factory" link in top nav

The top navigation bar includes a link to "Peli's Agent Factory" (a blog post). To a new user, this looks like a core feature/product section. It's actually a blog post. The naming is informal/personal and doesn't clearly signal its nature.

Suggestion: Either label it as "Blog: Agent Factory" or move it to be a sub-item under "Blog".

7. add-wizard command format is opaque

gh aw add-wizard githubnext/agentics/daily-repo-status uses a 3-part slash format (owner/repo/workflow-name) that isn't immediately obvious. The explanation comes after the command.

Suggestion: Reverse the order—explain the format first, then show the command.

---

🟢 What Worked Well

• ✅ Home page is clear: The tagline "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions" clearly explains the value proposition at a glance.
• ✅ CTA buttons are prominent: "Quick Start with CLI" and "Creating Workflows" are immediately visible on the home page.
• ✅ 10-minute estimate in the Quick Start sets expectations well.
• ✅ Prerequisites list is comprehensive and links to external resources.
• ✅ Step-by-step numbered structure in Quick Start is easy to follow.
• ✅ Security warning at the top of the home page (early development / use with caution) is honest and well-placed.
• ✅ Most Common Commands table at the top of the CLI page is excellent for quick reference.
• ✅ Alternative install methods (curl script, pinned version) are documented clearly.

---

Recommendations

Quick wins 🚀

1. Define frontmatter inline at first use in Quick Start
2. Add a 2-3 sentence description of daily-repo-status before the install command
3. Add a prerequisite callout explaining that Copilot requires a second PAT
4. Move the .lock.yml explanation to a <details> block in Step 2

Longer-term improvements 🔧

5. Split the CLI Commands page into beginner/intermediate/advanced sections
6. Rename or relocate the "Peli's Agent Factory" top-nav link to clarify it's a blog post
7. Add terminal screenshots showing what the add-wizard interactive flow looks like

---

Screenshots

📎 [home.png] — Home page

📎 [quick-start.png] — Quick Start guide

📎 [cli.png] — CLI Commands page

---

References: §26799922159

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · sonnet46 2.7M · ◷

• expires on Jun 3, 2026, 5:22 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #36577.