📚 Documentation Noob Test Report - December 25, 2025

[github-actions[bot]]

Executive Summary

As a complete beginner attempting to get started with GitHub Agentic Workflows, I navigated through the documentation site and identified several critical issues that would block or significantly slow down new users trying to adopt this tool.

Pages Visited:

• Home page: (redacted)
• Introduction > Overview: (redacted)
• Quick Start: (redacted)
• CLI Commands: (redacted)
• Creating Workflows: (redacted)
• Example - IssueOps: (redacted)

Overall Impression:
The documentation has solid technical content, but the getting started experience is confusing for beginners. The homepage doesn't clearly explain what the tool does, navigation is overwhelming, and critical concepts like "compilation" and "lock files" are introduced without sufficient explanation.

---

🔴 Critical Issues (Block Getting Started)

1. Homepage Provides No Clear Introduction

Issue: The homepage shows only "RESEARCH PROTOTYPE" without any explanation of what GitHub Agentic Workflows actually does.

Impact: A new user landing on the homepage cannot immediately understand:

• What this tool is
• What problems it solves
• Whether it's relevant to their needs
• How to get started

Expected: The homepage should have:

• A clear one-sentence description (e.g., "Write AI-powered GitHub automation in natural language")
• A prominent "Quick Start" or "Get Started" button
• A brief example showing markdown → automation
• Key benefits (why use this vs. regular GitHub Actions)

Recommendation: Add a hero section with:

# GitHub Agentic Workflows
Write AI-powered GitHub automation in natural language

Turn simple markdown descriptions into intelligent GitHub Actions powered by AI agents.
No complex YAML. No intricate scripting. Just describe what you want to happen.

[Quick Start →] [View Examples →]

2. Missing Prerequisites Section in Quick Start

Issue: The Quick Start guide jumps straight into installation without listing what you need beforehand.

Impact: New users don't know if they need:

• A GitHub account (obviously, but should be stated)
• GitHub CLI already installed
• Admin access to a repository
• A personal access token
• Copilot license or API keys

Expected: A clear "Before You Begin" section listing:

• Required: GitHub account, GitHub CLI installed, repository admin access
• Required for Copilot engine: GitHub Copilot license
• Optional: Other AI engines (Claude, OpenAI) with their requirements

Recommendation: Add prerequisites section at the top of Quick Start:

## Before You Begin

You'll need:
- ✅ A GitHub account
- ✅ [GitHub CLI](https://cli.github.com/) installed
- ✅ Admin access to a GitHub repository
- ✅ One of the following:
  - GitHub Copilot license (recommended)
  - Anthropic API key for Claude
  - OpenAI API key for Codex

3. "Compilation" Concept Not Explained

Issue: The docs frequently mention "compiling" workflows and ".lock.yml" files without explaining what compilation means in this context.

Impact: New users see commands like gh aw compile and wonder:

• Why do I need to compile markdown?
• What is a .lock.yml file?
• Do I edit the .lock.yml or the .md file?
• When do I need to recompile?

Found in: Quick Start, CLI Commands, Creating Workflows guides all use "compile" without definition.

Expected: Early explanation that:

• You write workflows in .md (markdown) files
• gh aw compile converts them to .lock.yml (GitHub Actions YAML)
• You edit the .md file, never the .lock.yml
• Think of it like compiling code: .md is source, .lock.yml is compiled output

Recommendation: Add a "Key Concepts" callout in Quick Start:

:::note[Key Concept: Compilation]
You write workflows in **markdown files** (.md). The `gh aw compile` command 
converts them into **GitHub Actions YAML** (.lock.yml).

- ✏️ Edit: `.md` files (your source)
- 🔒 Don't edit: `.lock.yml` files (generated output)
- 🔄 Recompile: After changing any `.md` file

Think of it like compiling code—markdown is your source, YAML is the compiled output.
:::

4. Overwhelming Sidebar Navigation

Issue: The left sidebar has 50+ links organized into categories like "Design Patterns," "Examples," "Reference," etc. For a beginner, this is overwhelming.

Impact: Users don't know where to start or what's important vs. optional.

Expected:

• Clear visual hierarchy showing "Start here" content
• Collapsed sections by default
• A "Getting Started" or "Basics" section at the top

Recommendation: Reorganize sidebar:

📚 Getting Started (expanded by default)
  - Overview
  - Quick Start
  - Your First Workflow
  - Key Concepts

📖 Guides (collapsed)
  - Creating Workflows
  - Security Best Practices
  - ...

💡 Examples (collapsed)
  - ...

📘 Reference (collapsed)
  - ...

5. "Safe Outputs" and "Frontmatter" Without Context

Issue: Terms like "safe-outputs" and "frontmatter" appear in examples without explanation.

Impact: Beginners reading the Quick Start example see:

safe-outputs:
  add-comment:

And wonder:

• What is a safe output?
• Why can't I just tell the AI to comment directly?
• What are the other safe output types?
• When do I need this?

Expected: Either inline tooltips, hover definitions, or a callout explaining these terms when first encountered.

Recommendation: Add a glossary callout in first occurrence:

:::tip[New Term: Safe Outputs]
Safe outputs are how AI agents interact with GitHub without direct write access.
Instead of giving the AI permission to modify your repository directly, you declare
what actions it can take (create issues, add comments, etc.).

Common safe outputs:
- `create-issue`: Create a new issue
- `add-comment`: Comment on an issue/PR  
- `create-pull-request`: Create a PR

[Learn more about safe outputs →](../reference/safe-outputs/)
:::

---

🟡 Confusing Areas (Slow Down Learning)

6. CLI Commands Page Is Reference, Not Tutorial

Issue: The CLI Commands page lists every command but doesn't teach workflow.

Impact: Users see 20+ commands and don't know:

• Which commands do I actually need?
• What's the typical workflow?
• Which commands are for beginners vs. advanced users?

Current State: This section DOES exist, but it's buried after "Installation" and "Global Options". Should be at the top.

Recommendation: Reorder the CLI page:

1. Common Workflows (show typical usage patterns)
2. Quick Reference (table of common commands)
3. Installation
4. Detailed Commands Reference

7. Examples Don't Show Complete Context

Issue: Example workflows show code snippets without showing the full file structure or where things go.

Impact: A beginner looks at an example and doesn't know:

• Do I create this as .github/workflows/example.md?
• Do I need to compile it first?
• What happens after I commit it?

Recommendation: Add "How to Use This Example" section to all example pages showing file paths, compilation steps, and testing instructions.

8. Terminology Inconsistency

Issue: Docs use "AI agent," "agent," "engine," and "Copilot" interchangeably.

Impact: Beginners wonder if these are different things.

Recommendation: Add to glossary and use consistent terminology:

• "AI engine" for the service (Copilot, Claude)
• "AI agent" or "agent" for the executor
• Link to glossary on first use

9. Missing "What Can Go Wrong" Section

Issue: Docs focus on the happy path without showing common errors.

Impact: When something fails, users don't know if it's normal or what to do.

Recommendation: Add troubleshooting callouts in Quick Start and link to detailed troubleshooting guide.

---

🟢 What Worked Well

10. Introduction > Overview Is Excellent

Positive: The Overview page is the BEST introduction to the tool.

Why it works:

• Clear explanation of what agentic workflows are
• Comparison to traditional automation
• Simple example with explanation
• Explains key security concepts

Recommendation: Make this content more prominent! Consider:

• Moving some of this to the homepage
• Linking to it prominently from Quick Start
• Using it as the "What is this?" introduction

11. CLI Commands Has Good "Common Workflows" Section

Positive: The CLI page includes a "Common Workflows for Beginners" section showing typical command combinations.

Why it works:

• Shows realistic usage patterns
• Groups related commands
• Explains what each command does

Issue: It's buried too deep in the page. Should be at the very top.

12. Glossary Exists

Positive: There's a comprehensive glossary at /reference/glossary/.

Why it works:

• Defines all key terms
• Good reference for confused users

Issue: Not linked from other pages when terms are first introduced.

Recommendation: Add inline links to glossary terms throughout docs.

---

🎯 Prioritized Recommendations

Quick Wins (Fix Today)

1. Add homepage hero section explaining what the tool is
2. Add prerequisites to Quick Start (GitHub CLI, tokens, etc.)
3. Reorder CLI page to show common workflows first
4. Add "Key Concept: Compilation" callout in Quick Start

Short Term (Fix This Week)

5. Add inline glossary links for first use of technical terms
6. Add "How to Use" sections to all examples
7. Create "Your First Workflow" tutorial page (step-by-step)
8. Add common errors to Quick Start

Long Term (Nice to Have)

9. Reorganize sidebar with clearer hierarchy
10. Add interactive examples (copy-paste ready)
11. Create video walkthrough (5 minutes)
12. Add "Why use this?" comparison to homepage

---

🏆 Success Metrics

If these issues are fixed, a new user should be able to:

✅ Understand what the tool does in <30 seconds (homepage)
✅ Know what they need before starting (<1 minute)
✅ Create and run their first workflow (<10 minutes)
✅ Troubleshoot basic errors (without asking for help)

---

Test Conducted By: GitHub Copilot Agent (Documentation Noob Testing)
Date: December 25, 2025
Environment: Local preview server (npm run preview)
Method: Manual navigation and content analysis

AI generated by Documentation Noob Tester

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 3 days ago.