feat(tools): enhance skill validator with tag quality, workflow, prereq, and safety checks

[KennyUMN]

What

Extends tools/validate-skill.py with four new quality checks requested in #86:

1. Tag quality — flags generic stop-words and filename-split tags that provide no agent routing value (e.g. analyzing, with, block, logs). These are words split from the skill filename rather than meaningful cybersecurity discovery terms — the exact bugs identified in the NLPM audit: 7 mechanical bugs found (tags, stub workflows) #49 NLPM audit.

2. Workflow completeness — verifies presence of required sections (workflow/instructions/steps) and recommends When to Use, Prerequisites, and Output sections. Also includes stub detection: flags skills that list code-based prerequisites (e.g. pip install boto3) but have no code blocks in the workflow body.

3. Prerequisite consistency — extracts library/tool names from the Prerequisites section and checks whether each listed library actually appears in the workflow body. Warns when a prerequisite is declared but never used.

4. Safety gates — identifies high-risk skills (red team, pentest, malware, credential access, phishing simulation, C2, exploit) by subdomain, tag, and name keywords. Flags those that lack authorization, scope, or legal-notice language.

Why

Issue #86 explicitly requests these checks. Issue #49 (NLPM audit) found 7 mechanical bugs across the skill catalog — word-split tags, stub workflows with prerequisites listed but no code — that a validator with these checks would have caught before merge. The existing validator only checked frontmatter field presence; it could not detect these quality issues.

Changes

• tools/validate-skill.py — extended with new check functions, new CLI flags (--check, --strict), and a frontmatter parser fix for nested YAML structures
• tools/README.md — updated documentation for new checks and CLI flags

Frontmatter parser fix

The parser had a bug where nested YAML structures (e.g. mitre_f3: with nested name: keys) would corrupt top-level fields — a nested name: 'Account Manipulation: Account Linking' would overwrite the skill's actual name: field. Fixed by only processing top-level (indent=0) lines as frontmatter fields.

Testing

Validated against 16 sample skills covering high-quality skills, stub skills from issue #49, high-risk skills with authorization language, and skills with nested YAML frontmatter:

Total: 17  Passed: 16  Failed: 1  Warnings: 15

Results match expectations:

• 5 stub warnings (code prereqs but no code blocks) — catches the NLPM audit: 7 mechanical bugs found (tags, stub workflows) #49 stub bugs
• 2 prerequisite consistency warnings (libraries listed but not referenced in workflow)
• 4 missing output section warnings
• 2 subdomain alias warnings (security-operations to soc-operations, red-team to red-teaming)
• 0 false positive name errors (parser fix working)
• High-risk skills with authorization language correctly NOT flagged

New CLI flags

# Run only specific checks
python tools/validate-skill.py --all --check tags,body,prereqs,safety

# Treat warnings as errors (useful for CI)
python tools/validate-skill.py --all --strict

Refs #86

[Copilot]

Pull request overview

Enhances tools/validate-skill.py from a frontmatter-only validator into a broader “skill quality” validator, adding checks for tag quality, workflow completeness, prerequisite usage consistency, and high-risk safety/authorization language, plus new CLI flags to select checks and enforce strictness.

Changes:

• Added new quality checks (tags, body, prereqs, safety) and CLI flags (--check, --strict) to tools/validate-skill.py.
• Modified the frontmatter parser to avoid nested-YAML key collisions.
• Updated tools/README.md to document the new checks and flags.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
tools/validate-skill.py	Adds new validation checks + CLI options; updates frontmatter parsing logic.
tools/README.md	Documents the new validator capabilities and CLI flags.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mukul975]

Thanks for extending the validator with the tag-quality, workflow, prereq, and safety checks (#86) — these are useful additions. Two things before merge:

1. Rebase on main. PR #87 (the nested-name misparse fix) has now been merged into tools/validate-skill.py — the same file this PR rewrites. Please rebase onto current main so your additive checks layer on top of the merged fix rather than reverting it. Right now they will conflict.
2. After rebase, a few review points:
   • Tag-quality stopword check: confirm the false-positive surface (legit tags that happen to contain a flagged word) is acceptable.
   • Python 3.9 compatibility: no walrus in comprehensions, no match/case, no X | None hints — the repo targets 3.9.
   • Make sure the new checks don't duplicate what the merged #87 validator already does.

Note: there are competing validator PRs (#91 proposes a separate skill_quality_validator.py). The repo should converge on one validator — extending the existing validate-skill.py here is the preferred path, so this PR is the one to carry forward once rebased.

[KennyUMN]

Rebased onto current main — all three conflict points addressed:

1. Rebase done. The branch now sits cleanly on top of current main (including the nested-name misparse fix from #87).

2. Conflict resolution details:

   • REQUIRED_FIELDS now includes version, author, and license added by #87.
   • The duplicate indent-guard that #87 added to parse_frontmatter was dropped from our version since the equivalent check already exists earlier in the same loop body — no functional change, single clear guard.
   • Our new validate_skill() and main() functions incorporate the .bak directory skip from #87's main().

3. Tag-quality false positives: The stopword list flags words like forensics, analysis, detection when they appear as standalone tags with no domain specificity. We're comfortable with this surface — a tag like forensics genuinely provides no routing value compared to digital-forensics or memory-forensics. The list is conservative enough that no real routing tags are flagged.

4. Python 3.9 compatibility: Confirmed — no walrus operators, no match/case, no X | None type union hints. The file uses stdlib only.

5. No duplication with #87: The merged validator does only frontmatter field presence + subdomain checks. Our additions (tag quality, workflow completeness, prereq consistency, safety gates) are entirely new check functions that run after the base frontmatter checks.