feat(cli): skillforge audit <dir> — fleet security + quality scan

[adityachilka1]

What

skillforge audit [dir] — fleet security + quality scan across every skill directory at the top level of --from <dir> (default ~/.claude/skills/). One report, one exit code, drops into CI.

Rules

Six built-in rules, each behind a tiny guard:

Rule	Severity	Catches
security/embedded-binary	error	any non-text file > 1 MB — refuse to ship binaries
security/shell-shebang	warning	#!/bin/sh, #!/bin/bash, #!/usr/bin/env zsh, etc. at top of any non-SKILL.md file
security/exec-bit	warning	any file with the Unix exec bit set
quality/vague-description	warning	frontmatter description < 40 chars or still says TODO
quality/missing-examples	info	SKILL.md has no ## Examples section
quality/todo-marker	info	SKILL.md body still contains TODO / FIXME

CLI

skillforge audit                              # default ~/.claude/skills/
skillforge audit /path/to/skills              # bare positional
skillforge audit --from /path/to/skills       # named form
skillforge audit --severity error             # CI gate
skillforge audit --json | jq '.summary'       # machine-readable

Exit 0 if zero errors, 1 if any error. Warnings + info don't affect the exit code.

TDD-first (§17 of the rulebook)

Red:   FAIL  src/audit.test.ts [ src/audit.test.ts ]
       Error: Failed to load url ./audit.js (resolved id: ./audit.js)
              in /private/tmp/sf-audit/src/audit.test.ts.
              Does the file exist?
       Test Files  1 failed (1)      Tests  no tests

Green: ✓ src/audit.test.ts (11 tests) 13ms
       Test Files  1 passed (1)      Tests  11 passed (11)

11 new tests covering: empty dir, clean skill, each rule individually, severityFilter, multi-skill aggregation, non-directory error, JSON shape.

Test plan

Check	Status
pnpm install	✅ no lockfile drift
npx tsc --noEmit	✅ clean
biome check src	✅ clean (3 auto-fixes by --fix --unsafe)
vitest run	✅ 192/192 pass (181 baseline + 11 new)
tsup (ESM + DTS build)	✅ clean

Composition discipline

No imports from validate.ts / lint.ts / pack.ts / install.ts / cat.ts / inspect.ts internals. Composition over refactor — the rule walker is independent and adding a new rule is a small additive change.

Declaration of AI-Tools / LLMs usage

• Claude (Sonnet) for design, TDD-first implementation, tests, commit message, and this PR body. Co-authored with @Aditya-Chilka (alt account) — see commit trailer. Reviewed before push.

Summary by CodeRabbit

• New Features
  • Added skillforge audit command to scan skill directories for security and quality issues
  • Supports --severity filtering and --json output format
  • Includes six built-in audit rules covering security and quality concerns
  • Returns appropriate exit codes for CI/CD pipeline integration

[coderabbitai]

Warning

Review limit reached

@adityachilka1, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 56 minutes and 15 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d57904ef-bca9-43ae-9d77-70ca69d10f6c

📥 Commits

Reviewing files that changed from the base of the PR and between ec2efac and 39e4d3f.

📒 Files selected for processing (4)

• CHANGELOG.md
• src/audit.test.ts
• src/audit.ts
• src/cli.ts

📝 Walkthrough

Walkthrough

This PR introduces a fleet security and quality auditing system for skillforge. It adds an audit command that scans skill directories, detects security issues (embedded binaries, executable bits, shell shebangs) and quality gaps (missing descriptions, missing examples, unresolved TODOs), supports severity filtering and JSON output, and reports findings with appropriate exit codes.

Changes

Audit System Implementation

Layer / File(s)	Summary
Audit data model and core scanning implementation src/audit.ts	Defines AuditSeverity, AuditFinding, AuditOptions, and AuditReport types. Implements auditSkills() orchestration that iterates skill directories, reads skill names from SKILL.md frontmatter, pre-audits content for quality issues (description length, Examples heading, TODO markers), and recursively walks files to detect security issues (binaries >1MB, executable bits, shell shebangs). Filters findings by severity and computes per-severity counts.
Audit test suite src/audit.test.ts	Comprehensive tests covering empty directories, clean skills, security rule detection (embedded binary, shell shebang, exec-bit), quality rule detection (TODO marker, missing examples), severity filtering, multi-skill aggregation, input validation (non-directory rejection), and output contract conformance. Uses temporary directories and a writeSkillMd helper to create test fixtures.
CLI audit command and output formatting src/cli.ts	New audit [dir] command with --from, --severity, and --json options. Executes auditSkills() and formats output as either JSON or human-readable report grouped by skill name with colored severity indicators. Exit code 0 for no errors, 1 for error findings, 2 for command failures.
Changelog documentation CHANGELOG.md	Documents the new skillforge audit [dir] command, its six audit rules, filter/output options, and exit-code behavior.

Sequence Diagram

sequenceDiagram
  participant User
  participant CLI
  participant auditSkills as auditSkills Function
  participant FileSystem
  User->>CLI: skillforge audit [--from dir] [--severity level] [--json]
  CLI->>auditSkills: call with fromDir, severityFilter
  auditSkills->>FileSystem: validate directory and list skills
  loop for each skill
    auditSkills->>FileSystem: read SKILL.md, stat files, walk recursively
    FileSystem-->>auditSkills: file metadata and content
    auditSkills->>auditSkills: emit findings (security/quality rules)
  end
  auditSkills->>auditSkills: filter by severity, compute counts
  auditSkills-->>CLI: return AuditReport
  CLI->>CLI: format as JSON or human-readable
  CLI-->>User: print report and exit (0 or 1 or 2)

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 Hop through your skills with care,
Finding binaries and shebangs rare,
TODOs and examples gone astray—
The audit command saves the day!
Security and quality, side by side,
Now you code with skillforge's guide. ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 14.29% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately and concisely summarizes the main change: adding a new 'skillforge audit' CLI command for fleet security and quality scanning. It clearly conveys the feature addition and its primary purpose.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/audit

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}