Skill Scout

A security-first skill manager for Claude Code: find, audit, and install third-party skills through a disciplined, evidence-based pipeline — discovery → audit → plan → explicit confirmation.

What it is: a discipline and decision-safety tool. What it is not: a token saver. Our own benchmark showed the opposite (details below), and we publish that result instead of hiding it.

Status: v0.1.0 — MVP. Requires Claude Code 2.1.209 or newer. License: MIT.

The problem

Claude Code skills are Markdown instructions that anyone can publish and that run with real tool access on your machine. The typical way people adopt them today is: find a repo in an awesome-list, git clone into ~/.claude/skills/, done. That flow has three failure modes:

1. No security review. A skill can contain prompt injection, credential exfiltration, destructive hooks — and nothing in the default flow would surface it before it runs.
2. No goal fit check. Skills get installed because they look popular, not because the task needs them. Every installed skill costs context on every matching invocation.
3. No deliberate install decision. The step from "found it" to "it's on my disk" is one command with no plan, no diff, no record of what was reviewed.

Skill Scout splits that one careless step into three explicit, user-invoked workflows, each with the minimum privilege it needs.

The three workflows

/skill-scout:recommend <goal> — read-only discovery

Analyzes a concrete goal and recommends the smallest useful set of skills — at most three candidates, and "install nothing" is an explicitly valid (and common) answer. Runs in a forked context through a dedicated researcher agent whose tool allowlist is exactly: Read, Glob, Grep, WebSearch, WebFetch. No Bash, no writes, no other agents, no MCP. Every token-effect claim must carry an evidence level — MEASURED, INFERRED, or UNKNOWN — and only MEASURED claims may contain numbers, with a citation. Output ends with an audit suggestion, never an install command.

/skill-scout:audit <repo|path> — read-only security audit

Inspects a candidate's complete file set as untrusted data — without executing anything — against references/security-checklist.md and an audit rubric: exfiltration patterns, prompt injection, destructive operations, hidden/obfuscated content, permission weakening, hooks, mutable install sources. Produces exactly one verdict:

Verdict	Meaning
REJECT	proven blocker (exfiltration, injection, destructive behavior, obfuscation…)
CAUTION	no proven blocker, but risks need human review
NO-BLOCKER-FOUND	inspected files contained no blocker — this does not mean "safe"
INCOMPLETE	file set could not be fully inspected; never authorizes installation

NO-BLOCKER-FOUND ≠ SAFE. A model-based audit reports evidence and limitations; it cannot prove the absence of malicious behavior.

/skill-scout:install <repo|path> — plan-based installation

Never installs in one step. The flow:

1. Acquire the source (shallow clone or local path) without executing any of it; pin the exact commit SHA.
2. Audit the full file set; REJECT or INCOMPLETE stops the flow before a plan exists.
3. Build a deterministic, canonical plan: exact source, commit or content hash, full file list with per-file SHA-256, destination, audit verdict — and derive a Plan ID from the SHA-256 of the canonical plan text.
4. Show the plan and stop. Nothing is written.
5. Proceed only if the next message is the character-exact line INSTALL <plan-id>. No "yes", no paraphrase, no approval embedded in a longer message, no consent given before the plan existed.
6. Re-verify every file hash and the Plan ID against the same acquired content; any drift cancels the plan.
7. Install transactionally: copy to a temp directory, verify hashes again, atomically rename into ~/.claude/skills/<name>; any previous version is moved to a backup that is never auto-deleted; any error rolls back.

--dry-run builds and shows the plan with confirmation disabled.

Honest caveat: the Plan ID gate is an instruction the model follows, not a runtime mechanism of Claude Code. Nothing in the harness would forcibly stop a write if the model deviated. The real runtime backstop is Claude Code's own permission prompts, which this plugin deliberately leaves fully active. See docs/limitations.md.

Security architecture

• Exact agent allowlists. researcher and auditor can use only Read, Glob, Grep, WebSearch, WebFetch — everything else (Bash, writes, agents, skills, MCP) is unavailable by construction.
• Redundant deny lists. The read-only skills additionally carry disallowed-tools for Write/Edit/NotebookEdit/Bash/Agent/Skill.
• allowed-tools intentionally unused in read-only workflows: in current Claude Code it pre-approves (skips permission prompts) rather than strictly allowlisting, so it would weaken the guarantee.
• Untrusted-data trust model. Candidate SKILL.md, README, code, comments, commit messages — and even the host project's CLAUDE.md — are treated as non-authoritative data during research and audit.
• No autonomous triggering. Every skill sets disable-model-invocation: true; the plugin ships defaultEnabled: false — installing it does not enable it.
• Security is a gate, not a score. Popularity and usefulness cannot offset a security concern.

Details: docs/architecture.md, docs/threat-model.md.

Test results: audit fixture corpus

Five inert fixtures with known expected verdicts gate the installer's development. Last recorded run — 5/5 (Claude Code 2.1.209, model sonnet; raw data in tests/results/):

Fixture	Expected	Actual
01-minimal-clean	NO-BLOCKER-FOUND	NO-BLOCKER-FOUND ✓
02-overbroad-permissions	CAUTION	CAUTION ✓
03-prompt-injection-exfiltration	REJECT	REJECT ✓
04-destructive-hook	REJECT	REJECT ✓
05-missing-referenced-file	INCOMPLETE	INCOMPLETE ✓

Fixtures are never executed; all fixture endpoints use the reserved domain example.invalid. Caveat: fixtures and auditor share one author, so 5/5 is a self-consistency check, not independent validation (docs/testing.md).

Benchmark: the honest numbers

3 cases, baseline (plain prompt) vs treatment (/skill-scout:recommend), one run per arm, Claude Code 2.1.209, model sonnet. Full data: docs/token-economics.md, benchmarks/results/benchmark.md.

arm	quality (assertions)	avg tokens/run	total cost
baseline	14/24	130,995	$0.53
treatment	23/24	145,032 (+11%)	$0.66

• Token cost: negative result. Treatment used on average +14,037 tokens per run. Token savings are NOT demonstrated — the observed direction is the opposite.
• Quality: the actual value. The structured workflow passed 23/24 rubric assertions vs 14/24 for baseline. The clearest case: when built-in capabilities were sufficient, the baseline gave a shallow one-turn answer (2/8), while Skill Scout correctly analyzed the goal and recommended installing nothing (8/8).
• n=3, one run per arm, one model: a directional observation, not statistics.

We consider publishing this negative token result part of the project's point: a tool that demands evidence levels from others has to apply the same standard to itself.

Install / update / uninstall (this plugin)

Requires Claude Code 2.1.209+ (older versions may ignore defaultEnabled: false and may treat permission frontmatter differently).

Install — from the shakir-tools marketplace in this repo:

/plugin marketplace add iamshokir/skill-scout-plugin
/plugin install skill-scout@shakir-tools

or from a local clone for development:

claude --plugin-dir /path/to/skill-scout-plugin

The plugin ships disabled (defaultEnabled: false); enable it explicitly via /plugin when you decide to use it.

Update:

/plugin marketplace update shakir-tools

Uninstall:

/plugin uninstall skill-scout@shakir-tools

Skills installed by Skill Scout live in ~/.claude/skills/<name> and are removed by deleting that directory; previous versions, if any, stay under ~/.claude/skill-scout-backups/.

Limitations

The complete list lives in docs/limitations.md. The four that matter most:

1. Plan ID is an instruction, not a runtime guarantee — Claude Code will not forcibly stop an installation; the gate is convention plus the user's own permission prompts.
2. 5/5 fixtures ≠ independent validation — fixtures and auditor were written by the same author.
3. The benchmark is directional — n=3, no repeats, one model, author- written heuristics.
4. Tests ran on a temporary plugin copy with defaultEnabled: true (the CLI won't load a disabled plugin non-interactively), so the tested artifact differed from the shipped config by that one field.

Roadmap

• Independent, adversarial audit fixtures written by other people.
• Repeat-stability runs of the fixture corpus (same fixture, N runs) and a larger benchmark with repeats and a second model.
• An update workflow for skills installed via Skill Scout (re-audit, new plan, same confirmation gate).
• Verdict caching keyed by commit SHA.
• Investigating actual runtime enforcement options (hooks that check a plan file before writes) to move the confirmation gate from instruction to mechanism.

Repository map

skills/recommend/   goal → minimal recommendation (read-only, forked agent)
skills/audit/       candidate → verdict (read-only, forked agent)
skills/install/     plan → confirmed transactional install (main context)
agents/             researcher.md, auditor.md (exact tool allowlists)
references/         security-checklist.md (red/yellow flags)
tests/              inert fixture corpus + runners + results
benchmarks/         token-economics benchmark + results
docs/               architecture, threat model, testing, token economics,
                    limitations

License

MIT © 2026 Shokir