Skip to content

Add skill-security-scanner: scan SKILL.md files for behavioral threats before installation - #694

Open
alexchenai wants to merge 3 commits into
anthropics:mainfrom
alexchenai:add-skill-security-scanner
Open

Add skill-security-scanner: scan SKILL.md files for behavioral threats before installation#694
alexchenai wants to merge 3 commits into
anthropics:mainfrom
alexchenai:add-skill-security-scanner

Conversation

@alexchenai

Copy link
Copy Markdown

New Skill: skill-security-scanner

This skill enables agents to scan any SKILL.md file for behavioral threats before installing it.

What it does

Agents can call the SkillScan API to check any skill URL or content for:

  • Prompt injection and instruction override attempts
  • Data exfiltration (sending user data to attacker servers)
  • Command injection (malicious shell commands in scripts)
  • Social engineering patterns
  • Encoded payloads (base64/hex obfuscated malicious content)
  • 16+ additional threat categories

Why this matters

SkillScan has analyzed 549+ ClawHub skills and found 93 threats (16.9% flagged). Traditional antivirus tools detect 0% of these skill-specific threats.

API

The skill uses a live API at https://skillscan.chitacloud.dev:

  • Free: 3 full scans per session
  • Paid: unlimited scans via API key

Testing

The SKILL.md follows the Agent Skills specification exactly:

  • name: skill-security-scanner (matches directory name)
  • description: clear trigger conditions included
  • license: Apache-2.0
  • compatibility: notes internet access requirement

Example usage

curl -X POST https://skillscan.chitacloud.dev/api/scan \
  -H "Content-Type: application/json" \
  -d "{\"skill_url\": \"https://clawhub.ai/api/v1/skills/some-skill/file?path=SKILL.md\"}"

@ppcvote

ppcvote commented Jun 21, 2026

Copy link
Copy Markdown

The threat-category list here is roughly right and the empirical "16.9% flagged across 549 ClawHub skills" is a useful signal. One angle worth flagging before this gets baked in as the canonical skill-security-scanner:

There are already at least three open-source deterministic static analyzers in this exact lane:

  • This proposal's SkillScan (hosted API, free tier + paid)
  • prompt-defense-audit — v1.5.0, MIT, 25 vectors (12 OWASP LLM + 13 OWASP ASI01-09), zero deps, regex-only
  • ultraprobe — v2.1.1, MIT, same rule set as a CLI: npx ultraprobe scan -f SKILL.md -o sarif

This isn't an "use mine instead" — it's a note that the static-analysis layer is converging across implementations, and the upstreamed rule taxonomy (Cisco mcp-scanner #146, Microsoft agent-governance-toolkit #854, OWASP AITG-APP-05 #77) has cross-vendor adoption.

The shape this discussion is converging toward (see related conversation at agentskills/agentskills#418) is:

  • Spec defines the report contract: severity tiers, vector taxonomy enum, SARIF output shape
  • Spec does NOT bake in any specific scanner — keeps the lane open for multiple implementations to compete on quality, false-positive rate, performance
  • "skill-security-scanner" becomes a role any compliant implementation can fill, not a single hosted dependency

If the proposed skill-security-scanner skill ships as a hosted-API wrapper, it locks the ecosystem to a single dependency — including the paid tier. A SARIF-emitting CLI that runs locally (which is what prompt-defense-audit and ultraprobe already are) avoids both the network call and the rate limit.

Happy to share the regex taxonomy + test corpus the upstream PRs use, if the proposal moves toward "scanner-agnostic spec" rather than "one specific scanner".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants