Add web-search-plus skill (multi-provider web search + extraction, no MCP)

.claude/skills/add-web-search-plus/REMOVE.md

@@ -0,0 +1,68 @@
+# Remove Web Search Plus
+
+Idempotent — safe to run even if some steps were never applied. Reverses the
+copied container skill, the integration test, and the Dockerfile edit.
+
+## 1. Delete the copied skill files and test
+
+```bash
+rm -rf container/skills/web-search-plus
+rm -f src/wsp-skill.test.ts
+```
+
+## 2. Revert the Dockerfile edit
+
+In `container/Dockerfile`, delete the `python3 \` line this skill added to the
+system-deps `apt-get install` block (between `git \` and `tini \`). Skip if
+already gone. Leave every other package line untouched.
+
+## 3. Dependencies
+
+The engine is pure-stdlib Python and `wsp` is a shell script — there is no npm
+or bun package to uninstall. Step 2 removes the only dependency surface
+(the apt `python3` install).
+
+## 4. Per-group key files and cache
+
+Remove the key file and cache from each group that configured a provider key,
+unless another integration uses them:
+
+```bash
+rm -f groups/<folder>/.wsp.env
+rm -rf groups/<folder>/.wsp-cache
+```
+
+If a provider key was stored in the OneCLI vault solely for wsp, delete that
+secret in the OneCLI dashboard.
+
+For any group with an explicit skills list in container config (not `"all"`),
+remove `"web-search-plus"` from that list.
+
+## 5. Rebuild and restart
+
+Run from your NanoClaw project root:
+
+```bash
+pnpm run build && ./container/build.sh
+source setup/lib/install-slug.sh
+
+# macOS
+launchctl kickstart -k gui/$(id -u)/$(launchd_label)
+
+# Linux
+systemctl --user restart $(systemd_unit)
+```
+
+## Verification
+
+After removal, the skill's guards no longer apply (their files are gone).
+Confirm it is fully unwired:
+
+```bash
+ls container/skills/web-search-plus 2>/dev/null        # no such directory
+grep -n "python3" container/Dockerfile                 # no output
+ls src/wsp-skill.test.ts 2>/dev/null                   # no such file
+```
+
+New containers spawned after the restart no longer show `web-search-plus`
+under `/app/skills/`.

.claude/skills/add-web-search-plus/SKILL.md

@@ -0,0 +1,167 @@
+---
+name: add-web-search-plus
+description: Add Web Search Plus (wsp) to agent containers — multi-provider web search and URL extraction on self-managed provider keys instead of the metered WebSearch tool. 13 providers (Serper, Brave, Tavily, Exa, Linkup, Firecrawl, Perplexity, SearXNG, ...) with auto-routing, fallback chains, and caching. Pure-stdlib Python engine; adds python3 to the agent image.
+---
+
+# Web Search Plus (wsp)
+
+Gives every agent container a `wsp` CLI for web search and page extraction
+through the Web Search Plus engine: provider auto-routing, fallback chains,
+parallel extraction, response caching. Calls go out on self-managed provider
+keys — not the metered `WebSearch` tool — at a fraction of the cost.
+
+This is a utility skill: it ships its code in this folder and copies it into
+the project. The engine is pure-stdlib Python (12 `.py` files, no pip, no
+venv); the only runtime requirement is a `python3` interpreter, which this
+skill adds to the agent image.
+
+## Install
+
+### Pre-flight
+
+If all of the following are already present, skip to **Configure a provider key**:
+
+- `container/skills/web-search-plus/SKILL.md`
+- `container/skills/web-search-plus/bin/wsp` (executable)
+- `container/skills/web-search-plus/engine/search.py` (plus the other engine files)
+- `src/wsp-skill.test.ts`
+- a `python3 \` line in the system-deps `apt-get install` block of `container/Dockerfile`
+
+Missing pieces — continue below. All steps are idempotent; re-running is safe.
+
+### 1. Copy the container skill files
+
+Wholesale copies (owned entirely by this skill — user edits to these files
+won't survive a re-run, as designed):
+
+```bash
+mkdir -p container/skills/web-search-plus
+cp -R .claude/skills/add-web-search-plus/resources/web-search-plus/. container/skills/web-search-plus/
+chmod +x container/skills/web-search-plus/bin/wsp
+```
+
+This lands the agent-facing `SKILL.md`, the `bin/wsp` wrapper, and the
+`engine/` directory (12 `.py` files + `LICENSE`). `container/skills/` is
+mounted read-only into every agent container at `/app/skills/`, so agents see
+the tool at `/app/skills/web-search-plus/bin/wsp`.
+
+### 2. Add python3 to the container Dockerfile
+
+The base image is Node-only; the engine needs a Python interpreter. In
+`container/Dockerfile`, add one line to the system-deps `apt-get install
+-y --no-install-recommends` block, between `git \` and `tini \`:
+
+```dockerfile
+        python3 \
+```
+
+Idempotent: if the line is already in the block, skip this step.
+
+`python3` is deliberately **not** version-pinned. The `ARG <X>_VERSION` pin
+pattern in this Dockerfile is for pnpm-installed CLIs; the apt system-deps
+block installs unpinned distro packages by house style (chromium, git, curl —
+all unpinned), because Debian's archive drops superseded point releases and a
+pinned `python3=3.x.y-z` breaks rebuilds at random. Reproducibility for apt
+packages comes from the base image, and the engine targets the stdlib of any
+python3 ≥ 3.9, so the distro's stable version is the correct dependency.
+
+### 3. Copy the integration-point test
+
+```bash
+cp .claude/skills/add-web-search-plus/resources/wsp-skill.test.ts src/wsp-skill.test.ts
+```
+
+`wsp-skill.test.ts` is the guard for this skill's one functional integration
+point, the Dockerfile python3 dependency: a CLI binary installed via apt is
+not importable, so neither a behavior test nor the build leg can see it, and a
+structural test of the Dockerfile line is the correct guard. The test scopes
+its assertion to the apt-get install block (a `python3` mention elsewhere
+can't keep it green) and also asserts the copied wrapper, engine entry, and
+container SKILL.md are present and executable — red on a half-applied or
+half-removed skill.
+
+The skill reaches into no host or container TypeScript (no barrel edit, no
+`main()` call), so there is no code-wiring test to ship; the engine files are
+pure adds.
+
+### 4. Build and validate
+
+```bash
+pnpm run build                              # host typecheck (unaffected, must stay green)
+pnpm exec vitest run src/wsp-skill.test.ts  # Dockerfile + skill-file guards
+./container/build.sh                        # agent image — bakes python3 in
+```
+
+All must be clean before proceeding. To confirm the interpreter actually
+landed in the image:
+
+```bash
+docker run --rm --entrypoint python3 nanoclaw-agent:latest --version
+```
+
+Then restart the service so new sessions use the new image:
+
+```bash
+source setup/lib/install-slug.sh
+systemctl --user restart $(systemd_unit)              # Linux
+# or: launchctl kickstart -k gui/$(id -u)/$(launchd_label)  # macOS
+```
+
+### 5. Enable for groups
+
+Groups with the default skill selection (`skills: "all"` in container config)
+pick up `web-search-plus` automatically at the next container spawn. For a
+group with an explicit skills list, add `"web-search-plus"` to that list —
+`ncl groups config get` shows the current selection; until an `ncl` verb for
+skill selection lands, the in-tree query wrapper (`pnpm exec tsx scripts/q.ts`)
+is the sanctioned way to update the `skills` JSON column.
+
+## Configure a provider key
+
+This skill never handles credential values: apply copies no keys, sets no env
+vars, and threads nothing into the container. The engine routes to a provider
+when that provider's key variable is set in the per-group key file
+`/workspace/agent/.wsp.env` (inside the container; `groups/<folder>/.wsp.env`
+on the host, covered by the `groups/*` git-ignore).
+
+**Preferred: the OneCLI gateway holds the real key.** The engine's HTTP layer
+is stdlib `urllib`, which honors the `HTTPS_PROXY` / `SSL_CERT_FILE`
+environment the gateway sets in every agent container, so provider calls route
+through the gateway like all other outbound HTTPS. Store the key in the OneCLI
+vault as a custom secret scoped to the provider's API host (e.g.
+`api.search.brave.com` with header `X-Subscription-Token` for Brave), and set
+the placeholder in the key file so the engine routes to that provider:
+
+```bash
+# inside the agent container / group workspace
+cat > /workspace/agent/.wsp.env <<'EOF'
+BRAVE_API_KEY=onecli-managed
+EOF
+chmod 600 /workspace/agent/.wsp.env
+```
+
+The gateway injects the real credential per request; no key material enters
+the container or the repo.
+
+**Fallback: real key in the key file.** On installs without a vault entry for
+the provider, put the real key in `.wsp.env` instead of the placeholder. Same
+file, same permissions. One working provider is enough; more keys add fallback
+resilience. The full provider/key-variable table is in the agent-facing
+[SKILL.md](resources/web-search-plus/SKILL.md) this skill installs.
+
+## Next steps
+
+In a wired agent, run `wsp doctor` (offline provider/key report), then a live
+`wsp search "test query"` — the JSON response's `routing.provider` shows which
+provider served it. If every provider reports `missing_api_key`, the key file
+is absent or unreadable; if `python3: command not found`, the running
+container predates the image rebuild in step 4.
+
+## Recipe entry
+
+Independent — no dependency on other skills and no ordering constraint. Needs
+the agent-image rebuild (step 4) after apply; when composing several
+image-touching skills in one recipe run, one rebuild at the end covers them
+all.
+
+To back this skill out, follow [REMOVE.md](REMOVE.md).

.claude/skills/add-web-search-plus/resources/web-search-plus/SKILL.md

@@ -0,0 +1,124 @@
+---
+name: web-search-plus
+description: Multi-provider web search and URL extraction via your own provider keys. Use INSTEAD of the built-in WebSearch tool for every web search, and instead of WebFetch when a page needs real content extraction — WebSearch is metered against the SDK credit pool; wsp runs on self-managed keys at a fraction of the cost. Supports 13 providers (Serper, Brave, Tavily, Exa, Linkup, Firecrawl, Perplexity, SearXNG, ...) with smart auto-routing and fallback.
+allowed-tools: Bash(/app/skills/web-search-plus/bin/wsp:*)
+---
+
+# Web Search Plus (wsp)
+
+Search the web and extract page content through the Web Search Plus engine —
+provider auto-routing, fallback chains, parallel extraction, response caching.
+All calls go out on your own provider keys, **not** the metered WebSearch tool.
+
+The wrapper lives at `/app/skills/web-search-plus/bin/wsp` (called `wsp` below).
+
+## Quick start
+
+```bash
+wsp search "anthropic claude opus 4.8 release notes"   # auto-routed search, JSON out
+wsp search "graz hifi events" --max-results 5          # any engine flag passes through
+wsp search "breaking news openai" --type news --time-range day
+wsp extract https://example.com/article                # clean markdown from URL(s)
+wsp explain "some query"                               # dry-run: show routing decision
+wsp doctor                                             # offline provider/key/cooldown status
+```
+
+`wsp search` accepts the full engine flag surface after the query — e.g.
+`--provider brave` (pin one provider), `--type news|images|videos|places|shopping`,
+`--time-range hour|day|week|month|year`, `--include-domains`, `--exclude-domains`,
+`--mode research` (multi-provider research with extraction). For the full flag
+surface run `python3 /app/skills/web-search-plus/engine/search.py --help`.
+
+## Output handling
+
+- Output is **JSON on stdout — parse it, don't regex it.** Results are in
+  `results[]` (`title`, `url`, `snippet`/`content`); extraction output carries
+  per-URL `content` in markdown.
+- The `routing` object tells you what happened: `routing.provider` (who served
+  it), `routing.chain_tried` (fallback path), `routing.reason`.
+- On failure you get `"error": "All providers failed"` plus per-provider
+  `provider_errors` (e.g. `missing_api_key`, rate limits, cooldowns) — read
+  them before retrying; a missing key won't fix itself.
+- **Don't extract every search hit.** Snippets are usually enough; run
+  `wsp extract` only on the 1–3 URLs you actually need full content from.
+- Results are cached (default TTL applies). `--no-cache` forces a live call.
+
+## Provider keys
+
+The engine routes to a provider when that provider's key variable is **set**.
+Keys live in the per-group key file `/workspace/agent/.wsp.env` (mode 0600) —
+**never** in this skill directory and never in git.
+
+**Preferred: OneCLI gateway holds the real key.** The engine's HTTP layer is
+stdlib `urllib`, which honors the `HTTPS_PROXY` and `SSL_CERT_FILE` environment
+the gateway sets in every agent container — provider calls route through the
+gateway like any other outbound HTTPS. Store the provider key in the OneCLI
+vault as a custom secret scoped to the provider's API host (e.g.
+`api.search.brave.com`, header `X-Subscription-Token`), and put the placeholder
+in the key file so the engine routes to that provider:
+
+```bash
+cat > /workspace/agent/.wsp.env <<'EOF'
+BRAVE_API_KEY=onecli-managed
+EOF
+chmod 600 /workspace/agent/.wsp.env
+```
+
+The gateway replaces the credential at request time; no real key enters the
+container.
+
+**Fallback: real key in the key file.** On installs without a vault entry for
+the provider, put the real key in `/workspace/agent/.wsp.env` instead of the
+placeholder. Same file, same permissions; the workspace is git-ignored.
+
+One working provider beats ten configured ones. Two routing gotchas when
+picking which key to configure:
+
+- **Brave is excluded from auto-routing by the engine's default `auto_allow`
+  config.** With only `BRAVE_API_KEY` set, a plain `wsp search` reports
+  `no_available_providers` — pin it with `--provider brave`, or configure at
+  least one auto-allowed provider (Serper, Tavily, Linkup, Exa, Firecrawl,
+  You, SearXNG).
+- **`wsp extract` needs an extract-capable provider** (Tavily, Exa, Linkup,
+  Firecrawl, Parallel, You). Brave and Serper are search-only.
+
+Supported key variables (one per provider, no key pools):
+
+| Variable | Provider |
+|----------|----------|
+| `SERPER_API_KEY` | Serper (Google-style search/news/shopping/places) |
+| `SERPBASE_API_KEY` | Serpbase |
+| `BRAVE_API_KEY` | Brave Search |
+| `TAVILY_API_KEY` | Tavily |
+| `QUERIT_API_KEY` | Querit |
+| `LINKUP_API_KEY` | Linkup (cheap clean extraction) |
+| `EXA_API_KEY` | Exa (neural/keyword search) |
+| `FIRECRAWL_API_KEY` | Firecrawl (search + scrape) |
+| `PARALLEL_API_KEY` | Parallel |
+| `PERPLEXITY_API_KEY` | Perplexity (answer-style search) |
+| `KILOCODE_API_KEY` | Kilo (Perplexity via Kilo) |
+| `YOU_API_KEY` | You.com |
+| `SEARXNG_INSTANCE_URL` | SearXNG (keyless, self-hosted, $0/search) |
+
+Override the key-file location with `WSP_ENV_FILE`. Cache and provider-health
+state live in `/workspace/agent/.wsp-cache/` (override with `WSP_CACHE_DIR`).
+
+## Troubleshooting
+
+Run `wsp doctor` first — it reports, offline, every provider's key presence,
+cooldown state, and the cache status.
+
+- `missing_api_key` for every provider → no key file, or wrong path: check
+  `ls -la /workspace/agent/.wsp.env`.
+- One provider keeps failing with 401/422 → key invalid (or the gateway has no
+  secret for that host while the key file holds a placeholder); the engine puts
+  it in cooldown and falls through the chain. Fix the key or the vault entry.
+- `python3: command not found` → the container image predates this skill;
+  the host needs an image rebuild (`./container/build.sh`).
+
+## Engine
+
+The engine under `engine/` is vendored from
+[`hermes-web-search-plus`](https://github.com/robbyczgw-cla/hermes-web-search-plus)
+(v2.4.0 @ `373024c`). Pure stdlib, zero Python dependencies. Engine bugs are
+fixed upstream and pulled in via sync commits to this skill, never patched here.