Merge pull request #3 from ZenRows/fix/cli-taxonomy-primitives

Replace old product taxonomy with canonical primitives (Fetch / Extract / Batch / Browser Sessions)

Author: claude[bot]

README.md

@@ -4,8 +4,8 @@
 workflows, recipes, and evals layer for giving AI agents reliable access to
 protected web data through Zenrows cloud infrastructure.**
 
-It makes the Zenrows Universal Scraper API, Scraping Browser, and Residential
-Proxies installable and usable directly from AI agents and developer workflows —
+It makes the four Zenrows primitives — Fetch, Extract, Batch, and Browser
+Sessions — installable and usable directly from AI agents and developer workflows —
 so an agent can reliably access protected web data without hand-rolling anti-bot
 handling, proxies, or browser rendering.
 
@@ -21,25 +21,25 @@ through AI agents, developers, and teams.
 ## 2. Why Zenrows
 
 Normal fetch fails. Generic scrapers fail. Browser-first tools are expensive.
-The Zenrows **Universal Scraper API** retrieves protected pages reliably and
-structures them, while the **Scraping Browser** is there for the rare cases that
+Zenrows **Fetch** retrieves protected pages reliably and **Extract** structures
+them, while **Browser Sessions** are there for the rare cases that
 need a real browser. Zenrows wins when the workflow runs over thousands or
 millions of URLs.
 
 ## 3. Product architecture
 
-The core product is the Zenrows **Universal Scraper API**
+The core product is Zenrows **Fetch and Extract**
 (`GET https://api.zenrows.com/v1/`). The CLI exposes it two ways: `zenrows fetch`
 retrieves a protected page, and `zenrows extract` turns it into structured data
 (JSON / CSS / Markdown). Both call the same endpoint — `extract` is just
 that endpoint with extraction parameters, not a separate product.
 
 | Command | What it does | Status (this build) |
 | --- | --- | --- |
-| `zenrows fetch` | Universal Scraper API — retrieve a protected page | **available** — `GET https://api.zenrows.com/v1/` |
-| `zenrows extract` | Universal Scraper API — structured extraction (Autoparse / CSS / Markdown) | **available** — same `/v1/` |
-| `zenrows batch` | Batch Scraper API — fan out over many URLs | beta — cloud works with beta access; local validate/estimate always |
-| `zenrows browser` | Scraping Browser (CDP) / MCP escalation | experimental |
+| `zenrows fetch` | Fetch — retrieve a protected page | **available** — `GET https://api.zenrows.com/v1/` |
+| `zenrows extract` | Extract — `extract=auto` (domain-gated open beta; falls back to Autoparse) / CSS / Markdown | **beta** — same `/v1/` |
+| `zenrows batch` | Batch — fan out over many URLs | beta — cloud works with beta access; local validate/estimate always |
+| `zenrows browser` | Browser Sessions REST API (same backend as MCP `browser_*`) | **available** — escalation-only; bills by bandwidth + time |
 | `zenrows mcp` | MCP server config (remote + local) | **available** |
 | Zenrows CLI | this repo | available |
 
@@ -53,7 +53,8 @@ matrix.
 ```bash
 npx -y @zenrows/cli init
 zenrows fetch https://httpbin.io/html # auto-provisions a Free plan account on first use
-zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse
+zenrows extract https://www.owler.com/company/meltwater # extract=auto on an enabled domain
+zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse # Autoparse (any domain)
 ```
 
 No API key up front: on your first cloud call the toolkit creates a free,
@@ -110,14 +111,16 @@ zenrows fetch <url> --proxy-country us --wait-for ".price"
 ## 8. Extract
 
 ```bash
-zenrows extract <url> --autoparse
+zenrows extract https://www.owler.com/company/meltwater                # extract=auto (enabled domain)
+zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse   # Autoparse (any domain)
 zenrows extract <url> --css '{"title":"h1","price":".price"}' --validate
+zenrows extract <url> --outputs emails,links                            # built-in output filters → JSON
 zenrows extract <url> --output markdown
 ```
 
 ## 9. Batch (beta)
 
-The Zenrows **Batch Scraper API** (`https://async.api.zenrows.com/v1`) fans a
+Zenrows **Batch** (`https://async.api.zenrows.com/v1`) fans a
 protected fetch/extract out over many URLs. It is a real product in
 **beta**: the cloud subcommands work once your API key has
 beta access; without it the API returns `BATCH_ACCESS_DENIED`. Local spec
@@ -138,8 +141,17 @@ locally or fan out with `zenrows fetch` per URL.
 
 ## 10. Browser Sessions
 
-Escalation only, gated by `policy.allow_browser`. Backed by the Zenrows Scraping
-Browser (CDP) and the `@zenrows/mcp` `browser_*` tools.
+Escalation only — **prefer `fetch`/`extract` for the vast majority of cases**;
+they cost less. Use the browser for logins, forms, and multi-step JS flows that
+Protected Fetch can't handle. Drives the managed Browser Sessions REST API
+(`mcp.zenrows.com/browser/sessions/*`, same backend as the `@zenrows/mcp`
+`browser_*` tools). `zenrows browser connect` prints the raw CDP wss URL for
+bring-your-own Playwright/Puppeteer.
+
+**Billing & lifecycle:** sessions bill by **bandwidth + session time** and
+**auto-terminate after 15 minutes** — `run <script.json>` closes automatically;
+close interactive sessions with `zenrows browser close`. On by default; opt out
+with `zenrows policy set allow_browser false`.
 
 ## 11. MCP
 
@@ -216,8 +228,8 @@ Reports write `input.json`, `results.json`, `report.md`, `failures.jsonl`,
 - `.zenrows/account.json` holds no secret — only the accountId, Free-period info, and
   claim link.
 - `.zenrows/policy.json` enforces credit/page/concurrency limits and domain allow/deny.
-- Destructive `uninstall` requires `--yes`. Browser and experimental are off by
-  default.
+- Destructive `uninstall` requires `--yes`. Browser is on by default; opt out with
+  `zenrows policy set allow_browser false`. Experimental features are off by default.
 
 ## 19. Capability matrix
 

docs/capabilities.md

@@ -7,8 +7,9 @@ the single source of truth that prevents hallucinated execution.
 
 - **available** — a documented endpoint exists and the command does real work.
 - **available-but-needs-confirmation** — likely available; verify per account.
-- **experimental** — exists but gated (e.g. browser, behind `policy.allow_browser`).
-- **beta** — real product in beta; limited access (local spec / validation works today, cloud execution needs beta access).
+- **experimental** — exists but not yet promoted to a stable status.
+- **beta** — open beta; usable by any key. Product-specific limits (e.g. Extract
+  domain coverage) are handled by adapters / API errors, not by blocking the CLI.
 - **planned** — no documented endpoint yet; local spec / validation only.
 - **not-implemented** / **deprecated** — not usable.
 
@@ -18,15 +19,8 @@ Classification is based on the public Zenrows documentation:
 
 | Capability | Backend evidence | Status |
 | --- | --- | --- |
-| `protected_fetch` | Universal Scraper API `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available |
-| `extract` | Same `/v1/` endpoint via `autoparse`, `css_extractor`, `response_type=markdown\|plaintext` | available |
-| `batch` | Zenrows Batch Scraper API `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — real product in beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. Local JSONL spec validation + credit estimation work with no key. | beta |
-| `browser` | Zenrows Scraping Browser (CDP) + `@zenrows/mcp` `browser_*` tools; no managed REST sessions API | experimental |
+| `protected_fetch` | Fetch `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available |
+| `extract` | Extract `GET https://api.zenrows.com/v1/` via `extract=auto` (domain-gated open beta; CLI falls back to `autoparse`), plus `autoparse`, `css_extractor`, `outputs`, `response_type=markdown\|plaintext` | beta |
+| `batch` | Batch `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — open beta. Local JSONL spec validation + credit estimation work with no key. | beta |
+| `browser` | Browser Sessions REST API `https://mcp.zenrows.com/browser/sessions/*` (Bearer; same backend as `@zenrows/mcp` `browser_*`); CDP via `zenrows browser connect`. Escalation-only (prefer fetch/extract); on by default, opt out via `policy.allow_browser`; bills by bandwidth + session time (15-min max) | available |
 | `mcp` | Hosted `https://mcp.zenrows.com/mcp` + local `npx -y @zenrows/mcp` | available |
-
-## Important honesty note
-
-`protected_fetch` and `extract` are the **same** product: a single `/v1/`
-Universal Scraper API. "Extract" is not a separate endpoint — it is parameters
-on that endpoint (`autoparse` / `css_extractor` / `response_type`). The CLI keeps
-them as separate commands only for ergonomics.

evals/extract-smoke/README.md

@@ -1,9 +1,9 @@
 # Eval: extract-smoke
 
-Reproducible smoke test for Extract (Autoparse).
+Reproducible smoke test for Extract (`extract=auto`).
 
-- **Target:** `https://www.scrapingcourse.com/ecommerce/` (public scraping demo)
-- **Config:** `autoparse=true`
+- **Target:** `https://www.owler.com/company/meltwater` (enabled domain, real company page)
+- **Config:** `extract=auto`
 - **Success criteria:** HTTP 2xx and a non-empty body
 - **Cost:** ~1× basic request
 

evals/extract-smoke/spec.json

@@ -1,10 +1,10 @@
 {
-  "description": "Smoke test: Extract returns structured output for a known-good ecommerce demo.",
+  "description": "Smoke test: Extract (extract=auto) returns structured output for an enabled company page.",
   "steps": [
     {
       "kind": "extract",
-      "url": "https://www.scrapingcourse.com/ecommerce/",
-      "options": { "method": "autoparse", "jsRender": false },
+      "url": "https://www.owler.com/company/meltwater",
+      "options": { "method": "extract", "jsRender": false },
       "expect": { "minLength": 2 }
     }
   ]

registry/capabilities.json

@@ -1,50 +1,50 @@
 {
-  "$comment": "Honest capability matrix derived from confirmed Zenrows docs. Every command checks status here before attempting a cloud call. 'available' = a documented endpoint exists today; 'planned' = no documented endpoint yet (local spec / unavailable behavior only); 'experimental' = exists but gated behind policy.",
+  "$comment": "Capability matrix for the Zenrows CLI. Every command consults this file before attempting a cloud call, so the CLI never fakes behavior for primitives the backend does not expose. Status values and classification rationale live in docs/capabilities.md; entries must stay in sync with the Capability type in src/types/index.ts.",
   "capabilities": {
     "protected_fetch": {
       "key": "protected_fetch",
       "label": "Protected Fetch",
       "status": "available",
       "command": "zenrows fetch",
-      "backend": "GET https://api.zenrows.com/v1/",
+      "backend": "Fetch — GET https://api.zenrows.com/v1/",
       "requiresAuth": true,
-      "notes": "Universal Scraper API. Confirmed params: mode=auto (Adaptive Stealth), js_render, premium_proxy, proxy_country, wait, wait_for, js_instructions, custom_headers, session_id, original_status, allowed_status_codes, block_resources, response_type, screenshot."
+      "notes": "Supported params: mode=auto (Adaptive Stealth), js_render, premium_proxy, proxy_country, wait, wait_for, js_instructions, custom_headers, session_id, original_status, allowed_status_codes, block_resources, response_type, screenshot."
     },
     "extract": {
       "key": "extract",
-      "label": "Extract (Autoparse / CSS / Markdown)",
-      "status": "available",
+      "label": "Extract (extract=auto / Autoparse / CSS / Markdown)",
+      "status": "beta",
       "command": "zenrows extract",
-      "backend": "GET https://api.zenrows.com/v1/ (autoparse, css_extractor, response_type)",
+      "backend": "Extract — GET https://api.zenrows.com/v1/ (extract=auto, autoparse, css_extractor, outputs, response_type)",
       "requiresAuth": true,
-      "notes": "Structured extraction runs on the same /v1/ endpoint via autoparse=true, css_extractor, and response_type=markdown|plaintext. There is no separate /extract endpoint."
+      "notes": "Open beta. Default is extract=auto (domain-gated); CLI falls back to autoparse on AUTH010. Autoparse / CSS / outputs / markdown work on any domain."
     },
     "batch": {
       "key": "batch",
       "label": "Batch (beta)",
       "status": "beta",
       "command": "zenrows batch",
-      "backend": "Batch Scraper API — https://async.api.zenrows.com/v1 (X-API-Key header; separate host from the scraper /v1/)",
+      "backend": "Batch — https://async.api.zenrows.com/v1 (X-API-Key header; separate host from the Fetch/Extract /v1/)",
       "requiresAuth": true,
-      "notes": "The Zenrows Batch Scraper API is a real product in beta. The cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → BATCH_ACCESS_DENIED. Local value always works with no key: `zenrows batch estimate` validates JSONL job specs and estimates credit cost."
+      "notes": "Open beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) call the Batch API; `zenrows batch estimate` works locally with no API key."
     },
     "browser": {
       "key": "browser",
-      "label": "Browser (Scraping Browser)",
-      "status": "experimental",
+      "label": "Browser Sessions",
+      "status": "available",
       "command": "zenrows browser",
-      "backend": "Scraping Browser (CDP) + @zenrows/mcp browser tools",
+      "backend": "Browser Sessions REST API — https://mcp.zenrows.com/browser/sessions/* (Bearer auth; same backend as @zenrows/mcp browser_* tools)",
       "requiresAuth": true,
-      "notes": "Zenrows Scraping Browser and the @zenrows/mcp browser_* tools exist. There is no managed REST 'sessions' API in the public docs, so this is gated as experimental and escalation-only (policy.allow_browser=false by default)."
+      "notes": "Browser Sessions is a GA Zenrows product (formerly Scraping Browser). Drives its managed REST API directly (create/verb/close over HTTP with Authorization: Bearer) — no CDP client or browser dependency needed. Escalation-only: prefer fetch/extract (they cost less) for the vast majority of cases. On by default; opt out with policy.allow_browser=false. Sessions bill by bandwidth + session time and auto-terminate after 15 minutes. For raw CDP control, `zenrows browser connect` prints the wss://browser.zenrows.com endpoint for your own Playwright/Puppeteer."
     },
     "mcp": {
       "key": "mcp",
       "label": "MCP",
       "status": "available",
       "command": "zenrows mcp",
-      "backend": "remote https://mcp.zenrows.com/mcp + local npx -y @zenrows/mcp",
+      "backend": "Remote https://mcp.zenrows.com/mcp + local `npx -y @zenrows/mcp`",
       "requiresAuth": true,
-      "notes": "Both a hosted remote MCP server and a local STDIO server (@zenrows/mcp, ZENROWS_API_KEY env) are documented."
+      "notes": "Hosted remote MCP server plus a local STDIO server (`npx -y @zenrows/mcp`, authenticated via the ZENROWS_API_KEY environment variable)."
     }
   }
 }

registry/skills.json

@@ -26,7 +26,7 @@
       "name": "extract",
       "type": "skill",
       "description": "Turn protected pages into structured data with Autoparse / CSS / Markdown.",
-      "status": "available",
+      "status": "beta",
       "requires_backend_capabilities": ["extract"],
       "requires_auth": true,
       "version": "0.1.0",
@@ -36,7 +36,7 @@
     {
       "name": "batch-jobs",
       "type": "skill",
-      "description": "Scale protected fetch/extract over many URLs with the Batch Scraper API (beta): submit/track/collect jobs with beta access; validate + estimate specs locally with no key.",
+      "description": "Scale protected fetch/extract over many URLs with Batch (beta): submit/track/collect jobs with beta access; validate + estimate specs locally with no key.",
       "status": "beta",
       "requires_backend_capabilities": ["batch"],
       "requires_auth": true,
@@ -47,8 +47,8 @@
     {
       "name": "interact-browser",
       "type": "skill",
-      "description": "Escalate to a browser (Scraping Browser / MCP) only when fetch/extract cannot do the job.",
-      "status": "experimental",
+      "description": "Escalate to Browser Sessions (REST API / MCP browser_*) only when fetch/extract cannot do the job.",
+      "status": "available",
       "requires_backend_capabilities": ["browser"],
       "requires_auth": true,
       "version": "0.1.0",

registry/templates.json

@@ -3,7 +3,7 @@
     {
       "name": "protected-fetch-node",
       "type": "template",
-      "description": "Minimal Node.js project calling the Zenrows Universal Scraper API.",
+      "description": "Minimal Node.js project calling Zenrows Fetch.",
       "status": "available",
       "requires_backend_capabilities": ["protected_fetch"],
       "requires_auth": true,
@@ -25,7 +25,7 @@
     {
       "name": "batch-jsonl-pipeline",
       "type": "template",
-      "description": "JSONL job-spec scaffold for high-scale workloads on the Batch Scraper API: submit/track/collect with beta access, validate + estimate locally with no key (beta).",
+      "description": "JSONL job-spec scaffold for high-scale workloads on Batch: submit/track/collect with beta access, validate + estimate locally with no key (beta).",
       "status": "beta",
       "requires_backend_capabilities": [],
       "requires_auth": false,

skills/batch-jobs/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: batch-jobs
-description: Scale protected fetch/extract over many URLs via the Batch Scraper API (beta). Cloud create/status/results/cancel/wait/retry-failed work with beta access; estimate/validate run locally with no key.
+description: Scale protected fetch/extract over many URLs via Batch (beta). Cloud create/status/results/cancel/wait/retry-failed work with beta access; estimate/validate run locally with no key.
 version: 0.1.0
 requires_backend_capabilities: [batch]
 ---
@@ -11,7 +11,7 @@ Process large workloads reliably and asynchronously. Batch is where Zenrows'
 high-scale anti-bot advantage becomes obvious — Zenrows wins when the workflow
 runs over thousands, millions, or recurring sets of URLs.
 
-> Status: **beta**. The Zenrows Batch Scraper API is a real
+> Status: **beta**. The Zenrows **Batch** is a real
 > product in beta and runs on a separate host
 > (`async.api.zenrows.com/v1`). The cloud subcommands work once your account has
 > beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. The

skills/compliance-policy/SKILL.md

@@ -15,16 +15,16 @@ zenrows policy show
 zenrows policy set max_credits_per_run 5000
 zenrows policy set blocked_domains "example.com,foo.test"
 zenrows policy set allowed_domains "mysite.com"      # non-empty = allow-list mode
-zenrows policy set allow_browser true                # enable escalation
+zenrows policy set allow_browser false               # opt OUT of browser (on by default)
 zenrows policy set allow_experimental true
 ```
 
 ## Hard rules
 - **Never** print or commit API keys. Keys live in `.zenrows/secrets.json`
   (0600, gitignored) and are redacted from logs and run artifacts.
-- Respect `allowed_domains` / `blocked_domains` (→ `POLICY_BLOCKED_DOMAIN`).
-- Stay under `max_credits_per_run` / `max_pages_per_run` / `max_concurrency`.
+- Respect `allowed_domains` / `blocked_domains` — enforced by the CLI (→ `POLICY_BLOCKED_DOMAIN`).
+- `max_credits_per_run` / `max_pages_per_run` / `max_concurrency` are **advisory budgets** surfaced by `zenrows status` (not hard-enforced by the CLI today) — self-limit against them.
 - Confirm destructive `uninstall` with `--yes`.
-- Browser and experimental commands are **off by default**.
+- **Experimental** commands are off by default (`allow_experimental`). **Browser is on by default** (opt out with `allow_browser=false`).
 
 See [[cost-control]] and [[trace-debug]].

skills/cost-control/SKILL.md

@@ -9,7 +9,7 @@ requires_backend_capabilities: []
 
 Prefer the **cheapest reliable** configuration; escalate only with evidence.
 
-## Cost multipliers (Universal Scraper API)
+## Cost multipliers (Fetch)
 - Basic request: **1×**
 - JS rendering (`js_render`): **5×**
 - Premium proxies (`premium_proxy`): **10×**