From 741c3b1e83733517c8af57cde7309556f7cf52f3 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 31 Jul 2026 11:10:09 +0000 Subject: [PATCH 1/2] Replace old product taxonomy with canonical primitives Update descriptive/label/help/comment text across the CLI, registry, skills, templates, workflows, and docs to use the four canonical primitives: - "Universal Scraper API" -> Fetch / Extract / "Fetch and Extract" (umbrella) - "Scraping Browser" -> Browser Sessions - "Residential Proxies" (standalone) -> folded into the primitives - "Batch Scraper API" -> "Batch" in prose/labels, "Batch API" in error strings Text-only: no command names, flags, endpoints, or hosts changed. Typecheck passes. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NsXfB6dR3rRX64K3NwyuoD --- README.md | 24 +++++++++---------- docs/capabilities.md | 8 +++---- registry/capabilities.json | 12 +++++----- registry/skills.json | 4 ++-- registry/templates.json | 4 ++-- skills/batch-jobs/SKILL.md | 4 ++-- skills/cost-control/SKILL.md | 2 +- skills/extract/SKILL.md | 2 +- skills/interact-browser/SKILL.md | 2 +- skills/protected-fetch/SKILL.md | 4 ++-- skills/zenrows/SKILL.md | 6 ++--- src/adapters/batch.ts | 4 ++-- src/adapters/extract.ts | 2 +- src/adapters/protected-fetch.ts | 4 ++-- src/cli/commands/batch.ts | 4 ++-- src/cli/commands/browser.ts | 10 ++++---- src/cli/commands/extract.ts | 6 ++--- src/cli/commands/fetch.ts | 4 ++-- src/cli/commands/usage.ts | 2 +- src/core/batch-api.ts | 18 +++++++------- src/core/config.ts | 4 ++-- src/core/http.ts | 4 ++-- src/core/usage.ts | 2 +- templates/batch-jsonl-pipeline/README.md | 4 ++-- templates/protected-fetch-node/README.md | 2 +- workflows/competitor-intelligence/WORKFLOW.md | 4 ++-- 26 files changed, 73 insertions(+), 73 deletions(-) diff --git a/README.md b/README.md index 02fb5a1..634d4e8 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,8 @@ workflows, recipes, and evals layer for giving AI agents reliable access to protected web data through Zenrows cloud infrastructure.** -It makes the Zenrows Universal Scraper API, Scraping Browser, and Residential -Proxies installable and usable directly from AI agents and developer workflows — +It makes the four Zenrows primitives — Fetch, Extract, Batch, and Browser +Sessions — installable and usable directly from AI agents and developer workflows — so an agent can reliably access protected web data without hand-rolling anti-bot handling, proxies, or browser rendering. @@ -21,14 +21,14 @@ through AI agents, developers, and teams. ## 2. Why Zenrows Normal fetch fails. Generic scrapers fail. Browser-first tools are expensive. -The Zenrows **Universal Scraper API** retrieves protected pages reliably and -structures them, while the **Scraping Browser** is there for the rare cases that +Zenrows **Fetch** retrieves protected pages reliably and **Extract** structures +them, while **Browser Sessions** are there for the rare cases that need a real browser. Zenrows wins when the workflow runs over thousands or millions of URLs. ## 3. Product architecture -The core product is the Zenrows **Universal Scraper API** +The core product is Zenrows **Fetch and Extract** (`GET https://api.zenrows.com/v1/`). The CLI exposes it two ways: `zenrows fetch` retrieves a protected page, and `zenrows extract` turns it into structured data (JSON / CSS / Markdown). Both call the same endpoint — `extract` is just @@ -36,10 +36,10 @@ that endpoint with extraction parameters, not a separate product. | Command | What it does | Status (this build) | | --- | --- | --- | -| `zenrows fetch` | Universal Scraper API — retrieve a protected page | **available** — `GET https://api.zenrows.com/v1/` | -| `zenrows extract` | Universal Scraper API — structured extraction (Autoparse / CSS / Markdown) | **available** — same `/v1/` | -| `zenrows batch` | Batch Scraper API — fan out over many URLs | beta — cloud works with beta access; local validate/estimate always | -| `zenrows browser` | Scraping Browser (CDP) / MCP escalation | experimental | +| `zenrows fetch` | Fetch — retrieve a protected page | **available** — `GET https://api.zenrows.com/v1/` | +| `zenrows extract` | Extract — structured extraction (Autoparse / CSS / Markdown) | **available** — same `/v1/` | +| `zenrows batch` | Batch — fan out over many URLs | beta — cloud works with beta access; local validate/estimate always | +| `zenrows browser` | Browser Sessions (CDP) / MCP escalation | experimental | | `zenrows mcp` | MCP server config (remote + local) | **available** | | Zenrows CLI | this repo | available | @@ -117,7 +117,7 @@ zenrows extract --output markdown ## 9. Batch (beta) -The Zenrows **Batch Scraper API** (`https://async.api.zenrows.com/v1`) fans a +Zenrows **Batch** (`https://async.api.zenrows.com/v1`) fans a protected fetch/extract out over many URLs. It is a real product in **beta**: the cloud subcommands work once your API key has beta access; without it the API returns `BATCH_ACCESS_DENIED`. Local spec @@ -138,8 +138,8 @@ locally or fan out with `zenrows fetch` per URL. ## 10. Browser Sessions -Escalation only, gated by `policy.allow_browser`. Backed by the Zenrows Scraping -Browser (CDP) and the `@zenrows/mcp` `browser_*` tools. +Escalation only, gated by `policy.allow_browser`. Backed by Zenrows Browser +Sessions (CDP) and the `@zenrows/mcp` `browser_*` tools. ## 11. MCP diff --git a/docs/capabilities.md b/docs/capabilities.md index bc0637e..934b4c7 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -18,15 +18,15 @@ Classification is based on the public Zenrows documentation: | Capability | Backend evidence | Status | | --- | --- | --- | -| `protected_fetch` | Universal Scraper API `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available | +| `protected_fetch` | Fetch and Extract `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available | | `extract` | Same `/v1/` endpoint via `autoparse`, `css_extractor`, `response_type=markdown\|plaintext` | available | -| `batch` | Zenrows Batch Scraper API `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — real product in beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. Local JSONL spec validation + credit estimation work with no key. | beta | -| `browser` | Zenrows Scraping Browser (CDP) + `@zenrows/mcp` `browser_*` tools; no managed REST sessions API | experimental | +| `batch` | Zenrows Batch `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — real product in beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. Local JSONL spec validation + credit estimation work with no key. | beta | +| `browser` | Zenrows Browser Sessions (CDP) + `@zenrows/mcp` `browser_*` tools; no managed REST sessions API | experimental | | `mcp` | Hosted `https://mcp.zenrows.com/mcp` + local `npx -y @zenrows/mcp` | available | ## Important honesty note `protected_fetch` and `extract` are the **same** product: a single `/v1/` -Universal Scraper API. "Extract" is not a separate endpoint — it is parameters +Fetch and Extract API. "Extract" is not a separate endpoint — it is parameters on that endpoint (`autoparse` / `css_extractor` / `response_type`). The CLI keeps them as separate commands only for ergonomics. diff --git a/registry/capabilities.json b/registry/capabilities.json index 87e75d8..980613b 100644 --- a/registry/capabilities.json +++ b/registry/capabilities.json @@ -8,7 +8,7 @@ "command": "zenrows fetch", "backend": "GET https://api.zenrows.com/v1/", "requiresAuth": true, - "notes": "Universal Scraper API. Confirmed params: mode=auto (Adaptive Stealth), js_render, premium_proxy, proxy_country, wait, wait_for, js_instructions, custom_headers, session_id, original_status, allowed_status_codes, block_resources, response_type, screenshot." + "notes": "Fetch and Extract. Confirmed params: mode=auto (Adaptive Stealth), js_render, premium_proxy, proxy_country, wait, wait_for, js_instructions, custom_headers, session_id, original_status, allowed_status_codes, block_resources, response_type, screenshot." }, "extract": { "key": "extract", @@ -24,18 +24,18 @@ "label": "Batch (beta)", "status": "beta", "command": "zenrows batch", - "backend": "Batch Scraper API — https://async.api.zenrows.com/v1 (X-API-Key header; separate host from the scraper /v1/)", + "backend": "Batch — https://async.api.zenrows.com/v1 (X-API-Key header; separate host from the Fetch/Extract /v1/)", "requiresAuth": true, - "notes": "The Zenrows Batch Scraper API is a real product in beta. The cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → BATCH_ACCESS_DENIED. Local value always works with no key: `zenrows batch estimate` validates JSONL job specs and estimates credit cost." + "notes": "The Zenrows Batch is a real product in beta. The cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → BATCH_ACCESS_DENIED. Local value always works with no key: `zenrows batch estimate` validates JSONL job specs and estimates credit cost." }, "browser": { "key": "browser", - "label": "Browser (Scraping Browser)", + "label": "Browser Sessions", "status": "experimental", "command": "zenrows browser", - "backend": "Scraping Browser (CDP) + @zenrows/mcp browser tools", + "backend": "Browser Sessions (CDP) + @zenrows/mcp browser tools", "requiresAuth": true, - "notes": "Zenrows Scraping Browser and the @zenrows/mcp browser_* tools exist. There is no managed REST 'sessions' API in the public docs, so this is gated as experimental and escalation-only (policy.allow_browser=false by default)." + "notes": "Zenrows Browser Sessions and the @zenrows/mcp browser_* tools exist. There is no managed REST 'sessions' API in the public docs, so this is gated as experimental and escalation-only (policy.allow_browser=false by default)." }, "mcp": { "key": "mcp", diff --git a/registry/skills.json b/registry/skills.json index 7d19f2e..227b34d 100644 --- a/registry/skills.json +++ b/registry/skills.json @@ -36,7 +36,7 @@ { "name": "batch-jobs", "type": "skill", - "description": "Scale protected fetch/extract over many URLs with the Batch Scraper API (beta): submit/track/collect jobs with beta access; validate + estimate specs locally with no key.", + "description": "Scale protected fetch/extract over many URLs with Batch (beta): submit/track/collect jobs with beta access; validate + estimate specs locally with no key.", "status": "beta", "requires_backend_capabilities": ["batch"], "requires_auth": true, @@ -47,7 +47,7 @@ { "name": "interact-browser", "type": "skill", - "description": "Escalate to a browser (Scraping Browser / MCP) only when fetch/extract cannot do the job.", + "description": "Escalate to a browser (Browser Sessions / MCP) only when fetch/extract cannot do the job.", "status": "experimental", "requires_backend_capabilities": ["browser"], "requires_auth": true, diff --git a/registry/templates.json b/registry/templates.json index 4717f83..d65b411 100644 --- a/registry/templates.json +++ b/registry/templates.json @@ -3,7 +3,7 @@ { "name": "protected-fetch-node", "type": "template", - "description": "Minimal Node.js project calling the Zenrows Universal Scraper API.", + "description": "Minimal Node.js project calling Zenrows Fetch.", "status": "available", "requires_backend_capabilities": ["protected_fetch"], "requires_auth": true, @@ -25,7 +25,7 @@ { "name": "batch-jsonl-pipeline", "type": "template", - "description": "JSONL job-spec scaffold for high-scale workloads on the Batch Scraper API: submit/track/collect with beta access, validate + estimate locally with no key (beta).", + "description": "JSONL job-spec scaffold for high-scale workloads on Batch: submit/track/collect with beta access, validate + estimate locally with no key (beta).", "status": "beta", "requires_backend_capabilities": [], "requires_auth": false, diff --git a/skills/batch-jobs/SKILL.md b/skills/batch-jobs/SKILL.md index ee31df6..e6bc028 100644 --- a/skills/batch-jobs/SKILL.md +++ b/skills/batch-jobs/SKILL.md @@ -1,6 +1,6 @@ --- name: batch-jobs -description: Scale protected fetch/extract over many URLs via the Batch Scraper API (beta). Cloud create/status/results/cancel/wait/retry-failed work with beta access; estimate/validate run locally with no key. +description: Scale protected fetch/extract over many URLs via Batch (beta). Cloud create/status/results/cancel/wait/retry-failed work with beta access; estimate/validate run locally with no key. version: 0.1.0 requires_backend_capabilities: [batch] --- @@ -11,7 +11,7 @@ Process large workloads reliably and asynchronously. Batch is where Zenrows' high-scale anti-bot advantage becomes obvious — Zenrows wins when the workflow runs over thousands, millions, or recurring sets of URLs. -> Status: **beta**. The Zenrows Batch Scraper API is a real +> Status: **beta**. The Zenrows **Batch** is a real > product in beta and runs on a separate host > (`async.api.zenrows.com/v1`). The cloud subcommands work once your account has > beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. The diff --git a/skills/cost-control/SKILL.md b/skills/cost-control/SKILL.md index 903a041..2ed6aef 100644 --- a/skills/cost-control/SKILL.md +++ b/skills/cost-control/SKILL.md @@ -9,7 +9,7 @@ requires_backend_capabilities: [] Prefer the **cheapest reliable** configuration; escalate only with evidence. -## Cost multipliers (Universal Scraper API) +## Cost multipliers (Fetch and Extract) - Basic request: **1×** - JS rendering (`js_render`): **5×** - Premium proxies (`premium_proxy`): **10×** diff --git a/skills/extract/SKILL.md b/skills/extract/SKILL.md index aad6239..215e561 100644 --- a/skills/extract/SKILL.md +++ b/skills/extract/SKILL.md @@ -11,7 +11,7 @@ Convert protected pages into structured data. The value is **protected page access + extraction**, not generic LLM parsing. > Honest note: there is no separate `/extract` endpoint. Extraction runs on the -> same Universal Scraper API (`/v1/`) via `autoparse`, `css_extractor`, and +> same Fetch and Extract API (`/v1/`) via `autoparse`, `css_extractor`, and > `response_type`. ## Methods (available today) diff --git a/skills/interact-browser/SKILL.md b/skills/interact-browser/SKILL.md index 3fbdba1..8ed6a64 100644 --- a/skills/interact-browser/SKILL.md +++ b/skills/interact-browser/SKILL.md @@ -14,7 +14,7 @@ or Extract is insufficient. > Status: **experimental**, gated behind `policy.allow_browser` (default false). > There is no managed REST "sessions" API; browser workflows run through the -> Zenrows **Scraping Browser** (CDP — connect Playwright/Puppeteer) and the +> Zenrows **Browser Sessions** (CDP — connect Playwright/Puppeteer) and the > `@zenrows/mcp` `browser_*` tools (navigate, click, fill, screenshot, …). ## Rules diff --git a/skills/protected-fetch/SKILL.md b/skills/protected-fetch/SKILL.md index aa21460..9af8a46 100644 --- a/skills/protected-fetch/SKILL.md +++ b/skills/protected-fetch/SKILL.md @@ -8,8 +8,8 @@ requires_backend_capabilities: [protected_fetch] # Protected Fetch Retrieve pages that normal fetch, generic scrapers, or naive browser tools -cannot. This is the **core primitive** — backed by the Zenrows Universal Scraper -API (`GET https://api.zenrows.com/v1/`). +cannot. This is the **core primitive** — backed by Zenrows **Fetch** +(`GET https://api.zenrows.com/v1/`). ## When to use - You have a known URL and want its content (HTML, Markdown, text, or a PDF). diff --git a/skills/zenrows/SKILL.md b/skills/zenrows/SKILL.md index e4b84ae..51c8d0d 100644 --- a/skills/zenrows/SKILL.md +++ b/skills/zenrows/SKILL.md @@ -66,14 +66,14 @@ Run `zenrows status` for the live capability matrix. As of this toolkit: | Protected Fetch | `zenrows fetch` | available (`GET /v1/`) | | Extract (Autoparse/CSS/Markdown) | `zenrows extract` | available (same `/v1/`) | | Batch | `zenrows batch` | beta (validate specs locally) | -| Browser | `zenrows browser` | experimental (Scraping Browser / MCP) | +| Browser | `zenrows browser` | experimental (Browser Sessions / MCP) | | MCP | `zenrows mcp` | available (remote + local server) | -Protected Fetch and Extract are the same Universal Scraper API used two ways. +Protected Fetch and Extract are the same API (`GET /v1/`) used two ways. The command consults the capability matrix before any cloud call — it never fakes success. -## Cost model (Universal Scraper API) +## Cost model (Fetch and Extract) Relative multipliers: basic **1×**, JS rendering **5×**, premium proxies **10×**, both **25×**. `mode=auto` charges only for the configuration that succeeds. diff --git a/src/adapters/batch.ts b/src/adapters/batch.ts index 1999d6e..78b0042 100644 --- a/src/adapters/batch.ts +++ b/src/adapters/batch.ts @@ -1,7 +1,7 @@ /** * Batch Jobs adapter. * - * Status: `beta` — the Zenrows Batch Scraper API is in beta. + * Status: `beta` — the Zenrows Batch API is in beta. * With beta access the cloud subcommands run for real (see `core/batch-api.ts`). * Without access the API returns 403 → BATCH_ACCESS_DENIED. This adapter owns * the local, no-network pieces: validating a JSONL job spec, estimating credit @@ -176,7 +176,7 @@ function assertProxyCountryPremium(params: Record, where: strin code: "PARAM_PROXY_COUNTRY_REQUIRES_PREMIUM", message: `proxy_country needs premium proxies (or mode=auto) — ${where}.`, likely_cause: - "The Batch Scraper API only geolocates the proxy when premium_proxy=true, or in Adaptive Stealth Mode (mode=auto).", + "Batch only geolocates the proxy when premium_proxy=true, or in Adaptive Stealth Mode (mode=auto).", next_action: "Add premium_proxy (10x cost) alongside proxy_country, or set mode=auto — geo-targeting works there without the flag.", suggested_commands: ["zenrows batch create jobs.jsonl --premium-proxy --proxy-country us"], diff --git a/src/adapters/extract.ts b/src/adapters/extract.ts index a39138a..feb4fad 100644 --- a/src/adapters/extract.ts +++ b/src/adapters/extract.ts @@ -2,7 +2,7 @@ * Extract adapter. * * IMPORTANT (honest mapping): there is no separate `/extract` endpoint. - * Structured extraction runs on the same `/v1/` Universal Scraper API via: + * Structured extraction runs on the same `/v1/` Fetch and Extract API via: * - autoparse=true → automatic structured JSON (available) * - css_extractor= → selector-based field extraction (available) * - outputs= → built-in output filters → JSON (available) diff --git a/src/adapters/protected-fetch.ts b/src/adapters/protected-fetch.ts index 6280b00..2e280d0 100644 --- a/src/adapters/protected-fetch.ts +++ b/src/adapters/protected-fetch.ts @@ -1,5 +1,5 @@ /** - * Protected Fetch adapter → Zenrows Universal Scraper API (`GET /v1/`). + * Protected Fetch adapter → Zenrows Fetch (`GET /v1/`). * * Maps toolkit options to confirmed API parameters and enforces the * auto/manual contract: in Adaptive Stealth Mode (mode=auto), `js_render` and @@ -34,7 +34,7 @@ export interface FetchOptions { cssExtractor?: string; autoparse?: boolean; /** - * Comma-separated Universal Scraper API output filters (e.g. "emails,links", + * Comma-separated Fetch and Extract output filters (e.g. "emails,links", * or "*" for all available fields). Returns structured JSON. Standalone: not combined with * autoparse / css_extractor / response_type. */ diff --git a/src/cli/commands/batch.ts b/src/cli/commands/batch.ts index 46a102d..99af8de 100644 --- a/src/cli/commands/batch.ts +++ b/src/cli/commands/batch.ts @@ -1,5 +1,5 @@ /** - * `zenrows batch` — Batch Scraper API (status: beta). + * `zenrows batch` — Batch API (status: beta). * * Local (no key, always works): `estimate`/`create --dry-run`-style spec * validation + credit estimate. Cloud (needs a key + Batch beta access): @@ -18,7 +18,7 @@ import { printError, writeOut } from "../output.ts"; export const batch: Command = { name: "batch", - summary: "Run JSONL batch jobs on the Zenrows Batch Scraper API (beta).", + summary: "Run JSONL batch jobs on Zenrows Batch (beta).", usage: "zenrows batch |status |results |cancel |wait |retry-failed >", help: [ "Local (no key):", diff --git a/src/cli/commands/browser.ts b/src/cli/commands/browser.ts index 1b36e83..d5639a5 100644 --- a/src/cli/commands/browser.ts +++ b/src/cli/commands/browser.ts @@ -2,7 +2,7 @@ * `zenrows browser` — Interact / Browser Sessions (status: experimental). * * Browser is an ESCALATION layer, not the default. There is no documented REST - * "sessions" API; Zenrows exposes the Scraping Browser (CDP) and browser_* + * "sessions" API; Zenrows exposes Browser Sessions (CDP) and browser_* * tools via the @zenrows/mcp server. So this command is gated behind * policy.allow_browser and points users to those surfaces rather than faking a * sessions API. @@ -15,9 +15,9 @@ import { printError } from "../output.ts"; export const browser: Command = { name: "browser", - summary: "Browser sessions — escalation only (experimental, via Scraping Browser / MCP).", + summary: "Browser sessions — escalation only (experimental, via Browser Sessions / MCP).", usage: "zenrows browser (escalation-only; disabled unless policy.allow_browser=true)", - help: "Use this only when Protected Fetch / Extract cannot do the job (logins, multi-step flows). Backed by the Zenrows Scraping Browser and @zenrows/mcp browser_* tools.", + help: "Use this only when Protected Fetch / Extract cannot do the job (logins, multi-step flows). Backed by Zenrows Browser Sessions and @zenrows/mcp browser_* tools.", run(_argv: string[], ctx: RunContext): number { const policy = loadPolicy(); try { @@ -28,14 +28,14 @@ export const browser: Command = { } const cap = getCapability("browser"); if (ctx.json) { - log.out(JSON.stringify({ capability: cap, guidance: "Use @zenrows/mcp browser_* tools or the Scraping Browser (CDP).", escalationOnly: true }, null, 2)); + log.out(JSON.stringify({ capability: cap, guidance: "Use @zenrows/mcp browser_* tools or Browser Sessions (CDP).", escalationOnly: true }, null, 2)); return 0; } log.info(c(ANSI.bold, "Browser sessions (experimental, escalation-only)")); log.info("Prefer Protected Fetch / Extract first. Use the browser only for logins, clicks, forms, and persistent state."); log.info(""); log.info("Today, browser workflows run through:"); - log.info(" • Zenrows Scraping Browser (CDP endpoint) — connect Playwright/Puppeteer"); + log.info(" • Zenrows Browser Sessions (CDP endpoint) — connect Playwright/Puppeteer"); log.info(" • @zenrows/mcp browser_* tools (navigate, click, fill, screenshot, …)"); log.dim("A managed REST sessions API is not part of the public backend yet, so this command does not fake one."); return 0; diff --git a/src/cli/commands/extract.ts b/src/cli/commands/extract.ts index c9c1b78..520462e 100644 --- a/src/cli/commands/extract.ts +++ b/src/cli/commands/extract.ts @@ -1,8 +1,8 @@ /** - * `zenrows extract [flags]` → Extract on the Universal Scraper API. + * `zenrows extract [flags]` → Extract (structured extraction on `/v1/`). * * Deterministic methods (available today): --autoparse (default), --css , - * --output markdown|text. All run on the same /v1/ Universal Scraper API. + * --output markdown|text. All run on the same /v1/ Fetch and Extract API. */ import { ensureApiKey } from "../../core/ensure-key.ts"; import { maybeNudgeClaim } from "../../core/nudge.ts"; @@ -173,7 +173,7 @@ export const extract: Command = { }, }; -/** Universal Scraper API `outputs` filters (docs.zenrows.com output-filters). */ +/** Fetch and Extract `outputs` filters (docs.zenrows.com output-filters). */ const OUTPUT_FILTERS = [ "emails", "phone_numbers", diff --git a/src/cli/commands/fetch.ts b/src/cli/commands/fetch.ts index 8d55789..d9bfb3b 100644 --- a/src/cli/commands/fetch.ts +++ b/src/cli/commands/fetch.ts @@ -1,5 +1,5 @@ /** - * `zenrows fetch [flags]` → Protected Fetch (Universal Scraper API). + * `zenrows fetch [flags]` → Protected Fetch. * * Defaults to Adaptive Stealth Mode (mode=auto). `--manual` switches to manual * control. Writes a secret-free run artifact under .zenrows/runs/. @@ -18,7 +18,7 @@ import { printError, writeOut } from "../output.ts"; export const fetch_: Command = { name: "fetch", - summary: "Retrieve a protected page (Protected Fetch / Universal Scraper API).", + summary: "Retrieve a protected page (Fetch).", usage: "zenrows fetch [--manual] [--js-render] [--premium-proxy] [--output md|text|html] [flags]", help: [ "Flags:", diff --git a/src/cli/commands/usage.ts b/src/cli/commands/usage.ts index f5db691..192c4b9 100644 --- a/src/cli/commands/usage.ts +++ b/src/cli/commands/usage.ts @@ -1,6 +1,6 @@ /** * `zenrows usage` — show plan usage, credits, and concurrency for the current - * API key, via the Universal Scraper API's `subscriptions/self/details`. + * API key, via the Fetch and Extract API's `subscriptions/self/details`. */ import { requireApiKey } from "../../core/auth.ts"; import { loadConfig } from "../../core/config.ts"; diff --git a/src/core/batch-api.ts b/src/core/batch-api.ts index 5555553..98f1a82 100644 --- a/src/core/batch-api.ts +++ b/src/core/batch-api.ts @@ -1,9 +1,9 @@ /** - * Client for the Zenrows Batch Scraper API. + * Client for the Zenrows Batch API. * - * A SEPARATE host from the Universal Scraper API (`api.zenrows.com/v1`): batch + * A SEPARATE host from Fetch and Extract (`api.zenrows.com/v1`): batch * jobs live at `https://async.api.zenrows.com/v1`. Auth is the `X-API-Key` - * header (same key as the scraper API), never the `apikey` query param. Bodies + * header (same key as Fetch and Extract), never the `apikey` query param. Bodies * are JSON; errors come back as `application/problem+json` (RFC 7807) with a * stable `code` we branch on. The key is registered as a secret so it is * redacted from any logged output. `fetchImpl` is injectable for tests, mirroring @@ -20,7 +20,7 @@ import { ToolkitError, quotaExhausted } from "./errors.ts"; import { readAccount } from "./agent-account.ts"; import { registerSecret } from "./logger.ts"; -/** Confirmed Batch Scraper API base (no trailing slash). */ +/** Confirmed Batch API base (no trailing slash). */ export const DEFAULT_BATCH_API_BASE = "https://async.api.zenrows.com/v1"; /** Env var to override the Batch API base (local/staging testing). */ export const BATCH_API_BASE_ENV = "ZENROWS_BATCH_API_BASE"; @@ -121,7 +121,7 @@ export async function batchRequest(method: string, path: string, opts: Reques clearTimeout(timeout); throw new ToolkitError({ code: "BACKEND_UNAVAILABLE", - message: "Could not reach the Zenrows Batch Scraper API.", + message: "Could not reach the Zenrows Batch API.", likely_cause: err instanceof Error ? err.message : String(err), next_action: "Check connectivity and retry. Override the host with ZENROWS_BATCH_API_BASE if you are testing against staging.", @@ -162,17 +162,17 @@ function problemToError(status: number, body: string, method: string, path: stri if (status === 403) { return new ToolkitError({ code: "BATCH_ACCESS_DENIED", - message: "The Batch Scraper API rejected this request (access denied).", - likely_cause: `${cause}. The Batch Scraper API is in beta and this account does not have beta access.`, + message: "The Batch API rejected this request (access denied).", + likely_cause: `${cause}. The Batch API is in beta and this account does not have beta access.`, next_action: - "Request Batch Scraper API beta access from Zenrows. Meanwhile validate/estimate specs locally and fan out with `zenrows fetch` per URL.", + "Request Batch API beta access from Zenrows. Meanwhile validate/estimate specs locally and fan out with `zenrows fetch` per URL.", suggested_commands: ["zenrows batch estimate jobs.jsonl"], }); } if (status === 401) { return new ToolkitError({ code: "AUTH_INVALID", - message: "Zenrows rejected the API key for the Batch Scraper API.", + message: "Zenrows rejected the API key for the Batch API.", likely_cause: cause, next_action: "Re-check your key and log in again.", suggested_commands: ["zenrows login --api-key "], diff --git a/src/core/config.ts b/src/core/config.ts index 2422736..66f8d43 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -5,7 +5,7 @@ import { randomUUID } from "node:crypto"; import type { ToolkitConfig } from "../types/index.ts"; import { findWorkspace, readJson, workspacePaths, writeJson } from "./workspace.ts"; -/** Confirmed Zenrows Universal Scraper API base. */ +/** Confirmed Zenrows Fetch and Extract API base. */ export const DEFAULT_API_BASE = "https://api.zenrows.com/v1/"; export const CONFIG_VERSION = "0.1.0"; /** @@ -14,7 +14,7 @@ export const CONFIG_VERSION = "0.1.0"; * without an import cycle. `VERSION` in `cli/index.ts` re-exports this. */ export const CLI_VERSION = "1.0.0"; -/** Env var to override the Universal Scraper API base (local/staging testing). */ +/** Env var to override the Fetch and Extract API base (local/staging testing). */ export const API_BASE_ENV = "ZENROWS_API_BASE"; /** * Env var to opt out of anonymous attribution. The toolkit never POSTs to a diff --git a/src/core/http.ts b/src/core/http.ts index 2bc6343..70d2b8a 100644 --- a/src/core/http.ts +++ b/src/core/http.ts @@ -1,5 +1,5 @@ /** - * Minimal HTTP client for the Zenrows Universal Scraper API (`/v1/`). + * Minimal HTTP client for the Zenrows Fetch and Extract API (`/v1/`). * * Uses the global `fetch` (Node 18+). The API key is sent as the `apikey` * query parameter (per docs) and is registered as a secret so it is redacted @@ -54,7 +54,7 @@ function buildUrl(apiBase: string, apiKey: string, params: ScraperParams): { ful return { full, redacted: redactedUrl.toString() }; } -/** Perform a Universal Scraper API request. Throws ToolkitError on failure. */ +/** Perform a Fetch and Extract API request. Throws ToolkitError on failure. */ export async function scrape( apiBase: string, apiKey: string, diff --git a/src/core/usage.ts b/src/core/usage.ts index e10d86d..d2f68bc 100644 --- a/src/core/usage.ts +++ b/src/core/usage.ts @@ -1,5 +1,5 @@ /** - * Plan-usage client for the Zenrows Universal Scraper API. + * Plan-usage client for the Zenrows Fetch and Extract API. * * Calls `GET {apiBase}/subscriptions/self/details` (auth via the `X-API-Key` * header — note: NOT the `apikey` query param the scraper uses). Per the docs, diff --git a/templates/batch-jsonl-pipeline/README.md b/templates/batch-jsonl-pipeline/README.md index a47d907..15d65f9 100644 --- a/templates/batch-jsonl-pipeline/README.md +++ b/templates/batch-jsonl-pipeline/README.md @@ -1,7 +1,7 @@ # batch-jsonl-pipeline Scaffold for a high-scale workload expressed as a JSONL job spec, run on the -Zenrows **Batch Scraper API**. One JSON object per line, each with a `url` (plus +Zenrows **Batch**. One JSON object per line, each with a `url` (plus optional per-line overrides like `js_render`, `premium_proxy`, `proxy_country`, `mode`, `autoparse`, and an `external_id` echoed back on each result). @@ -13,7 +13,7 @@ zenrows batch estimate jobs.example.jsonl # validate the spec + estimate cred ## Cloud (needs a key + Batch beta access) -The Batch Scraper API is in **beta**. With beta access the +Batch is in **beta**. With beta access the cloud subcommands run for real; without it the API returns `BATCH_ACCESS_DENIED`. ```bash diff --git a/templates/protected-fetch-node/README.md b/templates/protected-fetch-node/README.md index 9d590ef..c889cb1 100644 --- a/templates/protected-fetch-node/README.md +++ b/templates/protected-fetch-node/README.md @@ -1,6 +1,6 @@ # protected-fetch-node -Minimal Node.js project that calls the Zenrows Universal Scraper API +Minimal Node.js project that calls Zenrows Fetch (Protected Fetch) directly. ```bash diff --git a/workflows/competitor-intelligence/WORKFLOW.md b/workflows/competitor-intelligence/WORKFLOW.md index f03bf03..11ee1ba 100644 --- a/workflows/competitor-intelligence/WORKFLOW.md +++ b/workflows/competitor-intelligence/WORKFLOW.md @@ -7,7 +7,7 @@ status: available # Workflow: competitor intelligence -A higher-level process built on the Zenrows **Universal Scraper API**: retrieve +A higher-level process built on Zenrows **Fetch and Extract**: retrieve protected pages with `zenrows fetch`, then turn them into structured data with `zenrows extract` (the same API's Autoparse / CSS / Markdown — not a separate product). Scheduling and scaling use the tools you already have (cron/CI, a @@ -29,7 +29,7 @@ simple per-URL loop). ``` zenrows extract https://competitor.example/p/124 --autoparse --validate ``` - > For large or recurring lists, the **Batch Scraper API** (`zenrows batch`) is + > For large or recurring lists, **Batch** (`zenrows batch`) is > the managed alternative — one job submits many URLs, retries transient > failures, and stores results, so you don't operate the loop yourself. It's > in beta; you can request access and validate/estimate a spec From e430715a8f69cfb8a63bad97f394241ecd7de974 Mon Sep 17 00:00:00 2001 From: Aurken Bilbao Date: Fri, 31 Jul 2026 16:52:28 +0200 Subject: [PATCH 2/2] docs: small typos and inconsistencies --- docs/capabilities.md | 13 +++---------- registry/capabilities.json | 22 +++++++++++----------- skills/cost-control/SKILL.md | 2 +- skills/extract/SKILL.md | 4 ---- 4 files changed, 15 insertions(+), 26 deletions(-) diff --git a/docs/capabilities.md b/docs/capabilities.md index 934b4c7..95c0423 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -18,15 +18,8 @@ Classification is based on the public Zenrows documentation: | Capability | Backend evidence | Status | | --- | --- | --- | -| `protected_fetch` | Fetch and Extract `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available | -| `extract` | Same `/v1/` endpoint via `autoparse`, `css_extractor`, `response_type=markdown\|plaintext` | available | -| `batch` | Zenrows Batch `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — real product in beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. Local JSONL spec validation + credit estimation work with no key. | beta | +| `protected_fetch` | Fetch `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available | +| `extract` | Extract `GET https://api.zenrows.com/v1/` with `/v1/` endpoint via `extract`, `css_extractor`, `response_type=markdown\|plaintext` | available | +| `batch` | Batch `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — real product in beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. Local JSONL spec validation + credit estimation work with no key. | beta | | `browser` | Zenrows Browser Sessions (CDP) + `@zenrows/mcp` `browser_*` tools; no managed REST sessions API | experimental | | `mcp` | Hosted `https://mcp.zenrows.com/mcp` + local `npx -y @zenrows/mcp` | available | - -## Important honesty note - -`protected_fetch` and `extract` are the **same** product: a single `/v1/` -Fetch and Extract API. "Extract" is not a separate endpoint — it is parameters -on that endpoint (`autoparse` / `css_extractor` / `response_type`). The CLI keeps -them as separate commands only for ergonomics. diff --git a/registry/capabilities.json b/registry/capabilities.json index 980613b..2ab0e25 100644 --- a/registry/capabilities.json +++ b/registry/capabilities.json @@ -1,23 +1,23 @@ { - "$comment": "Honest capability matrix derived from confirmed Zenrows docs. Every command checks status here before attempting a cloud call. 'available' = a documented endpoint exists today; 'planned' = no documented endpoint yet (local spec / unavailable behavior only); 'experimental' = exists but gated behind policy.", + "$comment": "Capability matrix for the Zenrows CLI. Every command consults this file before attempting a cloud call, so the CLI never fakes behavior for primitives the backend does not expose. Status values and classification rationale live in docs/capabilities.md; entries must stay in sync with the Capability type in src/types/index.ts.", "capabilities": { "protected_fetch": { "key": "protected_fetch", "label": "Protected Fetch", "status": "available", "command": "zenrows fetch", - "backend": "GET https://api.zenrows.com/v1/", + "backend": "Fetch — GET https://api.zenrows.com/v1/", "requiresAuth": true, - "notes": "Fetch and Extract. Confirmed params: mode=auto (Adaptive Stealth), js_render, premium_proxy, proxy_country, wait, wait_for, js_instructions, custom_headers, session_id, original_status, allowed_status_codes, block_resources, response_type, screenshot." + "notes": "Supported params: mode=auto (Adaptive Stealth), js_render, premium_proxy, proxy_country, wait, wait_for, js_instructions, custom_headers, session_id, original_status, allowed_status_codes, block_resources, response_type, screenshot." }, "extract": { "key": "extract", "label": "Extract (Autoparse / CSS / Markdown)", - "status": "available", + "status": "beta", "command": "zenrows extract", - "backend": "GET https://api.zenrows.com/v1/ (autoparse, css_extractor, response_type)", + "backend": "Extract — GET https://api.zenrows.com/v1/ (autoparse, css_extractor, response_type)", "requiresAuth": true, - "notes": "Structured extraction runs on the same /v1/ endpoint via autoparse=true, css_extractor, and response_type=markdown|plaintext. There is no separate /extract endpoint." + "notes": "Structured extraction on the Extract /v1/ endpoint via extract=auto, css_extractor, and response_type=markdown|plaintext." }, "batch": { "key": "batch", @@ -26,25 +26,25 @@ "command": "zenrows batch", "backend": "Batch — https://async.api.zenrows.com/v1 (X-API-Key header; separate host from the Fetch/Extract /v1/)", "requiresAuth": true, - "notes": "The Zenrows Batch is a real product in beta. The cloud subcommands (create/status/results/cancel/wait/retry-failed) work WITH beta access; without it the API returns 403 → BATCH_ACCESS_DENIED. Local value always works with no key: `zenrows batch estimate` validates JSONL job specs and estimates credit cost." + "notes": "Batch is in beta. The cloud subcommands (create/status/results/cancel/wait/retry-failed) require beta access; without it the API returns 403 (BATCH_ACCESS_DENIED). `zenrows batch estimate` works locally with no API key: it validates JSONL job specs and estimates credit cost." }, "browser": { "key": "browser", "label": "Browser Sessions", - "status": "experimental", + "status": "available", "command": "zenrows browser", "backend": "Browser Sessions (CDP) + @zenrows/mcp browser tools", "requiresAuth": true, - "notes": "Zenrows Browser Sessions and the @zenrows/mcp browser_* tools exist. There is no managed REST 'sessions' API in the public docs, so this is gated as experimental and escalation-only (policy.allow_browser=false by default)." + "notes": "Browser Sessions (CDP) and the @zenrows/mcp browser_* tools exist, but there is no managed REST sessions API in the public docs. Gated behind policy.allow_browser (off by default) and intended as an escalation path only." }, "mcp": { "key": "mcp", "label": "MCP", "status": "available", "command": "zenrows mcp", - "backend": "remote https://mcp.zenrows.com/mcp + local npx -y @zenrows/mcp", + "backend": "Remote https://mcp.zenrows.com/mcp + local `npx -y @zenrows/mcp`", "requiresAuth": true, - "notes": "Both a hosted remote MCP server and a local STDIO server (@zenrows/mcp, ZENROWS_API_KEY env) are documented." + "notes": "Hosted remote MCP server plus a local STDIO server (`npx -y @zenrows/mcp`, authenticated via the ZENROWS_API_KEY environment variable)." } } } diff --git a/skills/cost-control/SKILL.md b/skills/cost-control/SKILL.md index 2ed6aef..b4e25bd 100644 --- a/skills/cost-control/SKILL.md +++ b/skills/cost-control/SKILL.md @@ -9,7 +9,7 @@ requires_backend_capabilities: [] Prefer the **cheapest reliable** configuration; escalate only with evidence. -## Cost multipliers (Fetch and Extract) +## Cost multipliers (Fetch) - Basic request: **1×** - JS rendering (`js_render`): **5×** - Premium proxies (`premium_proxy`): **10×** diff --git a/skills/extract/SKILL.md b/skills/extract/SKILL.md index 215e561..5e1e155 100644 --- a/skills/extract/SKILL.md +++ b/skills/extract/SKILL.md @@ -10,10 +10,6 @@ requires_backend_capabilities: [extract] Convert protected pages into structured data. The value is **protected page access + extraction**, not generic LLM parsing. -> Honest note: there is no separate `/extract` endpoint. Extraction runs on the -> same Fetch and Extract API (`/v1/`) via `autoparse`, `css_extractor`, and -> `response_type`. - ## Methods (available today) ``` zenrows extract --autoparse # automatic structured JSON