chore!: unify argument validation to zod by vladfrangu · Pull Request #3935 · apify/crawlee

vladfrangu · 2026-07-30T11:35:27Z

Replaces the remaining ow-based argument validation with zod across all packages and reworks how validation results are consumed and reported.

Closes #3716

What changed

ow is gone — every argument check now goes through parseArgument(value, schema, label?) from @crawlee/utils, backed by shared zod schemas (schemas, exported via @crawlee/utils/internal). The @sapphire/shapeshift checks in @crawlee/fs-storage were converted too, so a single validation library remains.
Parse results are used everywhere — option defaults moved from destructuring into the schemas (.default(...)), and call sites destructure the typed parse result. parseArgument returns TValue & z.output<TSchema>, so call sites keep their declared TS types while gaining the defaults.
Schemas are built once — all per-call schemas are hoisted to module scope; crawler/launcher classes build their strict options schema once as a static optionsSchema next to optionsShape. The urlPatternSchema for include/exclude lives in enqueue_links/shared.ts, next to the type it validates.
Specific validators instead of anyObject — class-typed options use z.instanceof(...) (BaseHttpClient, Configuration, EventManager), interface-typed ones use duck-typed objectWithKeys validators (storageBackend, requestManager, logger, …), and element-typed arrays use the new schemas.arrayOf(item, 'numbers').

Error messages

ArgumentValidationError (replacing ow's ArgumentError) renders one line per issue: the expected type, the received type and value folded into one clause, the offending field path, and the validated interface:

// v3 (ow) — first issue only
Expected property `maxRequestRetries` to be of type `number` but received type `string` in object `HttpCrawlerOptions`

// v4 (zod) — every issue, one line each
Invalid input: expected number, received the string `many` at `maxRequestRetries` in `HttpCrawlerOptions`
Invalid input: expected an array of numbers, received the number `500` at `additionalHttpErrorStatusCodes` in `HttpCrawlerOptions`
Invalid input: expected boolean, received the string `yes` at `retryOnBlocked` in `HttpCrawlerOptions`

Details worth knowing:

Union failures expand into one line per failed arm (zod's own message is a bare "Invalid input").
NaN is named as itself, an empty string renders as '', and arrays name their element type (expected an array of URL patterns) — none of which ow or stock zod reported.
new Request('https://…') gets a targeted hint pointing at the { url } object form.
For programmatic handling, the error exposes zod's structured output: error.issues and the raw ZodError as a typed cause.

The migration is documented in the v4 upgrading guide (docs/upgrading/upgrading_v4.md), including a rename-cheat-sheet entry.

Notes

Custom HTTP clients must now extend BaseHttpClient from @crawlee/http-client rather than just implementing the interface (all shipped clients already do; LazyDefaultHttpClient was converted). Same applies to test mocks — Object.create(BaseHttpClient.prototype) works.
One caveat of consuming parse results: zod object schemas return a pruned plain copy, so options holding class instances are validated with passthrough schemas (z.custom-based) to keep their prototypes — there are comments at the relevant schemas.
Fixes a few latent gaps surfaced along the way: Request.state now accepts RequestState.SKIPPED (validated via z.enum(RequestState)), and the publish-time catalog inlining covers optionalDependencies.
ArgumentValidationError and its formatter are intentionally kept close to the copy in refactor!: validate arguments with zod instead of ow apify-client-js#986 — a follow-up may extract them into a shared package.

…gument results Message format matches #3716: the zod issue message plus the offending field path and the received value. parseArgument drops the label argument and returns the parsed (default-applied) value, typed as the input intersected with the schema output.

…nce validators Options are now destructured from the parse result with defaults declared in the schemas, all per-call schemas are hoisted to module scope, and instance-like options are validated with specific validators (instanceof for classes, objectWithKeys for interfaces) instead of anyObject.

Crawler and launcher classes build their strict options schema once as a static (optionsSchema) and consume the parse result. Adds missing optionsShape entries (requestManager, ignoreIframes, ignoreShadowRoots) and makes LazyDefaultHttpClient extend BaseHttpClient so it passes the new instanceof validation.

Consumers were mixing the class from @crawlee/http-client with the same-named interface from @crawlee/types, producing aliased duplicate imports (BaseHttpClient_2) in the public API reports. Since httpClient options are now validated with instanceof, the class is the actual contract — import it everywhere. Only the definition site keeps the interface (the class implements it).

…tionError Matches the formatter in apify-client-js#986 - a union's own message is a bare "Invalid input"; the offending arm and field only appear in the nested errors.

- validate Request.state with z.enum(RequestState), so SKIPPED (7) is accepted - hoist the shared urlPatternSchema into enqueue_links/shared.ts (was duplicated 4x) - move stranded imports above the schema consts in concurrency_system.ts - use safeParse instead of parse + try/catch for schema defaults in Configuration - inline catalog versions in optionalDependencies too when publishing

- parseArgument/ArgumentValidationError accept an optional label naming the validated interface, appended to every error line the way ow's errors ended with "in object `HttpCrawlerOptions`"; wired up at all option-bag call sites (crawlers, launchers, enqueueLinks, storages) - bare custom-schema failures now name the received type before the value ("expected number, received string at `x`, got `3`"), skipped when the two would read the same (NaN) - new Request(...) rejects a bare URL string with a hint pointing at the `{ url }` object form, and labels its errors with RequestOptions

"expected number, received the string `3` at `maxRequestRetries`" instead of a dangling ", got `3`" after the location. Messages that never name a received type (regex, min/max, enums) keep the plain suffix; NaN is named as itself and an empty string is made visible.

`schemas.arrayOf(item, 'numbers')` reports a top-level type miss as "expected an array of numbers, received the number `500`" instead of zod's bare "expected array"; per-element messages are unchanged. Adopted at every element-typed array option (status codes, urls, include/exclude patterns, mime types, domains).

…ause

… errors Also drop dead launchContext destructure defaults — BrowserCrawler's schema already defaults it to an empty object.

…nt instance

…ages

…lidation error

barjin

Thank you @vladfrangu !

The diff looks like what I'd expect it to look like, I don't see much else to review 😅 If I understood correctly, this went through rounds of LLM reviews, right? Since the CI is green, I say let's merge and see what feedback we get. The longer we wait, the more conflicts there will be.

barjin · 2026-08-12T13:25:25Z

+        launcher: schemas.anyObject.optional(),
    };

+    protected static override optionsSchema = z.strictObject(PlaywrightCrawler.optionsShape);


According to my findings on another issue (#3549 ), this will run during the import, which can add ~50ms to the @crawlee/playwright import (and other packages that use this pattern). Perhaps we can lazy-initialize this?

By no means is this something we have to solve now, though.

Make an issue to track this and assign it to me please 🙏

your wish is my command #4025

janbuchar

What @barjin said

Replaces the remaining `ow`-based argument validation with `zod` across all packages and reworks how validation results are consumed and reported. Closes #3716 - **`ow` is gone** — every argument check now goes through `parseArgument(value, schema, label?)` from `@crawlee/utils`, backed by shared zod schemas (`schemas`, exported via `@crawlee/utils/internal`). The `@sapphire/shapeshift` checks in `@crawlee/fs-storage` were converted too, so a single validation library remains. - **Parse results are used everywhere** — option defaults moved from destructuring into the schemas (`.default(...)`), and call sites destructure the typed parse result. `parseArgument` returns `TValue & z.output<TSchema>`, so call sites keep their declared TS types while gaining the defaults. - **Schemas are built once** — all per-call schemas are hoisted to module scope; crawler/launcher classes build their strict options schema once as a `static optionsSchema` next to `optionsShape`. The `urlPatternSchema` for `include`/`exclude` lives in `enqueue_links/shared.ts`, next to the type it validates. - **Specific validators instead of `anyObject`** — class-typed options use `z.instanceof(...)` (`BaseHttpClient`, `Configuration`, `EventManager`), interface-typed ones use duck-typed `objectWithKeys` validators (`storageBackend`, `requestManager`, `logger`, …), and element-typed arrays use the new `schemas.arrayOf(item, 'numbers')`. `ArgumentValidationError` (replacing ow's `ArgumentError`) renders one line per issue: the expected type, the received type and value folded into one clause, the offending field path, and the validated interface: ```text // v3 (ow) — first issue only Expected property `maxRequestRetries` to be of type `number` but received type `string` in object `HttpCrawlerOptions` // v4 (zod) — every issue, one line each Invalid input: expected number, received the string `many` at `maxRequestRetries` in `HttpCrawlerOptions` Invalid input: expected an array of numbers, received the number `500` at `additionalHttpErrorStatusCodes` in `HttpCrawlerOptions` Invalid input: expected boolean, received the string `yes` at `retryOnBlocked` in `HttpCrawlerOptions` ``` Details worth knowing: - Union failures expand into one line per failed arm (zod's own message is a bare "Invalid input"). - `NaN` is named as itself, an empty string renders as `''`, and arrays name their element type (``expected an array of URL patterns``) — none of which ow or stock zod reported. - `new Request('https://…')` gets a targeted hint pointing at the `{ url }` object form. - For programmatic handling, the error exposes zod's structured output: `error.issues` and the raw `ZodError` as a typed `cause`. The migration is documented in the v4 upgrading guide (`docs/upgrading/upgrading_v4.md`), including a rename-cheat-sheet entry. - Custom HTTP clients must now **extend `BaseHttpClient`** from `@crawlee/http-client` rather than just implementing the interface (all shipped clients already do; `LazyDefaultHttpClient` was converted). Same applies to test mocks — `Object.create(BaseHttpClient.prototype)` works. - One caveat of consuming parse results: zod object schemas return a pruned plain copy, so options holding class instances are validated with passthrough schemas (`z.custom`-based) to keep their prototypes — there are comments at the relevant schemas. - Fixes a few latent gaps surfaced along the way: `Request.state` now accepts `RequestState.SKIPPED` (validated via `z.enum(RequestState)`), and the publish-time catalog inlining covers `optionalDependencies`. - `ArgumentValidationError` and its formatter are intentionally kept close to the copy in apify/apify-client-js#986 — a follow-up may extract them into a shared package. --------- Co-authored-by: Martin Adámek <banan23@gmail.com>

…atalog dependency Two upstream v4 changes landed after the last E2E run and broke the suite: - the enqueueLinks split (#4010) changed the context helper's return value to the addRequestsBatched result, so the *-enqueue-links fixtures now assert on `addedRequests` being empty instead of deep-equality with the old shape - the zod validation unification (#3935) introduced a `catalog:` dependency, which npm cannot resolve when the platform builds the actor image; the E2E package-copy step now rewrites catalog deps to their pinned versions from pnpm-workspace.yaml, the same way it already rewrites `workspace:` deps

Replaces the remaining `ow`-based argument validation with `zod` across all packages and reworks how validation results are consumed and reported. Closes #3716 - **`ow` is gone** — every argument check now goes through `parseArgument(value, schema, label?)` from `@crawlee/utils`, backed by shared zod schemas (`schemas`, exported via `@crawlee/utils/internal`). The `@sapphire/shapeshift` checks in `@crawlee/fs-storage` were converted too, so a single validation library remains. - **Parse results are used everywhere** — option defaults moved from destructuring into the schemas (`.default(...)`), and call sites destructure the typed parse result. `parseArgument` returns `TValue & z.output<TSchema>`, so call sites keep their declared TS types while gaining the defaults. - **Schemas are built once** — all per-call schemas are hoisted to module scope; crawler/launcher classes build their strict options schema once as a `static optionsSchema` next to `optionsShape`. The `urlPatternSchema` for `include`/`exclude` lives in `enqueue_links/shared.ts`, next to the type it validates. - **Specific validators instead of `anyObject`** — class-typed options use `z.instanceof(...)` (`BaseHttpClient`, `Configuration`, `EventManager`), interface-typed ones use duck-typed `objectWithKeys` validators (`storageBackend`, `requestManager`, `logger`, …), and element-typed arrays use the new `schemas.arrayOf(item, 'numbers')`. `ArgumentValidationError` (replacing ow's `ArgumentError`) renders one line per issue: the expected type, the received type and value folded into one clause, the offending field path, and the validated interface: ```text // v3 (ow) — first issue only Expected property `maxRequestRetries` to be of type `number` but received type `string` in object `HttpCrawlerOptions` // v4 (zod) — every issue, one line each Invalid input: expected number, received the string `many` at `maxRequestRetries` in `HttpCrawlerOptions` Invalid input: expected an array of numbers, received the number `500` at `additionalHttpErrorStatusCodes` in `HttpCrawlerOptions` Invalid input: expected boolean, received the string `yes` at `retryOnBlocked` in `HttpCrawlerOptions` ``` Details worth knowing: - Union failures expand into one line per failed arm (zod's own message is a bare "Invalid input"). - `NaN` is named as itself, an empty string renders as `''`, and arrays name their element type (``expected an array of URL patterns``) — none of which ow or stock zod reported. - `new Request('https://…')` gets a targeted hint pointing at the `{ url }` object form. - For programmatic handling, the error exposes zod's structured output: `error.issues` and the raw `ZodError` as a typed `cause`. The migration is documented in the v4 upgrading guide (`docs/upgrading/upgrading_v4.md`), including a rename-cheat-sheet entry. - Custom HTTP clients must now **extend `BaseHttpClient`** from `@crawlee/http-client` rather than just implementing the interface (all shipped clients already do; `LazyDefaultHttpClient` was converted). Same applies to test mocks — `Object.create(BaseHttpClient.prototype)` works. - One caveat of consuming parse results: zod object schemas return a pruned plain copy, so options holding class instances are validated with passthrough schemas (`z.custom`-based) to keep their prototypes — there are comments at the relevant schemas. - Fixes a few latent gaps surfaced along the way: `Request.state` now accepts `RequestState.SKIPPED` (validated via `z.enum(RequestState)`), and the publish-time catalog inlining covers `optionalDependencies`. - `ArgumentValidationError` and its formatter are intentionally kept close to the copy in apify/apify-client-js#986 — a follow-up may extract them into a shared package. --------- Co-authored-by: Martin Adámek <banan23@gmail.com>

…atalog dependency Two upstream v4 changes landed after the last E2E run and broke the suite: - the enqueueLinks split (#4010) changed the context helper's return value to the addRequestsBatched result, so the *-enqueue-links fixtures now assert on `addedRequests` being empty instead of deep-equality with the old shape - the zod validation unification (#3935) introduced a `catalog:` dependency, which npm cannot resolve when the platform builds the actor image; the E2E package-copy step now rewrites catalog deps to their pinned versions from pnpm-workspace.yaml, the same way it already rewrites `workspace:` deps

vladfrangu changed the base branch from master to v4 July 30, 2026 11:36

vladfrangu force-pushed the chore/unify-to-zod branch 2 times, most recently from 48101f6 to acb4aab Compare July 30, 2026 13:10

vladfrangu added the t-tooling Issues with this label are in the ownership of the tooling team. label Aug 4, 2026

vladfrangu force-pushed the chore/unify-to-zod branch 2 times, most recently from e9dee4f to 5d71608 Compare August 11, 2026 11:41

vladfrangu marked this pull request as ready for review August 11, 2026 13:35

vladfrangu requested review from B4nan, barjin and janbuchar August 11, 2026 16:12

vladfrangu added 20 commits August 12, 2026 12:55

chore: unify argument validation from ow to zod

25ef997

chore: regenerate TypeScript API reports

ce8b83e

test: adapt storage test mocks to instance-based validation

97cf3b6

feat(utils): expand union issues into per-arm lines in ArgumentValida…

fc90961

…tionError Matches the formatter in apify-client-js#986 - a union's own message is a bare "Invalid input"; the offending arm and field only appear in the nested errors.

docs: document the zod validation errors in the v4 upgrading guide

29f85e4

test: align RequestList message assertion with the folded received cl…

50cc9f7

…ause

fix: restore options labels in Playwright/Puppeteer/Stagehand crawler…

9b7c3cd

… errors Also drop dead launchContext destructure defaults — BrowserCrawler's schema already defaults it to an empty object.

fix(jsdom): validate JSDOMCrawler options like the sibling crawlers

ce709b8

fix(core): validate SitemapRequestLoader httpClient as a BaseHttpClie…

45fd102

…nt instance

feat(utils): fold the received value into plainObject/typedArray mess…

debc549

…ages

fix(utils): elide long received strings in validation error messages

74d3e78

fix(playwright): name transactionalStorage in the adaptive crawler va…

0110e6a

…lidation error

vladfrangu added 4 commits August 12, 2026 12:56

chore: parse catalog versions containing spaces in the copy script

54b915d

chore: regenerate jsdom public API report

6f7916a

test: expect the elided long key in the KVS invalid-key error

29f9e2b

chore: adapt to the ignoreTlsErrors rename after rebasing on v4

eeb92da

vladfrangu force-pushed the chore/unify-to-zod branch from 6e8f04e to eeb92da Compare August 12, 2026 10:01

barjin approved these changes Aug 12, 2026

View reviewed changes

janbuchar approved these changes Aug 12, 2026

View reviewed changes

barjin mentioned this pull request Aug 12, 2026

Minimize import-time zod schema building #4025

Open

B4nan changed the title ~~chore: unify argument validation to zod~~ chore!: unify argument validation to zod Aug 12, 2026

chore: merge v4 into chore/unify-to-zod

f51a8dc

B4nan approved these changes Aug 12, 2026

View reviewed changes

B4nan merged commit 6f4a5ca into v4 Aug 12, 2026
8 checks passed

B4nan deleted the chore/unify-to-zod branch August 12, 2026 14:59

B4nan mentioned this pull request Aug 12, 2026

Consolidate runtime validation #3112

Closed

Navigation Menu

Uh oh!

chore!: unify argument validation to zod - #3935

chore!: unify argument validation to zod#3935
B4nan merged 25 commits into
v4from
chore/unify-to-zod

vladfrangu commented Jul 30, 2026 •

edited

Loading

Uh oh!

barjin left a comment •

edited

Loading

Uh oh!

barjin Aug 12, 2026 •

edited

Loading

Uh oh!

vladfrangu Aug 12, 2026

Uh oh!

barjin Aug 12, 2026

Uh oh!

janbuchar left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

vladfrangu commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Error messages

Notes

Uh oh!

barjin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

barjin Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vladfrangu Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

barjin Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

janbuchar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vladfrangu commented Jul 30, 2026 •

edited

Loading

barjin left a comment •

edited

Loading

barjin Aug 12, 2026 •

edited

Loading