feat: replace idcac-playwright with @duckduckgo/autoconsent - #4044
Draft
barjin wants to merge 216 commits into
Draft
feat: replace idcac-playwright with @duckduckgo/autoconsent#4044barjin wants to merge 216 commits into
barjin wants to merge 216 commits into
Conversation
BREAKING CHANGE: The project is now native ESM without a CJS alternative. This is fine since all supported node versions allow `require(esm)`. Also all the dependencies are updated to the latest versions, including cheerio v1.
BREAKING CHANGE: The crawler following options are removed: - `handleRequestFunction` -> `requestHandler` - `handlePageFunction` -> `requestHandler` - `handleRequestTimeoutSecs` -> `requestHandlerTimeoutSecs` - `handleFailedRequestFunction` -> `failedRequestHandler`
BREAKING CHANGE: The crawling context no longer includes the `Error` object for failed requests. Use the second parameter of the `errorHandler` or `failedRequestHandler` callbacks to access the error. Previously, the crawling context extended a `Record` type, allowing to access any property. This was changed to a strict type, which means that you can only access properties that are defined in the context.
….retireOnBlockedStatusCodes` BREAKING CHANGE: `additionalBlockedStatusCodes` parameter of `Session.retireOnBlockedStatusCodes` method is removed. Use the `blockedStatusCodes` crawler option instead.
….retireOnBlockedStatusCodes` BREAKING CHANGE: `additionalBlockedStatusCodes` parameter of `Session.retireOnBlockedStatusCodes` method is removed. Use the `blockedStatusCodes` crawler option instead.
also tries to bump better-sqlite3 to latest version to have prebuilds for node 22
- closes #2479 - closes #3106 - closes #3107 - closes #3078 In my opinion, it makes a lot of sense to do the remaining changes in a separate PR. - [x] Introduce a `ContextPipeline` abstraction - [x] Update crawlers to use it - [x] Make sure that existing tests pass - [ ] Refine the `ContextPipeline.compose` signature and the semantics of `BasicCrawlerOptions.contextPipelineEnhancer` to maximize DX - [x] Write tests for the `contextPipelineEnhancer` - [x] Resolve added TODO comments (fix immediately or make issues) - [ ] Update documentation The `context-pipeline` branch introduces a fundamental architectural change to how Crawlee crawlers build and enhance the crawling context passed to request handlers. The core motivation is to fix the composition and extensibility nightmare in the current crawler hierarchy. 1. **Rigid inheritance hierarchy**: Crawlers were stuck in a brittle inheritance chain where each layer manipulated the context object while assuming that it already satisfied its final type. Multiple overrides of `BasicCrawler` lifecycle methods made the execution flow even harder to follow. 2. **Context enhancement via monkey-patching**: Manual property assignment (`crawlingContext.page = page`, `crawlingContext.$ = $`) scattered everywhere. It was a mess to follow and impossible to reason about. 3. **Cleanup coordination**: Resource cleanup was handled by separate `_cleanupContext` methods that were not co-located with the initialization. 4. **Extension mechanism was broken**: The `CrawlerExtension.use()` API tried to let you extend crawlers (the ones based on `HttpCrawler`) by overwriting properties - completely type-unsafe and fragile as hell. Introduces `ContextPipeline` - a **middleware-based composition pattern** where: - Each crawler layer defines how it enhances the context through explicit `action` functions - Cleanup logic is co-located with initialization via optional `cleanup` functions - Type safety is maintained through TypeScript generics that track context transformations - The pipeline executes middleware sequentially with proper error handling and guaranteed cleanup Declarative middleware composition with co-located cleanup: ```typescript contextPipeline.compose({ action: async (context) => ({ page, $ }), cleanup: async (context) => { await page.close(); } }) ``` The `ContextPipeline<TBase, TFinal>` tracks type transformations through the chain: ```typescript ContextPipeline<CrawlingContext, CrawlingContext> .compose<{ page: Page }>(...) // ContextPipeline<CrawlingContext, CrawlingContext & { page: Page }> .compose<{ $: CheerioAPI }>(...) // ContextPipeline<CrawlingContext, CrawlingContext & { page: Page, $: CheerioAPI }> ``` The `CrawlerExtension.use()` is gone. New approach via `contextPipelineEnhancer`: ```typescript new BasicCrawler({ contextPipelineEnhancer: (pipeline) => pipeline.compose({ action: async (context) => ({ myCustomProp: ... }) }) }) ``` The current way to express a context pipeline middleware has some shortcomings (`ContextPipeline.compose`, `BasicCrawlerOptions.contextPipelineEnhancer`). I suggest resolving this in another PR. For most legitimate use cases, this should be non-breaking. Those who extend the Crawler classes in non-trivial ways may need to adjust their code though - the non-public interface of `BasicCrawler` and `HttpCrawler` changed quite a bit. The pipeline uses `Object.defineProperties` for each middleware. Is this a serious performance consideration? --------- Co-authored-by: Martin Adámek <banan23@gmail.com>
Extracts `ProxyConfiguration` to `BasicCrawler` (related to discussion under #2917). Pass the `ProxyConfiguration` instance to the `SessionPool` for new `Session` object creation. Store and read the `ProxyInfo` from the `Session` instance instead of calling the `ProxyConfiguration` methods in the crawlers. closes #3198
Phasing out `got-scraping`-specific interfaces in favour of native `fetch` API. Related to #3071
Fixes build toolchain errors caused by the recent rebase onto the current `master` ([more details here](https://apify.slack.com/archives/C02JQSN79V4/p1764373034961859)). The largest thing is probably updating the dependency versions in `package.json` - if `turborepo` doesn't find the matching version in the local workspace, it will build against the package pulled from `npm` (which doesn't match the v4 API at this point).
Related to #3275 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Fixes the response header handling in `GotScrapingHttpClient` (`got-scraping` headers contain unexpected `Symbol`s and HTTP2 pseudoheaders). Fixes omission from one of the previous commits - `GotScrapingHttpClient.stream` now uses proxy correctly again. Closes #2917
Gets the E2E test suite running against v4. The suite hadn't been run since the v4 rewrite and everything failed on startup. After these changes the MEMORY run passes locally end to end, and getting there surfaced a few real regressions in the packages themselves. - `LinkeDOMCrawler`'s `enqueueLinks` helper referenced the global `document` (which doesn't exist in Node) instead of the parsed window, so every call crashed at runtime. - `ErrorSnapshotter.saveHTMLSnapshot()` returned the record key with a v3-style `.html` suffix, so the follow-up `getPublicUrl()` lookup missed and `firstErrorHtmlUrl` never made it into the crawler statistics. - `JSDOMCrawlingContext`/`LinkeDOMCrawlingContext` didn't override `enqueueLinks`, exposing the strict urls-required signature even though the runtime helper extracts URLs from the parsed document. - `LinkeDOMCrawler` can now be constructed without arguments, like the other crawlers. - Bumped the pinned `apify` SDK to 4.0.0-beta.22 (beta.19 imports `snakeCaseToCamelCase` from `@crawlee/utils`, which no longer exists there). - Adapted `tools.mjs` to the fs-storage on-disk layout (extensionless key-value records; the short-lived `__default__` directory alias it originally targeted was a bug, fixed in #4013) and to the `@crawlee/utils` exports split. - Migrated test actors to the v4 APIs: the `logger` option with `ApifyLogAdapter` instead of `log`, hooks reading `gotoOptions` from the crawling context, `session.setCookie()`, a custom `SessionPool` instead of `sessionPoolOptions`, the WHATWG `Response` returned by `sendRequest`, `registerDeferredCleanup` for dataset writes that must survive a throwing handler, and explicit enqueue strategies now that `include` globs are ANDed with the default same-hostname strategy. - The ignore-ssl test now configures TLS verification on the http client, because the crawler-level `ignoreSslErrors` option is not wired to the default client in v4. That dangling option deserves a separate fix or removal, since it currently does nothing. - The impit test pins session fingerprints, since the random default fingerprint overrides the client's browser impersonation. - Added ES2022 to the actor tsconfigs' `lib` (a bare `["DOM"]` drops the ES lib and broke compilation on `ErrorOptions`). - Skipped the zero-concurrency queue test: it stages a stuck queue through the v3 client-side `inProgress` set, which the rewritten queue doesn't have. - Fixed the camoufox fetch retry loop fetching 5x even on success, and removed a duplicate `apify` dependency key that silently downgraded the curl-impersonate actor to SDK v3. - Commented out the LOCAL storage matrix entry in the workflow, as `@apify/storage-local` doesn't support v4.
Replaces the remaining `ow`-based argument validation with `zod` across all packages and reworks how validation results are consumed and reported. Closes #3716 - **`ow` is gone** — every argument check now goes through `parseArgument(value, schema, label?)` from `@crawlee/utils`, backed by shared zod schemas (`schemas`, exported via `@crawlee/utils/internal`). The `@sapphire/shapeshift` checks in `@crawlee/fs-storage` were converted too, so a single validation library remains. - **Parse results are used everywhere** — option defaults moved from destructuring into the schemas (`.default(...)`), and call sites destructure the typed parse result. `parseArgument` returns `TValue & z.output<TSchema>`, so call sites keep their declared TS types while gaining the defaults. - **Schemas are built once** — all per-call schemas are hoisted to module scope; crawler/launcher classes build their strict options schema once as a `static optionsSchema` next to `optionsShape`. The `urlPatternSchema` for `include`/`exclude` lives in `enqueue_links/shared.ts`, next to the type it validates. - **Specific validators instead of `anyObject`** — class-typed options use `z.instanceof(...)` (`BaseHttpClient`, `Configuration`, `EventManager`), interface-typed ones use duck-typed `objectWithKeys` validators (`storageBackend`, `requestManager`, `logger`, …), and element-typed arrays use the new `schemas.arrayOf(item, 'numbers')`. `ArgumentValidationError` (replacing ow's `ArgumentError`) renders one line per issue: the expected type, the received type and value folded into one clause, the offending field path, and the validated interface: ```text // v3 (ow) — first issue only Expected property `maxRequestRetries` to be of type `number` but received type `string` in object `HttpCrawlerOptions` // v4 (zod) — every issue, one line each Invalid input: expected number, received the string `many` at `maxRequestRetries` in `HttpCrawlerOptions` Invalid input: expected an array of numbers, received the number `500` at `additionalHttpErrorStatusCodes` in `HttpCrawlerOptions` Invalid input: expected boolean, received the string `yes` at `retryOnBlocked` in `HttpCrawlerOptions` ``` Details worth knowing: - Union failures expand into one line per failed arm (zod's own message is a bare "Invalid input"). - `NaN` is named as itself, an empty string renders as `''`, and arrays name their element type (``expected an array of URL patterns``) — none of which ow or stock zod reported. - `new Request('https://…')` gets a targeted hint pointing at the `{ url }` object form. - For programmatic handling, the error exposes zod's structured output: `error.issues` and the raw `ZodError` as a typed `cause`. The migration is documented in the v4 upgrading guide (`docs/upgrading/upgrading_v4.md`), including a rename-cheat-sheet entry. - Custom HTTP clients must now **extend `BaseHttpClient`** from `@crawlee/http-client` rather than just implementing the interface (all shipped clients already do; `LazyDefaultHttpClient` was converted). Same applies to test mocks — `Object.create(BaseHttpClient.prototype)` works. - One caveat of consuming parse results: zod object schemas return a pruned plain copy, so options holding class instances are validated with passthrough schemas (`z.custom`-based) to keep their prototypes — there are comments at the relevant schemas. - Fixes a few latent gaps surfaced along the way: `Request.state` now accepts `RequestState.SKIPPED` (validated via `z.enum(RequestState)`), and the publish-time catalog inlining covers `optionalDependencies`. - `ArgumentValidationError` and its formatter are intentionally kept close to the copy in apify/apify-client-js#986 — a follow-up may extract them into a shared package. --------- Co-authored-by: Martin Adámek <banan23@gmail.com>
…0.0 in the camoufox template The rebase onto master carried over master's renovate bump of camoufox-js to ^0.12.0, but the template (and the repo root) still pin the Playwright version whose bundled Firefox matches camoufox-js 0.11.
…tion The rebase onto master replaced the RobotsTxtFile factory bodies with master's versions (which keep the URL for enqueue-strategy filtering), dropping the @ts-ignore comments v4 needs because robots-parser's CJS default export is not callable under nodenext module resolution.
The domain matching in the restored `filterUrl`/`matchesEnqueueStrategy` helpers (carried over from master) uses tldts, which had been dropped from the package manifest during the rebase.
- requestQueue → requestManager in enqueueLinks options - await the now-async createRequestQueueMock - sessionPoolOptions → sessionPool instance in the redirect-cookie test - config → configuration in purgeDefaultStorages options - SitemapRequestList → SitemapRequestLoader in loader tests - handleCloudflareChallenge lost its session parameter in v4 - drop duplicate imports
- markRequestHandled → markRequestAsHandled on SitemapRequestLoader - await the now-async RequestQueue.getTotalCount() - transformRequestFunction skips now report the dedicated 'transform' reason - robots.txt mock needs getCrawlDelay - statistics/session-pool single-persistence tests observe KeyValueStore.setValue instead of the persistState methods RecoverableState replaced - pass an explicit logger to Sitemap.load in the aggregated-warning test
Moves the `requestManager`-bound enqueueing logic into `BasicCrawlerContext.addRequests`, and each DOM-aware crawler now exposes its own `extractLinks()` plus an `enqueueLinks()` that composes `extractLinks` + `addRequests`. This aligns the JS implementation with what Python does, to some extent. Closes #3081
The transplanted enqueueLinks split reverted a few master-carried behaviors in BasicCrawler; this restores them on top of the new design: - stop capturing statistics before teardown again, so the crawler state is saved before the final persistence event fires (prevents double persistence) - teardown() only emits an explicit PERSIST_STATE event for externally-managed event managers, and tears the owned session pool down with persistState matching event manager ownership (an unset flag previously fell back to the `persistState = true` default, double-persisting the pool) - the enqueue limit log distinguishes an explicit `limit` from the remaining maxRequestsPerCrawl budget again - adapt the master-carried tests to the addRequests() API; drop the explicit-undefined override tests for options that no longer exist on it
… tests Follow-up to the BasicCrawler.stats → statistics rename (#4028) for two master-carried tests it could not have known about.
…atalog dependency Two upstream v4 changes landed after the last E2E run and broke the suite: - the enqueueLinks split (#4010) changed the context helper's return value to the addRequestsBatched result, so the *-enqueue-links fixtures now assert on `addedRequests` being empty instead of deep-equality with the old shape - the zod validation unification (#3935) introduced a `catalog:` dependency, which npm cannot resolve when the platform builds the actor image; the E2E package-copy step now rewrites catalog deps to their pinned versions from pnpm-workspace.yaml, the same way it already rewrites `workspace:` deps
…oudflare fixture camoufox-js releases bundle a specific Firefox build that must match the one expected by the pinned playwright version (0.11 ↔ 1.60). The fixture pinned camoufox-js ^0.12.0 next to playwright 1.60.0, so the Cloudflare challenge kept failing on the platform even with the updated challenge markup handling (#4019) in place — master validates that fix with the 0.11/1.60 pairing.
The v3.18 release blog post and the 3.18 versioned upgrading guide linked to the current (unversioned) API reference. That resolved fine on master, where the current API was 3.18, but on v4 the docs build fails: StorageClient was renamed and RequestValidationError's page moved. Version-pinned API links are the established pattern in versioned content (see the 3.17 guide).
…ixture Cloudflare serves its challenge pages with a 403 status. On v3, handleCloudflareChallenge() received the session and removed 403 from the session pool's blocked status codes itself; v4 dropped that mechanism when the hook was redesigned, so challenged requests died in throwOnBlockedRequest() on every retry and the solver only ever got a single attempt. Solving the challenge is probabilistic, which is why the fixture passes on master (where retries reach the solver) and kept failing here. blockedStatusCodes is a public crawler option in v4, so the fixture opts out of 403 explicitly. Whether handleCloudflareChallengeHook() should handle this automatically again is a follow-up design question.
impit and fs-storage-native ship platform binaries as optionalDependencies, so --omit=optional (common in v3 Docker templates) breaks the install.
…on a read-after-write race The statistics record is persisted during crawler teardown and the platform key-value store is eventually consistent, so reading it immediately after the run can miss it. That crashed the whole test with a TypeError on stats.requestsFinished (seen in cheerio-curl-impersonate-ts) even though the actor run itself succeeded. The lookup now retries for up to ~30 seconds and falls back to an empty object, so a genuinely missing record fails the assertions cleanly.
The MPL-2.0 licensed autoconsent is actively maintained and compatible with Crawlee's Apache-2.0 license, so it ships as a regular dependency instead of an optional peer dependency. Unlike idcac, it goes through the consent flow and opts out of all optional cookies (or opts in, with the new mode option) rather than just hiding the modal.
barjin
marked this pull request as draft
August 17, 2026 12:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the unmaintained, GPL-licensed
idcac-playwrightoptional peer dependency incloseCookieModalswith the actively maintained, MPL-2.0 (Apache-2.0 compatible) autoconsent, which goes through the consent flow instead of just hiding the modal. Closes #3987.