diff --git a/docs/superpowers/specs/2026-08-20-prebid-ad-latency-optimizations-design.md b/docs/superpowers/specs/2026-08-20-prebid-ad-latency-optimizations-design.md new file mode 100644 index 000000000..9f670b502 --- /dev/null +++ b/docs/superpowers/specs/2026-08-20-prebid-ad-latency-optimizations-design.md @@ -0,0 +1,236 @@ +# Prebid Ad-Latency and Auction-Load Optimizations — Design + +**Date:** 2026-08-20 (revised 2026-08-23, round 8) +**Status:** Draft (Lever A gated on a burst-trace prerequisite; Lever C gated on discovery) +**Scope:** Client-side auction properties (server-side ad templates inactive). Measurements come from a pilot news property; identifying details stay out of this document per repository policy, and sanitized measurement artifacts live outside the spec. The pilot rollout is scoped to the Fastly adapter (see the activation matrix). + +## Problem + +On properties running the client-side auction path (`creative_opportunities.enabled = false`), ad delivery leaves measurable headroom. A three-run instrumented baseline against a local Trusted Server proxying the pilot property's origin (article page, consent resolved, reader-style scrolling) measured: + +| Milestone | Time | +| -------------------------------------------------------------------- | ------ | +| Prebid bundle + shim installed (deferred head scripts, execute ~DCL) | ~2.1 s | +| `DOMContentLoaded` | ~2.1 s | +| `window.load` | ~3.0 s | +| First Trusted Server `/auction` request | ~3.3 s | +| Publisher's first `requestBids` | ~4.8 s | +| First non-empty ad render | ~5.0 s | + +Observed costs, each owned by a lever below: + +1. **The first Trusted Server auction fires ~1.2 s after `DOMContentLoaded`** and did not pass through `pbjs.requestBids`. The leading in-repo hypothesis is the `window.tsjs.requestAds` path, which builds its payload from the TSJS registry, POSTs `/auction` directly, and **immediately renders returned creatives** — fetch and render are coupled there. Lever C's discovery must confirm or refute this before any design commitment. +2. **Publisher `requestBids` bursts issue one `/auction` POST each.** The baseline captured two calls 1 ms apart producing two POSTs 2 ms apart. The POSTs run **concurrently**, so this is a load-reduction hypothesis, not a first-render latency lever. The baseline did **not** record the burst calls' option keys, ad-unit codes, bidder entries, timeouts, payload sizes, or callback behavior — the properties that decide merge eligibility — so Lever A carries a trace prerequisite (below). +3. **The first direct GAM ad request pays fresh connection setup** to `securepubads.g.doubleclick.net`. GPT scripts themselves are first-party proxied (the script guard rewrites the cascade), so only the direct ad request path can benefit from a warmed connection. + +Out of scope: re-enabling server-side ad templates, GPT lazy-load fetch margins (publisher-coordinated), the publisher's own ad-framework init latency, and server-side auction duration tuning (PBS `tmax`). + +## Billing and impression integrity (applies to every lever) + +Nothing in this design may create impression, win, or billing signals for ad units that **no caller requested to auction**. Signals differ per delivery path: + +- **Client `/auction` → Prebid adapter path:** the `/auction` response serializer does not propagate explicit `nurl`/`burl` to this consumer; win notification is owned by Prebid/GAM rendering. Early or coalesced auctions on this path are targeting-only by construction. Within a merged auction, **cross-caller delivery is accepted behavior** (see Lever A's shared-auction section): every merged unit was requested by _some_ constituent caller. What remains forbidden is delivery of units absent from every constituent call. +- **`tsjs.requestAds` path:** fetch and render are currently coupled. If discovery selects this path for Lever C, fetch must be **split from render** first; an early fetch must never trigger its render half. +- **Server-side notices:** some PBS deployments fire win/billing notices server-side, outside browser control. Any early-auction design must state whether the upstream configuration can bill on auction rather than render; properties where that is true are **excluded** from early auctions until the upstream policy is confirmed render-tied. +- **The PUC render bridge** consumes the server-template `tsjs.bids` path — inactive in this scope; its beacon semantics are not this spec's integrity boundary. +- A prefetched bid that is never consumed expires without firing any beacon. +- Guardrails use **per-path computable signals**; no cross-path "beacons per impression" universal metric is claimed. + +## Design + +### Lever A — `requestBids` coalescing window (opt-in; load-reduction hypothesis) + +**Phase 0 — trace prerequisite (blocks implementation).** Capture a sanitized trace of the production burst calls recording: full option-key set, per-call ad-unit codes and bid entries, effective timeouts, payload sizes, and **callback behavior** — specifically whether callbacks use code-scoped targeting and slot-scoped refresh, or unscoped targeting / bare `pubads.refresh()`. Implementation proceeds only if the observed calls satisfy every admission predicate below **and** the callback discipline is compatible with the shared-auction behavior: + +- If the burst is **same-code duplicate auctions** (supported today; the disjoint-code rule refuses to merge them), Lever A as specified reduces nothing — the follow-up is identical-request deduplication as a separate design, or dropping the lever. +- If callbacks perform **bare refreshes**, the first caller's bare refresh consumes the one-shot pending-delivery state for _all_ merged units, and a second caller's subsequent refresh can be classified as an independent refresh and start a **synthetic auction** — potentially cancelling the load reduction. Phase 0 must estimate the net `/auction` effect under the observed callback pattern; if the net is not clearly positive, the lever stops here. + +**Objective:** reduce `/auction` request count for bursty, disjoint-unit publisher call patterns. Downstream bidder-call reduction is a **hypothesis to measure**. Explicitly not a first-render latency lever; rollout must verify render latency does not regress. + +**Config:** `[integrations.prebid] request_bids_coalesce_ms` — `u32`, default `0`, validated `0..=250`. Injected as `requestBidsCoalesceMs`, omitted when `0`. Default `0` preserves the existing synchronous pass-through path and public observable behavior (the bundle bytes necessarily change when the coalescer ships). + +**Coalescing-config lifetime.** The injected coalescing config carries an issue timestamp (diagnostic) and a maximum document lifetime (24 h). The contract enforces the 24 h bound, fail-closed, with no skew allowance: at config receipt the shim latches both the wall-clock time and the monotonic (`performance.now()`) reading, and **evaluates the expiry check on every wrapped `requestBids` call and every flush** (not only on lifecycle events). Coalescing self-disables when **any** of these holds: monotonic age ≥ 24 h; elapsed wall time since receipt ≥ 24 h; or the wall clock reads **earlier than the latched receipt time or the previous check** (a backward jump — fail closed, since elapsed time is no longer provable). A suspended monotonic clock is covered by the wall-time comparison at the next call/flush; a negatively skewed wall clock is covered by the backward-jump rule. The enforced lifetime is therefore **at most 24 h of provable elapsed time**, which is what Drain's additive bound uses. + +**Snapshot at enqueue.** The coalescer snapshots each admitted request the way Prebid itself does on entry: shallow-copy the request object and snapshot the ad-unit **array membership and order** (retaining unit references). Later additions of request keys or array push/splice do not retroactively change the issued call; unit-object mutations are caught by dispatch-time revalidation. + +**Admission rules.** A call is held only when all of: + +- its request object consists solely of `adUnits`, `timeout`, and `bidsBackHandler`, with a non-empty explicit `adUnits` array; +- `timeout` is absent or a finite positive integer, **and** the resulting auction-time budget stays above the solo-dispatch threshold: a call whose effective timeout is less than `window + 150 ms` dispatches solo; +- it is not one of the shim's own synthetic refresh auctions (their GPT watchdog starts when the wrapper returns); +- no ad unit contains a bid entry for a configured client-side bidder; +- ad-unit codes are non-empty strings, **unique within the call**, and **disjoint from every code already pending**; +- structural safety holds: every ad unit is measurable by a **side-effect-free size estimator** that walks own enumerable _data_ properties only. Units carrying accessors, custom `toJSON`, or otherwise unmeasurable values dispatch solo — publisher getter/`toJSON` code must never execute during admission (regression-tested with a stateful `toJSON`); +- batch bounds hold after admission: at most 4 pending calls, at most 32 total ad units, and an estimated unit payload ≤ 160 KiB UTF-8. + +**Authoritative size bound (adapter seam).** The estimator above is a cheap pre-filter; the **authoritative** bound lives in the adapter's `buildRequests`, where the final body — including then-current EIDs and context — exists. Mechanics and guarantees: + +- **Boundary ownership:** the coalescer assigns each merged segment an **internal auction ID**, passes it through the merged `requestBids` call, and keys constituent boundaries by it in a module-scoped map. `buildRequests` reads the ID from the **auction-scoped `bidderRequest`** it receives (Prebid deep-clones ad units after enrichment, so unit-attached markers are unreliable and are not used). Boundaries are stored as **code → constituent membership**, never numeric index ranges: between dispatch-time revalidation and `buildRequests`, Prebid hooks, ad-unit validation, and `beforeRequestBids` listeners can still remove or reorder units (or add a client bidder), so `buildRequests` partitions the units it actually receives by code membership. A received code that belongs to no constituent, or a duplicated code, marks the partition unreliable: the auction dispatches as a **single unsplit descriptor** with a diagnostic counter — never a mis-split. Post-handoff mutation (unit removal, reorder, and injected client bidder via a `beforeRequestBids` listener) is an explicit test. **Cleanup covers every path:** the entry is deleted when `buildRequests` consumes it, on synchronous dispatch failure of the segment, and at auction end/settlement as a backstop. The map is bounded at 16 entries, but **live entries are never evicted** — evicting an active auction's entry before its deferred `buildRequests` runs would silently lose split boundaries and telemetry. When the map is full, **new calls dispatch solo** (no new merges are created) until entries release through the cleanup paths above. A test proves no boundary marker leaks into bidder params or the wire payload, and a test covers each cleanup path. +- **EID snapshot (merged auctions only; defined lifetime):** for a **merged** auction, EID collection materializes the collected EIDs into plain JSON data **at collection time** — walking own enumerable **data** properties with one accessor read and **never invoking custom `toJSON`** — and that snapshot is the only EID representation any of that auction's consumers see: transport serialization, split descriptors, callback bookkeeping, and the `ts-eids` cookie sync. The snapshot is keyed by the internal auction ID in its own map with its own lifetime — it **outlives the boundary record** (which is consumed at `buildRequests`) and is deleted only when the auction settles and its callback bookkeeping/cookie sync completes. Overlapping auctions settling in reverse order each hold an independent snapshot (tested). **Scope:** solo dispatches and the `window 0` path keep today's collection behavior byte-for-byte — snapshot and drop semantics apply only where merging is active, so the flag-off path is unchanged. +- **Non-serializable EID policy (explicit):** an EID entry whose materialization throws is **dropped from that merged auction's EID set**, incrementing a diagnostic counter; the auction proceeds without it. Nothing else changes shape: no forced merged descriptor, no thrown auction, no per-body divergence. Because the snapshot is plain data, splitting is always serialization-safe. +- **Splitting:** if the final body exceeds 192 KiB (64 KiB margin under the endpoint's 256 KiB limit) and contains multiple constituents, `buildRequests` splits along constituent boundaries into multiple transport descriptors — Prebid dispatches each as a separate HTTP request within the _same_ auction, preserving the one-auction/one-event-stream contract. +- **Narrowed guarantee + singleton rule:** the "no oversized request" guarantee applies to **multi-constituent** descriptors only. A single-constituent descriptor whose body exceeds the limit (e.g., through large common EIDs/context) cannot be split further and **dispatches as-is — exactly the behavior an oversized solo call has today** (sent, possibly answered 413, resolving as a no-bid auction). Oversized-common-EIDs is an explicit test case. + +**Ineligible arrivals — one rule for all classes:** any call failing any admission predicate **first synchronously flushes the pending batch, then dispatches solo**. Nothing overtakes an earlier caller; event order is preserved. + +**Deadlines — a reduced auction-time budget, not an absolute deadline.** Prebid starts its auction timer only after request hooks and FPD enrichment, so no wrapper can guarantee completion by `arrival + timeout`; this design shapes the _budget_: + +- At enqueue, capture the live `pbjs.getConfig('bidderTimeout')`. If it is missing or not a finite positive integer, calls without an explicit timeout dispatch solo. +- Each call's nominal deadline is `arrival + effectiveTimeout`; queue residence counts against the budget. +- Compatibility: calls merge only while `maxDeadline − minDeadline ≤ 50 ms`; an incompatible arrival flushes first. Later-arriving compatible callers accept the batch's earlier shared budget and shared `timedOut` result — documented behavior. +- A monotonic scheduler flushes at `min(windowEnd, earliestDeadline − 100 ms)` and re-arms if a new caller tightens the earliest deadline. +- The dispatched timeout is `earliestDeadline − now`, floored at **50 ms** — never `0`. The floor is reachable only through timer overshoot, which means the auction runs up to ~50 ms past the nominal deadline — accepted and documented. + +**Dispatch-time revalidation and order-preserving eviction.** All predicates are re-checked at dispatch against the live unit objects. Revalidation walks the queue **in arrival order** and dispatches **contiguous eligible segments**, with each invalid call dispatched solo in its queue position: A(valid), B(now-invalid), C(valid) → merged-[A], solo-B, merged-[C]. A synchronous dispatch failure of one segment rejects only that segment's facade promises; **later segments still dispatch in order and settle** (tested). + +**Queue lifecycle (reentrancy-safe).** At flush, the pending batch is **atomically detached** from the queue _before_ the underlying `requestBids` is invoked; settlement handlers own only the detached batch and never touch newer queue state. A constituent callback may re-enter `requestBids` and start a new batch while the first settles — the detached-batch rule makes that safe, and the reentrancy test proves the second batch dispatches and settles. + +**Dispatch.** One underlying `requestBids` per eligible segment with: the segment's units in arrival order (constituent boundaries attached for the adapter seam); the floored shared budget; and a combined `bidsBackHandler` that: + +1. runs Trusted Server bookkeeping for every constituent call first — each call keeps **its own registration ID**, preserving today's per-caller throw-rollback; +2. invokes each caller's original handler in arrival order with callback `this` and the exact three arguments `(bids, timedOut, auctionId)`. When `bids` is an object it is partitioned to the caller's codes; `undefined` cancelled-auction values pass through unaltered. A throwing handler rolls back only its own registration and does not block later handlers; after **all** handlers have run, the **first captured error is thrown synchronously** so Prebid's existing catch-and-log path handles it — observably matching today's lone-call behavior. No global asynchronous rethrow. Facade promises still resolve. + +**Shared-auction behavior (documented, accepted).** Partitioned callbacks do not partition Prebid's global auction state: unscoped `setTargetingForGPTAsync()` applies targeting for every returned unit and a bare `pubads.refresh()` can deliver a co-merged caller's units — attributed as publisher delivery because all constituent bookkeeping registers first. Operators enabling the flag accept this; acceptance tests treat cross-caller delivery within a merged auction as permitted. + +**Promise semantics.** Every held call returns a facade promise settling with the values described above. Rejection fan-out (synchronous dispatch failure) is per detached segment and is defensive unit coverage — Prebid's public promise is resolve-only, so it cannot be proven against the real artifact. One `auctionInit`/`auctionEnd` event stream per merged segment replaces N — a documented, operator-visible analytics change. **The visible `requestBids` calls themselves also change:** anything hooking or wrapping `pbjs.requestBids` (analytics adapters, publisher wrappers) observes fewer, later, merged invocations. Phase 0 must capture whether any such hook exists on the property, and the real-artifact suite documents the observable difference. + +**Coalescing telemetry (required for rollout).** Merged dispatches carry a dedicated, bounded, untrusted metadata object as a **top-level sibling of** `config` in the `/auction` body (never inside the `allowed_context_keys`-filtered context): `coalesced: { group, size, part, parts }` with a **typed, bounded schema**: `group` is a client-generated UUIDv4 string (exactly 36 characters, format-validated); `size` is the constituent count; splits run along constituent boundaries, so the full cross-field constraint is `1 <= part <= parts <= size <= 4` (with `size >= 2`). Semantics: absent = solo/legacy; **logical merges are counted by distinct `group`**, so split descriptors never overcount; rows sharing a `group` must agree on `size`/`parts` (**group consistency**, validated in the rollup — a disagreeing group is counted invalid, not merged); any field failing its type, bound, or cross-field constraint drops the **whole object as absent, never clamped** — and every such drop increments a **`coalesced_invalid` counter** carried through the observation context, event row, and rollup, so malformed merged traffic cannot masquerade as solo and silently satisfy drain. Server side: the endpoint parses and validates the object, `AuctionObservationContext` and the auction event summary row carry the fields, the Tinybird datasource gains the columns — **schema deployed before the emitting binary** — and a named rollup (`auction_coalescing_daily`: logical merges, constituent totals, split counts per property/day) plus a dashboard panel constitute the continuous guardrail; a raw column alone does not. Endpoint, telemetry, sink-serialization, schema, and rollup tests are part of the implementation. + +### Lever B — GAM preconnect hint (opt-in) + +**Config:** `[integrations.gpt] gam_preconnect` — `bool`, default `false` — plus a required-when-enabled `[integrations.gpt.gam_preconnect_approval]` **embedded manifest envelope**, so validation needs no fetch and has no fetch-failure branch. The envelope carries the manifest content itself (id, configuration version, jurisdiction inventory, verified matrix summary, accepted tail, approver, `valid_until`), canonically encoded as **RFC 8785 (JCS) JSON**, with an Ed25519 signature over the canonical bytes. Trust and rotation: the verifying public keys ship in the binary's pinned trust set (two active keys during rotation overlap; a manifest verifies against either). At config load the server verifies: signature valid against a pinned key; the manifest's **configuration-version digest** (SHA-256 over the canonical encoding of the effective integration configuration, computed by the server at load) matches the digest the manifest records; and jurisdiction scope matches the deployed scope declaration. **Expiry is rechecked at every emission**, not only at load — GPT `head_inserts` builds synchronously from `Settings`, so the emission path compares current server time against `valid_until` per document; a long-lived adapter (Axum/Cloudflare hold startup state) therefore stops emitting at expiry without a reload. Any validation failure leaves the hint un-emitted with a warning — an approval that does not describe the running configuration proves nothing and must not enable emission. + +When enabled, GPT `head_inserts` emits `` **before the GPT bootstrap inserts** (asserted by a transformed-HTML ordering test), without `crossorigin` — ad requests are cookie-credentialed. Browsers may partially perform or skip hints; best-effort by nature. + +**Scope of claim:** GPT scripts (including `pubads_impl`) are first-party proxied; the hint can only affect the **first direct ad request**, and the claim is "may reduce" its connection setup. + +**Browser coverage (blocking gates, two distinct scopes):** the flag emits to every browser, and per-engine emission restriction would require request-scoped UA gating (out of scope). Two gates with different scopes follow: + +- **Privacy/conformance gate — a finite, named matrix, not a family claim.** One pinned build per engine family does not cover product, OS network-stack, WebView, fork, or version differences, so the gate is a **defined product/OS matrix**: for each engine family (Chromium/Blink, WebKit, Gecko), every product/OS combination above 1% property traffic share — **including Android WebView when the property serves it** — at pinned current stable versions. Each matrix cell must pass the invariant; a cell that cannot be verified blocks enablement — a traffic threshold is never a privacy waiver within the matrix. The **residual tail below the matrix floor is explicitly risk-accepted in the approval artifact** (named, with rationale: request-write behavior is an engine property and the tail embeds the verified engines), and any observed violation anywhere — matrix or tail — is an immediate rollback. +- **Performance gate — material engines.** Reuse/benefit acceptance is required per engine **above a 5% traffic share** on the property; engines below 5% are documented as performance-unverified in the approval (their conformance is still required above). + +The unit of verification is the **matrix cell — product × OS × pinned version × consent scenario** (consent-resolved, GPC-set, CMP-unresolved, CMP-denied) — not the engine: sample floors (at least 20 cold-start runs), event predicates, and acceptance rules are all defined and satisfied **per cell**, using engine-appropriate low-level tooling (Chromium: NetLog; WebKit/Firefox: their native network logging). Current repository browser coverage is Chromium-only, so building this matrix is part of the lever's cost. + +**Governance (required before any property enables it):** + +- The flag is a property-level boolean and GPT head insertion has no request-scoped jurisdiction or consent input; the approval must therefore cover **every jurisdiction served by the deployed configuration**. +- **The approval is a durable, signed manifest** recording: configuration version, jurisdiction inventory and unknown-jurisdiction handling, the verified product/OS matrix and the explicitly risk-accepted tail, approver, date, expiry, a re-approval trigger when the served scope changes, and — explicitly — acceptance of the **pre-consent DNS/TCP/TLS disclosure inherent to preconnect**: the hint establishes a connection to the GAM host (disclosing the reader's IP address to it) before any consent signal resolves, an inherent property of the HTML preconnect algorithm that no request-write invariant removes. The signed manifest **is** the config's embedded `gam_preconnect_approval` envelope, so runtime enforcement and the artifact cannot drift apart. +- **Verification protocol:** pinned browser build per engine, fresh profile per run, cold cache and socket pools, capture mode stated explicitly, raw logs treated as sensitive (retained only for the acceptance window). **Redaction must not destroy evidence:** the capture preserves the **complete hint-transport dependency graph** — every connection to the GAM host including coalesced HTTP/2/HTTP/3 connections, socket-to-request joins, and the initiator attribution proving the first ad request came from GPT — and sanitization to the GAM host happens only **after** the invariant is evaluated over that full graph, never before. For Chromium: NetLog with the speculative socket joined to the first ad request via source IDs, using a **versioned parser algorithm with fixtures and one controlled end-to-end test**; equivalent engine-appropriate tooling for others. Current audit tooling has no NetLog surface — building this capture path is part of the lever's implementation cost. +- **Enforceable invariant:** no HTTP request writes on **any connection attributable to the hint** — not only the socket the first ad request later reuses — before the normal first GAM ad request; defined as no HTTP/2 or HTTP/3 HEADERS/DATA frames **and** no HTTP/1 request writes, on every `securepubads.g.doubleclick.net` connection the browser opened speculatively. "Normal first GAM request" = the first `securepubads.g.doubleclick.net` ad request initiated by GPT for the document. **Terminal observation horizon:** a run in which no GAM ad request ever occurs is not indeterminate — it observes all hint-attributed connections for the full scripted session plus 60 s of idle, and passes only if zero request writes occurred in that window. Verified including GPC-set, CMP-unresolved, and CMP-denied cases; connection-level protocol frames (settings, pings) are inherent to preconnect and permitted. +- Rollback trigger: any observed request write before the normal ad request disables the flag — an immediate-stop conformance failure regardless of performance. + +### Lever C — earlier first auction (discovery first; design contingent) + +**Status: not implementation-ready.** No design option is selected; discovery produces the inputs for a follow-up spec revision. + +**Discovery contract:** + +- **Attribute the 3.3 s `/auction` request.** Named hypothesis: `tsjs.requestAds`. If confirmed, acceleration is mis-scoped until fetch is split from render, and the prior question becomes whether an independent non-Prebid auction should be **deduplicated or removed** rather than accelerated. +- **Typed direct-path result.** `requestAds` returns `void` and `sendAuction` collapses network failure, parse failure, and legitimate emptiness into `[]`. A fetch/render split requires a private result type — `{outcome, bids, completedAt}` — preserving the public callback lifecycle. +- **Transport constraint (`keepalive`).** `sendAuction` sends with `keepalive: true`, and the Fetch Standard's 64 KiB keepalive quota is **aggregate across unfinished keepalive requests, not per body** — two concurrent 40 KiB requests can fail even though each fits. A per-body size gate therefore cannot make `keepalive` safe; the private producer **drops `keepalive`** for its POSTs, with a test covering concurrent-request aggregation at the boundary. +- **Provider/mediator privacy audit (blocking, all of them).** The redaction obligation is not Prebid-specific: the Prebid provider debug-logs the full normalized OpenRTB request, the APS provider `trace!`-logs its request body **and** exposes it as client-visible debug metadata when `debug` is on, and mediators see the same bytes. Before any fingerprint/preflight work ships, discovery delivers an **audit of every enabled provider and mediator across logs, error reports, telemetry, and client-visible debug output**, with blocking redaction tests. Retained diagnostics must be **bounded metadata or navigation-keyed values** — a stable content hash is linkable across navigations and is not an acceptable diagnostic form. +- **Transport ownership and the Fastly constraint.** The Prebid adapter returns a request descriptor (Prebid core owns that HTTP operation); the repository-owned `sendAuction` is the separate core API. Server-side single-flight at the common auction boundary is the preferred _shape_, **but it currently has no viable Fastly owner**: Fastly application state is rebuilt per request, and the platform abstraction exposes KV/cache/HTTP but no atomic pending-join primitive. **Discovery exit criterion:** prove a cross-instance, atomic, _pending-only_ join with zero retention after settlement on Fastly. If Fastly cannot supply that contract, server-side single-flight is off the table for the pilot and the candidate becomes authenticated client-side coordination or a different architecture. An ordinary persistent-cache lookup is not an acceptable substitute — it _is_ the completed-response reuse this spec forbids. +- **Navigation-scoped reservation (both architectures).** Client- or server-side, sharing requires: an opaque, authenticated, **single-navigation reservation**; exactly one intended early/normal pair per reservation; server revalidation of hidden inputs (HttpOnly EC identity, geo, server-resolved EIDs, headers, provider mode, configuration version) on the joining request; a canonical key derived from the fully normalized provider input plus relevant headers/settings — never a content-only or stable-identity key, which could join different documents or users with identical units and expose a one-shot bid across contexts. **Key/fingerprint hygiene:** the key is a **versioned keyed digest** (server-keyed HMAC over the normalized input) combined with the navigation nonce; it lives only for the pending lifetime, and neither the digest input nor the raw digest ever appears in logs, URLs, or telemetry — telemetry may carry only a truncated, non-reversible diagnostic form. +- **Provider-free signed preflight (the only workable token issuance).** A digest computed during the early provider execution cannot reach the browser before that response settles — when pending-only joining is already closed — and a stateless instance cannot compare a joiner against inputs it never saw. The reservation token must therefore be issued by a **provider-free preflight**: it normalizes inputs, computes the digest, and immediately returns a **versioned, server-signed token carrying the input digest, navigation nonce, and expiry** — executing no provider call. **Both sides bind to the token:** the provider-executing early request presents it too, and the server revalidates the early request's inputs against the attested digest **before executing the provider** — a producer whose inputs mutated after preflight executes as an unshared fresh auction (its result is never joinable), so a joiner can never match a token whose producer ran different bytes. The joiner presents the same token; **any** instance validates the signature statelessly and compares its own revalidated inputs against the attested digest. **Single-use is enforced by the pending-join state owner** — the same atomic cross-instance pending-join primitive the Fastly exit criterion must prove; no other component claims that responsibility. The pure side-effect-free `prepare + fingerprint` capability is required in **both** architectures (it is what the preflight runs client-side-initiated, and what the server join compares). Discovery must deliver **replay, input-mismatch (joiner and producer sides), navigation-change, and expiry tests** for this protocol — without it, client coordination is rejected. +- **Single-consumer sharing via a private fetch-only producer.** The public `requestAds` path renders every returned creative, and one bid must never reach two rendering consumers. Sharing is therefore restricted to a **private, fetch-only speculative producer**: its result carries no render obligation, and each response is **consumable exactly once** — the arbitration state machine hands the settled result to at most one consumer (the joining Prebid auction _or_ nothing), never to the public direct-path renderer and a second consumer. The public call keeps its normal consumer and its result is never shared. Removing or changing the public path's render remains permissible only as an explicitly approved breaking migration. +- **Pending-only state machine (for a private speculative producer).** Joining is allowed **only while Pending**. If no waiter attaches before settlement, the private result is **discarded immediately**; a later call starts a fresh auction; the pending-to-settled attachment race is atomic and tested. The measured gap (early ~3.3 s, publisher ~4.8 s) makes settled-before-join the _likely_ case — expected benefit is modest and must be measured before further investment. A full **outcome transition table** is a discovery deliverable: for each of bid / no-bid / consent-denial / failure / timeout / invalidation / attachment-race, define the consumer, deadline, and terminal state. +- **Client-side feasibility gate.** Prebid owns transport and request-scoped bid IDs, and the repository `sendAuction` returns flattened bids, losing raw response and outcome information. The invariant is **exactly one provider-executing `/auction` request** — not one HTTP request, which would contradict the server-revalidation requirement above. A lightweight, authenticated **join/claim request that executes no provider call** is permitted, and must be proven side-effect-free (no upstream contact, no billing-relevant state). Client-side coordination is admissible only after a **real-artifact feasibility proof**: one provider execution serving the single consumer while preserving bid-request IDs, APS admission, callbacks, promises, events, timeout semantics, targeting, and global bid state. If no supported seam exists, authenticated client coordination is also off the table — and Lever C may be infeasible in every architecture, which is an acceptable discovery outcome. +- **Server-side pure-plan seam.** Server request normalization currently inserts a fresh correlation UUID, and the provider contract exposes only a side-effecting `request_bids` — a fingerprint containing correlation randomness never matches, and invoking a provider to learn its exact bytes already contacts the upstream. **Discovery exit criterion:** a side-effect-free `prepare + fingerprint` capability for **every enabled provider and mediator**, or server-side sharing is off the table. +- **Cancellation honesty.** `sendAuction` has no `AbortSignal` and uses `keepalive`; client detach discards the local result but does not prove the server or upstream provider stopped. The design distinguishes **detach/result-discard** from **proven upstream cancellation**, adds generation guards against late targeting/render, and states whether a fresh auction may overlap an invalidated one. +- **Server-seam event semantics** (if a server join is ever built): share only provider/orchestrator execution after each waiter's inputs are normalized; serialize request-specific responses separately; emit one leader auction event plus a joined-waiter event; never replay the leader's correlation data. +- **No completed-response reuse.** The `/auction` response carries no bid lifetime and the adapter stamps a fresh `ttl: 300` at interpretation; reuse would silently renew lifetimes. If ever supported, the response must carry completion time and per-bid expiry, with TTL set to remaining lifetime. +- **Consent parity.** The `/auction` body carries no consent envelope; the server reconstructs consent from cookies and `Sec-GPC`. Discovery must define cookie parity or a validated consent envelope, and `/auction` needs a response signal distinguishing consent-denied from legitimate no-bid; consent-denied results are never shareable. +- **Input ownership tables** per path (TSJS registry generations, Prebid ad-unit generations, navigation generation, render targets); GPT slot targeting is an auction input only if discovery proves it affects request bytes. +- **Billing integrity** per the section above; split fetch from render before any reuse on the `requestAds` path. + +## Config-blob compatibility + +Integration settings are retained as raw JSON in the pushed blob, so an explicitly configured `0`/`false` is present in the blob; only omitted keys are absent. `PrebidIntegrationConfig` and `GptConfig` do not `deny_unknown_fields`, so older binaries tolerate blobs carrying the new keys. Compatibility tests carry the **non-default values** (`request_bids_coalesce_ms = 50`, `gam_preconnect = true` with a full `gam_preconnect_approval` envelope table) through a full blob into the legacy struct shapes, plus present-leaf and absent-leaf environment-overlay tests for **each** new field and the envelope table. The public configuration guide's "unknown TOML keys fail" statement gains an explicit exception note for forward-compatible integration leaves. + +**Adapter activation/rollback matrix.** Injected client config is read once per document, so within an adapter a config change affects new documents only — and "new navigation" is not "new document": client-path HTML can be browser-cached for 60 s. Across adapters: Fastly instances are effectively per-request (config push suffices). Axum builds shared state at startup (restart required). Cloudflare holds startup state (redeploy/restart). **Spin parses an embedded example config at build time — a restart cannot activate a changed flag; it requires a source-config change plus rebuild/redeploy (or future runtime config loading).** The pilot rollout is scoped to Fastly. + +## Testing + +**Vitest (shim), Lever A:** + +- two mergeable calls → one underlying `requestBids`, segment units in arrival order, per-caller partitioned maps, handlers in arrival order with preserved `this` and exact `(bids, timedOut, auctionId)`; cancelled-auction `undefined` values pass through unpartitioned; +- bookkeeping-before-callbacks with per-caller registration IDs; a throwing first handler rolls back only its own registration, later handlers run, no facade promise rejects, and the first captured error is **synchronously** thrown after all handlers (asserted to land in Prebid's catch path, matching the existing lone-call regression); +- every ineligible-arrival class flushes the pending batch first, then dispatches solo — ordering asserted; +- order-preserving eviction: merged-[A], solo-B, merged-[C]; **segment-failure isolation:** segment A's underlying dispatch throws synchronously, segments B and C still dispatch in order and settle; +- shared-auction scenario: unscoped `setTargetingForGPTAsync()` plus bare and mixed-slot `refresh()` from the first callback, asserting documented cross-caller delivery, pending-state consumption, synthetic-auction classification, and net `/auction` count; +- reentrancy: a constituent callback enqueues a new batch while the first settles; the detached-batch rule proven — second batch dispatches and settles un-clobbered; +- deadline math: sub-threshold, zero/negative/`NaN`/`Infinity`, missing/invalid captured `bidderTimeout`, mid-hold config change, earlier-deadline re-arm, overshoot floor (never `0`); +- snapshot semantics: post-enqueue request-key additions and array push/splice do not alter the issued call; unit-object mutations caught at revalidation; +- **side-effect-free estimator:** a stateful `toJSON`/getter is never invoked at admission (regression test); accessor-bearing units dispatch solo; +- **adapter split:** `buildRequests` reads boundaries via the internal auction ID from the auction-scoped `bidderRequest` and splits an over-limit merged body along call boundaries (boundary−1/boundary/boundary+1, multibyte content, large EIDs), each descriptor carrying correct `coalesced` group/part metadata; an **oversized single constituent** (large common EIDs) dispatches unsplit as today; **no boundary marker appears in bidder params or the wire payload**; a **stateful EID getter** is read exactly once per merged auction and a **custom `toJSON` is never invoked** (data-property walk), the transport body, split descriptors, and cookie sync all observe the same snapshot, overlapping reverse-settling auctions hold independent snapshots, the `window 0`/solo path collects EIDs exactly as today, and a **throwing EID materialization** drops that entry, increments the diagnostic counter, and the auction still dispatches; +- coalescing-config lifetime: expiry is checked on every wrapped call and flush — a simulated suspend (frozen `performance.now()`, advanced `Date`) disables via the wall-time comparison, a forward-skewed `Date` alone disables no earlier than the monotonic bound, and a **backward `Date` jump disables immediately** (fail closed); +- boundary-map capacity: a queue admitted while the map is at 16 live entries flushes/dispatches solo without evicting any live entry, and merging resumes after cleanup releases entries; +- `requestBids` observability: exact timing and payload of the merged underlying call as seen by an external wrapper, in **both installation orders** (coalescer wraps the analytics wrapper; analytics wrapper wraps the coalescer); +- window `0` / absent config leaves the existing suite untouched. + +**Real-artifact coverage (external bundle):** two real calls → one fetch, two thenables, callback-before-promise ordering, partitioned results, a single event sequence, shared timeout/auction-id semantics; a merged body exceeding the limit where `buildRequests` returns multiple descriptors within one auction, **including split-response aggregation** (each descriptor's response contributes its bids to the one auction result) and a **partial-413 case** (one descriptor rejected 413 while siblings succeed — the auction settles, surviving bids are delivered, no global error); a **throwing constituent callback** proving no global error surfaces and all facades resolve; and a merged dispatch where some constituents have **no handler**, proving bookkeeping still runs for them. + +**Rust:** + +- config defaults and bounds; injected-config serialization omits `requestBidsCoalesceMs` at `0`; injected coalescing config carries `issuedAt` + lifetime; +- `/auction` endpoint parses and validates the `coalesced` object (absent; valid group/size/part; malformed, out-of-bound, and cross-field-violating values dropped as absent, never clamped); `AuctionObservationContext` and the event summary row carry the fields; sink serialization includes them; the Tinybird columns, the `auction_coalescing_daily` rollup, and the schema-before-binary ordering are covered; +- **deployment-skew telemetry fixtures:** an old-emitter row (no `coalesced` fields) ingests cleanly into the new schema, and a binary-rollback row (new schema deployed, old emitter restored) reads as solo/legacy — both directions fixtured; +- **invalid-metadata propagation:** a dropped `coalesced` object increments `coalesced_invalid` through the observation context, event row, and rollup; **group consistency** — rows sharing a `group` with disagreeing `size`/`parts` are counted invalid, not merged — both fixtured; +- `gam_preconnect = true` with a valid embedded approval envelope emits the link without `crossorigin` **before** the GPT bootstrap inserts; `false` emits nothing; enabled with a **missing envelope, signature failure, unknown signing key, configuration-digest or jurisdiction-scope mismatch, or expired `valid_until`** emits nothing and warns — each failure mode fixtured; **expiry is enforced per emission** (a valid-at-load manifest stops emitting once `valid_until` passes, tested with a stepped clock); +- new-schema → legacy-schema blob test with non-default values; present/absent env-leaf tests for `request_bids_coalesce_ms`, `gam_preconnect`, and the `gam_preconnect_approval` envelope table. + +**Lever B engine matrix:** for each engine family in the conformance gate, the matrix carries a versioned capture parser with fixtures, the privacy-invariant conformance run set (including the no-GAM-request horizon case), and one controlled end-to-end run — per engine, not Chromium-only. + +**Documentation checklist:** `trusted-server.example.toml`, configuration tables (including the unknown-keys exception note and the preconnect approval keys), Prebid and GPT integration guides, environment-overlay leaf behavior for the new fields, and the **adapter activation/rollback matrix published in the operator documentation** (not only in this spec) so operators of non-Fastly adapters know a config change alone does not activate or roll back these flags. + +**Lever C tests are defined with its design after discovery.** Discovery itself must additionally deliver: + +- **Passive measurement first — with instrumentation that can actually compute it.** Existing telemetry cannot: auction telemetry UUIDs are fresh per request and unrelated across a document's requests, and a start-time gap alone proves neither pending overlap nor input equivalence. Discovery therefore adds a **privacy-safe document span** — a navigation-keyed (never identity-keyed) client record carrying early-request start/settlement, publisher `requestBids` invocation time, an input-equivalence indicator, and invalidation events — and the attach-before-settle potential is computed from these spans. No speculative executions are added to measure the opportunity. +- **Consumption telemetry with terminal outcomes:** produced, attached-before-settle, **admitted by Prebid**, **rendered**, and discarded-on-settle counts — attachment alone does not prove consumption. +- **An explicit go/no-go threshold:** Lever C proceeds past discovery only if the passively measured attach-before-settle rate is ≥20% and the modeled median first-render improvement is ≥100 ms; below either, the lever is dropped without further design investment. +- **Hard load non-regression gates:** on navigations where the speculative result settles before any join, a private producer adds an execution the page already paid for elsewhere (public direct call + speculative + fresh Prebid auction). Total `/auction` requests per navigation and upstream provider executions per navigation **must not increase** versus baseline, and the accounting includes the coordination overhead itself — preflight/claim request counts, bytes, server CPU, retries, and added latency — these gates bind regardless of the attach-rate gate passing. + +## Measurement methodology + +Acceptance uses a reproducible harness (local Viceroy against the pilot origin, pre-seeded consent, identical scripted scroll) with **randomized, balanced AB/BA pair ordering**, explicit warm/cold connection conditions, and sample sizes derived from pilot variance with **quantile-specific power calculations**: a pilot batch of ≥10 pairs seeds variance for **median** gates only; gates at p95 require a pilot batch of **≥30 pairs** (tail variance cannot be seeded from 10). Acceptance batches are powered at 80% for the stated effect at the stated quantile, never fewer than 20 pairs for median gates and never fewer than **50 pairs for p95 gates**. Decisions use **named paired estimators and a named interval method**: median differences via the Hodges–Lehmann estimator; quantile differences via the paired bootstrap; intervals for **continuous paired metrics** are **BCa bootstrap intervals with 10,000 resamples**; intervals for **binomial proportions** (socket reuse, merge success) use the **one-sided Wilson score bound** — BCa degenerates on all-success/all-failure samples and is not used for proportions; non-inferiority is stated per metric and quantile on a **ratio scale** (treatment/baseline at the stated quantile, one-sided 95% upper bound ≤ 1.05). **No-render handling:** a pair whose treatment arm produces no render within the harness window scores that arm at the **numeric measurement-window cap** (the harness observation window duration — a defined numeric censoring value usable in quantile and ratio estimators, conservative); pairs with no render in either arm are excluded but capped — more than 10% such pairs invalidates the batch. + +**Denominators:** a _run_ is one full harness execution (fresh context); a _navigation_ is one document load; a **candidate burst** is ≥2 `requestBids` calls within the configured window on one navigation (pre-admission); an **eligible burst** is a candidate burst whose calls pass every admission predicate. + +Numeric gates: + +- **Lever A retains if:** candidate-burst coverage is reported (share of candidate bursts that are eligible — this is the publisher-pattern fact); the one-sided 95% Wilson lower bound on **merge success among eligible bursts is ≥95%** (implementation health — eligible bursts should merge absent runtime failure, and each failure is diagnosed); the **relative drop in total `/auction` requests per navigation** — estimated as the paired per-navigation count ratio (treatment/baseline), BCa-bootstrapped over pairs — has a one-sided 95% **upper bound ≤ 0.80** (equivalently, the lower bound on the drop is ≥20%; a point estimate of 20% with a bound near zero does not pass); and first-non-empty-render non-inferiority holds at p50 and p95 on the ratio scale (one-sided upper bound ≤ 1.05). Otherwise the flag returns to `0`. +- **Lever B retains if:** the one-sided 95% **lower confidence bound** on cold-start speculative-socket reuse for the first GAM ad request exceeds 70% — the denominator is runs where a first GAM ad request occurred; a run with no speculative socket counts as a reuse failure; **parser-indeterminate runs count as reuse failures in the primary analysis** (the optimistic excluded-run figure may be reported as a sensitivity analysis only), and a batch with more than 10% indeterminate runs is invalid regardless — the request-write invariant holds in 100% of runs (any violation = immediate rollback); and the benefit gate passes: the one-sided 95% **lower bound** of the paired setup-time improvement (Hodges–Lehmann, BCa interval) **exceeds the 30 ms minimum worthwhile effect** (a point estimate of 30 ms with a confidence interval reaching near zero does not pass). Median (location) effects use Hodges–Lehmann; **p95 non-inferiority uses the paired bootstrap quantile estimator** on the ratio scale (one-sided upper bound ≤ 1.05), for first-render (p50, p95) and `load`-event (p50). Socket reuse alone does not retain the flag. +- **B is measured with identical Lever A state in both arms** — the comparison is X vs. X+B, stated in the results. +- **Causality boundary:** the pre-enable paired harness runs are the causal experiment and the **only basis for retention** — the acceptance gates above are evaluated on them, before enablement. The production hold checks are **non-causal guardrails** (a globally enabled property has no contemporaneous control arm): they can trigger rollback and nothing else. +- **Hold decision rule:** synthetic checks run **daily** (batches of at least 10 pairs) as regression/conformance detectors, with conformance failures acting immediately at any point. "Hold completion" is not a retention decision: the hold **completes** when day 7 passes with no rollback trigger, and the reporting-based guardrails (fill/revenue, 48 h maturation delay) for the full day 1–7 window are evaluated once matured — **at day 9** — before the hold is declared clean. No sequential peeking on the synthetic batches; the day-9 reporting evaluation is the single final read. + +**Live control limits (per signal, fixed before enablement):** every live guardrail carries a written tuple — estimator, fixed threshold, **rollback-bound polarity**, minimum daily volume, and missing/stale-data action — so a rollback decision is reproducible from the recorded rules alone. The polarity is the opposite of acceptance: acceptance proved benefit via a lower bound clearing a threshold, but copying that rule live would let mere daily-sample **uncertainty** trigger rollback. A live metric triggers rollback only on **evidence of degradation** — its one-sided 95% **upper** bound (for benefit metrics) falls below the minimum acceptable level, or a conformance invariant fails (those act immediately, no statistics involved). **Batching by metric feasibility:** daily 10-pair batches evaluate only median-level and proportion metrics; p95-gated metrics are evaluated once on the accumulated hold pool (≥70 pairs by day 7) at the day-7/day-9 read. **Repeated-look control:** the seven daily looks share a Bonferroni-split alpha (0.05/7 per look) so the hold's family-wise false-rollback rate stays at 5%. Fill/revenue alerts require a minimum volume of 1,000 impressions per property-day — a below-volume day is **inconclusive**, and inconclusive or missing/stale (>48 h late) reporting days extend the hold rather than pass it, with two consecutive inconclusive days escalating to manual review. + +**Live guardrails and their sources:** + +| Signal | Source | Type / cadence | +| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | +| Merged-auction count / drain state | `coalesced` group counting via the `auction_coalescing_daily` rollup | new; continuous | +| `/auction` volume per property | existing server telemetry | existing; continuous | +| Fill/revenue, bid rate, timeout rate | ad-server / PBS reporting vs. 7-day pre-enable baseline; alert on >5% adverse move; owner: property operator; reporting maturation delay 48 h | existing; daily | +| First-render / per-slot latency | scheduled synthetic harness runs against production | synthetic; daily during holds | +| Lever B socket reuse + setup time | scheduled engine-appropriate capture runs | synthetic; daily during the B hold | +| Consent-denied network activity (Lever B) | scheduled capture runs (GPC / unresolved / denied) | synthetic; daily during the B hold | + +**Drain (Lever A):** three requirements make the bound hold end to end. + +- **Injection implies a full-stack cache bound.** The 60 s freshness term is only valid when every cache layer honors it. Response classes that retain origin/CDN cache policy (bot, prefetch, consent-denied/unresolved, non-GET — today's request-scoped cache-cap skips) **must not inject either lever's client config**. Any HTML response that does inject it carries: browser `Cache-Control: max-age=60` (or stricter), **capped or removed CDN-targeted directives** (`Surrogate-Control`, `Fastly-Surrogate-Control`, `CDN-Cache-Control`, `Cloudflare-CDN-Cache-Control`), a removed `Expires`, and — critically — **no revivable validators**: `ETag`/`Last-Modified` are stripped (today conditional-header stripping applies only on ad-stack paths), because an origin `304` would otherwise revive previously transformed flag-carrying HTML past every `max-age`. A test asserts the implication in both directions across all named headers. +- **Emission stops before approval expiry can be outlived.** Lever B's per-emission expiry check uses `valid_until − (observed propagation + 60 s freshness)` as its effective cutoff, so no cached hint-carrying HTML can still be served when the approval lapses. +- **Document lifetime is enforced per call, fail-closed.** Per the lifetime contract above, expiry is evaluated on every wrapped call and flush against both the monotonic age and elapsed wall time since receipt, with backward wall-clock jumps disabling coalescing outright. Suspension gaps (monotonic clocks may exclude suspended time in Firefox/WebKit) are caught by the wall-time comparison at the next call; over-disabling is safe, and no path extends the lifetime past 24 h of provable elapsed time. +- **Propagation is observed across the served scope.** After the disabling push, the effective **rendered** configuration must be observed across the POP scope actually serving the property (enumerated from recent traffic logs, or via platform config-version observability) — a fixed small probe count does not establish convergence for an eventually consistent config store; ~5 minutes is the expectation, the observation is the bound. + +Drain completion is then the **additive** bound: observed propagation + 60 s HTML freshness + the 24 h enforced document lifetime after the flag returns to `0` — and this **time bound is authoritative**. The telemetry check is confirmatory, not gating: because auction telemetry is best-effort and can drop rows, the `auction_coalescing_daily` rollup showing **zero logical merges while overall `/auction` row volume confirms pipeline liveness** corroborates drain; the invalid-metadata signal is compared against its **trailing pre-disable baseline** (a server-observed rate, not a value clients can set) and an anomaly **escalates to investigation without extending drain** — `coalesced` metadata is client-supplied, so a rule that let malformed traffic hold drain open indefinitely would hand an attacker a veto. Lever B rollback completion is the same observed propagation + 60 s HTML freshness. + +## Rollout + +One lever at a time, Fastly pilot only, each independently config-reversible for newly generated documents: + +1. Land binary (including the telemetry field and Tinybird schema, deployed schema-first); all flags default off. +2. Lever A Phase 0 trace on the pilot property (including callback-discipline capture). If predicates hold: run the **pre-enable paired acceptance harness**; only if its gates pass, enable `request_bids_coalesce_ms = 50` alone; hold per the hold decision rule (rollback-only guardrails, day-9 reporting read); on any trigger return to `0` and confirm drain per the definition above. +3. With A's hold clean or A fully drained: run Lever B's **pre-enable paired acceptance harness**; only if its gates pass, enable `gam_preconnect` alone under its governance artifact; hold per the hold decision rule with the daily synthetic reuse/setup and consent-network checks as rollback triggers — the live checks never retain the flag, they can only roll it back; conformance failures act immediately. +4. Lever C follows its own spec revision after discovery.