feat(self-driving): wire up the Replay Vision scanners step - #1055
Open
Twixes wants to merge 2 commits into
Open
feat(self-driving): wire up the Replay Vision scanners step#1055Twixes wants to merge 2 commits into
Twixes wants to merge 2 commits into
Conversation
The wizard glue for step 6c, whose HOW lives in the context-mill skill: the OAuth scope, the prompt STEP + task-list entry, and a sidebar tip. The scope object is `replay_scanner`, not `vision-scanners-*` — those are MCP tool names, and requesting one grants nothing, so the step would 403 on every call. Create/update also require `session_recording:read` alongside `replay_scanner:write` (the API pairs them, since a scanner's config indirectly exposes recording contents); that one is already granted for the step-2 usage probes, so only the `replay_scanner` pair is net-new. Like `product_enablement:write`, it needs the manual prod OAuth-ceiling edit before a token can carry it — README and the §7 checklist updated. The prompt carries only what the wizard owns: order, the never-abort contract, and the two constraints too load-bearing to leave to the skill alone — don't edit the skeletons' locked fields, and keep the scanners' queries disjoint. ARCHITECTURE §10 documents why that second one matters: overlapping scanners describe one defect twice and self-corroborate past the promotion threshold. The scanner tip earns its slot because a scanner is the least familiar thing in the run — an LLM watching recordings on a cadence, and the only part of setup that spends Replay Vision quota.
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
… scopes The live wizard OAuth apps' `OAuthApplication.scopes` use the `@default` sentinel (US prod: `@default,llm_gateway:read,wizard_session:read, wizard_session:write`), not the exhaustive literal list this repo's docs described. `@default` resolves to `UNPRIVILEGED_SCOPES` — every public (non-privileged, non-internal, non-hidden) obj:action scope — and auto-tracks new ones (`posthog/scopes.py`, `resolve_ceiling`). So `replay_scanner:*` — and `product_enablement:write`, and every other self-driving addition — are already inside the ceiling. There is **no manual prod ceiling edit** for any of them; I had it wrong, and so did the README (§ "OAuth app scope ceiling"), ARCHITECTURE §3/§7/§9/§10. Only a privileged/internal/hidden object would ever need a per-app edit, and the wizard requests none. Rewrites the README ceiling section to explain `@default` and how to verify (`seed_oauth_app_scopes --dry-run`), flips §7 item 1 from "add ten net-new objects" to "no action needed, here's why + how to check", and drops the "one manual step outstanding" banners from §9 (product_enablement) and §10 (replay_scanner). The only real ordering constraint left is wizard-release- before-mcp-publish, and a token predating the scope is fixed by a reconnect, not a ceiling edit. Also notes the EU prod app: a per-region DB row referenced via `WIZARD_CLOUD_RUN_OAUTH_CLIENT_ID`, not committed here, seeded the same `@default,…` way.
edwinyjlim
reviewed
Aug 5, 2026
edwinyjlim
left a comment
edwinyjlim
left a comment
Member
There was a problem hiding this comment.
lgtm but will let someone from self-driving verify behavior
sortafreel
approved these changes
Aug 5, 2026
Contributor
🦔 ReviewHog reviewed this pull requestNothing worth raising this time, so here's a calming picture instead:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Human tl;dr: Pairs with PostHog/context-mill#313. This adds the scaffolding for the new replay vision setup step in scanner. Scoping changes are key, then a little bit of text updates.
What
The wizard glue for a new self-driving step 6c: create Replay Vision scanners with
emits_signals: trueso on-screen breakage a recording reveals reaches the inbox, alongside what sources and scouts already pull. Three small surfaces, since the HOW lives in the skill:program-scopes.ts—replay_scanner:read/replay_scanner:write.prompt.ts— STEP 6c + its task-list entry.content/tips.ts— a Scanner tip.The scope name is the thing to get right
The scope object is
replay_scanner, notvision-scanners-*. Those are MCP tool names — requesting one grants nothing and the step 403s on every call. (ReplayScannerViewSet.scope_object.)Create/update also require
session_recording:readalongsidereplay_scanner:write— the API pairs them viadangerously_get_required_scopes, since a scanner's config indirectly exposes recording contents. That one is already granted for the step-2 usage probes, so only thereplay_scannerpair is net-new to the requested set.No prod OAuth-ceiling edit is needed — this corrects an earlier version of this PR (and some stale repo docs, fixed in the second commit). The live wizard apps'
OAuthApplication.scopesuse the@defaultsentinel — US prod is@default,llm_gateway:read,wizard_session:read,wizard_session:write.@defaultresolves toUNPRIVILEGED_SCOPES(every public non-privileged/non-internal/non-hidden scope) and auto-tracks new ones, soreplay_scanner:*is already inside the ceiling. Verify withpython manage.py seed_oauth_app_scopes --client-id <id> --scopes @default,… --dry-runon the posthog side. A ceiling edit is only ever needed for a privileged/internal/hidden object, which this isn't.Not a blocker either: the endpoints sit behind the
replay-visionflag and 404 when off, but its first release condition isproperties: [], rollout_percentage: 100— everyone. The skill still treats a 404 as a recorded follow-up.What the prompt carries
Only what the wizard owns — order, the never-abort contract, and the two constraints too load-bearing to leave to the skill alone: don't edit the skeletons' locked fields, and keep the scanners' queries disjoint.
That second one is documented in ARCHITECTURE §10.3 and is the non-obvious part of this whole change.
emits_signalsappends a fixed defect turn to every scan, identical regardless of the scanner's own prompt — so prompts don't differentiate scanners, queries do. And since grouping matches signals semantically rather than by source, two scanners over the same sessions describe one defect twice and sum0.5 + 0.5to exactlyWEIGHT_THRESHOLD. Overlapping scanners self-promote, defeating the half-weight design that's meant to require corroboration.Sequencing
mcp-publish.Reversed, the agent reads step 6c before the release ships the requested scope, and 403s. Note the fix there is a reconnect (refresh tokens keep their original grant), not a ceiling edit — nothing manual on the prod OAuth app. Everything here is inert until the skill ships, so this half is safe to land first.
Testing
pnpm build && pnpm test && pnpm lint— 1719 pass, 0 lint errors. Three new tests: STEP 6c present and ordered 6b → 6c → 7, its task-list entry exists (a STEP with no task silently drops off the TUI progress view), and the scope additions coverreplay_scanner:*+session_recording:readwithout weakening the base set.