feat(self-driving): add Replay Vision scanners as step 6c - #313
Open
Twixes wants to merge 1 commit into
Open
Conversation
Self-driving sets up the pull side of the inbox (sources, scouts) but not the push side. This adds step 6c: create Replay Vision scanners with `emits_signals: true`, so on-screen breakage a recording reveals — a blank screen, a dead button, a broken layout — reaches the inbox too. No new contract needed; `replay_vision`/`scanner_finding` is already a registered, self-authorizing signal source, so the flag on the scanner IS the config. Two locked `monitor` skeletons. Each pins the trust-critical bits (`scanner_type`, `emits_signals`, base prompt) and leaves the agent two blanks it's uniquely placed to fill from the repo: the `query` and a one-line product-context sentence. The count is derived, not chosen. `emits_signals` appends a *fixed* defect turn to every scan — the same one regardless of the scanner's own prompt — so prompts don't differentiate scanners, queries do. And because grouping matches signals semantically rather than by source, two scanners over the same sessions describe one defect twice and sum 0.5 + 0.5 to the promotion threshold: overlapping scanners manufacture their own corroboration. So the two here filter on deliberately different axes (URL vs `$rageclick`), and "widen one, narrow the other" is a hard rule. A third "blocked conversion" monitor would have been scanner 1 under another name; its real value was the targeting, which folded into scanner 1's query instead. Also: chain 6b → 6c → 7, note in step 4 that scanners need no source row, and scope the `signals-scout-replay-vision` pick in step 6 to scanners the team already had (6c runs later, so its scanners have no observations yet).
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pairs with PostHog/wizard#1055 — that side adds the OAuth scope and the prompt STEP. Neither half works alone.
What
Self-driving currently sets up the pull side of the inbox — signal sources and scouts. This adds the push side: a new step 6c that creates Replay Vision scanners with
emits_signals: true, so breakage a recording reveals but no event records — a blank screen, a dead button, a broken layout, a spinner that never resolves — reaches the inbox too.No new backend contract.
replay_vision/scanner_findingis already a registered, self-authorizing signal source: theemits_signalsflag on the scanner is the per-source config, so there's noSignalSourceConfigrow to create (noted explicitly in step 4, since the enum listsreplay_visionand it's an easy wrong turn).Two skeletons, and why two
Each skeleton locks the trust-critical bits —
scanner_type,emits_signals, the base prompt — and leaves the agent exactly two blanks it's uniquely placed to fill from the repo: thequeryand a one-line product-context sentence. Curated perspective, tailored targeting.sampling_rate: 0.5.$rageclick-gated.sampling_rate: 1.0.The count is derived, not picked, and this is the part worth reviewing:
emits_signalsappends a fixed defect-hunting turn to every scan (signals_step.jinja), identical regardless of the scanner's own prompt. So prompts don't differentiate scanners — queries do. The skill says so outright, so the agent spends its effort on targeting rather than wording.temporal/grouping.py), and scanner findings emit at0.5against aWEIGHT_THRESHOLDof1.0. Two scanners matching the same session therefore describe one defect twice and sum to exactly the promotion threshold — overlapping scanners manufacture their own corroboration, which is the one thing the half-weight design exists to prevent.Hence two scanners on deliberately different filter axes (where the user is vs what they did), plus a hard "widen one, narrow the other" rule. An earlier draft had a third "Blocked conversion" monitor; its query was a subset of scanner 1's, so the pair would have double-scanned the same sessions. Its real value was never the prompt wording but the targeting — insisting the agent find the actual completion path — so that folded into scanner 1's query instead.
Also
6b → 6c → 7;description.md9→10 steps.7-report.mdgains a scanners section — including that this is the only part of setup spending Replay Vision quota.6-scouts.md: thesignals-scout-replay-visionpick is now scoped to scanners the team already had. 6c runs after step 6, so its new scanners have no observations for that scout to read yet.Merging
Land the wizard side (#1055) and cut its npm release before adding
mcp-publishhere. If this skill ships first, the agent reads step 6c before the wizard requests thereplay_scannerscope and 403s.To be clear, this is not gated on any manual prod OAuth-ceiling edit — an earlier version of this note said it was, wrongly. The wizard apps' ceiling uses the
@defaultsentinel, which already coversreplay_scanner:*(see wizard#1055 for the mechanics). The only sequencing constraint is wizard-release-before-publish; a token predating the scope is fixed by a reconnect, not a ceiling edit.Testing
pnpm buildrenders the chain correctly as 10 steps with 6c between 6b and 7.pnpm test137 pass.pnpm security-scan:skillsreports zero findings inself-driving-setup(the pre-existing failures are in other skills, untouched here).