Skip to content

(DONT-REVIEW-YET) feat(self-driving): Add the self-driving program to set up PostHog Self-driving#642

Open
sortafreel wants to merge 42 commits into
mainfrom
feat/product-autonomy
Open

(DONT-REVIEW-YET) feat(self-driving): Add the self-driving program to set up PostHog Self-driving#642
sortafreel wants to merge 42 commits into
mainfrom
feat/product-autonomy

Conversation

@sortafreel

@sortafreel sortafreel commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Problem

  • We didn't have a program like that

Changes

  • Now we do
  • A new self-driving command and program, gated behind its own program id. It requires the base posthog-integration flow to have run first.
  • A 8-step agent run: check access, read project state, connect GitHub, enable signal sources, offer issue-tracker integrations, configure the scout fleet, design custom scouts, write the report.
  • Its own intro screen (with prerequisite detection), a Signals-flavored set of sidebar tips, and a clean outro that links straight to the inbox.
  • Twelve extra OAuth scopes (task, integration, signal scout, a few read-only product probes, external data source, llm skill) that are requested only when this program runs.

Shared Wizard changes — all optional, nothing else is affected

  • Sidebar tips during the run. A program can supply its own tips; everyone else keeps the default set.
CleanShot 2026-06-19 at 19 13 04
  • Intro subtitle. A screen can pass its own subtitle line; unset means the usual generic copy.
CleanShot 2026-06-19 at 17 29 00
  • Outro link and "what's next". The outro can now show a link and a short next-steps list. Self-driving uses these for the inbox URL and a few inbox actions. Programs that don't set them get the outro they already have.
CleanShot 2026-06-19 at 17 29 21
CleanShot 2026-06-19 at 23 19 44
  • Per-option descriptions on multi-select questions. An option can carry a one-line explanation shown under its label. Questions without descriptions look and space identically to before.
CleanShot 2026-06-19 at 15 38 37
  • Clickable, copyable links in questions. Opt in with richLinks and long URLs render as terminal hyperlinks (with a clipboard fallback) instead of wrapping into garbage. Off by default, so other prompts are untouched.
CleanShot 2026-06-19 at 16 01 56

Test plan

@github-actions

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci basic-integration
  • /wizard-ci error-tracking-upload-source-maps
  • /wizard-ci misc
  • /wizard-ci revenue

Test an individual app:

  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
Show more apps
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci error-tracking-upload-source-maps/android
  • /wizard-ci error-tracking-upload-source-maps/flutter
  • /wizard-ci error-tracking-upload-source-maps/ios
  • /wizard-ci error-tracking-upload-source-maps/next
  • /wizard-ci error-tracking-upload-source-maps/next-no-posthog
  • /wizard-ci error-tracking-upload-source-maps/node-raw
  • /wizard-ci error-tracking-upload-source-maps/node-rollup
  • /wizard-ci error-tracking-upload-source-maps/node-rollup-typescript-plugin
  • /wizard-ci error-tracking-upload-source-maps/node-webpack
  • /wizard-ci error-tracking-upload-source-maps/nuxt-3-6
  • /wizard-ci error-tracking-upload-source-maps/nuxt-4-3
  • /wizard-ci error-tracking-upload-source-maps/react-native
  • /wizard-ci error-tracking-upload-source-maps/react-vite
  • /wizard-ci error-tracking-upload-source-maps/rust
  • /wizard-ci misc/quack-quack
  • /wizard-ci revenue/stripe

Results will be posted here when complete.

sortafreel commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

sortafreel and others added 12 commits June 11, 2026 16:22
…ons is reachable

evaluateAskCap rejected every call once askCallCount reached
ASK_BATCH_THRESHOLD (3), and the rejected call never advanced the
counter — so the nudge repeated forever and any maxQuestions above 3
was unreachable. Track the nudge in the server closure and pass it to
evaluateAskCap so the batch-your-questions error fires exactly once,
after which sequential asks proceed up to the real cap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New skill-based program that sets up PostHog Product Autonomy
(Signals) for an already-onboarded project. The agent runs the
context-mill product-autonomy-setup skill to enable the right signal
sources, require the GitHub integration, confirm org AI-data-processing
approval, and materialize + tune the scout fleet, then writes
posthog-product-autonomy-report.md.

- detect step requires posthog-setup-report.md; custom intro screen
  renders the typed detect errors
- scope additions: task:read/write (signal source configs live under
  the task scope object), integration:read, signal_scout:read/write
- requestDeepLink learns optional {purpose, path} so postRun can mint
  an inbox deep link for the outro
- no keep-skills step: the transient setup skill is removed in postRun
- .gitignore re-includes src/lib/ so user-global lib/ ignore patterns
  cannot silently drop program sources from commits

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sortafreel sortafreel force-pushed the wizard/add-logs-fix-local-rins branch from 9fcc815 to 6f1c211 Compare June 11, 2026 14:22
@sortafreel sortafreel force-pushed the feat/product-autonomy branch from 2d86ad7 to af15908 Compare June 11, 2026 14:22
@sortafreel sortafreel changed the base branch from wizard/add-logs-fix-local-rins to graphite-base/642 June 11, 2026 14:25
@sarahxsanders

Copy link
Copy Markdown
Collaborator

hey @sortafreel!! just a heads up, once the CLI overhaul merges tomorrow AM you're gonna hit a bin.ts conflict when you rebase. it rewrites the .use() command chain and tweaks command.ts.

resolution should be straightforward, just re-add your import { autonomyCommand } and .use(autonomyCommand) to the new chain. your command itself needs no changes: autonomyCommand is a flat Command and the overhaul only added optional fields to the interface, so it stays compatible :)

@sortafreel

sortafreel commented Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

Validated review of this PR (self-review)

I re-ran the AI review report against the actual code on this branch: each of the 22 findings was checked by a reviewer agent and then adversarially re-checked by a second agent (44 agents total). Kept = a real problem that could plausibly happen in production; dropped = wrong about the code or purely theoretical. The adversarial pass corrected 9 findings (notably walking three original must_fix items down to should_fix once the real blast radius was traced).

Each kept finding is posted as an inline comment on the diff (reviewer voice) with a threaded reply in an agent-written author voice (labelled Agent response — these replies were drafted by the validation agents, not by me) — both under my account, so the reasoning is visible on the line.

Result: 20 kept, 2 dropped.

Kept

ID Title Severity Verdict
1-1-1 Stale "do-not-commit" scratch handoff docs encode pre-rename product-autonomy naming should_fix ✅ keep
1-2-1 self-driving command forwards --signup/--ci without rejecting modes that break the run should_fix ✅ keep
2-1-1 Committed scratch plan documents a stale, over-privileged OAuth/AI-consent contract should_fix ✅ keep
2-2-1 Self-driving OAuth scopes exceed the documented app ceiling; no guard test should_fix ✅ keep
2-2-2 No MCP-tool authorization boundary for self-driving's broad write scopes (prompt text is the only limit) should_fix ✅ keep
2-3-1 OSC 8 rich links render agent-controlled URLs without stripping C0/C1 control bytes should_fix ✅ keep
2-4-2 Rich-link mode auto-copies agent-controlled URLs to the clipboard with no host validation should_fix ✅ keep
2-7-1 Setup-report prerequisite uses existsSync (follows symlinks, accepts dirs) before agent reads it as ground truth should_fix ✅ keep
2-7-2 Custom-scout step (STEP 7) lacks a content constraint, leaving scout bodies as an unscanned secret/PII egress channel should_fix ✅ keep
3-5-1 Rich-link URLs can be unrecoverable when terminal lacks both OSC 8 and clipboard should_fix ✅ keep
3-6-1 Self-driving inbox link lost from scrollback after outro is dismissed should_fix ✅ keep
3-7-2 Transient self-driving setup skill survives every abort path (postRun cleanup never runs) should_fix ✅ keep
3-7-3 Self-driving wizard_ask uses 5-min default timeout despite required browser handoffs (GitHub App install, Linear OAuth) should_fix ✅ keep
1-3-1 Prompt AI-consent field reads stale local user after the opt-in gate clears nice_to_have ✅ keep
1-5-1 URL punctuation trimming can corrupt link targets ending in stripped chars nice_to_have ✅ keep
1-6-1 Self-driving Tips pane hidden ~70s behind the shared agent-skill Learn deck nice_to_have ✅ keep
1-7-2 Prompt's "never polls to confirm a connection" blanket-disables the silent verify the design requires for the GitHub Issues / Linear auto-connect path nice_to_have ✅ keep
2-3-2 requestDeepLink forwards opts.path into the authenticated-redirect API without validation nice_to_have ✅ keep
3-4-2 askQuestionSchema accepts unbounded option arrays and description strings that the multi-select overlay renders without any height clamp nice_to_have ✅ keep
3-7-1 Self-driving completion bypasses runner graceful shutdown (no keep-skills step) nice_to_have ✅ keep

Dropped

ID Title Verdict
3-5-2 Clipboard subprocess has no timeout/kill, but cannot hang or zombie the CLI ❌ drop
1-7-1 Self-driving prompt does not claim AI approval is guaranteed; consent is gate-owned ❌ drop

3-5-2 — Clipboard subprocess has no timeout/kill, but cannot hang or zombie the CLI ❌

The mechanism is accurately described — no timeout, close-only resolve, and the effect ignores but doesn't kill the child — but the finding's headline harm is refuted by the code. It cannot "hang the CLI" or "keep the wizard process alive after the overlay exits": every termination path calls process.exit() (ExitScreen.tsx:12; start-tui.ts:92-95; wizard-abort.ts:93), and process.exit() tears down any lingering child immediately. start-tui.ts:88-91 documents exactly this defense against background handles producing a "zombie wizard." The call is also fire-and-forget (void copyToClipboard(...) in WizardAskScreen.tsx:55), so it never blocks the UI thread — a stuck helper just leaves copied false. The only genuine residual is Linux-specific, cosmetic degradation: a hung wl-copy stops us trying xclip/xsel, so the "Link copied" hint never shows — but the URL is still rendered as plain text and as an OSC 8 link, so the user can always paste it. macOS/Windows have a single command and no fallback chain to degrade. The lingering subprocess during the session is real but harmless (killed at exit). I'll add a 1s timeout + kill and ignore-stdout stdio as a small robustness/hygiene improvement, but this is nice-to-have, not a should_fix CLI-hang risk.

1-7-1 — Self-driving prompt does not claim AI approval is guaranteed; consent is gate-owned ❌

Agree this should be dropped. The quoted text ("approval is guaranteed... do not abort") does not exist in the prompt, and the cited lines are the GitHub step. Org AI consent is deliberately not the prompt's job: withAiOptInGate parks the runner before any source leaves the machine on the interactive path, using the strict is_ai_data_processing_approved===true reading (ai-opt-in-gate.ts), and ARCHITECTURE.md §6.3 spells out that neither prompt nor skill has an AI step anymore. The CI auto-consent the finding leans on (gate: session.ci || aiApproved; LoggingUI auto-resolves) is the documented intended behavior for the wizard's own Claude usage, and it cannot silently succeed: a CI self-driving run aborts at the first wizard_ask-needing step with [ABORT] requires-interactive-mode (a handled abort case), and --ci is disabled in published builds anyway. The only residual — that the backend Signals emit gate also fail-closes on org approval, independent of the wizard's gate — is a backend emit-gating concern, not a bug in this prompt, and isn't reachable today on this beta-flagged interactive flow.

Comment thread PRODUCT-AUTONOMY-HANDOFF.md Outdated
Comment thread src/commands/self-driving.ts
Comment thread PRODUCT-AUTONOMY-PLAN.md Outdated
Comment thread src/lib/oauth/program-scopes.ts
Comment thread src/lib/oauth/program-scopes.ts
Comment thread src/ui/tui/primitives/link-helpers.ts
Comment thread src/ui/tui/screens/WizardAskScreen.tsx
Comment thread src/lib/programs/self-driving/detect.ts Outdated
Comment thread src/lib/programs/self-driving/prompt.ts
Comment thread src/ui/tui/primitives/LinkText.tsx
Comment thread src/lib/programs/self-driving/index.ts
Comment thread src/lib/programs/self-driving/index.ts
Comment thread src/lib/programs/self-driving/index.ts
Comment thread src/lib/agent/agent-runner.ts
Comment thread src/ui/tui/primitives/link-helpers.ts
Comment thread src/ui/tui/screens/RunScreen.tsx
Comment thread src/lib/programs/self-driving/prompt.ts
Comment thread src/utils/provisioning.ts
Comment thread src/lib/wizard-tools.ts
Comment thread src/lib/programs/self-driving/steps.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants