Add clarifying-questions step to the dispatch wizard (PK-283) - #64
Draft
posthog[bot] wants to merge 1 commit into
Draft
Add clarifying-questions step to the dispatch wizard (PK-283)#64posthog[bot] wants to merge 1 commit into
posthog[bot] wants to merge 1 commit into
Conversation
The interactive dispatch wizard previously only guarded against an empty task string, forwarding vague or risky instructions straight to the agent and burning a run on throwaway work. Add an optional LLM-backed clarification step between the task prompt and the confirm/dispatch: it reviews the instruction, surfaces concerns, and offers to fold answers to a few clarifying questions back into the refined prompt. The check follows the repo's port pattern (ClarifierPort behind an interface, real port wrapping the Anthropic SDK, pure parse/refine helpers unit-tested with fakes). It is fail-open — a missing Anthropic key or any error leaves the original prompt untouched so a dispatch is never blocked. Also add best-effort PostHog (posthog-node) instrumentation for the wizard, since the CLI previously captured nothing. Analytics no-ops when unconfigured and never throws. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Generated-By: PostHog Code Task-Id: 4dc8a91f-4df5-4258-9d18-d8dcd0c09977
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The interactive dispatch wizard (
src/commands/dispatch-interactive.ts) previously only checked that the task string wasn't empty before forwarding it todispatch()— a vague, ambiguous, or risky instruction was dispatched as-is, burning an agent run on throwaway work.This adds an optional LLM-backed clarification step between the task prompt and the confirm/dispatch:
ClarifierPortbehind an interface, a real port wrapping the Anthropic SDK, and pureparseAssessment/refinePrompthelpers that are unit-tested with fakes (no key needed).It also adds best-effort PostHog instrumentation (
posthog-node) to the wizard, since the CLI previously captured nothing. Analytics no-ops when unconfigured (POSTHOG_API_KEYunset) and never throws.Why
PK-283 (backlog): the wizard fires tasks off without ever pushing back on flawed or underspecified instructions. Catching those before spending an agent run is a quality win, and instrumenting the flow makes it measurable for the first time.
Test plan
npm test— 47 pass, including newtest/clarify.test.tscovering JSON parsing (fences, malformed → fail-open, question cap) and prompt refinement.npm run buildandtsc --noEmitclean.captureno-ops without a key; clarifier parse/refine helpers produce expected output.Created with PostHog Desktop from this inbox report.