Fix wizard orchestrator remarks: stale tool names + doc gaps - #272
Draft
posthog[bot] wants to merge 1 commit into
Draft
Fix wizard orchestrator remarks: stale tool names + doc gaps#272posthog[bot] wants to merge 1 commit into
posthog[bot] wants to merge 1 commit into
Conversation
Fixes surfaced by clustering `wizard remark` telemetry (task_type=self-driving, integrate) from the last several hours: - self-driving: `products-enable` MCP tool doesn't exist (confirmed via live tool search) — add a soft-degrade instruction so the step records a follow-up instead of failing. - self-driving: `signals-scout-config-*` / `signals-scout-project-profile-get` are deprecated aliases (confirmed via `info`) — switch references to the current `scout-config-*` / `scout-project-profile-get` names. - integration (old flow): add explicit JSON-escaping guidance for the notebook-mirroring step, matching the escaping note already present in integration-v2's notebook skill — this was the single highest-volume complaint cluster. - commandments.yaml: resolve a real contradiction in the Python section — the general rule said "always use the Posthog() class constructor", but the Flask/FastAPI/Django sections (and their example apps) correctly use the module-level `posthog.api_key` API for those frameworks. Scoped the general rule to standalone scripts/CLIs. Left for later (needs the wizard repo, not available in this checkout): a possible mismatch between the audit skill's `init-not-duplicated` check ID and the wizard's ledger seed list. Generated-By: PostHog Code Task-Id: 5d7dc36c-18c9-447f-93d1-4e9722437e5b
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes derived from clustering
wizard remarktelemetry (self-driving and integrate task types) over the last several hours — each verified against the live PostHog MCP tool schema or the actual example-app code before fixing:products-enable: this MCP tool does not exist (confirmed via livesearch/toolslisting, and independently reported by ~16 orchestrator runs). Added a soft-degrade instruction so the step records a follow-up instead of stalling.signals-scout-config-*,signals-scout-project-profile-get: confirmed viainfoto be deprecated aliases (DEPRECATED: renamed to scout-config-*). Switched all references to the currentscout-config-*/scout-project-profile-getnames.notebooks-createJSON payload instead of building it as one escaped JSON value.Posthog()class constructor", but the Flask/FastAPI/Django sections (and their example apps inexample-apps/) correctly use the module-levelposthog.api_keyAPI for those frameworks. Scoped the general rule to standalone scripts/CLIs so it no longer conflicts with the framework-specific guidance.Why
An orchestrator remarks-triage loop reads
wizard remarktelemetry, clusters recurring complaints, and verifies each against the live tool schema/code before fixing — this avoids acting on any single agent's possibly-wrong self-report. Left out of scope: a report that the audit skill'sinit-not-duplicatedcheck ID isn't in the wizard's seeded ledger — the check ID is legitimate on the context-mill side, so the mismatch likely needs a wizard-repo fix (not available in this checkout).Test plan
npm test— 137 tests passnpm run build— full build succeeds, no errorsCreated with PostHog Code