feat: ai-observability test apps + evaluator rubric - #3358
Open
marco-g-pm wants to merge 1 commit into
Open
Conversation
Nine PostHog-less fixtures for `wizard ai-observability`, one per instrumentation shape (wrapper, tracing processor, experimental_telemetry, manual capture) across Python and Node. Each README states the session -> trace -> span -> generation tree the app must produce; runs are graded on emitted events, not the diff. Adds the ai-observability rubric to pr-evaluator and registers it, plus the manifest entry so wizard-ci infers the command from the app path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
marco-g-pm
marked this pull request as ready for review
August 4, 2026 21:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Nine PostHog-less test fixtures for
wizard ai-observability, plus an AIO-specific rubric for the PR evaluator.Matches the context-mill updates PostHog/context-mill#319 and the docs updates PostHog/posthog#75976
Test apps (
apps/ai-observability/)Each app exists to test one thing the others don't:
anthropic/python-weather— the baseline: tool span, single-trace inferred sessionanthropic/node-weather— multi-turn: one trace per turn, one session across turnsopenai/python-weather— same flow, OpenAI shape; canary for stale SDK resolvesopenai/node-weather— same flow, single-turn Nodeopenai/python-docs-rag— embeddings; hardest session (no field to read)groq/node-chat— provider named bybaseURL, not packageopenai-agents/python-travel-triage— tracing processor; the SDK emits the treevercel-ai/nextjs-support-chat— per-request identity; framework bootstrapmanual-capture/node-http-chat— hand-built tree; must reuse the existing clientThe four weather apps implement the identical
get_weatherround trip from Anthropic's tool-use docs, so a diff between any two isolates one variable (SDK, language, or conversation structure).Each README states the
session → trace → span → generationtree the app must produce. Runs are graded on emitted events, not the diff — every failure mode observed while building these produced plausible-looking code and a broken tree (session on a dropped key, fresh trace id per call, missing flush).Conventions: wrapper-first (
posthog.ai.*/@posthog/ai, per-call identity) with three legitimate exceptions (openai-agents→ tracing processor,vercel-ai→experimental_telemetry,manual-capture→ nothing to wrap); every app gets a session — the graded property is cardinality; spans come from tool registration, never hand-authored wrappers.Evaluator (
services/pr-evaluator/)prompts/evaluation-ai-observability.md— 28-item rubric over the existing four dimensions, targeting the looks-right-data-wrong failure class: exact$ai_session_idkey, shared trace id per request, identity scope (per-call vs Resource), flush on CLIs, dependency versions that actually provide the imported modules, no fabricated structureai-observabilitycommand id;wizard-ciinfers it from the app path🤖 Generated with Claude Code