feat(ai-observability): rebuild the skill around the SDK wrapper and the session tree - #319
Open
marco-g-pm wants to merge 15 commits into
Open
feat(ai-observability): rebuild the skill around the SDK wrapper and the session tree#319marco-g-pm wants to merge 15 commits into
marco-g-pm wants to merge 15 commits into
Conversation
…etup step The skill's success bar was 'one $ai_generation event appears', so runs that only did the OTel bootstrap passed while shipping flat, disconnected generations — no trace grouping, no session, no user attribution, and non-LLM steps invisible. Verified by running the skill twice against one unchanged fixture: one run built the tree, one didn't, both 'passed'. - 1-begin: map the app's logical structure up front (what is a session, a trace, a span-worthy step, the distinct-id source) - new 4-nesting: build the tree per mechanism — enclosing span (OTel), shared posthog_trace_id (wrapper SDKs), explicit $ai_parent_id (manual-capture); calls out the silent ai.* span-name drop rule and the Resource-vs-per-span session propagation caveat - 5-verify (was 4-verify): success is the tree — one trace holding the request's generations and spans, session grouping, person attribution; a lone generation is 'bootstrap verified, nesting unverified' - shared_docs: add sessions.md and spans.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One line per request so local wizard runs against the dev server are observable — you can watch the menu + zip fetches arrive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Round-2 delta from re-running the skill on the support-triage fixture: generations + trace grouping landed but session, user attribution, and the non-LLM span were silently dropped at ingest. The 4-nesting doc taught the client-side filter (name OR attribute key prefixed gen_ai./llm./ai./traceloop.) as the survival rule. The server (rust/capture/src/otel/fan_out.rs) keeps a span only when an attribute KEY starts with gen_ai./ai./traceloop./pydantic_ai. — names are never consulted. So the doc's own example (span named ai.support_request carrying only $ai_session_id + posthog.distinct_id) died at ingest. - OTel branch rewritten per goal: trace grouping = enclosing span (any name, zero attrs, needn't survive); session + user = two Resource attributes in the bootstrap (denylist is only host./process./os./ telemetry.); non-LLM steps = $ai_span via posthog.capture() sharing the OTel trace id; per-request-on-OTel = opentelemetry-processor- baggage, never hand-rolled SpanProcessors - new decision gate: wrapper (per-request identity) → OTel + Resource (existing OTel / multi-backend) → manual capture; imports must run in the project's env or you switch paths - 5-verify: prove-the-import rule (posthog 6.9.3 ships no posthog.ai.otel; 7.29.0 does), failure-mode table updated to the attribute-key rule - do-not list: ai.* naming rule removed (neither necessary nor sufficient), session-only-on-hand-span added Verified against posthog monorepo rust capture source and the 6.9.3 / 7.29.0 wheels on PyPI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…/wiring on it
The round-2 gate lived in 4-nesting.md — but by then the agent had
already declared OTel packages (step 2) and written the OTel bootstrap
(step 3), so OTel was still the de-facto default and the wrapper choice
meant ripping out finished work.
- 1-begin: 'Choose the mechanism' gate right after the structure map —
existing-OTel/multi-backend → OTel; variant has a PostHog wrapper
(verified: posthog 7.29.0 ships posthog.ai.{openai,anthropic,gemini,
langchain,openai_agents,claude_agent_sdk}; @posthog/ai ships
{openai,anthropic,gemini,vercel}) → wrapper; else OTel; no vendor
SDK → manual capture. Explicit 'do not default to OTel'.
- 2-install: packages per mechanism; wrapper path = posthog /
@posthog/ai + posthog-node, zero OTel packages
- 3-otel-setup.md renamed 3-instrument.md: wrapper client-swap section
added, OTel + manual sections scoped per path
- 4-nesting: gate section replaced with a pointer to 1-begin
- description.md: steps and principles updated to match
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ssion-is-the-job Supersedes the enclosing-span/hand-authored-span model. Two rules now drive the skill: spans come from the app's own tool registration, and the wizard's job is the bootstrap plus the session id. - 1-begin: variant picker now reaches all 68 variants — agent frameworks before providers, and a baseURL/base_url check so OpenAI-compatible gateways (Groq, OpenRouter, Together, ...) stop misrouting to openai-*. The four structure questions collapse to the two that matter: does the app register tools, and what identifies one conversation. - 4-nesting: rewritten. Spans are never hand-authored; a tools-less app correctly has none. Sessions are always set, and cardinality is the graded property. Documents that a raw provider SDK + OTel instrumentor emits no tool spans at all — it patches only the SDK's own methods and records tool use as attributes on the generation. - 5-verify: grades what lands in PostHog rather than what the diff contains. - commandments: ai_observability carve-out for rules that assume a Posthog() client the OTel path never creates, and for the PII rule that contradicts $ai_input carrying the user's prompt. - skill-generator: dedupe tags before collecting commandments — variants re-declare group-level tags, which doubled every carve-out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…able The four mapping questions collapsed to two last commit, which dropped the distinct-id source entirely — while 5-verify still graded attribution. Step 1 now asks for conversation and user together (they share a placement rule), and step 4 states wiring both as the task rather than showing the distinct id only incidentally in code samples. Also cuts the 15-row baseURL host table. Verified all 17 gateway variants install identical packages and the identical OpenAIInstrumentor bootstrap, so the routing choice affects only how the provider is named in the report — not the generated code. The claim that misrouting breaks cost attribution was wrong: $ai_provider is derived at runtime, not from the variant picked. Portkey's extra portkey-ai package is the one real difference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…itself Feedback from an openai-python run: the generated install wrapped a five-line bootstrap in an init function with a module global and a custom env guard. Three of the four causes were ours. - 3-instrument: new 'Match the doc's shape' rule. Copy the install doc's block at module level, change only the values. No init function, no globals, no extra env scaffolding. Names the actual trigger: when a Resource value isn't available at module scope, that means the app needs per-call identity, not a deferred bootstrap. - 4-nesting: identity placement now prefers per-call wherever the mechanism offers it, with a per-mechanism table. Resource is documented as the raw-OTel constraint it is (the instrumentor owns the span; OTel doesn't inherit attributes parent->child) rather than the default. Removes a duplicated Resource block, and adds the $ai_session_id charset restriction from sessions.md that the rewrite had missed. - commandments: the 'all' missing-configuration rule mandates a custom raise, which is what produced the guard block. Carve-out notes that a direct os.environ lookup already fails loudly and idiomatically, so no separate presence check is warranted around a short bootstrap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Matches PostHog/posthog#75976 ("update docs to match SDK use over OTel"), which rewrites all 40 AIO installation pages: OTel out (-207 mentions, -92 PostHogSpanProcessor, -46 TracerProvider), wrapper client + manual $ai_span in (+69). The stated reason is that OTel makes the full session tree hard to build and maintain, which is this skill's whole deliverable. - 2-install: providers and gateways now install just the PostHog SDK beside the vendor SDK. No opentelemetry packages at all. OTel packages narrowed to the opentelemetry-* variants and LlamaIndex. - 3-instrument: wrapper client is the default mechanism; OTel demoted to the variants whose docs still call for it. Adds the gateway base_url shape. - 4-nesting: rewritten. Identity is per call (posthog_distinct_id, posthog_trace_id, posthog_properties). Trace id is per turn and must be shared across a turn's calls. Reverses the previous rule against authoring spans: the docs now require capturing each tool execution as an $ai_span event, since the wrapper never sees the app's dispatch loop. Framework variants still emit their own. - 1-begin: restores the gateway-provider consequence, now real for the right reason - the wrapper reports openai whatever host it targets, and pricing keys on $ai_model + $ai_provider (see PostHog/posthog-python#822). - 5-verify: expectations follow the new tree, including a $ai_provider check. The three upstream PRs are still open, so the bundled install pages in each zip remain the old OTel content until they merge and the docs are re-fetched. 3-instrument carries a note that the wrapper shape wins over a stale bundled page; it can come out once #75976 lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
REVERT BEFORE PostHog/posthog#75976 MERGES. The SDK-first procedure landed in d14ddb2, but the bundled install pages still came from posthog.com, which serves master — so every zip paired wrapper instructions with an OTel reference doc and the agent got contradictory material. posthog.com renders those pages from the monorepo onboarding .tsx at build time (gatsby onPostBuild -> generateRawMarkdownPages), so there is no local or preview URL carrying the new content: `gatsby develop` never emits the .md files, and the posthog.com PR preview sources the monorepo from master. Pointing docs_urls straight at the PR branch's raw .tsx is the only way to see the real new content before merge. fetchDoc does no format validation, so the TSX inlines fine; urlToFilename appends .md, hence openai.tsx.md. Pinned (7 URLs, 6 variants — the ones with workbench fixtures): openai-python, anthropic-python, vercel-ai, manual-capture, groq-node, openai-agents. groq-node takes two files because groq.tsx is now just a config object and _snippets/openai-compatible.tsx carries the steps. On merge the branch is deleted and these 404. fetchDocOnce treats 404 as non-retryable, and the stale-cache fallback only helps where an entry already exists, so leaving these in place breaks the build — and these URLs compile into manifest.json, which the wizard fetches. Revert this commit, then `rm -rf .docs-cache && npm run build`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PostHog/posthog#75976 rewrote 40 onboarding pages to the SDK wrapper but did not touch aws-bedrock.tsx, which still bootstraps a TracerProvider and instruments the AWS SDK (opentelemetry-instrumentation-botocore in Python, @opentelemetry/instrumentation-aws-sdk in Node). Bedrock has no wrapper client, so it cannot follow the new default. d14ddb2 claimed OTel was left to the opentelemetry-* variants and LlamaIndex only, and that every direct provider uses the wrapper. Both were wrong for Bedrock. Corrected in the 2-install package guidance and the 3-instrument mechanism table. Found by diffing the skill's 43 variant frameworks against the 42 onboarding pages on the PR branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… branch" This reverts commit 7fa1837.
3-instrument told the agent that a bundled page still showing a TracerProvider bootstrap "predates the SDK-first rewrite" and that the wrapper wins. That was true only while the docs were unmerged. Now that posthog.com serves the rewritten pages it is actively wrong: AWS Bedrock legitimately shows a TracerProvider (it was never part of #75976 and has no wrapper client), so the note would push a Bedrock run onto a mechanism that does not exist for it. Follows the revert of the PR-branch pin in 85f516d. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The repo is pnpm — main tracks pnpm-lock.yaml and no package-lock.json. An npm install during this branch's work generated one, and since it is not gitignored it got committed in b3c0dec, adding 2815 lines of noise to the diff for zero dependency change (package.json itself was never touched). pnpm-lock.yaml is unmodified, so nothing about the dependency set changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three files overlapped: - references/3-otel-setup.md — modify/delete. Kept deleted: this branch replaces it with 3-instrument.md. Main's only change (#296) was repointing one manual-capture doc reference to its canonical URL, and the rewritten files carry no /docs/ai-observability/* paths at all, so nothing was lost. - context/commandments.yaml — auto-merged. Main's new rust/wordpress sections and this branch's ai_observability carve-outs are additive and both survive. - scripts/dev-server.js — auto-merged. Main's /agents/ route fix and this branch's request logging touch different lines. 140 tests pass (up from 138 — main added two). Full build emits all 68 ai-observability zips with the 1-begin..5-verify step files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…in half The bundle size gate failed at 8,644,625 bytes against an 8 MB limit. Main builds 7,978,436 and passes, so the 666 KB this branch added caused it. The step files had grown from 14,554 to 25,843 bytes per variant, times 68. Rewrote all four steps with the ste-writing skill (ASD-STE100 controlled English) and its bundled linter. Every file now passes the strict gate: 1-begin.md 0.26 per 100 words, longest sentence 15 words 2-install.md 0.00 16 3-instrument.md 0.00 20 4-verify.md 0.32 18 The three flagged items left are false positives: the word "begin" in the step title, and "generation of" where generation is the event type. Structure: 4-nesting.md folds into 3-instrument.md, and 5-verify.md returns to 4-verify.md. Identity and tool spans belong with the client swap, not in a separate step. The published docs now carry the code and property tables, so the skill keeps only what the docs cannot know: which variant matches this codebase, which four facts to read from it, the cardinality bar, the gateway provider override, and how to grade what reaches PostHog. Per variant: 25,843 -> 11,255 bytes. Bundle: 8,245,278, under the limit by 139 KB. That headroom is thinner than main's 400 KB. Dropping python.md (32,589 bytes in every Python variant, mostly flags and surveys) is the next lever if more room is needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rewrites the
ai-observabilityskill to follow PostHog/posthog#75976, which moved the AI Observability installation docs off OpenTelemetry and onto the PostHog SDK wrappers.Why
The docs PR states the reason directly: "OTel doesn't give the best developer experience to end up with a full session tree (session, traces, spans, generations). It makes it difficult to capture sessions in an easy to develop, maintain, and understand way." That session tree is exactly what this skill exists to produce, so the skill had to follow.
Counting tokens across that diff shows how total the reversal is:
opentelemetryPostHogSpanProcessorTracerProviderposthog_trace_id$ai_spanopenai.tsxends up with zero OTel mentions, and all fifteen OpenAI-compatible gateways collapse into one shared snippet.What changed
The skill previously stopped at the bootstrap: a successful run captured individual
$ai_generationevents but never built the session → trace → span → generation tree. Running it twice against one unchanged fixture produced the full tree once and bootstrap-only the other time — both passing the skill's own bar. The steps are now:1-begin— variant picker reaches all 68 variants (frameworks before providers,baseURLcheck so gateways stop misrouting toopenai-*). Asks what identifies the conversation, the user, and one turn.2-install— providers and gateways install just the PostHog SDK beside the vendor SDK. No OpenTelemetry packages.3-instrument(renamed from3-otel-setup) — wrapper client is the default mechanism. Copy the doc's shape: no init function, no module globals, no extra env scaffolding.4-nesting(new) — identity is per call; the trace id is per turn and shared across that turn's calls. Tool executions are captured as$ai_spanevents, since the wrapper never sees the app's dispatch loop.5-verify(renamed from4-verify) — grades what lands in PostHog rather than what the diff contains.Two supporting fixes:
commandments.yaml— carve-outs for rules that assume aPosthog()client this skill's paths don't create, and for the PII rule that contradicts$ai_inputdeliberately carrying the user's prompt.skill-generator.js— dedupe tags before collecting commandments. Variants re-declare group-level tags, which was doubling every rule. Covered by a test.Verified against source, not assumption
opentelemetry-instrumentation-botocore. Found by diffing this skill's 43 variant frameworks against the 42 onboarding pages.opentelemetry-instrumentation-anthropicwheel: it patches only the vendor SDK's own methods, andtracer.start_spanappears exactly twice in the package. Tool use is recorded as attributes on the generation, never as a child span. This is why manual$ai_spancapture is now the documented answer.$ai_provider.posthog.ai.openaireportsopenaiwhatever host it targets, and PostHog prices by$ai_model+$ai_provider— see feat(ai): add posthog_provider_override to the OpenAI wrapper posthog-python#822.Not done
Not yet exercised end to end against a workbench fixture. A local run is what surfaced most of the findings above, but the most recent attempt failed for an unrelated reason: with the skills registry unreachable, the wizard presents
load_skill_menuwith a truncated category enum, so the agent concluded the skill did not exist and improvised. Worth a green run before this leaves draft.Two doc pages have no variant here —
vercel-ai-gateway(a normal OpenAI-compatible gateway, config-only to add) andeve. Left alone since new variants change the wizard's command surface.🤖 Generated with Claude Code