RFD for local telemetry recording by gmemuriuki · Pull Request #273 · symposium-dev/symposium

gmemuriuki · 2026-08-11T11:18:37Z

What does this PR do?

Define a consented, local-only telemetry contract covering resolution, sessions, commands, hook reliability, and Claude skill activation. Document privacy boundaries, storage behavior, user controls, extensibility, and the staged implementation plan.

Disclosure questions

AI disclosure.

The AI tool authored large parts of the code

Questions for reviewers.

jackh726

A few thoughts

jackh726 · 2026-08-13T18:54:36Z

+| Q1  | Do opted-in installations return after first observed use?          | Deduplicate `session_start` by `retention_subject` and measure observed cohort days D1, D7, and D30.                               |
+| Q2  | Which plugins and skills resolve?                                   | Count `extension_resolution` occurrences and scoped subjects by public extension and safe witnessed path.                          |
+| Q3  | Which public packages and versions occur, and what do they resolve? | Count `package_resolution` by public coordinate and `extension_match`; use paths carried by `extension_resolution`.                |
+| Q4  | Are Symposium and plugin hooks failing or slow?                     | Use daily invocation/attempt counters, outcomes, fixed latency histograms, and complete identified-session impact counts.          |
+| Q5  | Which agents, versions, and platforms have reach?                   | Keep configured-agent observations separate from observed hook sessions.                                                           |
+| Q6  | Which command surfaces are used?                                    | Count completed built-ins and eligible public plugin commands without arguments.                                                   |
+| Q7  | Which resolved public skills does Claude actually activate?         | Count completed `extension_invocation_metrics` and complete identified-session counts by public skill and safe resolution subject. |


Hmm, did you come up with these questions? These overall seem okay, but I don't (for example) really understand what "Do opted-in installatons return after first observed use?" means.

Yes, I pictured that we want telemetry to answer two questions:

Does Symposium work, for whom, and at what price?

Does it help?

So this RFD is for answering the first question. These are basically sub-questions under the umbrella.

What the first question mean is that we'd want to measure whether we are being invoked again after first observed use. I imagine we'd want to know that symposium still is configured and working properly after being installed, and participating in agent sessions.

I'll change the word "return" to "invoked" to help make it clearer.

jackh726 · 2026-08-13T18:54:42Z

+
+## `enable`
+
+`enable` presents the exact disclosure below. Interactive `cargo agents init` uses the same text. Both default to no.


Do you think we need the exact printed text here?

Not necessarily this. I think we'd want a shorter version of this and still present our intentions clearly. This here is something I thought I could start with and hear what you all think about it.

jackh726 · 2026-08-13T18:58:21Z

+
+### Identifiers and correlation boundaries
+
+On first enabled recording, telemetry atomically creates a random 32-byte identity key in `state.toml`. It derives the first 128 bits of HMAC-SHA-256 over a domain, locally anchored 30-day window, and exact dimension:


Not sure if I've missed it: how are these stored? I would expect that we would need to store them for consistency over the period, but they probably should be stored with the anonymized data? And, possibly, in a very different place so there's less of a chance of somebody accidentally giving us that "key"

No you haven't missed anything. Initially the plan was to have them in the telemetry folder (which is shareable). Now, I agree with you, it makes sense to have them in the config, plus have them anonymized.

jackh726 · 2026-08-13T19:00:27Z

+
+Safe nodes are public `package` and `extension` coordinates, `all` contributors, the successful `any` branch, an opaque `not` marker, or `opaque` with a fixed reason (`private_source`, `non_package_predicate`, or `limit`). Shell commands, paths, environment values, custom predicate details, workspace members, wildcards, private names, and a negated child never enter the path.
+
+Paths are bounded to 8 levels, 16 evidence leaves, and 4 KiB; an over-limit subtree becomes `opaque: limit`. Full sync builds safe evidence for successful installations because the generated attribution index needs it even when telemetry is disabled; only an enabled recorder serializes that evidence as telemetry. Existing cached booleans for non-package predicates may synthesize `opaque: non_package_predicate`, but caching an entire `PredicateSet` would lose successful branches and witnesses and requires revisiting this design.


That 8 levels: I assume that the parent-most level is just the total of the rest of the path?
i.e.

/d1/d2/d3/d4/d5/d6/d7/d8/d9/ would have /d1/d2/ be one "level" as far as recording?

No, the purpose here is to have some sort of an evidence tree that describes why a plugin or skill matched it's predicate.

Something like:
let's say we have an extension with this condition:

all( depends-on(tokio), any( depends-on(serde) custom(has-runtime-support) )

if tokio and serde match, the evidence would look like this:

all # level 1 | _ package(tokio, version) # level 2 (leaf) | _ any( # level 2 |_ package(serde, version)) # level 3 (leaf)

We don't record directory components or filesystems path.

I'll reword the text to be more clear.

jackh726 · 2026-08-13T19:01:49Z

+This matrix reports current adapter capability and test coverage, not telemetry priority. The producer contract is agent-neutral; unavailable values remain optional or unsupported as shown.
+
+
+| Agent          | Configuration | `SessionStart` | Session id | Fresh/resume | `Stop` | Skill invocation            |
+| -------------- | ------------- | -------------- | ---------- | ------------ | ------ | --------------------------- |
+| Claude Code    | yes           | yes            | yes        | yes          | yes    | attempted/completed/failed |
+| Codex CLI      | yes           | yes            | yes        | yes          | no     | unsupported                 |
+| GitHub Copilot | yes           | yes            | no         | no           | no     | unsupported                 |
+| Gemini CLI     | yes           | yes            | yes        | no           | no     | unsupported                 |
+| Kiro           | yes           | yes            | yes        | no           | no     | unsupported                 |
+| OpenCode       | yes           | no             | n/a        | n/a          | no     | unsupported                 |
+| Goose          | yes           | no             | n/a        | n/a          | no     | unsupported                 |


Hmm. Is this "Implementation status" or "Expected coverage". The former I would expect to just be in a separate section.

In general, I'm not sure if this makes sense as part of a design doc anyways.

I pictured it as version 1 producer coverage, it makes sense here instead of not having one.

jackh726 · 2026-08-13T19:05:50Z

+
+Recorders make one non-waiting exclusive-lock attempt. Contention drops the entire buffered batch or aggregate observation. Event batches serialize before one append so concurrent lines cannot interleave. Snapshot updates use same-directory temporary replacement; no `fsync` is promised, so a crash can still lose the latest update. Contribution counts detect state/snapshot divergence and permanently mark affected daily session counts incomplete.
+
+The event file, aggregate snapshot, and reserved maximum-size `storage_limit` row share 8 MiB per day. Aggregate metrics receive at most 512 KiB. An oversized metric update is dropped without stopping low-volume events; an ordinary batch that cannot fit is replaced by the daily marker and ordinary recording stops for that day. Relationship batches are never split.


Hmm, is it worth having this storage limit? At least like this? 8 MiB seems both small (given storage typically available) and reasonable (given it's just text).

Yes. This is some sort of a safety ceiling/ boundary. Not an expected volume or preallocation.

With 8MiB, through 31 day that is approximately 248MiB, which is fine.

(I do find myself out of storage and I have to delete the target folder(s) every time I open my pc. So, by chance I run to a folder that says it stores telemetry and is occupying like 1GB or so, I wouldn't even wait for the program to do the housekeeping. I delete the files there and then. 😆 😆..)

My point here is maybe this is small but we need to have a limit, something could go wrong and we get a bug that is invoking recording in a loop, or such kind of things.

jackh726 · 2026-08-13T19:07:49Z

+
+### Does a completed skill activation mean the skill helped?
+
+No. It means Claude successfully activated the installed skill. It does not show whether Claude followed the instructions or whether the task result improved. That causal question requires a controlled evaluation comparing equivalent runs with and without the skill.


Claude -> an agent

Aah yeah. I'll generalize. Thanks.

jackh726 · 2026-08-13T19:10:13Z

+      - [Proposed: What Symposium records](./rfds/telemetry-recording/proposed-data-collected.md)
+      - [Proposed: `cargo agents telemetry`](./rfds/telemetry-recording/proposed-reference-telemetry.md)
+      - [Proposed: Telemetry configuration](./rfds/telemetry-recording/proposed-configuration-telemetry.md)


I'm not sure these need "proposed"

Totally. Fixing them asap.

Define a consented, local-only telemetry contract covering resolution, sessions, commands, hook reliability, and Claude skill activation. Document privacy boundaries, storage behavior, user controls, extensibility, and the staged implementation plan.

gmemuriuki · 2026-08-14T11:58:38Z

Thanks Jack. Anything that is not clear, please flag it. Happy to reword it.

Clarify consent coverage, agent-neutral skill activation attribution terminology, private state handling, retention and implementation boundaries. Co-authored-by: Codex <codex@openai.com>

gmemuriuki force-pushed the telemetry/rfd-recording branch from 71c37b4 to 765d445 Compare August 11, 2026 12:34

jackh726 reviewed Aug 13, 2026

View reviewed changes

gmemuriuki force-pushed the telemetry/rfd-recording branch from 765d445 to ddb4b42 Compare August 14, 2026 11:24

docs(rfd): refine telemetry recording design

0fb0aca

Clarify consent coverage, agent-neutral skill activation attribution terminology, private state handling, retention and implementation boundaries. Co-authored-by: Codex <codex@openai.com>

gmemuriuki requested a review from jackh726 August 17, 2026 09:41


		## `enable`

		`enable` presents the exact disclosure below. Interactive `cargo agents init` uses the same text. Both default to no.


		### Identifiers and correlation boundaries

		On first enabled recording, telemetry atomically creates a random 32-byte identity key in `state.toml`. It derives the first 128 bits of HMAC-SHA-256 over a domain, locally anchored 30-day window, and exact dimension:


		Safe nodes are public `package` and `extension` coordinates, `all` contributors, the successful `any` branch, an opaque `not` marker, or `opaque` with a fixed reason (`private_source`, `non_package_predicate`, or `limit`). Shell commands, paths, environment values, custom predicate details, workspace members, wildcards, private names, and a negated child never enter the path.

		Paths are bounded to 8 levels, 16 evidence leaves, and 4 KiB; an over-limit subtree becomes `opaque: limit`. Full sync builds safe evidence for successful installations because the generated attribution index needs it even when telemetry is disabled; only an enabled recorder serializes that evidence as telemetry. Existing cached booleans for non-package predicates may synthesize `opaque: non_package_predicate`, but caching an entire `PredicateSet` would lose successful branches and witnesses and requires revisiting this design.


		Recorders make one non-waiting exclusive-lock attempt. Contention drops the entire buffered batch or aggregate observation. Event batches serialize before one append so concurrent lines cannot interleave. Snapshot updates use same-directory temporary replacement; no `fsync` is promised, so a crash can still lose the latest update. Contribution counts detect state/snapshot divergence and permanently mark affected daily session counts incomplete.

		The event file, aggregate snapshot, and reserved maximum-size `storage_limit` row share 8 MiB per day. Aggregate metrics receive at most 512 KiB. An oversized metric update is dropped without stopping low-volume events; an ordinary batch that cannot fit is replaced by the daily marker and ordinary recording stops for that day. Relationship batches are never split.


		### Does a completed skill activation mean the skill helped?

		No. It means Claude successfully activated the installed skill. It does not show whether Claude followed the instructions or whether the task result improved. That causal question requires a controlled evaluation comparing equivalent runs with and without the skill.

Uh oh!

Conversation

gmemuriuki commented Aug 11, 2026

What does this PR do?

Uh oh!

jackh726 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmemuriuki Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmemuriuki Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmemuriuki Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmemuriuki commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gmemuriuki Aug 14, 2026 •

edited

Loading

gmemuriuki Aug 14, 2026 •

edited

Loading

gmemuriuki Aug 14, 2026 •

edited

Loading