Skip to content

Rya core (Track A): retry/repair, credentials, streaming chat, Langfuse datasets#2

Open
lokesh-danu wants to merge 11 commits into
mainfrom
track-a-core
Open

Rya core (Track A): retry/repair, credentials, streaming chat, Langfuse datasets#2
lokesh-danu wants to merge 11 commits into
mainfrom
track-a-core

Conversation

@lokesh-danu

@lokesh-danu lokesh-danu commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Re-scoped from #1 to contain ONLY the Track A core-runtime primitives:

  • A1 declarative tool-call retry + repair
  • A2 id-secrecy output guard (scrub master ids)
  • A3 per-user connection credential + reconnect-on-401 (store upsert, plaintext-credential deploy gate, needs_reconnect terminal outcome)
  • A5 in-turn deterministic student-key adoption

Commits (4), ordered guard → store → runtime → sdk; all changes are under src/ + root tests/.

The csa-counsellor example (Track B) is being moved to another repo, so it is intentionally not part of this PR. Supersedes #1.

lokesh-danu and others added 4 commits July 23, 2026 09:38
Add a secrecy output guard that rewrites configured secret-id patterns to a
safe token instead of blocking the turn — the runtime form of production's
scrubMasterIds(). Unlike grounding (which fails closed), secrecy scrubs and
lets the redacted value through so the model keeps working.

- secrecy_scrub_text / secrecy_scrub (per-leaf over dicts/lists) / secrecy_check
- _compile_secrecy with a policy-keyed cache; a malformed pattern is skipped,
  never fatal, so one broken rule can't take down every tool call
- opt-in via `secrecy.enabled` in rya.guard.yaml; applied by callers at the
  tool boundary and on outbound (wired in a later commit)

Tests port chatstudyabroad/lib/utils.test.ts (scrubMasterIds): a Crizac master
id (3-8 letters + 8+ digits) is scrubbed; numeric CAMS ids, passports, phones,
years, and money are preserved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add upsert_connection to both FileStore and PostgresStore, keyed on
(provider, owner). A login handler that re-mints a per-user bearer must not
leave a stale duplicate the runtime could later inject, so the upsert rewrites
the existing active connection in place — preserving id/createdAt, stamping
updatedAt — rather than appending a new one.

- FileStore: scan connections_dir for the existing (provider, owner, active)
  doc and rewrite its file; else create.
- PostgresStore: SELECT ... FOR UPDATE on `owner IS NOT DISTINCT FROM %s`, then
  UPDATE or INSERT.

Backs the ctx.connections.upsert SDK path (wired in a later commit).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…oy gate (A3)

A per-user connection can expire mid-turn. Surface that as a clean, distinct
outcome the caller can act on (log in again and retry) instead of a generic
failure — mirroring production's CrizacAuthError -> reconnect prompt. No
auto-refresh.

- engine: an E_CONNECTION_EXPIRED RyaError sets run.status = needs_reconnect
  with its own run.needs_reconnect trace step (not run.failed); status added to
  the observability export gate.
- turns: needs_reconnect added to TERMINAL_RUN_STATUSES.
- api/app: needs_reconnect added to the manual run-status allow-list.
- readiness: `rya deploy --check` now blocks E_PLAINTEXT_SECRET_AT_REST when a
  require_user provider has a stored connection sitting unencrypted at rest —
  seal() degrades to plaintext without key material, so this fails closed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…edential enforcement (A1/A2/A5/A3)

Rewrite the _Tools.call backend path so retry, self-heal, secrecy, and adoption
all flow through one journaled step, backend-agnostic across agent/http/mock —
so a replay after an approval pause returns the memoized result and never
re-runs a retry or repair.

A1 — declarative retry + repair:
- manifest RetryDecl (max_attempts / backoff / on), with a YAML-1.1 `on: true`
  rescue so an unquoted `on:` key is never a silent no-op.
- @agent.repair("<tool>") callback; a RyaRecoverableToolError self-heals once
  with a patched input (tool.repair trace step). Repair (domain) and retry
  (transient) are orthogonal budgets.
- _http_tool tags a 5xx (http_status) and a socket timeout (E_TIMEOUT) so the
  retry classifier can see them; RyaError carries http_status.

A2 — secrecy wiring: the scrub runs on every tool result before it re-enters
the loop / journal / trace, and on every outbound message before it leaves.
ctx.guard.check_secrecy exposed for handler-side assertions.

A5 — adoption: a manifest `adopt: {field: scope.key}` copies a successful
result field into scoped memory so a later pinned tool in the same turn adopts
it (e.g. create_lead -> student_state.camsId).

A3 — per-user credential: credential resolution is scoped to egress backends
only (a local @agent.tool leaf never receives the secret, so provider: on it is
governance metadata); require_user fails closed with E_NO_IDENTITY when no
verified sub; _http_tool maps a 401 on a credentialed request to
E_CONNECTION_EXPIRED; ctx.connections.upsert mints/refreshes the caller's
connection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lokesh-danu and others added 2 commits July 23, 2026 17:47
rya is a general-purpose agent runtime, so the core primitives and their
tests should not carry a specific customer's business vocabulary. Replace
Crizac/CAMS/counsellor/chatstudyabroad references (comments, test provider
names, fixture field names, a secrecy-policy id) with neutral equivalents
(external CRM, account id, user, opaque_master_id, session_state). Behaviour
is unchanged: comment-only in src, self-contained renames in tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The governed agent loop seeded itself with only the current message, so
follow-ups ("compare those", "#2") had no earlier context to resolve
against. Add an optional keyword-only `history` param to _LLM.run that
seeds the loop with prior user/assistant turns ahead of the current
message. Blank-content and non-user/assistant rows are filtered so the
provider never receives an empty or unnormalizable message.

Backward-compatible: defaults to None (existing callers and journals
unchanged; the number/order of _step calls is unaffected). The caller
owns the window size, e.g. via ctx.sessions.history(...).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@lokesh-danu
lokesh-danu marked this pull request as ready for review July 24, 2026 06:14
@lokesh-danu
lokesh-danu requested review from arshadshk and vaibs-d July 24, 2026 06:14
@lokesh-danu

Copy link
Copy Markdown
Collaborator Author

support conversation history in ctx.llm.run

lokesh-danu and others added 3 commits July 24, 2026 17:39
Handlers that need to call an upstream API directly (not through a declared
`url:` tool) had no way to get at a connection's bearer - only leaf tools got
credential injection. Add ctx.connections.secret(provider, scopes=...), the
handler-side analogue: same per-user resolution and scope-intersection
enforcement as tool injection, and the resolved secret is seeded into the
redaction vault so it can never surface in a trace or log.

Deliberately not journaled - a plaintext credential must never touch the run
journal - so a replay re-resolves it live instead of memoizing it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
chat() gained an optional on_token callback: when set, the anthropic provider
streams the response over SSE and emits assistant text deltas as they arrive,
instead of blocking for the full completion. Tool-use blocks are rebuilt from
input_json_delta fragments so a tool-calling step behaves identically to the
non-streaming path - only the final text is additionally streamed. Providers
without a streaming chat path ignore on_token and return the full result at
once, so passing it is always safe.

_anthropic_messages/_anthropic_tools are pulled out of _anthropic_chat so the
streaming and non-streaming paths build the request body from the same code
and can never diverge.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…taset

Rya could only push runs/scores to Langfuse; there was no way to run the
agent over an existing Langfuse Dataset. Add `rya eval --langfuse-dataset
<name>`: fetches the dataset's items, fires a real engine run per item
(--trigger-type/--payload-defaults control how an item's `input` maps to an
event), and links each run's trace back to its dataset item as a
dataset-run-item under a named run - so the run shows up under
Datasets -> runs in the Langfuse UI, one row per item.

An item may carry a Rya `expect` block under its metadata to be scored with
the same SCORERS as local eval cases (extracted local-suite scoring into
_score_expect so both paths share it); a failing item still exits non-zero
like the local suite, so a Langfuse dataset can gate a deploy too.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@lokesh-danu lokesh-danu changed the title Rya core primitives (Track A): retry+repair, id-secrecy scrub, per-user connection creds, in-turn adoption Rya core (Track A): retry/repair, credentials, streaming chat, Langfuse datasets Jul 24, 2026
@lokesh-danu

Copy link
Copy Markdown
Collaborator Author

handler-side credentials, streaming Anthropic chat, Langfuse dataset evals

  • ctx.connections.secret(provider, scopes=...) — handler-side analogue of tool
    credential injection: same per-user resolution + scope-intersection enforcement
    as declared url: tools, and the resolved secret is vaulted so it can never
    surface in a trace or log. Deliberately not journaled — a replay re-resolves it
    live rather than memoizing a plaintext credential.
  • chat(..., on_token=...) — optional streaming callback on the LLM provider
    layer; the anthropic provider now streams assistant text deltas over SSE while
    assembling the identical {text, toolCalls, model, provider, usage} shape.
    Providers without a streaming path ignore on_token and return the full result
    at once, so passing it is always safe.
  • rya eval --langfuse-dataset <name> — pulls an existing Langfuse dataset's
    items, fires a real engine run per item, and links each run's trace back to its
    dataset item as a dataset-run-item (shows up under Datasets → runs in the
    Langfuse UI). An item may carry a Rya expect block under its metadata to be
    scored with the same scorers as local eval cases; a failing item still exits
    non-zero, so a Langfuse dataset can gate a deploy too.

The repo had no linting at all - style and import hygiene were whatever each
file happened to land on. Add ruff (one tool for both lint and format),
configured entirely in pyproject.toml: line-length 120 (the tree only has 31
lines over it), target py310 to match requires-python, and the F/E/W/I/UP/B/
C4/SIM/RUF rule set. Three ignores, each with a stated reason - E501 (the
formatter owns wrapping), B008 (Typer/FastAPI declare params as calls in
defaults, 48 intentional hits), SIM105 - plus per-file E402 for tests/ (conftest
sets sys.path before importing rya) and examples/.

Config only: no source file is touched by this commit. The tree is NOT clean
against it yet - `ruff check` reports 739 findings (562 safe-autofixable,
mostly UP045 Optional -> `| None`) and `ruff format --check` would rewrite 88
of 136 files. Fixing those is deliberately deferred and will happen gradually,
file by file as they're touched, so real changes aren't buried under thousands
of lines of churn.

CI is also pending: there is no .github/ in this repo, and wiring a workflow
now would only fail red. Once the tree is closer to clean - or as a
changed-files-only gate - lint/format/pytest should run there.

docs/devex.md documents the commands and this convergence plan.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lokesh-danu

Copy link
Copy Markdown
Collaborator Author

ruff as linter + formatter — config only, cleanup and CI still pending

The repo had no linting at all, so style and import hygiene were whatever each
file happened to land on. 8def09f adds ruff
(one tool for both lint and format), configured entirely in pyproject.toml:

  • line-length = 120 — the tree only has 31 lines over it
  • target-version = "py310" to match requires-python
  • rule set F, E, W, I, UP, B, C4, SIM, RUF
  • three ignores, each with a stated reason: E501 (the formatter owns
    wrapping), B008 (Typer/FastAPI declare params as calls in defaults — 48
    intentional hits), SIM105
  • per-file E402 for tests/ (conftest sets sys.path before importing rya)
    and examples/
  • ruff>=0.16 in the dev extra; docs/devex.md documents the commands

No source file is touched by this commit — nothing in the diff is
behavioural, so it doesn't affect review of the Track A commits.

Deliberately deferred

The tree is not clean against this config yet. Current baseline:

check result
ruff check 739 findings, 562 safe-autofixable
ruff format --check 88 of 136 files would be rewritten

Largest buckets: UP045 Optional[X]X \| None (364), UP006 (95),
E702 semicolon one-liners (78), I001 unsorted imports (53), B904
raise-without-from (48).

Fixing these is intentionally not in this PR. A repo-wide autofix +
reformat would be a ~7k-line diff that buries the actual runtime changes here.
The plan is to converge file by file as they're touched.

CI is also pending. There is no .github/ in this repo, and wiring a
workflow now would only fail red. Once the tree is closer to clean — or as a
changed-files-only gate — lint/format/pytest should run there. Happy to do that
as a follow-up PR.

A real agent turn exported a trace that was hard to read: the model steps
themselves were contentless, most observations were empty rows, everything
was flat, and the reported latency was wrong.

Root causes, all in the export path:

- The governed agent loop (ctx.llm.run) emits its model steps with kind
  `llm.chat`, but `_LLM_KINDS` only listed `llm.respond`/`model.call`. Those
  steps fell through to the generic branch and were exported as contentless
  EVENTs — no prompt, no response, no model, no tokens. Only the single-shot
  sidecar (llm.respond) rendered as a GENERATION, so the interesting calls
  were the ones missing. The same mismatch made run_usage skip the whole
  compose loop, silently under-reporting tokens and cost.
- Every step records its return value, but the exporter only mapped
  input/output for LLM and tool steps; everything else dumped it into
  `metadata.result`, so memory/session/ui observations rendered blank.
- No parent was ever set, so observations were flat siblings of the trace.
- Steps carried only `ts`, which is second-resolution, so a whole turn
  collapsed onto one instant and could not even be ordered reliably.
- The trace output read `run["result"]`, which the engine never populates —
  the handler's return value lives on `run["output"]`.

Changes:

- Recognise `llm.chat` as an LLM call in both the exporter and usage
  metering, and journal the messages array + model parameters so the
  generation shows the real prompt it was sent.
- Record a true wall-clock span (`startedAt`/`endedAt`, ms) around every
  journaled step. `store.now_iso()` is deliberately untouched: it has 63
  call sites and its second-resolution format is compared when scheduling.
- Build a real tree — run root -> agent loop -> model step -> the tools that
  step called. ctx.llm.run stamps a `loopId` on its steps and on the tool
  calls they trigger; the loop span is synthesised at export time, so no
  extra journaled step is created and approval-pause/resume replay is
  unaffected.
- Give every observation input/output, and demote runtime bookkeeping
  (memory/session/file/connection/knowledge) to DEBUG under one collapsed
  `context` span, so the top level shows the agent's work rather than the
  runtime's. Durable operations are exported as SPANs because Langfuse only
  accepts an endTime on SPAN and above; instantaneous markers stay EVENTs.
  Failures carry level=ERROR plus a statusMessage.
- Populate trace-level `output` (from run["output"]), `sessionId` (recovered
  from the session step, which enables Langfuse's Sessions view for
  multi-turn agents), `userId` (new, set from the run's identity), and tags.
- Mirror the same hierarchy and real timings in the OTLP exporter so
  Phoenix/Tempo/Datadog show the identical shape.

Runs recorded before this change have no startedAt/loopId and still export,
falling back to the previous seq-derived ordering (covered by a test).

Verified against a live Langfuse instance: latency now reflects reality
(10.1s for a turn previously reported as 0.02s), no non-EVENT observation
has null input/output, and generations carry model parameters and usage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant