Rya core (Track A): retry/repair, credentials, streaming chat, Langfuse datasets#2
Rya core (Track A): retry/repair, credentials, streaming chat, Langfuse datasets#2lokesh-danu wants to merge 11 commits into
Conversation
Add a secrecy output guard that rewrites configured secret-id patterns to a safe token instead of blocking the turn — the runtime form of production's scrubMasterIds(). Unlike grounding (which fails closed), secrecy scrubs and lets the redacted value through so the model keeps working. - secrecy_scrub_text / secrecy_scrub (per-leaf over dicts/lists) / secrecy_check - _compile_secrecy with a policy-keyed cache; a malformed pattern is skipped, never fatal, so one broken rule can't take down every tool call - opt-in via `secrecy.enabled` in rya.guard.yaml; applied by callers at the tool boundary and on outbound (wired in a later commit) Tests port chatstudyabroad/lib/utils.test.ts (scrubMasterIds): a Crizac master id (3-8 letters + 8+ digits) is scrubbed; numeric CAMS ids, passports, phones, years, and money are preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add upsert_connection to both FileStore and PostgresStore, keyed on (provider, owner). A login handler that re-mints a per-user bearer must not leave a stale duplicate the runtime could later inject, so the upsert rewrites the existing active connection in place — preserving id/createdAt, stamping updatedAt — rather than appending a new one. - FileStore: scan connections_dir for the existing (provider, owner, active) doc and rewrite its file; else create. - PostgresStore: SELECT ... FOR UPDATE on `owner IS NOT DISTINCT FROM %s`, then UPDATE or INSERT. Backs the ctx.connections.upsert SDK path (wired in a later commit). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…oy gate (A3) A per-user connection can expire mid-turn. Surface that as a clean, distinct outcome the caller can act on (log in again and retry) instead of a generic failure — mirroring production's CrizacAuthError -> reconnect prompt. No auto-refresh. - engine: an E_CONNECTION_EXPIRED RyaError sets run.status = needs_reconnect with its own run.needs_reconnect trace step (not run.failed); status added to the observability export gate. - turns: needs_reconnect added to TERMINAL_RUN_STATUSES. - api/app: needs_reconnect added to the manual run-status allow-list. - readiness: `rya deploy --check` now blocks E_PLAINTEXT_SECRET_AT_REST when a require_user provider has a stored connection sitting unencrypted at rest — seal() degrades to plaintext without key material, so this fails closed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…edential enforcement (A1/A2/A5/A3)
Rewrite the _Tools.call backend path so retry, self-heal, secrecy, and adoption
all flow through one journaled step, backend-agnostic across agent/http/mock —
so a replay after an approval pause returns the memoized result and never
re-runs a retry or repair.
A1 — declarative retry + repair:
- manifest RetryDecl (max_attempts / backoff / on), with a YAML-1.1 `on: true`
rescue so an unquoted `on:` key is never a silent no-op.
- @agent.repair("<tool>") callback; a RyaRecoverableToolError self-heals once
with a patched input (tool.repair trace step). Repair (domain) and retry
(transient) are orthogonal budgets.
- _http_tool tags a 5xx (http_status) and a socket timeout (E_TIMEOUT) so the
retry classifier can see them; RyaError carries http_status.
A2 — secrecy wiring: the scrub runs on every tool result before it re-enters
the loop / journal / trace, and on every outbound message before it leaves.
ctx.guard.check_secrecy exposed for handler-side assertions.
A5 — adoption: a manifest `adopt: {field: scope.key}` copies a successful
result field into scoped memory so a later pinned tool in the same turn adopts
it (e.g. create_lead -> student_state.camsId).
A3 — per-user credential: credential resolution is scoped to egress backends
only (a local @agent.tool leaf never receives the secret, so provider: on it is
governance metadata); require_user fails closed with E_NO_IDENTITY when no
verified sub; _http_tool maps a 401 on a credentialed request to
E_CONNECTION_EXPIRED; ctx.connections.upsert mints/refreshes the caller's
connection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rya is a general-purpose agent runtime, so the core primitives and their tests should not carry a specific customer's business vocabulary. Replace Crizac/CAMS/counsellor/chatstudyabroad references (comments, test provider names, fixture field names, a secrecy-policy id) with neutral equivalents (external CRM, account id, user, opaque_master_id, session_state). Behaviour is unchanged: comment-only in src, self-contained renames in tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The governed agent loop seeded itself with only the current message, so
follow-ups ("compare those", "#2") had no earlier context to resolve
against. Add an optional keyword-only `history` param to _LLM.run that
seeds the loop with prior user/assistant turns ahead of the current
message. Blank-content and non-user/assistant rows are filtered so the
provider never receives an empty or unnormalizable message.
Backward-compatible: defaults to None (existing callers and journals
unchanged; the number/order of _step calls is unaffected). The caller
owns the window size, e.g. via ctx.sessions.history(...).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
support conversation history in
|
Handlers that need to call an upstream API directly (not through a declared `url:` tool) had no way to get at a connection's bearer - only leaf tools got credential injection. Add ctx.connections.secret(provider, scopes=...), the handler-side analogue: same per-user resolution and scope-intersection enforcement as tool injection, and the resolved secret is seeded into the redaction vault so it can never surface in a trace or log. Deliberately not journaled - a plaintext credential must never touch the run journal - so a replay re-resolves it live instead of memoizing it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
chat() gained an optional on_token callback: when set, the anthropic provider streams the response over SSE and emits assistant text deltas as they arrive, instead of blocking for the full completion. Tool-use blocks are rebuilt from input_json_delta fragments so a tool-calling step behaves identically to the non-streaming path - only the final text is additionally streamed. Providers without a streaming chat path ignore on_token and return the full result at once, so passing it is always safe. _anthropic_messages/_anthropic_tools are pulled out of _anthropic_chat so the streaming and non-streaming paths build the request body from the same code and can never diverge. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…taset Rya could only push runs/scores to Langfuse; there was no way to run the agent over an existing Langfuse Dataset. Add `rya eval --langfuse-dataset <name>`: fetches the dataset's items, fires a real engine run per item (--trigger-type/--payload-defaults control how an item's `input` maps to an event), and links each run's trace back to its dataset item as a dataset-run-item under a named run - so the run shows up under Datasets -> runs in the Langfuse UI, one row per item. An item may carry a Rya `expect` block under its metadata to be scored with the same SCORERS as local eval cases (extracted local-suite scoring into _score_expect so both paths share it); a failing item still exits non-zero like the local suite, so a Langfuse dataset can gate a deploy too. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
handler-side credentials, streaming Anthropic chat, Langfuse dataset evals
|
The repo had no linting at all - style and import hygiene were whatever each file happened to land on. Add ruff (one tool for both lint and format), configured entirely in pyproject.toml: line-length 120 (the tree only has 31 lines over it), target py310 to match requires-python, and the F/E/W/I/UP/B/ C4/SIM/RUF rule set. Three ignores, each with a stated reason - E501 (the formatter owns wrapping), B008 (Typer/FastAPI declare params as calls in defaults, 48 intentional hits), SIM105 - plus per-file E402 for tests/ (conftest sets sys.path before importing rya) and examples/. Config only: no source file is touched by this commit. The tree is NOT clean against it yet - `ruff check` reports 739 findings (562 safe-autofixable, mostly UP045 Optional -> `| None`) and `ruff format --check` would rewrite 88 of 136 files. Fixing those is deliberately deferred and will happen gradually, file by file as they're touched, so real changes aren't buried under thousands of lines of churn. CI is also pending: there is no .github/ in this repo, and wiring a workflow now would only fail red. Once the tree is closer to clean - or as a changed-files-only gate - lint/format/pytest should run there. docs/devex.md documents the commands and this convergence plan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ruff as linter + formatter — config only, cleanup and CI still pendingThe repo had no linting at all, so style and import hygiene were whatever each
No source file is touched by this commit — nothing in the diff is Deliberately deferredThe tree is not clean against this config yet. Current baseline:
Largest buckets: Fixing these is intentionally not in this PR. A repo-wide autofix + CI is also pending. There is no |
A real agent turn exported a trace that was hard to read: the model steps themselves were contentless, most observations were empty rows, everything was flat, and the reported latency was wrong. Root causes, all in the export path: - The governed agent loop (ctx.llm.run) emits its model steps with kind `llm.chat`, but `_LLM_KINDS` only listed `llm.respond`/`model.call`. Those steps fell through to the generic branch and were exported as contentless EVENTs — no prompt, no response, no model, no tokens. Only the single-shot sidecar (llm.respond) rendered as a GENERATION, so the interesting calls were the ones missing. The same mismatch made run_usage skip the whole compose loop, silently under-reporting tokens and cost. - Every step records its return value, but the exporter only mapped input/output for LLM and tool steps; everything else dumped it into `metadata.result`, so memory/session/ui observations rendered blank. - No parent was ever set, so observations were flat siblings of the trace. - Steps carried only `ts`, which is second-resolution, so a whole turn collapsed onto one instant and could not even be ordered reliably. - The trace output read `run["result"]`, which the engine never populates — the handler's return value lives on `run["output"]`. Changes: - Recognise `llm.chat` as an LLM call in both the exporter and usage metering, and journal the messages array + model parameters so the generation shows the real prompt it was sent. - Record a true wall-clock span (`startedAt`/`endedAt`, ms) around every journaled step. `store.now_iso()` is deliberately untouched: it has 63 call sites and its second-resolution format is compared when scheduling. - Build a real tree — run root -> agent loop -> model step -> the tools that step called. ctx.llm.run stamps a `loopId` on its steps and on the tool calls they trigger; the loop span is synthesised at export time, so no extra journaled step is created and approval-pause/resume replay is unaffected. - Give every observation input/output, and demote runtime bookkeeping (memory/session/file/connection/knowledge) to DEBUG under one collapsed `context` span, so the top level shows the agent's work rather than the runtime's. Durable operations are exported as SPANs because Langfuse only accepts an endTime on SPAN and above; instantaneous markers stay EVENTs. Failures carry level=ERROR plus a statusMessage. - Populate trace-level `output` (from run["output"]), `sessionId` (recovered from the session step, which enables Langfuse's Sessions view for multi-turn agents), `userId` (new, set from the run's identity), and tags. - Mirror the same hierarchy and real timings in the OTLP exporter so Phoenix/Tempo/Datadog show the identical shape. Runs recorded before this change have no startedAt/loopId and still export, falling back to the previous seq-derived ordering (covered by a test). Verified against a live Langfuse instance: latency now reflects reality (10.1s for a turn previously reported as 0.02s), no non-EVENT observation has null input/output, and generations carry model parameters and usage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Re-scoped from #1 to contain ONLY the Track A core-runtime primitives:
Commits (4), ordered guard → store → runtime → sdk; all changes are under
src/+ roottests/.The csa-counsellor example (Track B) is being moved to another repo, so it is intentionally not part of this PR. Supersedes #1.