Fix attestation credential delivery and test coverage (#4411) - #4420
Merged
Trecek merged 10 commits intoJul 30, 2026
Merged
Conversation
…edential Make arming and delivery of the recipe execution attestation inseparable: - one typed credential value object (RecipeExecutionCredential) and build_recipe_execution_credential producer consumed by both renderers; - one completion-receipt renderer _render_completion_receipt routed from both branches of build_completion_response; - install-site registry ratchet RECIPE_EXECUTION_INSTALL_SITE_REGISTRY with AST guard test_execution_install_delivery; - differential cross-delivery-mode reachability test test_attestation_delivery_reachability; - build-time fitness assertion test_completion_receipt_fitness for the receipt with credential against every backend bound; - loud structured failure recipe_initialization_receipt_altered when the response budget rewrites the receipt (was: silent return enforced); - error message constants RECIPE_EXECUTION_ATTESTATION_MISSING_MESSAGE and RECIPE_EXECUTION_INACTIVE_MESSAGE name their remedy and replace the "standalone mode cannot claim recipe attestation" literal; - instruction surface updates to _MCP_RETRY_INSTRUCTION and sous-chef/SKILL.md document the attestation forwarding rule. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
The canonical install-site digest was previously not asserted anywhere, so any drift in the registry (added/removed entry, edited _asdict shape) would go undetected. Mirror the RESPONSE_BACKSTOP_EXEMPTION_REGISTRY digest test pattern at tests/core/test_type_constants.py:54-71. No production source change.
The real open_kitchen driver (_mock_fmcp_ctx + _open_kitchen_patched) was duplicated only in test_serve_idempotence.py. Move it byte-identically into _helpers.py, the designated shared-builder module (tests/server/AGENTS.md:10), and replace the local copies with a re-import. Add _credit_initialization_sections and _pull_step_section so the reachability rewrite can credit pages to the initialization without re-implementing pagination. Drop the now-unused AsyncMock/MagicMock/patch imports from test_serve_idempotence.py. All 11 pre-existing test functions in that file keep passing — the helpers were not referenced by any other module. No production source change.
The previous suite contained three synthetic dict-shape tests over hand-built payloads that called none of the real tools. Replace it with a single parametrized 3-mode reachability test plus two focused tests: * REQ-031: drive open_kitchen (remediation, ENVELOPE), credit every section through get_recipe_section, then complete_recipe_initialization, asserting the credential appears on the completion receipt alone — no payload.json read. * REQ-032: derive run_skill inputs exclusively from the receipt and the pulled step body, then prove the executor was reached (not merely a deny string avoided) by asserting the runner call count grew. * REQ-033: parametrize over RecipeDeliveryMode with match/case dispatch. ORDINARY_INLINE reads body.recipe_execution; ATTESTED_INLINE is driven at finalize_recipe_delivery (no MCP tool accepts host attestation evidence); ENVELOPE asserts the credential is absent from the first response and present on the completion receipt. A forbid_artifact_reads fixture patches the consuming module's bound name (tools_recipe.load_recipe_artifact) to prove nothing reads the persisted artifact — the poison is armed after the last pull so get_recipe_section still serves pages. Register the new test in tests/_test_filter.py under LAYER_CASCADE_CONSERVATIVE['recipe'] so test-cascade-map_guard picks it up. No production source change.
…error The completion-enforcement contract asserts that an altered receipt returns recipe_initialization_receipt_altered with response_budget_error surfaced. Previously this field was asserted neither as the empty string (no budget-error context) nor as the extracted value (when the altered payload was a parseable dict containing 'error'). Add both assertions inside the existing test_completion_is_server_owned_and_commits_ready_only_after_enforcement. A membership-style check would pass against a hardcoded empty string; the =='' form does not, so the new code cannot degrade to a vacuous assertion. The extraction branch is exercised by a second altered call before the existing success path. No production source change.
The install-site registry's delivery_surface values are promises to the rest of the system that the credential reaches the caller via those tools. A typo'd or retired surface name would silently break the catalog without breaking any existing test. Add a fourth test method to TestExecutionInstallDelivery that resolves the delivery_surface set against the IL-0 tool registry via autoskillit.core.all_tool_names() (NOT the FastMCP mcp singleton — pulling that into tests/arch/ would activate the mcp._transforms clear/restore machinery in unrelated workers). Both current surfaces (open_kitchen, complete_recipe_initialization) are in TOOL_REGISTRY, so the test passes on landing and fails on any future drift. No production source change.
… enumeration The previous fitness test hand-built three receipts with fake digests, asserted on one bound (response_max_bytes), and reused no helper — a vacuous size check. Replace it with a single parametrized test that, for every bundled recipe: * Builds the canonical artifact payload and a real execution credential through prepare_recipe_delivery_generation + build_recipe_execution_credential. * Persists under tmp_path with persist_recipe_artifact. * Renders the completion receipt through the production _render_completion_receipt (no synthetic digests). * Asserts the receipt fits response_max_bytes AND every backend's general output token limit, reusing _backend_capabilities() and _generic_backstop_bound_bytes() from tests/contracts/test_delivery_bound_fitness. * Asserts the measured credential matches the real credential: same key set, same execution_id, same template_digests dict — so the test cannot degrade to a size check on a credential-less receipt. backend=None is deliberate: prepare_recipe_delivery_generation compiles the snapshot without consulting the backend, while the bound loop covers every registered backend explicitly. prepared.canonical_artifact_payload (not the raw payload) is required because persist_recipe_artifact validates flow_records/recipe_flow. No production source change.
- _pull_step_section unwraps parsed[step_name] because _extract_step_body
wraps the step in {step_name: step_obj}; consumers read step_body['with'].
- ATTESTED_INLINE test uses _payload(...) since real remediation (453 KB
rendered) exceeds the 56,750 byte attested budget — measured in plan
was body-only and missed flow_records/metadata added by finalize.
…n test_no_delivery_mode_omits_the_attestation_credential
Trecek
deleted the
bounded-open-kitchen-never-delivers-the-attestation-values-r/4411
branch
July 30, 2026 01:36
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part A makes arming and delivery of the recipe-execution credential inseparable: a typed credential value object with one producer consumed by both renderers; a shared completion-receipt renderer; an install-site registry ratchet that fails when an
install_recipe_executioncall's declaring entry does not name a response builder calling the sanctioned producer; a differential cross-delivery-mode reachability test driven against the real tool functions; a build-time fitness assertion across every bundled recipe; and structured failures naming their own remedy. Part A also remediates seven test requirements trimmed during implementation (digest stability, real-driver reachability, attestedrun_skillusing only delivered values, parametrized 3-mode differential,response_budget_errorpinning, registered delivery surfaces, fitness on real receipts), all driven against the real tool drivers with nopayload.jsonreads and no synthetic hand-built dicts.Individual Group Plans
Plan 1 — Rectify: Install-Implies-Deliver — the Recipe Execution Credential (Part A)
run_skillarms a hard attestation requirement the moment a recipe execution is installed(
tools_execution.py:884-892). The values that satisfy it —execution_idand the per-stepinvocation_template_digest— are rendered by a different function, in a different call,and only on some delivery paths. On the bounded (
delivery_bound_spill: True) path therequirement is armed by
complete_recipe_initialization(_recipe_initialization.py:241) andthat call's six-key receipt carries none of them (
:254-311). The orchestrator is required tosend a value it was never handed.
The bug class is broader than one missing key:
Part A makes arming and delivery inseparable:
RecipeExecutionCredential) with one producer(
build_recipe_execution_credential), consumed by both renderers. Two hand-maintaineddict literals become zero.
build_completion_response,so the two renderings cannot silently diverge — the exact recurrence the investigation
warns about (Test Gap make-script-skill: Add pipeline data-flow quality checks #7).
RECIPE_EXECUTION_INSTALL_SITE_REGISTRY) modelleddirectly on the
#4413canonical-JSON producer/consumer ratchet: an AST guard that failswhen any call to
install_recipe_executionexists whose declaring entry does not name aresponse builder that calls the sanctioned credential producer. You cannot add a new way
to install an execution without declaring — and proving — which response delivers its
credential.
functions in every delivery mode and asserts the credential is reachable from responses
alone, then completes a real attested
run_skill. Nopayload.jsonread anywhere.assertion over every bundled recipe, plus conversion of the silent
enforced != finalized.renderedswallow into a loud structured failure.when a server-required
run_skillparameter is named by no shipped instruction.Part B (a separate task) covers the destructive named re-open and its diagnostics.
Plan 2 — Remediate Trimmed Test Coverage for the Recipe Execution Credential (#4411 Part A)
Part A's production changes all landed — 36 of 43 requirements are
COVEREDby audit round 1 — butseven test requirements were trimmed or substituted during implementation. The
NO GOauthorityrecords them as
MISSING:REQ-021 (no digest-stability test for the install-site registry), REQ-031 (reachability file
holds synthetic tests over hand-built dicts), REQ-032 (no
test_attested_run_skill_succeeds_using_only_delivered_values),REQ-033 (no parametrized 3-mode differential test), REQ-035 (
response_budget_errornever asserted),REQ-039 (no
test_registered_delivery_surfaces_are_real_tools), and REQ-043 (fitness test hand-buildsreceipts with fake digests, checks one bound, reuses no helper).
Test-only. No production source file changes. Every symbol the new tests need already exists
and is already exported through
autoskillit.core.Key design decisions: drive the real tools with the driver already in the repo (
_open_kitchen_patchedmoved into
tests/server/_helpers.py); mode is chosen by backend capability plus real payload size,never by a stub;
ATTESTED_INLINEis driven at the finalizer (no MCP tool accepts host attestation);nothing reads
payload.json(enforced structurally via monkeypatch); the attestedrun_skillusesonly caller-visible values; the fitness test builds real receipts through the real renderer.
Closes #4411
Implementation Plan
Plan files:
/home/talon/projects/autoskillit-runs/remediation-20260729-093623-404122/.autoskillit/temp/rectify/rectify_attestation_credential_delivery_2026-07-29_103426_part_a.md/home/talon/projects/autoskillit-runs/remediation-20260729-093623-404122/.autoskillit/temp/make-plan/remediate_attestation_delivery_test_coverage_2026-07-29_155013.md🤖 Generated with Claude Code via AutoSkillit
Token Usage Summary
* Step used a non-Anthropic provider; caching behavior may differ.
Token Efficiency
Model Usage Breakdown