diff --git a/changelog.d/sp-4an.2.1.md b/changelog.d/sp-4an.2.1.md new file mode 100644 index 0000000..0005a10 --- /dev/null +++ b/changelog.d/sp-4an.2.1.md @@ -0,0 +1,42 @@ +# sp-4an.2.1 + +## Added + +- Run records on the storage contract: `StatifierPersistence.Storage.Adapter` + gains `insert_run/2`, `fetch_run/2`, and `update_run/2` callbacks with + `run_record`/`run_status` types and the `:run_exists` / `:run_not_found` + error arms; `StatifierPersistence.Storage.InMemory` implements them. +- Guarded run access on the facade: `StatifierPersistence.Storage.insert_run/5`, + `update_run/5`, `fetch_run/2`, and `load_run_position/3` (identity-guarded, + with the `:run_position_missing` arm for a run persisted without a + position). +- Run-record conformance tests in + `StatifierPersistence.Testing.StorageConformance`, so downstream adapters + inherit the same contract checks. +- The run lifecycle as a library: `StatifierPersistence.Runs.create/4` and + `step/5` drive the load -> re-stamp -> step -> execute -> persist loop + over durable run records, handing effects to a host-supplied + `StatifierPersistence.Executor` (behaviour or arity-2 fun) and returning + the host-facing `StatifierPersistence.Run` struct; events to a terminal + run come back as `{:discarded, run}`. +- Failure semantics on the loop: executor failures on actionable effects + re-enter the chart as `error.communication` events (single wave per step, + observational failures discarded); effect execution is at-least-once, with + a failed persist re-driving the same event and re-emitting the same + effects under identical deterministic keys; budget exhaustion persists a + `:failed` run (position untouched) and returns + `{:error, {:budget_exhausted, payload}}`. +- `StatifierPersistence.Runs.fail/4`, the host-driven abandonment: marks an + active run `:failed` with a reason, leaves the stored position untouched, + and discards on a terminal run - backed by the new status-only writer + `StatifierPersistence.Storage.update_run_status/4`. +- Pluggable per-run serialization: the `StatifierPersistence.Serialization` + behaviour (`with_run/3`), selected per lifecycle call with + `serialization: {module, config}` on `Runs.create/4`, `step/5`, and + `fail/4`. The default strategy, + `StatifierPersistence.Serialization.AdapterLock`, delegates to the new + optional adapter callback + `StatifierPersistence.Storage.Adapter.lock_run/3` (implemented by + `InMemory`, conformance-tested when exported) and refuses with + `{:error, {:serialization, :not_supported}}` when the adapter does not + export it. diff --git a/docs/adr/0003-storage-adapter-behaviour-and-the-identity-guard.md b/docs/adr/0003-storage-adapter-behaviour-and-the-identity-guard.md index 3ae46ba..d840c2b 100644 --- a/docs/adr/0003-storage-adapter-behaviour-and-the-identity-guard.md +++ b/docs/adr/0003-storage-adapter-behaviour-and-the-identity-guard.md @@ -2,7 +2,10 @@ Status: accepted (2026-08-21) - amended 2026-08-21 (sp-5qa Phase 4: adds the optional per-test isolation callback, `isolate/1`, to the behaviour's -contract surface, alongside the conformance suite decision 5 already names) +contract surface, alongside the conformance suite decision 5 already names); +amended 2026-08-22 (sp-4an.2.1 Phase 5: adds the optional per-run lock +callback, `lock_run/3`, to the behaviour's contract surface, as the seam +ADR-0004 decision 5's default serialization strategy delegates to) ## Context @@ -47,9 +50,10 @@ it. This is why the chart record's payload is opaque here - see Decision 1. `fetch_chart/2`, `save_position/2`, `fetch_position/2` - takes and returns binaries plus engine identity strings. No callback receives a `Statifier.Machine.t()` and none returns a `Statifier.MachineState.t()`. (The -optional `isolate/1` this ADR's 2026-08-21 amendment adds is the one -exception, by design: it carries no chart or position data at all - see the -amendment under Decision 5.) The chart record's +optional `isolate/1` and `lock_run/3` this ADR's 2026-08-21 and 2026-08-22 +amendments add are the exceptions, by design: neither carries chart or +position data at all - see the amendments under Decision 5.) The chart +record's `chart_blob` is opaque to this layer: this record does not choose between `Statifier.Chart.to_binary/1`'s envelope and a host's own retained source, because both satisfy the only property the layer needs - given the blob @@ -110,6 +114,23 @@ in `lib/` for a downstream adapter to reuse - is what makes the hook possible in the first place, so it belongs with that decision rather than as a new one.)* +*(Amended 2026-08-22, sp-4an.2.1 Phase 5: the behaviour gains a second +optional callback, `c:StatifierPersistence.Storage.Adapter.lock_run/3` - +mutual exclusion per `run_id`, running the given fun while the exclusion is +held and releasing it on any exit, a raise escaping the fun included, so +the lock cannot leak. It is the seam ADR-0004 decision 5's default +serialization strategy (`StatifierPersistence.Serialization.AdapterLock`) +delegates to, and the strategy refuses with +`{:error, {:serialization, :not_supported}}` when an adapter does not +export it; an Ecto adapter implements it as a transaction-scoped row lock +(sp-4an.3). Like `isolate/1` it is declared `@optional_callbacks` with no +default implementation, carries no chart or position data, and decodes +nothing, so decision 1's blobs-only rule holds; the conformance suite +exercises it only when the adapter under test exports it, the same +`function_exported?/3` shape the isolate amendment records - which is why +this widening, like that one, is recorded here with decision 5 rather than +as a new decision.)* + ## Consequences - An adapter author has no code path in which to be careless about the diff --git a/docs/adr/0004-run-lifecycle-executor-seam-and-serialization.md b/docs/adr/0004-run-lifecycle-executor-seam-and-serialization.md new file mode 100644 index 0000000..4638f81 --- /dev/null +++ b/docs/adr/0004-run-lifecycle-executor-seam-and-serialization.md @@ -0,0 +1,179 @@ +# ADR-0004: Run lifecycle, the executor seam, and per-run serialization + +Status: accepted (2026-08-22) + +## Context + +The charter's loop is this package's reason to exist: load a persisted +position, step it, execute the effects, persist (sp-4an.2, restating the +charter's scope bullets 2 and 3). sp-4an.1 shipped the substrate - the +blobs-only adapter behaviour and the guarded facade (ADR-0003) over the +keys ADR-0002 fixed - but nothing in the package yet knows what a run is, +calls the interpreter, or executes an effect. This record fixes the +contracts the loop code will encode: the durable run record, the loop's +order, the seam through which effects reach a host, and the seam through +which concurrent deliveries to one run are ordered. + +The engine facts this record leans on, verified against the vendored pin +(`deps/statifier/`, `mix.lock`): + +- `Statifier.Interpreter.initialize/2` returns an untagged + `{MachineState.t(), [Effect.t()]}` pair and cannot fail + (`deps/statifier/lib/statifier/interpreter.ex:259-260`). Creating a run + therefore always has a machine state in hand, even when that state is + already terminal or budget-exhausted. +- `Statifier.Interpreter.handle_event/2` returns + `{:ok, MachineState.t(), [Effect.t()]} | {:error, :not_running}`, with + the `running: false` refusal as the head clause + (`interpreter.ex:477-502`). Terminality is a typed refusal at the core, + never an exception. +- `Statifier.Interpreter.deliver_internal/5` is st-ADR-0039's re-entry + seam: the one door through which an out-of-loop failure becomes an + internal `error.*` event, delegating to the same two internal-queue + writers the core's own executable content uses and folding to quiescence + (`interpreter.ex:505-545`). This package never constructs `error.*` + events by hand. +- `Statifier.Position.to_binary/1` refuses only `:unidentified_chart` and + does not check quiescence; that check belongs to `export/1` + (`deps/statifier/lib/statifier/position.ex:105-117, 267-277`). + Quiescence before persist is therefore this loop's own assertion, not + something upstream enforces for it. +- st-ADR-0064 makes `from_binary/2` drop `routes` and `invoke_types` + unconditionally on decode (`position.ex:164-180`), so re-stamping both + on every load is structural, not a convention: the loop can assert the + fields arrive `nil` and fail loudly if upstream ever regresses. +- st-ADR-0054 decision 3's deterministic dedup key + (`{scope, send_id, macrostep, microstep, round, c_index, owner, + ordinal}`) and st-ADR-0059's `timer_counter` ordinal make at-least-once + honest: a re-driven step re-emits effects carrying identical keys, so + idempotency can live with the consumer. +- The interpreter moduledoc's "Rehydrating a position" recipe + (`interpreter.ex:43-92`) is this loop's resume spec: `from_binary/2`, + then `put_routes/2` + `put_invoke_types/2`, then an advance entry; no + `initialize/2` call on the resume path, ever. + +Accepted records that bound the design: this repo's ADR-0002 (runs +vocabulary, engine identities verbatim, surrogate keys are Ecto-layer +only) and ADR-0003 (blobs-only behaviour, guard in the facade, engine +identities as the only keys); upstream's st-ADR-0052/0054/0059/0060/0064 +(identity, effect-vocabulary consumption, timer ordinal, resume +semantics, blob field drops), adopted by reference per ADR-0001. + +## Decision + +**1. The run record owns its current position.** A run is the durable +unit: `%{run_id, status, content_hash, identity_blob, position_blob, +failure}`. Storing the position on the run row (rather than a second +lookup into the sp-4an.1 position table) makes the persist tail one +adapter write, makes the per-run lock cover exactly the bytes it +protects, and matches ADR-0002 decision 4/5's `statifier_runs` sketch. +The sp-4an.1 chart/position callbacks stand unchanged for hosts +persisting sessions without the lifecycle. `position_blob` is nullable: a +run that fails at creation (budget exhaustion during `initialize/2`) has +no quiescent position to store, and persisting a non-quiescent one is the +bug the loop exists to prevent. The adapter behaviour gains three +callbacks - `insert_run/2`, `fetch_run/2`, `update_run/2` - and two error +arms, `:run_exists` and `:run_not_found`. ADR-0003's blobs-only rule +binds all three: no callback decodes a blob, validates a status +transition, or performs an identity check - the facade and the lifecycle +own those. + +**2. Run keys and statuses stay this layer's style: opaque and total.** +`run_id` is a caller-supplied opaque string stored verbatim - a host +identity in ADR-0002 decision 1's category, not a surrogate this layer +generates. `status :: :active | :completed | :failed` and +`failure :: String.t() | nil` (a short reason; structured detail is not +portably storable and belongs in host telemetry). Uniqueness is the +adapter's: `insert_run/2` refuses a duplicate with +`{:error, :run_exists}`, which is what makes create-exactly-once +checkable without a lock. The identity guard extends structurally, not by +convention: the new facade functions mirror sp-4an.1 exactly - writers +derive `content_hash` and `identity_blob` from the machine state's own +`Machine.identity/1` (never a caller value) and refuse +`:unidentified_chart`; the read path (`load_run_position/3`) reuses the +same identity pre-check and `Position.from_binary/2` as +`load_position/3`, so ADR-0003 decision 2's claim - no adapter ever holds +both sides of the guard - stays true for run records too. + +**3. The loop's order is the contract.** A step is: liveness check on the +run record -> load (guarded) -> re-stamp `routes`/`invoke_types` +unconditionally (with the nil tripwire from st-ADR-0064: the fields are +pattern-matched `nil` before stamping, so an upstream regression fails +loudly here, not silently downstream) -> step via +`Interpreter.handle_event/2` -> execute effects via the executor seam -> +consume `:done` and `:budget_exhausted` into run status -> assert +`MachineState.internal_queue_empty?/1` -> persist. At-least-once effect +execution is a property, not a bug: a crash between step and persist +re-drives the same event and re-emits the same effects with identical +deterministic keys (st-ADR-0054 decision 3, st-ADR-0059), and the loop +never dedupes - idempotency is the consumer's. An event delivered to a +terminal run is discarded with a typed `{:discarded, run}` result, never +an exception and never a silent step; the check runs on the run record +before any position decode, with `handle_event/2`'s `:not_running` arm as +the structural backstop. + +**4. The executor seam is the effect vocabulary and nothing else.** A +behaviour with one required callback, +`execute(effect :: Statifier.Effect.t(), context :: map()) :: +:ok | {:error, term()}`, invoked per effect in list order; an arity-2 fun +is accepted anywhere a module is. Only the public core effect vocabulary +crosses the seam - never Session instruction tuples (st-ADR-0054 decision +1). The loop consumes `:done` and `:budget_exhausted` itself and hands +everything else over. Failures map by upstream's own classification axis +(st-ADR-0051's table, st-ADR-0039's seam): the core raises +`error.execution` itself at planning time before any effect is emitted, +so every failure an executor can report is a failure to reach or act on +the outside world after the core accepted the effect - and re-enters +uniformly as `error.communication` through +`Interpreter.deliver_internal/5`, for actionable effects of both the +invoke class (`:invoke`, `:cancel_invoke`, `:autoforward`) and the send +class (`:send`, `:send_delayed`, `:cancel`). Failures on observational +effects (`:log`, `:datamodel_*`, trace) are discarded, because +observation must never steer a run. This package never mints +`error.execution`. Re-entry is single-wave per step: effects emitted by +the error re-entries are executed, but their failures are not re-entered +again - they surface in the returned run's step result - so a +deterministically failing executor cannot loop the library. + +**5. Per-run serialization is a pluggable strategy, not a property of the +loop.** A behaviour `StatifierPersistence.Serialization` with +`with_run(config, run_id, fun) :: {:ok, term()} | {:error, term()}`; the +loop runs its whole load-to-persist tail inside `with_run/3`. The default +strategy, `StatifierPersistence.Serialization.AdapterLock`, delegates to +a new optional adapter callback `lock_run/3` (declared like `isolate/1`), +and refuses with `{:error, {:serialization, :not_supported}}` when the +adapter does not export it. The Ecto adapter implements `lock_run/3` as a +row lock (sp-4an.3); a job-queue host later swaps the strategy without +touching the loop - the ordering guarantee moves, the API does not. That +no-API-change swap is the acceptance test for this shape. + +**6. Completion is chart-driven.** The `:done` effect is the only path to +`:completed`; there is no public `complete/2`. A host that must end a run +early has `fail/4` (abandonment with a reason), the only host-driven +terminal transition, and it involves no interpreter call - abandonment is +a host decision about the run, not a chart transition. + +## Consequences + +- What would reopen this record: an effect the lifecycle must consume + beyond the two named (`:done`, `:budget_exhausted`); a serialization + strategy that cannot express its guarantee as `with_run/3`; upstream + moving quiescence enforcement into `to_binary/1`, which would make + decision 3's assertion redundant or conflicting. `caller_context` + (st-ADR-0063) landing upstream is NOT a reopener: effect structs pass + through the seam verbatim, so the field arrives here for free when the + pin moves. +- Deliberately omitted, on the same unexercised-contract reasoning + ADR-0003's Consequences records: effect deduplication (at-least-once is + the contract and a deduping loop would hide it), run deletion and + `delete_position`, position history, and an operator-forced manual + complete (a deliberate future API, recorded then, if a real embedder + needs it). +- Durable timers and async invoke execution stop at the seam: + `:send_delayed`, `:cancel`, `:invoke`, `:cancel_invoke`, `:autoforward` + cross it and scheduling them durably is statifier_oban's charter + (st-ADR-0054) or the host's. Resume restores position, not liveness + (st-ADR-0060 decision 7). +- The Ecto adapter (sp-4an.3) inherits three run callbacks, two error + arms, and an optional `lock_run/3` to implement as a transaction-scoped + row lock, all conformance-tested through the sp-4an.1 suite. diff --git a/docs/adr/README.md b/docs/adr/README.md index 554bbdb..bcf1ae7 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -5,6 +5,7 @@ | [0001](0001-record-architecture-decisions.md) | Record architecture decisions | accepted | | [0002](0002-configurable-keys-and-table-names.md) | Storage keys and table names are host-configurable at compile time (UXID default, `statifier_` prefix, runs vocabulary); engine identities are not | accepted | | [0003](0003-storage-adapter-behaviour-and-the-identity-guard.md) | The storage adapter stores opaque blobs keyed by engine identities; the identity guard lives above every adapter and cannot be skipped | accepted | +| [0004](0004-run-lifecycle-executor-seam-and-serialization.md) | The run record owns its position, the loop's order is the contract, effects cross a host-executor seam (failures re-enter as `error.communication`), and per-run serialization is a pluggable strategy | accepted | New ADRs: next number, same three-section format (Context, Decision, Consequences). Pick the number against a freshly fetched remote. A bare diff --git a/docs/plans/260822-sp-4an.2.1-run-lifecycle-executor-seam-stepper.md b/docs/plans/260822-sp-4an.2.1-run-lifecycle-executor-seam-stepper.md new file mode 100644 index 0000000..e93a0fe --- /dev/null +++ b/docs/plans/260822-sp-4an.2.1-run-lifecycle-executor-seam-stepper.md @@ -0,0 +1,1042 @@ +# Run lifecycle, executor seam, and the stepper loop Implementation Plan + +## Overview + +Build the loop this package exists to package: run lifecycle as a library +(create/step/fail on durable run records), the +load -> re-stamp -> step -> execute -> assert-quiescent -> persist loop with +effect execution delegated to a host executor seam, and a pluggable per-run +serialization strategy with the adapter row lock as the library default. +Bead: sp-4an.2.1 (first task of the sp-4an.2 epic). + +## Current State Analysis + +sp-4an.1 (via sp-5qa, merged as PR #2) landed the blob layer this work sits +on: + +- `lib/statifier_persistence/storage/adapter.ex` - the behaviour: opaque + blobs keyed by engine identities, five callbacks plus optional + `isolate/1` (ADR-0003). +- `lib/statifier_persistence/storage.ex` - the guarded facade; + `load_position/3` is the only supported blob-to-`MachineState` path, and + its doc already says re-stamping `routes`/`invoke_types` "is the + stepper's job (sp-4an.2), not this function's" + (`lib/statifier_persistence/storage.ex:160-165`). +- `lib/statifier_persistence/storage/in_memory.ex` - the reference adapter. +- `lib/statifier_persistence/testing/{charts,storage_conformance}.ex` - the + fixtures and the conformance case template sp-4an.3's Ecto adapter reuses. + +Nothing in the package yet knows what a run is, calls the interpreter, or +executes an effect. The engine surface was verified directly against the +vendored pin (`mix.lock:18`, statifier-ex `68b814a`, which includes +st-ADR-0064): + +- `deps/statifier/lib/statifier/interpreter.ex:259-260` - + `initialize(machine, opts) :: {MachineState.t(), [Effect.t()]}`, an + untagged pair that cannot fail. +- `deps/statifier/lib/statifier/interpreter.ex:477-502` - + `handle_event(machine_state, event) :: + {:ok, MachineState.t(), [Effect.t()]} | {:error, :not_running}`, with the + `running: false` refusal as the head clause. +- `deps/statifier/lib/statifier/interpreter.ex:528-545` - + `deliver_internal(machine_state, kind, name, origin, opts)` returning + `handle_event/2`'s own shape: st-ADR-0039's re-entry seam, the one door + through which an out-of-loop failure becomes an internal `error.*` event. +- `deps/statifier/lib/statifier/machine_state.ex:361-384` - the struct; + `status :: :running | :done` (`:450`); `put_routes/2` (`:801-803`), + `put_invoke_types/2` (`:810-812`), + `internal_queue_empty?/1` (`:680-681`). No `running?/1` wrapper exists; + `running` and `status` are read as plain fields. +- `deps/statifier/lib/statifier/effect.ex:120-132` - the core effect + vocabulary, every effect a `{tag, %Struct{}}` pair; producers table at + `:23-44`. `:done` and `:budget_exhausted` are the two lifecycle-outcome + effects. +- `deps/statifier/lib/statifier/position.ex:105-117, 164-180` - + `to_binary/1` refuses only `:unidentified_chart` and does NOT check + quiescence (that check belongs to `export/1`, `:267-277`); `from_binary/2` + unconditionally blanks `routes` and `invoke_types` on decode + (st-ADR-0064). Quiescence before persist is therefore this loop's own + assertion, not something upstream enforces for it. + +Accepted records that bound the design: this repo's ADR-0002 (runs +vocabulary, engine identities verbatim, surrogate keys are Ecto-layer only) +and ADR-0003 (blobs-only behaviour, guard in the facade, engine identities +as the only keys); upstream's st-ADR-0052/0054/0059/0060/0064 (identity, +effect-vocabulary consumption, timer ordinal, resume semantics, blob field +drops). + +## Desired End State + +A host with no live Session process can run a chart across process +boundaries: + +```elixir +{:ok, store} = StatifierPersistence.Storage.new(InMemory, []) +{:ok, machine} = Statifier.compile(source, []) + +{:ok, run, _ms} = + StatifierPersistence.Runs.create(store, "order-1234", machine, + executor: MyApp.Executor, routes: routes, invoke_types: invoke_types) + +{:ok, run, _ms} = + StatifierPersistence.Runs.step(store, "order-1234", machine, event, + executor: MyApp.Executor, routes: routes, invoke_types: invoke_types) + +run.status in [:active, :completed, :failed] +``` + +with these properties, each verified by a test named in a phase below: + +- Every load is identity-guarded (structurally, via the sp-4an.1 facade). +- `routes`/`invoke_types` are re-stamped unconditionally on every load, and + the loop trips (raises `MatchError` in dev/test via a pattern match) if + `from_binary/2` ever stops blanking them. +- Only the public core effect vocabulary crosses the executor seam - never + Session instruction tuples (st-ADR-0054 decision 1). +- `:done` and `:budget_exhausted` never reach the executor; the lifecycle + consumes them into run status. +- A non-quiescent `MachineState` is never persisted. +- An event delivered to a terminal run is discarded with a typed + `{:discarded, run}` result, never an exception and never a silent step. +- Executor-reported failures re-enter the chart as `error.communication` + events through st-ADR-0039's seam, per st-ADR-0051's classification + (see Key Discoveries). +- Concurrent deliveries to one run are ordered by a pluggable serialization + strategy, adapter lock by default. + +### Key Discoveries: + +- `deps/statifier/lib/statifier/interpreter.ex:43-92` - the moduledoc's + "Rehydrating a position" recipe is this loop's spec: `from_binary/2`, + then `put_routes/2` + `put_invoke_types/2`, then an advance entry; no + `initialize/2` call on the resume path, ever. +- `deps/statifier/lib/statifier/position.ex:267-277` - quiescence is + `export/1`'s check (`{:error, :internal_queue_not_empty}`), not + `to_binary/1`'s. The bead description's step 5 stands: the loop asserts + `MachineState.internal_queue_empty?/1` itself before persisting. +- `deps/statifier/lib/statifier/interpreter.ex:505-545` - + `deliver_internal/5` delegates to `raise_internal/4` / `raise_platform/4` + and folds to quiescence, so the error re-entry path produces the same + queue state an in-loop `raise` would (st-ADR-0039). This package never + constructs `error.*` events by hand. +- st-ADR-0051's failure table + (`deps/statifier/docs/adr/0051-invoke-handlers-are-registered-per-session.md:92-95`) + fixes the classification axis: a type naming no registered handler is + `error.execution` (a planning-time rejection the core raises itself, + before any effect is emitted - `session/effects.ex:345,362` are that + path); a registered handler failing to reach its service is + `error.communication` (`deps/statifier/lib/statifier/session.ex:1971`, + `invoke_error/4`). Every failure an executor can report through this + package's seam is by construction the second row - the core accepted the + effect before emitting it - so executor failures re-enter uniformly as + `error.communication`, and this package never mints `error.execution`. + (This corrects the epic description's "invoke start failure maps to + error.execution" line; upstream owns the vocabulary and its record says + communication.) +- st-ADR-0054 decision 3 fixes the deterministic dedup key + (`{scope, send_id, macrostep, microstep, round, c_index, owner, + ordinal}`) and st-ADR-0059 the `timer_counter` ordinal, which is why the + loop can be at-least-once and never dedupe: every re-emitted effect + carries identical keys, and idempotency is the consumer's. +- `deps/statifier/lib/statifier/interpreter.ex:479` - delivery to a + terminal `MachineState` is `{:error, :not_running}` at the core; the + lifecycle catches terminality earlier, on the run record, so a discarded + event never pays the position decode. +- ADR-0002 decision 5 - the vocabulary is runs, not sessions; ADR-0003 + decision 3 - this layer's keys are verbatim opaque strings, never + generated here. Together they settle the inherited open question from + the sp-5qa plan ("position key: engine session id, or a run id"): the + run key is a caller-supplied opaque `run_id`, stored verbatim, and the + run record carries the position, so no second key scheme appears. + +## What We're NOT Doing + +- **Durable timers and async invoke execution.** `:send_delayed`, `:cancel`, + `:invoke`, `:cancel_invoke`, `:autoforward` cross the executor seam and + stop there; scheduling them durably is statifier_oban's charter + (st-ADR-0054) or the host's. This package never holds a timer. +- **Restoring liveness on resume.** In-flight timers and live invoked + children re-establish through the host's durable machinery (st-ADR-0060 + decision 7). The loop restores position only. +- **Effect deduplication.** At-least-once is the contract; every effect + carries deterministic keys (st-ADR-0054 decision 3) and idempotency is + the executor's/consumer's. A deduping loop would hide the contract. +- **Ecto, row locks in SQL, schemas, migrations.** sp-4an.3. This plan + defines the `lock_run/3` seam the Ecto adapter will implement with + `SELECT ... FOR UPDATE`; it ships only the in-memory implementation. +- **A job-queue serialization strategy.** The strategy behaviour exists so + the first production embedder can move to per-run job-queue serialization + later without an API change; shipping one now would be speculation with + no consumer. +- **A public `complete/2`.** Completion is chart-driven: the `:done` effect + is the only path to `:completed`. A host that must end a run early has + `fail/4` (abandonment with a reason). If a real embedder needs + operator-forced completion, that is a deliberate future API, recorded + then. +- **Key generation.** `run_id` is caller-supplied and opaque, like every + key at this layer (ADR-0003 decision 3). The `uxid` dependency and + surrogate keys stay in sp-4an.3 (ADR-0002 decision 2). +- **`delete_position` / run deletion, and position history.** Nothing in + this loop reads either; unexercised callbacks stay out of the contract + (same reasoning ADR-0003's Consequences records). +- **`caller_context` (st-ADR-0063).** Decided upstream but not on any + effect struct at the pinned SHA; an open upstream PR implements it. The + loop passes effect structs through verbatim, so the field arrives here + for free when the pin moves. Nothing in this plan reads or waits for it. +- **Multi-node coordination beyond the strategy seam.** The first + production embedder runs the stepper on a single elected leader node; + the strategy behaviour is the extension point, not a distributed lock + shipped here. + +## Implementation Approach + +Five ideas carry the design; ADR-0004 (Phase 1) records them. + +**1. The run record owns its current position.** A run is the durable unit: +`%{run_id, status, content_hash, identity_blob, position_blob, failure}`. +Storing the position on the run row (rather than a second lookup into the +sp-4an.1 position table) makes the persist tail one adapter write, makes +the per-run lock cover exactly the bytes it protects, and matches ADR-0002 +decision 4/5's `statifier_runs` sketch. The sp-4an.1 chart/position +callbacks stand unchanged for hosts persisting sessions without the +lifecycle. `position_blob` is nullable: a run that fails at creation +(budget exhaustion during `initialize/2`) has no quiescent position to +store, and persisting a non-quiescent one is the bug the loop exists to +prevent. + +**2. Keys and statuses stay this layer's style: opaque and total.** +`run_id` is a caller-supplied opaque string stored verbatim - a host +identity in ADR-0002 decision 1's category, not a surrogate this layer +generates. `status :: :active | :completed | :failed` and +`failure :: String.t() | nil` (a short reason; structured detail is not +portably storable and belongs in host telemetry). Uniqueness is the +adapter's: `insert_run/2` refuses a duplicate with `{:error, :run_exists}`, +which is what makes create-exactly-once checkable without a lock. + +**3. The identity guard extends structurally, not by convention.** The new +facade functions mirror sp-4an.1 exactly: writers derive `content_hash` and +`identity_blob` from the machine state's own `Machine.identity/1` (never a +caller value) and refuse `:unidentified_chart`; the read path +(`load_run_position/3`) reuses the same identity pre-check and +`Position.from_binary/2` as `load_position/3`, so ADR-0003 decision 2's +claim - no adapter ever holds both sides of the guard - stays true for run +records too. + +**4. The executor seam is the effect vocabulary and nothing else.** A +behaviour with one required callback, +`execute(effect :: Statifier.Effect.t(), context :: map()) :: +:ok | {:error, term()}`, invoked per effect in list order; an arity-2 fun +is accepted anywhere a module is. The loop consumes `:done` and +`:budget_exhausted` itself and hands everything else over. Failures map by +upstream's own classification axis (st-ADR-0051's table, st-ADR-0039's +seam): the core raises `error.execution` itself at planning time before +any effect is emitted, so every failure an executor can report is a +failure to reach or act on the outside world after the core accepted the +effect - and re-enters uniformly as `error.communication` through +`Interpreter.deliver_internal/5`, for actionable effects of both the +invoke class (`:invoke`, `:cancel_invoke`, `:autoforward`) and the send +class (`:send`, `:send_delayed`, `:cancel`). Failures on observational +effects (`:log`, `:datamodel_*`, trace) are discarded, because observation +must never steer a run. This package never mints `error.execution`. Re-entry is single-wave per step: +effects emitted by the error re-entries are executed, but their failures +are not re-entered again - they surface in the returned run's step result - +so a deterministically failing executor cannot loop the library. + +**5. Serialization is a strategy, not a property of the loop.** A behaviour +`StatifierPersistence.Serialization` with +`with_run(config, run_id, fun) :: {:ok, term()} | {:error, term()}`; the +loop runs its whole load-to-persist tail inside `with_run/3`. The default +strategy, `StatifierPersistence.Serialization.AdapterLock`, delegates to a +new optional adapter callback `lock_run/3` (declared like `isolate/1`), and +refuses with `{:error, {:serialization, :not_supported}}` when the adapter +does not export it. The Ecto adapter implements `lock_run/3` as a row lock +(sp-4an.3); a job-queue host later swaps the strategy without touching the +loop - the ordering guarantee moves, the API does not. + +Phase ordering follows the coverage floor and the sp-5qa precedent: the ADR +first (docs only, green by construction), then each code phase lands with +its own tests. Phases 3-5 each consume the previous phase's surface, but +each leaves the gate green and the package coherent on its own, so the epic +can pause between any two phases with a mergeable branch. + +## Phase 1: Record the lifecycle contract as ADR-0004 + +### Overview + +Write the decision record before the code that encodes it (ADR-0001's +Consequences). Docs only; independently committable. + +### Changes Required: + +#### 1. The ADR + +**File**: `docs/adr/0004-run-lifecycle-executor-seam-and-serialization.md` +**Changes**: New file, Context / Decision / Consequences, plain ASCII +punctuation matching 0001-0003. Content: + +- **Context**: the charter's loop restated by reference; the verified + engine facts this record leans on (initialize cannot fail; handle_event's + `:not_running` arm; `deliver_internal/5` as st-ADR-0039's seam; + `to_binary/1` not checking quiescence, so the loop must; st-ADR-0064's + unconditional blanking making the re-stamp structural; st-ADR-0054/0059's + deterministic keys making at-least-once honest). +- **Decision 1 - the run record owns its current position** (approach idea + 1, including nullable `position_blob` and why). Names the three new + adapter callbacks `insert_run/2`, `fetch_run/2`, `update_run/2`, the new + error arms `:run_exists` / `:run_not_found`, and states that ADR-0003's + blobs-only rule binds them (no callback decodes, no callback guards). +- **Decision 2 - run keys and statuses** (approach idea 2). +- **Decision 3 - the loop's order is the contract**: + liveness check on the run record -> load (guarded) -> re-stamp + unconditionally (with the nil tripwire) -> step -> execute via the seam -> + consume `:done`/`:budget_exhausted` -> assert quiescence -> persist. + At-least-once stated as a property with the dedup-key citation; the loop + never dedupes. Terminal delivery is `{:discarded, run}`. +- **Decision 4 - the executor seam and error re-entry** (approach idea 4, + including the single-wave rule and the observational-effect rule). +- **Decision 5 - per-run serialization is a pluggable strategy** (approach + idea 5), with `AdapterLock`/`lock_run/3` as the default and the no-API- + change swap as the acceptance test for the shape. +- **Decision 6 - completion is chart-driven**: `:done` is the only path to + `:completed`; `fail/4` is the only host-driven terminal transition. +- **Consequences**: what reopens the record (an effect the lifecycle must + consume beyond the two named; a strategy that cannot express its + guarantee as `with_run/3`; upstream moving quiescence enforcement into + `to_binary/1`; `caller_context` landing is NOT a reopener - structs pass + through verbatim); the deliberate omissions (dedupe, delete, history, + manual complete). + +#### 2. The index + +**File**: `docs/adr/README.md` +**Changes**: add the 0004 row in the existing table style. + +### Success Criteria: + +#### Automated Verification: +- [x] Full quality gate passes (`mix quality`). +- [x] `docs/adr/0004-run-lifecycle-executor-seam-and-serialization.md` + exists with `## Context`, `## Decision`, `## Consequences`. +- [x] `docs/adr/README.md` links the 0004 file. +- [x] No typographic dashes or curly quotes in either file + (`grep -nP '[\x{2010}-\x{2015}\x{2018}\x{2019}\x{201C}\x{201D}]'` + returns nothing). +- [x] The terminology scan from the umbrella's + `docs/terminology-firewall.md` finds nothing in the diff. + +#### Manual Verification: +- [ ] The record reads as a decision; citing "ADR-0004 decision 3" ends the + argument about the loop's order. +- [ ] Cross-repo citations use `st-ADR-NNNN`; bare `ADR-NNNN` is only this + repo's own. +- [ ] Nothing contradicts ADR-0002/0003 or st-ADR-0052/0054/0059/0060/0064; + upstream decisions are adopted by reference, not restated. + +**Implementation Note**: Use the project's loop gate between edits while +iterating; run the full gate as the phase gate. In interactive execution, +pause here for the human to confirm the manual testing before moving to the +next phase. In looped (`--loop`) execution, this phase's Automated +Verification gates advancement automatically (via `/wurk:commit --auto`), and +Manual Verification items are deferred and surfaced once at the end instead +of blocking here. + +--- + +## Phase 2: Run records - adapter callbacks, facade, conformance + +### Overview + +The durable substrate: run records in the adapter behaviour, the guarded +facade functions over them, the in-memory implementation, and the +conformance additions sp-4an.3 inherits. No interpreter call yet. + +### Changes Required: + +#### 1. The behaviour + +**File**: `lib/statifier_persistence/storage/adapter.ex` +**Changes**: add + +```elixir +@type run_status :: :active | :completed | :failed + +@type run_record :: %{ + run_id: run_id(), + status: run_status(), + content_hash: content_hash(), + identity_blob: binary(), + position_blob: binary() | nil, + failure: String.t() | nil + } + +@type run_id :: String.t() +``` + +extend `@type error` with `:run_exists | :run_not_found`, and add three +callbacks with `@doc` contracts: + +- `insert_run(opts(), run_record()) :: :ok | {:error, error()}` - refuses a + duplicate `run_id` with `{:error, :run_exists}`; the refusal must be + atomic with the write (this is what create-exactly-once rests on). +- `fetch_run(opts(), run_id()) :: {:ok, run_record()} | {:error, error()}` - + `{:error, :run_not_found}` rather than nil or a raise; blobs returned + byte-identical. +- `update_run(opts(), run_record()) :: :ok | {:error, error()}` - + `{:error, :run_not_found}` when no run exists for the id; full-record + overwrite (no partial update surface). + +`@doc` prose restates ADR-0003's rules as they bind these: no callback +decodes a blob, validates a status transition, or performs an identity +check - the facade and the lifecycle own those. + +#### 2. The in-memory adapter + +**File**: `lib/statifier_persistence/storage/in_memory.ex` +**Changes**: add `runs` map to `state`, implement the three callbacks. +`insert_run/2` uses `Agent.get_and_update/2` so the exists-check and write +are one atomic state transition. + +#### 3. The facade + +**File**: `lib/statifier_persistence/storage.ex` +**Changes**: add, mirroring the existing writers/readers exactly: + +```elixir +@type run_write_opt :: {:failure, String.t() | nil} | {:position, :persist | :skip} + +@spec insert_run(t(), Adapter.run_id(), MachineState.t(), Adapter.run_status(), + opts :: [run_write_opt()]) :: :ok | {:error, error()} +@spec update_run(t(), Adapter.run_id(), MachineState.t(), Adapter.run_status(), + opts :: [run_write_opt()]) :: :ok | {:error, error()} +@spec fetch_run(t(), Adapter.run_id()) :: {:ok, Adapter.run_record()} | {:error, error()} +@spec load_run_position(t(), Adapter.run_id(), Machine.t()) :: + {:ok, MachineState.t()} | {:error, error()} +``` + +Writers always take a `MachineState` - even a failed-at-birth run has one, +because `Interpreter.initialize/2` cannot fail - and derive `content_hash` +and `identity_blob` from that state's own machine +(`{:error, :unidentified_chart}` refusal, never a caller hash). `opts` +carries `failure:` (default nil) and `position: :persist | :skip` (default +`:persist`): `position_blob` is encoded and stored exactly when the loop +had a quiescent state to store, and is written `nil` under `:skip` on +insert. Update under `:skip` preserves the stored blob - and because the +adapter's `update_run/2` is a full-record overwrite, the facade implements +that by fetching the current record and carrying its `position_blob` +forward verbatim (safe under Phase 5's lock; until then it is what the +budget-exhaustion arm in Phase 4 needs and documents). `load_run_position/3` reuses the existing +`precheck_identity/2` and `Position.from_binary/2` path and adds +`{:error, :run_position_missing}` (new facade arm) for a run whose +`position_blob` is nil. + +#### 4. Conformance suite + +**File**: `lib/statifier_persistence/testing/storage_conformance.ex` +**Changes**: add run-record tests: insert/fetch round trip byte-identical; +duplicate insert -> `:run_exists`; update of missing -> `:run_not_found`; +fetch of missing -> `:run_not_found`; nil `position_blob` round-trips as +nil; status and failure round-trip verbatim. + +#### 5. Tests + +**Files**: `test/statifier_persistence/storage_test.exs` (facade arms: +unidentified machine refusal, `load_run_position/3` guard mismatch reusing +the two-chart fixtures, `:run_position_missing`), +`test/statifier_persistence/storage/in_memory_test.exs` (anything +InMemory-specific), conformance runs via the existing +`in_memory_conformance_test.exs` unchanged. + +### Success Criteria: + +#### Automated Verification: +- [x] Full quality gate passes (`mix quality`), coverage at or above 90. +- [x] The blob layer still decodes nothing: + `grep -nE '\bStatifier\.(Position|Machine|MachineState)[A-Za-z.]*\.[a-z_]+\(' lib/statifier_persistence/storage/adapter.ex lib/statifier_persistence/storage/in_memory.ex` + returns nothing. +- [x] The cross-revision mismatch test on `load_run_position/3` asserts a + returned `{:error, {:identity_mismatch, _, _}}`; no `assert_raise` on + any guard path. +- [x] Every new test asserting `lib/` behavior carries its one-line + `# sabotage: ...` note, sabotage-verified. +- [x] `mix.exs` `deps/0` unchanged. + +#### Manual Verification: +- [ ] The three `@callback` docs are sufficient to write the Ecto run table + without reading `in_memory.ex`. +- [ ] `insert_run/2`'s atomicity requirement is stated strongly enough that + an Ecto implementer reaches for a unique index, not a + check-then-insert. +- [ ] ADR-0004 decision 1 and the shipped callback surface agree exactly. + +**Implementation Note**: Use the project's loop gate between edits while +iterating; run the full gate as the phase gate. In interactive execution, +pause here for the human to confirm the manual testing before moving to the +next phase. In looped (`--loop`) execution, this phase's Automated +Verification gates advancement automatically (via `/wurk:commit --auto`), and +Manual Verification items are deferred and surfaced once at the end instead +of blocking here. + +--- + +## Phase 3: The executor seam and the happy-path loop + +### Overview + +The centerpiece: `StatifierPersistence.Executor` (the seam), +`StatifierPersistence.Run` (the host-facing struct), and +`StatifierPersistence.Runs` with `create/4` and `step/5` covering the +happy path - including `:done` consumption, the re-stamp tripwire, +quiescence assertion, and terminal discard. Failure semantics beyond +`:done` land in Phase 4; serialization wraps in Phase 5 (until then the +loop documents no concurrency guarantee). + +### Changes Required: + +#### 1. The executor behaviour + +**File**: `lib/statifier_persistence/executor.ex` +**Changes**: new module. + +```elixir +defmodule StatifierPersistence.Executor do + @callback execute(effect :: Statifier.Effect.t(), context :: context()) :: + :ok | {:error, term()} + + @type context :: %{run_id: String.t(), content_hash: String.t()} + @type t :: module() | (Statifier.Effect.t(), context() -> :ok | {:error, term()}) +end +``` + +With a module-private `run/3` helper normalizing module-or-fun. `@doc` +states: effects arrive in list order, one call per effect, only the public +core vocabulary ever arrives (st-ADR-0054 decision 1), `:done` and +`:budget_exhausted` never arrive, at-least-once redelivery is the contract +and idempotency (by the st-ADR-0054 decision 3 key) is the implementer's. + +#### 2. The run struct + +**File**: `lib/statifier_persistence/run.ex` +**Changes**: new module: `%StatifierPersistence.Run{run_id, status, +content_hash, failure}`, plus `from_record/1`. No position bytes on the +struct - the host-facing value answers "what state is this run in", not +"what are its bytes". + +#### 3. The lifecycle + +**File**: `lib/statifier_persistence/runs.ex` +**Changes**: new module. + +```elixir +@spec create(Storage.t(), run_id, Machine.t(), opts) :: + {:ok, Run.t(), MachineState.t()} | {:error, error()} +@spec step(Storage.t(), run_id, Machine.t(), Event.t(), opts) :: + {:ok, Run.t(), MachineState.t()} | {:discarded, Run.t()} | {:error, error()} +``` + +`opts`: `executor:` (required, `Executor.t()`), `routes:`, `invoke_types:` +(host-supplied per call - per-drive snapshots per st-ADR-0048/0051, never +read back from storage), `initialize:` (create only, passed to +`Interpreter.initialize/2`). + +`create/4`: `Interpreter.initialize(machine, opts[:initialize] || [])` -> +shared tail. Uniqueness comes from `insert_run/2`'s `:run_exists` arm, not +a pre-check. + +`step/5`, in ADR-0004 decision 3's order: + +1. `Storage.fetch_run/2`; terminal status -> `{:discarded, Run.from_record(record)}` + before any decode. +2. `Storage.load_run_position/3` - guarded load. +3. The tripwire + re-stamp: + + ```elixir + %MachineState{routes: nil, invoke_types: nil} = machine_state + + machine_state = + machine_state + |> MachineState.put_routes(opts[:routes]) + |> MachineState.put_invoke_types(opts[:invoke_types]) + ``` + + The bare match IS the tripwire (st-ADR-0064 makes blanking + unconditional; if upstream ever regresses, this fails loudly here, not + silently downstream). +4. `Interpreter.handle_event/2`. `{:error, :not_running}` maps to + `{:discarded, run}` too - it means the stored position went terminal + without the run record catching up, which the persist tail then repairs + by updating status (belt for step 1's suspenders; test covers it by + hand-writing a run record whose status lies). +5. Shared tail: partition effects - `{:done, %Done{}}` and + `{:budget_exhausted, _}` to the lifecycle, the rest to the executor in + order (Phase 3 treats executor `{:error, _}` as collect-only; re-entry + is Phase 4); status from the state (`status: :done` -> `:completed`, + else `:active`); assert `MachineState.internal_queue_empty?/1` (a false + here without `:budget_exhausted` is a loop bug - raise, deliberately, + since it is unreachable through upstream's own quiescence fold); + `insert_run`/`update_run` with the encoded position. + +#### 4. Tests + +**File**: `test/statifier_persistence/runs_test.exs` +**Changes**: new, against `InMemory` with a recording executor +(`test/support/recording_executor.ex`, an Agent-backed module implementing +the behaviour; `test/support` is outside the coverage floor and the +`Testing` namespace stays adapter-conformance-only). Cover: create +persists an `:active` run whose blob decodes to the initialized +configuration; create on existing id -> `:run_exists`; step advances and +persists; effects reach the executor in order and exclude +`:done`/`:budget_exhausted`; a chart reaching top-level final -> +`:completed` and `{:done, _}` consumed; step on completed run -> +`{:discarded, %Run{status: :completed}}` and the executor never invoked; +fun-as-executor accepted; routes/invoke_types stamped (chart with a +`` whose delivery depends on the stamp - assert via emitted effect); +missing run -> `:run_not_found`. + +Fixture charts: extend `lib/statifier_persistence/testing/charts.ex` only +if the existing two charts cannot express "reaches final on event X"; +otherwise add SCXML strings local to the test. + +### Success Criteria: + +#### Automated Verification: +- [x] Full quality gate passes (`mix quality`), coverage at or above 90. +- [x] `grep -n "rescue\|raise " lib/statifier_persistence/runs.ex` shows no + rescue and no raise other than the two deliberate invariant + violations named above (the bare-match tripwire compiles to no + `raise`; the quiescence assertion may use one explicit raise with a + message naming it a loop bug). +- [x] The executor-order test asserts the exact effect list order, not set + membership. +- [x] Every new test asserting `lib/` behavior is sabotage-verified with + its `# sabotage: ...` note; the tripwire test's mutation is the loop + skipping the re-stamp. +- [x] `grep -rn "StatifierPersistence.Testing" lib/ --include=*.ex | grep -v "lib/statifier_persistence/testing/"` + returns nothing (one-way namespace rule holds). + +#### Manual Verification: +- [ ] In `iex -S mix`: create a run, step it to final across two separate + `Storage.new/2` handles (simulating a restart between steps), and + confirm the second handle resumes purely from stored bytes. +- [ ] The `Runs` moduledoc's loop description matches ADR-0004 decision 3 + word for word where it quotes the order. +- [ ] Confirm no Session instruction tuples can reach the executor by + reading the partition against `deps/statifier/lib/statifier/effect.ex`'s + `@type core`. + +**Implementation Note**: Use the project's loop gate between edits while +iterating; run the full gate as the phase gate. In interactive execution, +pause here for the human to confirm the manual testing before moving to the +next phase. In looped (`--loop`) execution, this phase's Automated +Verification gates advancement automatically (via `/wurk:commit --auto`), and +Manual Verification items are deferred and surfaced once at the end instead +of blocking here. + +--- + +## Phase 4: Failure semantics - error re-entry, budget exhaustion, fail/4 + +### Overview + +Errors are events, end to end: executor failures re-enter the chart, +budget exhaustion becomes a failed run, and the host gets `fail/4` for +abandonment. At-least-once gets its proving test. + +### Changes Required: + +#### 1. Error re-entry in the loop + +**File**: `lib/statifier_persistence/runs.ex` +**Changes**: replace Phase 3's collect-only handling. After the primary +effect pass, for each collected failure, classify by tag: + +- `:invoke`, `:cancel_invoke`, `:autoforward`, `:send`, `:send_delayed`, + `:cancel` -> + `Interpreter.deliver_internal(state, :platform, "error.communication", origin, opts)` + (st-ADR-0051's table: a failure after the core accepted the effect is + the failed-communication row, for both classes; the core alone mints + `error.execution`, at planning time) +- `:log`, `:datamodel_change`, `:datamodel_init`, trace tags -> discarded. + +`origin` mirrors the shapes upstream's session builds on its own +failed-communication paths (`deps/statifier/lib/statifier/session.ex:1971` +region, `invoke_error/4` / the ADR-0039 decision 4 write) - read +`Cause.origin()`'s type at implementation time and reuse its constructors +verbatim; this package invents no origin vocabulary. Each `deliver_internal` +returns new state + effects; those effects go through the executor too +(same partition), but their failures are NOT re-entered (single wave, +ADR-0004 decision 4) - they are dropped after a `:telemetry`-free log via +the returned state only, i.e. the step result reflects the post-re-entry +state and the run persists as that state dictates. `deliver_internal` +returning `{:error, :not_running}` (the re-entry itself hit a final state) +ends the wave; the tail then reads status normally. + +#### 2. Budget exhaustion + +**Changes** (same file): `{:budget_exhausted, %BudgetExhausted{}}` in the +lifecycle partition -> run status `:failed`, +`failure: "budget_exhausted: rounds"`, position NOT +persisted (`position: :skip`) - the pre-step blob (or nil at create) +remains, and the record explains why. Returns +`{:error, {:budget_exhausted, effect}}` to the caller after persisting the +status, so the host sees both the durable state and the reason. + +#### 3. `fail/4` + +**Changes** (same file): + +```elixir +@spec fail(Storage.t(), run_id, String.t(), opts) :: + {:ok, Run.t()} | {:discarded, Run.t()} | {:error, error()} +``` + +Terminal-status check first (discard, same as step); otherwise +`update_run` to `:failed` with the reason, position untouched +(`position: :skip`). No interpreter involvement - abandonment is a host +decision about the run, not a chart transition. + +#### 4. Tests + +**File**: `test/statifier_persistence/runs_test.exs` (extend), fixtures as +needed. Cover: an executor failing an `:invoke` start -> chart observes +`error.communication` (fixture chart with a transition on it, per +st-ADR-0051's failed-communication row) and the persisted position +reflects the error-handling state; a failing `:send` -> the same +`error.communication` re-entry; a failing `:log` -> run unaffected; single-wave: an +executor failing every effect including re-entry effects terminates and +persists; budget exhaustion (fixture chart with an eventless loop or a +raise cycle - upstream's `max_macrostep_rounds` opt on +`MachineState.new/2` keeps the fixture small) -> `:failed` run, prior blob +intact, `{:error, {:budget_exhausted, _}}` returned; `fail/4` on active -> +`:failed` with reason; `fail/4` on terminal -> `{:discarded, _}`. + +**At-least-once proof**: wrap `InMemory` in a test adapter whose +`update_run/2` fails once with `{:error, {:adapter, :injected}}` (a +delegating module in `test/support/`). Step -> error surfaces; step again +with the same event -> succeeds, and the recording executor received the +same effect list twice with identical deterministic keys +(`send_id`/`ordinal`/`invoke_id` fields equal across the two deliveries). +This is the bead's "a crash between step and persist re-drives the same +event and re-emits the same effects" made executable. + +### Success Criteria: + +#### Automated Verification: +- [x] Full quality gate passes (`mix quality`), coverage at or above 90. +- [x] The at-least-once test asserts field-level equality of the + deterministic keys across redelivery, not just list length. +- [x] `grep -n "error\.execution" lib/` returns nothing (this package + never mints `error.execution`, per st-ADR-0051's table), and + `grep -n "error\.communication" lib/statifier_persistence/runs.ex` + shows the name only as a `deliver_internal/5` argument - never a + hand-built `%Statifier.Event{}` construction. +- [x] Every new test asserting `lib/` behavior is sabotage-verified with + its `# sabotage: ...` note; the re-entry test's mutation is dropping + the failure classification (treating invoke failure as observational). + +#### Manual Verification: +- [ ] The origin shapes passed to `deliver_internal/5` match what + `deps/statifier/lib/statifier/session.ex` builds on its own + failed-communication paths (the ADR-0039 decision 4 writes), field + for field. +- [ ] The `failure` strings are useful in a psql console: short, prefixed, + no inspect-dump of internals. +- [ ] Read the single-wave rule's implementation against ADR-0004 decision + 4 and confirm a deterministically failing executor cannot recurse. + +**Implementation Note**: Use the project's loop gate between edits while +iterating; run the full gate as the phase gate. In interactive execution, +pause here for the human to confirm the manual testing before moving to the +next phase. In looped (`--loop`) execution, this phase's Automated +Verification gates advancement automatically (via `/wurk:commit --auto`), and +Manual Verification items are deferred and surfaced once at the end instead +of blocking here. + +--- + +## Phase 5: Pluggable per-run serialization + +### Overview + +The ordering guarantee as a strategy: the behaviour, the adapter-lock +default, the in-memory lock, and the loop running its tail inside +`with_run/3`. + +### Changes Required: + +#### 1. The strategy behaviour + +**File**: `lib/statifier_persistence/serialization.ex` +**Changes**: new module. + +```elixir +defmodule StatifierPersistence.Serialization do + @callback with_run(config :: term(), run_id :: String.t(), fun :: (-> result)) :: + {:ok, result} | {:error, term()} + when result: var +end +``` + +`@doc` states the guarantee a strategy must provide: for one `run_id`, two +`with_run/3` bodies never overlap, and completed bodies are observed in +execution order. It explicitly does NOT promise cross-run ordering or +fairness. Strategy selection is a `step`/`create`/`fail` opt: +`serialization: {module, config}`, defaulting to +`{AdapterLock, store}`. + +#### 2. The default strategy + +**File**: `lib/statifier_persistence/serialization/adapter_lock.ex` +**Changes**: `with_run({%Storage{} = store}, run_id, fun)` delegates to +`store.adapter.lock_run(store.opts, run_id, fun)` when exported, else +`{:error, {:serialization, :not_supported}}`. + +#### 3. The adapter callback + +**File**: `lib/statifier_persistence/storage/adapter.ex` +**Changes**: optional callback (joins `isolate/1` in +`@optional_callbacks`): + +```elixir +@callback lock_run(opts(), run_id(), (-> result)) :: + {:ok, result} | {:error, error()} + when result: var +``` + +`@doc`: mutual exclusion per `run_id`; the callback runs `fun` while +holding the exclusion and releases on any exit (including a raise +escaping `fun` - the lock must not leak); an Ecto adapter implements it as +a transaction-scoped row lock (sp-4an.3). + +#### 4. The in-memory lock + +**File**: `lib/statifier_persistence/storage/in_memory.ex` +**Changes**: implement `lock_run/3` with a per-run lock table in the +Agent's state (`locks :: %{run_id => reference()}`): acquire via +`Agent.get_and_update/2` (insert-if-absent), spin with a small sleep on +contention, release in an `after` block. Simple and honest for a reference +adapter; the `@doc` says a production adapter should prefer its backend's +native lock. + +#### 5. Wire the loop + +**File**: `lib/statifier_persistence/runs.ex` +**Changes**: `create/4`, `step/5`, `fail/4` run their fetch-to-persist tail +inside `with_run/3`; `{:error, {:serialization, _}}` surfaces unchanged. +Remove Phase 3's "no concurrency guarantee" doc note. + +#### 6. ADR-0003 amendment + conformance + +**Files**: `docs/adr/0003-storage-adapter-behaviour-and-the-identity-guard.md` +(dated amendment under Status recording `lock_run/3` joining the optional +contract surface, exactly as the `isolate/1` amendment models), +`lib/statifier_persistence/testing/storage_conformance.ex` (when the +adapter exports `lock_run/3`: two concurrent `with_run` bodies on one run +id never overlap - assert via a shared Agent recording enter/exit pairs; +lock released after a raising fun). + +#### 7. Tests + +**File**: `test/statifier_persistence/runs_test.exs` (extend): two +processes stepping one run concurrently produce a final persisted state +equal to some serial order of the two events (assert via a chart whose +final datamodel or configuration distinguishes orders from interleavings); +`serialization: {module, config}` override honored (a test strategy that +records invocation); missing `lock_run` adapter + default strategy -> +`{:error, {:serialization, :not_supported}}`. + +### Success Criteria: + +#### Automated Verification: +- [x] Full quality gate passes (`mix quality`), coverage at or above 90. +- [x] The concurrency test runs the two steps in genuinely concurrent + processes (`Task.async`), not sequenced by the test. +- [x] The lock-release-on-raise conformance test passes against `InMemory`. +- [x] Every new test asserting `lib/` behavior is sabotage-verified with + its `# sabotage: ...` note; the overlap test's mutation is `lock_run` + running `fun` without the exclusion. +- [x] The ADR-0003 amendment names `lock_run/3` and the file stays free of + typographic punctuation. + +#### Manual Verification: +- [ ] Read `with_run/3`'s guarantee prose against what a Postgres + `SELECT ... FOR UPDATE` inside a transaction actually provides - + sp-4an.3 must be able to implement it without weasel words. +- [ ] Confirm a job-queue strategy (single consumer per run id) can satisfy + the same `@callback` shape with no loop change - the no-API-change + swap the charter asks for. +- [ ] The InMemory spin-lock cannot deadlock a test run (bounded sleep, no + nested `with_run` on the same run id inside the loop). + +**Implementation Note**: Use the project's loop gate between edits while +iterating; run the full gate as the phase gate. In interactive execution, +pause here for the human to confirm the manual testing before moving to the +next phase. In looped (`--loop`) execution, this phase's Automated +Verification gates advancement automatically (via `/wurk:commit --auto`), and +Manual Verification items are deferred and surfaced once at the end instead +of blocking here. + +--- + +## Testing Strategy + +### Unit Tests: + +- `test/statifier_persistence/storage_test.exs` - the new facade arms + (guarded run loads, unidentified refusals, `:run_position_missing`). +- `test/statifier_persistence/runs_test.exs` - the lifecycle and loop: + happy path, discard, error re-entry, budget exhaustion, at-least-once + redelivery, concurrency. +- Conformance (`lib/statifier_persistence/testing/storage_conformance.ex`, + run via the existing InMemory conformance test) - run-record storage + semantics and, when exported, `lock_run/3` semantics, so sp-4an.3 + inherits both. +- `test/support/` - the recording executor and the failure-injecting + adapter wrapper; outside the coverage floor by `coveralls.json`. + +Key edge cases: event to a completed run (before and after decode - the +lying-status record); create on an existing run id; executor failure on +every effect class including during the re-entry wave; budget exhaustion +at create and at step; adapter write failure between execute and persist +(the at-least-once proof); identity mismatch on a run load; a run record +with nil position blob reached by a step. + +### Manual Testing Steps: + +1. In `iex -S mix`: create a run against `InMemory`, step it through a + multi-state chart across two separate `Storage.new/2` handles, confirm + resume-from-bytes and final `:completed`. +2. Deliver one more event to the completed run; confirm `{:discarded, _}` + and the executor untouched. +3. Wire an executor that fails invoke starts; watch the chart take its + `error.execution` transition and the run persist the error state. +4. Run the umbrella's terminology scan from `docs/terminology-firewall.md` + over the full diff before any push. + +## References + +- Bead: `sp-4an.2.1` (task; epic `sp-4an.2` carries the contract detail and + the 2026-08-21 st-otr0/ADR-0064 notes) +- Prior plan (house pattern): + `docs/plans/260821-sp-5qa-storage-behaviour-and-identity-guard.md` - its + settled question 2 is resolved here by Key Discoveries' last bullet +- This repo's ADRs: `docs/adr/0002-configurable-keys-and-table-names.md`, + `docs/adr/0003-storage-adapter-behaviour-and-the-identity-guard.md` +- Upstream contract (vendored at `deps/statifier/`, pin `68b814a`, + `mix.lock:18`): + - `deps/statifier/lib/statifier/interpreter.ex:43-92, 259-260, 477-502, 505-545` + - `deps/statifier/lib/statifier/machine_state.ex:361-384, 450, 680-681, 801-812` + - `deps/statifier/lib/statifier/effect.ex:23-44, 120-144` + - `deps/statifier/lib/statifier/position.ex:105-117, 164-180, 267-277` + - `deps/statifier/lib/statifier/session/effects.ex:299-362` (failure + classification to mirror) + - `deps/statifier/docs/persistence.md`, `docs/durable-timers.md` + - st-ADR-0035, 0039, 0048, 0051, 0052, 0054, 0059, 0060, 0063, 0064 +- Existing facade to mirror: `lib/statifier_persistence/storage.ex:110-197` +- Gate configuration: `.quality.exs`, `coveralls.json` + +## Deferred Manual Verification + +Manual verification items are deferred during looped (--loop) execution and +surfaced here once, rather than blocking after each phase. Confirm these +before considering the plan fully landed. + +### Phase 1 + +- [ ] The record reads as a decision; citing "ADR-0004 decision 3" ends the + argument about the loop's order. +- [ ] Cross-repo citations use `st-ADR-NNNN`; bare `ADR-NNNN` is only this + repo's own. +- [ ] Nothing contradicts ADR-0002/0003 or st-ADR-0052/0054/0059/0060/0064; + upstream decisions are adopted by reference, not restated. + +**Implementation Note**: Use the project's loop gate between edits while +iterating; run the full gate as the phase gate. In interactive execution, +pause here for the human to confirm the manual testing before moving to the +next phase. In looped (`--loop`) execution, this phase's Automated +Verification gates advancement automatically (via `/wurk:commit --auto`), and +Manual Verification items are deferred and surfaced once at the end instead +of blocking here. + +--- + +### Phase 2 + +- [ ] The three `@callback` docs are sufficient to write the Ecto run table + without reading `in_memory.ex`. +- [ ] `insert_run/2`'s atomicity requirement is stated strongly enough that + an Ecto implementer reaches for a unique index, not a + check-then-insert. +- [ ] ADR-0004 decision 1 and the shipped callback surface agree exactly. + +**Implementation Note**: Use the project's loop gate between edits while +iterating; run the full gate as the phase gate. In interactive execution, +pause here for the human to confirm the manual testing before moving to the +next phase. In looped (`--loop`) execution, this phase's Automated +Verification gates advancement automatically (via `/wurk:commit --auto`), and +Manual Verification items are deferred and surfaced once at the end instead +of blocking here. + +--- + +### Phase 3 + +- [ ] In `iex -S mix`: create a run, step it to final across two separate + `Storage.new/2` handles (simulating a restart between steps), and + confirm the second handle resumes purely from stored bytes. +- [ ] The `Runs` moduledoc's loop description matches ADR-0004 decision 3 + word for word where it quotes the order. +- [ ] Confirm no Session instruction tuples can reach the executor by + reading the partition against `deps/statifier/lib/statifier/effect.ex`'s + `@type core`. + +**Implementation Note**: Use the project's loop gate between edits while +iterating; run the full gate as the phase gate. In interactive execution, +pause here for the human to confirm the manual testing before moving to the +next phase. In looped (`--loop`) execution, this phase's Automated +Verification gates advancement automatically (via `/wurk:commit --auto`), and +Manual Verification items are deferred and surfaced once at the end instead +of blocking here. + +--- + +### Phase 4 + +- [ ] The origin shapes passed to `deliver_internal/5` match what + `deps/statifier/lib/statifier/session.ex` builds on its own + failed-communication paths (the ADR-0039 decision 4 writes), field + for field. +- [ ] The `failure` strings are useful in a psql console: short, prefixed, + no inspect-dump of internals. +- [ ] Read the single-wave rule's implementation against ADR-0004 decision + 4 and confirm a deterministically failing executor cannot recurse. + +**Implementation Note**: Use the project's loop gate between edits while +iterating; run the full gate as the phase gate. In interactive execution, +pause here for the human to confirm the manual testing before moving to the +next phase. In looped (`--loop`) execution, this phase's Automated +Verification gates advancement automatically (via `/wurk:commit --auto`), and +Manual Verification items are deferred and surfaced once at the end instead +of blocking here. + +--- + +### Phase 5 + +- [ ] Read `with_run/3`'s guarantee prose against what a Postgres + `SELECT ... FOR UPDATE` inside a transaction actually provides - + sp-4an.3 must be able to implement it without weasel words. +- [ ] Confirm a job-queue strategy (single consumer per run id) can satisfy + the same `@callback` shape with no loop change - the no-API-change + swap the charter asks for. +- [ ] The InMemory spin-lock cannot deadlock a test run (bounded sleep, no + nested `with_run` on the same run id inside the loop). + +**Implementation Note**: Use the project's loop gate between edits while +iterating; run the full gate as the phase gate. In interactive execution, +pause here for the human to confirm the manual testing before moving to the +next phase. In looped (`--loop`) execution, this phase's Automated +Verification gates advancement automatically (via `/wurk:commit --auto`), and +Manual Verification items are deferred and surfaced once at the end instead +of blocking here. + +--- diff --git a/lib/statifier_persistence/executor.ex b/lib/statifier_persistence/executor.ex new file mode 100644 index 0000000..02094e0 --- /dev/null +++ b/lib/statifier_persistence/executor.ex @@ -0,0 +1,55 @@ +defmodule StatifierPersistence.Executor do + @moduledoc """ + The seam through which a stepped run's effects reach the host (ADR-0004 + decision 4). + + An executor is a module implementing this behaviour, or an arity-2 fun + accepted anywhere a module is. `StatifierPersistence.Runs` invokes it once + per effect, in the effect list's own order, for every effect the lifecycle + does not consume itself. + """ + + @typedoc """ + What `c:execute/2` receives alongside each effect: the run's + caller-supplied id and the content hash of the chart revision it runs - + enough to key idempotency storage and telemetry without another lookup. + """ + @type context :: %{run_id: String.t(), content_hash: String.t()} + + @typedoc """ + An executor: a module implementing this behaviour, or an arity-2 fun with + `c:execute/2`'s own signature, accepted anywhere a module is (ADR-0004 + decision 4). + """ + @type t :: module() | (Statifier.Effect.t(), context() -> :ok | {:error, term()}) + + @doc """ + Executes one effect against the outside world. + + The contract, per ADR-0004 decisions 3 and 4: + + - Effects arrive in list order, one call per effect. + - Only the public effect vocabulary (`t:Statifier.Effect.t/0`) ever + arrives - never Session instruction tuples (st-ADR-0054 decision 1). + - `:done` and `:budget_exhausted` never arrive: the lifecycle consumes + both into run status itself. + - At-least-once redelivery is the contract: a crash between step and + persist re-drives the same event and re-emits the same effects carrying + identical deterministic keys (st-ADR-0054 decision 3, st-ADR-0059's + `timer_counter` ordinal). The loop never dedupes; idempotency by that + key is the implementer's. + """ + @callback execute(effect :: Statifier.Effect.t(), context :: context()) :: + :ok | {:error, term()} + + # Package-internal: normalizes the module-or-fun shapes of `t:t/0` into + # one call. `StatifierPersistence.Runs` is the only intended caller. + @doc false + @spec run(executor :: t(), effect :: Statifier.Effect.t(), context :: context()) :: + :ok | {:error, term()} + def run(executor, effect, context) when is_atom(executor), + do: executor.execute(effect, context) + + def run(executor, effect, context) when is_function(executor, 2), + do: executor.(effect, context) +end diff --git a/lib/statifier_persistence/run.ex b/lib/statifier_persistence/run.ex new file mode 100644 index 0000000..811962e --- /dev/null +++ b/lib/statifier_persistence/run.ex @@ -0,0 +1,42 @@ +defmodule StatifierPersistence.Run do + @moduledoc """ + The host-facing view of a durable run: what state is this run in - never + what its bytes are. The position blob stays on the stored record + (ADR-0004 decision 1), loaded only through the guarded + `StatifierPersistence.Storage.load_run_position/3` path; this struct + carries the fields a host reads to decide what to do with a run. + """ + + alias StatifierPersistence.Storage.Adapter + + @enforce_keys [:run_id, :status, :content_hash] + defstruct [:run_id, :status, :content_hash, :failure] + + @type t :: %__MODULE__{ + run_id: Adapter.run_id(), + status: Adapter.run_status(), + content_hash: Adapter.content_hash(), + failure: String.t() | nil + } + + @doc """ + Builds the host-facing struct from a stored + `t:StatifierPersistence.Storage.Adapter.run_record/0`, dropping the two + blob fields (`identity_blob`, `position_blob`) and carrying everything + else verbatim. + """ + @spec from_record(Adapter.run_record()) :: t() + def from_record(%{ + run_id: run_id, + status: status, + content_hash: content_hash, + failure: failure + }) do + %__MODULE__{ + run_id: run_id, + status: status, + content_hash: content_hash, + failure: failure + } + end +end diff --git a/lib/statifier_persistence/runs.ex b/lib/statifier_persistence/runs.ex new file mode 100644 index 0000000..6fcf7ba --- /dev/null +++ b/lib/statifier_persistence/runs.ex @@ -0,0 +1,536 @@ +defmodule StatifierPersistence.Runs do + @moduledoc """ + The run lifecycle: create and step durable runs with no live Session + process, the loop this package exists to package. + + A step runs in ADR-0004 decision 3's order, and the order is the + contract: liveness check on the run record -> load (guarded) -> re-stamp + `routes`/`invoke_types` unconditionally (with the nil tripwire from + st-ADR-0064: the fields are pattern-matched `nil` before stamping, so an + upstream regression fails loudly here, not silently downstream) -> step + via `Interpreter.handle_event/2` -> execute effects via the executor + seam -> consume `:done` and `:budget_exhausted` into run status -> assert + `MachineState.internal_queue_empty?/1` -> persist. + + Effect execution is at-least-once: a crash between step and persist + re-drives the same event and re-emits the same effects with identical + deterministic keys (st-ADR-0054 decision 3, st-ADR-0059), and this loop + never dedupes - idempotency is the consumer's. `:done` is the only path + to `:completed` (ADR-0004 decision 6); an event delivered to a terminal + run is discarded with a typed `{:discarded, run}` result, never an + exception and never a silent step. + + Executor failures on actionable effects re-enter the chart as + `error.communication` events through `Statifier.Interpreter.deliver_internal/5` + (st-ADR-0039's seam), per st-ADR-0051's failed-communication row: the core + alone mints the planning-time execution-error events, before any effect is + emitted, so every failure an executor can report re-enters uniformly as + `error.communication` (ADR-0004 + decision 4). Failures on observational effects are discarded. Re-entry is + single-wave per step: effects the re-entries emit are executed too, but + their failures are not re-entered again, so a deterministically failing + executor cannot loop this library. + + Concurrent deliveries to one run are ordered by a pluggable per-run + serialization strategy (ADR-0004 decision 5): every entry point runs its + fetch-to-persist tail inside the strategy's + `c:StatifierPersistence.Serialization.with_run/3`, selected per call with + `serialization: {module, config}` and defaulting to + `{StatifierPersistence.Serialization.AdapterLock, store}` - the adapter's + own optional `lock_run/3`. A strategy refusal surfaces unchanged as + `{:error, {:serialization, reason}}`. + """ + + alias Statifier.{Event, Interpreter, Machine, MachineState} + + alias Statifier.Effect.{ + Autoforward, + BudgetExhausted, + Cancel, + CancelInvoke, + Invoke, + Send, + SendDelayed + } + + alias Statifier.Machine.Identity + alias StatifierPersistence.{Executor, Run, Storage} + alias StatifierPersistence.Serialization.AdapterLock + alias StatifierPersistence.Storage.Adapter + + @typedoc "A run's caller-supplied opaque key (ADR-0004 decision 2)." + @type run_id :: Adapter.run_id() + + @typedoc """ + This module's error vocabulary: the facade's arms, unflattened, plus the + `{:budget_exhausted, payload}` arm returned after a budget-exhausted step + or create has persisted its `:failed` run record, plus the serialization + strategy's own refusal, surfaced unchanged + (`{:serialization, :not_supported}` from the default strategy over an + adapter with no `lock_run/3`). + """ + @type error :: + Storage.error() + | {:budget_exhausted, BudgetExhausted.t()} + | {:serialization, term()} + + @typedoc """ + Options `create/4` and `step/5` accept: + + - `executor:` (required) - the `t:StatifierPersistence.Executor.t/0` + every non-lifecycle effect is handed to, in list order. + - `routes:` - the `t:Statifier.Send.Routes.t/0` snapshot stamped onto the + loaded position before the step; host-supplied per call, never read + back from storage (st-ADR-0048). Defaults to `nil`, "no determination + made". + - `invoke_types:` - the `t:Statifier.Invoke.Types.t/0` snapshot, stamped + the same way (st-ADR-0051). Defaults to `nil`, "the built-in set only". + - `initialize:` (`create/4` only) - passed to + `Statifier.Interpreter.initialize/2` unchanged. + - `serialization:` - the `{module, config}` per-run serialization + strategy the fetch-to-persist tail runs inside (ADR-0004 decision 5; + `fail/4` accepts it too). Defaults to + `{StatifierPersistence.Serialization.AdapterLock, store}`. + """ + @type opt :: + {:executor, Executor.t()} + | {:routes, MachineState.routes()} + | {:invoke_types, MachineState.invoke_types()} + | {:initialize, keyword()} + | {:serialization, {module(), term()}} + + @doc """ + Creates a run: `Statifier.Interpreter.initialize/2` (which cannot fail), + then the shared persist tail - effects through the executor seam, + `:done`/`:budget_exhausted` consumed into run status, quiescence + asserted, the record inserted with its encoded position. + + Create-exactly-once rests on the adapter's atomic `:run_exists` refusal + (ADR-0004 decision 2), not on a pre-check here: creating an existing + `run_id` returns `{:error, :run_exists}`. + + A create whose `initialize/2` exhausts its macrostep budget persists a + `:failed` run with no position blob (there is no quiescent position to + store - ADR-0004 decision 1) and then returns + `{:error, {:budget_exhausted, payload}}`, so the caller sees both the + durable state and the reason. + """ + @spec create(store :: Storage.t(), run_id :: run_id(), machine :: Machine.t(), opts :: [opt()]) :: + {:ok, Run.t(), MachineState.t()} | {:error, error()} + def create(%Storage{} = store, run_id, %Machine{} = machine, opts) do + executor = Keyword.fetch!(opts, :executor) + + {machine_state, effects} = + Interpreter.initialize(machine, Keyword.get(opts, :initialize, [])) + + serialized(store, run_id, opts, fn -> + persist_tail(store, run_id, machine_state, effects, executor, :insert) + end) + end + + @doc """ + Delivers one external event to a run, in ADR-0004 decision 3's order (the + moduledoc quotes it). + + An event delivered to a terminal run returns `{:discarded, run}` from the + run record alone, before any position decode. `handle_event/2`'s + `{:error, :not_running}` arm is the structural backstop for a run record + whose `:active` status lies about a terminal stored position: it discards + too, and repairs the record's status to `:completed` on the way out. + """ + @spec step( + store :: Storage.t(), + run_id :: run_id(), + machine :: Machine.t(), + event :: Event.t(), + opts :: [opt()] + ) :: {:ok, Run.t(), MachineState.t()} | {:discarded, Run.t()} | {:error, error()} + def step(%Storage{} = store, run_id, %Machine{} = machine, %Event{} = event, opts) do + executor = Keyword.fetch!(opts, :executor) + + serialized(store, run_id, opts, fn -> + step_tail(store, run_id, machine, event, opts, executor) + end) + end + + @spec step_tail(Storage.t(), run_id(), Machine.t(), Event.t(), [opt()], Executor.t()) :: + {:ok, Run.t(), MachineState.t()} | {:discarded, Run.t()} | {:error, error()} + defp step_tail(store, run_id, machine, event, opts, executor) do + case Storage.fetch_run(store, run_id) do + {:ok, %{status: status} = run_record} when status in [:completed, :failed] -> + {:discarded, Run.from_record(run_record)} + + {:ok, _run_record} -> + with {:ok, machine_state} <- Storage.load_run_position(store, run_id, machine) do + step_loaded(store, run_id, machine_state, event, opts, executor) + end + + {:error, _reason} = error -> + error + end + end + + @doc """ + Abandons a run: the only host-driven terminal transition (ADR-0004 + decision 6). No interpreter is involved - abandonment is a host decision + about the run, not a chart transition - so the stored position is left + untouched and only the record's status and failure reason change. + + A terminal run is discarded, same as `step/5`: `{:discarded, run}`. + `reason` is the short string stored as the run's `failure` - keep it a + prefixed, console-readable reason, not an inspect dump. + + `opts` accepts `serialization:` only - the same `{module, config}` + strategy `create/4` and `step/5` take, with the same default. + """ + @spec fail(store :: Storage.t(), run_id :: run_id(), reason :: String.t(), opts :: keyword()) :: + {:ok, Run.t()} | {:discarded, Run.t()} | {:error, error()} + def fail(%Storage{} = store, run_id, reason, opts \\ []) when is_binary(reason) do + serialized(store, run_id, opts, fn -> + fail_tail(store, run_id, reason) + end) + end + + @spec fail_tail(Storage.t(), run_id(), String.t()) :: + {:ok, Run.t()} | {:discarded, Run.t()} | {:error, error()} + defp fail_tail(store, run_id, reason) do + case Storage.fetch_run(store, run_id) do + {:ok, %{status: status} = run_record} when status in [:completed, :failed] -> + {:discarded, Run.from_record(run_record)} + + {:ok, run_record} -> + with :ok <- Storage.update_run_status(store, run_id, :failed, failure: reason) do + {:ok, Run.from_record(%{run_record | status: :failed, failure: reason})} + end + + {:error, _reason} = error -> + error + end + end + + # Runs `fun` - one entry point's whole fetch-to-persist tail - inside the + # selected serialization strategy's `with_run/3` (ADR-0004 decision 5), + # unwrapping the strategy's `{:ok, result}` envelope back to the tail's + # own result. A strategy refusal (`{:error, {:serialization, _}}` from + # the default over an adapter with no `lock_run/3`) surfaces unchanged. + # Nothing inside any tail calls back into this function, so `with_run/3` + # never nests on one run id. + @spec serialized(Storage.t(), run_id(), keyword(), (-> result)) :: + result | {:error, error()} + when result: term() + defp serialized(store, run_id, opts, fun) do + {strategy, config} = Keyword.get(opts, :serialization, {AdapterLock, store}) + + case strategy.with_run(config, run_id, fun) do + {:ok, result} -> result + {:error, _reason} = error -> error + end + end + + @spec step_loaded( + Storage.t(), + run_id(), + MachineState.t(), + Event.t(), + [opt()], + Executor.t() + ) :: {:ok, Run.t(), MachineState.t()} | {:discarded, Run.t()} | {:error, error()} + defp step_loaded(store, run_id, machine_state, event, opts, executor) do + # The bare match IS the st-ADR-0064 tripwire: `from_binary/2` blanks + # both fields unconditionally on decode, so if upstream ever stops, + # this fails loudly here rather than silently resuming a stale + # snapshot downstream. + %MachineState{routes: nil, invoke_types: nil} = machine_state + + machine_state = + machine_state + |> MachineState.put_routes(opts[:routes]) + |> MachineState.put_invoke_types(opts[:invoke_types]) + + case Interpreter.handle_event(machine_state, event) do + {:ok, machine_state, effects} -> + persist_tail(store, run_id, machine_state, effects, executor, :update) + + {:error, :not_running} -> + repair_terminal(store, run_id, machine_state) + end + end + + # The stored position went terminal without the run record catching up + # (the record said `:active`, `handle_event/2` said `:not_running`). + # Discard the event and repair the record's status: `:done` is the only + # chart-driven terminal state (ADR-0004 decision 6), so the repaired + # status is `:completed`. `position: :skip` carries the stored blob + # forward untouched - nothing stepped. + @spec repair_terminal(Storage.t(), run_id(), MachineState.t()) :: + {:discarded, Run.t()} | {:error, error()} + defp repair_terminal(store, run_id, machine_state) do + with :ok <- Storage.update_run(store, run_id, machine_state, :completed, position: :skip) do + {:discarded, run(run_id, :completed, Machine.identity(machine_state.machine))} + end + end + + # The shared tail of `create/4` and `step/5`, in ADR-0004 decision 3's + # order: partition `:done`/`:budget_exhausted` to the lifecycle and hand + # everything else to the executor in list order, derive the run status, + # assert quiescence, persist. The identity refusal runs first - the same + # `:unidentified_chart` arm the facade's writers return - so no effect is + # executed for a run that cannot be persisted at all. + @spec persist_tail( + Storage.t(), + run_id(), + MachineState.t(), + [Statifier.Effect.t()], + Executor.t(), + :insert | :update + ) :: {:ok, Run.t(), MachineState.t()} | {:error, error()} + defp persist_tail(store, run_id, machine_state, effects, executor, write) do + case Machine.identity(machine_state.machine) do + nil -> + {:error, :unidentified_chart} + + identity -> + {lifecycle, executable} = Enum.split_with(effects, &lifecycle_effect?/1) + context = %{run_id: run_id, content_hash: identity.content_hash} + + failures = execute_effects(executable, executor, context) + + {machine_state, lifecycle} = + reenter_failures(machine_state, failures, executor, context, lifecycle) + + status = run_status(machine_state, lifecycle) + :ok = assert_quiescent(machine_state, lifecycle) + + with :ok <- write_run(write, store, run_id, machine_state, status, lifecycle) do + tail_result(run_id, status, identity, lifecycle, machine_state) + end + end + end + + # The persisted-run return: `{:ok, ...}` for a live or completed run, + # `{:error, {:budget_exhausted, payload}}` AFTER the `:failed` record is + # durable, so the caller sees both the state and the reason. + @spec tail_result( + run_id(), + Adapter.run_status(), + Identity.t(), + [Statifier.Effect.t()], + MachineState.t() + ) :: {:ok, Run.t(), MachineState.t()} | {:error, error()} + defp tail_result(run_id, status, identity, lifecycle, machine_state) do + case budget_effect(lifecycle) do + nil -> {:ok, run(run_id, status, identity), machine_state} + %BudgetExhausted{} = payload -> {:error, {:budget_exhausted, payload}} + end + end + + # ADR-0004 decision 4's error re-entry: each executor failure on an + # actionable effect re-enters the chart as `error.communication` through + # `Interpreter.deliver_internal/5` - st-ADR-0051's failed-communication + # row, the only row an executor failure can be, because the core accepted + # the effect before emitting it. Observational failures are discarded: + # observation must never steer a run. Effects the re-entries emit go + # through the executor too, but their failures are NOT re-entered (single + # wave), so a deterministically failing executor cannot recurse here. + # `{:error, :not_running}` (a re-entry itself reached a final state) ends + # the wave, as does a re-entry exhausting the macrostep budget - the tail + # then reads status normally. A wave is never opened into a state the + # primary pass already reported budget-exhausted. + @spec reenter_failures( + MachineState.t(), + [{Statifier.Effect.t(), term()}], + Executor.t(), + Executor.context(), + [Statifier.Effect.t()] + ) :: {MachineState.t(), [Statifier.Effect.t()]} + defp reenter_failures(machine_state, failures, executor, context, lifecycle) do + if budget_exhausted?(lifecycle) do + {machine_state, lifecycle} + else + Enum.reduce_while(failures, {machine_state, lifecycle}, fn {effect, _reason}, acc -> + reenter_one(effect, acc, executor, context) + end) + end + end + + @spec reenter_one( + Statifier.Effect.t(), + {MachineState.t(), [Statifier.Effect.t()]}, + Executor.t(), + Executor.context() + ) :: {:cont | :halt, {MachineState.t(), [Statifier.Effect.t()]}} + defp reenter_one(effect, {_machine_state, _lifecycle} = acc, executor, context) do + case reentry_origin(effect) do + :observational -> {:cont, acc} + {origin, opts} -> deliver_reentry(acc, origin, opts, executor, context) + end + end + + @spec deliver_reentry( + {MachineState.t(), [Statifier.Effect.t()]}, + Statifier.Event.Cause.origin(), + keyword(), + Executor.t(), + Executor.context() + ) :: {:cont | :halt, {MachineState.t(), [Statifier.Effect.t()]}} + defp deliver_reentry({machine_state, lifecycle} = acc, origin, opts, executor, context) do + case Interpreter.deliver_internal( + machine_state, + :platform, + "error.communication", + origin, + opts + ) do + {:ok, machine_state, wave_effects} -> + {wave_lifecycle, wave_executable} = Enum.split_with(wave_effects, &lifecycle_effect?/1) + + # Single wave: these failures are dropped, never re-entered. + _wave_failures = execute_effects(wave_executable, executor, context) + + lifecycle = lifecycle ++ wave_lifecycle + flow = if budget_exhausted?(wave_lifecycle), do: :halt, else: :cont + {flow, {machine_state, lifecycle}} + + {:error, :not_running} -> + {:halt, acc} + end + end + + # The origin each re-entry carries, mirroring the shapes upstream's + # session builds on its own failed-communication paths - this package + # invents no origin vocabulary (every arm below is a + # `t:Statifier.Event.Cause.origin/0` constructor): + # + # - `:invoke` -> `{:invoke, state_index, invoke_index}` with no opts, + # exactly as `Statifier.Session.invoke_error/4` builds it + # (`deps/statifier/lib/statifier/session.ex`, the st-ADR-0039 decision 4 + # write). + # - `:send`/`:send_delayed` -> `{:content, c_index, owner}` with the + # failing send's `sendid`, exactly as `Statifier.Session.origin_of/1` + # and `communication_error/4` build it. + # - `:cancel` -> the same `{:content, c_index, owner}` arm (the `` + # element's own content node carries both fields for exactly this + # identity), with no `sendid` - there is no failing `` here to + # name one. + # - `:cancel_invoke`/`:autoforward` -> `{:state, state_index}`, the + # platform-raised-with-no-content-node arm, since neither payload + # carries an `invoke_index` to name the `{:invoke, _, _}` arm with. + # + # Everything else is observational (`:log`, `:datamodel_change`, + # `:datamodel_init`, `:trace`) and its failure is discarded. + @spec reentry_origin(Statifier.Effect.t()) :: + {Statifier.Event.Cause.origin(), keyword()} | :observational + defp reentry_origin({:invoke, %Invoke{state_index: state_index, invoke_index: invoke_index}}), + do: {{:invoke, state_index, invoke_index}, []} + + defp reentry_origin({:send, %Send{c_index: c_index, owner: owner, send_id: send_id}}), + do: {{:content, c_index, owner}, [sendid: send_id]} + + defp reentry_origin({:send_delayed, %SendDelayed{} = payload}), + do: {{:content, payload.c_index, payload.owner}, [sendid: payload.send_id]} + + defp reentry_origin({:cancel, %Cancel{c_index: c_index, owner: owner}}), + do: {{:content, c_index, owner}, []} + + defp reentry_origin({:cancel_invoke, %CancelInvoke{state_index: state_index}}), + do: {{:state, state_index}, []} + + defp reentry_origin({:autoforward, %Autoforward{state_index: state_index}}), + do: {{:state, state_index}, []} + + defp reentry_origin(_observational_effect), do: :observational + + @spec lifecycle_effect?(Statifier.Effect.t()) :: boolean() + defp lifecycle_effect?({:done, _payload}), do: true + defp lifecycle_effect?({:budget_exhausted, _payload}), do: true + defp lifecycle_effect?(_effect), do: false + + @spec execute_effects([Statifier.Effect.t()], Executor.t(), Executor.context()) :: + [{Statifier.Effect.t(), term()}] + defp execute_effects(effects, executor, context) do + effects + |> Enum.reduce([], fn effect, failures -> + case Executor.run(executor, effect, context) do + :ok -> failures + {:error, reason} -> [{effect, reason} | failures] + end + end) + |> Enum.reverse() + end + + # `:done` is the only path to `:completed` (ADR-0004 decision 6); + # `:budget_exhausted` - from the primary pass or from a re-entry wave - + # is the only chart-driven path to `:failed`. + @spec run_status(MachineState.t(), [Statifier.Effect.t()]) :: Adapter.run_status() + defp run_status(machine_state, lifecycle_effects) do + cond do + budget_exhausted?(lifecycle_effects) -> :failed + machine_state.status == :done -> :completed + true -> :active + end + end + + @spec budget_exhausted?([Statifier.Effect.t()]) :: boolean() + defp budget_exhausted?(lifecycle_effects), + do: Enum.any?(lifecycle_effects, &match?({:budget_exhausted, _payload}, &1)) + + @spec budget_effect([Statifier.Effect.t()]) :: BudgetExhausted.t() | nil + defp budget_effect(lifecycle_effects) do + Enum.find_value(lifecycle_effects, fn + {:budget_exhausted, %BudgetExhausted{} = payload} -> payload + _effect -> nil + end) + end + + # A non-quiescent state without `:budget_exhausted` cannot come out of + # `initialize/2` or `handle_event/2` - both fold to quiescence - so a + # false here is a bug in this loop, not a caller error, and it raises + # deliberately rather than persisting a position the resume recipe cannot + # honor. + @spec assert_quiescent(MachineState.t(), [Statifier.Effect.t()]) :: :ok + defp assert_quiescent(machine_state, lifecycle_effects) do + if budget_exhausted?(lifecycle_effects) or MachineState.internal_queue_empty?(machine_state) do + :ok + else + raise "loop bug: non-quiescent MachineState reached the persist tail " <> + "without :budget_exhausted - upstream's quiescence fold makes this unreachable" + end + end + + # A budget-exhausted state is not quiescent, so its position is never + # persisted: `:skip` stores `nil` on insert and carries the stored blob + # forward on update (ADR-0004 decision 1). The failure string is short + # and prefixed - a psql-console reason, not an inspect dump. + @spec write_run( + :insert | :update, + Storage.t(), + run_id(), + MachineState.t(), + Adapter.run_status(), + [Statifier.Effect.t()] + ) :: :ok | {:error, error()} + defp write_run(write, store, run_id, machine_state, status, lifecycle_effects) do + {position, failure} = + case budget_effect(lifecycle_effects) do + nil -> {:persist, nil} + %BudgetExhausted{budget: budget} -> {:skip, "budget_exhausted: #{budget} rounds"} + end + + opts = [position: position, failure: failure] + + case write do + :insert -> Storage.insert_run(store, run_id, machine_state, status, opts) + :update -> Storage.update_run(store, run_id, machine_state, status, opts) + end + end + + @spec run(run_id(), Adapter.run_status(), Identity.t()) :: Run.t() + defp run(run_id, status, identity) do + %Run{ + run_id: run_id, + status: status, + content_hash: identity.content_hash, + failure: nil + } + end +end diff --git a/lib/statifier_persistence/serialization.ex b/lib/statifier_persistence/serialization.ex new file mode 100644 index 0000000..fe685fa --- /dev/null +++ b/lib/statifier_persistence/serialization.ex @@ -0,0 +1,32 @@ +defmodule StatifierPersistence.Serialization do + @moduledoc """ + The per-run serialization strategy behaviour (ADR-0004 decision 5): + the seam through which concurrent deliveries to one run are ordered. + + `StatifierPersistence.Runs` runs its whole fetch-to-persist tail inside + `c:with_run/3`, so the ordering guarantee lives in the strategy, not in + the loop. Strategy selection is a `StatifierPersistence.Runs` option on + `create/4`, `step/5`, and `fail/4` - `serialization: {module, config}` - + defaulting to `{StatifierPersistence.Serialization.AdapterLock, store}`, + which delegates to the storage adapter's optional + `c:StatifierPersistence.Storage.Adapter.lock_run/3`. A host that orders + deliveries some other way (a single job-queue consumer per run id, for + one) swaps the strategy without touching the loop: the guarantee moves, + the API does not. + """ + + @doc """ + Runs `fun` under this strategy's per-run exclusion for `run_id`, + returning `{:ok, fun.()}` or the strategy's own refusal. + + The guarantee a strategy must provide: for one `run_id`, two `with_run/3` + bodies never overlap, and completed bodies are observed in execution + order - a body that finishes before another starts is durable before the + later body loads. It explicitly does NOT promise cross-run ordering or + fairness: bodies for different run ids may interleave freely, and a + contended run id may serve waiters in any order. + """ + @callback with_run(config :: term(), run_id :: String.t(), fun :: (-> result)) :: + {:ok, result} | {:error, term()} + when result: var +end diff --git a/lib/statifier_persistence/serialization/adapter_lock.ex b/lib/statifier_persistence/serialization/adapter_lock.ex new file mode 100644 index 0000000..a40339a --- /dev/null +++ b/lib/statifier_persistence/serialization/adapter_lock.ex @@ -0,0 +1,30 @@ +defmodule StatifierPersistence.Serialization.AdapterLock do + @moduledoc """ + The default `StatifierPersistence.Serialization` strategy (ADR-0004 + decision 5): per-run ordering as the storage adapter's own lock. + + Its `config` is the `StatifierPersistence.Storage` handle itself. + `with_run/3` delegates to the adapter's optional + `c:StatifierPersistence.Storage.Adapter.lock_run/3` when the adapter + exports it, and refuses with `{:error, {:serialization, :not_supported}}` + when it does not - an adapter without a lock provides no ordering, and + a silent fallback here would be the loop pretending otherwise. + """ + + @behaviour StatifierPersistence.Serialization + + alias StatifierPersistence.Storage + + @impl StatifierPersistence.Serialization + @spec with_run(config :: term(), run_id :: String.t(), fun :: (-> result)) :: + {:ok, result} | {:error, term()} + when result: term() + def with_run(%Storage{} = store, run_id, fun) do + if Code.ensure_loaded?(store.adapter) and + function_exported?(store.adapter, :lock_run, 3) do + store.adapter.lock_run(store.opts, run_id, fun) + else + {:error, {:serialization, :not_supported}} + end + end +end diff --git a/lib/statifier_persistence/storage.ex b/lib/statifier_persistence/storage.ex index c822c23..872bfad 100644 --- a/lib/statifier_persistence/storage.ex +++ b/lib/statifier_persistence/storage.ex @@ -3,19 +3,21 @@ defmodule StatifierPersistence.Storage do The guarded entry point from a storage adapter to a `Statifier.MachineState.t()`. - Every load runs through `load_position/3`, and every load is checked - against the exact chart revision that produced the stored position - (ADR-0003 decision 2). No adapter callback ever holds both the stored - identity and a caller-supplied `Statifier.Machine.t()` at the same time - (`StatifierPersistence.Storage.Adapter`'s moduledoc), so the guard cannot - be skipped or weakened per adapter - it lives here, above every one of - them, and nowhere else in this package decodes a position blob. - - `save_chart/3` and `save_position/3` derive a chart's `content_hash` and - `identity_blob` from `Machine.identity/1` on the machine they are given - - never from a caller-supplied value - and refuse an unidentified machine - with `{:error, :unidentified_chart}` rather than writing a row a later - load has no way to check. + Every load runs through `load_position/3` or `load_run_position/3`, and + every load is checked against the exact chart revision that produced the + stored position (ADR-0003 decision 2). No adapter callback ever holds + both the stored identity and a caller-supplied `Statifier.Machine.t()` at + the same time (`StatifierPersistence.Storage.Adapter`'s moduledoc), so + the guard cannot be skipped or weakened per adapter - it lives here, + above every one of them, and nowhere else in this package decodes a + position blob. + + Every writer taking a machine or machine state - `save_chart/3`, + `save_position/3`, `insert_run/5`, `update_run/5` - derives a chart's + `content_hash` and `identity_blob` from `Machine.identity/1` on the + machine it is given - never from a caller-supplied value - and refuses an + unidentified machine with `{:error, :unidentified_chart}` rather than + writing a row a later load has no way to check. Every function here returns an error tuple instead of throwing; nothing in this module ever downgrades a failure to a default value. @@ -39,9 +41,22 @@ defmodule StatifierPersistence.Storage do Adapter.error() | :not_a_statifier_blob | :unidentified_chart + | :run_position_missing | {:unsupported_format_version, term()} | {:identity_mismatch, Identity.t(), Identity.t() | nil} + @typedoc """ + Options the run writers (`insert_run/5`, `update_run/5`) accept: + + - `failure:` - the short reason stored on a `:failed` run; defaults to + `nil`. + - `position:` - `:persist` (the default) encodes the given machine state + with `Position.to_binary/1` and stores it as the run's `position_blob`; + `:skip` stores `nil` on insert and carries the currently stored blob + forward verbatim on update. + """ + @type run_write_opt :: {:failure, String.t() | nil} | {:position, :persist | :skip} + @doc """ Initializes `adapter` with `opts` and returns the handle every other function in this module takes as its first argument. @@ -176,6 +191,190 @@ defmodule StatifierPersistence.Storage do end end + @doc """ + Inserts a run record for `run_id`, keyed by the content hash and identity + envelope of `machine_state.machine`'s own `Machine.identity/1` - never a + caller-supplied hash - and refusing an unidentified machine with + `{:error, :unidentified_chart}` before calling the adapter. + + Writers always take a `MachineState`: even a run failed at creation has + one, because `Statifier.Interpreter.initialize/2` cannot fail (ADR-0004 + decision 1). Under `position: :persist` (the default) the state is + encoded with `Position.to_binary/1` and stored as the run's + `position_blob`; under `position: :skip` the blob is stored `nil` - the + arm for a run with no quiescent position to store. Uniqueness comes from + the adapter's `insert_run/2` `:run_exists` refusal, not a pre-check here. + """ + @spec insert_run( + store :: t(), + run_id :: Adapter.run_id(), + machine_state :: MachineState.t(), + status :: Adapter.run_status(), + opts :: [run_write_opt()] + ) :: :ok | {:error, error()} + def insert_run( + %__MODULE__{} = store, + run_id, + %MachineState{} = machine_state, + status, + opts \\ [] + ) do + case Machine.identity(machine_state.machine) do + nil -> + {:error, :unidentified_chart} + + identity -> + with {:ok, position_blob} <- insert_position_blob(machine_state, position_opt(opts)) do + run_record = run_record(run_id, status, identity, position_blob, opts) + store.adapter.insert_run(store.opts, run_record) + end + end + end + + @doc """ + Overwrites the run stored under `run_id` with a full record derived the + same way `insert_run/5` derives one: identity always from + `machine_state.machine`'s own `Machine.identity/1`, refusal of an + unidentified machine, `position_blob` encoded under `position: :persist` + (the default). + + Under `position: :skip` the currently stored `position_blob` is carried + forward verbatim: because the adapter's `update_run/2` is a full-record + overwrite, this function fetches the current record and reuses its blob + bytes unchanged, so a status-only update (a failed step, an abandonment) + never touches the stored position. Returns `{:error, :run_not_found}` + when no run exists for the id. + """ + @spec update_run( + store :: t(), + run_id :: Adapter.run_id(), + machine_state :: MachineState.t(), + status :: Adapter.run_status(), + opts :: [run_write_opt()] + ) :: :ok | {:error, error()} + def update_run( + %__MODULE__{} = store, + run_id, + %MachineState{} = machine_state, + status, + opts \\ [] + ) do + case Machine.identity(machine_state.machine) do + nil -> + {:error, :unidentified_chart} + + identity -> + with {:ok, position_blob} <- + update_position_blob(store, run_id, machine_state, position_opt(opts)) do + run_record = run_record(run_id, status, identity, position_blob, opts) + store.adapter.update_run(store.opts, run_record) + end + end + end + + @doc """ + Overwrites only the status and failure of the run stored under `run_id`, + carrying every other stored field - both blobs included - forward + verbatim. + + This is the writer for a host-driven terminal transition that has no + `MachineState` in hand (`StatifierPersistence.Runs.fail/4`, ADR-0004 + decision 6): nothing is derived, decoded, or re-encoded, so the identity + guard is preserved by construction - the stored `identity_blob` and + `position_blob` bytes never change. `opts` accepts `failure:` only + (default `nil`). Returns `{:error, :run_not_found}` when no run exists + for the id. + """ + @spec update_run_status( + store :: t(), + run_id :: Adapter.run_id(), + status :: Adapter.run_status(), + opts :: [run_write_opt()] + ) :: :ok | {:error, error()} + def update_run_status(%__MODULE__{} = store, run_id, status, opts \\ []) do + with {:ok, run_record} <- store.adapter.fetch_run(store.opts, run_id) do + updated = %{run_record | status: status, failure: Keyword.get(opts, :failure)} + store.adapter.update_run(store.opts, updated) + end + end + + @doc """ + Fetches the run record stored under `run_id`. + """ + @spec fetch_run(store :: t(), run_id :: Adapter.run_id()) :: + {:ok, Adapter.run_record()} | {:error, error()} + def fetch_run(%__MODULE__{} = store, run_id) do + store.adapter.fetch_run(store.opts, run_id) + end + + @doc """ + Fetches the run stored under `run_id` and rebuilds its position into a + `Statifier.MachineState.t()` walking `machine`, refusing a chart-revision + mismatch instead of silently resuming the wrong configuration. + + Runs in `load_position/3`'s order, with one extra arm: the same cheap + identity pre-check against the stored `identity_blob`, then + `{:error, :run_position_missing}` for a run whose `position_blob` is + `nil` (a run that failed at creation stores none - ADR-0004 decision 1), + then `Position.from_binary/2` as the authoritative check, its result + returned unchanged. + + Like `load_position/3`, the returned state carries `nil` for both + `routes` and `invoke_types` (st-ADR-0064); re-stamping them before the + next drive is the stepper's job, not this function's. + """ + @spec load_run_position( + store :: t(), + run_id :: Adapter.run_id(), + machine :: Machine.t() + ) :: {:ok, MachineState.t()} | {:error, error()} + def load_run_position(%__MODULE__{} = store, run_id, %Machine{} = machine) do + with {:ok, run_record} <- store.adapter.fetch_run(store.opts, run_id), + :ok <- precheck_identity(run_record.identity_blob, machine) do + case run_record.position_blob do + nil -> {:error, :run_position_missing} + position_blob -> Position.from_binary(position_blob, machine) + end + end + end + + @spec run_record( + Adapter.run_id(), + Adapter.run_status(), + Identity.t(), + binary() | nil, + [run_write_opt()] + ) :: Adapter.run_record() + defp run_record(run_id, status, identity, position_blob, opts) do + %{ + run_id: run_id, + status: status, + content_hash: identity.content_hash, + identity_blob: Identity.to_binary(identity), + position_blob: position_blob, + failure: Keyword.get(opts, :failure) + } + end + + @spec position_opt([run_write_opt()]) :: :persist | :skip + defp position_opt(opts), do: Keyword.get(opts, :position, :persist) + + @spec insert_position_blob(MachineState.t(), :persist | :skip) :: + {:ok, binary() | nil} | {:error, error()} + defp insert_position_blob(_machine_state, :skip), do: {:ok, nil} + defp insert_position_blob(machine_state, :persist), do: Position.to_binary(machine_state) + + @spec update_position_blob(t(), Adapter.run_id(), MachineState.t(), :persist | :skip) :: + {:ok, binary() | nil} | {:error, error()} + defp update_position_blob(_store, _run_id, machine_state, :persist), + do: Position.to_binary(machine_state) + + defp update_position_blob(store, run_id, _machine_state, :skip) do + with {:ok, run_record} <- store.adapter.fetch_run(store.opts, run_id) do + {:ok, run_record.position_blob} + end + end + @spec precheck_identity(identity_blob :: binary(), machine :: Machine.t()) :: :ok | {:error, :not_a_statifier_blob} diff --git a/lib/statifier_persistence/storage/adapter.ex b/lib/statifier_persistence/storage/adapter.ex index 8e4adc1..effe4b1 100644 --- a/lib/statifier_persistence/storage/adapter.ex +++ b/lib/statifier_persistence/storage/adapter.ex @@ -12,10 +12,11 @@ defmodule StatifierPersistence.Storage.Adapter do A chart is keyed by its content hash (`Statifier.Machine.Identity.content_hash`, verbatim); a position is keyed - by the engine session id (st-ADR-0008's `sess_` UXID), also verbatim. Both - are opaque strings to this layer: no callback here accepts or returns a - surrogate key, a table name, or a prefix (ADR-0002 decision 1, ADR-0003 - decision 3). + by the engine session id (st-ADR-0008's `sess_` UXID), also verbatim; a + run is keyed by a caller-supplied opaque `run_id`, also verbatim + (ADR-0004 decision 2). All are opaque strings to this layer: no callback + here accepts or returns a surrogate key, a table name, or a prefix + (ADR-0002 decision 1, ADR-0003 decision 3). """ @typedoc """ @@ -31,6 +32,36 @@ defmodule StatifierPersistence.Storage.Adapter do @typedoc "An engine session id (st-ADR-0008), verbatim." @type session_id :: String.t() + @typedoc """ + A run's key: a caller-supplied opaque string, stored verbatim (ADR-0004 + decision 2). A host identity in ADR-0002 decision 1's category - never a + surrogate this layer generates. + """ + @type run_id :: String.t() + + @typedoc """ + A run's lifecycle status (ADR-0004 decision 2). `:completed` and `:failed` + are terminal. No callback here validates a transition between them - the + lifecycle above the facade owns that. + """ + @type run_status :: :active | :completed | :failed + + @typedoc """ + A stored run (ADR-0004 decision 1): its caller-supplied key, its status, + the content hash and identity envelope of the chart it runs, the opaque + `position_blob` holding its current position - nullable, because a run + that fails at creation has no quiescent position to store - and a short + `failure` reason for a `:failed` run, `nil` otherwise. + """ + @type run_record :: %{ + run_id: run_id(), + status: run_status(), + content_hash: content_hash(), + identity_blob: binary(), + position_blob: binary() | nil, + failure: String.t() | nil + } + @typedoc """ A stored chart: its content hash, its identity envelope (`Statifier.Machine.Identity.to_binary/1`), and an opaque `chart_blob` @@ -57,14 +88,17 @@ defmodule StatifierPersistence.Storage.Adapter do } @typedoc """ - This layer's own refusal arms. `:chart_not_found` and `:position_not_found` - are the not-found arms every adapter must return instead of `nil` or a - raise; `{:adapter, term()}` carries a backend failure (a database down, a - timeout) that is not this layer's to interpret further. + This layer's own refusal arms. `:chart_not_found`, `:position_not_found`, + and `:run_not_found` are the not-found arms every adapter must return + instead of `nil` or a raise; `:run_exists` is `insert_run/2`'s refusal of + a duplicate `run_id`; `{:adapter, term()}` carries a backend failure (a + database down, a timeout) that is not this layer's to interpret further. """ @type error :: :chart_not_found | :position_not_found + | :run_exists + | :run_not_found | {:adapter, term()} @doc """ @@ -126,6 +160,54 @@ defmodule StatifierPersistence.Storage.Adapter do @callback fetch_position(opts(), session_id()) :: {:ok, StatifierPersistence.Storage.Adapter.position_record()} | {:error, error()} + @doc """ + Inserts `run_record`, refusing a duplicate `run_id` with + `{:error, :run_exists}`. + + The refusal must be atomic with the write: no interleaving of two + `insert_run/2` calls for the same `run_id` may let both return `:ok`. + Create-exactly-once rests on this callback alone, without a lock, so a + check-then-insert implemented as two separate operations does not satisfy + the contract - a SQL adapter reaches for a unique index (or equivalent + backend-native uniqueness) and maps its violation to + `{:error, :run_exists}`. + + This callback does not decode `position_blob` or `identity_blob`, does + not validate the status, and performs no identity check - the facade and + the lifecycle own those (ADR-0003 decisions 1 and 2, ADR-0004 + decision 1). + """ + @callback insert_run(opts(), StatifierPersistence.Storage.Adapter.run_record()) :: + :ok | {:error, error()} + + @doc """ + Fetches the run stored under `run_id`. + + Returns `{:error, :run_not_found}` when no run is stored under that id - + never `{:ok, nil}` and never a raise. The returned `identity_blob` and + `position_blob` must be byte-identical to what was stored (a stored `nil` + `position_blob` comes back as `nil`); an adapter must not normalize, + truncate, or re-encode them, and it does not decode them either (ADR-0003 + decision 1). + """ + @callback fetch_run(opts(), run_id()) :: + {:ok, StatifierPersistence.Storage.Adapter.run_record()} | {:error, error()} + + @doc """ + Overwrites the run stored under `run_record`'s `run_id` with the full + record. + + Returns `{:error, :run_not_found}` when no run exists for the id. This is + a full-record overwrite - there is no partial-update surface, so every + field in the stored row after this call is the given record's, including + a `nil` `position_blob`. Like the other run callbacks it decodes nothing, + validates no status transition, and performs no identity check - the + facade and the lifecycle own those (ADR-0003 decisions 1 and 2, ADR-0004 + decision 1). + """ + @callback update_run(opts(), StatifierPersistence.Storage.Adapter.run_record()) :: + :ok | {:error, error()} + @doc """ Optional per-test isolation hook (ADR-0003 amendment, 2026-08-21). @@ -141,5 +223,28 @@ defmodule StatifierPersistence.Storage.Adapter do """ @callback isolate(opts()) :: :ok | {:error, error()} - @optional_callbacks isolate: 1 + @doc """ + Optional per-run lock (ADR-0003 amendment, 2026-08-22; ADR-0004 + decision 5). + + Provides mutual exclusion per `run_id`: while one `lock_run/3` call for a + given `run_id` is running `fun`, no other `lock_run/3` call for the same + `run_id` may run its own. `fun` runs while the exclusion is held, and the + exclusion is released on ANY exit from `fun` - a normal return, a throw, + and a raise escaping `fun` alike. The lock must not leak: a raising `fun` + propagates to the caller, but the next `lock_run/3` for that `run_id` + must still acquire. + + This is the callback the default serialization strategy + (`StatifierPersistence.Serialization.AdapterLock`) delegates to; an + adapter that does not export it makes that strategy refuse with + `{:error, {:serialization, :not_supported}}`. An Ecto adapter implements + it as a transaction-scoped row lock - `SELECT ... FOR UPDATE` on the run + row inside a transaction that spans `fun` (sp-4an.3). + """ + @callback lock_run(opts(), run_id(), (-> result)) :: + {:ok, result} | {:error, error()} + when result: var + + @optional_callbacks isolate: 1, lock_run: 3 end diff --git a/lib/statifier_persistence/storage/in_memory.ex b/lib/statifier_persistence/storage/in_memory.ex index b569282..0e434c1 100644 --- a/lib/statifier_persistence/storage/in_memory.ex +++ b/lib/statifier_persistence/storage/in_memory.ex @@ -1,7 +1,8 @@ defmodule StatifierPersistence.Storage.InMemory do @moduledoc """ - The reference `StatifierPersistence.Storage.Adapter`: an Agent holding two - maps, charts keyed by content hash and positions keyed by session id. + The reference `StatifierPersistence.Storage.Adapter`: an Agent holding + three maps - charts keyed by content hash, positions keyed by session id, + and runs keyed by run id. It ships in `lib/`, not the test-only `support/` directory, for two reasons: the conformance template this package ships in `lib/` (this @@ -14,12 +15,20 @@ defmodule StatifierPersistence.Storage.InMemory do alias StatifierPersistence.Storage.Adapter - @typedoc "This adapter's state: the two maps `init/1` starts the Agent with." + @typedoc """ + This adapter's state: the three record maps `init/1` starts the Agent + with, plus the per-run lock table `lock_run/3` acquires through. + """ @type state :: %{ charts: %{Adapter.content_hash() => Adapter.chart_record()}, - positions: %{Adapter.session_id() => Adapter.position_record()} + positions: %{Adapter.session_id() => Adapter.position_record()}, + runs: %{Adapter.run_id() => Adapter.run_record()}, + locks: %{Adapter.run_id() => reference()} } + # How long a contended lock_run/3 sleeps between acquisition attempts. + @lock_spin_sleep_ms 5 + @doc """ Starts the backing Agent and returns `opts` with `:pid` merged in - the handle every other callback expects as its first argument. @@ -27,7 +36,7 @@ defmodule StatifierPersistence.Storage.InMemory do @impl Adapter @spec init(Adapter.opts()) :: {:ok, Adapter.opts()} | {:error, Adapter.error()} def init(opts) do - case Agent.start_link(fn -> %{charts: %{}, positions: %{}} end) do + case Agent.start_link(fn -> %{charts: %{}, positions: %{}, runs: %{}, locks: %{}} end) do {:ok, pid} -> {:ok, Keyword.put(opts, :pid, pid)} {:error, reason} -> {:error, {:adapter, reason}} end @@ -84,6 +93,116 @@ defmodule StatifierPersistence.Storage.InMemory do end end + @doc """ + Inserts `run_record` under its `run_id`, refusing a duplicate with + `{:error, :run_exists}`. + + The exists-check and the write run inside one `Agent.get_and_update/2` + call, so they are a single atomic state transition: two concurrent + inserts of the same `run_id` cannot both return `:ok`. + """ + @impl Adapter + @spec insert_run(Adapter.opts(), Adapter.run_record()) :: :ok | {:error, Adapter.error()} + def insert_run(opts, %{run_id: run_id} = run_record) do + Agent.get_and_update(pid(opts), fn state -> + if Map.has_key?(state.runs, run_id) do + {{:error, :run_exists}, state} + else + {:ok, put_in(state, [:runs, run_id], run_record)} + end + end) + end + + @doc """ + Fetches the run stored under `run_id`, or `:run_not_found`. + """ + @impl Adapter + @spec fetch_run(Adapter.opts(), Adapter.run_id()) :: + {:ok, Adapter.run_record()} | {:error, Adapter.error()} + def fetch_run(opts, run_id) do + case Agent.get(pid(opts), &get_in(&1, [:runs, run_id])) do + nil -> {:error, :run_not_found} + run_record -> {:ok, run_record} + end + end + + @doc """ + Overwrites the run stored under `run_record`'s `run_id` with the full + record, or refuses with `:run_not_found` when no run exists for the id. + """ + @impl Adapter + @spec update_run(Adapter.opts(), Adapter.run_record()) :: :ok | {:error, Adapter.error()} + def update_run(opts, %{run_id: run_id} = run_record) do + Agent.get_and_update(pid(opts), fn state -> + if Map.has_key?(state.runs, run_id) do + {:ok, put_in(state, [:runs, run_id], run_record)} + else + {{:error, :run_not_found}, state} + end + end) + end + + @doc """ + Runs `fun` under this adapter's per-run mutual exclusion for `run_id` + (the optional `c:StatifierPersistence.Storage.Adapter.lock_run/3`). + + Acquisition is an insert-if-absent on the Agent's lock table, one atomic + `Agent.get_and_update/2` transition; contention spins with a small + bounded sleep (#{@lock_spin_sleep_ms}ms) between attempts. The lock is + released in an `after` block, so any exit from `fun` - a raise included - + releases it; the raise itself propagates to the caller. + + Simple and honest for a reference adapter. A production adapter should + prefer its backend's native lock - the Ecto adapter implements this as a + transaction-scoped row lock (sp-4an.3). + """ + @impl Adapter + @spec lock_run(Adapter.opts(), Adapter.run_id(), (-> result)) :: + {:ok, result} | {:error, Adapter.error()} + when result: term() + def lock_run(opts, run_id, fun) do + token = acquire_lock(pid(opts), run_id) + + try do + {:ok, fun.()} + after + release_lock(pid(opts), run_id, token) + end + end + + @spec acquire_lock(pid(), Adapter.run_id()) :: reference() + defp acquire_lock(pid, run_id) do + token = make_ref() + + acquired? = + Agent.get_and_update(pid, fn state -> + if Map.has_key?(state.locks, run_id) do + {false, state} + else + {true, put_in(state, [:locks, run_id], token)} + end + end) + + if acquired? do + token + else + Process.sleep(@lock_spin_sleep_ms) + acquire_lock(pid, run_id) + end + end + + @spec release_lock(pid(), Adapter.run_id(), reference()) :: :ok + defp release_lock(pid, run_id, token) do + Agent.update(pid, fn state -> + case state.locks do + # Only the holder's own token releases: a stray release can never + # drop a lock some later acquirer holds. + %{^run_id => ^token} -> %{state | locks: Map.delete(state.locks, run_id)} + _other -> state + end + end) + end + @spec pid(Adapter.opts()) :: pid() defp pid(opts), do: Keyword.fetch!(opts, :pid) diff --git a/lib/statifier_persistence/testing/storage_conformance.ex b/lib/statifier_persistence/testing/storage_conformance.ex index d97e2bf..399dfd2 100644 --- a/lib/statifier_persistence/testing/storage_conformance.ex +++ b/lib/statifier_persistence/testing/storage_conformance.ex @@ -27,6 +27,13 @@ defmodule StatifierPersistence.Testing.StorageConformance do no such callback, like `StatifierPersistence.Storage.InMemory`, is unaffected: the check is a `function_exported?/3` guard, not a requirement. + + The optional `c:StatifierPersistence.Storage.Adapter.lock_run/3` gets the + same treatment at generation time: when the adapter under test exports + it, the suite generates the per-run lock tests (mutual exclusion of two + concurrent bodies, release after a raising fun); when it does not, they + are not generated at all - exporting the callback is what opts an + adapter into its contract. """ use ExUnit.CaseTemplate @@ -211,6 +218,201 @@ defmodule StatifierPersistence.Testing.StorageConformance do assert fetched.position_blob == position_blob end + # -- Adapter level: run records ------------------------------------ + + # sabotage: in the adapter under test's insert_run/2, store under a + # fixed key instead of run_id -> red, fetch_run/2 below returned + # {:error, :run_not_found} instead of the round-tripped record. + # Verified red, reverted. + test "adapter: round-trips an inserted run byte-identically", %{store: store} do + run_record = %{ + run_id: "run-conformance-a", + status: :active, + content_hash: "sha256:conformance-chart-a", + identity_blob: <<9, 0, 8, 255, 7>>, + position_blob: <<0, 255, 1, 2, 3, 0, 0, 254>>, + failure: nil + } + + assert :ok = @conformance_adapter.insert_run(store.opts, run_record) + + assert {:ok, ^run_record} = + @conformance_adapter.fetch_run(store.opts, "run-conformance-a") + end + + # sabotage: in the adapter under test's insert_run/2, drop the + # exists-check and always write with :ok -> red, the second insert + # below returned :ok instead of {:error, :run_exists}. Verified red + # (together with InMemoryTest's concurrent-insert test under this one + # mutation), reverted. + test "adapter: insert_run/2 refuses a duplicate run_id with :run_exists", %{store: store} do + run_record = %{ + run_id: "run-conformance-duplicate", + status: :active, + content_hash: "sha256:conformance-chart-a", + identity_blob: <<1, 2, 3>>, + position_blob: <<7, 8, 9>>, + failure: nil + } + + assert :ok = @conformance_adapter.insert_run(store.opts, run_record) + + assert {:error, :run_exists} = + @conformance_adapter.insert_run(store.opts, %{run_record | status: :failed}) + + assert {:ok, ^run_record} = + @conformance_adapter.fetch_run(store.opts, "run-conformance-duplicate") + end + + # sabotage: in the adapter under test's update_run/2, upsert on a + # missing run_id (write and return :ok) instead of refusing -> red, + # the update below returned :ok instead of {:error, :run_not_found}. + # Verified red, reverted. + test "adapter: update_run/2 reports :run_not_found for an unknown run_id", %{store: store} do + run_record = %{ + run_id: "run-conformance-update-missing", + status: :failed, + content_hash: "sha256:conformance-chart-a", + identity_blob: <<1, 2, 3>>, + position_blob: nil, + failure: "abandoned" + } + + assert {:error, :run_not_found} = + @conformance_adapter.update_run(store.opts, run_record) + + assert {:error, :run_not_found} = + @conformance_adapter.fetch_run(store.opts, "run-conformance-update-missing") + end + + # sabotage: in the adapter under test's fetch_run/2, return + # {:ok, a_placeholder_record} instead of :run_not_found for an + # unknown id -> red, this test's pattern match on + # {:error, :run_not_found} saw the placeholder. Verified red, + # reverted. + test "adapter: fetch_run/2 reports :run_not_found for an unknown run_id", %{store: store} do + assert {:error, :run_not_found} = + @conformance_adapter.fetch_run(store.opts, "run-conformance-missing") + end + + # sabotage: in the adapter under test's insert_run/2, normalize a nil + # position_blob to <<>> before storing -> red, the equality assertion + # on nil below saw "" instead. This is the arm ADR-0004 decision 1 + # makes nullable; an adapter must not paper over it. Verified red, + # reverted. + test "adapter: a nil position_blob round-trips as nil", %{store: store} do + run_record = %{ + run_id: "run-conformance-nil-blob", + status: :failed, + content_hash: "sha256:conformance-chart-a", + identity_blob: <<1, 2, 3>>, + position_blob: nil, + failure: "budget_exhausted: 100 rounds" + } + + assert :ok = @conformance_adapter.insert_run(store.opts, run_record) + + assert {:ok, fetched} = + @conformance_adapter.fetch_run(store.opts, "run-conformance-nil-blob") + + assert fetched.position_blob == nil + end + + # sabotage: in the adapter under test's update_run/2, keep the stored + # record's status and failure instead of overwriting them (a partial + # update) -> red, the fetch below returned the inserted :active/nil + # pair instead of the updated :failed/reason pair. Verified red, + # reverted. + test "adapter: update_run/2 overwrites the full record, status and failure verbatim", %{ + store: store + } do + inserted = %{ + run_id: "run-conformance-overwrite", + status: :active, + content_hash: "sha256:conformance-chart-a", + identity_blob: <<1, 2, 3>>, + position_blob: <<7, 8, 9>>, + failure: nil + } + + updated = %{ + inserted + | status: :failed, + position_blob: <<10, 11, 12>>, + failure: "abandoned: operator request" + } + + assert :ok = @conformance_adapter.insert_run(store.opts, inserted) + assert :ok = @conformance_adapter.update_run(store.opts, updated) + + assert {:ok, ^updated} = + @conformance_adapter.fetch_run(store.opts, "run-conformance-overwrite") + end + + # -- Adapter level: the optional per-run lock ---------------------- + # + # Generated only when the adapter under test exports the optional + # lock_run/3 (ADR-0003 amendment 2026-08-22, ADR-0004 decision 5) - + # the same shape as the isolate/1 hook: exporting the callback is + # what opts an adapter into its contract. + + if Code.ensure_loaded?(conformance_adapter) and + function_exported?(conformance_adapter, :lock_run, 3) do + # sabotage: in the adapter under test's lock_run/3, run fun without + # the exclusion ({:ok, fun.()} with no acquire) -> red, the two + # sleeping bodies below interleave and the enter/enter prefix + # breaks the paired pattern. Verified red (together with + # RunsTest's concurrent-step test under this one mutation), + # reverted. + test "adapter: lock_run/3 never overlaps two bodies for one run_id", %{store: store} do + {:ok, events} = Agent.start_link(fn -> [] end) + + body = fn tag -> + fn -> + Agent.update(events, &[{:enter, tag} | &1]) + Process.sleep(30) + Agent.update(events, &[{:exit, tag} | &1]) + tag + end + end + + tasks = + for tag <- [:first, :second] do + Task.async(fn -> + @conformance_adapter.lock_run(store.opts, "run-conformance-lock", body.(tag)) + end) + end + + assert [{:ok, _tag_a}, {:ok, _tag_b}] = Task.await_many(tasks, 5_000) + + recorded = events |> Agent.get(& &1) |> Enum.reverse() + assert [{:enter, one}, {:exit, one}, {:enter, other}, {:exit, other}] = recorded + assert one != other + end + + # sabotage: in the adapter under test's lock_run/3, release only on + # a normal return (move the release out of the after block, after + # {:ok, fun.()}) -> red, the raise leaks the lock and the + # reacquisition below times out (Task.yield returns nil). Verified + # red, reverted. + test "adapter: lock_run/3 releases the lock after a raising fun", %{store: store} do + assert_raise RuntimeError, "lock body boom", fn -> + @conformance_adapter.lock_run(store.opts, "run-conformance-lock-raise", fn -> + raise "lock body boom" + end) + end + + task = + Task.async(fn -> + @conformance_adapter.lock_run(store.opts, "run-conformance-lock-raise", fn -> + :reacquired + end) + end) + + assert {:ok, {:ok, :reacquired}} = Task.yield(task, 1_000) || Task.shutdown(task) + end + end + # -- Facade level -------------------------------------------------- # sabotage: in StatifierPersistence.Storage.save_position/3, drop the diff --git a/test/statifier_persistence/runs_test.exs b/test/statifier_persistence/runs_test.exs new file mode 100644 index 0000000..b76e945 --- /dev/null +++ b/test/statifier_persistence/runs_test.exs @@ -0,0 +1,682 @@ +defmodule StatifierPersistence.RunsTest do + use ExUnit.Case, async: true + + defmodule SpyStrategy do + @moduledoc false + # A pass-through StatifierPersistence.Serialization strategy whose + # config is the test pid: it reports every invocation, which is what + # lets a test assert the serialization: {module, config} opt is + # honored rather than the default strategy being hardwired. + @behaviour StatifierPersistence.Serialization + + @impl StatifierPersistence.Serialization + def with_run(test_pid, run_id, fun) do + send(test_pid, {:with_run, run_id}) + {:ok, fun.()} + end + end + + alias Statifier.Effect.{BudgetExhausted, Log, SendDelayed} + alias Statifier.Event + alias Statifier.Invoke.Types, as: InvokeTypes + alias Statifier.Machine + alias Statifier.MachineState + alias Statifier.Send.Routes + alias StatifierPersistence.{Run, Runs, Storage} + alias StatifierPersistence.Storage.InMemory + alias StatifierPersistence.Test.{FlakyAdapter, NoLockAdapter, RecordingExecutor} + alias StatifierPersistence.Testing.Charts + + # Reaches top-level final on "finish", executing two blocks in + # document order on the way - the exact-order assertion's fixture. + @final_chart_source """ + + + + + + + + + + """ + + # An immediate whose emission depends on the + # routes snapshot stamped before the step (ADR-0048): unreachable parent + # -> rejected in the core, no :send crosses the seam; reachable parent -> + # the :send effect is emitted. + @send_chart_source """ + + + + + + + + + """ + + # A registered whose start the executor can fail, and a + # transition on the error.communication re-entry (st-ADR-0051's + # failed-communication row). + @invoke_chart_source """ + + + + + + + + + + + """ + + # An immediate the executor can fail, with the error.communication + # transition waiting in the target state. + @send_error_chart_source """ + + + + + + + + + + + + """ + + # A the executor can fail - observational, so the run must not + # take the error.communication transition that would betray a re-entry. + @log_error_chart_source """ + + + + + + + + + + + + """ + + # The single-wave fixture: the primary :send failure re-enters (b -> c), + # c's onentry emits another :send whose failure must NOT re-enter - a + # second wave would land in d. + @wave_chart_source """ + + + + + + + + + + + + + + + + + + """ + + # Quiescent at "idle"; "boom" enters a raise cycle that spends whatever + # macrostep budget the run was created with. + @loop_after_event_source """ + + + + + + + + + + """ + + # The same raise cycle entered at initialization, for create-time + # exhaustion. + @loop_at_create_source """ + + + + + + + """ + + # A delayed with an author id: the deterministic-key fixture for + # the at-least-once proof (send_id from the author, ordinal from + # st-ADR-0059's timer_counter). + @delayed_send_chart_source """ + + + + + + + + + """ + + # The serialization proof's fixture: from s0, the two events reach + # order-specific final states (e1 then e2 -> s12; e2 then e1 -> s21), + # while a lost-update interleaving - both steps loading s0 - persists a + # one-event state (s1 or s2) that no serial order produces. The on + # every transition gives a slow executor something to stall on, widening + # the load-to-persist window an unserialized pair would interleave in. + @order_chart_source """ + + + + + + + + + + + + + + + """ + + setup do + {:ok, store} = Storage.new(InMemory, []) + start_supervised!(RecordingExecutor) + %{store: store} + end + + defp compile!(source) do + {:ok, machine} = Statifier.compile(source) + machine + end + + # The active leaf states as sorted string ids - the readable form of a + # configuration assertion. + defp active_ids(%MachineState{} = machine_state) do + machine_state + |> MachineState.active_leaf_states() + |> Enum.map(&Machine.id(machine_state.machine, &1)) + |> Enum.sort() + end + + # An executor failing exactly the given effect tags, recording every call + # through RecordingExecutor either way. + defp failing_executor(tags, reason \\ :down) do + fn effect, context -> + RecordingExecutor.execute(effect, context) + + if elem(effect, 0) in tags do + {:error, reason} + else + :ok + end + end + end + + describe "create/4" do + # sabotage: write_run/6 inserts with position: :skip -> red (:run_position_missing) + test "persists an :active run whose blob decodes to the initialized configuration", + %{store: store} do + {_source, machine} = Charts.chart_a() + + assert {:ok, %Run{run_id: "run-1", status: :active, failure: nil}, %MachineState{} = ms} = + Runs.create(store, "run-1", machine, executor: RecordingExecutor) + + assert {:ok, %{status: :active, failure: nil}} = Storage.fetch_run(store, "run-1") + assert {:ok, loaded} = Storage.load_run_position(store, "run-1", machine) + assert loaded.configuration == ms.configuration + end + + # sabotage: persist_tail/6 swallows write_run/6's result and returns {:ok, ...} -> red + test "on an existing run id returns {:error, :run_exists}", %{store: store} do + {_source, machine} = Charts.chart_a() + + assert {:ok, _run, _ms} = Runs.create(store, "run-1", machine, executor: RecordingExecutor) + + assert {:error, :run_exists} = + Runs.create(store, "run-1", machine, executor: RecordingExecutor) + end + end + + describe "step/5" do + # sabotage: write_run/6 updates with position: :skip -> red (stored config stays initial) + test "advances the position and persists it", %{store: store} do + {_source, machine} = Charts.chart_a() + {:ok, _run, initial} = Runs.create(store, "run-1", machine, executor: RecordingExecutor) + + assert {:ok, %Run{status: :active}, %MachineState{} = stepped} = + Runs.step(store, "run-1", machine, Event.external("go"), + executor: RecordingExecutor + ) + + refute stepped.configuration == initial.configuration + assert {:ok, loaded} = Storage.load_run_position(store, "run-1", machine) + assert loaded.configuration == stepped.configuration + end + + # sabotage: execute_effects/3 reverses the effect list before executing -> red + test "hands effects to the executor in list order, excluding :done", %{store: store} do + machine = compile!(@final_chart_source) + {:ok, _run, _ms} = Runs.create(store, "run-1", machine, executor: RecordingExecutor) + RecordingExecutor.reset() + + assert {:ok, _run, _ms} = + Runs.step(store, "run-1", machine, Event.external("finish"), + executor: RecordingExecutor + ) + + assert [{:log, %Log{label: "one"}}, {:log, %Log{label: "two"}}] = + RecordingExecutor.effects() + end + + # sabotage: run_status/2 drops the status == :done -> :completed arm -> red + test "a chart reaching top-level final completes the run and consumes {:done, _}", + %{store: store} do + machine = compile!(@final_chart_source) + {:ok, _run, _ms} = Runs.create(store, "run-1", machine, executor: RecordingExecutor) + + assert {:ok, %Run{status: :completed}, %MachineState{status: :done}} = + Runs.step(store, "run-1", machine, Event.external("finish"), + executor: RecordingExecutor + ) + + assert {:ok, %{status: :completed}} = Storage.fetch_run(store, "run-1") + refute Enum.any?(RecordingExecutor.effects(), &match?({:done, _}, &1)) + end + + # sabotage: Run.from_record/1 hardcodes status: :active -> red + test "on a completed run discards without invoking the executor", %{store: store} do + machine = compile!(@final_chart_source) + {:ok, _run, _ms} = Runs.create(store, "run-1", machine, executor: RecordingExecutor) + + {:ok, %Run{status: :completed}, _ms} = + Runs.step(store, "run-1", machine, Event.external("finish"), executor: RecordingExecutor) + + RecordingExecutor.reset() + + assert {:discarded, %Run{run_id: "run-1", status: :completed}} = + Runs.step(store, "run-1", machine, Event.external("finish"), + executor: RecordingExecutor + ) + + assert RecordingExecutor.calls() == [] + end + + # sabotage: repair_terminal/3 skips the Storage.update_run/5 repair -> red (record stays :active) + test "a run record whose :active status lies about a terminal position is discarded and repaired", + %{store: store} do + machine = compile!(@final_chart_source) + {:ok, _run, _ms} = Runs.create(store, "run-1", machine, executor: RecordingExecutor) + + {:ok, _run, terminal_ms} = + Runs.step(store, "run-1", machine, Event.external("finish"), executor: RecordingExecutor) + + # Hand-write the lie: a terminal stored position under an :active + # status, the state a crash between chart completion and record + # update would leave behind. + :ok = Storage.update_run(store, "run-1", terminal_ms, :active) + RecordingExecutor.reset() + + assert {:discarded, %Run{status: :completed}} = + Runs.step(store, "run-1", machine, Event.external("finish"), + executor: RecordingExecutor + ) + + assert RecordingExecutor.calls() == [] + assert {:ok, %{status: :completed}} = Storage.fetch_run(store, "run-1") + end + + # sabotage: Executor.run/3's fun clause returns :ok without calling the fun -> red + test "accepts an arity-2 fun as executor and passes the run context", %{store: store} do + machine = compile!(@final_chart_source) + parent = self() + + fun = fn effect, context -> + send(parent, {:executed, effect, context}) + :ok + end + + {:ok, _run, _ms} = Runs.create(store, "run-1", machine, executor: fun) + + {:ok, _run, _ms} = + Runs.step(store, "run-1", machine, Event.external("finish"), executor: fun) + + content_hash = Machine.identity(machine).content_hash + + assert_receive {:executed, {:log, %Log{label: "one"}}, + %{run_id: "run-1", content_hash: ^content_hash}} + end + + # sabotage: step_loaded/6 skips the put_routes/2 re-stamp -> red (the :send is emitted anyway) + test "stamps the routes snapshot onto the loaded position before the step", + %{store: store} do + machine = compile!(@send_chart_source) + + # Parent declared unreachable: the core rejects the against + # the stamped snapshot, so no :send effect crosses the seam. An + # unstamped (nil) snapshot would have emitted it - "no determination + # made" - which is what makes this assert the stamp itself. + {:ok, _run, _ms} = Runs.create(store, "run-blocked", machine, executor: RecordingExecutor) + RecordingExecutor.reset() + + {:ok, _run, _ms} = + Runs.step(store, "run-blocked", machine, Event.external("go"), + executor: RecordingExecutor, + routes: Routes.new() + ) + + refute Enum.any?(RecordingExecutor.effects(), &match?({:send, _}, &1)) + + # Parent declared reachable: the same chart emits the :send. + {:ok, _run, _ms} = Runs.create(store, "run-open", machine, executor: RecordingExecutor) + RecordingExecutor.reset() + + {:ok, _run, _ms} = + Runs.step(store, "run-open", machine, Event.external("go"), + executor: RecordingExecutor, + routes: Routes.new(parent?: true) + ) + + assert Enum.any?( + RecordingExecutor.effects(), + &match?({:send, %{event: "ping", target: "#_parent"}}, &1) + ) + end + + # sabotage: step/5's fetch error arm rewrites the reason -> red + test "on a missing run returns {:error, :run_not_found}", %{store: store} do + {_source, machine} = Charts.chart_a() + + assert {:error, :run_not_found} = + Runs.step(store, "absent", machine, Event.external("go"), + executor: RecordingExecutor + ) + end + end + + describe "error re-entry (ADR-0004 decision 4)" do + # sabotage: reentry_origin/1's :invoke clause returns :observational -> red (run stays in b) + test "a failed :invoke re-enters as error.communication and the persisted position reflects it", + %{store: store} do + machine = compile!(@invoke_chart_source) + executor = failing_executor([:invoke]) + invoke_types = InvokeTypes.new(types: ["myapp:enrich"]) + + {:ok, _run, _ms} = + Runs.create(store, "run-1", machine, executor: executor, invoke_types: invoke_types) + + assert {:ok, %Run{status: :active}, %MachineState{} = stepped} = + Runs.step(store, "run-1", machine, Event.external("go"), + executor: executor, + invoke_types: invoke_types + ) + + assert active_ids(stepped) == ["errored"] + assert {:ok, loaded} = Storage.load_run_position(store, "run-1", machine) + assert active_ids(loaded) == ["errored"] + end + + # sabotage: reentry_origin/1's :send clause returns :observational -> red (run stays in b) + test "a failed :send re-enters as error.communication the same way", %{store: store} do + machine = compile!(@send_error_chart_source) + executor = failing_executor([:send]) + + {:ok, _run, _ms} = Runs.create(store, "run-1", machine, executor: executor) + + assert {:ok, %Run{status: :active}, stepped} = + Runs.step(store, "run-1", machine, Event.external("go"), + executor: executor, + routes: Routes.new(parent?: true) + ) + + assert active_ids(stepped) == ["errored"] + assert {:ok, loaded} = Storage.load_run_position(store, "run-1", machine) + assert active_ids(loaded) == ["errored"] + end + + # sabotage: reentry_origin/1's catch-all re-enters as {:state, 0} -> red (run lands in errored) + test "a failed :log is discarded and the run is unaffected", %{store: store} do + machine = compile!(@log_error_chart_source) + executor = failing_executor([:log]) + + {:ok, _run, _ms} = Runs.create(store, "run-1", machine, executor: executor) + + assert {:ok, %Run{status: :active}, stepped} = + Runs.step(store, "run-1", machine, Event.external("go"), executor: executor) + + assert active_ids(stepped) == ["b"] + assert Enum.any?(RecordingExecutor.effects(), &match?({:log, %Log{label: "observed"}}, &1)) + end + + # sabotage: reenter_one/4 re-enters the wave's own failures recursively -> red (run lands in d) + test "re-entry is single-wave: a deterministically failing executor terminates in one wave", + %{store: store} do + machine = compile!(@wave_chart_source) + executor = failing_executor([:send, :log, :datamodel_change, :datamodel_init, :cancel]) + + {:ok, _run, _ms} = Runs.create(store, "run-1", machine, executor: executor) + + assert {:ok, %Run{status: :active}, stepped} = + Runs.step(store, "run-1", machine, Event.external("go"), + executor: executor, + routes: Routes.new(parent?: true) + ) + + # The primary :send failure re-entered (b -> c); c's onentry :send + # failed too, and that failure was dropped, not re-entered - a second + # wave would have landed in d. + assert active_ids(stepped) == ["c"] + assert {:ok, loaded} = Storage.load_run_position(store, "run-1", machine) + assert active_ids(loaded) == ["c"] + end + end + + describe "budget exhaustion" do + # sabotage: write_run/6 persists the exhausted position (always {:persist, nil}) -> red + test "at step: run fails, prior blob stays intact, the error surfaces after the persist", + %{store: store} do + machine = compile!(@loop_after_event_source) + + {:ok, _run, created} = + Runs.create(store, "run-1", machine, + executor: RecordingExecutor, + initialize: [max_macrostep_rounds: 5] + ) + + assert {:error, {:budget_exhausted, %BudgetExhausted{budget: 5}}} = + Runs.step(store, "run-1", machine, Event.external("boom"), + executor: RecordingExecutor + ) + + assert {:ok, %{status: :failed, failure: "budget_exhausted: 5 rounds"}} = + Storage.fetch_run(store, "run-1") + + # The pre-step blob remains: the loaded position is the one create + # persisted, not the non-quiescent exhausted state. + assert {:ok, loaded} = Storage.load_run_position(store, "run-1", machine) + assert loaded.configuration == created.configuration + assert active_ids(loaded) == ["idle"] + end + + # sabotage: persist_tail/6 returns {:ok, ...} with no budget_effect/1 case -> red + test "at create: run fails with no position blob and the error surfaces", %{store: store} do + machine = compile!(@loop_at_create_source) + + assert {:error, {:budget_exhausted, %BudgetExhausted{budget: 4}}} = + Runs.create(store, "run-1", machine, + executor: RecordingExecutor, + initialize: [max_macrostep_rounds: 4] + ) + + assert {:ok, %{status: :failed, failure: "budget_exhausted: 4 rounds"}} = + Storage.fetch_run(store, "run-1") + + assert {:error, :run_position_missing} = + Storage.load_run_position(store, "run-1", machine) + end + end + + describe "fail/4" do + # sabotage: fail/4 returns {:ok, ...} without calling Storage.update_run_status/4 -> red + test "on an active run persists :failed with the reason, position untouched", + %{store: store} do + {_source, machine} = Charts.chart_a() + {:ok, _run, created} = Runs.create(store, "run-1", machine, executor: RecordingExecutor) + + assert {:ok, %Run{status: :failed, failure: "operator: abandoned"}} = + Runs.fail(store, "run-1", "operator: abandoned") + + assert {:ok, %{status: :failed, failure: "operator: abandoned"}} = + Storage.fetch_run(store, "run-1") + + assert {:ok, loaded} = Storage.load_run_position(store, "run-1", machine) + assert loaded.configuration == created.configuration + end + + # sabotage: fail/4 drops the terminal-status head clause -> red ({:ok, ...} on a completed run) + test "on a terminal run discards without writing", %{store: store} do + machine = compile!(@final_chart_source) + {:ok, _run, _ms} = Runs.create(store, "run-1", machine, executor: RecordingExecutor) + + {:ok, %Run{status: :completed}, _ms} = + Runs.step(store, "run-1", machine, Event.external("finish"), executor: RecordingExecutor) + + assert {:discarded, %Run{status: :completed}} = + Runs.fail(store, "run-1", "operator: abandoned") + + assert {:ok, %{status: :completed, failure: nil}} = Storage.fetch_run(store, "run-1") + end + + # sabotage: fail/4's fetch error arm rewrites the reason -> red + test "on a missing run returns {:error, :run_not_found}", %{store: store} do + assert {:error, :run_not_found} = Runs.fail(store, "absent", "operator: abandoned") + end + end + + describe "per-run serialization (ADR-0004 decision 5)" do + # sabotage: InMemory.lock_run/3 runs fun without the exclusion + # ({:ok, fun.()} with no acquire) -> red (both steps load s0 and the + # persisted state is a one-event s1/s2, which no serial order produces) + test "two concurrent steps on one run serialize to some order of the two events", + %{store: store} do + machine = compile!(@order_chart_source) + + # The slow executor stalls each step between load and persist, so an + # unserialized pair reliably overlaps there instead of racing past + # each other by luck. + executor = fn _effect, _context -> + Process.sleep(50) + :ok + end + + {:ok, _run, _ms} = Runs.create(store, "run-1", machine, executor: executor) + + tasks = + for event <- ["e1", "e2"] do + Task.async(fn -> + Runs.step(store, "run-1", machine, Event.external(event), executor: executor) + end) + end + + results = Task.await_many(tasks, 10_000) + assert Enum.all?(results, &match?({:ok, %Run{status: :active}, %MachineState{}}, &1)) + + # The final persisted state is one of the two serial orders' results, + # never an interleaving's. + assert {:ok, loaded} = Storage.load_run_position(store, "run-1", machine) + assert active_ids(loaded) in [["s12"], ["s21"]] + end + + # sabotage: Runs.serialized/4 ignores the :serialization opt and always + # uses the {AdapterLock, store} default -> red (no {:with_run, _} + # message ever arrives) + test "the serialization: {module, config} override is honored on every entry point", + %{store: store} do + {_source, machine} = Charts.chart_a() + serialization = {SpyStrategy, self()} + + assert {:ok, _run, _ms} = + Runs.create(store, "run-1", machine, + executor: RecordingExecutor, + serialization: serialization + ) + + assert_receive {:with_run, "run-1"} + + assert {:ok, _run, _ms} = + Runs.step(store, "run-1", machine, Event.external("go"), + executor: RecordingExecutor, + serialization: serialization + ) + + assert_receive {:with_run, "run-1"} + + assert {:ok, %Run{status: :failed}} = + Runs.fail(store, "run-1", "operator: abandoned", serialization: serialization) + + assert_receive {:with_run, "run-1"} + end + + # sabotage: AdapterLock.with_run/3 falls back to {:ok, fun.()} when the + # adapter exports no lock_run/3 -> red (create returns {:ok, ...} and + # the run exists) + test "an adapter without lock_run/3 under the default strategy is refused" do + {:ok, store} = Storage.new(NoLockAdapter, []) + {_source, machine} = Charts.chart_a() + + assert {:error, {:serialization, :not_supported}} = + Runs.create(store, "run-1", machine, executor: RecordingExecutor) + + # The refusal precedes the tail: nothing was inserted. + assert {:error, :run_not_found} = Storage.fetch_run(store, "run-1") + end + end + + describe "at-least-once redelivery" do + # sabotage: persist_tail/6 swallows write_run/6's error and returns {:ok, ...} -> red + test "a failed persist re-drives the same event and re-emits identical deterministic keys" do + {:ok, store} = Storage.new(FlakyAdapter, []) + machine = compile!(@delayed_send_chart_source) + + {:ok, _run, _ms} = Runs.create(store, "run-1", machine, executor: RecordingExecutor) + RecordingExecutor.reset() + + # First step: the effect executes, then the injected adapter failure + # lands exactly where a crash between execute and persist would. + assert {:error, {:adapter, :injected}} = + Runs.step(store, "run-1", machine, Event.external("go"), + executor: RecordingExecutor + ) + + # Re-driving the same event succeeds and re-emits the same effect. + assert {:ok, %Run{status: :active}, _ms} = + Runs.step(store, "run-1", machine, Event.external("go"), + executor: RecordingExecutor + ) + + assert [{:send_delayed, %SendDelayed{} = first}, {:send_delayed, %SendDelayed{} = second}] = + RecordingExecutor.effects() + + # Field-level equality of the deterministic keys (st-ADR-0054 + # decision 3, st-ADR-0059), not just list length. + assert first.send_id == "ping" + assert second.send_id == first.send_id + assert second.ordinal == first.ordinal + + assert {second.macrostep, second.microstep, second.round} == + {first.macrostep, first.microstep, first.round} + + assert second == first + end + end +end diff --git a/test/statifier_persistence/storage/in_memory_test.exs b/test/statifier_persistence/storage/in_memory_test.exs index d7e4c09..487f4ae 100644 --- a/test/statifier_persistence/storage/in_memory_test.exs +++ b/test/statifier_persistence/storage/in_memory_test.exs @@ -44,4 +44,32 @@ defmodule StatifierPersistence.Storage.InMemoryTest do assert :ok = InMemory.save_chart(first_opts, chart_record) assert {:error, :chart_not_found} = InMemory.fetch_chart(second_opts, "sha256:lifecycle") end + + # sabotage: in InMemory.insert_run/2, drop the exists-check inside + # Agent.get_and_update/2 and always write with :ok -> red, all 25 + # concurrent inserts returned :ok instead of exactly one. Verified red + # (together with the conformance suite's duplicate-insert test under + # this one mutation), reverted. + test "insert_run/2 admits exactly one of many concurrent inserts for one run_id" do + {:ok, opts} = InMemory.init([]) + + run_record = %{ + run_id: "run-atomic", + status: :active, + content_hash: "sha256:lifecycle", + identity_blob: <<1, 2, 3>>, + position_blob: <<7, 8, 9>>, + failure: nil + } + + results = + 1..25 + |> Task.async_stream(fn _index -> InMemory.insert_run(opts, run_record) end, + max_concurrency: 25 + ) + |> Enum.map(fn {:ok, result} -> result end) + + assert Enum.count(results, &(&1 == :ok)) == 1 + assert Enum.count(results, &(&1 == {:error, :run_exists})) == 24 + end end diff --git a/test/statifier_persistence/storage_test.exs b/test/statifier_persistence/storage_test.exs index 94de358..94e5b0c 100644 --- a/test/statifier_persistence/storage_test.exs +++ b/test/statifier_persistence/storage_test.exs @@ -1,8 +1,12 @@ defmodule StatifierPersistence.StorageTest do use ExUnit.Case, async: true + alias Statifier.Machine + alias Statifier.Machine.Identity + alias Statifier.MachineState alias StatifierPersistence.Storage alias StatifierPersistence.Storage.InMemory + alias StatifierPersistence.Testing.Charts defmodule FailingAdapter do @moduledoc false @@ -22,6 +26,15 @@ defmodule StatifierPersistence.StorageTest do @impl StatifierPersistence.Storage.Adapter def fetch_position(_opts, _session_id), do: {:error, {:adapter, :boom}} + + @impl StatifierPersistence.Storage.Adapter + def insert_run(_opts, _run_record), do: {:error, {:adapter, :boom}} + + @impl StatifierPersistence.Storage.Adapter + def fetch_run(_opts, _run_id), do: {:error, {:adapter, :boom}} + + @impl StatifierPersistence.Storage.Adapter + def update_run(_opts, _run_record), do: {:error, {:adapter, :boom}} end # The conformance suite (test/statifier_persistence/storage/in_memory_conformance_test.exs, @@ -49,4 +62,183 @@ defmodule StatifierPersistence.StorageTest do test "new/2 returns the adapter's own init/1 failure unchanged" do assert {:error, {:adapter, :boom}} = Storage.new(FailingAdapter, []) end + + # -- Run records: the facade arms Phase 2 adds ---------------------- + + describe "run records" do + setup do + {:ok, store} = Storage.new(InMemory, []) + %{store: store} + end + + # sabotage: in Storage.insert_run/5, replace the + # store.adapter.insert_run(store.opts, run_record) call with a bare :ok + # that never writes -> red, fetch_run/2 below returned + # {:error, :run_not_found} instead of the record. Verified red, + # reverted. + test "insert_run/5 and load_run_position/3: the guarded round trip", %{store: store} do + {_source, machine} = Charts.chart_a() + + machine_state = + MachineState.new(machine, session_id: "sess_run_round_trip", datamodel: %{"count" => 1}) + + assert :ok = Storage.insert_run(store, "run-round-trip", machine_state, :active) + + assert {:ok, record} = Storage.fetch_run(store, "run-round-trip") + assert %{status: :active, failure: nil} = record + assert record.content_hash == Machine.identity(machine).content_hash + + assert {:ok, loaded} = Storage.load_run_position(store, "run-round-trip", machine) + assert loaded.configuration == machine_state.configuration + assert loaded.datamodel == machine_state.datamodel + assert loaded.status == machine_state.status + end + + # sabotage: in Storage.insert_run/5, replace the nil-identity refusal + # arm with a write of a dummy-identity record (content_hash "dummy", + # empty identity_blob) -> red, this test's assertion that the insert is + # refused failed (it returned :ok). Verified red, reverted. + test "insert_run/5 refuses an unidentified machine, and nothing is written", %{store: store} do + machine_state = + MachineState.new(Charts.unidentified_machine(), session_id: "sess_run_unidentified") + + assert {:error, :unidentified_chart} = + Storage.insert_run(store, "run-unidentified", machine_state, :active, + position: :skip + ) + + assert {:error, :run_not_found} = Storage.fetch_run(store, "run-unidentified") + end + + # sabotage: in Storage.update_run/5, replace the nil-identity refusal + # arm with a fetch-and-overwrite of the stored record's status and + # failure -> red, this test's assertion that the update is refused + # failed (it returned :ok). Verified red, reverted. + test "update_run/5 refuses an unidentified machine", %{store: store} do + {_source, machine} = Charts.chart_a() + machine_state = MachineState.new(machine, session_id: "sess_run_update_unidentified") + + assert :ok = Storage.insert_run(store, "run-update-unidentified", machine_state, :active) + + unidentified_state = + MachineState.new(Charts.unidentified_machine(), + session_id: "sess_run_update_unidentified" + ) + + assert {:error, :unidentified_chart} = + Storage.update_run(store, "run-update-unidentified", unidentified_state, :failed, + position: :skip + ) + end + + # sabotage: in Storage.load_run_position/3, replace the whole with-chain + # (fetch_run -> precheck_identity/2 -> nil arm -> Position.from_binary/2) + # with a body that fetches the run record and unconditionally returns + # {:ok, MachineState.new(machine)} -> red, this test saw a plain + # {:ok, _} instead of {:identity_mismatch, _, _} (the round-trip, + # unidentified, and missing-position tests in this describe went red + # under the same mutation). Verified red, reverted. + test "load_run_position/3 refuses a different chart revision, not raised", %{store: store} do + {_source_a, machine_a} = Charts.chart_a() + {_source_b, machine_b} = Charts.chart_b() + + refute Identity.matches?(Machine.identity(machine_a), Machine.identity(machine_b)) + + machine_state = MachineState.new(machine_a, session_id: "sess_run_mismatch") + + assert :ok = Storage.insert_run(store, "run-mismatch", machine_state, :active) + + assert {:error, {:identity_mismatch, expected, actual}} = + Storage.load_run_position(store, "run-mismatch", machine_b) + + assert expected.content_hash == Machine.identity(machine_a).content_hash + assert actual.content_hash == Machine.identity(machine_b).content_hash + end + + # sabotage: in Storage's private precheck_identity/2, delete the + # %Machine{identity: nil} -> {:error, :unidentified_chart} clause -> + # red, Identity.matches?/2 is total and this test saw an + # {:identity_mismatch, _, nil} tuple instead of :unidentified_chart. + # Verified red, reverted. + test "load_run_position/3 refuses an unidentified machine as :unidentified_chart", %{ + store: store + } do + {_source, machine} = Charts.chart_a() + machine_state = MachineState.new(machine, session_id: "sess_run_load_unidentified") + + assert :ok = Storage.insert_run(store, "run-load-unidentified", machine_state, :active) + + assert {:error, :unidentified_chart} = + Storage.load_run_position( + store, + "run-load-unidentified", + Charts.unidentified_machine() + ) + end + + # sabotage: in Storage.load_run_position/3, change the nil + # position_blob arm to return {:error, :run_not_found} instead of + # :run_position_missing -> red, this test's pattern match on the + # dedicated arm saw the wrong error. Verified red, reverted. + test "load_run_position/3 reports :run_position_missing for a nil position_blob", %{ + store: store + } do + {_source, machine} = Charts.chart_a() + machine_state = MachineState.new(machine, session_id: "sess_run_no_position") + + assert :ok = + Storage.insert_run(store, "run-no-position", machine_state, :failed, + position: :skip, + failure: "budget_exhausted: 100 rounds" + ) + + assert {:ok, %{position_blob: nil, failure: "budget_exhausted: 100 rounds"}} = + Storage.fetch_run(store, "run-no-position") + + assert {:error, :run_position_missing} = + Storage.load_run_position(store, "run-no-position", machine) + end + + # sabotage: in Storage's private update_position_blob/4, change the + # :skip clause to return {:ok, nil} instead of fetching the current + # record and carrying its position_blob forward -> red, the equality + # assertion on stored_blob below saw nil. Verified red, reverted. + test "update_run/5 under position: :skip carries the stored blob forward verbatim", %{ + store: store + } do + {_source, machine} = Charts.chart_a() + machine_state = MachineState.new(machine, session_id: "sess_run_skip_carry") + + assert :ok = Storage.insert_run(store, "run-skip-carry", machine_state, :active) + assert {:ok, %{position_blob: stored_blob}} = Storage.fetch_run(store, "run-skip-carry") + assert is_binary(stored_blob) + + assert :ok = + Storage.update_run(store, "run-skip-carry", machine_state, :failed, + position: :skip, + failure: "abandoned: operator request" + ) + + assert {:ok, updated} = Storage.fetch_run(store, "run-skip-carry") + assert %{status: :failed, failure: "abandoned: operator request"} = updated + assert updated.position_blob == stored_blob + end + + # sabotage: two mutations together, because either layer alone + # backstops the other - update_position_blob/4's :skip clause returning + # {:ok, nil} without the fetch AND InMemory.update_run/2 upserting on a + # missing run_id -> red, the update below returned :ok instead of + # {:error, :run_not_found}. Verified red, both reverted. + test "update_run/5 under position: :skip reports :run_not_found for an unknown run", %{ + store: store + } do + {_source, machine} = Charts.chart_a() + machine_state = MachineState.new(machine, session_id: "sess_run_skip_missing") + + assert {:error, :run_not_found} = + Storage.update_run(store, "run-skip-missing", machine_state, :failed, + position: :skip + ) + end + end end diff --git a/test/support/flaky_adapter.ex b/test/support/flaky_adapter.ex new file mode 100644 index 0000000..c92f7a0 --- /dev/null +++ b/test/support/flaky_adapter.ex @@ -0,0 +1,56 @@ +defmodule StatifierPersistence.Test.FlakyAdapter do + @moduledoc """ + A delegating `StatifierPersistence.Storage.Adapter` wrapping + `StatifierPersistence.Storage.InMemory` whose `update_run/2` fails exactly + once with `{:error, {:adapter, :injected}}`, then delegates normally. + + The at-least-once proof's fixture: the injected failure lands between + effect execution and persist, exactly where a crash would, so re-driving + the same event must re-emit the same effects with identical deterministic + keys (st-ADR-0054 decision 3, st-ADR-0059). + """ + + @behaviour StatifierPersistence.Storage.Adapter + + alias StatifierPersistence.Storage.InMemory + + @impl true + def init(opts) do + with {:ok, inner} <- InMemory.init(opts) do + {:ok, trip} = Agent.start_link(fn -> false end) + {:ok, %{inner: inner, trip: trip}} + end + end + + @impl true + def save_chart(%{inner: inner}, chart_record), do: InMemory.save_chart(inner, chart_record) + + @impl true + def fetch_chart(%{inner: inner}, content_hash), do: InMemory.fetch_chart(inner, content_hash) + + @impl true + def save_position(%{inner: inner}, position_record), + do: InMemory.save_position(inner, position_record) + + @impl true + def fetch_position(%{inner: inner}, session_id), + do: InMemory.fetch_position(inner, session_id) + + @impl true + def insert_run(%{inner: inner}, run_record), do: InMemory.insert_run(inner, run_record) + + @impl true + def fetch_run(%{inner: inner}, run_id), do: InMemory.fetch_run(inner, run_id) + + @impl true + def lock_run(%{inner: inner}, run_id, fun), do: InMemory.lock_run(inner, run_id, fun) + + @impl true + def update_run(%{inner: inner, trip: trip}, run_record) do + if Agent.get_and_update(trip, fn tripped -> {tripped, true} end) do + InMemory.update_run(inner, run_record) + else + {:error, {:adapter, :injected}} + end + end +end diff --git a/test/support/no_lock_adapter.ex b/test/support/no_lock_adapter.ex new file mode 100644 index 0000000..b88706d --- /dev/null +++ b/test/support/no_lock_adapter.ex @@ -0,0 +1,39 @@ +defmodule StatifierPersistence.Test.NoLockAdapter do + @moduledoc """ + A delegating `StatifierPersistence.Storage.Adapter` wrapping + `StatifierPersistence.Storage.InMemory` that implements every required + callback and deliberately does NOT export the optional `lock_run/3`. + + The fixture for the default serialization strategy's refusal arm: + `StatifierPersistence.Serialization.AdapterLock` over this adapter must + return `{:error, {:serialization, :not_supported}}`. + """ + + @behaviour StatifierPersistence.Storage.Adapter + + alias StatifierPersistence.Storage.InMemory + + @impl true + defdelegate init(opts), to: InMemory + + @impl true + defdelegate save_chart(opts, chart_record), to: InMemory + + @impl true + defdelegate fetch_chart(opts, content_hash), to: InMemory + + @impl true + defdelegate save_position(opts, position_record), to: InMemory + + @impl true + defdelegate fetch_position(opts, session_id), to: InMemory + + @impl true + defdelegate insert_run(opts, run_record), to: InMemory + + @impl true + defdelegate fetch_run(opts, run_id), to: InMemory + + @impl true + defdelegate update_run(opts, run_record), to: InMemory +end diff --git a/test/support/recording_executor.ex b/test/support/recording_executor.ex new file mode 100644 index 0000000..8131eeb --- /dev/null +++ b/test/support/recording_executor.ex @@ -0,0 +1,45 @@ +defmodule StatifierPersistence.Test.RecordingExecutor do + @moduledoc """ + An Agent-backed `StatifierPersistence.Executor` implementation that + records every `{effect, context}` pair it receives, in call order, and + always answers `:ok`. + + Registered under this module's own name, so `execute/2` needs no handle + beyond the behaviour's own arguments. Start it per test with + `start_supervised!/1`; tests within one module run serially, so the + single name never collides. + """ + + @behaviour StatifierPersistence.Executor + + use Agent + + @spec start_link(term()) :: Agent.on_start() + def start_link(_opts \\ []) do + Agent.start_link(fn -> [] end, name: __MODULE__) + end + + @impl StatifierPersistence.Executor + def execute(effect, context) do + Agent.update(__MODULE__, &[{effect, context} | &1]) + :ok + end + + @doc "Every recorded `{effect, context}` pair, oldest first." + @spec calls() :: [{Statifier.Effect.t(), StatifierPersistence.Executor.context()}] + def calls do + Agent.get(__MODULE__, &Enum.reverse/1) + end + + @doc "Every recorded effect, oldest first." + @spec effects() :: [Statifier.Effect.t()] + def effects do + Enum.map(calls(), fn {effect, _context} -> effect end) + end + + @doc "Drops everything recorded so far." + @spec reset() :: :ok + def reset do + Agent.update(__MODULE__, fn _recorded -> [] end) + end +end