Adds the run lifecycle and the stepper loop

changelog.d/sp-4an.2.1.md

@@ -0,0 +1,42 @@
+# sp-4an.2.1
+
+## Added
+
+- Run records on the storage contract: `StatifierPersistence.Storage.Adapter`
+  gains `insert_run/2`, `fetch_run/2`, and `update_run/2` callbacks with
+  `run_record`/`run_status` types and the `:run_exists` / `:run_not_found`
+  error arms; `StatifierPersistence.Storage.InMemory` implements them.
+- Guarded run access on the facade: `StatifierPersistence.Storage.insert_run/5`,
+  `update_run/5`, `fetch_run/2`, and `load_run_position/3` (identity-guarded,
+  with the `:run_position_missing` arm for a run persisted without a
+  position).
+- Run-record conformance tests in
+  `StatifierPersistence.Testing.StorageConformance`, so downstream adapters
+  inherit the same contract checks.
+- The run lifecycle as a library: `StatifierPersistence.Runs.create/4` and
+  `step/5` drive the load -> re-stamp -> step -> execute -> persist loop
+  over durable run records, handing effects to a host-supplied
+  `StatifierPersistence.Executor` (behaviour or arity-2 fun) and returning
+  the host-facing `StatifierPersistence.Run` struct; events to a terminal
+  run come back as `{:discarded, run}`.
+- Failure semantics on the loop: executor failures on actionable effects
+  re-enter the chart as `error.communication` events (single wave per step,
+  observational failures discarded); effect execution is at-least-once, with
+  a failed persist re-driving the same event and re-emitting the same
+  effects under identical deterministic keys; budget exhaustion persists a
+  `:failed` run (position untouched) and returns
+  `{:error, {:budget_exhausted, payload}}`.
+- `StatifierPersistence.Runs.fail/4`, the host-driven abandonment: marks an
+  active run `:failed` with a reason, leaves the stored position untouched,
+  and discards on a terminal run - backed by the new status-only writer
+  `StatifierPersistence.Storage.update_run_status/4`.
+- Pluggable per-run serialization: the `StatifierPersistence.Serialization`
+  behaviour (`with_run/3`), selected per lifecycle call with
+  `serialization: {module, config}` on `Runs.create/4`, `step/5`, and
+  `fail/4`. The default strategy,
+  `StatifierPersistence.Serialization.AdapterLock`, delegates to the new
+  optional adapter callback
+  `StatifierPersistence.Storage.Adapter.lock_run/3` (implemented by
+  `InMemory`, conformance-tested when exported) and refuses with
+  `{:error, {:serialization, :not_supported}}` when the adapter does not
+  export it.

docs/adr/0003-storage-adapter-behaviour-and-the-identity-guard.md

@@ -2,7 +2,10 @@
 
 Status: accepted (2026-08-21) - amended 2026-08-21 (sp-5qa Phase 4: adds the
 optional per-test isolation callback, `isolate/1`, to the behaviour's
-contract surface, alongside the conformance suite decision 5 already names)
+contract surface, alongside the conformance suite decision 5 already names);
+amended 2026-08-22 (sp-4an.2.1 Phase 5: adds the optional per-run lock
+callback, `lock_run/3`, to the behaviour's contract surface, as the seam
+ADR-0004 decision 5's default serialization strategy delegates to)
 
 ## Context
 
@@ -47,9 +50,10 @@ it. This is why the chart record's payload is opaque here - see Decision 1.
 `fetch_chart/2`, `save_position/2`, `fetch_position/2` - takes and returns
 binaries plus engine identity strings. No callback receives a
 `Statifier.Machine.t()` and none returns a `Statifier.MachineState.t()`. (The
-optional `isolate/1` this ADR's 2026-08-21 amendment adds is the one
-exception, by design: it carries no chart or position data at all - see the
-amendment under Decision 5.) The chart record's
+optional `isolate/1` and `lock_run/3` this ADR's 2026-08-21 and 2026-08-22
+amendments add are the exceptions, by design: neither carries chart or
+position data at all - see the amendments under Decision 5.) The chart
+record's
 `chart_blob` is opaque to this layer: this record does not choose between
 `Statifier.Chart.to_binary/1`'s envelope and a host's own retained source,
 because both satisfy the only property the layer needs - given the blob
@@ -110,6 +114,23 @@ in `lib/` for a downstream adapter to reuse - is what makes the hook
 possible in the first place, so it belongs with that decision rather than as
 a new one.)*
 
+*(Amended 2026-08-22, sp-4an.2.1 Phase 5: the behaviour gains a second
+optional callback, `c:StatifierPersistence.Storage.Adapter.lock_run/3` -
+mutual exclusion per `run_id`, running the given fun while the exclusion is
+held and releasing it on any exit, a raise escaping the fun included, so
+the lock cannot leak. It is the seam ADR-0004 decision 5's default
+serialization strategy (`StatifierPersistence.Serialization.AdapterLock`)
+delegates to, and the strategy refuses with
+`{:error, {:serialization, :not_supported}}` when an adapter does not
+export it; an Ecto adapter implements it as a transaction-scoped row lock
+(sp-4an.3). Like `isolate/1` it is declared `@optional_callbacks` with no
+default implementation, carries no chart or position data, and decodes
+nothing, so decision 1's blobs-only rule holds; the conformance suite
+exercises it only when the adapter under test exports it, the same
+`function_exported?/3` shape the isolate amendment records - which is why
+this widening, like that one, is recorded here with decision 5 rather than
+as a new decision.)*
+
 ## Consequences
 
 - An adapter author has no code path in which to be careless about the

docs/adr/0004-run-lifecycle-executor-seam-and-serialization.md

@@ -0,0 +1,179 @@
+# ADR-0004: Run lifecycle, the executor seam, and per-run serialization
+
+Status: accepted (2026-08-22)
+
+## Context
+
+The charter's loop is this package's reason to exist: load a persisted
+position, step it, execute the effects, persist (sp-4an.2, restating the
+charter's scope bullets 2 and 3). sp-4an.1 shipped the substrate - the
+blobs-only adapter behaviour and the guarded facade (ADR-0003) over the
+keys ADR-0002 fixed - but nothing in the package yet knows what a run is,
+calls the interpreter, or executes an effect. This record fixes the
+contracts the loop code will encode: the durable run record, the loop's
+order, the seam through which effects reach a host, and the seam through
+which concurrent deliveries to one run are ordered.
+
+The engine facts this record leans on, verified against the vendored pin
+(`deps/statifier/`, `mix.lock`):
+
+- `Statifier.Interpreter.initialize/2` returns an untagged
+  `{MachineState.t(), [Effect.t()]}` pair and cannot fail
+  (`deps/statifier/lib/statifier/interpreter.ex:259-260`). Creating a run
+  therefore always has a machine state in hand, even when that state is
+  already terminal or budget-exhausted.
+- `Statifier.Interpreter.handle_event/2` returns
+  `{:ok, MachineState.t(), [Effect.t()]} | {:error, :not_running}`, with
+  the `running: false` refusal as the head clause
+  (`interpreter.ex:477-502`). Terminality is a typed refusal at the core,
+  never an exception.
+- `Statifier.Interpreter.deliver_internal/5` is st-ADR-0039's re-entry
+  seam: the one door through which an out-of-loop failure becomes an
+  internal `error.*` event, delegating to the same two internal-queue
+  writers the core's own executable content uses and folding to quiescence
+  (`interpreter.ex:505-545`). This package never constructs `error.*`
+  events by hand.
+- `Statifier.Position.to_binary/1` refuses only `:unidentified_chart` and
+  does not check quiescence; that check belongs to `export/1`
+  (`deps/statifier/lib/statifier/position.ex:105-117, 267-277`).
+  Quiescence before persist is therefore this loop's own assertion, not
+  something upstream enforces for it.
+- st-ADR-0064 makes `from_binary/2` drop `routes` and `invoke_types`
+  unconditionally on decode (`position.ex:164-180`), so re-stamping both
+  on every load is structural, not a convention: the loop can assert the
+  fields arrive `nil` and fail loudly if upstream ever regresses.
+- st-ADR-0054 decision 3's deterministic dedup key
+  (`{scope, send_id, macrostep, microstep, round, c_index, owner,
+  ordinal}`) and st-ADR-0059's `timer_counter` ordinal make at-least-once
+  honest: a re-driven step re-emits effects carrying identical keys, so
+  idempotency can live with the consumer.
+- The interpreter moduledoc's "Rehydrating a position" recipe
+  (`interpreter.ex:43-92`) is this loop's resume spec: `from_binary/2`,
+  then `put_routes/2` + `put_invoke_types/2`, then an advance entry; no
+  `initialize/2` call on the resume path, ever.
+
+Accepted records that bound the design: this repo's ADR-0002 (runs
+vocabulary, engine identities verbatim, surrogate keys are Ecto-layer
+only) and ADR-0003 (blobs-only behaviour, guard in the facade, engine
+identities as the only keys); upstream's st-ADR-0052/0054/0059/0060/0064
+(identity, effect-vocabulary consumption, timer ordinal, resume
+semantics, blob field drops), adopted by reference per ADR-0001.
+
+## Decision
+
+**1. The run record owns its current position.** A run is the durable
+unit: `%{run_id, status, content_hash, identity_blob, position_blob,
+failure}`. Storing the position on the run row (rather than a second
+lookup into the sp-4an.1 position table) makes the persist tail one
+adapter write, makes the per-run lock cover exactly the bytes it
+protects, and matches ADR-0002 decision 4/5's `statifier_runs` sketch.
+The sp-4an.1 chart/position callbacks stand unchanged for hosts
+persisting sessions without the lifecycle. `position_blob` is nullable: a
+run that fails at creation (budget exhaustion during `initialize/2`) has
+no quiescent position to store, and persisting a non-quiescent one is the
+bug the loop exists to prevent. The adapter behaviour gains three
+callbacks - `insert_run/2`, `fetch_run/2`, `update_run/2` - and two error
+arms, `:run_exists` and `:run_not_found`. ADR-0003's blobs-only rule
+binds all three: no callback decodes a blob, validates a status
+transition, or performs an identity check - the facade and the lifecycle
+own those.
+
+**2. Run keys and statuses stay this layer's style: opaque and total.**
+`run_id` is a caller-supplied opaque string stored verbatim - a host
+identity in ADR-0002 decision 1's category, not a surrogate this layer
+generates. `status :: :active | :completed | :failed` and
+`failure :: String.t() | nil` (a short reason; structured detail is not
+portably storable and belongs in host telemetry). Uniqueness is the
+adapter's: `insert_run/2` refuses a duplicate with
+`{:error, :run_exists}`, which is what makes create-exactly-once
+checkable without a lock. The identity guard extends structurally, not by
+convention: the new facade functions mirror sp-4an.1 exactly - writers
+derive `content_hash` and `identity_blob` from the machine state's own
+`Machine.identity/1` (never a caller value) and refuse
+`:unidentified_chart`; the read path (`load_run_position/3`) reuses the
+same identity pre-check and `Position.from_binary/2` as
+`load_position/3`, so ADR-0003 decision 2's claim - no adapter ever holds
+both sides of the guard - stays true for run records too.
+
+**3. The loop's order is the contract.** A step is: liveness check on the
+run record -> load (guarded) -> re-stamp `routes`/`invoke_types`
+unconditionally (with the nil tripwire from st-ADR-0064: the fields are
+pattern-matched `nil` before stamping, so an upstream regression fails
+loudly here, not silently downstream) -> step via
+`Interpreter.handle_event/2` -> execute effects via the executor seam ->
+consume `:done` and `:budget_exhausted` into run status -> assert
+`MachineState.internal_queue_empty?/1` -> persist. At-least-once effect
+execution is a property, not a bug: a crash between step and persist
+re-drives the same event and re-emits the same effects with identical
+deterministic keys (st-ADR-0054 decision 3, st-ADR-0059), and the loop
+never dedupes - idempotency is the consumer's. An event delivered to a
+terminal run is discarded with a typed `{:discarded, run}` result, never
+an exception and never a silent step; the check runs on the run record
+before any position decode, with `handle_event/2`'s `:not_running` arm as
+the structural backstop.
+
+**4. The executor seam is the effect vocabulary and nothing else.** A
+behaviour with one required callback,
+`execute(effect :: Statifier.Effect.t(), context :: map()) ::
+:ok | {:error, term()}`, invoked per effect in list order; an arity-2 fun
+is accepted anywhere a module is. Only the public core effect vocabulary
+crosses the seam - never Session instruction tuples (st-ADR-0054 decision
+1). The loop consumes `:done` and `:budget_exhausted` itself and hands
+everything else over. Failures map by upstream's own classification axis
+(st-ADR-0051's table, st-ADR-0039's seam): the core raises
+`error.execution` itself at planning time before any effect is emitted,
+so every failure an executor can report is a failure to reach or act on
+the outside world after the core accepted the effect - and re-enters
+uniformly as `error.communication` through
+`Interpreter.deliver_internal/5`, for actionable effects of both the
+invoke class (`:invoke`, `:cancel_invoke`, `:autoforward`) and the send
+class (`:send`, `:send_delayed`, `:cancel`). Failures on observational
+effects (`:log`, `:datamodel_*`, trace) are discarded, because
+observation must never steer a run. This package never mints
+`error.execution`. Re-entry is single-wave per step: effects emitted by
+the error re-entries are executed, but their failures are not re-entered
+again - they surface in the returned run's step result - so a
+deterministically failing executor cannot loop the library.
+
+**5. Per-run serialization is a pluggable strategy, not a property of the
+loop.** A behaviour `StatifierPersistence.Serialization` with
+`with_run(config, run_id, fun) :: {:ok, term()} | {:error, term()}`; the
+loop runs its whole load-to-persist tail inside `with_run/3`. The default
+strategy, `StatifierPersistence.Serialization.AdapterLock`, delegates to
+a new optional adapter callback `lock_run/3` (declared like `isolate/1`),
+and refuses with `{:error, {:serialization, :not_supported}}` when the
+adapter does not export it. The Ecto adapter implements `lock_run/3` as a
+row lock (sp-4an.3); a job-queue host later swaps the strategy without
+touching the loop - the ordering guarantee moves, the API does not. That
+no-API-change swap is the acceptance test for this shape.
+
+**6. Completion is chart-driven.** The `:done` effect is the only path to
+`:completed`; there is no public `complete/2`. A host that must end a run
+early has `fail/4` (abandonment with a reason), the only host-driven
+terminal transition, and it involves no interpreter call - abandonment is
+a host decision about the run, not a chart transition.
+
+## Consequences
+
+- What would reopen this record: an effect the lifecycle must consume
+  beyond the two named (`:done`, `:budget_exhausted`); a serialization
+  strategy that cannot express its guarantee as `with_run/3`; upstream
+  moving quiescence enforcement into `to_binary/1`, which would make
+  decision 3's assertion redundant or conflicting. `caller_context`
+  (st-ADR-0063) landing upstream is NOT a reopener: effect structs pass
+  through the seam verbatim, so the field arrives here for free when the
+  pin moves.
+- Deliberately omitted, on the same unexercised-contract reasoning
+  ADR-0003's Consequences records: effect deduplication (at-least-once is
+  the contract and a deduping loop would hide it), run deletion and
+  `delete_position`, position history, and an operator-forced manual
+  complete (a deliberate future API, recorded then, if a real embedder
+  needs it).
+- Durable timers and async invoke execution stop at the seam:
+  `:send_delayed`, `:cancel`, `:invoke`, `:cancel_invoke`, `:autoforward`
+  cross it and scheduling them durably is statifier_oban's charter
+  (st-ADR-0054) or the host's. Resume restores position, not liveness
+  (st-ADR-0060 decision 7).
+- The Ecto adapter (sp-4an.3) inherits three run callbacks, two error
+  arms, and an optional `lock_run/3` to implement as a transaction-scoped
+  row lock, all conformance-tested through the sp-4an.1 suite.

docs/adr/README.md

@@ -5,6 +5,7 @@
 | [0001](0001-record-architecture-decisions.md) | Record architecture decisions | accepted |
 | [0002](0002-configurable-keys-and-table-names.md) | Storage keys and table names are host-configurable at compile time (UXID default, `statifier_` prefix, runs vocabulary); engine identities are not | accepted |
 | [0003](0003-storage-adapter-behaviour-and-the-identity-guard.md) | The storage adapter stores opaque blobs keyed by engine identities; the identity guard lives above every adapter and cannot be skipped | accepted |
+| [0004](0004-run-lifecycle-executor-seam-and-serialization.md) | The run record owns its position, the loop's order is the contract, effects cross a host-executor seam (failures re-enter as `error.communication`), and per-run serialization is a pluggable strategy | accepted |
 
 New ADRs: next number, same three-section format (Context, Decision,
 Consequences). Pick the number against a freshly fetched remote. A bare