Adds a restart-surviving demo host

README.md

@@ -61,6 +61,17 @@ which brings up `postgres:17` on `localhost:5432` with user/password
 `config/test.exs` for the defaults) if a server is already running
 elsewhere.
 
+## Surviving a restart
+
+`docs/restart-demo.md` walks through the demo embedder that drives this
+package's whole surface across a simulated restart with no Session
+process: persist mid-run with a pending durable timer and an in-flight
+async invocation, drop everything volatile, cold-boot from the run id
+alone, and finish with zero duplicate side effects and a replay that
+reproduces the path. The executable version lives in
+`test/statifier_persistence/demo/restart_demo_test.exs` (and its
+Postgres variant beside it).
+
 ## The contract this package builds on
 
 The persisted-position story is already specified upstream, and this package

docs/adr/0004-run-lifecycle-executor-seam-and-serialization.md

@@ -196,3 +196,24 @@ exit from `fun` (a raise included) releases them with the transaction.
 The rowless hole and the fix are pinned by a live two-connection test
 outside the SQL sandbox, whose single shared connection would otherwise
 serialize the callers by ownership and mask a broken lock.
+
+## Validation note (2026-08-22, sp-4an.4): driven end to end by a demo embedder
+
+Decisions 3, 4 and 6 were exercised end to end by a demo embedder
+(`test/statifier_persistence/demo/`, walkthrough in
+`docs/restart-demo.md`) running a multi-step chart across a simulated
+restart with no Session process - persist mid-run with a pending durable
+timer and an in-flight async invocation, drop everything volatile, boot
+from the run id alone, recover, finish - over both adapters, with the
+executor call log asserted on exact contents and a replay reproducing
+the path struct for struct. Not an amendment: nothing decided here
+changes.
+
+One finding about the API surface: a byte-identical replay needs the
+session id, the one input `Runs.create/4` otherwise generates fresh
+(`MachineState.new/2` stamps it into the `:datamodel_init` effect's
+`_sessionid`/`_ioprocessors` system variables). The existing
+`initialize: [session_id: ...]` pass-through already covers it - no new
+surface needed - but a host that wants replayable runs must record that
+id alongside its input tape, which `docs/restart-demo.md` now says out
+loud.

docs/restart-demo.md

@@ -0,0 +1,105 @@
+# Surviving a restart: the demo host
+
+`test/statifier_persistence/demo/` is a demo embedder that runs a
+multi-step chart across a simulated restart with no `Statifier.Session`
+process at all: persist mid-run, drop every volatile process and struct,
+cold-boot from the run id alone, continue, finish. It exists to validate
+this package's whole surface - the identity guard (ADR-0003), the run
+lifecycle and executor seam (ADR-0004), and both storage adapters - the
+way the charter demands: driven by an embedder-shaped pipeline, not by
+unit tests alone.
+
+Every claim below is asserted by a test; when the prose and the tests
+disagree, the tests win.
+
+- `test/statifier_persistence/demo/restart_demo_test.exs` - the scenario
+  over `Storage.InMemory`, with the full per-stage assertions.
+- `test/statifier_persistence/demo/restart_demo_ecto_test.exs` - the same
+  scenario bodies over `Storage.Ecto` and real Postgres.
+- `test/support/demo/` - the host the tests drive: `Host`, `Ledger`,
+  `Runtime`, `EnrichHandler`, `Scenario`.
+
+## The chart and the kill point
+
+The chart (`test/support/demo/scenario.ex`) walks
+`intake -> enriching -> cooling -> settling -> settled`, with an
+`escalated` state as the negative target - reaching it means the host
+lost a race it exists to control, and the tests assert it is never
+entered.
+
+The kill point is `enriching`, chosen so the restart happens with the
+maximum in flight:
+
+- a **pending durable timer** - `enriching`'s onentry armed a 900s
+  `sla-timer`;
+- an **in-flight async invocation** - the `myapp:enrich` invoke started a
+  real worker process.
+
+At that moment the volatile runtime is stopped - every worker pid and
+armed in-memory timer dies with it - and a fresh host is booted from
+nothing but the run id.
+
+## What survives, and what the host must rebuild
+
+| Piece | Lives in | Survives the restart |
+|---|---|---|
+| the run's position and status | `StatifierPersistence.Storage` (InMemory / Postgres) | yes |
+| the chart blob | `StatifierPersistence.Storage` | yes |
+| the host's own timer and invocation rows | `Demo.Ledger` (stands in for the embedder's tables) | yes |
+| armed timers, live worker pids | `Demo.Runtime` (supervisor over volatile state) | no - stopped and rebuilt |
+| the `%Host{}` struct | plain struct | no - rebuilt by `boot/4` |
+| the compiled `%Machine{}` | recompiled from the stored chart blob | no - `Chart.from_binary/1` on boot |
+| the handler palette (`invoke_types`, `invoke_handlers`) | per-deployment declaration | no - re-supplied on every step (st-ADR-0064) |
+
+st-ADR-0060's rule is "resume restores position, not liveness", and the
+demo makes each half of that visible:
+
+1. **Position restores.** `boot/4` re-reads the run record and the chart
+   blob, recompiles a freshly interned machine, and the next step's
+   identity guard proves the pair still match. The restart tests assert
+   the boot really re-read stored bytes rather than reusing a carried
+   struct.
+2. **Timers do not.** Nothing about a pending delayed send survives in
+   the position - `delay_ms` is relative and no wall-clock instant is
+   stored. `recover/1` re-arms every open timer from the host's own
+   durable rows; the `sla-timer` that fires after the restart was armed
+   before it, and fires from that durable row.
+3. **Invocations do not.** The position's `active_invocations` records
+   *what was invoked*, never a pid. `recover/1` re-establishes each
+   invocation the engine still considers active and the ledger still
+   shows open, by re-running the ADR-0051 handler's `start/2` - the
+   engine is the liveness authority, the host's rows are the payload
+   source. The tests assert the post-restart worker is a live pid
+   different from the dead pre-restart one.
+4. **Nothing runs twice.** Re-arming and re-establishing are idempotent
+   on their durable keys (`{run_id, ordinal}` for timers, `invoke_id`
+   for invocations), and recovery goes through the host's own hands, not
+   back through the executor seam - so the executor call log after the
+   restarted run is exactly the straight-through run's, asserted on
+   exact contents.
+5. **The inputs are the host's to record.** `boot/4` restores no input
+   tape; the recorder carries it. A replay of the recorded events -
+   plus the one generated create input, the session id - against a
+   fresh store reproduces the identical configuration sequence and the
+   identical effect structs, field for field.
+
+## Where every effect goes
+
+Nothing the host does to the outside world happens outside the
+`StatifierPersistence.Executor` seam: one arity-2 fun
+(`Demo.Host.executor/1`) receives every effect, records it, and
+dispatches it - durable row first, volatile arm/spawn second. The two
+deliberate exceptions are cold-boot obligations, not chart effects:
+`boot/4`'s chart-fetched marker and `recover/1`'s re-establishment,
+both of which exist precisely because no step emits them.
+
+## The boundary
+
+The durable timer store here is the *host's own* (`Demo.Ledger`), and
+the mock clock is the demo's one departure from a production host - the
+gate must stay fast and deterministic. A production embedder hands the
+same `SendDelayed`/`Cancel` effects to a real scheduler instead; that is
+statifier_oban's charter (st-ADR-0054). The seam is identical either
+way, which is the point: this package stops at the effect vocabulary,
+and everything the demo rebuilds by hand is exactly the work a durable
+scheduler package takes on.

test/statifier_persistence/demo/restart_demo_ecto_test.exs

@@ -0,0 +1,142 @@
+defmodule StatifierPersistence.Demo.RestartDemoEctoTest do
+  @moduledoc """
+  The demo scenarios re-run against `StatifierPersistence.Storage.Ecto`
+  over real Postgres (the ADR-0005 harness), so the demo proves the loop,
+  not the `InMemory` adapter (Phase 4 of
+  `docs/plans/260822-sp-4an.4-restart-demo-host.md`).
+
+  The per-stage assertions live in `StatifierPersistence.Demo.RestartDemoTest`;
+  this variant drives the identical scenario bodies (`Scenario` names no
+  storage module) and asserts the same outcomes, plus the one thing only
+  this variant can prove - that the post-restart boot re-read the chart
+  and the run from the database, under the default `AdapterLock`
+  serialization's advisory-plus-row lock on every step.
+
+  `async: false` per the plan: the demo drives several `Storage.new/2`
+  handles in one test over a single sandbox-checked-out connection.
+  """
+  use ExUnit.Case, async: false
+
+  alias Statifier.Effect.{Cancel, CancelInvoke, DatamodelInit, Invoke, SendDelayed}
+  alias StatifierPersistence.Demo.{Host, Ledger, Scenario}
+  alias StatifierPersistence.EctoHosts
+  alias StatifierPersistence.Run
+  alias StatifierPersistence.Storage
+
+  @adapter Storage.Ecto
+  @adapter_opts [persistence: EctoHosts.Default, sandbox: true]
+
+  setup do
+    # The conformance suite's shape: normalize the opts through
+    # `Storage.new/2`, then check this test process out its own sandboxed
+    # connection/transaction.
+    {:ok, store} = Storage.new(@adapter, @adapter_opts)
+    :ok = @adapter.isolate(store.opts)
+    :ok
+  end
+
+  # The scenario bodies and the mutations are shared with the InMemory
+  # variant (`RestartDemoTest`); the plan's Testing Strategy table records
+  # that Phase 4 reuses those notes rather than adding new mutations. The
+  # per-test notes below name which shared mutation reds each test.
+
+  # sabotage: shared with RestartDemoTest's straight-through test -
+  # Runs.run_status/2 returning :active for a :done machine state reds the
+  # `status: :completed` assertion here identically (same scenario body).
+  # Run and confirmed red on the InMemory variant, reverted.
+  test "drives the chart straight through over Postgres" do
+    result = Scenario.straight_through({@adapter, @adapter_opts})
+
+    assert result.configs == [["enriching"], ["cooling"], ["settling"]]
+    refute Enum.any?(result.configs, &("escalated" in &1))
+    assert %Run{status: :completed} = Host.run(result.host)
+
+    calls = result.ledger |> Ledger.calls() |> Enum.map(fn {effect, _context} -> effect end)
+
+    assert [
+             {:datamodel_init, %DatamodelInit{}},
+             {:send_delayed, %SendDelayed{send_id: "sla-timer", event: "sla.breach"}},
+             {:invoke, %Invoke{type: "myapp:enrich", invoke_id: invoke_id}},
+             {:cancel_invoke, %CancelInvoke{invoke_id: invoke_id}},
+             {:send_delayed, %SendDelayed{send_id: "reminder-timer", event: "reminder"}},
+             {:cancel, %Cancel{send_id: "reminder-timer"}}
+           ] = calls
+  end
+
+  # sabotage: shared with RestartDemoTest's restart test - Runs.write_run/6
+  # passing position: :skip on the :update path leaves the stored blob in
+  # intake, so `config_at_kill == ["enriching"]` reds here identically
+  # (same scenario body). Run and confirmed red on the InMemory variant,
+  # reverted.
+  test "resumes from a simulated restart over Postgres" do
+    result = Scenario.across_restart({@adapter, @adapter_opts})
+
+    # Kill point, node death, and re-established liveness - same claims
+    # the InMemory variant pins in full.
+    assert result.config_at_kill == ["enriching"]
+    refute result.alive_after_stop
+    assert is_pid(result.pid_after_recover)
+    refute result.pid_after_recover == result.pid_before
+
+    # The post-restart boot genuinely re-read from Postgres: the
+    # `{:chart_fetched, _}` marker is present, and the freshly recompiled
+    # machine's identity matches the stored run record's - this is the
+    # plan's "confirm `boot/4` issues a `fetch_run` and a `fetch_chart`"
+    # check, asserted rather than observed by hand.
+    assert {:chart_fetched, content_hash} =
+             result.ledger
+             |> Ledger.side_effects()
+             |> Enum.find(&match?({:chart_fetched, _}, &1))
+
+    assert content_hash == result.host_after_boot.machine.identity.content_hash
+    assert %Run{content_hash: ^content_hash} = Host.run(result.host_after_boot)
+
+    # The tail finishes with the exact same executor call log - nothing
+    # re-emitted across the restart, now with every step's write behind
+    # `Storage.Ecto.lock_run/3`.
+    assert result.configs == [["cooling"], ["settling"]]
+    assert %Run{status: :completed} = Host.run(result.host)
+
+    calls = result.ledger |> Ledger.calls() |> Enum.map(fn {effect, _context} -> effect end)
+
+    assert [
+             {:datamodel_init, %DatamodelInit{}},
+             {:send_delayed, %SendDelayed{send_id: "sla-timer", event: "sla.breach"}},
+             {:invoke, %Invoke{type: "myapp:enrich", invoke_id: invoke_id}},
+             {:cancel_invoke, %CancelInvoke{invoke_id: invoke_id}},
+             {:send_delayed, %SendDelayed{send_id: "reminder-timer", event: "reminder"}},
+             {:cancel, %Cancel{send_id: "reminder-timer"}}
+           ] = calls
+  end
+
+  # sabotage: shared with RestartDemoTest's replay test - the same
+  # write_run position: :skip mutation diverges the loaded original
+  # configs from the replay's returned ones, redding the sequence
+  # comparison here identically (same scenario body and comparison). Run
+  # and confirmed red on the InMemory variant, reverted.
+  test "replays the recorded tape over Postgres" do
+    result = Scenario.across_restart({@adapter, @adapter_opts})
+
+    {:ok, replay_store} = Storage.new(@adapter, @adapter_opts)
+    {:ok, replay_ledger} = Ledger.start_link([])
+    replay_run_id = "restart-demo-replay-#{System.unique_integer([:positive])}"
+
+    # Same recorded-inputs discipline as the InMemory variant: the session
+    # id is the one generated create input, re-supplied to the replay.
+    {:ok, final_position} = Host.position(result.host)
+    original_session_id = final_position.datamodel["_sessionid"]
+
+    replay_result =
+      Scenario.replay(Host.tape(result.host), {replay_store, replay_ledger}, replay_run_id,
+        initialize: [session_id: original_session_id]
+      )
+
+    original_configs = [result.config_at_kill] ++ result.configs ++ [Host.config(result.host)]
+    assert Enum.drop(replay_result.configs, 1) == original_configs
+
+    original_effects =
+      result.ledger |> Ledger.calls() |> Enum.map(fn {effect, _context} -> effect end)
+
+    assert replay_result.effects == original_effects
+  end
+end