Serializes datamodel_change as a wire type

changelog.d/sui-h92.md

@@ -0,0 +1,7 @@
+### Added
+
+- Serializes statifier's `DatamodelChange` effect as the new
+  `effect.datamodel_change` wire type, so consumers can observe datamodel
+  values as they are written instead of only the variable names
+  `session.datamodel` carries. The format version stays 1; new types are
+  additive under the wire format's must-ignore rule.

docs/wire-format.md

@@ -20,7 +20,7 @@ interpreter's author needs first.
 - MUST, when tracing is enabled, emit the nine `trace.*` types at the phase
   boundaries Appendix D names, with the envelope fields this document
   requires.
-- MAY emit further message families (this document also specifies nine
+- MAY emit further message families (this document also specifies ten
   `effect.*` types and four `session.*` lifecycle types) and MAY add fields
   to any message beyond what this document requires.
 
@@ -503,10 +503,14 @@ pass, including when nothing was finalized or forwarded.
 | finalized | array of strings | always - `invoke_id`s of every invocation `<finalize>` ran for |
 | forwarded | array of strings | always - `invoke_id`s of every invocation `event` was autoforwarded to, in the pass's own walk order |
 
-## The nine `effect.*` schemas
+## The ten `effect.*` schemas
 
-The nine core (non-trace) statifier effects, mapped one-to-one onto their
-own namespace. ADR-0005 leaves non-trace effect naming to this document
+The core (non-trace) statifier effects, mapped one-to-one onto their
+own namespace: the nine originals plus `effect.datamodel_change`
+(`Statifier.Effect.DatamodelChange`, `st-oef3`). The engine's eleventh
+core effect, `Statifier.Effect.DatamodelInit`, is the one exception to
+the one-to-one mapping - it serializes as `session.datamodel`, below.
+ADR-0005 leaves non-trace effect naming to this document
 ("their own namespaces as consumers need them"); this document uses one
 `effect.*` family rather than several separate top-level namespaces,
 because a bare top-level `done` type would sit confusingly next to
@@ -622,6 +626,60 @@ Payload for `<cancel>` (spec 6.3).
 | c_index | integer | present only when known |
 | owner | owner object | present only when known |
 
+### `effect.datamodel_change`
+
+Emitted once per successful datamodel write (`st-oef3`): an `<assign>`, a
+`<data>` binding during the binding fold, a `<send idlocation>` write, an
+`<invoke idlocation>` write, or an empty-`<finalize>` auto-assign. A failed
+write emits nothing - the datamodel did not change, and the failure is
+already on the error channel. Together with `session.datamodel`'s starting
+snapshot, the sequence of these messages reconstructs the datamodel at any
+point in the run from the stream alone.
+
+| Field | Type | Presence |
+|---|---|---|
+| location_path | array of strings and integers | always - the resolved write path; see below |
+| location_source | string | always - the raw author string that named the location (`items[i].name` as written) |
+| new_value | value | present unless the write stored the unbound sentinel; a stored null is present as `null` (three-way absence rule) |
+| prior_value | value | present unless nothing stood at the path before the write; a previously stored null is present as `null` (three-way absence rule) |
+| d_index | integer | present only for a `<data>` binding - resolves through `session.start`'s `data` table |
+| c_index | integer | present only when a content node performed the write (`<assign>`, `<send idlocation>`) |
+| owner | owner object | present only when known - which construct performed the write |
+
+**`location_path` is a heterogeneous JSON array.** Each segment is either a
+string - an object key, the variable name first - or an integer, a 0-based
+array index: `user.items[0].name` resolved against `i = 0` arrives as
+`["user", "items", 0, "name"]`. JSON's own typing carries the distinction -
+the string `"0"` is a key, the number `0` is an index - so no tagging or
+escaping is needed, and a consumer applies the segments in order to its own
+copy of the datamodel to reproduce the write. It is the *resolved* path:
+an index expression like `[i]` has already been evaluated by the engine,
+which is what makes the path applicable without the pre-assignment
+datamodel. `location_source` is the raw author string kept alongside for
+display; neither substitutes for the other.
+
+**`new_value` and `prior_value` follow the `_event.data` three-way rule**,
+because the engine genuinely distinguishes unbound from null here
+(statifier ADR-0037 spells unbound as `:undefined`): key absence means the
+slot was or became unbound - for `prior_value`, that nothing stood at the
+path before the write, the common case for a first assignment - while JSON
+`null` means a genuinely stored null.
+
+**`d_index` and `c_index` are mutually exclusive** on this message: a
+`d_index` means the write was a `<data>` binding, which belongs to no
+content block and therefore also carries no `owner`; a `c_index` means a
+content node performed it. The two runner-side writes - the
+empty-`<finalize>` auto-assign and `<invoke idlocation>` - carry neither,
+identified by `owner` alone (`"finalize"` and `"invoke"` kinds
+respectively).
+
+**Versioning decision, recorded rather than left implicit:** this type
+joined the format after version 1 shipped with 23 types. Adding a type is
+exactly what the conformance section's MUST-ignore rule makes additive, so
+**the format version stays 1** - unlike `session.datamodel`, which kept
+version 1 because its type string was already reserved, this one keeps it
+because new types never bump the version at all.
+
 ## Origins
 
 `Statifier.Event.Cause.origin/0`'s eight variants, each a tagged object
@@ -638,17 +696,18 @@ with `"kind"` naming the tuple's tag:
 | "invoke" | state_index (integer), invoke_index (integer) | one of an `<invoke>`'s own arguments failed to evaluate |
 | "finalize" | state_index (integer), invoke_index (integer) | an empty `<finalize>`'s own auto-assign write failed |
 
-A `d_index` anywhere in this format - here or on a future
-`Effect.DatamodelChange` message - resolves through `session.start`'s
+A `d_index` anywhere in this format - here or on
+`effect.datamodel_change` - resolves through `session.start`'s
 `data` table, the same way a `state_index` resolves through `states` and
 a `t_index` through `transitions`.
 
 ## Owners
 
-`Statifier.Machine.Content.owner/0`'s four variants, plus the one case
+`Statifier.Machine.Content.owner/0`'s four variants, plus the case
 `Statifier.Effect.Trace.ContentExecuted` widens it with for a top-level
-`<script>`, together the five variants an `owner` field may take anywhere
-in this format:
+`<script>` and the case `Statifier.Effect.DatamodelChange` widens it with
+for `<invoke idlocation>`, together the six variants an `owner` field may
+take anywhere in this format:
 
 | "kind" | Fields | Meaning |
 |---|---|---|
@@ -657,6 +716,7 @@ in this format:
 | "transition" | t_index (integer) | a transition's own executable content |
 | "finalize" | state_index (integer), invoke_index (integer) | an `<invoke>`'s own `<finalize>` block |
 | "global_script" | index (integer) | a top-level `<script>`, run at load time - only ever appears as `trace.content_executed`'s owner, since a top-level script belongs to no `<onentry>`/`<onexit>`/transition block |
+| "invoke" | state_index (integer), invoke_index (integer) | an `<invoke idlocation>` write - only ever appears as `effect.datamodel_change`'s owner, since the write belongs to no content block |
 
 ## The `session.*` types
 
@@ -688,10 +748,10 @@ as `{"$undefined": true}`, not only the ones declared without a value: a
 `<data id="count" expr="41 + 1"/>` is present under `count` as
 `{"$undefined": true}` here, and is assigned its `42` afterwards. This
 message therefore names the datamodel's variables reliably, but is not a
-source for their values. The assignments that follow arrive as statifier's
-`Effect.DatamodelChange`, which this producer does not serialize yet;
-`sui-h92` owns that, and until it lands a consumer can observe the
-variable names but never a value.
+source for their values. The assignments that follow arrive as
+`effect.datamodel_change` messages (`sui-h92`), one per successful write,
+binding fold included - a consumer folds them over this snapshot to hold
+the current datamodel at any point in the run.
 
 This message keys the datamodel by variable name; `session.start`'s
 `data` table keys the same elements by `d_index` and carries the `id`
@@ -808,7 +868,7 @@ would show it in the shape its schema above describes.
 
 ## Type index
 
-One row per type this document defines - 23 rows: 9 `trace.*`, 9
+One row per type this document defines - 24 rows: 9 `trace.*`, 10
 `effect.*`, and 5 `session.*` (the four lifecycle types plus
 `session.datamodel`). This table's first column is a machine boundary: a
 drift test parses exactly this table's backtick-quoted `type` strings and
@@ -836,6 +896,7 @@ that test rather than drifting silently.
 | `effect.send` | effect | `<send>` fires immediately |
 | `effect.send_delayed` | effect | `<send>` with a delay is scheduled |
 | `effect.cancel` | effect | `<cancel>` runs |
+| `effect.datamodel_change` | effect | a datamodel location is successfully written |
 | `session.start` | session | a session's stream opens |
 | `session.halted` | session | the session reports `{:halted, reason}` |
 | `session.terminated` | session | the session's process exits |
@@ -858,8 +919,9 @@ that test rather than drifting silently.
   above, closed by ADR-0044.
 - `st-1xwh` - the upstream effect (`Statifier.Effect.DatamodelInit`) behind
   `session.datamodel`, above.
-- `st-oef3` - the upstream gap behind `Statifier.Effect.DatamodelChange`
-  (a `<data>`/`<assign>` value change, distinct from `session.datamodel`'s
-  one-time starting snapshot) having no wire representation yet.
+- `st-oef3` - the upstream effect (`Statifier.Effect.DatamodelChange`)
+  behind `effect.datamodel_change` - a per-write value change, distinct
+  from `session.datamodel`'s one-time starting snapshot; serialized here
+  by `sui-h92`.
 - `sui-qay` - the gap behind `session.start`'s location tables carrying no
   attribute-level spans.

lib/statifier_ui/trace/normalizer.ex

@@ -10,7 +10,8 @@ defmodule StatifierUI.Trace.Normalizer do
 
   Anything from the engine that occupies a *value* position -
   `Event.data`, `Log.value`, `Trace.Done.donedata`, `Effect.Done.donedata`,
-  `Send.data`, `SendDelayed.data`, `Invoke.params`, `Invoke.content` - goes
+  `Send.data`, `SendDelayed.data`, `Invoke.params`, `Invoke.content`,
+  `DatamodelChange.new_value`, `DatamodelChange.prior_value` - goes
   through `StatifierUI.Value.encode/1`; its `{:error, _}` is propagated,
   never rescued. Everything structural is reduced first (decision 6):
   `MapSet`s to sorted lists (`configuration/1`), atoms to `"kind"`-tagged
@@ -25,7 +26,11 @@ defmodule StatifierUI.Trace.Normalizer do
 
   `_event.data` keeps ADR-0005's three-way rule unchanged: `:undefined`
   omits the key, `nil` is present as JSON `null`, `%{}` is present as `{}`
-  (`put_event_data/2`). Every other nullable field in this vocabulary has no
+  (`put_defined/3`). `DatamodelChange.new_value` and
+  `DatamodelChange.prior_value` follow the same three-way rule, because they
+  share `_event.data`'s property: the engine spells "unbound" as
+  `:undefined` there (ADR-0037), so `nil` genuinely means a stored null.
+  Every other nullable field in this vocabulary has no
   engine-side way to distinguish "no value" from "a genuinely null value" -
   `Statifier.Effect.Log.value` and the two `donedata` fields default to
   plain `nil` for "nothing here", never `:undefined` - so `put_present/3`
@@ -38,6 +43,7 @@ defmodule StatifierUI.Trace.Normalizer do
   alias Statifier.Effect.BudgetExhausted
   alias Statifier.Effect.Cancel
   alias Statifier.Effect.CancelInvoke
+  alias Statifier.Effect.DatamodelChange
   alias Statifier.Effect.DatamodelInit
   alias Statifier.Effect.Done
   alias Statifier.Effect.Invoke
@@ -77,6 +83,7 @@ defmodule StatifierUI.Trace.Normalizer do
     "trace.invoke_pass",
     "trace.finalize_autoforward",
     "effect.log",
+    "effect.datamodel_change",
     "effect.done",
     "effect.budget_exhausted",
     "effect.invoke",
@@ -94,11 +101,12 @@ defmodule StatifierUI.Trace.Normalizer do
 
   @doc """
   The closed, sorted list of every `type` string this format defines - the
-  vocabulary's single definition site in code. 23 entries: 9 `trace.*`, 9
-  `effect.*`, and 5 `session.*` (the four lifecycle types plus
-  `session.datamodel`, emitted once per session from
+  vocabulary's single definition site in code. 24 entries: 9 `trace.*`, 10
+  `effect.*` (the nine core effects plus `effect.datamodel_change`, from
+  `Statifier.Effect.DatamodelChange`), and 5 `session.*` (the four
+  lifecycle types plus `session.datamodel`, emitted once per session from
   `Statifier.Effect.DatamodelInit`). `docs/wire-format.md`'s type index
-  table is the same 23, and `test/statifier_ui/trace/wire_format_spec_test.exs`
+  table is the same 24, and `test/statifier_ui/trace/wire_format_spec_test.exs`
   asserts the two sets are equal.
   """
   @spec types() :: [String.t()]
@@ -177,6 +185,7 @@ defmodule StatifierUI.Trace.Normalizer do
   defp decompose({:send, payload}), do: core_message(payload)
   defp decompose({:send_delayed, payload}), do: core_message(payload)
   defp decompose({:cancel, payload}), do: core_message(payload)
+  defp decompose({:datamodel_change, payload}), do: core_message(payload)
   defp decompose({:datamodel_init, payload}), do: datamodel_message(payload)
   defp decompose({tag, _payload}), do: {:error, {:unknown_effect, tag}}
   defp decompose(other), do: {:error, {:unknown_effect, other}}
@@ -243,9 +252,26 @@ defmodule StatifierUI.Trace.Normalizer do
 
   defp trace_message(payload), do: {:error, {:unknown_effect, {:trace, payload.__struct__}}}
 
-  # -- The nine effect.* payloads --------------------------------------------
+  # -- The ten effect.* payloads ----------------------------------------------
 
   @spec core_message(struct()) :: {:ok, decomposed()} | {:error, term()}
+  defp core_message(%DatamodelChange{} = p) do
+    # `location_path` is emitted structurally, not through
+    # `StatifierUI.Value.encode/1`: its segments are only strings (object
+    # keys) and integers (array indexes), both JSON-native, and the path is
+    # an identity like `c_index`, not a value.
+    base =
+      %{"location_path" => p.location_path, "location_source" => p.location_source}
+      |> put_present("d_index", p.d_index)
+      |> put_present("c_index", p.c_index)
+
+    with {:ok, base} <- put_owner(base, p.owner),
+         {:ok, base} <- put_defined(base, "new_value", p.new_value),
+         {:ok, base} <- put_defined(base, "prior_value", p.prior_value) do
+      {:ok, {"effect.datamodel_change", p.macrostep, p.microstep, nil, base}}
+    end
+  end
+
   defp core_message(%Log{} = p) do
     base = put_present(%{}, "label", p.label)
 
@@ -389,7 +415,7 @@ defmodule StatifierUI.Trace.Normalizer do
   defp event(%Event{} = ev) do
     base = %{"name" => ev.name, "type" => Atom.to_string(ev.type)}
 
-    with {:ok, base} <- put_event_data(base, ev.data),
+    with {:ok, base} <- put_defined(base, "data", ev.data),
          {:ok, base} <- put_cause(base, ev.cause) do
       base =
         base
@@ -469,7 +495,11 @@ defmodule StatifierUI.Trace.Normalizer do
     {:ok, %{"kind" => "finalize", "state_index" => state_index, "invoke_index" => invoke_index}}
   end
 
-  @spec owner(Statifier.Machine.Content.owner() | Trace.ContentExecuted.owner()) ::
+  @spec owner(
+          Statifier.Machine.Content.owner()
+          | Trace.ContentExecuted.owner()
+          | DatamodelChange.owner()
+        ) ::
           {:ok, map()} | {:error, term()}
   defp owner({:onentry, state_index, ordinal}) do
     {:ok, %{"kind" => "onentry", "state_index" => state_index, "ordinal" => ordinal}}
@@ -491,7 +521,11 @@ defmodule StatifierUI.Trace.Normalizer do
     {:ok, %{"kind" => "global_script", "index" => index}}
   end
 
-  @spec put_owner(map(), Statifier.Machine.Content.owner() | nil) ::
+  defp owner({:invoke, state_index, invoke_index}) do
+    {:ok, %{"kind" => "invoke", "state_index" => state_index, "invoke_index" => invoke_index}}
+  end
+
+  @spec put_owner(map(), Statifier.Machine.Content.owner() | DatamodelChange.owner() | nil) ::
           {:ok, map()} | {:error, term()}
   defp put_owner(map, nil), do: {:ok, map}
 
@@ -515,16 +549,18 @@ defmodule StatifierUI.Trace.Normalizer do
 
   # -- Absence ----------------------------------------------------------------
 
-  # `_event.data`'s three-way rule (decision 5, unchanged): `:undefined`
-  # omits the key, everything else - including `nil` - goes through
+  # The three-way rule (decision 5, unchanged): `:undefined` omits the key,
+  # everything else - including `nil` - goes through
   # `StatifierUI.Value.encode/1` and is kept, even when the encoded result is
-  # JSON `null`.
-  @spec put_event_data(map(), term()) :: {:ok, map()} | {:error, term()}
-  defp put_event_data(map, :undefined), do: {:ok, map}
+  # JSON `null`. Applies where the engine genuinely distinguishes unbound
+  # (`:undefined`) from a stored null (`nil`): `_event.data`, and
+  # `DatamodelChange`'s `new_value`/`prior_value`.
+  @spec put_defined(map(), String.t(), term()) :: {:ok, map()} | {:error, term()}
+  defp put_defined(map, _key, :undefined), do: {:ok, map}
 
-  defp put_event_data(map, data) do
-    with {:ok, encoded} <- Value.encode(data) do
-      {:ok, Map.put(map, "data", encoded)}
+  defp put_defined(map, key, value) do
+    with {:ok, encoded} <- Value.encode(value) do
+      {:ok, Map.put(map, key, encoded)}
     end
   end