docs(orchestrator): document controller correctness

submitqueue/orchestrator/controller/README.md

@@ -0,0 +1,213 @@
+# Controller Correctness Under At-Least-Once Delivery
+
+SubmitQueue controllers run on an at-least-once queue. A delivery may be repeated after any completed database write or queue publish, including after the controller returned from an external dependency but before it recorded the result.
+
+The core mental model is:
+
+> A message is a request to reconcile durable state toward a postcondition, not a command that runs exactly once.
+
+## What durable state tells us
+
+An entity state records domain progress. It does not prove that the controller's complete fanout was published or processed.
+
+For example, `BatchStateScored` proves that the score and state transition were persisted together. It does not prove that request logs and the `speculate` message were published.
+
+Therefore:
+
+```text
+durable state identifies which transition occurred
+fanout completion is unknown
+redelivery repairs fanout by replaying it idempotently
+```
+
+## Controller reconciliation loop
+
+Every controller should classify the latest durable state before writing:
+
+```text
+Process(message):
+    entity = load latest durable state
+    progress = classify(message, entity)
+
+    switch progress:
+
+    case NeedsTransition:
+        result = compute transition from entity
+
+        if CAS(expected version, new state and result) loses:
+            reload and classify again
+
+        entity = locally apply the committed transition
+        continue to fanout
+
+    case TransitionAppliedNeedsFanout:
+        skip the state write
+        continue to fanout
+
+    case DownstreamAlreadyProgressed:
+        return success
+
+    case TransitionSuperseded:
+        return success or invoke the designated recovery owner
+
+    case InvalidState:
+        return error
+
+    outputs = derive the complete required fanout from durable data
+
+    for output in stable order:
+        publish output with a stable logical message ID
+        if publish fails:
+            return error
+
+    return success
+```
+
+The progress classifications mean:
+
+| Classification | Meaning |
+|---|---|
+| `NeedsTransition` | This controller's durable state change has not happened. |
+| `TransitionAppliedNeedsFanout` | The state change is durable, but fanout may be partial or absent. |
+| `DownstreamAlreadyProgressed` | Durable state proves a downstream stage consumed the handoff. |
+| `TransitionSuperseded` | Cancellation, failure, or another outcome made this transition unnecessary. |
+| `InvalidState` | The state violates the controller's lifecycle contract. |
+
+## Persist before publishing
+
+For local state transitions followed by queue fanout:
+
+```text
+persist state
+publish complete fanout
+ack delivery
+```
+
+Do not acknowledge after the state write but before fanout completes.
+
+If the process crashes after the write, redelivery observes the target state, skips the write, and republishes the complete fanout.
+
+## Replay the complete fanout
+
+Controllers generally do not query which individual output messages were published. They assume fanout may be incomplete and replay the entire required set.
+
+```text
+previous attempt:
+    output A published
+    output B published
+    output C missing
+
+redelivery:
+    output A duplicate, no-op
+    output B duplicate, no-op
+    output C inserted
+```
+
+This converges because queue messages have stable identities. Repeated publication must use the same topic, partition key, message ID, and logical payload.
+
+If a payload changes by entity version, operation generation, or result revision, that value must be part of the message ID. Reusing an ID for different payloads is incorrect because an existing queue message remains authoritative.
+
+## Use CAS for transitions, not validation
+
+Optimistic locking answers:
+
+> Has this row changed since I read it?
+
+It does not answer:
+
+> Is this lifecycle transition valid?
+
+A controller must validate the current state before writing. Otherwise a redelivery can load a later state and successfully regress it using the latest version:
+
+```text
+database: Speculating v3
+redelivery reads Speculating v3
+CAS writes Scored v4
+```
+
+The controller should write only from states it owns:
+
+```text
+Created -> Scored
+```
+
+The current storage contract guards the expected version. On a version conflict, reload before deciding whether the transition is still needed, already completed, or superseded.
+
+Version arithmetic follows the [storage optimistic-locking contract](../../extension/storage/README.md): compute the new version in the controller and update the in-memory entity only after the write succeeds.
+
+## Example: score
+
+The score controller's state classification should be explicit:
+
+| Batch state | Classification | Behavior |
+|---|---|---|
+| `Created` | `NeedsTransition` | Compute the score and CAS to `Scored`. |
+| `Scored` | `TransitionAppliedNeedsFanout` | Preserve the score and republish logs plus `speculate`. |
+| `Speculating`, `Merging` | `DownstreamAlreadyProgressed` | Acknowledge without regressing the batch. |
+| `Cancelling`, terminal | `TransitionSuperseded` | Follow cancellation or terminal-state ownership. |
+
+If publishing logs fails after the state reaches `Scored`, the controller returns an error. Redelivery reloads `Scored`, skips rescoring, republishes all logs, and then republishes `speculate`. Existing messages deduplicate and missing messages are inserted.
+
+## Terminal and superseded states
+
+Do not treat every terminal state as proof that this controller completed.
+
+For each terminal or superseding state, document which component owns any remaining fanout or cleanup. A controller may return immediately only when:
+
+- Its work is no longer semantically required, or
+- Another controller explicitly owns reconciliation.
+
+The [DLQ reconciliation controllers](dlq/README.md) demonstrate this distinction: an already-failed request skips its CAS but republishes its terminal log, and an already-failed batch repeats member fanout.
+
+## External effects
+
+Queue deduplication cannot resolve an external effect whose outcome is unknown:
+
+```text
+provider accepts operation
+controller crashes before recording provider result
+```
+
+Correct execution requires one of:
+
+- A caller-supplied idempotency key accepted by the provider.
+- A stable operation identity that the controller can look up.
+- A documented acceptance of duplicate or orphaned work when duplicates are harmless.
+
+Without one of these properties, no redelivery algorithm can determine whether repeating the effect is safe.
+
+## Error and acknowledgement rules
+
+- Return `nil` only after every required publish in the current attempt succeeds.
+- Return an error after partial fanout rather than acknowledging success. The consumer will retry or route the message to its DLQ according to error classification and retry policy.
+- Let configured classifiers determine whether infrastructure failures are retryable.
+- Do not mark every publish error retryable merely because replay would be convenient.
+- Ensure the DLQ owner can drive the entity to an honest terminal state when retry is exhausted.
+
+See the [consumer error contract](../../../platform/consumer/README.md) and the [orchestrator workflow](../../../doc/rfc/submitqueue/workflow.md).
+
+## Controller review checklist
+
+Every controller should make these answers obvious:
+
+1. What stable logical operation does the input represent?
+2. Which states mean the transition is needed, already applied, downstream-progressed, superseded, or invalid?
+3. Which exact state transitions does this controller own?
+4. What durable data is needed to reconstruct fanout?
+5. What is the complete fanout, and in what order is it replayed?
+6. Are all output message IDs stable and payload-specific?
+7. What happens after a CAS conflict?
+8. Which component owns terminal-state recovery?
+9. Can any external effect be deduplicated or reconciled?
+10. What happens when retries are exhausted?
+
+The concise correctness model is:
+
+```text
+load and classify durable progress
+    -> perform the missing state transition once
+    -> replay complete fanout until accepted
+    -> acknowledge
+```
+
+See the [SQL queue RFC](../../../doc/rfc/sql-queue-rfc.md) for delivery, redelivery, ordering, and idempotent-publish semantics.