docs(rfc): add queue-scoped speculation RFC

[behinddwalls]

Summary

Why?

The current speculation design (doc/rfc/submitqueue/speculation.md) materializes a batch's full 2^N speculation tree up front, persists it, and re-walks all of it on every event — and each per-batch pass additionally triggers a queue-wide prioritize round. The enumerate/score/select seams force the materialized tree through every contract boundary, which caps how far the dependency limit can ever scale and couples per-event cost to tree size.

What?

Adds an RFC proposing a queue-scoped execution model for speculation: a queue-keyed speculation pass (pure function of ground truth, one pass per dirty signal, self-arming tick backstop) replaces the batch-keyed speculate and prioritize stages; persistence inverts to bets-not-candidates (bets keyed by content-addressed path IDs with a correctness-required attempt counter, an executor-owned path→build record, and a per-queue book as bet index and membership linearization point); extension moves to a single Speculator seam with a composed Belief/Generator/Admitter default. The RFC records the adversarial validation of each load-bearing assumption (single-writer, crash choreography under no-transactions NoSQL, merge safety, cost ceiling), the design decisions with rejected alternatives, a shadow-mode cutover test, and open questions.

Merge finalization is strict: the pass advances a batch to merge only when its predecessors have actually merged (observed via mergesignal). speculation.md's optimistic merge hand-off is deliberately dropped and deferred to its own design, recorded as an open question, to keep merge-executor concerns out of the speculation pass.

[albertywu]

Early signal

Thanks, I spent some time working through this. My early signal is that I prefer this direction over continuing the materialized-tree stack.

My mental model is now:

state-owning controllers persist facts
-> publish a queue-level poke
-> speculate reloads the current queue state
-> derives the desired bets
-> reconciles persisted intent and asynchronous execution toward that plan

This feels similar to a level-based reconciliation controller. The event tells speculation where to look, but current stored state determines what to do.

What I like

• The scope matches the decision. Build capacity is queue-wide, so admission should be queue-wide rather than per-batch selection followed by another queue-wide prioritization stage.
• Persisting bets rather than candidates is a much better model. Virtual candidates are derived state. Persisting only work that was actually dispatched removes a large amount of tree maintenance and reconciliation.
• Producers become simpler. Score, buildsignal, mergesignal, cancel, and DLQ controllers only update the state they own and poke the queue. They do not need to identify every affected dependent or prescribe downstream actions.
• The asynchronous model is understandable. Bets represent durable intent, execution records identify attempts, and build entities contain the latest persisted CI observation. The speculation pass reads local state rather than synchronously polling every CI build.
• Attempts provide a reasonable compensation model. If a decision must be reversed, the system cancels the current attempt. If the same path becomes desirable again, it creates a new attempt rather than trying to reverse cancellation.
• Cancelling work continuing to occupy budget is the correct conservative behavior.
• Strict merge finalization is a good initial boundary. Merge is irreversible, so waiting for terminal dependency facts keeps optimistic merge concerns out of the first implementation.
• Removing dependent fanout eliminates a meaningful class of partial-failure and missed-wakeup problems.

Concerns I think should be resolved in the RFC

External side-effect idempotency

The reconciliation loop can only supervise external work that it knows exists.

The current execution-record sequence still appears to be:

observe no execution record
-> trigger CI
-> persist execution record

A crash after triggering CI but before persisting the record leaves an unknown build. A later pass can trigger another one.

I think the design needs a durable pre-trigger reservation plus a runner-level idempotency key derived from (pathID, attempt), or an equivalent lookup mechanism.

The same issue exists for committing merges. Runway needs a durable request/result record written before calling Merger.Merge, or the merger must guarantee request-ID idempotency. Strict finalization prevents unsafe readiness decisions, but it does not prevent the same committing request from executing twice.

Tick multiplication

If every delivered poke schedules an occurrence-unique delayed tick, then a burst of ten pokes creates ten permanent tick chains. Each tick schedules its own successor.

I think the tick needs singleton semantics per queue and scheduling epoch. Multiple passes should converge on the same next tick rather than each creating an independent chain.

Book bounds

The RFC says the book stays small because bets are budget-bounded, but passed bets hold no budget and remain book-listed until their batches become terminal.

During a merge backlog, passed bets can accumulate while new builds continue to be admitted. The book then grows with queue depth or merge backlog rather than CI budget.

It may be cleaner for the book to track only bets that could still hold CI capacity, with the winning passed bet stored separately on the batch or in a head-to-winning-bet mapping.

Path identity

Hashing only:

queue + head ID + ordered base IDs

does not appear sufficient because dependency ordering is unspecified, dependencies can change for the same head, and excluded dependencies are part of the path's meaning.

I would define a versioned canonical encoding containing the complete dependency universe or graph generation, deterministic ordering, and included/excluded assumptions.

Policy versus correctness

The proposed Speculator owns both:

• replaceable policy such as beliefs, generation, ranking, and preemption;
• mandatory correctness such as fact reconciliation, refutation, merge readiness, no-viable-path failure, and cancel quiescence.

That boundary feels too broad. In particular, if a Generator may choose a narrower path space, it should not be able to influence the correctness conclusion that no viable path exists.

I would keep mandatory lifecycle and reconciliation rules in controller/core code and make only beliefs, ordering, admission, and optional preemption pluggable.

Determinism

The RFC says the same snapshot produces the same plan, but budget, depth, time-based fairness, model version, and injected signals can change independently from the snapshot.

If replayability and shadow-mode comparison are important, I think those values should be frozen into an immutable PlanningInput:

snapshot
+ budget
+ depth bound
+ planning time
+ policy/model version
+ fairness inputs

Things I would investigate next

1. Write the external-effect protocols first. Document the exact crash interleavings for build reservation, trigger, execution-record creation, cancellation, merge execution, and result publication.
2. Define the singleton tick protocol. Specify its durable identity and what happens when several passes race to schedule it.
3. Tighten book ownership and bounds. Decide exactly which bets belong in it and prove its maximum size.
4. Narrow the Speculator contract. Separate mandatory reconciliation and verdicts from swappable decision policy.
5. Specify canonical path identity. Include enough information to represent the complete truth assignment without relying on dependency slice ordering.
6. Define stale-plan outcomes. Expected CAS conflicts should cause reload/recompute, not poison-message handling or terminal DLQ reconciliation.
7. Measure the control-loop ceiling. Track p50/p99 pass duration, queue event rate, snapshot size, book size, stale-plan conflicts, and time from external fact persistence to corrective action.
8. Expand the shadow-mode plan. Snapshot-to-plan comparison validates planning, but not dispatch idempotency, cancellation, DLQ behavior, GC, tick scheduling, or merge execution. Those need failure-injection or integration validation.

Overall

I think the foundational choice is right: persist facts and actual side effects, then continuously derive the plan, rather than persisting and incrementally maintaining the entire possibility space.

I would pivot toward this design, but tighten the points above before committing to the interfaces and persistence model.

[albertywu]

Performance comparison

Let:

• Q: live batches
• K: batches affected by an event
• P: paths per materialized tree, potentially 2^depth
• R: total dependency references
• B: active bets
• C: candidates inspected lazily

Before

Per event: O(K * Q * P)
Storage:   O(Q * P)

Each affected batch walks its tree and triggers another queue-wide tree sweep.

After

Per poke: O(Q + R + B + C * depth)
Storage:  O(B)

Each poke reloads the queue, but candidates are generated only when needed.

Likely crossover

The new approach generally wins when:

K * Q * P > Q + R + B + C * depth

Scenario	Likely winner
Small queue, shallow trees, one affected batch	Previous may be slightly faster
Moderate fanout or deeper trees	New approach wins significantly
Huge queue with transitive dependency closure	Previous may win if trees are small
Huge queue and deep trees	New approach usually wins again

Example

Q=100, K=10, P=256
R=800, B=20, C=120, depth=8

Approximate work:

Before: ~258,000 units
After:  ~2,000 units

The main risk for the new approach is not candidate generation. It is repeatedly loading R, since transitive dependencies can grow to O(Q²).

Passes also serialize per queue:

utilization = poke rate * pass duration

If a pass takes 500 ms, two pokes per second saturate the queue. Coalescing redundant pokes would materially improve this.