parallel-workload: add a correctness oracle for read-then-write - #37918
Merged
Conversation
This was referenced Jul 29, 2026
aljoscha
force-pushed
the
aljoscha/occ-01-pw-oracle
branch
from
July 29, 2026 07:14
77441a8 to
1032317
Compare
aljoscha
changed the title
Contributor
Author
It has found bugs during iteration yes. But 100% would be good if we can integrate into a broader thing |
aljoscha
force-pushed
the
aljoscha/occ-01-pw-oracle
branch
from
July 29, 2026 09:46
1032317 to
74cb929
Compare
The workload could tell us that concurrent DML does not panic the server or raise an unexpected error, but nothing checked that the writes it issued actually happened. A run where every UPDATE silently did nothing looked exactly like a healthy one. ReadThenWriteCounter gives it an oracle. Every worker increments the same row of a dedicated single-row table, and each attempt is recorded as definitely committed, definitely not committed, or undecided, based on the error text. At the end of the run the counter's value has to satisfy committed <= v <= committed + unknown. Both bounds carry weight: the lower one catches a lost write, the upper one catches a write that was reported as failed but committed anyway. The oracle says nothing about how a read-then-write is sequenced, so it holds for the coordinator's lock-based path as much as for anything that replaces it. That is the point: it has to be able to pass before it is evidence about anything. Attempts run over pg wire only. On the HTTP and WS paths a client-side timeout or a dropped response leaves the outcome undecided far more often, which widens the bound until it stops proving anything. InsertSelectAction covers the read-dependent INSERT. A constant VALUES list is a blind write, so only a source query over a persisted collection exercises the read-then-write branch.
aljoscha
force-pushed
the
aljoscha/occ-01-pw-oracle
branch
from
July 29, 2026 10:57
74cb929 to
90ef255
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Part 1 of 7 in a stack that moves
DELETE,UPDATEandINSERT ... SELECToff the coordinator onto the session task, using optimistic concurrency
control. Design doc:
20260210_incremental_occ_read_then_write.md(lands in part 6).
That change replaces mutual exclusion on a table with conflict detection
after the fact, so the interesting failure mode is a lost update: two
concurrent mutations both read the same snapshot and one silently
overwrites the other. Parallel workload had no way to notice that. It
checks for panics and unexpected errors, and its existing assertions do not
constrain the value a table ends up with.
Description
ReadThenWriteCounteris a dedicated single-row table that every workerincrements with a read-then-write
UPDATE, next to a Python-side tally ofhow each attempt ended: definitely committed, definitely not committed, or
undecided. At the end of a run the counter's value has to satisfy
committed <= v <= committed + unknown. The lower bound catches a lostwrite, the upper bound catches a write reported as failed that committed
anyway.
The oracle says nothing about how a read-then-write is sequenced, so it
holds for the coordinator's lock-based path as much as for anything that
replaces it. That is deliberate. It has to be able to pass today before it
is evidence about anything, and it stays useful after this stack lands.
InsertSelectActioncovers the read-dependentINSERT. A constantVALUESlist is a blind write, so only a source query over a persisted collection
reaches the read-then-write branch, and pointing it at its own target is the
most contended shape such a statement can have.
Verification
Ran parallel workload against the current lock-based path with the oracle
enabled. Nightly runs it as part of the existing parallel workload suites.