parallel-workload: broaden SQL-surface and action coverage - #37590
Conversation
0510991 to
3fb6bcb
Compare
9afb0c6 to
00335db
Compare
94b6da0 to
008bea1
Compare
bce3057 to
617728a
Compare
Add ~34 new actions and query-generation breadth from a doc/user audit: full SELECT surface (GROUP BY/DISTINCT/outer+lateral joins/subqueries/ CTEs/WITH MUTUALLY RECURSIVE/table functions/temporal filters/aggregate families), DDL (CREATE OR REPLACE, DROP CASCADE, CREATE TYPE, network policies, ALTER CLUSTER SET), transactions (parameterized prepared statements, bounded staleness, read-only/DDL transactions), introspection (system-catalog reads, EXPLAIN ANALYZE/FILTER PUSHDOWN, full EXPLAIN matrix), load generator sources, sink options + Iceberg MODE APPEND, RBAC, connection/secret lifecycle, SHOW, and broader COMMENT. Verified end-to-end with a full mzcompose regression/ddl run (10423 queries, every new action exercised, no panics). Two pre-existing product bugs surfaced: ALTER NETWORK POLICY quoted-name resolution (action disabled) and the SS-324 ALTER SINK SET FROM stall (load generator tables excluded from sink inputs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The broadened coverage surfaced three product bugs. Each triggering action is disabled or narrowed so the nightly stays green, with a TODO to reenable once the bug is fixed. - Coordinator panic when ALTER CONNECTION re-alters a dependent sink's export connection (catalog_implications.rs:780, InvalidAlter turned into a panic by unwrap_or_terminate). AlterConnectionAction is disabled; ValidateConnectionAction and AlterSecretAction (Secret, not Connection) stay enabled. - Optimizer soft-panic on SUBSCRIBE ... UP TO over an object whose as_of has advanced to MAX, e.g. a finished bounded load generator source (dataflows.rs:97, non-empty until with as_of = MAX). FetchAction no longer emits UP TO; the safe AS OF AT LEAST 0 stays. - COPY FROM CSV cannot decode a large-year date that COPY TO CSV wrote, so the CSV date roundtrip is not closed for extreme years. CopyFromS3Action ignores that decode error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two further failures on the previous workaround commit, both worked around with a TODO pointing at the tracked bug. - The util.rs:263 coordinator panic recurred without ALTER CONNECTION: a GRANT/REVOKE or ALTER OWNER on a connection also emits a Connection(Altered) implication, which re-alters every dependent sink's export connection. When a dependent sink's relation_key_indices is momentarily out of sync between catalog and controller (a concurrent FROM reshape), the re-alter fails alter_compatible and InvalidAlter is turned into a panic. Removed the CONNECTION target from BroadPrivilegesAction (GRANT/REVOKE) and AlterOwnerAction (owner). ALTER CONNECTION stays disabled. COMMENT/SHOW/ VALIDATE on connections are unaffected (no Connection(Altered)). - DropTypeAction now ignores "does not exist": another worker or a CASCADE drop of the schema/database can drop the type first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Build 17159 repeat_row: EXPLAIN ANALYZE HINTS failed with "target replica failed or was dropped" when a concurrent DROP/reconfigure retired the target replica. services.log has no panic, so it is a client-side race, not a crash. Added the message to ExplainAnalyzeAction.errors_to_ignore. The build's other failure is the known DB-120 persist filter pushdown audit violation (tracked in FINDINGS-BUGS.md), left enabled since it catches a real correctness bug. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Build 17159 DDL Only: worker failed with "no valid schema selected" on CREATE SECRET pgpass<N>. The Pg/MySql/SqlServer source executor connects to the source's target user database (dbname=self.database) and runs an unqualified CREATE SECRET/CONNECTION there. A concurrent DropDatabaseCascade leaves that session with no schema to resolve. Added the message to the base DDL/DDL-only drop-race ignore list, next to the existing unknown schema / unknown database entries. Validated by a local ddl-only run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…c (17160) Nightly 17160 rename: thread 'coordinator' panicked at catalog/state.rs:1631 "OrdMap::index: invalid key". Backtrace is get_database <- resolve_full_name <- transact_op <- create_secret_finish. A staged create resolves its new item's full name at finish, but the item's database was dropped between staging and finish, so get_database (a panicking OrdMap index) hits a missing key. The source executor's unqualified CREATE SECRET pgpass<N> runs in the source's target user database, and a concurrent DROP DATABASE CASCADE of it triggers the race. Only CASCADE can drop a non-empty database, so DropDatabaseCascade is disabled with a TODO. See FINDINGS-BUGS.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
….. USING Local rename+naughty run: DELETE .. USING failed with "internal error: Non-positive multiplicity in DistinctBy". DELETE .. USING (added in this work) lowers to a semijoin whose DistinctBy can surface a negative-accumulation error for some generated WHERE clauses. This is the known class tracked in database-issues#9308, previously tolerated only in the RepeatRow scenario. DeleteAction now tolerates NEGATIVE_ACCUMULATION_ERRORS in all scenarios. See FINDINGS-BUGS.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rejections A short ddl-only run tripped the broken-action assertion for DropRoleAction: 0 successes with "cannot be dropped because some objects depend on it" counted as a real error. That is a legitimate RESTRICT rejection, not brokenness: AlterOwnerAction reassigns object ownership to random roles, so a role usually owns something and DROP ROLE is rejected. Longer runs eventually land a dependency-free role and succeed. Treat that message as ownership-style noise in the coverage check so the assertion stops false-positiving. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nic (17160)
Nightly 17160 0dt: thread 'coordinator' panicked at read_policy.rs:389
"missing compute collection: CollectionMissing". Backtrace is acquire_read_holds
<- sequence_explain_pushdown <- sequence_plan. EXPLAIN FILTER PUSHDOWN builds an
id_bundle then acquire_read_holds(..).expect("missing compute collection"); if a
referenced index/MV is dropped between planning and read-hold acquisition, the
expect panics. Normal peeks and EXPLAIN ANALYZE surface the drop as an error
instead, so this path is specific to EXPLAIN FILTER PUSHDOWN.
ExplainFilterPushdownAction disabled in both action lists with a TODO.
ExplainAnalyzeAction stays. See FINDINGS-BUGS.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Local 0dt run: DROP NETWORK POLICY "netpol-N" failed with "unknown network policy 'netpol-N'" when another worker dropped it first. The error carries the raw name (not the quoted form), so DROP resolves names correctly; this is a concurrency race, not the ALTER NETWORK POLICY quoted-name bug. Added "unknown network policy" to DropNetworkPolicyAction.errors_to_ignore. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… 17163 - rename: the broken-action assertion flagged ExplainAnalyzeAction. Like DropRoleAction, it has a strict runtime precondition (a hydrated MV/index on the active cluster) that renames/drops frequently deny, so a seed can see zero successes with only legitimate rejections. Both are now excluded from the assertion (they succeed in normal runs; not broken). - repeat_row: a negative-accumulation error expected in that scenario was missing from NEGATIVE_ACCUMULATION_ERRORS: the client-facing "... with non-zero accumulation in accumulable aggregate" (reduce.rs:1514, MaterializeInc#8106), distinct from the internal ReduceAccumulable log text already listed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ble,View} Nightly 10x flake-hunt (17171/17170): worker died with "KeyError: <Index object>" at DropIndexAction. A concurrent CASCADE drop's untrack_objects_in_schemas removes the object from the tracked collection (indexes is a set it reassigns; tables/views are lists it filters), and the Drop action then removes it again. DropIndex's except-branch remove hit KeyError; DropTable/DropView can hit ValueError in the window between a successful DROP and their remove. Use set.discard for indexes and guard the list removes with try/except ValueError. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…panic Nightly 10x flake-hunt (17169): thread panicked at frontend_peek.rs:1759 assert_read_holds_correct "storage read hold at .. is not enough for as_of ..". The frontend peek path chooses as_of then acquires read holds; compaction can advance the collection's since past as_of in that window (seen ~173ms), so the soft-assert fires and panics the coordinator. The classic coordinator peek path does not hit this. Gated by enable_frontend_peek_sequencing (default on). Pinned enable_frontend_peek_sequencing off in ADDITIONAL_SYSTEM_PARAMETER_ DEFAULTS and removed it from FlipFlags, with a TODO. Peeks still run via the classic path. See FINDINGS-BUGS.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…assertion Third coverage-check false-positive (DropWebhookSourceAction, after DropRole and ExplainAnalyze): a churny seed can see zero successes for any Drop* action when a concurrent CASCADE untrack/drop removes the target first, or RESTRICT rejects it, or (roles) AlterOwner leaves it with dependents. Per-action exclusions were whack-a-mole. Exclude all Drop* actions (by name) and ExplainAnalyze (hydrated- target precondition) from the assertion; it now targets CREATE/ALTER/write actions, where never-succeeding really does mean broken SQL. This also avoids the propagate-then-hang teardown that timed the run out. Drops still count in the printed coverage warnings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Nightly builds 17176-17185 were mostly clusterd OOM (Memory cgroup out of memory), concentrated in DDL/rename/cancel. clusterd hit ~9 GB anon-rss despite tiny data (MAX_ROWS=50), so the cost is arrangement/dataflow memory, which is copied per worker (and per process). The workload used up to scale=2,workers=2 / workers=4 (4x copies) with no per-replica bound. Cut the worker-copy multiplier: workload replicas now scale=1,workers=1 or scale=1,workers=2 (was up to 4x), and the multi-replica quickstart cluster (the default --replicas=2 peek target) drops from workers=4 to workers=1. Memory falls ~2-4x with modest loss of multi-worker coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each disabled or narrowed action now names the Linear issue tracking the product bug it works around, so the test parts can be re-enabled once each is fixed: - CLO-143 ALTER NETWORK POLICY quoted-name resolution (AlterNetworkPolicyAction) - SQL-517 connection export re-alter coord panic (AlterConnection/AlterOwner/BroadPrivileges) - SQL-518 get_database panic on concurrent DROP DATABASE CASCADE (DropDatabaseCascadeAction) - SQL-519 EXPLAIN FILTER PUSHDOWN missing-collection panic (ExplainFilterPushdownAction) - SQL-520 frontend-peek read-hold soft-panic (enable_frontend_peek_sequencing pin, FlipFlags) - SS-344 ALTER SINK SET FROM hang (db_objects_for_sinks, AlterIcebergSinkFromAction) - SS-345 COPY FROM CSV large-year date decode (CopyFromS3Action) - CLU-169 SUBSCRIBE UP TO as_of=MAX optimizer soft-panic (FetchAction) Comment-only change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Nightly 17201/17207/17209/17210/17212 kept OOM-killing clusterd (kernel
SIGKILL, "out of memory: Killed process (clusterd)") on the 24 GiB agent,
concentrated in repeat_row/cancel/rename. Attribution from services.log:
the repeatedly relaunched process was always the quickstart cluster (the
--replicas=2 peek/index target), grown unbounded until SIGKILL, with no
single query logged.
The replica sizes the workload uses (scale=1,workers=N) announce no
--heap-limit, so clusterd's memory limiter is disabled ("no heap limit
announced; disabling memory limiter") and nothing sheds before the cgroup
kills it. Switching to mem= sizes only trades the kill for the limiter's
"memory utilization exceeded configured limits", which ci-annotate-errors
also flags, so the only path to green is using less memory.
Data is tiny (was MAX_ROWS=50) but views nest, so a peek flattens into a
deep join whose intermediate scales as MAX_ROWS ** join_depth (50**5 is
~300M rows). Cut MAX_ROWS 50 -> 10: every arrangement/peek/join
intermediate shrinks, and the exponential blow-up drops by orders of
magnitude, while query-shape coverage (what this panic-hunting test cares
about) is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Builds 17205 and 17214 (repeat_row) failed a worker on `constant folding encountered reduce on collection with non-positive multiplicities` (SELECT DISTINCT / INTERSECT over a view whose repeat_row input has -1 diffs). This is the known negative-accumulation class (database-issues#9308): repeat_row intentionally creates non-positive multiplicities, and constant-folding a reduce/DISTINCT over such a collection surfaces this as a client error. It was just missing from the tolerated list, so add it alongside the existing constant-folding entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Upstream's Record/RecordList types joined DATA_TYPES for use nested in expressions, but a parameter or value cast to them fails with "cannot reference pseudo type pg_catalog.record". Draw prepared-statement parameter types from DATA_TYPES_FOR_COLUMNS, which excludes them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b262e24 to
04bf2be
Compare
|
Long time brewing, I think it's time to finally get this in. Green run: https://buildkite.com/materialize/nightly/builds/17630 & https://buildkite.com/materialize/nightly/builds/17631 |
bobbyiliev
left a comment
There was a problem hiding this comment.
Went through the diff and cross-checked against the current code. Great to see this much surface covered, the per-bug TODOs with issue links are really helpful. Most of my notes are small, and there is one I would like to fix before merge (the role lock ordering in Grant/RevokeRoleAction).
Two questions that are more decisions than line comments, so putting them here:
Negative accumulation tolerance (action.py L392): this now tolerates the whole negative-accumulation class in DML and DDL complexity, which is where most nightly runs live. Could we disable just the DELETE .. USING variant until database-issues#9308 is fixed and keep the assertion strict? Otherwise a genuine finding in those two complexities cannot fail a run anymore.
MAX_ROWS 500 -> 10 (database.py L70): that is a big drop, and it reads a bit against the "larger scale" goal of the PR. Do the DML and reduce/TopK paths still get useful coverage at 10 rows, or would something like 50 already be enough to keep clusterd under the cgroup?
I'll try to get it stable with 500 rows.
Yeah, reducing the scope is a great idea, trying. |
|
Lots of good stuff, looks like I need to run a new Fable review of this PR, and a few more rounds of CI. Converting to draft in the meantime. Edit: done. |
d77b7a2 to
b24525c
Compare
b24525c to
bf072eb
Compare
|
Ready for review again. New nightly run: https://buildkite.com/materialize/nightly/builds/17701 |
d8cc3bf to
74a63df
Compare
74a63df to
8efa5ec
Compare
Mostly parallel-workload followups to MaterializeInc#37590 and issues having been closed. Based on https://buildkite.com/materialize/nightly/builds/17774
Mostly parallel-workload followups to MaterializeInc#37590 and issues having been closed. Based on https://buildkite.com/materialize/nightly/builds/17774
Mostly parallel-workload followups to #37590 and issues having been closed or moved to Linear from my sweep yesterday. Based on https://buildkite.com/materialize/nightly/builds/17774
Substantially broadens what the parallel-workload stress test exercises,
from a doc and surface-area audit, so nightly runs cover far more of the SQL
and DDL surface, at larger scale, and find more product bugs. Every bug this
coverage surfaced is worked around (the triggering action is disabled or
narrowed, each with a TODO naming the tracked issue) so the nightly stays
green until the bug is fixed.
New coverage
Adds ~34 actions and query-generation breadth:
joins, subqueries, CTEs, WITH MUTUALLY RECURSIVE, table functions, temporal
filters, and the aggregate families. Generated
WHEREconjuncts areparenthesized so an
ORinside one part stays a single top-level conjunct.Boundary literals (float4
NaN/Infinity) are injected as expressionleaves to stress overflow and eval paths.
CREATE OR REPLACE,DROP ... CASCADE,CREATE TYPE, networkpolicies,
ALTER CLUSTER SET,ALTER ROLE SET/RESETsession defaults, andbroader
COMMENT.read-only and DDL transactions.
EXPLAIN ANALYZE, and the fullEXPLAINmatrix.MODE APPEND, andCREATE REPLACEMENT MATERIALIZED VIEWwith both theAPPLY REPLACEMENTand discard paths.SHOW.Incident 1136 coverage: replacement lifecycle and a sealed-shard oracle
Incident 1136 (a dropped replacement finalizing its live target's shard
after an envd restart) needed
CREATE REPLACEMENTand the resolvingDROPto happen on different connections across a restart. The old
ReplaceMaterializedViewActionresolved a replacement on the connectionthat created it, so it structurally could not hit this. The lifecycle is now
split into create/apply/drop actions that pick live replacements from
mz_internal.mz_replacements, so replacements created by other workers orleaked across kills and 0dt deploys are found and resolved too. Verified to
go red in one kill cycle on a build without the #37705 fix.
Detection does not rely on the (much later) tombstone panic but on a
client-side data-loss oracle,
SealedCollectionCheckAction: a live usertable or materialized view whose write frontier is the empty antichain has
had its shard sealed out from under it. Legitimate seals are classified
away: REFRESH AT views and bounded (UP TO) load generators seal when they
complete, sealing is transitive through readers (a catalog dependency walk),
and a view whose optimized plan reads no live storage collections (constant
folding, e.g. a join
ON false, drops inputs the catalog still records)completes on hydration, whatever its refresh schedule says (checked via
EXPLAIN OPTIMIZED PLAN AS JSON). Candidates are re-checked after 5s sincethe catalog and
mz_frontiersare not updated atomically. A guaranteedfinal oracle pass runs at the end of every run, including on the
wedged-worker exit path, so damage from a run's last seconds cannot escape.
Scale
Raises the per-object caps (databases, schemas, tables, views, indexes,
roles, sources, sinks, types, network policies, and columns) by roughly 10x,
so a run holds many more concurrent objects and exercises deeper dependency
graphs.
Product bugs surfaced
The new coverage found several pre-existing bugs. Each is worked around with a
TODO pointing at the tracked issue so the action can be re-enabled once fixed:
ALTER NETWORK POLICYcannot resolve quoted names.ALTER CONNECTION,GRANT/REVOKE, orALTER OWNER) reshapes a dependentsink's export connection while its key indices are momentarily out of sync.
get_databasepanic on a concurrentDROP DATABASE CASCADEduring a staged create's finish.
EXPLAIN FILTER PUSHDOWNmissing-collection panic when areferenced index/MV is dropped between planning and read-hold acquisition.
(
enable_frontend_peek_sequencingpinned off, peeks run the classic path).ALTER SINK SET FROMhang.COPY FROMCSV cannot decode a large-year date thatCOPY TOCSV wrote.
VALIDATE CONNECTIONtransiently reads an empty password duringa concurrent
ALTER SECRETrotation.SUBSCRIBE ... UP TOoptimizer soft-panic when the object'sas_ofhas advanced toMAX(e.g. a finished bounded load generator).CREATE REPLACEMENT MATERIALIZED VIEWaccepts a sealed target(e.g. a constant-folded view, which seals on hydration whatever its refresh
schedule says) that
APPLY REPLACEMENTthen refuses forever, leaving anunappliable pending replacement. Found by the sealed-shard oracle.
Known negative-accumulation errors (database-issues#9308) from
DELETE ... USINGand
repeat_roware tolerated in the scenarios that generate them and staystrict elsewhere, so a genuine finding still fails.
Coverage for two further directions is landed but left disabled, ready to flip
on once the bugs are fixed: a dependency-consistency oracle (SQL-521) and a
source read-hold sweep across restart (SS-346, PER-49).
Closes: QAR-139