Enforce unique indexes across all shards of a TRUNCATE time partition#6001
Enforce unique indexes across all shards of a TRUNCATE time partition#6001dorinhogea wants to merge 1 commit into
Conversation
roborivers
left a comment
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Success ✓.
Regression testing: Success ✓.
The first 10 failing tests are:
sc_truncate_multiddl_generated [db unavailable at finish] **quarantined**
sc_truncate [db unavailable at finish]
consumer_non_atomic_default_consumer_generated **quarantined**
remotecreate_twopc_generated
remotecreate
tunables
reco-ddlk-sql [timeout] **quarantined**
Time partitions spread data across multiple shard tables to implement data retention. Until now, unique indexes were only enforced within the shard being written, making it possible to insert the same key into different shards without conflict. A new 'partition_unique' tunable enables cross-shard unique enforcement for TRUNCATE-rollout partitions. When on, any write that would violate a unique constraint in any sibling shard is rejected with the same error that a within-shard violation produces. The enforcement covers both inserts and updates that change a key column. The feature carries a measurable write cost proportional to the number of sibling shards and unique indexes, driven by the extra index lookups per write and the larger BDB lock footprint per transaction. A performance test is included that quantifies both impacts separately. Known limitations: - Enabling the tunable does not validate pre-existing data; cross-shard duplicates already present are the user's responsibility to resolve. - UPSERT is not supported on time partitions. Signed-off-by: Dorin Hogea <dhogea@bloomberg.net>
roborivers
left a comment
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Error ⚠.
Regression testing: Success ✓.
The first 10 failing tests are:
sc_timepart **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
tunables
reco-ddlk-sql [timeout] **quarantined**
roborivers
left a comment
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Error ⚠.
Regression testing: Success ✓.
The first 10 failing tests are:
sc_resume_logicalsc_generated **quarantined**
sp_snapshot_generated
consumer_non_atomic_default_consumer_generated **quarantined**
reco-ddlk-sql [timeout] **quarantined**
partition_unique_perf [timeout]
2 participants
Time partitions spread data across multiple shard tables to implement data retention. Until now, unique indexes were only enforced within the shard being written, making it possible to insert the same key into different shards without conflict.
A new 'partition_unique' tunable enables cross-shard unique enforcement for TRUNCATE-rollout partitions. When on, any write that would violate a unique constraint in any sibling shard is rejected with the same error that a within-shard violation produces. The enforcement covers both inserts and updates that change a key column.
The feature carries a measurable write cost proportional to the number of sibling shards and unique indexes, driven by the extra index lookups per write and the larger BDB lock footprint per transaction. A performance test is included that quantifies both impacts separately.
Known limitations: