oplogPopulator: route oplog key through TransformObjectKey SMT when available#2741
Open
delthas wants to merge 1 commit into
Open
oplogPopulator: route oplog key through TransformObjectKey SMT when available#2741delthas wants to merge 1 commit into
delthas wants to merge 1 commit into
Conversation
Contributor
Hello delthas,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
... and 6 files with indirect coverage changes
@@ Coverage Diff @@
## development/9.4 #2741 +/- ##
===================================================
- Coverage 74.84% 74.79% -0.05%
===================================================
Files 199 199
Lines 13630 13671 +41
===================================================
+ Hits 10201 10225 +24
- Misses 3419 3436 +17
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
delthas
force-pushed
the
improvement/BB-768/oplog-partition-by-object-key
branch
from
650dd4e to
396d23b
Compare
delthas
force-pushed
the
improvement/BB-768/oplog-partition-by-object-key
branch
from
396d23b to
55af2e9
Compare
Contributor
Request integration branchesWaiting for integration branch creation to be requested by the user. To request integration branches, please comment on this pull request with the following command: Alternatively, the |
…vailable Probes the Kafka Connect cluster for the com.scality.kafka.connect.transforms.TransformObjectKey SMT plugin (shipped by Zenko via ZENKO-5274) on every reconciliation tick. When the plugin is loaded, the assembled connector config uses a key schema projecting documentKey._id (always populated on every change event) and adds the SMT to strip the arsenal master/version encoding, so master and all versions of the same S3 object hash to the same Kafka partition. When the plugin is absent (cluster on an older Connect image or during a rolling upgrade), the legacy fullDocument.value.key schema is used unchanged. Reconciliation flow is unchanged: the existing config-diff path PUTs the new config to /connectors/{name}/config in place, so no recreate, no resume-token loss. The probe is graceful on REST errors (falls back to legacy + a single warn log per state change). * lib/wrappers/KafkaConnectWrapper: add getConnectorPlugins. * extensions/oplogPopulator/constants: add defaultConnectorConfigSMT alongside defaultConnectorConfig; expose the SMT FQCN as transformObjectKeyClass. * extensions/oplogPopulator/modules/ConnectorsManager: probe in initializeConnectors and at the top of each scheduleConnectorUpdates tick; _getDefaultConnectorConfiguration consults _smtAvailable. Issue: BB-768
delthas
force-pushed
the
improvement/BB-768/oplog-partition-by-object-key
branch
from
55af2e9 to
3f61602
Compare
|
LGTM — clean design and solid edge-case handling. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
lib/wrappers/KafkaConnectWrapper: addgetConnectorPlugins()(GET /connector-plugins).extensions/oplogPopulator/constants.js: adddefaultConnectorConfigSMTnext to the legacydefaultConnectorConfig. The SMT variant projectsdocumentKey._idin the key schema and includes thetransforms.stripObjectKey.type=com.scality.kafka.connect.transforms.TransformObjectKeySMT +key.converter=StringConverter.extensions/oplogPopulator/modules/ConnectorsManager.js: probe Kafka Connect for the SMT plugin ininitializeConnectorsand at the top of everyscheduleConnectorUpdatestick._getDefaultConnectorConfigurationconsults_smtAvailableto pick the right base config. Single warn log per state transition; probe failures gracefully fall back to legacy.Context
BB-768 fixes per-S3-object partitioning on the oplog Kafka topic. The current key projects
fullDocument.value.key, which isnullonupdate/deleteevents (since BB-355 removedchange.stream.full.document=updateLookup). Update and delete events for any S3 object collapse onto a single partition while inserts spread across the topic — per-object ordering is broken across op types.The actual fix lives in the Zenko-side SMT plugin shipped via ZENKO-5274, which derives the key from
documentKey._id(always populated) and strips the arsenal master/version encoding so master and all versions of an S3 object hash to the same partition. This PR is the Backbeat side: it wires the connector config to use the SMT when the cluster's Kafka Connect image has the plugin installed.Probing each reconciliation tick means an upgrade rollout that bumps the Kafka Connect image after Backbeat auto-activates the SMT config on the very next tick, without operator action. Clusters whose Connect image does not yet have the SMT keep running on the legacy key schema unchanged. The transition flips an existing connector in place via
PUT /connectors/{name}/config— no recreate, no resume-token loss (the change touches the key schema + transforms only, not the pipeline match stage).Issue: BB-768