From e96df5fef184c6b13911d02d57b9a52b75f8f6b8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Piotr=20P=C5=82aczek?= Date: Tue, 23 Jun 2026 16:34:38 +0100 Subject: [PATCH 1/7] feat: add kafka api migration proposal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Assisted-by: Claude Sonnet 4.6 Signed-off-by: Piotr Płaczek --- proposals/116-kafka-api-migration.md | 197 +++++++++++++++++++++++++++ 1 file changed, 197 insertions(+) create mode 100644 proposals/116-kafka-api-migration.md diff --git a/proposals/116-kafka-api-migration.md b/proposals/116-kafka-api-migration.md new file mode 100644 index 0000000..6525eb7 --- /dev/null +++ b/proposals/116-kafka-api-migration.md @@ -0,0 +1,197 @@ +# 116 - Own the Kafka Protocol API Surface + +> [!NOTE] +> This proposal is based on the [associated investigation (kroxylicious#3996)](https://github.com/kroxylicious/kroxylicious/issues/3996). + +Kroxylicious should take ownership of the Kafka protocol classes - the generated `*Data` types, and the protocol infrastructure they depend on - placing everything under Kroxylicious-owned namespaces. This eliminates the project's exposure to Kafka internal package reorganisations and establishes the stable public API surface required for a 1.0 release. + +## Current situation + +Kroxylicious depends directly on classes from `kafka-clients` across its public and internal APIs: + +- **~198 generated `*Data` classes** (`org.apache.kafka.common.message.*`) appear in public filter interface method signatures. These are generated by Kafka's own `MessageGenerator` from JSON IDL spec files located at `clients/src/main/resources/common/message/` in the Kafka repo. +- **~16 protocol infrastructure classes** (`org.apache.kafka.common.protocol.*`) that the generated classes depend on: `ApiMessage`, `Message`, `Readable`, `Writable`, `ObjectSerializationCache`, `MessageSizeAccumulator`, the `types.*` schema model, etc. +- **~35 record classes** (`org.apache.kafka.common.record.*`) including `MemoryRecords`, `RecordBatch`, `Record`, etc. - hand-coded implementations of Kafka's on-wire/on-disk record format. +- **`ApiKeys`** - the enum that maps API key numbers to names and version ranges. +- **Scattered `common.*` types** - `Uuid`, `ApiException` subclasses, ACL types, SASL classes, utilities. + +Kroxylicious's `ByteBufAccessor` currently `implements Readable, Writable` from Kafka, bridging Netty's `ByteBuf` to Kafka's I/O abstraction. Any namespace change requires updating this bridge. + +The `kroxylicious-krpc-plugin` already parses the JSON IDL specs independently using Jackson (without depending on Kafka's generator) and drives FreeMarker templates to generate filter interfaces, invokers, and decoders. Its only Kafka dependency is `ApiKeys`, used in 3 places. + +## Motivation + +Kafka 4.3.0 moved 37 classes from `org.apache.kafka.common.record.*` to `org.apache.kafka.common.record.internal.*` ([KAFKA-20128](https://issues.apache.org/jira/browse/KAFKA-20128)). Upgrading Kroxylicious to 4.3.0 required import changes across 80 Java files in 9 modules. This is a symptom of a deeper problem: Kroxylicious depends on Kafka internal APIs that Kafka is free to move or remove at any time. + +Kafka does not consider these packages to be stable public API. The project has no obligation to maintain backward compatibility, and further reorganisations are likely. Each one forces a breaking change on every Kroxylicious filter developer - internal and external. + +For a 1.0 release, Kroxylicious needs a stable public API surface that the project controls. The Kafka protocol wire format itself is stable (governed by KIPs), but the Java classes representing that format are not. + +## Proposal + +### Scope: own the full API surface in one move + +Every type that appears in Kroxylicious's public API should live in a Kroxylicious namespace. This means a single breaking change for filter developers - one import migration - rather than a drip of repeated breakage as Kafka reorganises its internals. + +"Own the surface" does not mean "rewrite everything from scratch." The shaded and generated artifacts redistribute `kafka-clients` bytecode under the same Apache 2.0 licence. The original Apache Software Foundation copyright notices are preserved in the bytecode. The underlying implementations can be replaced with Kroxylicious-native ones incrementally, gated by the verification harness proving each replacement is correct. + +### Approach: shade `kafka-clients` via a dedicated repo + +The Maven Shade Plugin takes compiled classes from `kafka-clients` and republishes them under a Kroxylicious namespace. All cross-references between relocated classes are rewritten automatically at the bytecode level. This means Kafka package reorganisations - including moves to an `internal` subpackage like KAFKA-20128 - are completely invisible. + +A dedicated repo in the Kroxylicious GitHub org (`kroxylicious/kafka-patches`) stores the shading configuration and any generator patches needed for future enhancements. When generator enhancements are implemented, CI will apply the patches against the Kafka generator source, run the patched generator to produce enhanced `*Data` source, compile, and shade the result alongside the rest of `kafka-clients`. + +Published artifact versions use the Kafka version as a base with a qualifier suffix (e.g. `4.3.0-kroxy-0`, `4.3.0-kroxy-1` for Kroxylicious-only fixes, `4.4.0-kroxy-0` when bumping to a new Kafka release). This makes the Kafka baseline immediately visible in any dependency declaration. + +The `kroxylicious-krpc-plugin` currently depends on `kafka-clients` solely for `ApiKeys`. After this change it depends on the new artifact instead. Its role in generating filter interfaces, invokers, and decoders is otherwise unchanged. + +### Shading configuration + +The shading configuration in `kroxylicious/kafka-patches` uses a single broad relocation rule: + +- **Include:** `org.apache.kafka:kafka-clients` +- **Relocate:** `org.apache.kafka` → `io.kroxylicious.kafka` + +Everything in `kafka-clients` is relocated in one rule - `*Data` classes, protocol infrastructure, record classes, OAuth/SASL classes, `requests.*`, and anything Kafka adds in future releases. No per-package enumeration is needed. + +When Kafka relocates a class in a future release - as it did with `MemoryRecords` in KAFKA-20128 - an additional targeted rule is added to map the new upstream path back to its stable Kroxylicious location. For example: + +- **Stability rule:** `org.apache.kafka.common.record.internal.MemoryRecords` → `io.kroxylicious.kafka.common.record.MemoryRecords` + +Filter developers who import `io.kroxylicious.kafka.common.record.MemoryRecords` see no change at all. This is the key advantage over a direct `kafka-clients` dependency: Kafka can freely reorganise its internal package structure and the Kroxylicious namespace remains stable. These stability rules accumulate in `kroxylicious/kafka-patches` over time and serve as a permanent, inspectable record of every upstream reorganisation that would otherwise have been a breaking change. + +### Verification strategy + +The verification harness is a prerequisite, not a follow-up. Correctness must be provable at the point the artifact is first published and on every CI run thereafter. + +#### Wire-level round-trip tests + +For every spec and every supported protocol version: serialise a message with our generated classes, deserialise with Kafka's, and assert byte-level equality - then reverse the direction. This is the ground truth. If bytes we produce are consumed correctly by Kafka's classes and vice versa, we are wire-compatible. + +Exploratory work on the [`piotrpdev/kroxylicious#refactor/generate-data-class`](https://github.com/piotrpdev/kroxylicious/tree/refactor/generate-data-class) branch built a prototype of this infrastructure covering all 198 specs: parameterised fidelity tests at every valid version, randomised field-value tests, jqwik property-based tests for full-depth message instances, and a compile-time spec count guard. That prototype was designed around the FreeMarker template approach (see ["Rejected Alternatives"](#rejected-alternatives)), where generated classes were compiled at test time via `ToolProvider.getSystemJavaCompiler()` and loaded via `URLClassLoader`. With the shading approach the classes are already on the classpath at build time, so only the class-loading mechanism changes - the verification logic itself carries over unchanged. + +These tests live in `kroxylicious/kafka-patches` alongside the shading configuration, with `kafka-clients` as a test-scoped dependency. They run as part of that repo's CI pipeline - when `kafka-clients` is bumped, they immediately report any incompatibility before a new artifact is published. + +#### Spec drift detection + +When `kafka-clients` is bumped in `kroxylicious/kafka-patches`, the shaded artifact is rebuilt from the new jar. Any changes to the JSON IDL spec files are reflected in the shaded `*Data` classes automatically. The `EXPECTED_SPEC_COUNT` constant in the fidelity tests provides an additional guard: if the new `kafka-clients` version adds or removes spec files, the build fails until the constant is updated, forcing an explicit review of what changed. + +Not all spec changes are equal. Reviewers should evaluate by risk: + +| Change type | Risk | Why | +|-----------------------------------|------------|---------------------------------------------| +| Changed `flexibleVersions` | **High** | Changes wire encoding format | +| Removed or type-changed field | **High** | Silent deserialisation corruption | +| New field with `ignorable: false` | Medium | Non-default value + old version = exception | +| New spec file | Low-Medium | Generator may or may not handle it | +| New `validVersions` max | Low | Just more versions to test | +| New field with `ignorable: true` | Low | Safe at all versions | + +**Proactive early warning:** A scheduled GitHub Actions workflow (e.g. weekly) that detects new Kafka releases and opens a PR in `kroxylicious/kafka-patches` bumping the `kafka-clients` version automatically: + +1. Queries Maven Central's search API for the latest release of `org.apache.kafka:kafka-clients`. +2. Compares the result against the current `kafka-clients` version in `kroxylicious/kafka-patches`. +3. If a newer version exists and no open PR already targets it, opens a PR bumping the version. +4. CI runs on the PR - rebuilding the shaded artifact and running the fidelity tests against the new `kafka-clients`. + +If CI passes, the PR can be reviewed and merged with minimal effort. If CI fails, the PR surfaces exactly what broke - a fidelity test failure identifies a wire incompatibility; an `EXPECTED_SPEC_COUNT` failure flags a spec change requiring review. Either way, the problem is visible and actionable immediately, before anyone has to manually discover the new release. + +#### Drift in shaded classes + +With shading, upstream changes to hand-coded classes like `MemoryRecords` and `RecordBatch` are absorbed automatically when `kafka-clients` is bumped. Drift is detected through two complementary mechanisms: + +1. **Wire-level fidelity tests:** Any change to serialisation behaviour surfaces as a test failure. These tests serialise with our shaded classes and deserialise with the original `kafka-clients` classes (and vice versa), so any divergence in the binary format is caught immediately. + +2. **Release notes review:** The risk case is a behavioural change that does not affect the wire format - for example, a change to validation logic or error handling in `MemoryRecords`. This cannot be caught by tests alone and requires reviewing the `kafka-clients` changelog when bumping the version, which is part of the standard upgrade process. + +The randomised fidelity tests partially mitigate the second risk by exercising non-default field values, but changelog review remains essential. + +### Enhancements to the generator + +#### Type-safe request-response pairing + +This is an API design decision that must be made before the namespace is first published as stable - it cannot be added later without a breaking change. + +Since the generator knows the request/response pairing (via `ApiKeys`), it can emit typed interfaces: + +```java +interface ApiRequestMessage extends ApiMessage {} +interface ApiResponseMessage extends ApiMessage {} + +class ProduceRequestData implements ApiRequestMessage {} +``` + +This makes `FilterContext.sendRequest()` type-safe - `context.sendRequest(header, produceRequest)` returns `CompletionStage` with no cast required. The pairing information already exists in `ApiKeys`; emitting the `implements` clause is a small generator change. It is not required for correctness and can be deferred if it complicates the initial delivery, but the decision to include or exclude it must be made upfront. + +#### Post-parity improvements + +These can be delivered after the initial release: + +- **Javadoc from IDL `about` strings:** The JSON specs already include documentation strings on most fields. The generator can emit these as Javadoc on getters/setters. +- **`aliases` field in the IDL:** An optional field on spec field definitions (e.g. `"aliases": ["oldFieldName"]`) allowing Kroxylicious to maintain stable getter/setter names even if Kafka renames a field in a future IDL version. Requires schema model changes and generator work. + +### Migration for filter developers + +The namespace migration is a breaking change: all `org.apache.kafka.*` imports become `io.kroxylicious.kafka.*` (with the same sub-package hierarchy preserved). This requires: + +1. A migration guide documenting the package mapping +2. An automated migration script (e.g. `sed`-based) to ease the import update +3. Following the project's deprecation policy: the old imports are announced as deprecated in the CHANGELOG, and become eligible for removal no earlier than the third minor release after that announcement and at least three months later + +### Keeping current with upstream + +When Kafka releases a new version: + +1. Bump `kafka-clients` in `kroxylicious/kafka-patches`. +2. The build reshades the new `kafka-clients` jar - all classes are relocated automatically. +3. Run fidelity tests. If they pass, the new version is wire-compatible. If they fail, the failure identifies exactly which class changed behaviour. +4. If any classes were relocated in the new release, add stability rules to the shading configuration to keep the Kroxylicious namespace unchanged. +5. Update `EXPECTED_SPEC_COUNT` if the spec count changed. +6. Publish the new artifact and bump its version in the Kroxylicious repo. + + +## Affected projects + +All modules that directly reference `org.apache.kafka.*` classes require import updates. The modules with structural changes beyond import updates are: + +- `kroxylicious-api` - `ByteBufAccessor` updated to implement the new `Readable`/`Writable` interfaces; filter interface signatures updated to reference the new namespace +- `kroxylicious-runtime` - codec and Kafka exception mapper updated for the new namespace +- `kroxylicious-krpc-plugin` - FreeMarker templates updated to reference the new namespaces; `ApiKeys` dependency replaced by the new artifact + +Modules requiring import updates only: `kroxylicious-filters`, `kroxylicious-filter-test-support`, `kroxylicious-integration-test-support`. + +The shaded artifact published from `kroxylicious/kafka-patches` is consumed as a regular Maven dependency in the affected modules. `kroxylicious-docs` requires a migration guide. + +## Compatibility + +This is a breaking import change for filter developers - all `org.apache.kafka.*` references in filter code must be updated to `io.kroxylicious.kafka.*`. The wire protocol is unchanged; Kroxylicious continues to interoperate with all Kafka clusters it currently supports, and correctness is proven by the verification harness. `kafka-clients` becomes a dependency of `kroxylicious/kafka-patches` (for building and testing the shaded artifact) and is removed as a compile/runtime dependency of the core. + +## Rejected alternatives + +### Extend `kroxylicious-krpc-plugin` FreeMarker templates to generate `*Data` classes + +Exploratory work on the [`piotrpdev/kroxylicious#refactor/generate-data-class`](https://github.com/piotrpdev/kroxylicious/tree/refactor/generate-data-class) branch proved this is technically feasible - FreeMarker templates were written that generate compilable, wire-compatible `*Data` classes for all 198 specs, with 490 tests passing including byte-level fidelity tests against the real Kafka classes across all protocol versions. However, the templates are complex (~2,200 lines of FreeMarker) and represent a second, independent implementation of the serialisation logic which must be kept in sync with any changes Kafka makes to the wire encoding. The community call favoured using Kafka's battle-tested generator instead: it already produces correct code for all specs, comes with its own test suite, and staying close to Kafka's implementation makes absorbing upstream generator changes easier. The FreeMarker-based `kroxylicious-krpc-plugin` continues to generate filter interfaces, invokers, and decoders - its role is unchanged. + +### Narrow scope - fork only `record.*` and `protocol.*` infrastructure + +This directly addresses the 4.3 breakage without touching the `*Data` classes. However, it leaves the root problem partially in place and means filter developers would face a second breaking change when the `*Data` classes are eventually moved. The community call agreed that one breaking change is better than two. + +### Thin adapter/shim classes that delegate to Kafka implementations + +Wrapping each Kafka class with a Kroxylicious-owned adapter that delegates method calls to the underlying Kafka class. This breaks down for classes with nested types (e.g. `FetchRequestData.FetchTopic`) - the adapter must unwrap and rewrap at every boundary, making it neither cheap nor simple. Discussed and rejected in the community call. + +### Reflection-based class generation + +Automatically generating wrapper classes using reflection over the Kafka originals. This has the same nesting problem as adapters, plus additional fragility: reflection cannot capture generic type parameters, default values, or version-conditional logic. Rejected as too fragile and too complex. + +### Pure shading without patches + +Using the Maven Shade Plugin alone to relocate `kafka-clients` packages, without retaining any ability to modify `MessageGenerator`. This handles the initial namespace migration cleanly and is invisible to file moves in Kafka's source tree. However, it provides no access to `MessageGenerator` and therefore cannot support generator-level enhancements: `aliases`, Javadoc from IDL `about` strings, and type-safe request-response pairing all require modifying the generator and regenerating from source. The proposed approach uses shading for namespace relocation but first applies patches from `kroxylicious/kafka-patches` for when enhancements are needed. + +### Pure patch-based approach without shading (previous iteration) + +Cloning `apache/kafka` at a pinned tag and applying source-level patches to relocate classes before building and publishing. This was the previous iteration of this proposal. It was superseded because changing a Java file's `package` statement requires the file to live in a directory matching its new package for the compiler - meaning every package patch implicitly requires a file move too. This makes the approach a transformation pipeline rather than simple patches, with complexity that shading handles automatically at the bytecode level. + +### Fork the `apache/kafka` repository into the Kroxylicious GitHub org + +Maintaining a `kroxylicious/kafka` fork with Kroxylicious-specific commits on top of upstream. This gives a standard Git workflow (edit files, commit, push) and IDE support, and Git's merge/rebase machinery handles context drift more gracefully than patches do. However, it adds a repository to the org that must be kept in sync, the fork's diff from upstream is implicit rather than inspectable, and there is a risk of the fork accumulating unintentional drift. Storing generator patches in `kroxylicious/kafka-patches` achieves the same generator-modification capability with less overhead. From ae936be3cb7fd820f223317a284756e9bea30b28 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Piotr=20P=C5=82aczek?= Date: Wed, 24 Jun 2026 13:47:24 +0100 Subject: [PATCH 2/7] nit: the licence is comment in the source, so does not end up in the bytecode. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Piotr Płaczek Signed-off-by: Piotr Płaczek --- proposals/116-kafka-api-migration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/116-kafka-api-migration.md b/proposals/116-kafka-api-migration.md index 6525eb7..4007ae9 100644 --- a/proposals/116-kafka-api-migration.md +++ b/proposals/116-kafka-api-migration.md @@ -33,7 +33,7 @@ For a 1.0 release, Kroxylicious needs a stable public API surface that the proje Every type that appears in Kroxylicious's public API should live in a Kroxylicious namespace. This means a single breaking change for filter developers - one import migration - rather than a drip of repeated breakage as Kafka reorganises its internals. -"Own the surface" does not mean "rewrite everything from scratch." The shaded and generated artifacts redistribute `kafka-clients` bytecode under the same Apache 2.0 licence. The original Apache Software Foundation copyright notices are preserved in the bytecode. The underlying implementations can be replaced with Kroxylicious-native ones incrementally, gated by the verification harness proving each replacement is correct. +"Own the surface" does not mean "rewrite everything from scratch." The shaded and generated artifacts redistribute `kafka-clients` bytecode under the same Apache 2.0 licence. The original Apache Software Foundation copyright notices are preserved. The underlying implementations can be replaced with Kroxylicious-native ones incrementally, gated by the verification harness proving each replacement is correct. ### Approach: shade `kafka-clients` via a dedicated repo From e29f57a7eb1070ca27032c3b61fc2f9ee60f476b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Piotr=20P=C5=82aczek?= Date: Wed, 24 Jun 2026 13:54:30 +0100 Subject: [PATCH 3/7] It is not just moves and removals that are problematic. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Piotr Płaczek --- proposals/116-kafka-api-migration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/116-kafka-api-migration.md b/proposals/116-kafka-api-migration.md index 4007ae9..a89f62e 100644 --- a/proposals/116-kafka-api-migration.md +++ b/proposals/116-kafka-api-migration.md @@ -21,7 +21,7 @@ The `kroxylicious-krpc-plugin` already parses the JSON IDL specs independently u ## Motivation -Kafka 4.3.0 moved 37 classes from `org.apache.kafka.common.record.*` to `org.apache.kafka.common.record.internal.*` ([KAFKA-20128](https://issues.apache.org/jira/browse/KAFKA-20128)). Upgrading Kroxylicious to 4.3.0 required import changes across 80 Java files in 9 modules. This is a symptom of a deeper problem: Kroxylicious depends on Kafka internal APIs that Kafka is free to move or remove at any time. +Kafka 4.3.0 moved 37 classes from `org.apache.kafka.common.record.*` to `org.apache.kafka.common.record.internal.*` ([KAFKA-20128](https://issues.apache.org/jira/browse/KAFKA-20128)). Upgrading Kroxylicious to 4.3.0 required import changes across 80 Java files in 9 modules. This is a symptom of a deeper problem: Kroxylicious depends on Kafka internal APIs that Kafka is free to change, move or remove at any time. Kafka does not consider these packages to be stable public API. The project has no obligation to maintain backward compatibility, and further reorganisations are likely. Each one forces a breaking change on every Kroxylicious filter developer - internal and external. From c1125ea85947b89e0b2cf2f7a5ae9101fb428801 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Piotr=20P=C5=82aczek?= Date: Thu, 25 Jun 2026 15:44:10 +0100 Subject: [PATCH 4/7] Where a field is nullable, we'd need to test the null case too. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Piotr Płaczek --- proposals/116-kafka-api-migration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/116-kafka-api-migration.md b/proposals/116-kafka-api-migration.md index a89f62e..2dc02d2 100644 --- a/proposals/116-kafka-api-migration.md +++ b/proposals/116-kafka-api-migration.md @@ -68,7 +68,7 @@ The verification harness is a prerequisite, not a follow-up. Correctness must be For every spec and every supported protocol version: serialise a message with our generated classes, deserialise with Kafka's, and assert byte-level equality - then reverse the direction. This is the ground truth. If bytes we produce are consumed correctly by Kafka's classes and vice versa, we are wire-compatible. -Exploratory work on the [`piotrpdev/kroxylicious#refactor/generate-data-class`](https://github.com/piotrpdev/kroxylicious/tree/refactor/generate-data-class) branch built a prototype of this infrastructure covering all 198 specs: parameterised fidelity tests at every valid version, randomised field-value tests, jqwik property-based tests for full-depth message instances, and a compile-time spec count guard. That prototype was designed around the FreeMarker template approach (see ["Rejected Alternatives"](#rejected-alternatives)), where generated classes were compiled at test time via `ToolProvider.getSystemJavaCompiler()` and loaded via `URLClassLoader`. With the shading approach the classes are already on the classpath at build time, so only the class-loading mechanism changes - the verification logic itself carries over unchanged. +Exploratory work on the [`piotrpdev/kroxylicious#refactor/generate-data-class`](https://github.com/piotrpdev/kroxylicious/tree/refactor/generate-data-class) branch built a prototype of this infrastructure covering all 198 specs: parameterised fidelity tests at every valid version, randomised field-value tests (including explicit coverage of `null` for nullable fields), jqwik property-based tests for full-depth message instances, and a compile-time spec count guard. That prototype was designed around the FreeMarker template approach (see ["Rejected Alternatives"](#rejected-alternatives)), where generated classes were compiled at test time via `ToolProvider.getSystemJavaCompiler()` and loaded via `URLClassLoader`. With the shading approach the classes are already on the classpath at build time, so only the class-loading mechanism changes - the verification logic itself carries over unchanged. These tests live in `kroxylicious/kafka-patches` alongside the shading configuration, with `kafka-clients` as a test-scoped dependency. They run as part of that repo's CI pipeline - when `kafka-clients` is bumped, they immediately report any incompatibility before a new artifact is published. From 7925cb246c12c86431cf960029cbf64d08a7d211 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Piotr=20P=C5=82aczek?= Date: Thu, 25 Jun 2026 16:03:10 +0100 Subject: [PATCH 5/7] Presumably, we could codegen a ApiKeys from the RPC definitions themselves. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Piotr Płaczek --- proposals/116-kafka-api-migration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/116-kafka-api-migration.md b/proposals/116-kafka-api-migration.md index 2dc02d2..aaeff8e 100644 --- a/proposals/116-kafka-api-migration.md +++ b/proposals/116-kafka-api-migration.md @@ -43,7 +43,7 @@ A dedicated repo in the Kroxylicious GitHub org (`kroxylicious/kafka-patches`) s Published artifact versions use the Kafka version as a base with a qualifier suffix (e.g. `4.3.0-kroxy-0`, `4.3.0-kroxy-1` for Kroxylicious-only fixes, `4.4.0-kroxy-0` when bumping to a new Kafka release). This makes the Kafka baseline immediately visible in any dependency declaration. -The `kroxylicious-krpc-plugin` currently depends on `kafka-clients` solely for `ApiKeys`. After this change it depends on the new artifact instead. Its role in generating filter interfaces, invokers, and decoders is otherwise unchanged. +The `kroxylicious-krpc-plugin` currently depends on `kafka-clients` solely for `ApiKeys`. After this change it depends on the new artifact instead (or we could codegen an `ApiKeys` from the RPC definitions themselves). Its role in generating filter interfaces, invokers, and decoders is otherwise unchanged. ### Shading configuration From ccadf55cdd5c32d1cee2b64e3f14f40bb8c509f4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Piotr=20P=C5=82aczek?= Date: Thu, 25 Jun 2026 16:10:54 +0100 Subject: [PATCH 6/7] We've defined the persona name as 'Filter Authors'. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Piotr Płaczek --- proposals/116-kafka-api-migration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/116-kafka-api-migration.md b/proposals/116-kafka-api-migration.md index aaeff8e..bb11c07 100644 --- a/proposals/116-kafka-api-migration.md +++ b/proposals/116-kafka-api-migration.md @@ -23,7 +23,7 @@ The `kroxylicious-krpc-plugin` already parses the JSON IDL specs independently u Kafka 4.3.0 moved 37 classes from `org.apache.kafka.common.record.*` to `org.apache.kafka.common.record.internal.*` ([KAFKA-20128](https://issues.apache.org/jira/browse/KAFKA-20128)). Upgrading Kroxylicious to 4.3.0 required import changes across 80 Java files in 9 modules. This is a symptom of a deeper problem: Kroxylicious depends on Kafka internal APIs that Kafka is free to change, move or remove at any time. -Kafka does not consider these packages to be stable public API. The project has no obligation to maintain backward compatibility, and further reorganisations are likely. Each one forces a breaking change on every Kroxylicious filter developer - internal and external. +Kafka does not consider these packages to be stable public APIs. The project has no obligation to maintain backward compatibility, and further reorganisations are likely. Because these unstable Kafka classes are exposed through our public API, we cannot simply absorb these changes beneath our abstraction layer. Instead, every upstream reorganization forces a breaking change directly onto **Filter Authors** who rely on the Filter API to implement their own custom business logic. For a 1.0 release, Kroxylicious needs a stable public API surface that the project controls. The Kafka protocol wire format itself is stable (governed by KIPs), but the Java classes representing that format are not. From 2ac9d2b9985a31abac32672d4950a60a0b25402c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Piotr=20P=C5=82aczek?= Date: Thu, 25 Jun 2026 16:16:06 +0100 Subject: [PATCH 7/7] Shading can handle repackaging, but AFAIK, can't handle things like method renames MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Piotr Płaczek --- proposals/116-kafka-api-migration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/116-kafka-api-migration.md b/proposals/116-kafka-api-migration.md index bb11c07..3a04911 100644 --- a/proposals/116-kafka-api-migration.md +++ b/proposals/116-kafka-api-migration.md @@ -37,7 +37,7 @@ Every type that appears in Kroxylicious's public API should live in a Kroxylicio ### Approach: shade `kafka-clients` via a dedicated repo -The Maven Shade Plugin takes compiled classes from `kafka-clients` and republishes them under a Kroxylicious namespace. All cross-references between relocated classes are rewritten automatically at the bytecode level. This means Kafka package reorganisations - including moves to an `internal` subpackage like KAFKA-20128 - are completely invisible. +The Maven Shade Plugin takes compiled classes from `kafka-clients` and republishes them under a Kroxylicious namespace. All cross-references between relocated classes are rewritten automatically at the bytecode level. This means Kafka package reorganisations - including moves to an `internal` subpackage like KAFKA-20128 - are completely invisible. Breaking code changes such as method renames or signature changes are addressed by adding patches that overload the methods or wrap them appropriately. A dedicated repo in the Kroxylicious GitHub org (`kroxylicious/kafka-patches`) stores the shading configuration and any generator patches needed for future enhancements. When generator enhancements are implemented, CI will apply the patches against the Kafka generator source, run the patched generator to produce enhanced `*Data` source, compile, and shade the result alongside the rest of `kafka-clients`.