[26.2] Iceberg support for decoded message keys and headers#1804
[26.2] Iceberg support for decoded message keys and headers#1804kbatuigas wants to merge 10 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
d77121c to
0127264
Compare
0127264 to
f2690c7
Compare
|
|
||
| == Configure key, value, and header translation | ||
|
|
||
| For Redpanda clusters version 26.2 and later, in addition to the <<supported-iceberg-modes,supported modes>>, `redpanda.iceberg.mode` also accepts a section-based syntax that lets you independently configure how Redpanda translates the record key, value, and headers into the Iceberg table. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes are shorthand for common combinations of these sections. |
There was a problem hiding this comment.
@wdberkeley could you help confirm that this will also be live for the next Cloud release and that we don't need to conditionalize it for self-managed only? (Not sure which Cloud SME to tag)
|
|
||
| |=== | ||
|
|
||
| By default (`layout=flat`), Redpanda places each decoded value field as a top-level column in the generated table, alongside the `redpanda` system struct. If a decoded value field is named `redpanda`, Redpanda moves it into the `redpanda` system struct as a `data` field, to avoid colliding with the record metadata column of the same name. |
There was a problem hiding this comment.
@wdberkeley I think there is a discrepancy here with the TOI, could you please confirm the behavior for the system struct?
|
|
||
| == Iceberg: Decoded message keys and headers | ||
|
|
||
| You can now independently control how Redpanda translates a record's key, value, and headers into the Iceberg table using the `redpanda.iceberg.mode` topic property. A new section-based syntax lets you schema-decode message keys (the same way as values), store keys or values as UTF-8 strings, and decode header values from raw bytes to strings. |
There was a problem hiding this comment.
'lets you decode both keys and values based on a well known schema in the Schema Registry, store keys ....'
| === value_schema_id_prefix | ||
|
|
||
| Creates an Iceberg table whose structure matches the Redpanda schema for the topic, with columns corresponding to each field. You must register a schema in the xref:manage:schema-reg/schema-reg-overview.adoc[Schema Registry] and producers must write to the topic using the Schema Registry wire format. | ||
| Creates an Iceberg table whose structure matches the Redpanda schema for the topic, with columns corresponding to each field. You must register a schema in xref:manage:schema-reg/schema-reg-overview.adoc[Schema Registry] and producers must write to the topic using the Schema Registry wire format. |
There was a problem hiding this comment.
Does the 'wire format' still denote unambigously the byte prefix now that Confluent introduced the new approach @wdberkeley?
|
|
||
| [IMPORTANT] | ||
| ==== | ||
| Configuring anything beyond `key:mode=binary` and `headers:value_type=binary` requires every broker in the cluster to be running Redpanda version 26.2 or later. Until all brokers are upgraded, Redpanda rejects these configurations with an error. |
There was a problem hiding this comment.
I feel like we could put this further down. gets in the way a bit
| | `binary` | ||
| a| `string` stores the field as a UTF-8 string, replacing invalid bytes with the Unicode replacement character (`U+FFFD`). | ||
|
|
||
| See <<key-type-by-key-mode,Resulting `redpanda.key` type by `key` mode>> and <<value-type-by-value-mode,Resulting value field type by `value` mode>> for the resulting column types in the generated table. |
There was a problem hiding this comment.
Huh? This is confusing.
I think we're burying the lead here @kbatuigas .
Let's describe the power of the new section structure first and how it works, including the examples for key,value and header, and the 'layout' optiojs, before mentioning the equivalent resulting values using the 'mode' shorthands. This is really reference material that should be at the bottom , or maybe just above the 'validation rules'.
| ---- | ||
|
|
||
| NOTE: If a section-based configuration is equivalent to one of the modes described in <<supported-iceberg-modes,Supported Iceberg modes>>, `rpk topic describe` displays it using that mode's name instead of the section syntax. | ||
|
|
There was a problem hiding this comment.
@kbatuigas I wonder if we still need this warning in 26.2 docs taking up major real estate given it happened in 25.3.
wdberkeley
left a comment
wdberkeley
left a comment
There was a problem hiding this comment.
Looks good. Thanks for doing this... it's a lot of work, there's a lot of new stuff.
3 participants
Description
This pull request updates the documentation to introduce and clarify new capabilities for the
redpanda.iceberg.modetopic property, allowing users to independently configure how record keys, values, and headers are translated into Iceberg tables. It also improves guidance and references for configuring these options and updates related descriptions and links.Iceberg integration enhancements:
redpanda.iceberg.mode, enabling independent configuration of key, value, and header translation, including schema-decoding and UTF-8 string options.property-overrides.jsonto explain the new configuration options and reference the improved documentation.Documentation improvements:
Resolves https://redpandadata.atlassian.net/browse/2194
Review deadline: 23 July
Page previews
Checks