diff --git a/docs-data/property-overrides.json b/docs-data/property-overrides.json index 10195de9bc..c0bf16ad3a 100644 --- a/docs-data/property-overrides.json +++ b/docs-data/property-overrides.json @@ -1830,6 +1830,16 @@ ], "config_scope": "topic" }, + "redpanda.schema.registry.context": { + "description": "The xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry context] Redpanda uses to resolve the schemas for records in this topic, for example when translating the topic to an Iceberg table. Defaults to the Schema Registry default context (`.`).", + "related_topics": [ + "xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context]" + ], + "config_scope": "topic", + "category": "schema-registry", + "type": "string", + "default": "." + }, "redpanda.storage.mode": { "description": "The storage mode for a topic. Determines how topic data is stored and whether it is eligible for upload to object storage.\n\nAccepted values:\n\n* `local`: Topic data is stored only on the broker's local disk. Object storage upload is disabled for the topic, regardless of cluster-level Tiered Storage settings.\nifndef::env-cloud[]\n* `tiered`: Topic data is stored on local disk and also uploaded to object storage. Enables xref:manage:tiered-storage.adoc[Tiered Storage] for the topic.\n* `cloud`: Topic data is stored in object storage using the xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics] architecture. Local storage is used only as a write buffer.\nendif::[]\nifdef::env-cloud[]\n* `tiered`: Topic data is stored on local disk and also uploaded to object storage. Enables Tiered Storage for the topic.\n* `cloud`: Topic data is stored in object storage using the Cloud Topics architecture. Local storage is used only as a write buffer.\nendif::[]\n* `unset`: Specifies that the topic's storage mode is unset, regardless of the cluster default. The topic may still have Tiered Storage enabled through the legacy properties `redpanda.remote.read` and `redpanda.remote.write`.\n\nThis property overrides the cluster-wide config_ref:default_redpanda_storage_mode,true,properties/cluster-properties[] setting for individual topics.", "related_topics": [ diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index a0d27a5b3b..f7c123d79e 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -7,9 +7,20 @@ This topic includes new content added in version {page-component-version}. For a * xref:cloud-data-platform:get-started:whats-new-cloud.adoc[] * xref:cloud-data-platform:get-started:cloud-overview.adoc#redpanda-cloud-vs-self-managed-feature-compatibility[Redpanda Cloud vs Self-Managed feature compatibility] +== Iceberg translation with Schema Registry contexts + +Iceberg-enabled topics can now resolve schemas from a specific xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry context] instead of always using the default context. Set the new `redpanda.schema.registry.context` topic property to bind a topic's Iceberg translation to a context. Binding to a context is useful for multi-tenant or per-environment schema isolation. + +See xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context] for configuration examples and considerations for changing a topic's context. == Kafka 4.x client compatibility Redpanda version 26.2 validates a broader set of non-Java Apache Kafka® clients at their current versions that support Kafka 4.x, in addition to the Apache Kafka Java client. Newly validated clients include `confluent-kafka-go` and Sarama for Go, and `kafka-python` and `confluent-kafka-python` for Python. Validation confirms connectivity and correctness across core Kafka APIs, such as produce, consume, and transaction operations. See xref:develop:kafka-clients.adoc[Kafka Compatibility] for the full list of validated clients and known exceptions. + +== New configuration properties + +**Iceberg:** + +* xref:reference:properties/topic-properties.adoc#redpanda-schema-registry-context[`redpanda.schema.registry.context`]: Bind a topic's Iceberg translation to a specific Schema Registry context diff --git a/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc b/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc index 6104223a1c..0139bcbc85 100644 --- a/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc +++ b/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc @@ -27,6 +27,7 @@ Use this page to: If Redpanda encounters an error while writing a record to the Iceberg table, Redpanda by default writes the record to a separate DLQ Iceberg table named `~dlq`. The following can cause errors to occur when translating records in the `value_schema_id_prefix` and `value_schema_latest` modes to the Iceberg table format: - Redpanda cannot find the embedded schema ID in the Schema Registry. +- Redpanda cannot resolve the record's schema ID within the topic's configured Schema Registry context. For example, the schema exists only in a different context. See xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context]. - Redpanda fails to translate one or more schema data types to an Iceberg type. - In `value_schema_id_prefix` mode, you do not use the Schema Registry wire format with the magic byte. diff --git a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc index 672d12cb7e..fea7fbe211 100644 --- a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -109,6 +109,43 @@ value_schema_latest:subject=,protobuf_name= --topic-config redpanda.schema.registry.context= +---- + +.Set the Schema Registry context on an existing topic +[,bash] +---- +rpk topic alter-config --set redpanda.schema.registry.context= +---- + +The context name must start with a period (`.`), for example `.staging`. If you don't set this property, Redpanda resolves schemas in the default context (`.`). + +If Redpanda cannot resolve a record's schema within the configured context, it doesn't translate the record and instead writes it to a dead-letter queue (DLQ) table. See xref:manage:iceberg/iceberg-troubleshooting.adoc[Troubleshoot Iceberg Topics]. + +[IMPORTANT] +==== +Redpanda resolves schemas using the topic's current `redpanda.schema.registry.context` value, not the context that was active when a record was ingested. Changing the context on a topic that has pending, uncommitted Iceberg translation entries can cause resolution errors or unexpected DLQ routing for records still being translated. + +To change the Schema Registry context on a topic that is actively translating to Iceberg: + +. Disable Iceberg translation for the topic. +. Wait for pending translation entries to commit. +. Change `redpanda.schema.registry.context`. +. Re-enable Iceberg translation. +==== + == How Iceberg modes translate to table format ifndef::env-cloud[] diff --git a/modules/manage/pages/schema-reg/schema-reg-contexts.adoc b/modules/manage/pages/schema-reg/schema-reg-contexts.adoc index 85ebdfa7e8..1b070204c6 100644 --- a/modules/manage/pages/schema-reg/schema-reg-contexts.adoc +++ b/modules/manage/pages/schema-reg/schema-reg-contexts.adoc @@ -70,7 +70,12 @@ endif::env-cloud[] + [IMPORTANT] ==== -The `schema_registry_enable_qualified_subjects` property defaults to `false`, so you must explicitly enable it. After changing it, you must restart your brokers for the change to take effect. If qualified subjects are still being treated as literal names after enabling the flag, a broker restart is most likely still needed. +ifndef::env-cloud[] +In Redpanda 26.2 and later, `schema_registry_enable_qualified_subjects` is enabled by default. In earlier versions, it defaults to `false` and you must enable it explicitly. Changing this property requires a broker restart to take effect. If qualified subjects are still being treated as literal names, a broker restart is most likely still needed. +endif::env-cloud[] +ifdef::env-cloud[] +Enabling `schema_registry_enable_qualified_subjects` requires a broker restart to take effect. See <>. +endif::env-cloud[] ==== + * If you use Schema Registry ACLs, ensure the principal interacting with a context has the appropriate permissions. See <>. @@ -79,7 +84,6 @@ The `schema_registry_enable_qualified_subjects` property defaults to `false`, so * *Non-Java SerDe clients*: Not supported in Schema Registry contexts. * *Server-side schema ID validation*: Schema ID validation using Kafka record headers does not support contexts. However, schema ID validation using magic byte and prefix are supported. -* *Iceberg topics*: You cannot use schemas within a context for Iceberg Topics. * *`referencedby` endpoint*: `GET /subjects/\{subject\}/versions/\{version\}/referencedby` returns a list of bare schema IDs with no context information. When references span contexts, it is not possible to determine which context each returned ID belongs to. * *Cross-context isolation*: Contexts provide organizational and ID-space isolation, but do not prevent cross-context schema references. There is no mechanism to block schemas in one context from referencing schemas in another context. * *Default context cannot be deleted*: You cannot delete the default context (`.`). @@ -192,7 +196,7 @@ Unqualified references in schema definitions resolve to the same context as the == Enable Schema Registry contexts ifndef::env-cloud[] -To enable contexts, set the `schema_registry_enable_qualified_subjects` cluster configuration property to `true`: +In Redpanda 26.2 and later, Schema Registry contexts are enabled by default. In earlier versions, enable them by setting the `schema_registry_enable_qualified_subjects` cluster configuration property to `true`: [source,bash] ---- diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index e99bff7283..31d8d887b1 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -155,4 +155,10 @@ include::reference:partial$properties/topic-properties.adoc[tags=category-remote Integrate Redpanda topics as Iceberg tables. -include::reference:partial$properties/topic-properties.adoc[tags=category-iceberg-integration;!deprecated;!exclude-from-docs] \ No newline at end of file +include::reference:partial$properties/topic-properties.adoc[tags=category-iceberg-integration;!deprecated;!exclude-from-docs] + +== Schema Registry and Validation Properties + +The following properties control server-side schema ID validation and schema resolution for topics when using Schema Registry. + +include::reference:partial$properties/topic-properties.adoc[tags=category-schema-registry;!deprecated;!exclude-from-docs] \ No newline at end of file diff --git a/modules/reference/partials/properties/topic-properties.adoc b/modules/reference/partials/properties/topic-properties.adoc index d3261d2762..a4a3021cc9 100644 --- a/modules/reference/partials/properties/topic-properties.adoc +++ b/modules/reference/partials/properties/topic-properties.adoc @@ -1190,6 +1190,44 @@ endif::[] // end::category-iceberg-integration[] +// tag::category-schema-registry[] +=== redpanda.schema.registry.context + +The xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry context] Redpanda uses to resolve the schemas for records in this topic, for example when translating the topic to an Iceberg table. Defaults to the Schema Registry default context (`.`). + +[cols="1s,2a"] +|=== +| Property | Value + +| Type +| `string` + + + +| Default +| +ifdef::env-cloud[] +Available in the Redpanda Cloud Console +endif::[] +ifndef::env-cloud[] +`.` +endif::[] + +| Nullable +| No + +ifndef::env-cloud[] +| Restored on xref:manage:whole-cluster-restore.adoc[Whole Cluster Restore] +| Yes +endif::[] + +| Related topics +|xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context] + +|=== + +// end::category-schema-registry[] + // tag::category-schema-registry[] === redpanda.key.schema.id.validation