Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 46 additions & 51 deletions docs-data/property-overrides.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion modules/develop/pages/transactions.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,7 @@ Different transactions require different approaches to handling failures within
Transactions are supported on topics with compaction configured. The compaction process removes aborted transaction data from the log. The resulting compacted segment contains only committed data batches (and potentially harmless gaps in the offsets due to skipped batches).

ifndef::env-cloud[]
At a cluster-level, compaction is set when xref:reference:cluster-properties.adoc#log_cleanup_policy[`log_cleanup_policy`] or xref:reference:topic-properties.adoc#cleanuppolicy[`cleanup.policy`] are set to either `compact` or `compact,delete`.
At a cluster-level, compaction is set when xref:reference:cluster-properties.adoc#log_cleanup_policy[`log_cleanup_policy`] or xref:reference:properties/topic-properties.adoc#cleanup-policy[`cleanup.policy`] are set to either `compact` or `compact,delete`.

Optionally, you can enable removal of transactional control batches (commit and abort markers) during compaction by setting xref:reference:properties/cluster-properties.adoc#log_compaction_tx_batch_removal_enabled[`log_compaction_tx_batch_removal_enabled`] to `true`. When enabled, the topic's xref:reference:properties/topic-properties.adoc#delete-retention-ms[`delete.retention.ms`] setting is applied to these markers. For topics with a `compact` only cleanup policy, you must explicitly set `delete.retention.ms` at the topic level. This feature is not applied when Tiered Storage is enabled. See xref:manage:cluster-maintenance/compaction-settings.adoc#transactional-control-batch-removal[Transactional control batch removal].

Expand Down
2 changes: 1 addition & 1 deletion modules/get-started/pages/release-notes/redpanda.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ For end-to-end steps and prerequisites, see xref:manage:security/authentication.
**Storage mode:**

* xref:reference:properties/cluster-properties.adoc#default_redpanda_storage_mode[`default_redpanda_storage_mode`]: Set the default storage mode for new topics (`local`, `tiered`, `cloud`, or `unset`)
* xref:reference:properties/topic-properties.adoc#redpandastoragemode[`redpanda.storage.mode`]: Set the storage mode for an individual topic, superseding the legacy `redpanda.remote.read` and `redpanda.remote.write` properties
* xref:reference:properties/topic-properties.adoc#redpanda-storage-mode[`redpanda.storage.mode`]: Set the storage mode for an individual topic, superseding the legacy `redpanda.remote.read` and `redpanda.remote.write` properties

**Cloud Topics:**

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ While compaction reduces storage needs, Redpanda's compaction (just like Kafka's

== Configure a cleanup policy

A compaction policy may be applied to a cluster or to an individual topic. If both are set, the topic-level policy overrides the cluster-level policy. The cluster-level xref:reference:cluster-properties.adoc#log_cleanup_policy[`log_cleanup_policy`] and the topic-level xref:reference:topic-properties.adoc#cleanuppolicy[`cleanup.policy`] support the following three options:
A compaction policy may be applied to a cluster or to an individual topic. If both are set, the topic-level policy overrides the cluster-level policy. The cluster-level xref:reference:cluster-properties.adoc#log_cleanup_policy[`log_cleanup_policy`] and the topic-level xref:reference:topic-properties.adoc#cleanup-policy[`cleanup.policy`] support the following three options:

* `delete`: Records are deleted from the topic once the specified retention period (time and/or size allocations) is exceeded. This is the default mechanism and is analogous to disabling compaction.
* `compact`: This triggers only cleanup of records with multiple versions. A record that represents the only version for a given key is not deleted.
Expand Down Expand Up @@ -54,7 +54,7 @@ Where:
| Cluster
| The minimum ratio between the number of bytes in dirty segments and the total number of bytes in closed segments that must be reached before a partition's log is eligible for compaction in a compact topic.

| xref:reference:properties/topic-properties.adoc#mincleanabledirtyratio[`min.cleanable.dirty.ratio`]
| xref:reference:properties/topic-properties.adoc#min-cleanable-dirty-ratio[`min.cleanable.dirty.ratio`]
| Topic
| Topic-level override of the cluster-wide dirty ratio threshold.

Expand All @@ -70,11 +70,11 @@ Where:
| Cluster
| The minimum time in milliseconds that a message remains uncompacted in the log. Use to guarantee the minimum length of time that must pass after a message is written before it could be compacted. For example, to provide a lower bound on how long each message will remain in the (uncompacted) head.

| xref:reference:properties/topic-properties.adoc#maxcompactionlagms[`max.compaction.lag.ms`]
| xref:reference:properties/topic-properties.adoc#max-compaction-lag-ms[`max.compaction.lag.ms`]
| Topic
| The maximum amount of time in milliseconds that a message remains ineligible for compaction. Use to guarantee the maximum delay between the time a message is written and the time the message becomes eligible for compaction.

| xref:reference:properties/topic-properties.adoc#mincompactionlagms[`min.compaction.lag.ms`]
| xref:reference:properties/topic-properties.adoc#min-compaction-lag-ms[`min.compaction.lag.ms`]
| Topic
| The minimum time in milliseconds that a message remains uncompacted in the log. Use to guarantee the minimum length of time that must pass after a message is written before it could be compacted. For example, to provide a lower bound on how long each message will remain in the (uncompacted) head.
|===
Expand Down Expand Up @@ -106,7 +106,7 @@ Redpanda runs a scan every `log_compaction_interval_ms`. During each scan:

Compaction also enables deletion of existing records through tombstones. For example, as data is deleted from a source system, clients produce a tombstone record to the log. A tombstone contains a key and the value `null`. Tombstones signal to brokers and consumers that records with the same key prior to it in the log should be deleted.

You can specify how long Redpanda keeps these tombstones for compacted topics using both a cluster configuration property config_ref:tombstone_retention_ms,true,properties/cluster-properties[] and a topic configuration property xref:reference:properties/topic-properties.adoc#deleteretentionms[`delete.retention.ms`]. If both are set, the topic-level tombstone retention policy overrides the cluster-level policy.
You can specify how long Redpanda keeps these tombstones for compacted topics using both a cluster configuration property config_ref:tombstone_retention_ms,true,properties/cluster-properties[] and a topic configuration property xref:reference:properties/topic-properties.adoc#delete-retention-ms[`delete.retention.ms`]. If both are set, the topic-level tombstone retention policy overrides the cluster-level policy.

[NOTE]
====
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ The most commonly configured topic properties fall into these main categories:

Redpanda manages disk space through two main mechanisms: **compaction** (removing duplicate keys) and **retention** (removing old data).

Choose your cleanup strategy with xref:reference:properties/topic-properties.adoc#cleanuppolicy[`cleanup.policy`]:
Choose your cleanup strategy with xref:reference:properties/topic-properties.adoc#cleanup-policy[`cleanup.policy`]:
- `delete` - Remove old data based on time or size limits
- `compact` - Keep only the latest value for each key
- `compact,delete` - Combine both strategies
Expand All @@ -194,25 +194,25 @@ Choose your cleanup strategy with xref:reference:properties/topic-properties.ado

When using `cleanup.policy=compact` or `cleanup.policy=compact,delete`, configure:

- xref:reference:properties/topic-properties.adoc#mincleanabledirtyratio[`min.cleanable.dirty.ratio`] - Control when compaction triggers based on dirty data ratio
- xref:reference:properties/topic-properties.adoc#maxcompactionlagms[`max.compaction.lag.ms`] - Set maximum time before compaction is forced
- xref:reference:properties/topic-properties.adoc#mincompactionlagms[`min.compaction.lag.ms`] - Set minimum time before compaction can occur
- xref:reference:properties/topic-properties.adoc#min-cleanable-dirty-ratio[`min.cleanable.dirty.ratio`] - Control when compaction triggers based on dirty data ratio
- xref:reference:properties/topic-properties.adoc#max-compaction-lag-ms[`max.compaction.lag.ms`] - Set maximum time before compaction is forced
- xref:reference:properties/topic-properties.adoc#min-compaction-lag-ms[`min.compaction.lag.ms`] - Set minimum time before compaction can occur

=== Retention

When using `cleanup.policy=delete` or `cleanup.policy=compact,delete`, configure:

- xref:reference:properties/topic-properties.adoc#retention-bytes[`retention.bytes`] - Maximum size before cleanup (size-based retention)
- xref:reference:properties/topic-properties.adoc#retention-ms[`retention.ms`] - Maximum age before cleanup (time-based retention)
- xref:reference:properties/topic-properties.adoc#segmentbytes[`segment.bytes`] - Control how frequently cleanup can occur by setting segment size
- xref:reference:properties/topic-properties.adoc#segment-bytes[`segment.bytes`] - Control how frequently cleanup can occur by setting segment size

=== Performance

Essential performance tuning properties:

- xref:reference:properties/topic-properties.adoc#writecaching[`write.caching`] - Cache writes for lower latency with `acks=all`
- xref:reference:properties/topic-properties.adoc#maxmessagebytes[`max.message.bytes`] - Set maximum message size
- xref:reference:properties/topic-properties.adoc#replicationfactor[`replication.factor`] - Number of replicas for durability vs. performance
- xref:reference:properties/topic-properties.adoc#write-caching[`write.caching`] - Cache writes for lower latency with `acks=all`
- xref:reference:properties/topic-properties.adoc#max-message-bytes[`max.message.bytes`] - Set maximum message size
- xref:reference:properties/topic-properties.adoc#replication-factor[`replication.factor`] - Number of replicas for durability vs. performance

For complete details about all available topic properties, see xref:reference:properties/topic-properties.adoc[Topic Configuration Properties].

Expand Down
4 changes: 2 additions & 2 deletions modules/manage/pages/kubernetes/k-manage-topics.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -326,15 +326,15 @@ With `write_caching` enabled at the cluster level, Redpanda fsyncs to disk accor

=== Configure at topic level

To override the cluster-level setting at the topic level, set the topic-level property xref:reference:topic-properties.adoc#writecaching[`write.caching`]:
To override the cluster-level setting at the topic level, set the topic-level property xref:reference:properties/topic-properties.adoc#write-caching[`write.caching`]:

.`example-topic.yaml`
[,yaml,indent=0]
----
include::manage:example$kubernetes/topic-crds.feature[tags=write-caching-topic-example,indent=0]
----

With `write.caching` enabled at the topic level, Redpanda fsyncs to disk according to xref:reference:topic-properties.adoc#flushms[`flush.ms`] and xref:reference:topic-properties.adoc#flushbytes[`flush.bytes`], whichever is reached first.
With `write.caching` enabled at the topic level, Redpanda fsyncs to disk according to xref:reference:properties/topic-properties.adoc#flush-ms[`flush.ms`] and xref:reference:properties/topic-properties.adoc#flush-bytes[`flush.bytes`], whichever is reached first.

== Verify a topic

Expand Down
20 changes: 10 additions & 10 deletions modules/manage/pages/schema-reg/schema-id-validation.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -88,8 +88,8 @@ When <<enable-schema-id-validation,schema ID validation is enabled>>, Redpanda u
To customize the subject name strategy per topic, set the following client topic properties:

ifndef::env-cloud[]
* Set xref:reference:properties/topic-properties.adoc#redpandakeyschemavalidation[`redpanda.key.schema.id.validation`] to `true` to enable key schema ID validation for the topic, and set xref:reference:properties/topic-properties.adoc#redpandakeysubjectnamestrategy[`redpanda.key.subject.name.strategy`] to the desired subject name strategy for keys of the topic (default: `TopicNameStrategy`).
* Set xref:reference:properties/topic-properties.adoc#redpandavalueschemavalidation[`redpanda.value.schema.id.validation`] to `true` to enable value schema ID validation for the topic, and set xref:reference:properties/topic-properties.adoc#redpandavaluesubjectnamestrategy[`redpanda.value.subject.name.strategy`] to the desired subject name strategy for values of the topic (default: `TopicNameStrategy`).
* Set xref:reference:properties/topic-properties.adoc#redpanda-key-schema-id-validation[`redpanda.key.schema.id.validation`] to `true` to enable key schema ID validation for the topic, and set xref:reference:properties/topic-properties.adoc#redpanda-key-subject-name-strategy[`redpanda.key.subject.name.strategy`] to the desired subject name strategy for keys of the topic (default: `TopicNameStrategy`).
* Set xref:reference:properties/topic-properties.adoc#redpanda-value-schema-id-validation[`redpanda.value.schema.id.validation`] to `true` to enable value schema ID validation for the topic, and set xref:reference:properties/topic-properties.adoc#redpanda-value-subject-name-strategy[`redpanda.value.subject.name.strategy`] to the desired subject name strategy for values of the topic (default: `TopicNameStrategy`).

[NOTE]
====
Expand All @@ -98,17 +98,17 @@ The `redpanda.` properties have corresponding `confluent.` properties.
|===
| Redpanda property | Confluent property

| xref:reference:properties/topic-properties.adoc#redpandakeyschemavalidation[`redpanda.key.schema.id.validation`]
| xref:reference:properties/topic-properties.adoc#confluentkeyschemavalidation[`confluent.key.schema.validation`]
| xref:reference:properties/topic-properties.adoc#redpanda-key-schema-id-validation[`redpanda.key.schema.id.validation`]
| xref:reference:properties/topic-properties.adoc#confluent-key-schema-validation[`confluent.key.schema.validation`]

| xref:reference:properties/topic-properties.adoc#redpandakeysubjectnamestrategy[`redpanda.key.subject.name.strategy`]
| xref:reference:properties/topic-properties.adoc#confluentkeysubjectnamestrategy[`confluent.key.subject.name.strategy`]
| xref:reference:properties/topic-properties.adoc#redpanda-key-subject-name-strategy[`redpanda.key.subject.name.strategy`]
| xref:reference:properties/topic-properties.adoc#confluent-key-subject-name-strategy[`confluent.key.subject.name.strategy`]

| xref:reference:properties/topic-properties.adoc#redpandavalueschemavalidation[`redpanda.value.schema.id.validation`]
| xref:reference:properties/topic-properties.adoc#confluentvalueschemavalidation[`confluent.value.schema.validation`]
| xref:reference:properties/topic-properties.adoc#redpanda-value-schema-id-validation[`redpanda.value.schema.id.validation`]
| xref:reference:properties/topic-properties.adoc#confluent-value-schema-validation[`confluent.value.schema.validation`]

| xref:reference:properties/topic-properties.adoc#redpandavaluesubjectnamestrategy[`redpanda.value.subject.name.strategy`]
| xref:reference:properties/topic-properties.adoc#confluentvaluesubjectnamestrategy[`confluent.value.subject.name.strategy`]
| xref:reference:properties/topic-properties.adoc#redpanda-value-subject-name-strategy[`redpanda.value.subject.name.strategy`]
| xref:reference:properties/topic-properties.adoc#confluent-value-subject-name-strategy[`confluent.value.subject.name.strategy`]
|===
====
endif::[]
Expand Down
2 changes: 1 addition & 1 deletion modules/manage/partials/tiered-storage.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -1015,7 +1015,7 @@ See also: xref:manage:cluster-maintenance/disk-utilization.adoc#configure-messag

==== Compacted topics in Tiered Storage

When you set xref:reference:topic-properties.adoc#cleanuppolicy[`cleanup.policy`] for a topic to `compact`, nothing gets deleted from object storage based on retention settings. When set to `compact,delete`, compacted segments are deleted from object storage based on `retention.ms` and `retention.bytes`.
When you set xref:reference:properties/topic-properties.adoc#cleanup-policy[`cleanup.policy`] for a topic to `compact`, nothing gets deleted from object storage based on retention settings. When set to `compact,delete`, compacted segments are deleted from object storage based on `retention.ms` and `retention.bytes`.

For compacted topics, Redpanda compacts segments after they have been uploaded to object storage. Redpanda initially uploads all uncompacted segments. It then re-uploads the segments with compaction applied. It's likely that some segments in object storage are not compacted, but the Tiered Storage read path can manage this.

Expand Down
Loading
Loading