Fanout example with WASM #1563
Fanout example with WASM #1563
Conversation
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
📝 WalkthroughWalkthroughThis documentation update expands the data transforms guide with new content on routing and fan-out patterns. The changes add a dedicated multi-topic fan-out section that provides end-to-end examples in Go, Rust, and JavaScript, demonstrating how to route records from a single input topic to multiple output topics. The examples include Schema Registry integration, batched updates with routing fields, and per-topic schema ID handling. JavaScript guidance is updated to clarify that multi-topic fan-out is not supported by its SDK. Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Warning Review ran into problems🔥 ProblemsErrors were encountered while retrieving linked issues. Errors (1)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
modules/develop/pages/data-transforms/build.adoc (1)
432-437: Fix undefined identifiers in Go JSON validation example.
wandeare undefined; should usewriterandevent. This will fail for readers copying the snippet.🐛 Proposed fix
- if json.Valid(event.Record().Value) { - return w.Write(e.Record()) + if json.Valid(event.Record().Value) { + return writer.Write(event.Record()) } // Send invalid records to separate topic - return writer.Write(e.Record(), transform.ToTopic("invalid-json")) + return writer.Write(event.Record(), transform.ToTopic("invalid-json"))
🧹 Nitpick comments (2)
modules/develop/pages/data-transforms/build.adoc (2)
489-504: Use xref with empty brackets for link text.Project guidance prefers
xref:...[]so the target title renders automatically. Please update the new links in this block.♻️ Proposed fix
-xref:develop:data-transforms/configure.adoc[Configure the transform] with multiple output topics: +xref:develop:data-transforms/configure.adoc[] with multiple output topics: -NOTE: xref:manage:schema-reg/schema-reg-api.adoc[Register schemas in Schema Registry] before deploying the transform. +NOTE: xref:manage:schema-reg/schema-reg-api.adoc[] before deploying the transform.Based on learnings, “AsciiDoc linking: prefer using xref links with empty brackets (e.g., xref:section/target.adoc[]) because the title is pulled from the referenced document automatically. Avoid hard-coding link text; use xref:...[] to let the target document's title render as the link text when publishing.”
645-686: Refactorstatic mut+unsafetoOnceLockfor safer, idiomatic Rust.This example unnecessarily introduces unsafe code in the documentation. Since the Rust version requirement is "the latest stable version,"
std::sync::OnceLock(stabilized in Rust 1.70.0) is available and should be preferred as the idiomatic approach.♻️ Proposed fix (OnceLock)
-use std::collections::HashMap; -use std::error::Error; +use std::collections::HashMap; +use std::error::Error; +use std::sync::OnceLock; -// Schema IDs for each output topic obtained from Schema Registry -// Register schemas before deploying the transform using the {topic-name}-value naming convention -static mut SCHEMA_IDS: Option<HashMap<String, i32>> = None; +// Schema IDs for each output topic obtained from Schema Registry +// Register schemas before deploying the transform using the {topic-name}-value naming convention +static SCHEMA_IDS: OnceLock<HashMap<String, i32>> = OnceLock::new(); fn main() { let mut schema_ids = HashMap::new(); schema_ids.insert("orders".to_string(), 1); schema_ids.insert("inventory".to_string(), 2); schema_ids.insert("customers".to_string(), 3); - unsafe { - SCHEMA_IDS = Some(schema_ids); - } + let _ = SCHEMA_IDS.set(schema_ids); on_record_written(route_updates); } fn route_updates(event: WriteEvent, writer: &mut RecordWriter) -> Result<(), Box<dyn Error>> { let batch: BatchMessage = serde_json::from_slice(event.record.value().unwrap_or_default())?; - let schema_ids = unsafe { SCHEMA_IDS.as_ref().unwrap() }; + let schema_ids = SCHEMA_IDS.get().unwrap(); for update in batch.updates.iter() { if let Some(&schema_id) = schema_ids.get(&update.table) { write_update(update, schema_id, writer, &event)?; } }
Description
This pull request expands the documentation for data transforms by adding a new section on multi-topic fan-out using the Go and Rust SDKs, and enhances the explanation of writing to specific output topics. The changes provide detailed, practical examples for routing and distributing messages to multiple topics, including Schema Registry integration, and clarify SDK capabilities.
Enhancements to output topic routing documentation:
New multi-topic fan-out example with Schema Registry:
Resolves https://redpandadata.atlassian.net/browse/
Review deadline:
Page previews
Checks