feat: interactive Iceberg Mode Explorer with real C++→WASM engine#404
feat: interactive Iceberg Mode Explorer with real C++→WASM engine#404JakeSCahill wants to merge 1 commit into
Conversation
Turn the standalone v26.2 TOI explorer into a docs-integrated, versioned tool whose rule-bearing output is computed by Redpanda's real code, so it can't drift from the product. - src/js/27-iceberg-explorer.js: hydrates the `[iceberg-explorer]` block into the interactive tool (controls, config string, schema, sample record). Theme-aware CSS in src/css/iceberg-explorer.css. - iceberg-editor/: compiles Redpanda's real C++ Avro->Iceberg schema mapper (iceberg::type_to_iceberg from `dev`) to WebAssembly via Emscripten, with a thin Seastar shim. wasm-spike/ is the reproducible feasibility proof; wasm/ is the browser engine build (build.sh + Embind bindings). Fetched sources, deps, and build outputs are git-ignored (reproducible from the scripts). - Version-aware engine loader: loads iceberg-engine-<version> for the page's doc version, falling back to a bundle asset then to the interim JS engine. <body data-version> added in default.hbs to expose the version to JS. - Config DSL string stays JS but is pinned to Redpanda's own iceberg_mode_test.cc format vectors via tests/iceberg-dsl (fixed two drift bugs: missing value `layout`, wrong option order). - CI: build-iceberg-engine.yml builds a versioned engine per Redpanda release tag; iceberg-dsl-conformance.yml runs the DSL vectors and diffs upstream. Requires the `[iceberg-explorer]` block from redpanda-data/docs-extensions-and-macros. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
❌ Deploy Preview for docs-ui failed. Why did it fail? →
|
📝 WalkthroughWalkthroughAdds an interactive Iceberg Mode Explorer with deterministic DSL generation, schema and record rendering, and lazy loading of a versioned WebAssembly translation engine. The engine compiles production C++ Avro-to-Iceberg mapping code and exposes JSON through Embind. New scripts provide dependency fetching, source vendoring, staged WASM builds, and smoke tests. Documentation, preview wiring, static assets, styling, npm conformance tests, and GitHub Actions workflows are included. Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Browser
participant Explorer
participant WASMEngine
participant IcebergMapper
Browser->>Explorer: hydrate explorer
Explorer->>WASMEngine: load versioned engine
Explorer->>WASMEngine: avroToIcebergJson(schema)
WASMEngine->>IcebergMapper: map Avro schema to Iceberg type
IcebergMapper-->>WASMEngine: fields or error
WASMEngine-->>Explorer: JSON result
Explorer-->>Browser: render schema and translated record
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Checkov (3.3.8).github/workflows/build-iceberg-engine.ymlTraceback (most recent call last): .github/workflows/iceberg-dsl-conformance.ymlTraceback (most recent call last): package.jsonTraceback (most recent call last):
🔧 ESLint
ESLint install timed out. The project may have too many dependencies for the sandbox. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🧹 Nitpick comments (1)
iceberg-editor/wasm-spike/shim/seastar/util/defer.hh (1)
32-35: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winStore a decayed callable in the scope guard.
deferpreserves lvalue references, so a guard that outlives or is moved beyond its callable can invoke a dangling reference. Returndeferred_action<std::decay_t<Func>>instead.Proposed fix
+#include <type_traits> + template<typename Func> -deferred_action<Func> defer(Func&& f) { - return deferred_action<Func>(std::forward<Func>(f)); +deferred_action<std::decay_t<Func>> defer(Func&& f) { + return deferred_action<std::decay_t<Func>>(std::forward<Func>(f)); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@iceberg-editor/wasm-spike/shim/seastar/util/defer.hh` around lines 32 - 35, Update the defer function to decay Func when constructing and returning the deferred_action, so the scope guard owns the callable value rather than preserving an lvalue reference. Use std::decay_t<Func> consistently for the return type and deferred_action construction while preserving perfect forwarding of f.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/build-iceberg-engine.yml:
- Line 35: Update the workflow step defining REF and the downstream version
handling to pass event-controlled values through step-level env variables rather
than interpolating them in run scripts; validate both ref and derived version
against the accepted safe format before writing GITHUB_OUTPUT. Also set checkout
persist-credentials to false and restrict workflow permissions to the minimum
required, preserving the existing build behavior for valid inputs.
In @.github/workflows/iceberg-dsl-conformance.yml:
- Line 31: Update the actions/checkout@v4 step in the workflow to disable
credential persistence by setting persist-credentials to false. Leave the
checkout behavior otherwise unchanged.
In `@iceberg-editor/wasm-spike/env.sh`:
- Around line 10-11: The Redpanda engine inputs are not pinned to one immutable,
reproducible revision. In iceberg-editor/wasm-spike/env.sh lines 10-11, require
an explicit immutable tag or commit instead of defaulting to mutable dev and
record the resolved commit; in
iceberg-editor/wasm-spike/third_party/fetch-deps.sh lines 20-22, resolve the
Avro SHA and patch blobs from that same revision or consume a per-release
lockfile with checksums.
In `@iceberg-editor/wasm-spike/README.md`:
- Around line 13-15: Declare the fenced code block language for the Avro
conversion output by adding the appropriate text fence marker around the
existing message, preserving the output unchanged.
In `@iceberg-editor/wasm/iceberg_wasm.cc`:
- Around line 35-50: Update escape_json to encode every JSON control character,
not only quotes, backslashes, and newlines; handle tabs, carriage returns, and
other bytes below 0x20 with valid JSON escape sequences or Unicode escapes while
preserving existing escaping behavior.
In `@src/css/iceberg-explorer.css`:
- Around line 9-21: Insert a blank line between the final custom property
declaration, --ice-cyan, and the font-size declaration in the affected CSS rule
to satisfy stylelint’s declaration-empty-line-before requirement.
In `@src/js/27-iceberg-explorer.js`:
- Around line 411-421: Update applyInitialConfig to detect legacy strings using
the section-format delimiter rather than rejecting any string containing a
colon: treat strings without a semicolon as legacy, so value_schema_latest
options still select schema_latest while section-based configs continue through
the section parser.
- Around line 22-67: Reformat the multi-line object literals in VAL_SCHEMA,
RECORD, and AVRO_VALUE_SCHEMA to satisfy object-curly-newline: place each nested
object’s multiple properties on separate lines with matching braces, including
VAL_SCHEMA.shipping_address, RECORD.value and headers entries, and the nested
AVRO_VALUE_SCHEMA shipping_address record. Preserve all values and schema
structure.
In `@src/ui.yml`:
- Around line 4-5: Update the src/ui.yml static_files entries so
iceberg-engine.wasm and iceberg-engine.js are only referenced after being
generated by the web WASM build and available in src/static; alternatively
remove the entries until those artifacts are produced. Ensure the selected
approach occurs before the gulp bundle/package bundle:pack tasks copy static
assets.
---
Nitpick comments:
In `@iceberg-editor/wasm-spike/shim/seastar/util/defer.hh`:
- Around line 32-35: Update the defer function to decay Func when constructing
and returning the deferred_action, so the scope guard owns the callable value
rather than preserving an lvalue reference. Use std::decay_t<Func> consistently
for the return type and deferred_action construction while preserving perfect
forwarding of f.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1a0a94e1-703d-4d11-b26f-dab2f81711c2
📒 Files selected for processing (34)
.github/workflows/build-iceberg-engine.yml.github/workflows/iceberg-dsl-conformance.ymliceberg-editor/README.mdiceberg-editor/wasm-spike/.gitignoreiceberg-editor/wasm-spike/README.mdiceberg-editor/wasm-spike/build-avro-cpp.shiceberg-editor/wasm-spike/env.shiceberg-editor/wasm-spike/fetch-sources.shiceberg-editor/wasm-spike/poc_main.cciceberg-editor/wasm-spike/shim/base/seastarx.hiceberg-editor/wasm-spike/shim/base/vassert.hiceberg-editor/wasm-spike/shim/iceberg/conversion/conversion_outcome.hiceberg-editor/wasm-spike/shim/seastar/core/chunked_vector.hhiceberg-editor/wasm-spike/shim/seastar/core/sstring.hhiceberg-editor/wasm-spike/shim/seastar/util/bool_class.hhiceberg-editor/wasm-spike/shim/seastar/util/defer.hhiceberg-editor/wasm-spike/shim/seastar/util/log.hhiceberg-editor/wasm-spike/shim/seastar/util/variant_utils.hhiceberg-editor/wasm-spike/shim/ssx/sformat.hiceberg-editor/wasm-spike/stage1-datatypes.shiceberg-editor/wasm-spike/stage2-schema-avro.shiceberg-editor/wasm-spike/third_party/fetch-deps.shiceberg-editor/wasm/.gitignoreiceberg-editor/wasm/build.shiceberg-editor/wasm/iceberg_wasm.ccpackage.jsonpreview-src/iceberg-explorer-test.adocpreview-src/ui-model.ymlsrc/css/iceberg-explorer.csssrc/css/site.csssrc/js/27-iceberg-explorer.jssrc/layouts/default.hbssrc/ui.ymltests/iceberg-dsl/conformance-test.js
| - name: Resolve Redpanda ref and docs version | ||
| id: ver | ||
| run: | | ||
| REF="${{ github.event.inputs.redpanda_ref || github.event.client_payload.ref }}" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Do not interpolate event-controlled values into run scripts.
redpanda_ref can break out of this assignment and execute runner commands; the derived version is interpolated again at Line 64. Pass both values through step env, validate the accepted ref/version format before writing $GITHUB_OUTPUT, and set checkout persist-credentials: false with least-privilege workflow permissions.
Proposed direction
- - name: Resolve Redpanda ref and docs version
+ - name: Resolve Redpanda ref and docs version
id: ver
+ env:
+ REF: ${{ github.event.inputs.redpanda_ref || github.event.client_payload.ref }}
run: |
- REF="${{ github.event.inputs.redpanda_ref || github.event.client_payload.ref }}"
+ [[ "$REF" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] ||
+ { echo "Invalid Redpanda ref" >&2; exit 1; }
echo "ref=$REF" >> "$GITHUB_OUTPUT" - name: Stage versioned artifact
id: stage
+ env:
+ V: ${{ steps.ver.outputs.version }}
run: |
- V="${{ steps.ver.outputs.version }}"
mkdir -p dist📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| REF="${{ github.event.inputs.redpanda_ref || github.event.client_payload.ref }}" | |
| - name: Resolve Redpanda ref and docs version | |
| id: ver | |
| env: | |
| REF: ${{ github.event.inputs.redpanda_ref || github.event.client_payload.ref }} | |
| run: | | |
| [[ "$REF" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] || | |
| { echo "Invalid Redpanda ref" >&2; exit 1; } | |
| echo "ref=$REF" >> "$GITHUB_OUTPUT" | |
| - name: Stage versioned artifact | |
| id: stage | |
| env: | |
| V: ${{ steps.ver.outputs.version }} | |
| run: | | |
| mkdir -p dist |
🧰 Tools
🪛 zizmor (1.26.1)
[error] 35-35: code injection via template expansion (template-injection): may expand into attacker-controllable code
(template-injection)
[error] 35-35: code injection via template expansion (template-injection): may expand into attacker-controllable code
(template-injection)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/build-iceberg-engine.yml at line 35, Update the workflow
step defining REF and the downstream version handling to pass event-controlled
values through step-level env variables rather than interpolating them in run
scripts; validate both ref and derived version against the accepted safe format
before writing GITHUB_OUTPUT. Also set checkout persist-credentials to false and
restrict workflow permissions to the minimum required, preserving the existing
build behavior for valid inputs.
Source: Linters/SAST tools
| conformance: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
Disable credential persistence on checkout.
actions/checkout@v4 defaults to persisting the GITHUB_TOKEN in the runner's git config for the rest of the job. This job never needs to push/authenticate afterwards, so disabling persistence is a cheap hardening step.
🔒 Proposed fix
- uses: actions/checkout@v4
+ with:
+ persist-credentials: false
- uses: actions/setup-node@v4📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - uses: actions/checkout@v4 | |
| - uses: actions/checkout@v4 | |
| with: | |
| persist-credentials: false |
🧰 Tools
🪛 zizmor (1.26.1)
[warning] 31-31: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/iceberg-dsl-conformance.yml at line 31, Update the
actions/checkout@v4 step in the workflow to disable credential persistence by
setting persist-credentials to false. Leave the checkout behavior otherwise
unchanged.
Source: Linters/SAST tools
| export REDPANDA_REF="${REDPANDA_REF:-dev}" | ||
| export REDPANDA_REPO="${REDPANDA_REPO:-redpanda-data/redpanda}" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
Versioned engine inputs are not locked as one immutable release. The source ref defaults to mutable dev, while the Avro archive and patch source are selected independently; this prevents reproducing or attributing a generated engine to one Redpanda revision.
iceberg-editor/wasm-spike/env.sh#L10-L11: require an explicit immutable tag or commit for reproducible/release builds and record the resolved commit.iceberg-editor/wasm-spike/third_party/fetch-deps.sh#L20-L22: resolve the Avro SHA and patch blobs from that same revision, or consume a per-release lockfile with checksums.
📍 Affects 2 files
iceberg-editor/wasm-spike/env.sh#L10-L11(this comment)iceberg-editor/wasm-spike/third_party/fetch-deps.sh#L20-L22
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@iceberg-editor/wasm-spike/env.sh` around lines 10 - 11, The Redpanda engine
inputs are not pinned to one immutable, reproducible revision. In
iceberg-editor/wasm-spike/env.sh lines 10-11, require an explicit immutable tag
or commit instead of defaulting to mutable dev and record the resolved commit;
in iceberg-editor/wasm-spike/third_party/fetch-deps.sh lines 20-22, resolve the
Avro SHA and patch blobs from that same revision or consume a per-release
lockfile with checksums.
| ``` | ||
| OK: converted Avro record to iceberg::struct_type with 3 field(s) | ||
| ``` |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Declare the output fence language.
This fence triggers markdownlint MD040.
Proposed fix
-```
+```text
OK: converted Avro record to iceberg::struct_type with 3 field(s)</details>
<!-- suggestion_start -->
<details>
<summary>📝 Committable suggestion</summary>
> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
```suggestion
🧰 Tools
🪛 markdownlint-cli2 (0.23.0)
[warning] 13-13: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@iceberg-editor/wasm-spike/README.md` around lines 13 - 15, Declare the fenced
code block language for the Avro conversion output by adding the appropriate
text fence marker around the existing message, preserving the output unchanged.
Source: Linters/SAST tools
| void escape_json(const std::string& s, std::string& out) { | ||
| for (char c : s) { | ||
| switch (c) { | ||
| case '"': | ||
| out += "\\\""; | ||
| break; | ||
| case '\\': | ||
| out += "\\\\"; | ||
| break; | ||
| case '\n': | ||
| out += "\\n"; | ||
| break; | ||
| default: | ||
| out += c; | ||
| } | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Escape all JSON control characters.
Tabs, carriage returns, and other control bytes in a conversion error produce invalid JSON, causing the UI’s JSON parse to fail instead of rendering the engine error.
Proposed fix
case '\n':
out += "\\n";
break;
+ case '\r':
+ out += "\\r";
+ break;
+ case '\t':
+ out += "\\t";
+ break;
+ case '\b':
+ out += "\\b";
+ break;
+ case '\f':
+ out += "\\f";
+ break;
default:
- out += c;
+ if (static_cast<unsigned char>(c) < 0x20) {
+ static constexpr char hex[] = "0123456789abcdef";
+ const auto byte = static_cast<unsigned char>(c);
+ out += "\\u00";
+ out += hex[byte >> 4];
+ out += hex[byte & 0x0f];
+ } else {
+ out += c;
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| void escape_json(const std::string& s, std::string& out) { | |
| for (char c : s) { | |
| switch (c) { | |
| case '"': | |
| out += "\\\""; | |
| break; | |
| case '\\': | |
| out += "\\\\"; | |
| break; | |
| case '\n': | |
| out += "\\n"; | |
| break; | |
| default: | |
| out += c; | |
| } | |
| } | |
| void escape_json(const std::string& s, std::string& out) { | |
| for (char c : s) { | |
| switch (c) { | |
| case '"': | |
| out += "\\\""; | |
| break; | |
| case '\\': | |
| out += "\\\\"; | |
| break; | |
| case '\n': | |
| out += "\\n"; | |
| break; | |
| case '\r': | |
| out += "\\r"; | |
| break; | |
| case '\t': | |
| out += "\\t"; | |
| break; | |
| case '\b': | |
| out += "\\b"; | |
| break; | |
| case '\f': | |
| out += "\\f"; | |
| break; | |
| default: | |
| if (static_cast<unsigned char>(c) < 0x20) { | |
| static constexpr char hex[] = "0123456789abcdef"; | |
| const auto byte = static_cast<unsigned char>(c); | |
| out += "\\u00"; | |
| out += hex[byte >> 4]; | |
| out += hex[byte & 0x0f]; | |
| } else { | |
| out += c; | |
| } | |
| } | |
| } | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@iceberg-editor/wasm/iceberg_wasm.cc` around lines 35 - 50, Update escape_json
to encode every JSON control character, not only quotes, backslashes, and
newlines; handle tabs, carriage returns, and other bytes below 0x20 with valid
JSON escape sequences or Unicode escapes while preserving existing escaping
behavior.
| --ice-accent: var(--product-color, #e2401b); | ||
| --ice-muted: var(--color-gray-70, #6b7280); | ||
| --ice-border: var(--panel-border-color, #d0d3d9); | ||
| --ice-panel: var(--panel-background, #f7f8fa); | ||
| --ice-pre: var(--pre-background, #f0f1f4); | ||
| --ice-code: var(--code-font-color, var(--body-font-color)); | ||
| --ice-green: #1a7f5a; | ||
| --ice-blue: #2563c9; | ||
| --ice-yellow: #9a6b00; | ||
| --ice-purple: #7c3aed; | ||
| --ice-cyan: #0e7490; | ||
| font-size: var(--secondary-font-size, 0.9rem); | ||
| margin: 1.25rem 0; |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add blank line before font-size per stylelint declaration-empty-line-before.
🎨 Proposed fix
--ice-cyan: `#0e7490`;
+
font-size: var(--secondary-font-size, 0.9rem);
margin: 1.25rem 0;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| --ice-accent: var(--product-color, #e2401b); | |
| --ice-muted: var(--color-gray-70, #6b7280); | |
| --ice-border: var(--panel-border-color, #d0d3d9); | |
| --ice-panel: var(--panel-background, #f7f8fa); | |
| --ice-pre: var(--pre-background, #f0f1f4); | |
| --ice-code: var(--code-font-color, var(--body-font-color)); | |
| --ice-green: #1a7f5a; | |
| --ice-blue: #2563c9; | |
| --ice-yellow: #9a6b00; | |
| --ice-purple: #7c3aed; | |
| --ice-cyan: #0e7490; | |
| font-size: var(--secondary-font-size, 0.9rem); | |
| margin: 1.25rem 0; | |
| --ice-accent: var(--product-color, `#e2401b`); | |
| --ice-muted: var(--color-gray-70, `#6b7280`); | |
| --ice-border: var(--panel-border-color, `#d0d3d9`); | |
| --ice-panel: var(--panel-background, `#f7f8fa`); | |
| --ice-pre: var(--pre-background, `#f0f1f4`); | |
| --ice-code: var(--code-font-color, var(--body-font-color)); | |
| --ice-green: `#1a7f5a`; | |
| --ice-blue: `#2563c9`; | |
| --ice-yellow: `#9a6b00`; | |
| --ice-purple: `#7c3aed`; | |
| --ice-cyan: `#0e7490`; | |
| font-size: var(--secondary-font-size, 0.9rem); | |
| margin: 1.25rem 0; |
🧰 Tools
🪛 Stylelint (17.14.0)
[error] 20-20: Expected empty line before declaration (declaration-empty-line-before)
(declaration-empty-line-before)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/css/iceberg-explorer.css` around lines 9 - 21, Insert a blank line
between the final custom property declaration, --ice-cyan, and the font-size
declaration in the affected CSS rule to satisfy stylelint’s
declaration-empty-line-before requirement.
Source: Linters/SAST tools
| var VAL_SCHEMA = [ | ||
| { name: 'order_id', type: 'string' }, | ||
| { name: 'product', type: 'string' }, | ||
| { name: 'quantity', type: 'int' }, | ||
| { name: 'price', type: 'double' }, | ||
| { name: 'shipped', type: 'boolean' }, | ||
| { name: 'shipping_address', type: 'struct', fields: [ | ||
| { name: 'street', type: 'string' }, | ||
| { name: 'city', type: 'string' }, | ||
| { name: 'zip', type: 'string' }, | ||
| { name: 'tags', type: 'list<string>' }, | ||
| ] }, | ||
| ] | ||
| var RECORD = { | ||
| partition: 3, offset: 42, timestamp: 1720000000000, | ||
| value: { | ||
| order_id: 'ORD-98765', product: 'Redpanda Plushie', quantity: 3, | ||
| price: 29.99, shipped: false, | ||
| shipping_address: { street: '123 Main St', city: 'Seattle', zip: '98101', tags: ['residential', 'priority'] }, | ||
| }, | ||
| headers: [ | ||
| { key: 'content-type', value: 'application/json' }, | ||
| { key: 'x-request-id', value: 'req-abc-123' }, | ||
| { key: 'x-trace-id', value: 'trace-00f1e2d3' }, | ||
| ], | ||
| } | ||
|
|
||
| // The sample value schema expressed as real Avro JSON. When the WASM engine | ||
| // is available, this is fed to the REAL Redpanda C++ mapper to compute the | ||
| // Iceberg value fields (see below); otherwise the tool falls back to the | ||
| // VAL_SCHEMA rules above (interim engine). | ||
| var AVRO_VALUE_SCHEMA = JSON.stringify({ | ||
| type: 'record', name: 'order', fields: [ | ||
| { name: 'order_id', type: 'string' }, | ||
| { name: 'product', type: 'string' }, | ||
| { name: 'quantity', type: 'int' }, | ||
| { name: 'price', type: 'double' }, | ||
| { name: 'shipped', type: 'boolean' }, | ||
| { name: 'shipping_address', type: { type: 'record', name: 'shipping_address', fields: [ | ||
| { name: 'street', type: 'string' }, | ||
| { name: 'city', type: 'string' }, | ||
| { name: 'zip', type: 'string' }, | ||
| { name: 'tags', type: { type: 'array', items: 'string' } } | ||
| ] } } | ||
| ] | ||
| }) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
Fix ESLint object-curly-newline failures — this is currently breaking the build.
Pipeline logs confirm validate-build / build fails on object-curly-newline at line 28, and static analysis flags the same rule at lines 33, 36, 38, 39, 54. These are the multi-line object literals (VAL_SCHEMA's shipping_address, RECORD, AVRO_VALUE_SCHEMA) that pack multiple properties on one line while the surrounding object itself spans multiple lines.
🐛 Proposed reformat
var VAL_SCHEMA = [
{ name: 'order_id', type: 'string' },
{ name: 'product', type: 'string' },
{ name: 'quantity', type: 'int' },
{ name: 'price', type: 'double' },
{ name: 'shipped', type: 'boolean' },
- { name: 'shipping_address', type: 'struct', fields: [
- { name: 'street', type: 'string' },
- { name: 'city', type: 'string' },
- { name: 'zip', type: 'string' },
- { name: 'tags', type: 'list<string>' },
- ] },
+ { name: 'shipping_address',
+ type: 'struct',
+ fields: [
+ { name: 'street', type: 'string' },
+ { name: 'city', type: 'string' },
+ { name: 'zip', type: 'string' },
+ { name: 'tags', type: 'list<string>' },
+ ] },
]
var RECORD = {
- partition: 3, offset: 42, timestamp: 1720000000000,
+ partition: 3,
+ offset: 42,
+ timestamp: 1720000000000,
value: {
- order_id: 'ORD-98765', product: 'Redpanda Plushie', quantity: 3,
- price: 29.99, shipped: false,
+ order_id: 'ORD-98765',
+ product: 'Redpanda Plushie',
+ quantity: 3,
+ price: 29.99,
+ shipped: false,
shipping_address: { street: '123 Main St', city: 'Seattle', zip: '98101', tags: ['residential', 'priority'] },
},
headers: [
{ key: 'content-type', value: 'application/json' },
{ key: 'x-request-id', value: 'req-abc-123' },
{ key: 'x-trace-id', value: 'trace-00f1e2d3' },
],
}
...
var AVRO_VALUE_SCHEMA = JSON.stringify({
- type: 'record', name: 'order', fields: [
+ type: 'record',
+ name: 'order',
+ fields: [
{ name: 'order_id', type: 'string' },Note: the nested shipping_address record type inside AVRO_VALUE_SCHEMA (around line 60) has the same multi-prop/multiline shape and may also need reformatting — worth double-checking with the lint config once these are fixed.
🧰 Tools
🪛 GitHub Actions: validate-build / 0_build.txt
[error] 28-28: ESLint: 28:5 error Expected a line break after this opening brace (object-curly-newline).
🪛 GitHub Actions: validate-build / build
[error] 28-28: ESLint (lint:js) error: 28:5 Expected a line break after this opening brace (object-curly-newline).
🪛 GitHub Check: build
[failure] 54-54:
Object properties must go on a new line if they aren't all on the same line
[failure] 39-39:
Object properties must go on a new line if they aren't all on the same line
[failure] 38-38:
Object properties must go on a new line if they aren't all on the same line
[failure] 38-38:
Object properties must go on a new line if they aren't all on the same line
[failure] 36-36:
Object properties must go on a new line if they aren't all on the same line
[failure] 36-36:
Object properties must go on a new line if they aren't all on the same line
[failure] 33-33:
Expected a line break before this closing brace
[failure] 28-28:
Object properties must go on a new line if they aren't all on the same line
[failure] 28-28:
Object properties must go on a new line if they aren't all on the same line
[failure] 28-28:
Expected a line break after this opening brace
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/js/27-iceberg-explorer.js` around lines 22 - 67, Reformat the multi-line
object literals in VAL_SCHEMA, RECORD, and AVRO_VALUE_SCHEMA to satisfy
object-curly-newline: place each nested object’s multiple properties on separate
lines with matching braces, including VAL_SCHEMA.shipping_address, RECORD.value
and headers entries, and the nested AVRO_VALUE_SCHEMA shipping_address record.
Preserve all values and schema structure.
Sources: Linters/SAST tools, Pipeline failures
| // Apply an author-supplied initial config string (data-config) by mapping | ||
| // its sections back onto the controls. Best-effort; unknown tokens ignored. | ||
| function applyInitialConfig (str) { | ||
| if (!str) return | ||
| if (str.indexOf(':') === -1) { | ||
| // legacy string | ||
| if (str === 'key_value') { setRadioValue('val-mode', 'binary') } | ||
| else if (str === 'value_schema_id_prefix') { setRadioValue('val-mode', 'schema_id_prefix') } | ||
| else if (str.indexOf('value_schema_latest') === 0) { setRadioValue('val-mode', 'schema_latest') } | ||
| return | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Legacy-string detection breaks for legacy strings with options.
str.indexOf(':') === -1 is used to distinguish "legacy" DSL strings from the section-based format, but legacy strings can legitimately contain a colon (e.g. value_schema_latest:subject=my-topic-value, per the conformance vectors in tests/iceberg-dsl/conformance-test.js). Such strings fall through to the section-parser, where prefixFor['value_schema_latest'] is undefined, so the val-mode radio is silently never set to schema_latest. Section-format strings always contain ; (three sections are always joined), so that's a more reliable discriminant.
🐛 Proposed fix
function applyInitialConfig (str) {
if (!str) return
- if (str.indexOf(':') === -1) {
+ if (str.indexOf(';') === -1) {
// legacy string📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // Apply an author-supplied initial config string (data-config) by mapping | |
| // its sections back onto the controls. Best-effort; unknown tokens ignored. | |
| function applyInitialConfig (str) { | |
| if (!str) return | |
| if (str.indexOf(':') === -1) { | |
| // legacy string | |
| if (str === 'key_value') { setRadioValue('val-mode', 'binary') } | |
| else if (str === 'value_schema_id_prefix') { setRadioValue('val-mode', 'schema_id_prefix') } | |
| else if (str.indexOf('value_schema_latest') === 0) { setRadioValue('val-mode', 'schema_latest') } | |
| return | |
| } | |
| // Apply an author-supplied initial config string (data-config) by mapping | |
| // its sections back onto the controls. Best-effort; unknown tokens ignored. | |
| function applyInitialConfig (str) { | |
| if (!str) return | |
| if (str.indexOf(';') === -1) { | |
| // legacy string | |
| if (str === 'key_value') { setRadioValue('val-mode', 'binary') } | |
| else if (str === 'value_schema_id_prefix') { setRadioValue('val-mode', 'schema_id_prefix') } | |
| else if (str.indexOf('value_schema_latest') === 0) { setRadioValue('val-mode', 'schema_latest') } | |
| return | |
| } |
🧰 Tools
🪛 ast-grep (0.44.1)
[error] 416-416: React's useState should not be directly called
Context: setRadioValue('val-mode', 'binary')
Note: [CWE-710] Improper Adherence to Coding Standards. Security best practice.
(usestate-direct-usage)
[error] 417-417: React's useState should not be directly called
Context: setRadioValue('val-mode', 'schema_id_prefix')
Note: [CWE-710] Improper Adherence to Coding Standards. Security best practice.
(usestate-direct-usage)
[error] 418-418: React's useState should not be directly called
Context: setRadioValue('val-mode', 'schema_latest')
Note: [CWE-710] Improper Adherence to Coding Standards. Security best practice.
(usestate-direct-usage)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/js/27-iceberg-explorer.js` around lines 411 - 421, Update
applyInitialConfig to detect legacy strings using the section-format delimiter
rather than rejecting any string containing a colon: treat strings without a
semicolon as legacy, so value_schema_latest options still select schema_latest
while section-based configs continue through the section parser.
| - iceberg-engine.wasm | ||
| - iceberg-engine.js |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
fd -i 'iceberg-engine' .
fd -i 'blobl.wasm' src/static 2>/dev/null
cat src/ui.ymlRepository: redpanda-data/docs-ui
Length of output: 312
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## src tree around static files"
git ls-files 'src/*' 'src/**' | sed -n '1,120p'
echo "## package scripts and possible static-file validation"
python3 - <<'PY'
import json
from pathlib import Path
p=Path('package.json')
if p.exists():
data=json.loads(p.read_text())
for k,v in data.get('scripts',{}).items():
print(f"{k}: {v}")
PY
echo "## search for static_files usage"
rg -n "static_files|blobl\.wasm|iceberg-engine" . --glob '!node_modules' --glob '!dist' --glob '!build' --glob '!coverage'Repository: redpanda-data/docs-ui
Length of output: 7573
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## repository top-level"
git ls-files | sed -n '1,120p'
echo "## iceberg-editor wasm files"
fd -i 'iceberg-engine|build\.sh' iceberg-editor 2>/dev/null | sort
echo "## iceberg-editor/wasm/build.sh relevant sections"
wc -l iceberg-editor/wasm/build.sh 2>/dev/null || true
sed -n '1,110p' iceberg-editor/wasm/build.sh 2>/dev/null || true
sed -n '110,160p' iceberg-editor/wasm/build.sh 2>/dev/null || true
echo "## package scripts"
python3 - <<'PY'
import json
from pathlib import Path
data=json.loads(Path('package.json').read_text())
for name, script in data.get('scripts', {}).items():
print(f"{name}: {script}")
PY
echo "## search for build:wasm or wamc/emcc in workflows/package"
rg -n "build:wasm|wasm|emcc|wasm-pack|outDir|OUTDIR|iceberg-engine" package.json .github gulpfile.js --glob '!node_modules' || true
echo "## src static existence check"
python3 - <<'PY'
from pathlib import Path
files=['src/static/blobl.wasm','src/static/iceberg-engine.wasm','src/static/iceberg-engine.js']
for p in files:
path=Path(p)
print(f"{p}: {'exists_file' if path.is_file() else ('exists_dir' if path.exists() else 'missing')!r}")
PYRepository: redpanda-data/docs-ui
Length of output: 12044
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## compile-partial.js"
wc -l compile-partial.js
sed -n '1,220p' compile-partial.js
echo "## gulpfile build/static tasks"
wc -l gulpfile.js
sed -n '1,160p' gulpfile.js
echo "## build preview UI docs"
wc -l docs/modules/ROOT/pages/build-preview-ui.adoc
sed -n '1,180p' docs/modules/ROOT/pages/build-preview-ui.adoc
echo "## package.json build-related scripts"
python3 - <<'PY'
import json
from pathlib import Path
data=json.loads(Path('package.json').read_text())
for name in ['build','build:preview','build:deploy','deploy','compile:partials','bundle-react','serve:preview','publish:release']:
if name in data.get('scripts',{}):
print(f"{name}: {data['scripts'][name]}")
PY
echo "## static-files docs examples"
sed -n '35,75p' docs/modules/ROOT/pages/add-static-files.adocRepository: redpanda-data/docs-ui
Length of output: 15560
Add or produce the Iceberg Engine static assets before referencing them in static_files:.
src/ui.yml now promotes iceberg-engine.wasm and iceberg-engine.js, but neither file is checked into src/static/, and the UI bundle task does not generate them. Either omit these entries until the assets exist, or run iceberg-editor/wasm/build.sh web/check-in the resulting artifacts before gulp bundle/package bundle:pack copies src/static into the UI output.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/ui.yml` around lines 4 - 5, Update the src/ui.yml static_files entries so
iceberg-engine.wasm and iceberg-engine.js are only referenced after being
generated by the web WASM build and available in src/static; alternatively
remove the entries until those artifacts are produced. Ensure the selected
approach occurs before the gulp bundle/package bundle:pack tasks copy static
assets.
1 participant
What
Turns the standalone v26.2 "Iceberg Mode Explorer" TOI artifact into a docs-integrated, versioned interactive tool — and swaps its hand-written JavaScript rules for Redpanda's real C++ translation code compiled to WebAssembly.
An author drops the
[iceberg-explorer]block (companion PR below) on a page; this bundle hydrates it into the tool: config controls, the DSL config string, the resulting Iceberg table schema, and a sample translated record.Why WASM
The original re-typed the product's translation rules by hand, so it drifts the moment Redpanda changes. Running the real code in the browser (as the Bloblang playground already does) means the rule-bearing output can't drift, and a per-release build keeps every docs version correct with no manual upkeep.
Engine coverage (deliberate)
iceberg::type_to_iceberg)iceberg_mode_test.ccvectorsThe record path is Seastar-coroutine + serde-parser code with no synchronous entry point; a real-WASM port is a large, separate effort. Everything sits behind a
translate()seam, so any part can move to WASM later without UI changes.Changes
src/js/27-iceberg-explorer.js+src/css/iceberg-explorer.css— hydration module and theme-aware styling (follows the site's light/dark tokens).iceberg-editor/— Emscripten build of the real mapper.wasm-spike/is the reproducible feasibility proof (pinned toolchain: fmt 12.1.0, redpanda-data/avro fork + patches, thin Seastar shim);wasm/is the browser engine build (build.sh+ Embind). Fetched sources, third-party deps, and build outputs are git-ignored and reproducible from the scripts.iceberg-engine-<version>for the page's doc version → bundle asset → interim JS fallback.<body data-version>added todefault.hbs.tests/iceberg-dsl/— config-string conformance vectors copied from Redpanda'siceberg_mode_test.cc; caught and fixed two drift bugs (missing valuelayout, reversed option order).build-iceberg-engine.yml(build versioned engine per release tag) andiceberg-dsl-conformance.yml(run vectors + diff upstream).Companion PR
Needs the block macro: redpanda-data/docs-extensions-and-macros#219.
Test / preview
npm run test:iceberg-dsl→ 7/7 conformance vectors pass.gulp preview→ http://localhost:5252/iceberg-explorer-test.html (verified: real-engine schema, correct config strings, light/dark, two independent instances).iceberg-editor/wasm/build.sh {node|web}(needs emsdk).🤖 Generated with Claude Code