Skip to content

add document for consume_schema_strategy - #663

Open
yuzifeng1984 wants to merge 2 commits into
mainfrom
doc/issue-662-kafka-consume-schema-strategy
Open

add document for consume_schema_strategy#663
yuzifeng1984 wants to merge 2 commits into
mainfrom
doc/issue-662-kafka-consume-schema-strategy

Conversation

@yuzifeng1984

@yuzifeng1984 yuzifeng1984 commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Close #662

New in v3.4

@yuzifeng1984
yuzifeng1984 requested a review from gangtao August 4, 2026 09:21
@netlify

netlify Bot commented Aug 4, 2026

Copy link
Copy Markdown

Deploy Preview for elastic-elion-a958b6 ready!

Name Link
🔨 Latest commit ed84b6b
🔍 Latest deploy log https://app.netlify.com/projects/elastic-elion-a958b6/deploys/6a7406ec52dc9600086b4d4d
😎 Deploy Preview https://deploy-preview-663--elastic-elion-a958b6.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@gangtao

gangtao commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Review: consume_schema_strategy documentation

What checks out ✅

  • The /kafka-source#consume_schema_strategy link pattern into partial headings is well-established in this repo (#config_file, #named_collection are linked the same way), and underscore anchors match the existing convention.
  • The migration note calling out the default-behavior change is exactly the right thing to include for a breaking default.
  • The ssl_ca_pem typo fix and the grammar cleanup in kafka-schema-registry.md are nice drive-bys.

Issues

1. The raw-mode example is technically wrong (both copies). It strips the 5-byte Confluent header from an Avro-encoded payload, then routes it with json_extract_string(raw, ...). Avro binary is not JSON — json_extract_string will return nothing on it. Either the example should use a JSON-encoded payload scenario, or the routing should be shown on something that actually works on raw Avro bytes (e.g. schema-ID-based routing via a metadata column, or decoding in application logic as the prose suggests). As written, users will copy an MV that silently produces empty rows.

2. The two copies of the examples have already diverged. kafka-external-stream-read.md filters on json_extract_string(raw, '_schema_type') while kafka-external-stream.md filters on json_extract_string(raw, 'type'). Worse, both partials are included in the same rendered page: /kafka-source = Basics partial + Read partial, so the near-identical single/all/raw examples appear twice on one page. Suggest keeping the full reference in the settings section and reducing the Read-section addition to a short paragraph + link.

3. Read-side content now renders on the Kafka Sink page. The Basics partial (shared/kafka-external-stream.md) is imported by both kafka-external-stream-source.mdx and kafka-external-stream-sink.mdx, so ~100 lines about consumption strategy will appear on /kafka-sink too. If that's not intended, this content belongs in shared/kafka-external-stream-read.md (source-only) instead.

4. Internal contradiction on schema_subject_name requirement. The consume_schema_strategy table says single mode requires "schema_subject_name must be set" — but single is the default, and the subject_name_strategy table says the default TopicNameStrategy ignores schema_subject_name (subject is derived as <topic>-value). A reader can't tell what a plain registry-backed stream with no schema_subject_name does under the new default. Please clarify that the requirement only applies with RecordNameStrategy/TopicRecordNameStrategy.

5. No version marker. The PR body says "New in v3.4" and the all row says "preserves the pre-v3.4 behavior," but nowhere does the doc say the setting exists starting from 3.4. Repo convention is "Starting from Timeplus Enterprise X…" (see js-udf.md, py-udf.md). Users on 3.3 will try the setting and fail; kafka-schema-registry.md now also describes single-mode filtering as current behavior with no version gate. (enterprise-v3.4.md doesn't exist yet — the release-notes entry will presumably come separately, but the version marker here shouldn't wait for it.)

6. Minor: the Read partial's "see the Kafka External Stream settings" is a self-link on the /kafka-source page it renders into — harmless, but becomes unnecessary if the duplication in issue 2 is resolved.

Risk assessment

Medium for a docs PR: the copy-paste raw example would mislead users into a non-working pipeline (issue 1), and the on-page duplication (issue 2) plus sink-page bleed (issue 3) are structural. Recommend addressing issues 1–4 before merge, with 5 strongly encouraged.

🤖 Posted by Claude Code review

@yuzifeng1984

Copy link
Copy Markdown
Collaborator Author

Fixed the description for issue item 1 & 4
For item 2 & 3, remove the read examples in the shared content and keep only in the read page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add kafka external stream 'consume_schema_strategy' document

2 participants