feat(sinks): add validate_structure() for early template confinement checks by thomasqueirozb · Pull Request #25959 · vectordotdev/vector

thomasqueirozb · 2026-07-28T16:04:08Z

Summary

Implements RFC 2026-07-15 migration for sinks (parallel to PR #25953 which did transforms). Adds validate_structure() to SinkConfig trait and wires it into validation so that vector validate --no-environment catches structural errors like template confinement violations that would otherwise silently pass and only fail at boot.

Vector configuration

NA

How did you test this PR?

Ran make check-clippy - passes
Ran make fmt - passes
Ran cargo nextest run -p vector --lib "confinement" - all 69 tests pass
Code review verified all 22 sink implementations match their build() confinement logic

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Fixes: vector validate --no-environment skips template confinement check #25840 - vector validate --no-environment silently passing unconfined templates
Related: chore(transforms): add validate_structure and rename validate_env #25953 - transforms validate_structure PR

…checks

datadog-vectordotdev · 2026-07-28T16:06:48Z

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: d9fa031 | Docs | Datadog PR Page | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d172a7cfa6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…ate_structure

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 45539ea4cc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…y, axiom, humio) This adds validate_structure() implementations for wrapper sinks that delegate to HTTP or HEC sinks: - opentelemetry: Delegates to HttpSinkConfig::validate_structure() - axiom: Builds HttpSinkConfig and delegates validation - humio_logs: Delegates to HecLogsSinkConfig::validate_structure() - humio_metrics: Builds HumioLogsConfig and delegates to HEC validation These wrapper sinks were using the default validate_structure() which returns Ok(()), allowing unconfined templates to pass validation even though they would fail at boot. Now they properly delegate to their inner sink's validation. Also adds unit tests for each wrapper sink's confinement validation.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5dd8b9b0f6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

- Parse static URIs during validation to catch invalid URI syntax - Validate HTTP headers before splitting templates (malformed headers, auth conflicts) - Reject raw endpoint with auto_extract_timestamp in HEC logs during validation These validations catch errors at time that would otherwise fail at boot.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 35bb233832

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Add check for empty endpoints during structural validation so that `vector validate --no-environment` catches the same error that would fail at startup.

Validate content_type, content_encoding, cache_control, and metadata header values during structural validation to catch malformed headers that would otherwise fail at startup in RequestSettings::new().

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e6077b6057

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

- Check for empty labels (mirrors build() check) - Validate label names using valid_label_name() (mirrors build() check) - Add tests for both validation cases

- azure_blob: validate credential shape (connection_string/account_name/blob_endpoint combinations) - clickhouse: validate batch_encoding compatibility and arrow schema requirements - doris: check for empty endpoints - gcp_cloud_storage: validate header values (content_type, content_encoding, cache_control, metadata) - http: validate batch settings (max_events, max_bytes, timeout) - kafka: check batch option conflicts with librdkafka_options - loki: validate empty labels and label name format - mqtt: check empty client_id and user/password pairing - nats: parse server URLs during validation - prometheus_remote_write: validate endpoint URI and headers - redis: reject empty keys All validations mirror checks in build() so `vector validate --no-environment` catches errors that would otherwise fail at boot.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d2530d6247

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

- amqp: reject zero max_channels - aws_cloudwatch_logs: validate batch settings - elasticsearch: validate endpoint/endpoints mutual exclusivity - gcp_cloud_storage: fix default endpoint in tests - greptimedb_logs: parse endpoint URL - http: validate payload wrapper for JSON with delimited framing - loki: reject duplicate credentials (auth + endpoint auth) - pulsar: validate auth shape (name/token, oauth2 combinations) - redis: validate endpoints (non-empty, parseable URLs)

The url crate interprets 'localhost:4000' as a valid URL with scheme 'localhost'. Use 'http://' which actually fails to parse.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a52b3a6d19

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: da858354ef

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…cture

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8b598a84f8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 886abf3b0b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…onfig deserialization

…tructure

thomasqueirozb · 2026-07-29T20:15:33Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cf6357d672

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ecc5dc472e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-29T21:04:09Z

+            // Parse the authority part only (between :// and /)
+            // Use split to avoid UTF-8 slicing issues
+            if let Some(after_scheme) = uri_str.split("://").nth(1) {
+                let authority = after_scheme.split('/').next().unwrap_or("");


Stop dynamic URI auth scan at query delimiters

Fresh evidence after the path fix: this still only stops the authority scan at /, so a dynamic URI like https://api.example?email=alice@example.com/{{ id }} with top-level auth treats the query as part of the authority and rejects a valid config. Runtime URI parsing only extracts credentials from the authority, so stop at ?/# too or parse the literal prefix as a URI authority.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-29T21:04:10Z

+
+        // Validate encoding configuration (e.g., CSV with empty fields list)
+        // Mirrors build() in FileSink::new()
+        if let Err(e) = self.encoding.build(SinkType::StreamBased) {


Avoid reading protobuf descriptors during no-env validation

When a file sink uses encoding.codec = "protobuf" and encoding.protobuf.desc_file is only present on the deployment host, this new structural pass calls encoding.build(...); the Protobuf serializer build reads desc_file, so vector validate --no-environment now fails on the validation host even though startup would succeed where the file exists. Keep this check to serializers that do not need external files, or use a non-I/O structural validation path.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-29T21:04:10Z

+                    if uri.uri.host().is_none() {
+                        errors.push(format!("endpoint '{endpoint}': must include hostname"));


Reject non-HTTP Elasticsearch endpoints

For an endpoint such as ftp://example.com, this structural pass parses the URI and only verifies that a host exists, so no-environment validation succeeds. The same endpoint is later converted into Elasticsearch healthcheck and _bulk requests and sent through HttpClient, so startup or all sends fail for non-HTTP schemes; require the parsed scheme to be http or https here.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-29T21:04:10Z

+        } else {
+            // Check each endpoint has a host and no auth conflict
+            for endpoint in &self.endpoints {
+                if endpoint.uri.host().is_none() {


Reject non-HTTP Doris endpoints

When endpoints contains a value like ftp://fe:8030, this validator only checks that the parsed URI has a host, so the config passes validation. DorisSinkClient later rebuilds stream-load and healthcheck URIs from the endpoint's scheme and sends them via HttpClient, which leaves the sink unable to connect for non-HTTP schemes; check that each endpoint scheme is http or https before accepting it.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-29T21:04:10Z

+        if let Some(connstr) = &self.connection_string {
+            if let Err(e) = ParsedConnectionString::parse(connstr.inner()) {
+                errors.push(format!("connection_string: invalid format: {e}"));


Validate Azure connection-string URLs

For connection_string values that parse as key/value pairs but cannot produce a container URL, such as BlobEndpoint=not-a-url or a string without AccountName and without BlobEndpoint, this branch returns success because ParsedConnectionString::parse accepts the shape. build_client() immediately calls parsed.container_url(&container_name) and then Url::parse, both before any environment access, so no-environment validation can still accept configs startup rejects; mirror that container URL construction here.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-29T21:04:10Z

+        }
+
+        // Validate endpoint has a host (check before with_default_parts fills in defaults)
+        if self.endpoint.uri.host().is_none() {


Reject non-HTTP ClickHouse endpoints

For a ClickHouse endpoint such as ftp://localhost:8123, this validator accepts the config because it only checks that a host is present. Build then uses the same URI to construct healthcheck and insert requests and sends them through HttpClient, so the sink fails to connect for non-HTTP schemes; require the scheme to be http or https here as well.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-29T21:04:10Z

+
+        // Parse endpoint URL to catch malformed endpoints early
+        // (mirrors url::Url::parse in prepare_log_ingester_url)
+        if let Err(e) = url::Url::parse(&self.endpoint) {


Reject non-HTTP Greptime endpoints

An endpoint like ftp://localhost:4000 passes this check because Url::parse accepts the URL, but the sink later turns it into /health and /v1/events/logs requests sent by HttpClient, which only supports HTTP(S) destinations. Validate that the parsed endpoint scheme is http or https before returning success.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-29T21:04:10Z

        &self.acknowledgements.inner
    }
+
+    fn validate_structure(&self) -> std::result::Result<(), Vec<String>> {


Validate Splunk HEC endpoint schemes

This structural pass never validates endpoint, so a logs config with endpoint = "ftp://localhost:8088" passes even though request construction later appends /services/collector/... and sends the resulting URI via HttpClient, causing healthchecks and every batch to fail for non-HTTP schemes. Parse the endpoint here and require http or https before accepting the config.

Useful? React with 👍 / 👎.

pront

Preliminary feedback - will keep reviewing

pront · 2026-07-30T17:02:27Z

+    /// `vector validate --no-environment` and normal startup/reload.
+    pub fn validate_structure(
+        &self,
+    ) -> Result<(), Box<dyn std::error::Error + Send + Sync + 'static>> {


Use the existing vector_common::Error type wherever possible:

Suggested change

) -> Result<(), Box<dyn std::error::Error + Send + Sync + 'static>> {

) -> Result<(), vector_common::Error> {

pront · 2026-07-30T17:10:45Z

+/// `validate` runs twice at startup, which is intentional and cheap: it is pure by contract, so
+/// re-running it costs microseconds and avoids threading state from config compilation through to
+/// the topology builder.


Suggested change

/// `validate` runs twice at startup, which is intentional and cheap: it is pure by contract, so

/// re-running it costs microseconds and avoids threading state from config compilation through to

/// the topology builder.

/// Currently `validate` runs twice at startup, and this is accepted trade-off.

/// It is pure by contract, so, re-running it is very cheap and avoids threading

/// state from config compilation through to the topology builder.

feat(sinks): add validate_structure() for early template confinement …

f0f8dae

…checks

github-actions Bot added the domain: sinks Anything related to the Vector's sinks label Jul 28, 2026

thomasqueirozb added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Jul 28, 2026

thomasqueirozb added 2 commits July 28, 2026 12:10

changelog: add fragment for validate_structure sinks fix

5a37c1a

Fix changelog

d172a7c

thomasqueirozb removed the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Jul 28, 2026

thomasqueirozb marked this pull request as ready for review July 28, 2026 16:18

thomasqueirozb requested a review from a team as a code owner July 28, 2026 16:18

chatgpt-codex-connector Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread src/config/validation.rs

refactor(sinks): use if let Err instead of match Ok(_) => {} in valid…

ea410bc

…ate_structure

chatgpt-codex-connector Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread src/sinks/axiom/config.rs Outdated

Comment thread src/sinks/http/config.rs Outdated

Comment thread src/sinks/http/config.rs Outdated

Comment thread src/sinks/splunk_hec/logs/config.rs Outdated

thomasqueirozb force-pushed the sinks-validate-structure branch from 45539ea to 5dd8b9b Compare July 28, 2026 19:31

chatgpt-codex-connector Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread src/sinks/http/config.rs Outdated

Comment thread src/sinks/aws_cloudwatch_logs/config.rs Outdated

Comment thread src/sinks/redis/config.rs Outdated

Comment thread src/sinks/amqp/config.rs Outdated

Comment thread src/sinks/elasticsearch/config.rs Outdated

thomasqueirozb added 2 commits July 28, 2026 16:19

fix(doris sink): reject empty endpoints in validate_structure

235be5f

Add check for empty endpoints during structural validation so that `vector validate --no-environment` catches the same error that would fail at startup.

fix(gcs sink): validate header values in validate_structure

e6077b6

Validate content_type, content_encoding, cache_control, and metadata header values during structural validation to catch malformed headers that would otherwise fail at startup in RequestSettings::new().

chatgpt-codex-connector Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread src/sinks/greptimedb/logs/config.rs Outdated

Comment thread src/sinks/gcp/cloud_storage.rs Outdated

Comment thread src/sinks/gcp/cloud_storage.rs Outdated

Comment thread src/sinks/loki/config.rs Outdated

Comment thread src/sinks/pulsar/config.rs Outdated

thomasqueirozb added 2 commits July 28, 2026 16:43

feat(loki sink): add label validation to validate_structure

1bbab5f

- Check for empty labels (mirrors build() check) - Validate label names using valid_label_name() (mirrors build() check) - Add tests for both validation cases

chatgpt-codex-connector Bot reviewed Jul 28, 2026

View reviewed changes

thomasqueirozb added 2 commits July 29, 2026 11:11

fix(greptimedb): use URL that actually fails to parse in test

da85835

The url crate interprets 'localhost:4000' as a valid URL with scheme 'localhost'. Use 'http://' which actually fails to parse.

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

thomasqueirozb added 3 commits July 29, 2026 13:58

feat(sinks): add comprehensive structural validation for PR comments

374725c

fix(sinks): fix review issues in validate_structure implementations

8b598a8

Merge remote-tracking branch 'origin/master' into sinks-validate-stru…

886abf3

…cture

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

thomasqueirozb added 3 commits July 29, 2026 14:58

fix(kafka sink): use serde_json::from_value instead of try_into for c…

4081f7a

…onfig deserialization

feat(sinks): add encoder validation and endpoint checks to validate_s…

584a557

…tructure

fmt

cf6357d

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

feat(sinks): address remaining PR comments for validate_structure

ecc5dc4

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

thomasqueirozb marked this pull request as draft July 29, 2026 21:23

thomasqueirozb added 2 commits July 29, 2026 18:07

fix(sinks): correct structural validation checks

ec82dd1

Create and use ValidateSink to reuse validation code

d9fa031

pront reviewed Jul 30, 2026

View reviewed changes

		if uri.uri.host().is_none() {
		errors.push(format!("endpoint '{endpoint}': must include hostname"));

	) -> Result<(), Box<dyn std::error::Error + Send + Sync + 'static>> {
	) -> Result<(), vector_common::Error> {

Uh oh!

Conversation

thomasqueirozb commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vector configuration

How did you test this PR?

Is this a breaking change?

Does this PR include user facing changes?

References

Uh oh!

datadog-vectordotdev Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thomasqueirozb commented Jul 28, 2026 •

edited

Loading

datadog-vectordotdev Bot commented Jul 28, 2026 •

edited

Loading