feat(file sink): add parquet batch encoding to file sink

[sonnens]

Summary

add the 'parquet' batch encoder to the file sink

Vector configuration

  filetest:
    type: file
    path: /data/class_uid={{ .class_uid }}/{{class_name}}.parquet
    inputs:
      - ocsf
    encoding:
      codec: json
    batch:
      timeout_secs: 30
    batch_encoding:
      codec: parquet
      compression:
        algorithm: snappy
      schema_mode: relaxed
      schema_file: /data/schema/findings/detection_finding.parquet.schema

How did you test this PR?

with a parquet schema that includes struct & array, being fed json objects

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fa2e5e3ab9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9a8a39ca6f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Do not acknowledge filtered non-log events

When the upstream output advertises multiple event types (Vector's graph allows an all_bits output into a log-only sink), metric or trace events can still reach this Parquet path. ParquetSerializer::encode filters those non-log events out and returns Ok(()) (even with an empty buffer if the whole batch was non-log), but this success branch marks the finalizers for the original batch as Delivered, so end-to-end acknowledgements report dropped metric/trace events as successfully written and the sent counters include events that never made it into the Parquet file. Filter or reject non-log events before taking the batch finalizers, or mark the whole batch errored when any event is discarded by the encoder.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Confine paths before partitioning batches

When path contains event-controlled fields, an outside-base_dir value still becomes a live partition key here and is retained by batched_partitioned until the batch fills or the 300s default timeout expires; sending many distinct invalid paths can therefore create an unbounded set of batches that will only be rejected later in write_batch. Confine the rendered path in the partitioner and return None/error for rejected paths so malicious keys are dropped before they allocate batch state.

Useful? React with 👍 / 👎.