feat(buffers): add reject when_full mode for disk buffers

[PradeepSundarajan]

Summary

Adds a new reject value for a sink buffer's when_full option. Like drop_newest, it drops the newest event when the buffer is full, but it acknowledges the dropped event as errored instead of as a successful delivery. This lets a source with acknowledgements enabled signal the failure back to its client rather than silently reporting success for data that Vector actually discarded — for example, the http_server source returns 500 Internal Server Error instead of its configured success code when the buffer is full. This applies to both disk and memory buffers.

Implementation notes:

• New WhenFull::Reject variant.
• Disk buffers: wired through as an error_on_full flag on DiskBufferConfig (set only for reject). On the buffer-full shed paths, the disk writer resolves the shed record's finalizers as Errored (via the existing FinalizerGuard) instead of letting them resolve as delivered. Block/overflow paths are unchanged (finalizers reattached for retry).
• Memory buffers: the buffer sender marks a shed event's finalizers Errored in the Reject arm. For disk buffers this is a no-op — the writer has already errored and detached them.
• drop_newest is behaviorally unchanged (best-effort; shed events acknowledged as delivered) for both disk and memory.

Vector configuration

sources:
  http_in:
    type: http_server
    address: 0.0.0.0:8080
    # Required for buffer-full failures to surface as HTTP responses.
    acknowledgements:
      enabled: true

sinks:
  http_out:
    type: http
    inputs: [http_in]
    uri: https://example.com/ingest
    encoding:
      codec: json
    buffer:
      type: disk
      max_size: 268435488 # 256 MiB (minimum)
      # Drop the newest event when full AND signal failure to the source,
      # so http_in returns 500 instead of its success code.
      when_full: reject

How did you test this PR?

Added unit tests in vector-buffers:

• writer_marks_shed_record_errored_when_reject — a full disk reject buffer resolves the shed record's finalizers as Errored.
• test_sender_reject_memory_sheds_as_errored — a full memory reject buffer marks the shed event Errored.
• test_sender_drop_newest_memory_sheds_as_delivered — contrast: drop_newest remains best-effort and rolls a shed event up as Delivered.

Verified locally:

• cargo test -p vector-buffers — 109 passed, 2 pre-existing ignores (run single-threaded; the disk tests collide on file locks under in-process parallelism).
• cargo clippy -p vector-buffers --all-targets — clean (deny(pedantic)).
• vector-common (33) and vector-config (44) unit tests pass.

End-to-end, using a build of this branch against real Kafka (http_server → disk buffer with when_full: reject → kafka):

• With Kafka stopped, POSTs that fit the buffer returned 202; once the 256 MiB buffer filled, shed POSTs returned 500 (vector_buffer_events stayed at the accepted count — rejected events never entered the buffer).
• After restarting Kafka, all buffered (202) events drained to the topic with no loss, while the rejected (500) events were correctly refused for the client to retry.

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Changelog fragment added at changelog.d/reject_when_full.feature.md.
• No. A maintainer will apply the no-changelog label to this PR.

References

Notes

• Docs updated: added a reject section to website/content/en/docs/architecture/buffering-model.md and the generated when_full enum in website/cue/reference/components/generated/sinks.cue (run make generate-component-docs to confirm the generated file matches before merge).
• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[PradeepSundarajan]

I have read the CLA Document and I hereby sign the CLA

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 25837f7204

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Regenerate the full configuration reference for reject

Adding reject here updates the per-component sink metadata, but website/cue/reference/generated/configuration.cue still lists only block and drop_newest for sinks.*.buffer.when_full. As a result the generated public configuration reference remains out of sync and make check-generated-docs will rewrite/fail on this change; please regenerate and commit the full generated configuration docs along with this enum update.

AGENTS.md reference: AGENTS.md:L249-L255

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Record disk reject drops in buffer metrics

When a disk buffer in reject mode is full, this branch only errors the finalizers before returning the record as Full; because disk buffers provide their own instrumentation, the outer BufferSender does not install the UsageAccounting::DroppedNewest counters for this stage. In production this makes rejected disk-buffer events surface as client 500s without any corresponding buffer discarded/drop metric, so please update the ledger/usage accounting before returning here and in the recovered-record full path.

Useful? React with 👍 / 👎.