Skip to content

feat(aws_s3): add sqs.on_missing_object to dead-letter missing objects#4475

Open
peczenyj wants to merge 5 commits into
redpanda-data:mainfrom
peczenyj:aws-s3-sqs-on-missing-object
Open

feat(aws_s3): add sqs.on_missing_object to dead-letter missing objects#4475
peczenyj wants to merge 5 commits into
redpanda-data:mainfrom
peczenyj:aws-s3-sqs-on-missing-object

Conversation

@peczenyj

@peczenyj peczenyj commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Description

The SQS-driven aws_s3 input currently has a single, hard-coded behaviour when an SQS notification points at an S3 object that cannot be downloaded because it no longer exists (NoSuchKey / 404): it logs a warning, acks the notification (DeleteMessage) and discards it. The notification can therefore never reach an SQS dead-letter queue.

That is the correct default when delete_objects is enabled (a redelivered notification for an already-deleted object is expected and harmless), but it is wrong for pipelines that need a missing object to be treated as a delivery failure and dead-lettered for inspection/replay.

This PR adds an advanced enum field sqs.on_missing_object:

  • drop (default, the historical behaviour): warn, ack, discard.
  • nack: log an error and return the notification to the queue (the existing nack path) so an SQS redrive policy can dead-letter it after maxReceiveCount.

To avoid a foot-gun, combining on_missing_object: nack with delete_objects: true is rejected at config-parse time, since a redelivered notification for an object this input legitimately deleted would otherwise be dead-lettered.

The default value preserves existing behaviour exactly, so this change is fully backward compatible.

Changes

  • New sqs.on_missing_object string enum field (drop | nack), Advanced(), default drop.
  • Config-parse guard rejecting nack + delete_objects: true.
  • Missing-object branch in readSQSEvents now routes NoSuchKey to the nack path when nack is selected (ERROR log) instead of always dropping (WARN log).
  • Unit test covering the default, both values, and the rejected combination.
  • Regenerated aws_s3.adoc and a CHANGELOG entry.

Fixes #4474

peczenyj and others added 5 commits June 1, 2026 19:23
The SQS-driven aws_s3 input currently has a single hard-coded behaviour
when an object referenced by a notification cannot be downloaded because
it no longer exists (NoSuchKey/404): it warns, acks the notification and
discards it, so the notification can never reach an SQS dead-letter queue.

This adds an advanced enum field `sqs.on_missing_object`:

  - `drop` (default, the historical behaviour): warn, ack, discard.
  - `nack`: log an error and return the notification to the queue so an
    SQS redrive policy can dead-letter it after maxReceiveCount.

Combining `on_missing_object: nack` with `delete_objects: true` is
rejected at config-parse time, since a redelivered notification for an
object this input legitimately deleted would otherwise be dead-lettered.

The default preserves existing behaviour exactly (fully backward
compatible).

Fixes redpanda-data#4474

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

aws_s3 input: option to nack (dead-letter) SQS notifications for missing S3 objects

1 participant