dbt-fusion panics w "Arrow byte array offset overflow" during Elementary on_run_end even after successful model/test runs

[ezra-y-lee]

Describe the bug
After migrating high-volume dbt workloads from dbt-core (1.10.5) -- (and dbt-bigquery: 1.10.3) -- to dbt-fusion (2.0.0-preview.173), model builds and tests complete successfully, but the dbt-fusion engine panics during Elementary’s (version 0.22.1) on_run_end hook with Arrow’s byte array offset overflow. We confirmed that this is not an OOM related to our K8 instances--that is, raising pod memory to 18 GB only unmasked the engine panic. The failure is reproducible and occurs at the same step after successful Elementary uploads/inserts. Notably, when our stack was on dbt-core we did not observe these errors. However, we began observing this issue once migrated to dbt-fusion.

To Reproduce

• dbt work succeeds end-to-end (model + tests).
• Failure happens after successful Elementary on_run_end logging/uploads.

Panic message:

panic: panicked at .../arrow-rs-.../arrow-array/src/builder/generic_bytes_builder.rs:86:57:
byte array offset overflow

• Reproduced across retries with identical timing (~40s after Elementary insert activity).
• Example run: ac6bfd22 on 18 GB pods (-1-thncr, -2-5x8l2).

Per-attempt sequence

• example model builds — Succeeded / success (~68s).
• All tests Pass.
• Elementary on_run_end proceeds successfully:
• Elementary: Uploaded run results successfully
• Inserting 26 rows to ... test_result_rows
• Running insert query — outcome: success
• ~40s later → Fusion engine panics with byte array offset overflow.

Expected behavior

• After models and tests succeed, Elementary’s on_run_end should finish (or fail with a clear, non-fatal dbt/adapter error)—not crash the process.
• Large BigQuery result sets fetched via run_query / Elementary hooks should be handled safely (e.g. chunked batches, LargeUtf8, size limits with a recoverable error)—not an Arrow Rust panic (byte array offset overflow).
• A successful model + tests + Elementary upload/insert path should leave the run in a completed state (success or soft failure), so orchestrators (e.g. Dagster) are not forced into identical hard-crash retries.

Screenshots

Questions for Elementary Data:
Are there Fusion-specific mitigations in newer package versions (> 0.22.1) for on_run_end result handling volume?
Recommended pattern under Fusion for high-volume anomaly tests: reduce payload (days_back, dimensions), disable result-row storage, or skip certain on_run_end uploads?
Any guidance on avoiding run_query paths that materialize large string columns during handle_tests_results / metrics insertion?

Environment (please complete the following information):

dbt-core: 1.10.5
dbt-bigquery: 1.10.3
dbt-fusion: 2.0.0-preview.173
Elementary dbt package: 0.22.1 (vendored locally as dbt_elementary)
Warehouse: BigQuery
Runner: Dagster/K8s (18 GB memory limit)

[aw-stevens]

Possibly related — we're seeing what may be the same underlying problem on a newer Fusion build, with a different symptom.

Same shape as yours: dbt-fusion + BigQuery, on_run_end fails after a successful Elementary insert, fine on dbt-core. Ours doesn't panic though — it hangs indefinitely with no output until the orchestrator kills it on an inactivity timeout.

	#2309	#2318
Fusion	2.0.0-preview.173	2.0.0-preview.205
Elementary	0.22.1	0.25.1
Symptom	Arrow byte array offset overflow panic	indefinite hang, no error
Timing	~40s after insert activity	immediately after insert, never returns

We isolated ours by bisect to upload_run_results(), and specifically to after its INSERT — BigQuery shows the INSERT reaching DONE, then no further query is ever issued. The next statement is adapter.commit() at insert_rows.sql:78.

Flagging in case they share a root cause in the post-insert path — if they're distinct, ignore the noise.