perf(sheets): cap fan-out cell-matrix materialization to prevent OOM#1578
Open
zhengzhijiej-tech wants to merge 3 commits into
Open
perf(sheets): cap fan-out cell-matrix materialization to prevent OOM#1578zhengzhijiej-tech wants to merge 3 commits into
zhengzhijiej-tech wants to merge 3 commits into
Conversation
The +cells-set-style / +dropdown-set / +cells-batch-set-style / +dropdown-update shortcuts expand a single A1 range into a rows×cols matrix of per-cell maps client-side (the backing set_cell_range tool takes an explicit cells matrix). rangeDimensions() had no upper bound, so a tiny input like "A1:Z100000" balloons into ~2.6M heap maps (~900MB, doubled again by json.Marshal) and can OOM the process before the request is even sent. Add a 50000-cell safety cap (checkStampMatrixBudget) gating every fan-out materialization point, matching the documented but never-wired --max-cells default. Oversized ranges now fail fast with a clear validation error instead of allocating. Also preallocate the per-op slices now that the range count is known up front. Adds benchmarks + a boundary test as regression guards.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…f the cell-matrix cap) The single-range fan-out cap (maxStampMatrixCells) left three sibling ingress paths uncapped, each able to materialize an unbounded matrix or op set in memory before the request leaves: - +table-put / +workbook-create --sheets/--values: buildSheetMatrix builds the whole rows×cols matrix before slicing it into per-write batches; tablePutMaxCellsPerWrite only bounds the batch size, not the total input. Add tablePayload.checkCellBudget (1M-cell guardrail), enforced in validate() and in buildValuesPayload (the --values path bypasses validate()). - batch fan-out (+cells-batch-set-style / +dropdown-update): per-range checkStampMatrixBudget can't stop many ranges from summing past the cap. Add an aggregate cell budget (checkBatchStampBudget) and a shared maxBatchRanges (100) count cap in validateDropdownRanges — covering all fan-out commands and replacing the now-redundant +dropdown-delete count check. - +batch-update: cap --operations at maxBatchOperations (100) in translateBatchOperations. Adds boundary regression tests for each cap. go vet + gofmt clean; full shortcuts/sheets + backward suites green.
Add BenchmarkBuildSheetMatrix_* and TestTablePutMatrixPeakMemory mirroring the fan-out probes. Confirms the +table-put/+workbook-create ingress has the same OOM profile as the single-range stamp: 2.6M cells → ~917 MB / 5.3M allocs (+875 MB resident heap) materialized before the first write — now rejected up front by checkCellBudget.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The fan-out / stamp shortcuts (
+cells-set-style,+dropdown-set,+cells-batch-set-style,+dropdown-update) expand one A1 range into arows×colsmatrix of per-cell maps client-side, because the backingset_cell_rangetool takes an explicit cells matrix.rangeDimensions()enforced no upper bound, so a short input string explodes into millions of heap maps with no guard.Measured cost of materializing a single range in one shortcut call (
go test -bench+ aruntime.MemStatsprobe):A1:J10000A1:Z100000That is enough to OOM a normal CI container / laptop before the request is even sent — and
json.Marshaldoubles it again on the way out.Fix
Add a 50000-cell safety cap (
checkStampMatrixBudget) at every fan-out materialization point, beforefillCellsMatrixallocates. The value matches the documented--max-cellsdefault, which was never actually wired up in code. Oversized ranges now fail fast with a clear validation error pointing the user to narrow / split the range.Also preallocate the per-op slices (
make([]interface{}, 0, len(ranges))) now that the range count is known up front.After (same
A1:Z100000input): 917 MB / 5.3M allocs → 377 B / 5 allocs. Legitimate ranges (≤ 50000 cells) are unaffected.Tests
BenchmarkFillCellsMatrix_*+TestFanoutMatrixPeakMemorydocument the cost.TestStampMatrixBudgetCapcovers the 49998 / 50000 / 50001 / 2.6M boundaries.go vet,gofmt, and the fullshortcuts/sheets+backwardtest suites pass.Follow-up: sibling cap gaps
The single-range cap above left three sibling ingress paths that could materialize an unbounded matrix / op set the same way, before the request leaves:
+table-put/+workbook-create(--sheets/--values) —buildSheetMatrixbuilds the wholerows×colsmatrix in memory (measured: 2.6M cells → ~917 MB / 5.3M allocs, +875 MB resident heap — the same OOM profile as the single-range stamp) before slicing it intotablePutMaxCellsPerWrite-sized writes, sotablePutMaxCellsPerWritebounds only the per-write batch, not the total input. AddedtablePayload.checkCellBudget(1,000,000-cell guardrail), enforced invalidate()and inbuildValuesPayload(the--valuespath bypassesvalidate()).+cells-batch-set-style/+dropdown-update) — the per-rangecheckStampMatrixBudgetcan't stop many ranges from summing past the cap. Added an aggregate cell budget (checkBatchStampBudget, 50000) plus a sharedmaxBatchRanges(100) count cap invalidateDropdownRanges— covering all fan-out commands and replacing the now-redundant+dropdown-deletecount check.+batch-update— capped--operationsatmaxBatchOperations(100) intranslateBatchOperations.New boundary tests:
TestTablePutCellBudgetCap,TestBatchStampAggregateCap,TestBatchFanoutRangeCountCap,TestBatchOperationsCountCap, plus theTestTablePutMatrixPeakMemory/BenchmarkBuildSheetMatrix_*probes that produced the number above.go vet/gofmtclean; fullshortcuts/sheets+backwardsuites green.