turbo-tasks-backend: parent_count-driven garbage collection - #97282
Draft
lukesandberg wants to merge 4 commits into
Draft
turbo-tasks-backend: parent_count-driven garbage collection#97282lukesandberg wants to merge 4 commits into
lukesandberg wants to merge 4 commits into
Conversation
Contributor
Stats from current PR✅ No significant changes detected📊 All Metrics📖 Metrics GlossaryDev Server Metrics:
Build Metrics:
Change Thresholds:
⚡ Dev Server
📦 Dev Server (Webpack) (Legacy)📦 Dev Server (Webpack)
⚡ Production Builds
📦 Production Builds (Webpack) (Legacy)📦 Production Builds (Webpack)
📦 Bundle SizesBundle Sizes⚡ TurbopackClient Main Bundles
Server Middleware
Build DetailsBuild Manifests
Build Cache
📦 WebpackClient Main Bundles
Polyfills
Pages
Server Edge SSR
Middleware
Build DetailsBuild Manifests
Build Cache
🔄 Shared (bundler-independent)Runtimes
📎 Tarball URLCommit: 042f035 |
Contributor
Failing CI jobsCommit: 042f035 | About building and testing Next.js |
lukesandberg
changed the title
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
| // Never persisted: there is nothing on disk to tombstone, so drop it out of the | ||
| // next snapshot's scan entirely (clearing its modified bits + shard count). | ||
| // Eviction still removes the resident, soft-`deleted` task. | ||
| task.discard_modifications_for_gc_new_task(); |
Contributor
There was a problem hiding this comment.
GC clearing new_task on a never-persisted collected task, followed by resurrection before eviction, persists the task without its TaskCache (type→id) mapping, breaking cache dedup on the next session restore.
…ator GC mutates the task graph, so it needs the same exclusion against running operations that the snapshot already has. Generalize the coordinator from one exclusion (snapshot) to two (snapshot, GC): a second request bit, a shared `begin_exclusion` that both entry points delegate to, and `snapshot_pending` renamed to `exclusion_pending` now that it covers both. The important piece is `GcPhase::into_snapshot`: GC must hand its exclusion directly to the snapshot that tombstones what it collected. Swapping the flags under one lock leaves no window where an operation could start and resurrect a just-collected task. `suspend_point` now records the suspended operation unconditionally, because a GC phase can hand off to a snapshot that must persist it for replay.
A collected task is not erased immediately. It is flagged `deleted` and stays resident until the next snapshot tombstones its on-disk copy — that residency is what closes the resurrection window, since any `ctx.task` during the pass finds a live entry instead of restoring a zombie from disk. `resurrect_deleted` clears the flag and re-dirties the task if something connects to it first. `is_gc_collectible` is the authoritative predicate; `gc_maybe_collectible` is the storage-only half so the resident-map scan can use it without a guard. Its `is_restored(Meta)` gate is load-bearing: `parent_count` and the aggregation edges are Meta fields, so an evicted task reads them as 0/empty and would look collectible when its persisted Meta says otherwise. The collector discovers cascade candidates through two context buffers — tasks whose `parent_count` hit 0, and tasks that lost their last aggregation edge. Both are no-ops for every normal operation context. `TaskGuardImpl::category` becomes `Option` so `resident_task` can hand out a guard that restored no category at all: it touches only transient fields, and `check_access` must reject any Meta/Data read through it.
The collection pass. Seeds one job per resident-map shard, scans each for tasks passing the cheap collectibility pre-filter, and collects them in parallel over the self-feeding work queue: re-check under the write guard, soft-delete, scrub the outgoing edges via CleanupOldEdges, then cascade into whatever that freed. A task never persisted is dropped from the snapshot scan outright; there is nothing on disk to tombstone. Deletion completes across the following snapshot and eviction: the snapshot emits a `Delete` tombstone for each soft-deleted task, and eviction hard-deletes the entry, re-checking the flag under the shard write lock so a resurrection in between falls through to normal eviction instead. `prevent_gc()` becomes real, pinning the calling task through `transient_ref_count`. `OperationVc::task_id` lets a detached handle held outside the tracked graph pin its task the same way. Off by default; `TURBO_ENGINE_GC` enables it. Forced off on a ReadWrite backend with eviction disabled, where soft-deleted tasks would never be reclaimed.
Splits the GC tests out of `parent_count.rs` into `gc_collection.rs` (the collection cascade, pinning, and the erase-while-referenced invariant), leaving that file to the reference-counting tests. `gc_resurrection.rs` covers the races soft-deletion exists to close: an aggregation rebalance that must cascade in the same pass, a forward-dependency scrub racing a concurrent collect, and reconnecting a collected task before its tombstoning snapshot. `gc_stress.rs` is the anti-leak test — repeated re-rooting must return the resident count to a flat baseline rather than climbing per round — plus a wide per-task fan-out torn down in a single collect. `benches/gc.rs` (behind `TURBOPACK_BENCH_GC`) measures collection throughput over wide disconnected graphs.
lukesandberg
force-pushed
the
lukesandberg/gc-035-gc-collection
branch
from
August 13, 2026 06:52
8fb7af4 to
042f035
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds garbage collection, built on the
parent_countreference countingHow collection works
For removing edges we rely on the existing
CleanupOldEdgesoperation, though it is enhanced to collect the tasks that become collectible.A collected task is not erased immediately. It is flagged
deletedand stays resident until the next snapshot tombstones its on-disk copy. This opens up a race where a new reference to the task can be added after deletion. This is handled via newresurrect_deletedfunction that manages this when connecting children.Coordinator changes
GC mutates the graph, so it needs the same exclusion against running operations that the snapshot already has. The coordinator goes from one exclusion to two, with a shared
begin_exclusionboth entry points delegate to.The subtle piece is
GcPhase::into_snapshot: GC has to hand its exclusion directly to the snapshot that tombstones what it collected. Swapping the flags under one lock leaves no window where an operation could start and resurrect a just-collected task,Enabling it
Off by default;
TURBO_ENGINE_GCturns it on. Forced off on a ReadWrite backend with eviction disabled, where soft-deleted tasks would never actually be reclaimed.prevent_gc()becomes real, pinning the calling task throughtransient_ref_count.OperationVc::task_idlets a detached handle held outside the tracked graph pin its task the same way.Testing
gc_collection.rs— the collection cascade, pinning, and the erase-while-referenced invariant.gc_resurrection.rs— the races soft-deletion exists to close: an aggregation rebalance that must cascade in the same pass, a forward-dependency scrub racing a concurrent collect, and reconnecting a collected task before its tombstoning snapshot.gc_stress.rs— the anti-leak test (repeated re-rooting must return the resident count to a flat baseline rather than climbing per round), plus a wide per-task fan-out torn down in a single collect.benches/gc.rs, behindTURBOPACK_BENCH_GC, measures collection throughput over wide disconnected graphs.