feat(ci): Add revdep4, sequential halves driven by a two-ended bash work queue

[krlmlr]

Proposes revdep4, the queue engine: old and new halves of each reverse dependency checked sequentially, with the lost concurrency won back across packages by a custom bash work queue — one of two sibling proposals (the other is revdep3, #2856). Both reuse the good bits of revdep2 through a shared, engine-agnostic core, and are fully interoperable with each other.

The problem

revdep2 checks each reverse dependency against the released and the dev igraph simultaneously on one host. The PSOCK port collision (patched with R_PARALLEL_PORT) was one member of an open-ended interference class — shared /tmp, caches, locks, any singleton a check assumes it owns. Simultaneously checking the same package with two libraries is not a supported mode of operation.

What revdep4 does

Removes the simultaneity instead of isolating it: per package, the old half runs, then the new half — no overlap, ever. Parallelism moves to the package level: a shard runs workers packages at once (default 4, the runner's cores), each check in its own container, so different packages cannot collide either (stronger than revdep2, which only separated the two halves' ports).

The queue (revdep4/queue.sh, bash, flock-guarded cursors) consumes the shard's heaviest-first package list from both ends: one heavy lane takes the heaviest remaining package — the longest checks start first and are never the straggler discovered last — while the remaining light lanes drain the long tail of cheap packages from the other end; the two meet in the middle. Deadline-aware claiming defers what no longer fits; a crashed claim still writes an error manifest line through a four-rung fallback ladder (compare-one → --error rerun → printf-JSON template → supervisor sweep), so no claimed package can ever vanish silently; three check slices with an upload after each bound what a reclaimed runner can lose.

Heaviness — answering the design question directly: the queue key is expected check seconds — measured on this infrastructure by earlier revdepx runs (either workflow, youngest wins), else CRAN's T_total × a self-calibrating scale, else the cohort median. Package size and dependency count/size are deliberately not in the key: they price installation, which the shared universe image amortizes to zero (dependency count still gates the depfail screen). This is revdep2's proven cost model, re-priced for sequential halves (×2 per package, ÷workers per shard, with a max(heaviest, sum/workers) bound so a giant-dominated shard is not flattered). In the live runs the model self-corrected on its first opportunity: the second full run consumed the first's measured timings and cut 20 shards in one wave instead of 40 in two.

Sequential halves make per-half timings real: t_old/t_new become true measurements (revdep2 could only record the pair's wall clock for both), which sharpens the cost model over time.

The old half always runs; a stored old result is a second opinion only. Where the plan certifies an earlier run's old-version result as comparable — same revdep version, our CRAN version, container R, base-image tag, dependency fingerprint, within age — the fresh old check is compared against it and baseline_agrees records whether they match, with disagreements reported as drift. It never substitutes for the check: a fresh old is the only result whose provenance the run fully controls. (Baselines from a different platform — another R version, another base image — are never offered; the key walls them off.)

Pipeline (shared with revdep3): plan + base image (parallel) → build (dev binary inside the container) → universe image (whole dependency universe + sysreqs, delta-updated, age-bounded) → test shards (queue) → collect. Containers also bring per-check memory caps (default 6g; an OOM kills one container, not the runner — observed working live), low-memory check compiles (-g0, MAKEFLAGS=-j1 inside the container; both halves identical, CRAN flags restorable via REVDEPX_CHECK_FLAGS), and a pinned check platform (r-version, default 4.6.1 — at least one reverse dependency needs R-release) with qpdf/ghostscript/pandoc/TeX/Xvfb, the tcl/tk runtime (tcltk is a base R package, invisible to pak's sysreqs machinery) and Rust (six revdeps compile cargo crates). Dependency metadata spans CRAN and Bioconductor — the checked packages are CRAN reverse dependencies, but what they depend on may live in either repository.

Kept from revdep2 / dropped

Kept (via .github/workflows/revdepx/): the planner with all its inputs (packages/broken/retry-run/part/dry-run), the manifest schema and result vocabulary, baseline lineage, timings + calibration, three-slice uploads, resource sampler, chunked deadline-bounded installs, load test, comparison/diff/salvage, report/collect and the committed revdep/ record. Dropped: the port hack, the preflight job and all prebuilt-library artifacts/donor walks/tar machinery, host toolchain setup on shards, per-shard host installs (kept only as disaster-recovery fallback when the universe image is unavailable — a fallback the live runs exercised end-to-end, repeatedly). revdep2 itself is untouched by this PR.

Compatibility with revdep3

Both PRs ship .github/workflows/revdepx/ byte-for-byte identical (this PR adds revdep4.yaml + revdep4/; #2856 adds revdep3.yaml + revdep3/; merging both leaves one copy, no conflicts). One artifact family (revdepx-*), one manifest/baseline/timings schema, one comparison code path, one universe-image lineage, one concurrency group per ref. Either workflow reads the other's baselines as second opinions, consumes the other's timings, retries the other's runs, and starts from whichever universe image was refreshed last.

Live-test results

The full validation run (32158907637, most, depth 2, R 4.6.1) completed green end to end: 3435 reverse dependencies, 20 shards in one wave, ~6.5 h wall clock — 3231 ok, 21 newly broken, 49 fail to install, 121 with uninstallable dependencies, 6 new-half timeouts. revdep3's cross-run of the same parameters (32196879628) reproduced all 21 newly-broken on the other engine and resolved the residual buckets: five of the six "timeouts" were queue-worker core contention at the 1200 s floor, not dev slowdowns; the dmesg watch caught the kernel OOM-killing cc1plus at the derived ~3.4g per-check cap while the report said only "fails to install"; and the 121 uninstallable were all Bioconductor, dropped by the planner's CRAN-only metadata before pak (whose pinned repositories included Bioconductor all along) ever saw them.

Follow-up commits closed each cause: dependency metadata now merges the Bioconductor repositories (install union 4406 → 4675), the per-check cap defaults to 6g, check compiles run with -g0/-j1, failures.md carries the real tail of 00install.out (a container-path mismatch had left it "<00install.out file does not exist>"), and a detected compiler kill is named in the manifest message.

A targeted rerun of all 204 not-ok packages (32260705703, packages: broken) validated the lot in one pass: 171 of 204 ok — all 121 uninstallable-dependency packages resolved, 42 of 49 install failures installed (blavaan, gllvm and multinma among them, confirming the memory diagnosis), all 21 newly-broken reproduced. The universe job took the delta path for the first time (145 new packages on top of the published :latest, pushed in 50 min), and the new install-log capture identified the entire remaining residue at a glance: all six failures were sh: 1: rustc: not found — one missing toolchain, not memory.

A final rerun on the Rust-enabled base (32281237129) closed the loop: all six Rust packages now install and check ok, and the committed record survived a subset run intact at 3435 rows — validating the collector fix this rerun's predecessor exposed (a subset run used to replace the committed manifest with its own rows; rows outside a run's plan are now carried). It also surfaced one last sysreqs blind spot, fixed here: tcltk is a base R package, invisible to pak's sysreqs, so a universe built for a small package set shipped no libtcl and the three tcltk-using packages failed to install — the base image now carries the tcl/tk runtime it already had Xvfb for. Current committed record: 3410 ok · 20 newly broken · 3 failed (the tcltk trio, cured by the next run's base) · 1 timeout (ctmm) · 1 deferred. Two earlier newly-broken (c3net, ggm) passed on re-check — flaky rather than broken.

The stable newly-broken set (reproduced on both engines and on re-check): cfid, comato, cranly, dci, degreenet, ECoL, ggraph, glyrepr, GoodFitSBM, manynet, MetaNet, migraph, nat, netrics, R6causal, scistreer, SEMgraph, sfclust, tidygraph, vkR.

Notes for review

• Nine push-triggered live runs drove out every infrastructure bug on this branch, each fixed here: docker commit filling the runner's single disk killed two runners with their logs (now: a measured guard, a docker-stop watchdog, and a pre-commit evidence artifact uploaded before the copy starts — a dead runner loses even its streamed log lines); file.rename() across bind mounts silently dropped the dev binary from its artifact (EXDEV; now a checked copy); a 755 /tmp bind mount broke every apt call via apt-key's temp files (now 1777, reproduced and verified locally); each check slice truncated the shared manifest (file.create() truncates) and destroyed two thirds of a run's finished results (now create-if-absent; proven fixed by the final run's 4 deferred vs 2293 before); the runners' containerd image store made committed images unpushable OCI indexes and docker save a 1.3 kB shell (now: the classic graphdriver store via the daemon.json the workflow already writes, plus a 100 MB save floor and shard-prep falling through to a local build on any bad image); pak::pkg_sysreqs() over all 3435 packages was OOM-killed at 14 GB (now chunked, per-chunk fault-tolerant, with memory caps on build containers); Tk-based packages died installing headless (now Xvfb in the base image, used for installs, load tests and checks); and the one-at-a-time install salvage became three-way divide-and-conquer (5–29× fewer pak calls in simulation).
• The universe image could not publish under the containerd store, so three full runs ran entirely on the shard-local fallback (~35 min extra per shard) — the degraded path is production-proven. The classic-store fix is proven too: revdep3's validation run published the universe image to GHCR, all 40 of its shards pulled it in minutes, and the broken-packages rerun rebuilt it as a 50-min delta on top.
• The temporary push triggers that drove the live runs are reverted: both workflows are dispatch-only again with strong/depth-1 defaults restored and the check platform pinned at R 4.6.1.
• First push after merge creates the GHCR packages; org policy forbidding GITHUB_TOKEN package writes degrades gracefully to artifact transport, and a missing image to shard-local builds.

---

• By submitting this pull request, I assign the copyright of my contribution to The igraph development team.