Make distributed compilation work for OpenEmbedded/Yocto builds#2750
Closed
jetm wants to merge 43 commits into
Closed
Make distributed compilation work for OpenEmbedded/Yocto builds#2750jetm wants to merge 43 commits into
jetm wants to merge 43 commits into
Conversation
This was referenced Jun 23, 2026
AJIOB
reviewed
Jun 24, 2026
| // either succeeds (confirming a dist-only artifact) or reproduces the | ||
| // real error locally, so a remote failure never breaks a build that | ||
| // would compile fine locally. This only affects failing dist | ||
| // compiles; successful ones are returned unchanged above. |
Contributor
There was a problem hiding this comment.
This looks like a fix for #2700.
IMO, please, do a dedicated PR with that fix & test it to mitigate possible future regressions
Collaborator
Contributor
There was a problem hiding this comment.
Maybe yes. IMO, your code part should resolve that problem
Collaborator
There was a problem hiding this comment.
I actually haven't written any code in that area (yet, at least)
sylvestre
reviewed
Jun 24, 2026
| Some(dc) => dc, | ||
| None => { | ||
| debug!("[{}]: Compiling locally", out_pretty); | ||
| info!( |
Collaborator
There was a problem hiding this comment.
should be done in a different pr
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2750 +/- ##
===========================================
- Coverage 74.19% 37.22% -36.97%
===========================================
Files 71 56 -15
Lines 40505 16373 -24132
===========================================
- Hits 30053 6095 -23958
+ Misses 10452 10278 -174 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
get_signal did `status.signal().expect("must have signal")`, assuming the
Unix invariant that an ExitStatus with no exit code was terminated by a
signal. That does not always hold: an ExitStatus reconstructed for a
distributed compile (or an abnormal wait status such as WIFSTOPPED) can
report neither a code nor a signal. When that happened the expect() panicked
the compile task, which the server surfaced as a misleading "Failed to bind
socket" and, under load, repeatedly fell back to local compilation.
Return Option<i32> from get_signal and assign it straight into res.signal, so
a compile that reports neither code nor signal leaves res.signal unset
instead of crashing the in-flight task. The Windows arm returns None rather
than panicking; ExitStatus::code() is always Some there, so the signal branch
is never reached anyway.
Add a unit test covering a real terminating signal (SIGKILL) and the
neither-code-nor-signal case (WIFSTOPPED via from_raw), which previously
panicked.
Signed-off-by: Javier Tia <javier@peridio.com>
sccache-dist packages a toolchain's shared libraries by parsing `ldd` output, which resolves NEEDED libraries against the host's dynamic loader. Yocto/OpenEmbedded "uninative" cross toolchains ship a relocated glibc whose loader has a built-in search path pointing at its own sysroot. For those binaries `ldd` reports host paths (e.g. /usr/lib/libm.so.6), yet inside the build sandbox the relocated loader searches its own sysroot lib dir, where those libraries were never packaged. The remote compile then dies with "libm.so.6: cannot open shared object file" and silently falls back to local compilation, so distribution never actually runs. When a packaged executable's PT_INTERP lives outside the standard host loader directories, also bundle the interpreter's own directory. That directory holds the libc/libm the relocated loader resolves against, so they land at the absolute path the loader searches inside the sandbox. Standard host toolchains are untouched: their interpreter is under /lib, /lib64, or /usr/lib, so the existing ldd-only path is preserved. Signed-off-by: Javier Tia <javier@peridio.com>
sccache-dist ships the preprocessed input through the inputs packager keyed on the absolute, simplified path cwd.join(input) (CInputsPackager), but the distributed compile command referenced the raw parsed_args.input. For out-of-tree builds the input is relative (e.g. OpenEmbedded's ../sources/foo.c), so the command and the packaged input disagreed and the build-server compiled a path the inputs were never placed at, failing with "cc1: fatal error: ... No such file or directory". Transform the same absolute, simplified path in the dist command so it matches the packaged input. An absolute input is unchanged, since cwd.join of an absolute path returns it verbatim. Signed-off-by: Javier Tia <javier@peridio.com>
sccache logged the distribute-vs-local decision only at debug ("Compiling
locally", "Attempting distributed compilation"), while an infrastructure
fallback warned. At info a successful distribution and a local compile were
both silent, so the only visible dist signal was the failure path - leaving
no way to see the distribute/fallback ratio without the full debug firehose.
Diagnosing why a distributed build under-distributes meant guessing.
Promote the decision points to info and add a log on successful
distribution naming the server and exit code, so SCCACHE_LOG=info gives a
per-compile dist trace (attempt then distributed-on-server, compiled-locally,
or falling-back-with-reason). sccache only emits logs when SCCACHE_LOG is
set, so default runs are unaffected.
Signed-off-by: Javier Tia <javier@peridio.com>
A distributed compile the build-server rejects is often a distribution artifact, not a genuine compiler error: an object that .incbin's a binary the inputs packager does not ship - the kernel's vdso, embedded-config, and dtb wrappers - cannot be assembled remotely and returns non-zero, which failed the whole build with no recourse and forced the kernel to be excluded wholesale. Treat a non-zero remote result as a fallback trigger rather than a terminal error: recompile locally, which either succeeds (confirming a dist-only artifact) or reproduces the genuine error. A remote failure can no longer break a build that would compile locally. Only failing dist compiles are affected - successful ones are returned unchanged - so the kernel now distributes (1124/1128 compiles), with its handful of .incbin objects falling back to local. Signed-off-by: Javier Tia <javier@peridio.com>
A distributed compile can return a successful (exit 0) result yet omit a
declared output. glibc's ldconfig.o and sprof.o compile fine on the
build-server, but it does not return their `.o.dt` dependency file, so zipping
the compiler outputs fails fatally ("failed to open file ...o.dt: No such file")
with no recourse. One dropped output among 6427 forced glibc to be excluded from
distribution wholesale, even though 6425 of its compiles distribute cleanly.
Capture the declared output paths before the compilation is moved into the
packagers, and after a successful remote compile verify each one exists on disk.
A missing output is a distribution artifact, not a compiler error, so bail into
the existing local-recompile fallback (the same one that already salvages
non-zero remote results), which reproduces the full output set. Only compiles
whose dist output set is incomplete are affected; complete ones are returned
unchanged. glibc now distributes, with ldconfig.o and sprof.o falling back to
local.
Signed-off-by: Javier Tia <javier@peridio.com>
The lib test target links the thirtyfour dev-dependency, whose 0.36 release adds several `impl Add<_> for String`. With more than one `Add` impl for String in scope, the compiler no longer performs the &String -> &str and &Cow -> &str deref coercion that `s + &count` and `s + &p.to_string_lossy()` relied on, so `cargo test` fails to compile the lib target with E0277 in files unrelated to the change at hand. Spell the right-hand sides as &str explicitly (count.as_str(), to_string_lossy().as_ref()). The result is identical but no longer depends on coercion, so the test target builds regardless of which extra Add impls a dev-dependency happens to introduce. Signed-off-by: Javier Tia <javier@peridio.com>
When sccache packages a C/C++ toolchain for distributed compilation it asks the compiler for its sub-tools with -print-prog-name=as (and objcopy, cc1, ...). A gcc that cannot find a bundled tool returns the bare name `as`, which write_pkg resolved with which::which against the sccache daemon's PATH. For an OpenEmbedded native or cross build the daemon's PATH points at the build host, so the packaged assembler was the host's /usr/bin/as rather than the recipe's binutils -- a silent wrong-toolchain hazard that is the reason native, cross, and crosssdk classes have to be excluded from distribution. Thread the compile task's environment into CToolchainPackager and use it both when invoking the compiler for -print-*-name and when resolving a bare program name (which_in against the task's PATH instead of which::which against the daemon's). The dist-compile path supplies the real task env via into_dist_packagers; the standalone --package-toolchain command has no compilation env and keeps its prior daemon-PATH behavior. Signed-off-by: Javier Tia <javier@peridio.com>
SchedulerStatusResult only carried cluster aggregates (num_servers,
num_cpus, in_progress), so a client could see total capacity but not how
it was distributed: a two-server cluster reporting 64 cpus gave no way to
tell whether one node carried every job or the load was balanced.
Add a `servers` array of {id, num_cpus, in_progress}, populated from the
scheduler's live server map in handle_status. The bakar cluster-info
preflight already reads `servers` defensively via .get(), so this fills
in the per-node detail it renders while existing aggregate consumers stay
unaffected.
Signed-off-by: Javier Tia <javier@peridio.com>
When sccache runs as a compiler wrapper (`sccache <compiler> ...`) it
shares stderr with the wrapped compiler. A build's feature probes treat
any unexpected stderr as a compiler failure: libtool's -fPIC check
(_LT_COMPILER_OPTION) sets pic_flag="" whenever the probe compile emits
stderr, even on exit 0. With SCCACHE_LOG set, the client's own
env_logger records ("Attempting to read config file", "Server sent
CompileStarted") landed on that shared stderr, so an autotools package
configured with pic_flag="" and produced non-PIC objects that failed to
link into a shared library (relocation R_X86_64_32 ... recompile with
-fPIC). It looked concurrency-dependent because the client's startup
logging varies with server-connection timing, so the probe stderr only
sometimes diverged from libtool's expected boilerplate.
Initialize logging after parsing the command so the compile-wrapper case
can choose its target: route sccache's own records to SCCACHE_ERROR_LOG
when set, keep stderr only when it is an interactive terminal (so
`SCCACHE_LOG=debug sccache gcc ...` at a prompt still shows logs), and
discard them when stderr is captured by a build. The wrapped compiler's
forwarded stderr is untouched.
Signed-off-by: Javier Tia <javier@peridio.com>
e7df1d04 added the per-server `servers` array to SchedulerStatusResult but did not update the dist/system integration harness, whose `matches!` patterns enumerate every field explicitly. The test targets then failed to compile (E0027: pattern does not mention field `servers`), so `cargo build --all-targets` and the clippy pre-commit hook broke on a pre-existing, unrelated change. Add `servers: _` to both startup-wait patterns. Test-only; no effect on the binary. Signed-off-by: Javier Tia <javier@peridio.com>
A distributed PCH build produced a .gch that broke every consumer of the
header. erlang's BEAM JIT (asmjit) failed do_compile with "missing binary
operator before token (" at libstdc++'s `#elif ... && __has_builtin(...)`
once the asmjit.hpp.gch built under sccache-dist was force-included: the
.gch had been compiled on a remote builder from -fdirectives-only
preprocessed source, where __has_builtin evaluates differently under
-fpreprocessed than in a native build, so the cached macro state diverged
and poisoned the headers parsed after the PCH was restored. A locally
built .gch (original source, integrated cpp) compiles the same units
cleanly.
Treat a header-language input as precompiled-header generation and force
it local (dist_command = None), alongside the existing -v/--verbose and
ClangCUDA local-only cases. distcc refuses PCH for the same reason. The
PCH is one cheap compile per recipe and its many consumers still
distribute, so the throughput cost is nil while correctness is restored
for every PCH-using recipe, not just erlang.
A build server that stops is only removed once prune_servers sees its last_seen exceed the 90s heartbeat timeout, and prune runs only on the heartbeat and status paths. So --dist-status reports a stopped node as live for up to two minutes. Worse, handle_alloc_job picks servers purely by load and never checks last_seen, so it keeps dispatching compiles to a node that is gone; those fail and fall back to a local recompile, which silently skews dist timing measurements. Add a graceful deregister: the server installs a SIGTERM/SIGINT handler that flags a watcher thread to POST /api/v1/scheduler/deregister_server before exit, so a clean systemctl stop removes the node from the scheduler at once. The handler itself only does an async-signal-safe atomic store; the HTTP call runs on the watcher thread. The 90s timeout stays as the backstop for an ungraceful death (crash / power loss). Independently, skip any server stale beyond the heartbeat timeout in handle_alloc_job, so a job is never assigned to a node that has not been pruned yet - covering the crash case a deregister cannot. Signed-off-by: Javier Tia <javier@peridio.com>
avocado-distro's do_configure phase ships ~900 autoconf conftest and CMake try_compile feature-probe compiles to the build-server. Each is a tiny translation unit and many are designed to fail (detecting an absent feature), so the server returns no object and the client falls back to a local recompile. Those round-trips contend with real compiles for the build-server slots and bury the dist_errors metric, all for a result identical to a sub-second local compile. Force configure feature-probes local (dist_command = None), alongside the existing verbose, ClangCUDA, and precompiled-header cases. Detect autoconf by the conftest.<ext> input name and CMake try_compile by the CMakeScratch/CMakeTmp/TryCompile-* scratch directory rather than the generated probe source names (CheckSymbolExists.c, src.c, ...), which are not stable across CMake versions. A false positive only makes a compile run locally, still cached, so the detection errs safe. Signed-off-by: Javier Tia <javier@peridio.com>
A relocated toolchain interpreter (the OE buildtools SDK loader) already triggers bundling of its own lib dir so libc/libm resolve inside the build sandbox. But split-sysroot toolchains keep the compiler's dependency libraries -- libmpc/libmpfr/libgmp for cc1, libbfd/libsframe/libopcodes for as -- in <sysroot>/usr/lib, not beside the loader. ldd runs against the host loader, so it resolves those deps to host /usr/lib paths (or reports them "not found"), and they never land where the sandbox loader searches. cc1/as then fail with "error while loading shared libraries" and the distributed compile returns exit 1, forcing a local fallback -- every native/cross/crosssdk compile runs on the primary node while the secondaries sit idle. Bundle the shared libraries in <sysroot>/usr/lib (shallow, filtered to *.so*) alongside the interpreter lib dir so every NEEDED library resolves in-sandbox. Verified end to end: the OE buildtools gcc now distributes to the second node (dist_compiles, zero errors) where it previously fell back to local. Signed-off-by: Javier Tia <floss@jetm.me>
OpenSSL probes assembler support by running the compiler on /dev/null: `gcc -Wa,--help -c -o null.o -x assembler /dev/null`, then greps the assembler's help output for --noexecstack. Under an sccache wrapper this parsed as a cacheable assembler compile, but -Wa,--help makes the assembler print its help and exit without producing an object. sccache then aborted trying to zip the missing output and swallowed the probe's stdout, so OpenSSL saw no --noexecstack and assembled every .s with an executable stack. The resulting libcrypto.so.3 was rejected by a hardened host kernel, breaking later recipes that load it (python3-native). Refuse to cache a compile whose input is /dev/null. Such an input is never a real translation unit, so a transparent local exec costs nothing and lets the probe's output reach the caller, matching ccache's long-standing refusal to cache /dev/null. Signed-off-by: Javier Tia <floss@jetm.me>
The 2-node cluster leaves both build servers idle for long stretches of a Yocto cold build, but the existing logs cannot distinguish the three candidate causes: job-supply starvation (bitbake rarely runs enough concurrent do_compile phases), the client-side preprocessing tax that skews load onto the colocated node, and per-job round-trip latency on short compiles. Choosing which rework to pursue needs per-job and per-server timing data, not guesses. Add info-level instrumentation on both halves of the dist path, routed to SCCACHE_ERROR_LOG so it stays off the compiler's stderr. The scheduler logs every candidate server's load alongside the allocation decision (and on the no-capacity path), plus a one-line in-progress snapshot per status poll, exposing any first-hashed-server tie bias and the gate-full frequency. The client times put_toolchain, alloc, submit, and run+fetch separately, tracks an in-flight compile counter via an RAII guard so the count stays correct across every future exit, and emits a per-job summary naming the server that ran it. One instrumented build then tells supply starvation (low in-flight while nodes idle) from round-trip latency (run+fetch dominating) from a preprocessing bottleneck. Signed-off-by: Javier Tia <floss@jetm.me>
…ot stranded A server that fails a job assignment is dropped into the best_err bucket and only chosen when no error-free server has capacity. That demotion is absolute: `best.or(best_err)` takes any healthy server over an errored one regardless of load. On a two-node cluster the errored bucket is not symmetric - the build server co-located with the scheduler assigns over loopback and effectively never errors, while the network-remote node takes the occasional transient assignment error. So the remote node gets demoted, sits idle, and the local node absorbs the work. Instrumentation over one image build measured 1710 allocations that skipped an idle server for a busier healthy one, every single one on the remote node. Treat a recent error as a bounded load penalty instead of an absolute veto: a recently-errored server still wins when it is more than RECENT_ERROR_LOAD_PENALTY less loaded than the healthy alternative. A genuinely overloaded or persistently failing server is still avoided (its real load, or a fresh error each attempt, keeps it behind), but a transient blip no longer strands a node's idle capacity for the full remember-error window - and a mistaken pick costs only one wasted round-trip before the local fallback. Signed-off-by: Javier Tia <floss@jetm.me>
Rust compiles never distributed on the cluster - measured 0 of 659 while C/C++ distributed 51k. Every rust crate ran on the build server, came back successfully, and was then thrown away: RustOutputsRewriter expects the dep-info (.d) file among the returned outputs and, when none matches, bail!s "No outputs matched dep info file". That aborts the whole distributed compile, so the or_else path recompiles locally - stranding every rust compile back on the client while its remote result is discarded. The dep file only feeds cargo's rebuild tracking; a bitbake do_compile runs cargo from scratch and never consumes it across tasks. So a good remote object/rlib must not be discarded over it. Replace the bail with a warning that keeps the returned outputs and skips only the dep-info path rewrite. This is the same shape as the fork's existing dropped-output tolerance for glibc's .o.dt file, applied to the rust rewriter. Also drop the error!-level log of the full dep-file body on every rewrite to trace! - it fired on the success path and would flood the log now that the success path is actually reached. Signed-off-by: Javier Tia <floss@jetm.me>
Softening the rust dep-info rewriter (previous commit) was not enough to make rust distribute: a rust compile still fell back to local, now caught by the second guard. After a distributed compile the client verifies every declared output exists and, if one is missing, discards the result and recompiles locally. That guard ignored the `optional` flag, so a missing optional output forced a local recompile even though the output was declared droppable - stranding every rust compile on the client because the rust dep-info (.d) file, which the build server does not reliably return, was declared non-optional. Make the guard honor `optional`: only non-optional outputs must exist. The C dep file and glibc's dropped .o.dt stay non-optional, so their local-fallback (the reason that guard exists) is unchanged; a `-gsplit-dwarf` .dwo, already declared optional, no longer wrongly forces fallback. Then mark the rust .d optional, since it only feeds cargo rebuild tracking a from-scratch bitbake do_compile never consumes. With both, a distributed rust compile whose .d is absent keeps its object and rlib instead of recompiling locally. Signed-off-by: Javier Tia <floss@jetm.me>
Cold toolchain packaging walked the relocated interpreter's sysroot and called fs::metadata on every symlink to decide whether it pointed at a regular file. An OE recipe-sysroot-native tree carries sysroot-relative symlinks such as usr/bin/sg whose target resolves outside the packaged subtree, so metadata failed with ENOENT and the `?` aborted the entire add_dir_contents walk. The whole toolchain package failed to build, the client could not upload it, and every affected compile fell back to a local build instead of distributing. Rust recipes hit this on every cold cache because their toolchain is packaged fresh. Skip a symlink whose metadata cannot be resolved instead of failing, matching the swallow-on-error handling add_shared_libraries already uses for the identical check. A dangling link can never point at a file the package needs, and the walk is documented as best-effort, so dropping it is safe and keeps the package building. Signed-off-by: Javier Tia <floss@jetm.me>
A distributed rust compile against an OpenEmbedded custom target failed on every genuinely-remote build server with `error loading target specification: could not find specification for target "aarch64-avocado-linux-gnu"`. OE passes the target by name and points rustc at the spec JSON via RUST_TARGET_PATH; sccache-dist ships a target spec only when --target is a path, and never repoints RUST_TARGET_PATH, so the name-form target was invisible to the server and the remote rustc aborted. The client then fell back to a local recompile, keeping rust off the cluster (a colocated server on the client host sometimes succeeded only because the spec existed at its real host path). Resolve a bare --target NAME against the compile's own RUST_TARGET_PATH to the <name>.json it refers to, add that JSON to the shipped inputs, and rewrite RUST_TARGET_PATH in the remote environment to the shipped location so the server resolves the name exactly as the client did. The --target argument keeps its name form on purpose: rustc records the target's identity from that string, and the prebuilt std in the recipe sysroot was built name-form, so a path-form rewrite would make std fail to match with E0461. Built-in targets have no matching file and are left untouched. Recording the resolved spec also folds its content into the hash, so these compiles miss once as a name-form target previously did not invalidate the cache when the spec changed.
When a distributed compile exits non-zero the client logs only the exit code and falls back to a local recompile, discarding the remote compiler's stdout and stderr. That output is the only place the actual failure is visible - a missing target std, an input the packager did not ship, a sandbox path that does not resolve - so diagnosing a dist regression meant patching in a temporary dump and rebuilding the binary each time. Emit the remote stdout/stderr at debug on the non-zero path. SCCACHE_LOG=debug now surfaces the real remote error with no rebuild, while a normal SCCACHE_LOG=info run stays quiet (the one-line fallback reason is still logged at warn by the caller). Signed-off-by: Javier Tia <floss@jetm.me>
A distributed cross-compile against an OpenEmbedded custom target failed on the remote with `error[E0463]: can't find crate for core`/`std`. The target's prebuilt std/core rlibs live in <recipe-sysroot>/usr/lib/rustlib/<triple>/lib and are referenced via `-L`; that path is transformed and emitted to the server, but the rlibs themselves are dropped. write_inputs filters each crate-link dir to the crate names cargo discovered as rlib deps, and the implicit sysroot crates (core, std, alloc, compiler_builtins, proc_macro) are injected by rustc and never appear as externs, so the filter removes them and the remote rustc has no std. Exempt a target sysroot lib dir from that filter and ship all its rlibs. The dir is recognized by its `rustlib/<triple>/lib` tail, so an ordinary `-L dependency=target/<triple>/deps` search path is unaffected and still filtered to the discovered dep set. The rlibs enter the per-compile input set already scoped to this exact target, so no toolchain cache key changes are needed - unlike bundling them into the shared toolchain, whose key (weak_toolchain_key) is not target-specific and would alias std across targets built by the same host rustc. Signed-off-by: Javier Tia <floss@jetm.me>
When packaging dist inputs, sccache trims a dependency rlib to just its `rust.metadata.bin` ar member, since a crate only inspects a dependency's metadata. The trim loop scanned for that member and, if it never found one, appended nothing at all - silently omitting the rlib from the inputs tar. An OpenEmbedded target sysroot builds its std/core with split metadata: the metadata lives in a sibling `.rmeta` and the `.rlib` carries only object members, no `rust.metadata.bin`. Every such rlib was therefore dropped, so a distributed cross-compile reached the build server with the target rustlib directory present but empty. no_std crates compiled anyway; every crate that needs `std` failed remotely with `error[E0463]: can't find crate for std` and fell back to a local recompile, keeping most Rust off the cluster. Extract the metadata scan into trimmed_rlib_metadata, returning None when the rlib has no embedded metadata member, and ship the whole rlib in that case. Normal rlibs still trim to metadata; split-metadata rlibs now travel intact. Signed-off-by: Javier Tia <floss@jetm.me>
The dist input packager scans each crate link-path directory and parses a library's crate name by splitting the filename on its last `-` to drop the `-<metadata-hash>.rlib` suffix cargo emits. A file with no `-` fell into the skip branch and was dropped from the inputs entirely. An OpenEmbedded target sysroot names the std crate `libstd.rlib` with no hash suffix (every other crate there is `lib<name>-<hash>.rlib`). It was therefore never shipped, so a remote cross-compile had core and alloc but no std - and every crate that pulls in the std prelude failed with a cascade of "cannot find type `Option`/`Result`/`Vec`" and fell back to a local recompile. Fall back to the filename with its extension stripped when there is no `-` to split on, so a hashless `libstd.rlib` resolves to crate name `std` and ships. With this the recipe's Rust crates distribute across both nodes (21/21, zero failed distributed compiles) where std-dependent crates previously all fell back local. Signed-off-by: Javier Tia <floss@jetm.me>
The R0 dist instrumentation times put_toolchain, alloc, submit, and run+fetch per job, but not the client-side preprocessing that produces the shipped input. That leaves the W2 gap: the preprocessing tax is the leading theory for why the colocated node shows far more cc1 activity than the remote one, yet it is only inferred from cc1 process counts, never measured against the dist round-trip it precedes. Time the single cc1 -E invocation in generate_hash_key (the preprocess await in the C hasher) and thread the duration to the per-job line via a new Compilation::preprocess_duration accessor. The value is captured where the work happens rather than re-derived on the dist path, because preprocessing runs during hash-key generation, before the dist round-trip begins, so it cannot be timed inside do_dist_compile itself. A trait default returns None so only the C compiler, which does local preprocessing, carries a value; Rust and already-preprocessed inputs log 0ms. The number joins the existing dist-job done line as preprocess Nms and is not folded into the round-trip total. Signed-off-by: Javier Tia <floss@jetm.me>
The load model weighs every server as job_count/core_count with the raw hardware core count, which is blind to the colocated node also spending cores preprocessing (cc1 -E) for the whole cluster, packaging toolchains, and running bitbake and the scheduler. It therefore balances jobs ~50/50 onto a node that is really the busier of the two, and its both-idle tie is decided by hash-map iteration order - the same node wins every time, so a compile burst always starts on the pinned local node - while the load==0 early break truncates the logged candidate list, hiding the imbalance from the routing metrics. Add an optional advertised num_cpus to the server config: a colocated node can advertise fewer cores than it has so load_weight de-weights it and its admission cutoff drops, reserving cores for the preprocessing tax the scheduler cannot see. Remove the early break so every server is scanned (complete candidate log, no hidden misroutes) and break ties deterministically toward the node with more free admission slots - with the advertised-cores override the remote node wins, so bursts start off the busy local node. Signed-off-by: Javier Tia <floss@jetm.me>
The preprocessing wall - the client jobserver's cores saturated by local cc1 -E for the whole cluster while remote nodes idle - was only ever an inference from per-job timers. Nothing measured how many preprocesses run at once, so there was no way to confirm the jobserver token pool is the feed bottleneck rather than bitbake under-producing jobs. Wrap the preprocess call in an RAII counter mirroring DistInflightGuard and log "preprocess done in <n>ms (concurrent <k>)" so the concurrency is a time series. The guard drops the count as soon as preprocessing returns, before the dist round-trip, so it reflects preprocess pressure alone; gated on dist-client so a non-dist build is unchanged. Signed-off-by: Javier Tia <floss@jetm.me>
The advertised-cores override added earlier had to be a hardcoded absolute in each colocated node's server.conf (num_cpus = 20 for a 32-core PC1), which is wrong on any other cluster topology. A different node count or a differently-provisioned host would need the value hand-recomputed. Compute it instead. A build server on the scheduler's own host is the colocated orchestrator node - it also runs the client's preprocessing and bitbake - so it reserves a fraction of its cores; a server on a different host is a pure remote and advertises all of them. The fraction defaults to 0.35, overridable via colocated_reserve_fraction, and an explicit num_cpus still wins as an escape hatch. Everything derives from the node's own hardware, so the same binary and config are portable across clusters. The resolution lives in a pure advertised_cores() with unit tests; the harness gains the advertised-cores argument the server constructor now takes. Signed-off-by: Javier Tia <floss@jetm.me>
The dist-client and dist-server code is feature-gated, and upstream CI runs clippy on default features, so this code was never linted - it carried a batch of latent warnings that only surface with --features all,dist-server -D warnings. Add the trailing semicolons flagged by semicolon_if_nothing_returned in the heartbeat loop and the sccache-dist bin, inline a let-and-return in the rlib name parser, and allow large_enum_variant on the sccache-dist Command enum - it is built once at startup and matched once, so the size difference is irrelevant and boxing would only add indirection. No behavior change; this makes the tree clippy-clean so the new pre-commit gate passes. Signed-off-by: Javier Tia <floss@jetm.me>
The pre-commit config only ran nightly fmt and clippy with no failure gate, so a break reached CI before anyone noticed - a changed function signature or struct literal in tests/ slipped through because plain cargo check and makepkg never compile the integration tests. Add the CI gate locally: clippy with the same -D warnings and allows CI sets, but --features all,dist-server so the feature-gated dist code is actually linted (CI's clippy runs default features and never covers it), and cargo check --all-targets so the test harness is type-checked at commit time. Move cargo test and cargo audit to pre-push so per-commit turnaround stays fast. fmt stays on nightly to match the tree's existing formatting. Signed-off-by: Javier Tia <floss@jetm.me>
A distributed job that reached the Started state but never reported Complete leaked its jobs_assigned slot on the owning server. The Ready->Started transition removes the job from jobs_unclaimed, so the unclaimed reaper (which handles only Ready/Pending) can never see it again, and prune_servers reclaims jobs only from a server gone silent past the heartbeat timeout. On a live, still-heartbeating server these stuck-Started jobs accumulate until jobs_assigned reaches the admission ceiling, at which point load_weight returns MAX+1 and the scheduler stops routing to the node. Measured on the two-node cluster: the remote server pinned at 37/37 in-progress while idle (load 0.15, 0 cc1), its completion updates lost while clients backed up on the colocated node. Stamp each job with the time it entered its current state and, on every heartbeat, de-allocate any Started job the server has held past STARTED_COMPLETE_TIMEOUT (180s) - far longer than a real remote single-TU compile (~60s for a heavy LLVM object) yet short enough to reclaim a leaked slot briskly. Reaping only the scheduler's accounting never aborts the client's in-flight compile, which runs over a separate client-server channel, so an over-eager reap at worst mildly oversubscribes, which admission_ceiling already tolerates. This mirrors the prune_servers (dead servers) and unclaimed (stuck Ready/Pending) backstops already in the scheduler. Signed-off-by: Javier Tia <floss@jetm.me>
Diagnosing scheduler slot leaks meant a rebuild-with-debug-logging cycle each time, and the per-compile debug logging that surfaced the remote errors itself dragged the client daemon under load. There was no cheap, always-on signal to tell a feed/unclaimed leak (jobs allocated to a node but never claimed) apart from a lost-completion leak (jobs that reached Started but never reported Complete), nor to confirm the accounting is balanced at all. Add five cumulative atomic counters - started, completed, reaped_unclaimed, reaped_started, pruned - incremented at their lifecycle transitions, and emit one dist-accounting summary line per heartbeat (roughly every 30s per server, not per compile, so it adds no hot-path cost). A rising allocated-minus-started-minus-reaped_unclaimed isolates a feed/unclaimed leak; a rising started-minus-completed-minus-reaped_started isolates a lost-completion leak. The counters use Relaxed ordering since they are diagnostic, not synchronization. Signed-off-by: Javier Tia <floss@jetm.me>
The 180s Started-job reaper can discard a successful compile: it reclaims
the slot at the timeout, then the build server's run_job propagates a
Complete update for that job, handle_update_job_state hits bail!("Unknown
job"), run_job returns 500, and the client throws away the finished
object files and recompiles locally. Any translation unit that legitimately
compiles for longer than the timeout on a loaded remote node therefore
fails at the finish line and is strictly worse off than not distributing.
Treat a Started or Complete update for a job the scheduler no longer
tracks as a logged no-op success rather than an error, and stop failing
the build server's run_job when its terminal Complete update is rejected
(log and return the results anyway). This makes the reaper safe for a
compile of any duration: a reaped-then-completed job still delivers its
objects to the client. A non-terminal update for an unknown job still
warns. Covered by test_late_update_for_reaped_job_is_ok.
Signed-off-by: Javier Tia <floss@jetm.me>
The unclaimed-reservation reaper timed a job out from its allocation instant, so the toolchain-upload window (the Pending phase) ate into the claim budget: a job whose toolchain was still uploading could be reaped as a no-show before it ever became claimable, even though nothing was wrong. Reset the jobs_unclaimed timestamp when the job transitions Pending to Ready, so the timeout measures only the time a claimable job sits unclaimed. A job still uploading its toolchain is no longer counted against the reservation TTL. Covered by test_pending_to_ready_resets_unclaimed_clock. Signed-off-by: Javier Tia <floss@jetm.me>
A scheduler restart reminted job ids from zero while build servers kept running jobs assigned before the restart. The reused id collided with the survivor's stale job_toolchains entry, and handle_assign_job asserted the insert returned no prior value, so the rouille worker panicked and the server 500'd. The per-job toolchain map was also only ever pruned in handle_run_job, so an assign whose run_job never arrived leaked its entry forever. Seed the scheduler's job counter from wall-clock millis so a restart does not restart ids at zero; change the assign-time assert to a warn-and- overwrite so a residual collision degrades gracefully; and timestamp each job_toolchains entry, sweeping entries older than ABANDONED_TOOLCHAIN_TIMEOUT on each assign so abandoned entries cannot accumulate. Covered by test_job_count_seeded_nonzero. Signed-off-by: Javier Tia <floss@jetm.me>
The scheduler reaped a Started job purely on a fixed 180s timeout, so a genuinely-slow remote compile that ran longer than the timeout had its slot reclaimed while it was still running - the very case the reaper was meant to leave alone. There was no signal distinguishing a live long compile from a job whose completion was lost. Have each build server report the ids it is actively running in every heartbeat (a running-jobs set maintained by an RAII guard around handle_run_job, snapshotted into the heartbeat request each beat), and reap a Started job only once it is absent from two consecutive heartbeats of the owning server. A single dropped or racing heartbeat cannot reap a live job, and a compile the server keeps naming survives at any duration. The fixed timeout stays only as a backstop for a server that stops heartbeating its running set, and it too now skips any job named in the latest beat. This adds an active_jobs field to the heartbeat request, changing the bincode wire format: scheduler and all build servers must run the matching binary. Covered by test_liveness_lease_reap. Signed-off-by: Javier Tia <floss@jetm.me>
jetm
force-pushed
the
oe-bitbake-dist-support
branch
from
d4f3c6f to
771552d
Compare
A cold-read review of the Group 1 changes surfaced two must-fix bugs. The liveness lease could reap a freshly-Started job on the first heartbeat after it started: a job that transitions to Started between two beats is vacuously absent from the prior beat, so a heartbeat snapshot that races the Started-update-then-registry-insert window sees it missing from both beats and reaps it milliseconds into its compile. Add a LIVENESS_GRACE floor (45s, above the 30s heartbeat interval) on time-in-Started so only a job that has lived across a full beat period is reapable, and register the running job before sending the Started update so the window shrinks to pure network reordering. Seeding job_count from wall-clock millis (for restart-safe ids) also clobbered the dist-accounting allocated= counter, which reads the same field, breaking the allocated-started-reaped-live identity the leak detector depends on. Split the roles: job_count is a 0-based allocation counter again (logged as allocated=), and a new job_id_base carries the millis seed so minted ids stay unique across a restart. Also restrict the unknown-job update no-op to Started/Complete (a Pending/Ready update for an untracked job still errors), warn when the backstop protects a job past the timeout because the server still names it (wedged-build visibility), and document the toolchain-GC timeout coupling. Covered by test_liveness_lease_does_not_reap_within_grace and test_job_id_base_seeded_and_counter_zero. Signed-off-by: Javier Tia <floss@jetm.me>
The single client daemon is the throughput chokepoint of a full image build. It opened a fresh TCP+TLS connection on every coordination call (pool_max_idle_per_host(0) plus a Connection: close header), decoded multi-megabyte object blobs inline on the same async workers that poll every other in-flight request, and rebuilt the reqwest client under a Mutex held across the entire rebuild whenever a server certificate refreshed. Under make -jN the daemon backed up while build cores sat idle, which is why sccache-dist lost to plain ccache on the full image even though it wins on expensive single-object recipes like llvm. Re-enable keep-alive on all four dist client builders (a small idle pool plus a 90s idle timeout) and drop the Connection: close workaround. tiny_http 0.12 drains the request body on drop of the request reader, so a partially-read body on a reused connection cannot desync the next request framed on it. Deserialize responses above a config-selectable threshold via spawn_blocking so a large blob no longer stalls the reactor from polling other compiles, and hot-swap the TLS client through ArcSwap so a certificate refresh publishes the new client without holding a lock across the rebuild. The run_job compression level is now config-selectable with the default unchanged, so a no-compression setting can be measured on a fast LAN where CPU is scarcer than bandwidth. Keep-alive is the one intentional default behavior change; it carries an A5 revert marker in bincode_req_fut pointing at the tiny_http mozilla#151 risk so a soak-triggered rollback has an exact anchor. Signed-off-by: Javier Tia <floss@jetm.me>
The build server serialized all concurrent compiles behind two global locks. prepare_overlay_dirs held the toolchain_dir_map mutex across the whole gzip+untar and across eviction's remove_dir_all, so unpacking one toolchain blocked every other compile that needed any toolchain, cached or not. handle_submit_toolchain held the cache mutex across the entire upload io::copy, so one toolchain upload blocked every cache read for its whole duration. Give each toolchain its own preparation lock. The global map lock now only looks up or inserts the per-toolchain Arc<Mutex<()>> and picks eviction victims; the untar runs under the per-toolchain lock, and each victim's remove_dir_all runs after the map lock is dropped. Two requests for the same uncached toolchain still serialize on the same entry lock (re-checked after acquiring it) so no directory is ever half-written, while requests for different toolchains prepare in parallel. Stream the submit-toolchain upload into a NamedTempFile with no lock held, then re-acquire the cache lock only to graft the finished file in via a new hash-verified TcCache::insert_at. The assign-time duplicate-insert assert becomes a warn, matching the restart-safe accounting convention. Signed-off-by: Javier Tia <floss@jetm.me>
A review against the tiny_http-0.12.0 source found the G2 keep-alive change had one hole. do_submit_toolchain posts a chunked body (wrap_stream, no Content-Length), and tiny_http's chunked reader has no drop-drain, unlike the Content-Length EqualReader the change relied on. The server's submit handler has early returns that skip reading the body (cache fast-path, JobNotFound), so once Connection: close was dropped a partially-read chunked body could leave chunk bytes on a pooled connection and desync the next request framed on it. Two jobs racing a new toolchain hit this on every cold build. Re-add Connection: close on the do_submit_toolchain request only. Submits are rare and huge, so keep-alive there is worth ~nothing, and it makes the unread early-returns legal again without a server-side drain. Every other path sends a sized bincode body that tiny_http's EqualReader sweeps, so keep-alive stays on where it pays. Also address the review's promptly-fix items: delete eviction victims under their own prepare lock (try_lock-and-skip, which is deadlock-safe against a concurrent eviction where a held victim lock already means an active preparer that must not be deleted) and tolerate a stale leftover dir, so a victim being re-prepared cannot be deleted mid-untar and escalate into a lost archive; untar from an owned get_file handle so two different toolchains no longer serialize on the cache mutex; remove a hash-mismatched entry in insert_at instead of leaving it cached; and duplicate the A5 keep-alive revert marker into the blocking bincode_req path so a soak rollback re-closes both client paths. Signed-off-by: Javier Tia <floss@jetm.me>
The input packager built each tar entry with a ustar header and set the entry path with Header::set_path, which overruns the ustar name (100 B) and prefix (155 B) fields for deep build paths. gcc-runtime's libstdc++-v3 objects exceed both, so set_path returned "provided value is too long", the whole input tar failed, and every such compile fell back to local instead of distributing (observed as 50 gcc-runtime fallbacks on one cold core-image-minimal build). Build the entry headers with Header::new_gnu() and write them through a new pkg::append_tar_entry helper that routes to Builder::append_data, which emits a GNU @LongLink long-name entry when the path overflows the ustar fields and recomputes the checksum. Normal-length paths stay plain entries. The server untar side already reads @LongLink transparently via tar::Archive::unpack, so no server change is needed. Covers all three packagers (the two C input paths and the rust path). Signed-off-by: Javier Tia <floss@jetm.me>
Collaborator
|
please split this PR into smaller PRs. |
Author
|
Thanks @sylvestre, @AJIOB, and @ahartmetz for the reviews here. I'm closing this PR. Since I opened it, the scope has grown well beyond the original change: continued testing on real OpenEmbedded/Yocto builds surfaced a range of further distributed-compile problems and their fixes (toolchain packaging, Rust std and I'm classifying the work by concern and splitting it into a series of smaller, independently reviewable PRs, landing the self-contained fixes first. I expect to start opening them this week, once a final round of validation on large OE/Yocto builds finishes (those runs take a while to complete). I'll link the new PRs back here as they go up. Sorry for the noise here, on this PR and the earlier ones I closed. I got ahead of myself after some good early results on a handful of recipes, and opened them before I had tested broadly enough across the rest. |
Collaborator
|
no worries, i made that mistake so many times in my career ;) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This series makes
sccache --distusable end-to-end for OpenEmbedded/Yocto(BitBake) builds, where it previously panicked, failed, or silently fell back
to local on several distinct issues. Validated by building a qemuarm64
core-image-minimal entirely through a single-node sccache-dist cluster:
userspace do_compile distributes (e.g. busybox 509/509) and the kernel
distributes 1124/1128 compiles.
It combines and supersedes #2746 and #2747 into one OE/BitBake-support series.
Don't panic when a finished compile has neither code nor signal (Don't panic in get_signal when a finished compile has neither code nor signal #2746).
sccache-dist synthesizes an ExitStatus with neither exit code nor signal for
some abnormal remote compiles; get_signal panicked on .expect("must have
signal"). Return Option and handle the both-absent case.
Bundle the relocated interpreter's libdir into the dist toolchain package
(Bundle relocated interpreter's libdir into the dist toolchain package #2747). OE cross-toolchains use an absolute-path uninative loader whose
libc/libm live in the uninative sysroot, not the host paths ldd reports;
bundle the interpreter's libdir so the toolchain resolves in the sandbox.
Fix distributed compile of relative input paths. The dist command used the
raw input path while the inputs packager shipped the preprocessed content at
the absolute, simplified path, so out-of-tree builds (../sources/foo.c)
failed "No such file or directory" on the server. Use the same path in both.
Log distributed-compile decisions at info level. The distribute-vs-local
decision was debug-only and only failures warned; log it at info so
SCCACHE_LOG=info shows the distribute/fallback breakdown. Emitted only when
SCCACHE_LOG is set.
Fall back to local on distributed-compile failure. A remote failure is often
a distribution artifact (e.g. a kernel object that .incbin's a binary the
packager cannot ship) rather than a real compiler error; recompile locally
so a dist failure never breaks a build that compiles fine locally.
Draft: validated single-node so far; marking ready after a two-machine cluster
run confirms it.