fix(bootstrap): set dylib path for rustc tests

[HNO3Miracle]

Fixes #158526

compiler/rustc builds temporary test binaries when running ./x test --stage 2 compiler/rustc. With Cargo's new build-dir layout, dylibs such as rustc_driver can be emitted under Cargo's build/*/*/out directories rather than a directory that is already present in the test command's dynamic library lookup path.

Build the rustc test binaries once with cargo test --no-run, then scan the Cargo build directory for out directories containing dylibs and add them to the dynamic library lookup path used by the actual test run. This avoids target-specific rpath handling and uses the platform-specific lookup variable selected by bootstrap (LD_LIBRARY_PATH, DYLD_LIBRARY_PATH, PATH, etc.).

Tests:

• ./x fmt --all
• git diff --check
• ./x check src/bootstrap
• ./x test src/bootstrap

r? @bjorn3

[bjorn3]

-Wl,-rpath only works for ELF targets. The Apple linker has a different cli flag and Windows doesn't support rpath at all. Instead tests should run with the right dynamic linker library path set.

[HNO3Miracle]

I'm not entirely sure if this is the best approach, and I'd appreciate any feedback or suggestions.

[bjorn3]

I'm curious why cargo doesn't already add those dylibs. It should in principle do so for all dylibs that are dependencies of an executable. Or maybe that logic is broken in cargo with the new build dir layout?

View changes since the review

[HNO3Miracle]

I looked into Cargo a bit more, and I think this is likely related to the new build-dir layout indeed.

For compilation, Cargo does recursively add dependency dirs through lib_search_paths / add_dep_arg, so rustc can link against rustc_driver.

For running test binaries, however, Compilation::target_process only adds deps_output[&kind] to the dynamic library search path. With the new layout, deps_output is still a single HashMap<CompileKind, PathBuf>, and BuildRunner::prepare seems to repeatedly overwrite it while iterating units. So the runtime path can miss the per unit directory that contains librustc_driver.

This PR works around that in bootstrap by doing cargo test --no-run first and then adding the generated dylib-containing out dirs to the test command's dynamic library path. But I agree the more fundamental fix may belong in Cargo's new-layout runtime search path handling.