Enable autodiff in ci for all major os

[ZuseZ4]

View all comments

Follow-up attempt to #140064 after moving autodiff to dlopen.
It covers Linux (x86_64+aarch64), MacOS (aarch64), Windows (mingw-llvm aarch64+x86_64)

The follow-up distribution via rustup probably still needs a small fix, see #151063 (comment)

r? @Kobzol

closes #151243
closes #151063

[rustbot]

Kobzol is not on the review rotation at the moment.
They may take a while to respond.

[ZuseZ4]

@bors try jobs=dist-aarch64-apple

[rust-bors]

☀️ Try build successful (CI)
Build commit: f1988f0 (f1988f049b5de890c7b71b3da66d38620ed38fbd, parent: dfbfbf785f46d10103e80735a9ccb4765bd6dfe2)

[ZuseZ4]

@bors try jobs=dist-x86_64-linux

[rust-bors]

☀️ Try build successful (CI)
Build commit: 20e4cfa (20e4cfae177c9daec1a6b5ae5d1332821335dcbc, parent: 8387095803f21a256a9a772ac1f9b41ed4d5aa0a)

[ZuseZ4]

@bors try jobs=dist-aarch64-linux

[rust-bors]

☀️ Try build successful (CI)
Build commit: 22ac1fb (22ac1fb63228652dc40e63504ab188e2bae982bb, parent: 8387095803f21a256a9a772ac1f9b41ed4d5aa0a)

[ZuseZ4]

@bors try jobs=dist-aarch64-llvm-mingw

[rust-bors]

💔 Test for ea4964d failed: CI. Failed job:

• try - dist-aarch64-llvm-mingw (web logs, enhanced plaintext logs)

[rust-bors]

💔 Test for 033f25a failed: CI. Failed job:

• try - dist-aarch64-apple (web logs, enhanced plaintext logs)

[ZuseZ4]

Yeah, dist-x86_64-linux requires using lld, whereas dist-aarch64-apple will not recognize the lld setting because it calls cc, not clang. We should gate the -fuse-ld=lld flag, but not sure on what. Checking in the Enzyme dist component if our compiler is clang and if we have lld in the sysroot or somewhere feels hacky.

[Kobzol]

will not recognize the lld setting because it calls cc

Shouldn't GCC also support -fuse-ld=lld?

So is it even possible to link Enzyme without LLD? What is the problematic situation, Clang + BFD? Or Clang + old BFD?

[ZuseZ4]

You can generally build enzyme without lld. I can't reproduce these issues locally, and I can't download the temp CI artifacts from those failed runs, so I don't really know. Maybe you get more from these logs than I? https://github.com/rust-lang/rust/actions/runs/22168431338/job/64100800611

What we see is that we have clang on that linux builder, together with

-- The C compiler identification is Clang 22.1.0
-- The CXX compiler identification is Clang 22.1.0
-- Linker detection: GNU ld

The llm of my choice guessed that "llvm.thin-lto=true produced LLVM static archives (e.g., libLLVMSupport.a) whose members are LLVM bitcode. When you link an executable that pulls in those .a members, the linker must understand and compile LLVM bitcode at link time."

I didn't test thin-lto myself but it sounds somewhat sensible? And overwriting gnu ld with lld via the flag also worked.

Wrt. to whether gcc wouldn't handle the flag, are you sure that cc here resolves to gcc? From the apple build failure it looks more like an issue with Apple's clang fork, which based on quick google search seems to bring their own ld64 linker:

-- The C compiler identification is AppleClang 17.0.0.17000603
-- The CXX compiler identification is AppleClang 17.0.0.17000603
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc - broken
CMake Error at /opt/homebrew/share/cmake/Modules/CMakeTestCCompiler.cmake:67 (message):
  The C compiler

    "/usr/bin/cc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: '/Users/runner/work/rust/rust/build/aarch64-apple-darwin/enzyme/build/CMakeFiles/CMakeScratch/TryCompile-vEZ6k3'
    
    Run Build Command(s): /opt/homebrew/bin/ninja -v cmTC_9e641
    [1/2] /usr/bin/cc   -Wno-deprecated -ffunction-sections -fdata-sections -fPIC --target=arm64-apple-macosx -mmacosx-version-min=11  -arch arm64 -mmacosx-version-min=11 -MD -MT CMakeFiles/cmTC_9e641.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_9e641.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_9e641.dir/testCCompiler.c.o -c /Users/runner/work/rust/rust/build/aarch64-apple-darwin/enzyme/build/CMakeFiles/CMakeScratch/TryCompile-vEZ6k3/testCCompiler.c
    [2/2] : && /usr/bin/cc -Wno-deprecated -ffunction-sections -fdata-sections -fPIC --target=arm64-apple-macosx -mmacosx-version-min=11  -arch arm64 -mmacosx-version-min=11 -Wl,-search_paths_first -Wl,-headerpad_max_install_names  -fuse-ld=lld -static-libstdc++ CMakeFiles/cmTC_9e641.dir/testCCompiler.c.o -o cmTC_9e641   && :
    FAILED: [code=1] cmTC_9e641 
    : && /usr/bin/cc -Wno-deprecated -ffunction-sections -fdata-sections -fPIC --target=arm64-apple-macosx -mmacosx-version-min=11  -arch arm64 -mmacosx-version-min=11 -Wl,-search_paths_first -Wl,-headerpad_max_install_names  -fuse-ld=lld -static-libstdc++ CMakeFiles/cmTC_9e641.dir/testCCompiler.c.o -o cmTC_9e641   && :
    clang: error: invalid linker name in argument '-fuse-ld=lld'
    ninja: build stopped: subcommand failed.

[Kobzol]

Right, LLD definitely shouldn't be used on Apple, that's for sure. The LTO explanation makes sense, that is indeed the problem, if I recall correctly. Though we must have the same issue while building LLVM, right? Did you check the LLVM bootstrap step if it does something to work around that?

If yes, then ideally just copy that. If you can't find it, then I'd gate LLD on being on Linux +Thin LTO being enabled in the build. We could also detect Clang, bootstrap already does that else where, but I'm pretty sure that LLVM/LLD LTO won't work with GCC anyway, so probably it's not even needed.

[ZuseZ4]

fn configure_llvm(builder: &Builder<'_>, target: TargetSelection, cfg: &mut cmake::Config) {
    // ThinLTO is only available when building with LLVM, enabling LLD is required.
    // Apple's linker ld64 supports ThinLTO out of the box though, so don't use LLD on Darwin.
        if builder.config.llvm_thin_lto {
        cfg.define("LLVM_ENABLE_LTO", "Thin");
        if !target.contains("apple") {
            cfg.define("LLVM_ENABLE_LLD", "ON");
        }
    }

I guess the main problem is that now that we build Enzyme as its own dist component, the sensible defaults from the main bootstrap build aren't applied anymore, so I'm rediscovering them. I'll use that for Enzyme.

Ah, we raced.

[Kobzol]

I guess we could extract some shared logic that makes sense for both LLVM and LLD into a separate function and use it both in configure_llvm and in the LLD build step.

[ZuseZ4]

I am not sure that will be picked up by Enzyme, but it's late and worth a try:

@bors try jobs=dist-aarch64-linux,dist-aarch64-apple,dist-x86_64-linux,dist-aarch64-llvm-mingw

[rust-bors]

💔 Test for 5d94862 failed: CI. Failed job:

• try - dist-aarch64-linux (web logs, enhanced plaintext logs)

[ZuseZ4]

@bors try jobs=dist-aarch64-linux,dist-aarch64-apple,dist-x86_64-linux,dist-aarch64-llvm-mingw

[rust-bors]

☀️ Try build successful (CI)
Build commit: a84e1d3 (a84e1d351e69b6b0e668af5caeefa33ea9c3f026, parent: 59fd4ef94daa991e6797b5aa6127e824f3067def)

[ZuseZ4]

@bors try jobs=dist-x86_64-llvm-mingw

[rust-bors]

☀️ Try build successful (CI)
Build commit: 6211c2e (6211c2eadfd193f4d519380ee14be8116cb8d05e, parent: 59fd4ef94daa991e6797b5aa6127e824f3067def)

[ZuseZ4]

I guess we could extract some shared logic that makes sense for both LLVM and LLD into a separate function and use it both in configure_llvm and in the LLD build step.

That's already shared between llvm and lld. Enzyme ignores these LLVM_ variables, so we instead have to use -fuse-ld=lld, which in turn isn't needed for the LLVM/LLD build steps.

I don't think we can nicely share it between all three, llvm+lld+enzyme.

I tried all 5 targets and they build fine.