Cannot build `jit` branch due to missing `llvm/ADT/SmallVector.h` file

[pdarragh]

I assume @dvanhorn has some extra config locally that I'm missing, maybe? Not sure where to look, to be honest.

Racket: 9.0
macOS: 15.7.5 (Sequoia)
clang: Apple clang 17.0.0
CPU: Apple M1 Pro

I checked out the jit branch, then did raco setup a86 to re-build everything. This resulted in:

raco setup: pre-installing: <pkgs>/a86/a86
a86: building native JIT library in /Users/pdarragh/Development/Work/cmsc430/a86/a86/llvm-jit for macosx
gmake: llvm-config: No such file or directory
gmake: llvm-config: No such file or directory
mkdir -p build lib
clang++ -std=c++17 -O2 -g -fPIC -Iinclude  -c src/a86_jit.cpp -o build/a86_jit.o
src/a86_jit.cpp:10:10: fatal error: 'llvm/ADT/SmallVector.h' file not found
   10 | #include "llvm/ADT/SmallVector.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
gmake: *** [Makefile:36: build/a86_jit.o] Error 1
build-jit: native JIT build failed with exit code 2
  context...:
   /Applications/Racket v9.0/collects/setup/setup-core.rkt:934:2: do-install-part
   body of "/Applications/Racket v9.0/collects/setup/main.rkt"
   body of "/Applications/Racket v9.0/collects/raco/main.rkt"

which subsequently caused:

raco setup: running: <pkgs>/a86/a86/scribblings/a86.scrbl
ffi-lib: could not load foreign library
  path: /Users/pdarragh/Development/Work/cmsc430/a86/a86/./llvm-jit/lib/liba86_jit.dylib
  system error: dlopen(/Users/pdarragh/Development/Work/cmsc430/a86/a86/./llvm-jit/lib/liba86_jit.dylib, 0x0006): tried: '/Users/pdarragh/Development/Work/cmsc430/a86/a86/./llvm-jit/lib/liba86_jit.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/pdarragh/Development/Work/cmsc430/a86/a86/./llvm-jit/lib/liba86_jit.dylib' (no such file), '/Users/pdarragh/Development/Work/cmsc430/a86/a86/./llvm-jit/lib/liba86_jit.dylib' (no such file), '/Users/pdarragh/Development/Work/cmsc430/a86/a86/llvm-jit/lib/liba86_jit.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/pdarragh/Development/Work/cmsc430/a86/a86/llvm-jit/lib/liba86_jit.dylib' (no such file), '/Users/pdarragh/Development/Work/cmsc430/a86/a86/llvm-jit/lib/liba86_jit.dylib' (no such file)
  context...:
   /Applications/Racket v9.0/collects/ffi/unsafe.rkt:136:0: get-ffi-lib
   body of "/Users/pdarragh/Development/Work/cmsc430/a86/a86/interp.rkt"
   body of 'program
   /Applications/Racket v9.0/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
   /Applications/Racket v9.0/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
   /Applications/Racket v9.0/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
raco setup: --- installing collections ---                         [15:22:46]
raco setup: --- post-installing collections ---                    [15:22:46]
raco setup: --- summary of errors ---                              [15:22:46]
raco setup: error: during early install for <pkgs>/a86/a86
raco setup:   build-jit: native JIT build failed with exit code 2
raco setup: error: during building docs for <pkgs>/a86/a86/scribblings/a86.scrbl
raco setup:   ffi-lib: could not load foreign library
raco setup:     path: /Users/pdarragh/Development/Work/cmsc430/a86/a86/./llvm-jit/lib/liba86_jit.dylib
raco setup:     system error: dlopen(/Users/pdarragh/Development/Work/cmsc430/a86/a86/./llvm-jit/lib/liba86_jit.dylib, 0x0006): tried: '/Users/pdarragh/Development/Work/cmsc430/a86/a86/./llvm-jit/lib/liba86_jit.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/pdarragh/Development/Work/cmsc430/a86/a86/./llvm-jit/lib/liba86_jit.dylib' (no such file), '/Users/pdarragh/Development/Work/cmsc430/a86/a86/./llvm-jit/lib/liba86_jit.dylib' (no such file), '/Users/pdarragh/Development/Work/cmsc430/a86/a86/llvm-jit/lib/liba86_jit.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/pdarragh/Development/Work/cmsc430/a86/a86/llvm-jit/lib/liba86_jit.dylib' (no such file), '/Users/pdarragh/Development/Work/cmsc430/a86/a86/llvm-jit/lib/liba86_jit.dylib' (no such file)

[pdarragh]

Wait, do I just... not have LLVM? Can't believe I didn't think to check for that first. Let me brew install llvm and see if that resolves this issue.

That said, might be worth adding an explicit check to the setup process for that?

[pdarragh]

Okay yeah, so I had to brew install llvm, and then I adjusted the LDFLAGS and CPPFLAGS per the info from brew, but now I have:

raco setup: running: <pkgs>/a86/a86/scribblings/a86.scrbl
clang not found: either you have not installed clang
or it cannot be found in your PATH: /Users/pdarragh/.opam/default/bin:/Users/pdarragh/.pyenv/shims:/Applications/Racket v9.0/bin:/Users/pdarragh/.local/share/cargo/bin:/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/bin:/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/sqlite/bin:/opt/homebrew/opt/openssl@3/bin:/opt/homebrew/opt/llvm/bin:/opt/homebrew/sbin:/opt/homebrew/bin:/Users/pdarragh/.local/bin:/Users/pdarragh/.ghcup/bin:/Applications/Racket v9.0/bin:/Users/pdarragh/.local/share/cargo/bin:/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/bin:/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/sqlite/bin:/opt/homebrew/opt/openssl@3/bin:/opt/homebrew/sbin:/opt/homebrew/bin:/Users/pdarragh/.local/bin:/Users/pdarragh/.ghcup/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/TeX/texbin:/Applications/iTerm.app/Contents/Resources/utilities:/opt/homebrew/Cellar/emacs-plus@31/31.0.50/libexec/emacs/31.0.50/aarch64-apple-darwin24.6.0.
  context...:
   body of "/Users/pdarragh/Development/Work/cmsc430/a86/a86/interp.rkt"
   body of 'program
   /Applications/Racket v9.0/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
   /Applications/Racket v9.0/share/pkgs/sandbox-lib/racket/sandbox.rkt:921:7
   /Applications/Racket v9.0/share/pkgs/sandbox-lib/racket/sandbox.rkt:891:2: user-process
raco setup: --- installing collections ---                         [15:34:24]
raco setup: --- post-installing collections ---                    [15:34:24]
raco setup: --- summary of errors ---                              [15:34:24]
raco setup: error: during building docs for <pkgs>/a86/a86/scribblings/a86.scrbl
raco setup:   clang not found: either you have not installed clang
raco setup:   or it cannot be found in your PATH: /Users/pdarragh/.opam/default/bin:/Users/pdarragh/.pyenv/shims:/Applications/Racket v9.0/bin:/Users/pdarragh/.local/share/cargo/bin:/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/bin:/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/sqlite/bin:/opt/homebrew/opt/openssl@3/bin:/opt/homebrew/opt/llvm/bin:/opt/homebrew/sbin:/opt/homebrew/bin:/Users/pdarragh/.local/bin:/Users/pdarragh/.ghcup/bin:/Applications/Racket v9.0/bin:/Users/pdarragh/.local/share/cargo/bin:/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/bin:/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/sqlite/bin:/opt/homebrew/opt/openssl@3/bin:/opt/homebrew/sbin:/opt/homebrew/bin:/Users/pdarragh/.local/bin:/Users/pdarragh/.ghcup/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/TeX/texbin:/Applications/iTerm.app/Contents/Resources/utilities:/opt/homebrew/Cellar/emacs-plus@31/31.0.50/libexec/emacs/31.0.50/aarch64-apple-darwin24.6.0.

This doesn't make sense to me, because part of the quoted PATH includes /opt/homebrew/opt/llvm/bin, which is where my clang executable can now be found:

$ type -a clang
clang is /opt/homebrew/opt/llvm/bin/clang
clang is /usr/bin/clang

However, clang now reports its version:

$ clang --version
Homebrew clang version 22.1.4
Target: arm64-apple-darwin24.6.0
Thread model: posix
InstalledDir: /opt/homebrew/Cellar/llvm/22.1.4/bin
Configuration file: /opt/homebrew/Cellar/llvm/22.1.4/etc/clang/arm64-apple-darwin24.cfg

So I'm wondering if the check is actually looking for x86 clang instead of arm64 clang and failing for that reason. I'll continue to investigate. (I really want those faster compilation times!)

[pdarragh]

Okay, as I thought, a86/check-assembler.rkt has the following function to check the clang version:

a86/a86/check-assembler.rkt

Lines 37 to 45 in 9a763e9

	;; -> [Maybe String]
	(define (clang-version-string)
	(parameterize ([current-output-port (open-output-string)]
	[current-error-port (open-output-string)])
	(and (system (format "~aclang --version 2>&1"
	(if (macos?)
	"arch -x86_64 "
	"")))
	(get-output-string (current-output-port)))))

On my system, this runs arch -x86_64 clang --version, which produces:

$ arch -x86_64 clang --version          
arch: posix_spawnp: clang: Bad CPU type in executable

with status code 1. Maybe a good idea would be to revise the check to see (a) is there a clang executable at all, and then (b) can we invoke it with the arch -x86_64 prefix successfully. This could produce an error like "Detected clang (/path/to/clang) does not support x86-64 execution" or something.

[pdarragh]

Seems like this will require installing x86 Homebrew, unfortunately. For that, I did:

$ arch -x86_64 zsh
$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
$ /usr/local/bin/brew install llvm
$ exit

At this point, an x86-compatible Homebrew is installed under Rosetta 2, along with LLVM. The environment variables I configured:

# These are in [~/.zshrc].
export PATH="/usr/local/opt/llvm/bin:$PATH"
export LDFLAGS="-L/usr/local/opt/llvm/lib $LDFLAGS"
export CFLAGS="-I/usr/local/opt/llvm/include $CFLAGS"
export CPPFLAGS="-I/usr/local/opt/llvm/include $CPPFLAGS"

I think we no longer need to prefix arch -x86_64. In a fresh shell:

$ clang --version
Homebrew clang version 22.1.4
Target: x86_64-apple-darwin24.6.0
Thread model: posix
InstalledDir: /usr/local/Cellar/llvm/22.1.4/bin
Configuration file: /usr/local/Cellar/llvm/22.1.4/etc/clang/x86_64-apple-darwin24.cfg

At this point, I was able to raco setup a86 and get things working. (However, because I had a failed build already, I had to manually invoke make clean from within the a86/llvm-jit subdirectory; I think that should be incorporated into the pre-install.rkt script.)

[pdarragh]

Some downsides of this approach are:

• I now have two separate Homebrew installations.

• A lot of extra space was taken to support this. Before I had installed LLVM under Rosetta, I did a --dry-run to see what I would get:

  /usr/local/bin/brew install --dry-run llvm
  ==> Would install 1 formula:
  llvm
  ==> Downloading https://ghcr.io/v2/homebrew/core/llvm/manifests/22.1.4
  Already downloaded: /Users/pdarragh/Library/Caches/Homebrew/downloads/bb1c694280f7cda1d363021bafa9287b01e647ea67598cac7d6371fe8f161b72--llvm-22.1.4.bottle_manifest.json
  ==> Would install 10 dependencies for llvm:
  mpdecimal ca-certificates openssl@3 readline sqlite xz lz4 zstd python@3.14 z3
  

  It seems likely to me that, e.g., the Python installation probably doesn't need to be x86, and possibly some of the other things too. It's not the worst thing; just not my favorite.

• The /usr/local/ directory is now heavily polluted to serve a bunch of software I don't intend to use often.

• There's a whole new set of environment variable configuration to do.

I really wish the Homebrew folks had a plan to maybe incorporate supporting x86 under the native Apple Silicon Homebrew installation somehow, but I couldn't find any indication along these lines. In fact, it seems like they expect Rosetta 2 to eventually cease being supported by Apple because that's how things went back in the PowerPC transition, but that seems an unfounded analogy to me because Apple was PowerPC at that time; meanwhile, Intel CPUs are not going anywhere in the short term, so there will almost certainly always be some x86 software that needs support.

---

I think something that could be worth testing is installing Homebrew to a custom path. If that works, we could maybe consider incorporating that into the (as yet hypothetical) 430 CLI tool, i.e., we could install x86 Homebrew somewhere secret, install LLVM there as well, and then just reference that directly for building course software or something. Seems a little cumbersome, but it also relies less on hoping our "end users" understand how to configure all this correctly.

[pdarragh]

okay wow but the speed-up in execution time is totally worth it holy crap

[dvanhorn]

I also wonder if it might be easy to provide binaries so folks don't need to install everything.

[dvanhorn]

BTW, I think you want to be on the next branch, not jit. I think next fully subsumes jit and is set up to run the langs tests using the langs next branch. (I'll work to reconcile these two branches soon.). I'm going to look into pre-built libraries to make things easy on students.

[dvanhorn]

On next, the package now includes pre-built binaries for macos and linux. When you push, an action will build new binaries for you and generate a PR to include (once this lives on main that should probably be a workflow_dispatch). So I think we can safely avoid all of the complications around building the JIT library for students. Also nice for testers etc since there's significantly less set-up time now.

[pdarragh]

Wow, that's awesome!

Out of curiosity, does this distribute both prebuilt binaries for all users and then it just selects the relevant one?

[dvanhorn]

That's right. This adds about 100MB to the package.

[pdarragh]

Gotcha, okay, that doesn't seem so bad. And they rebuild every time we push? This increases git fetch time pretty noticeably... which isn't the biggest deal, I suppose, but it makes me wonder about alternative approaches.

Maybe we could use semester-based releases? When a release is created, the create action can be used to trigger a workflow to build the binaries. Then the course website would point students to the relevant release for their semester, they'd get the binaries, and we'd be free to continue making updates to the a86 library without causing problems for them and also without re-building the binaries every time we push. Or, at least, I think that's how it could work.

And I think there's a way to get releases to be made when you tag a commit, so you could manage everything from your local git and have GitHub do what's expected on push, but I haven't looked far enough to verify.

[dvanhorn]

I think we probably want a semester based branch for both a86 and langs. Have students install both a86 and langs with the relevant semester branch (if you just rely on langs to pull in its dependencies it will grab the main branch of a86). During the semester, we hopefully only make minor updates to that branch. In the meantime, we can keep working on main and other branches. The binaries in a86 are rebuilt on every push, but they don't automatically go into the repository. So you only accept the auto-generated PR when you want to binaries to be updated.

[pdarragh]

A strategy some large repositories use is to use a develop branch for pushing all changes to, and then only occasionally merging that into main. That way people can just point to the repository and get a working version of everything without having to specify a branch.

So my suggestion is to use the same process: next can be our development branch, and then occasionally it gets merged into main when we know things are working. What do you think?