Bonsai 8B running on AMD Strix Halo (gfx1151) via ROCm/HIP — 76.7 t/s

Bonsai 8B on AMD Strix Halo — ROCm/HIP Build & Benchmark

Successfully built and ran Bonsai 8B on AMD Strix Halo (gfx1151) using the PrismML llama.cpp fork with ROCm/HIP. Sharing the build fix and benchmark results for anyone else on AMD hardware.

Hardware

Component	Detail
APU	AMD Strix Halo (Ryzen AI Max+ 395)
GPU	Radeon 8060S (gfx1151)
Memory	128GB unified (GPU/system shared)
ROCm	7.2 (AMD clang 22.0.0)
PyTorch	2.9.1+rocm7.11.0
OS	Arch Linux (kernel 6.19.9)

Benchmark Results

Metric	CPU-only	ROCm/HIP (GPU)
Prompt	3.3 t/s	81.4 t/s
Generation	2.3 t/s	76.7 t/s

Model loads entirely into GPU memory at 1.15 GB — leaves plenty of headroom on the 128GB unified pool.

Build Fix for ROCm/HIP

The PrismML llama.cpp fork doesn't officially support ROCm yet, but the CUDA sources hipify cleanly with two fixes:

Problem 1: Missing hip_fp16.h

fatal error: 'hip/hip_fp16.h' file not found

Fix: Add -I/opt/rocm/include to HIP compiler flags.

Problem 2: Missing libamdhip64

ld.lld: error: unable to find library -lamdhip64

Fix: Add -L/opt/rocm/lib to linker flags.

Full CMake Command

CMAKE_PREFIX_PATH=/opt/rocm HIP_PATH=/opt/rocm cmake -B build \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS=gfx1151 \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_HIP_COMPILER=/opt/rocm/llvm/bin/clang++ \
  -DCMAKE_HIP_FLAGS="-I/opt/rocm/include" \
  -DCMAKE_EXE_LINKER_FLAGS="-L/opt/rocm/lib" \
  -DCMAKE_SHARED_LINKER_FLAGS="-L/opt/rocm/lib"

cmake --build build -j$(nproc)

Run Inference

./build/bin/llama-cli \
  -m Bonsai-8B-Q1_0_g128.gguf \
  -p "Your prompt here" \
  -n 256 \
  --temp 0.5 \
  --top-p 0.85 \
  --top-k 20 \
  -ngl 99

Notes

• All Q1_0_g128 CUDA kernels compiled cleanly through HIP without modification
• AOTriton is used for attention (confirmed in logs)
• The 1-bit model at 1.15 GB is an ideal fit for unified memory architectures like Strix Halo
• NPU (XDNA 2, 50 TOPS) could be an interesting future target for 1-bit inference

Feature Request

Consider adding official ROCm/HIP support to the build scripts. The two fixes above are trivial and would open Bonsai to the entire AMD GPU ecosystem. Happy to contribute a PR if interested.

Designed and built by the architect

Full Benchmark Suite — Bonsai 8B vs Qwen3-30B-A3B on Strix Halo

Ran llama-bench with multiple prompt/generation sizes, 3 runs averaged. Head-to-head against Qwen3-30B-A3B (Q4_K_M) which is the daily driver on this machine.

Hardware: Radeon 8060S (gfx1151), 128GB unified, 32 threads

Prompt Processing (tokens/sec)

Prompt Size	Bonsai 8B (1-bit, 1.07 GiB, ROCm/HIP)	Qwen3-30B-A3B (Q4_K, 17.28 GiB, Vulkan)
pp128	427	597
pp256	621	908
pp512	765	1,096
pp1024	710	947
pp2048	662	—

Text Generation (tokens/sec)

Gen Length	Bonsai 8B (1-bit, ROCm/HIP)	Qwen3-30B-A3B (Q4_K, Vulkan)
tg64	93.4	81.3
tg128	93.6	82.6
tg256	91.9	82.7
tg512	87.8	—

Analysis

• Bonsai wins text generation by 13% — 93.6 t/s vs 82.6 t/s, at 16x smaller model size
• Bonsai generation barely degrades from tg64→tg512 (93→88), very consistent
• Qwen3-MoE wins prompt processing due to architecture advantages
• Intelligence density is remarkable: ~87% generation speed at 6% of the size

For generation-heavy workloads (chat, code completion, voice agents), Bonsai 8B is actually the faster model on this hardware. ROCm/HIP build with the cmake fix from the original issue.

Designed and built by the architect

Full Bonsai Family Benchmarks — All Three Models on Strix Halo

Pulled down the 4B and 1.7B, benchmarked the entire family. llama-bench, 3 runs averaged, 32 threads, ROCm/HIP.

Hardware: Radeon 8060S (gfx1151), 128GB unified

Text Generation (tokens/sec)

Gen Length	Bonsai 1.7B (231 MB)	Bonsai 4B (540 MB)	Bonsai 8B (1.07 GB)
tg64	236	121	93
tg128	236	127	94
tg256	231	126	92
tg512	212	116	88

Prompt Processing (tokens/sec)

Prompt	Bonsai 1.7B	Bonsai 4B	Bonsai 8B
pp128	2,414	1,025	427
pp256	3,267	1,419	621
pp512	2,887	1,658	765
pp1024	2,762	1,551	710
pp2048	2,474	1,263	662

vs Qwen3-30B-A3B (Q4_K_M, 17.28 GB, Vulkan)

Model	tg128	pp512	Size
Bonsai 1.7B	236 t/s	2,887 t/s	231 MB
Bonsai 4B	127 t/s	1,658 t/s	540 MB
Bonsai 8B	94 t/s	765 t/s	1.07 GB
Qwen3-30B-A3B	83 t/s	1,096 t/s	17.28 GB

The 1.7B at 236 t/s generation is wild — all three Bonsai models combined (1.84 GB) use less memory than a single Qwen3-30B. ROCm/HIP build, same cmake fix from the original post.

Designed and built by the architect

[khosravipasha]

Thanks this is very cool, yeah love to support more gpus
there was on PR here in our fork.
PrismML-Eng/llama.cpp#5

note that we are currently in process of merging our current supported backend in main llama.cpp, after that is done I will look into adding more backend, but this is very nice for AMD GPUs, thanks for sharing the benchmarkrs.

[khosravipasha]

Also trying to add it to our release binaries to we have some rebuild binaries for AMD GPUs available in the meantime:

https://github.com/PrismML-Eng/llama.cpp/blob/ba7e817eecea53595fa9ee5ff4dce2d7febbd26b/.github/workflows/release-prism.yml#L190-L200

[khosravipasha]

We have AMD builds now in our latest release builds: https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b8201-ba7e817

And this PR will add it automatically to check for it https://github.com/PrismML-Eng/Bonsai-demo/pulls

Thanks @khosravipasha! Tested the new ROCm 7.2 pre-built (prism-b8201-ba7e817) on the same Strix Halo (gfx1151, 128GB unified):

Model	ROCm pre-built (tok/s)	Vulkan+FA compiled (tok/s)
Bonsai 8B (Q1_0)	pp512: 714.1 / tg128: 90.6	pp512: 330.1 / tg128: 103.7
Bonsai 4B (Q1_0)	tg128: 122.2	tg128: 148.3
Bonsai 1.7B (Q1_0)	tg128: 230.8	tg128: 260.0

ROCm build detects gfx1151 correctly and runs clean — nice work on the AMD support. Prompt processing is way faster on ROCm (714 vs 330 for 8B), but generation is ~11-18% slower than Vulkan+Flash Attention on this chip.

Sticking with Vulkan for gen throughput but keeping the ROCm build for testing. Great to have the option.

Setup: Arch Linux, ROCm 7.2.1, kernel 6.19.11

[khosravipasha]

@stampby awesome thanks for testing. Yeah, Vulkan is a good options too.

There is Vulkan PR here: PrismML-Eng/llama.cpp#9

Sending CUDA PR to main llama.cpp soon as well, after that's merged should be able to get AMD support in similar way they do there.

To clarify the hardware and backends:

All benchmarks above ran on the Strix Halo APU (Ryzen AI MAX+ 395) — the integrated Radeon 890M is gfx1151 (RDNA 3.5), same arch as the discrete 8060S. 128GB unified memory.

Vulkan backend uses the iGPU. The generation speed advantage (103.7 vs 90.6 tok/s on 8B) comes from Vulkan + Flash Attention compiled from source, running on that same integrated GPU. ROCm/HIP also targets the iGPU — both backends hit the same silicon, just different driver stacks.

A discrete Radeon 8060S with dedicated VRAM and more CUs should outperform these iGPU numbers on both backends.

Quick reference — Bonsai on Strix Halo iGPU (gfx1151):

Model	Vulkan+FA (gen tok/s)	ROCm (gen tok/s)	ROCm (pp512 tok/s)
8B	103.7	90.6	714.1
4B	148.3	122.2	—
1.7B	260.0	230.8	—

Now that the prism branch has Vulkan support (PR #9), looking forward to seeing if the prebuilt Vulkan binary matches these compiled numbers.

Designed and built by the architect

[khosravipasha]

@stampby
I just did a new release of bonsai demo with vulkan setup, have not tested it yet but now hopely bonsai demo should work out of the box with vulkan (only should use vulkan if there is no metal or cuda, etc)

#35

If you could test on your hardware with fresh install would be amazing.

https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b8796-e2d6742