No cpu-only build

[zcattacz]

llama.cpp (GGUF) — C/C++, runs on Mac (Metal), Linux/Windows (CUDA), and CPU.

Despite the README claimed to support CPU. Can't find any info on CPU only build for this fork. Maybe cpu-only is not on the plan?

[khosravipasha]

This demo focuses on Metal on Mac and GPU variations. CPU only build is possible. Note that the CPU implementations might not be the most optimized.

Saw a PR here to do cpu builds on Intel Macs: #4

Or for comprehensive build options can checkout the official llama.cpp build guides. There are too many different platform to cover in a simple demo: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md

[anogaia]

CPU builds would enable much wider adoption and experimenting.
Not everyone has a CUDA or METAL GPU, either, so adding an option for VULKAN would make it available to an even wider audience. After all, the whole point of this exercise is to show how an LLM can run on limited resources, is it not?

[zcattacz]

Hi @khosravipasha, thanks for the info. I compiled the fork with cmake -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_OPENBLAS=ON on an i5 box. Like you hinted, it didn't really work.

BONSAI_MODEL=4B ./scripts/run_llama.sh -c 8192 -p "Who are you? Introduce yourself in haiku0"
[WARN] Auto-fit not supported, falling back to -c 8192
llama.cpp-prism/bin/cpu/llama-cli
warning: no usable GPU found, --gpu-layers option will be ignored
warning: one possible reason is that llama.cpp was compiled without GPU support
warning: consult docs/build.md for compilation instructions

Loading model...
build      : b0-unknown
model      : Bonsai-4B.gguf
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read               add a text file

> Who are you? Introduce yourself in haiku0                                                                                                
 from, can can can the, can can can can can can even can the the the List, and shifting switched slow the List slow the the
(5min later, Ctrl+C)
[ Prompt: 0.1 t/s | Generation: 0.1 t/s ]                                                                                                  

> hi                                                                                                                                       
(5min later, Ctrl+C)
[ Prompt: 0.1 t/s | Generation: 0.1 t/s ]                                                                                                  

> /clear                                                                                                                                   
Chat history cleared.

> hi                                                                                                                                       
 and,.
 ( previously the same.
 ( the the List as. One Formatting. List, ( for, can can with and shifting switching shifting large large and similar,, ( also the same the, and like,, and similar,,., and shifting, and, and, and similar, or, and,, and, and,,. and,.
 Altern,
(10min later, Ctrl+C)
[ Prompt: 0.1 t/s | Generation: 0.1 t/s ]                                                                                                  

To verify the compiled llama.cpp is usable, I tried bin/cpu/llama-cli -c 8192 -m LFM2-2.6B-Q4_K_M.gguf.

build      : b0-unknown
model      : LFM2-2.6B-Q4_K_M.gguf
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read               add a text file

> hi                                                                                                                                       
Hello! How can I assist you today?
[ Prompt: 8.7 t/s | Generation: 8.4 t/s ]                                                                                                  

Actually Gemma3-4B-Q4_0_S gives about 3.54TPS, something seems off with the CPU impl of this 1-bit type.

[khosravipasha]

@zcattacz Thanks, yeah we realized there was an issue in x86 kernels, working on a fix now

PrismML-Eng/llama.cpp#8

[khosravipasha]

@anogaia We have some vulkan/opencl kernels as well, I need to test them further a bit and would be happy to release them in a branch.

[markd89]

CPU builds would enable much wider adoption and experimenting. Not everyone has a CUDA or METAL GPU, either, so adding an option for VULKAN would make it available to an even wider audience. After all, the whole point of this exercise is to show how an LLM can run on limited resources, is it not?

Also, a selling point of a small model is it fits in small hardware. If I've got NVDIA, I can do a larger model. If I have CPU only, I have to use a small model.

[khosravipasha]

We now have NEON build in our fork, there are few candidates for x86 that will be added in near future, for now can refer to the PRs in our llama.cpp fork. for example: PrismML-Eng/llama.cpp#10

We are working on porting our kernels to main llama.cpp as well.

[khosravipasha]

NEON and Metal builds are now in the main llama.cpp so don't need to use our fork

x86 and CUDA backend are pending PRs.

Should get decent speed on x86 after this PR in llama.cpp is merged
ggml-org/llama.cpp#21636