[Bug?] Failed to allocate CPU_REPACK buffer -> failed to load model (with `--usemmap`)

[alex-ie]

Describe the Issue
I have successfully used --usemmap to load the model GGUF of ~110% of free/available RAM. But loading the model 3x free RAM has failed (the model consisted of several GGUF files, does it matter for below?).

In terminal (numbers rounded):

done getting tensors: ... moved from CPU_REPACK, using CPU instead
ggml_aligned_malloc: insufficient memory (attempted to allocate 103 000 MB)
ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 108 000 000 000
alloc_tensor_range: failed to allocate CPU_REPACK buffer of size 108 000 000 000
llama_model_load: error loading model: unable to allocate CPU_REPACK buffer
llama_model_load_from_file_impl: failed to load model 

I have used --usemmap, why has the engine tried to allocate amount ~ total size of GGUF? Could it be a bug? If not, does such huge allocation necessity depend of model architecture maybe? Some 120 GB models can be loaded in 40 GB free RAM and some cannot? If so, what it depends on?

https://github.com/LostRuins/koboldcpp/wiki

mmap, or memory-mapped file I/O, maps files or devices into memory. It is a method of reducing the amount of RAM needed for loading the model, as parts can be read from disk into RAM on demand. You can enable it with --usemmap

Additional Information:
v1.112 Linux nocuda

[LostRuins]

Thanks! This seems to be a bug. I have applied this to fix it 66a7b5e

[alex-ie]

@LostRuins , thanks!

It works now, I got 0.1 t/s speed!

What happens now? Do you cherry-pick 66a7b5e5de07520b52918122f0084269ed4af1e1 commit from concendo_experimental to production branch?

Also, please confirm (and give advice to improve if needed) my testing procedure:

git checkout concedo_experimental
git pull
git pull origin concedo_experimental
make
make main
python koboldcpp.py --usemmap --model ...

TIA

[LostRuins]

It will be included in the next release

[alex-ie]

Thanks! Was my testing procedure correct?

[LostRuins]

Not really, when you do it this way you are building with no GPU acceleration at all. So it will be very very slow. You need CUDA/vulkan sdk installed and to build with support for it e.g. LLAMA_VULKAN=1

If you want an early test build, you can get one from here: https://github.com/LostRuins/koboldcpp/releases/tag/rolling

[alex-ie]

@LostRuins
I have started the habit to run koboldcpp.py from concedo_experimental branch for production use. It worked apparently fine until I have tried to run on older PC - it just exited, but one from release loaded the model.

For future debugging, I want to understand relationship between branches. I thought concedo_experimental is some cutting edge testing features added for testing, but seems it does not include all main branch have, correct? Or I need to run some additional make command for older CPUs support? TIA

[LostRuins]

concedo_experimental is just my latest newest branch. eventually the changes there will be merged into the concedo branch

[LostRuins]

Should now be fixed in the 1.114 latest release