Memory Issue

[Pengy-X]

Checks

• I have searched existing issues and discussions
• I can reproduce this with the latest main or release

Describe the bug

Whenever I use a model, regardless of the size of model, each request (either through Osaurus built-in chat window or an external application like BoltAI) causes the model to load an additional time. I tested with Qwen3 4B 2507 (around 2gb) and after each question, Osaurus' memory usage would multiply in size. For larger models like qwen3 30b, Osaurus consumed 40gb+ ram after just two messages and MacOS started using 6gb+ swap. Hitting "unload model" or "clear all" did not unload the model, and the only way to unload models was to quit Osaurus.

Steps to reproduce

No response

Osaurus version / commit

v0.5.54

macOS version

26.2

Apple Silicon chip

M4 Pro 48gb

Xcode version

26.2

Logs

Screenshots

No response

[tpae]

Thanks for reporting this issue. I think it's related to #192 . I'm doing iterations and you can follow it for progress.

[Pengy-X]

Thanks for the response, I appreciate the work going into Osaurus! I'd be happy to do any testing and report back results

[tpae]

Hey @Pengy-X , i made some updates to the memory management. Could you give it another test? Thank you

[Pengy-X]

I updated Osaurus and the memory buildup seems to happen still. After around 1.5k tokens the UI froze, and I tried again with some different prompts and the UI went blank after around 2k tokens. Tried another time and got 3k tokens with no ui issues, but the memory buildup was still there (all with built-in chat). Looking at activity monitor memory usage, I think it is not "model being loaded in multiple times" like I had previously thought, Osaurus just has some nonuniform jumps in memory usage starting from when a request is submitted.

[tpae]

Can you confirm these values?

Max Size = lower = slower but better memory management (set to 4k)

Can you share a video of the issue if possible?

[Pengy-X]

Not sure if github allows uploading video in comment, so here's a link to video: https://drive.proton.me/urls/BHD0YPNG64#oQEzXrF4HnUN

[tpae]

hey @Pengy-X , can you give it another test? it should be better now.

[Pengy-X]

It looks like memory is still growing with usage

[tpae]

Yeah. Model itself has a internal KV cache that grows each time you work with it. it should free it though when you clear the model cache itself. Try clicking on the RAM bar, you should be able to free the memory from there.

[Pengy-X]

I see, "unload" does properly clear the model, but then does the model need to be loaded again for a new request? I'm not sure how LM Studio is handling KV cache, but it seems to grow in smaller steps than osaurus, and the kv cache isn't quantized. I tried setting osaurus kv cache bits to 8192, then 4096, but the memory usage was still quite high

[tpae]

LM Studio uses different library (python), we use swift, there's going to be differences. Swift library isn't as maintained as the python version, I'm hoping that will change soon.