You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
You can start from unsloth's BF16 quantized model and quantize to Q4_K_4 or Q8R16 following the instruction. And 30b can barley run on 24GB RAM in Q4_K_4. But slow of course.
You can start from unsloth's BF16 quantized model and quantize to Q4_K_4 or Q8R16 following the instruction. And 30b can barley run on 24GB RAM in Q4_K_4. But slow of course.