Issue with converting and quantising deepseek-ai/DeepSeek-R1-Distill-Llama-8B

[jaycoolslm]

I've tried various mlx_lm.convert configs to manually convert this deepseek model, but always the output seems to be partial gibberish which isn't decoded propertly.

Things I've tried:

mlx_lm.convert \                                                                                                                        ✔
  --hf-path deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
  --mlx-path ./output_models/deepseek_r1_llama8b \
  -q

mlx_lm.convert \                                                                                                                        ✔
  --hf-path deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
  --mlx-path ./output_models/deepseek_r1_llama8b_mxfp4 \
  -q --q-mode mxfp4 --q-bits 4 --q-group-size 32 \
  --trust-remote-code


mlx_lm.convert \                                                                                                                        ✔
  --hf-path deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
  --mlx-path ./output_models/deepseek_r1_llama8b_affine \
  -q --q-mode affine --q-bits 4 --q-group-size 64 \
  --trust-remote-code

And the generated response generally looks like this:

>> hey there
ĊĊ</think>ĊĊHello!ĠHowĠcanĠIĠassistĠyouĠtoday?ĠðŁĺĬ

[awni]

It looks like the tokenizer decoder is getting changed due to underlying change in Transformers. I'm trying to find out if this is a Transformers bug or if we need to fix something: huggingface/transformers#43066

[awni]

Should be fixed in huggingface/transformers#42894