GLM-4.7-Flash is incoherent in further generations (KCPP 1.107 dev build)

[aoleg]

Just reporting that https://github.com/LostRuins/koboldcpp/actions/runs/21299565080 introduced a bug causing garbled thinking/output at least on GLM 4.7 Flash and GLM 4.5 Air models regardless of whether Flash Attention is on or off. Rolling back to https://github.com/LostRuins/koboldcpp/actions/runs/21210595986 fixes the issue, so it was likely something merged in between. Could it be b70d251 ?

[LostRuins]

i'll try doing a new CI build
https://github.com/LostRuins/koboldcpp/actions/runs/21349616387

if still not fixed, might have to merge some changes

[aoleg]

Still bugged. The first interaction goes well, but the second one...

Generating (673 / 4096 tokens)
[15:11:18] CtxLimit:2773/32768, Amt:673/4096, Init:0.07s, Process:0.26s (3911.20T/s), Generate:8.40s (80.13T/s), Total:8.66s
Output:  -

NULLmulti particularly_t'tikip gruesmulti与发展|'tобы对社会 vit的不同apimulti11 \
multi0 Dmulti2'tmulti的具体_fmulti_to'tmult_kSERVER11'tmulti1'tmulti20 gardenmulti2't virtualmulti_fmulti_fmulti0_f'tmulti11110'tmulti foreachmulti_kmulti particularly_kmultiungapi particularly_tmult particularlyу.environ'tmulti particular适-
mult_tomult1便 \
...
 virtual ...
rf't.environmulti11'tmulti1tnfn0multi1printfmult_t't carnmultiio virtual Enumerablemult1обыeng """
 particip2multi(...)
...
 vibrmulti10multi virtual Assign camp| particularly_t virtual_d |
 vacc vit fiction BUG differs_j -```

[aoleg]

Model: unsloth\GLM-4.7-Flash-GGUF\GLM-4.7-Flash-UD-Q6_K_XL.gguf
Same model, same chat, latest llama.cpp b7836 - fine with and without Flash Attention.
Same model, 21349616387 build, FA = ON: first reply OK, garbage on second reply
Same model, 21349616387 build, FA = OFF: first reply OK, garbage on second reply - but it's a different kind of garbage that looks different, e.g. see below.

Finally, the https://github.com/LostRuins/koboldcpp/actions/runs/21210595986 build. Obviously FA = OFF; first message OK, second message OK. Subsequent messages also OK. So 21210595986 is the last working build for GLM 4.7 Flash (but llama.cpp is ahead with the performance patches and KV cache optimizations at this point).

it's getting on my friends and it's sister. And I'm a bit? I'd say that guy was the. I guess that's a little bit, maybe. I've got it's right? I'm not a total. I'm so make your typical. Maybe. Just's the next I think the only. I'm the way, but I just
I\n me. I mean she doesn't I’m a guy. 'I }
What do the way I guess I think I'm a bit. And it's so me to`
I think, I'm. I’m going back at the only. I can you'd be my nose. I am I guess I guess. I've never I can you get out. And I guess I don't look at my head. But, but I guess. I might have a date. I can'ts face, I'm a nice. I think. I know! I guess, though it was. No, I'd just I’m face. It's just like that I could you know, maybe to度```

[LostRuins]

Sorry, I haven't had time to test out GLM 4.7 Flash yet. I did a tentative quick merge with https://github.com/LostRuins/koboldcpp/actions/runs/21402388144 but if you had problems with the previous build, this might not work for you either.

May I know what GPU you are using?

[aoleg]

I'm using an NVIDIA RTX 5090 (CUDA backend).

[aoleg]

Tested https://github.com/LostRuins/koboldcpp/actions/runs/21402388144, no change. First interaction OK (including re-generation), second interaction produces garbage:

Generating (273 / 8192 tokens)
[14:37:51] CtxLimit:8045/32768, Amt:273/8192, Init:0.01s, Process:0.27s (2567.67T/s), Generate:3.46s (78.79T/s), Total:3.73s
Output: 2 | |
 |
 |

 | |
 |

 |

[LostRuins]

@aoleg just to confirm, if you run it with --nofastforward it works find correct?

[LostRuins]

Also worth keeping an eye on ggml-org#19128, output kinda looks like yours

1 small test to try also - add [gMASK]<sop> at the start of your system prompt just to test

[aoleg]

Running with --nofastforward results in a similar output to the working build https://github.com/LostRuins/koboldcpp/actions/runs/21210595986

The thing is, the output itself even with 21210595986 is less coherent than with the latest llama.cpp builds; it only works with flash attention disabled (and then it is very slow).

[stepfunction83]

I can confirm that with the latest CI build for Windows, the default settings, and latest unsloth GGUF, I was able to run it fine for 2 generations and then it completely broke as described above.

[lolxdmainkaisemaanlu]

I am having the same issue, It's not even a repeat issue or it rambling about something coherent - it just degrades into number spamming and basically random stuff.

EDIT: I tested on the latest build for windows and also changed the prompt on SillyTavern - It's working fine now!

[aoleg]

Can't confirm. Build 21516841434, same result on second message:

Processing Prompt [BATCH] (374 / 374 tokens)
Generating (210 / 4096 tokens)
[00:16:25] CtxLimit:4542/32768, Amt:210/4096, Init:0.01s, Process:0.03s (12466.67T/s), Generate:5.09s (41.29T/s), Total:5.12s
Output:  | |
 |

 |

 |
|| |

 |

|
对身体|r |

 |

 |

 |

| |
|
u|}
|r| ||}```

[LostRuins]

Okay I tested the latest experimental with a RTX4090 on windows, no issue.
I even rented a RTX5090 on a linux cloud VM to test. No issues there either, both with and without flash attention.

@aoleg are you using the KoboldAI Lite UI? Can you try that the latest actions build with the preset "Simple Logical" just to rule out any sampler issues? Also, maybe you can compare with the vulkan backend and see if that works.

@stepfunction83 what GPU are you using?

[LostRuins]

Latest release is out