Notable performance regression in token generation speed (vulkan, windows FlashAttention).

[pqnet]

Describe the Issue
Notable performance regression in decode performance (token generation).

Observed with Vulkan, Windows, FlashAttention

Additional Information:
1.101.1:

Processing Prompt [BLAS] (1948 / 1948 tokens)
Generating (100 / 100 tokens)
[21:54:43] CtxLimit:2048/2048, Amt:100/100, Init:0.64s, Process:11.53s (169.02T/s), Generate:7.72s (12.96T/s), Total:19.24s
Benchmark Completed - v1.101.1 Results:
======
Flags: NoAVX2=False Threads=8 HighPriority=False Cuda_Args=None Tensor_Split=None BlasThreads=8 BatchSize=512 FlashAttention=True KvCache=0
Timestamp: 2026-01-20 20:54:43.101811+00:00
Backend: koboldcpp_vulkan.dll
Layers: 999
Model: openai_gpt-oss-20b-MXFP4
MaxCtx: 2048
GenAmount: 100
-----
ProcessingTime: 11.525s
ProcessingSpeed: 169.02T/s
GenerationTime: 7.719s
GenerationSpeed: 12.96T/s
TotalTime: 19.244s
Output:  1 1 1 1
-----

1.106.2:

Processing Prompt [BATCH] (1948 / 1948 tokens)
Generating (100 / 100 tokens)
[21:56:38] CtxLimit:2048/2048, Amt:100/100, Init:0.68s, Process:17.02s (114.48T/s), Generate:25.39s (3.94T/s), Total:42.41s
Benchmark Completed - v1.106.2 Results:
======
Flags: NoAVX2=False Threads=8 HighPriority=False Cuda_Args=None Tensor_Split=None BlasThreads=8 BatchSize=512 FlashAttention=True KvCache=0
Timestamp: 2026-01-20 20:56:38.645364+00:00
Backend: koboldcpp_vulkan.dll
Layers: 999
Model: openai_gpt-oss-20b-MXFP4
MaxCtx: 2048
GenAmount: 100
-----
ProcessingTime: 17.016s
ProcessingSpeed: 114.48T/s
GenerationTime: 25.394s
GenerationSpeed: 3.94T/s
TotalTime: 42.410s
Output:  1 1 1 1
-----

Last working release 1.101.1. I have observed the same regression on other models (no matter if dense/moe/small/large), and other machines (this is intel ARC, have observed it also on AMD Strix Halo). I seem to remember the issue was observable since 102 but right now the benchamrk on 102 and 103 crashes my computer.

For reference, this is 1.106.2 without FA (token generation performance is the same as flashattention enabled, but context prefill performance is better):

Processing Prompt [BATCH] (1948 / 1948 tokens)
Generating (100 / 100 tokens)
[22:01:17] CtxLimit:2048/2048, Amt:100/100, Init:0.66s, Process:7.38s (264.14T/s), Generate:27.27s (3.67T/s), Total:34.65s
Benchmark Completed - v1.106.2 Results:
======
Flags: NoAVX2=False Threads=8 HighPriority=False Cuda_Args=None Tensor_Split=None BlasThreads=8 BatchSize=512 FlashAttention=False KvCache=0
Timestamp: 2026-01-20 21:01:17.980829+00:00
Backend: koboldcpp_vulkan.dll
Layers: 999
Model: openai_gpt-oss-20b-MXFP4
MaxCtx: 2048
GenAmount: 100
-----
ProcessingTime: 7.375s
ProcessingSpeed: 264.14T/s
GenerationTime: 27.272s
GenerationSpeed: 3.67T/s
TotalTime: 34.647s
Output:  1 1 1 1
-----

[LostRuins]

does it only happen on mxfp4? or all quants?

[LostRuins]

I wonder if its related to #1915

[pqnet]

does it only happen on mxfp4? or all quants?

i empirically tested some k quants too (Q2_K, Q3_K, Q4_K are the one I use most often), i think quant is not relevant here, looks like a memory transfer thing since it mostly affects memory bound operations.

[pqnet]

I wonder if its related to #1915

I seem to remember this problem existed for a while. Let me do a run on 1.104 too to check how it performs

[pqnet]

These are the results for 1.104:
No FA:

Processing Prompt [BATCH] (1948 / 1948 tokens)
Generating (100 / 100 tokens)
[14:35:35] CtxLimit:2048/2048, Amt:100/100, Init:0.65s, Process:6.86s (283.88T/s), Generate:23.96s (4.17T/s), Total:30.82s
Benchmark Completed - v1.104 Results:
======
Flags: NoAVX2=False Threads=8 HighPriority=False Cuda_Args=None Tensor_Split=None BlasThreads=8 BatchSize=512 FlashAttention=False KvCache=0
Timestamp: 2026-01-21 13:35:35.799351+00:00
Backend: koboldcpp_vulkan.dll
Layers: 999
Model: openai_gpt-oss-20b-MXFP4
MaxCtx: 2048
GenAmount: 100
-----
ProcessingTime: 6.862s
ProcessingSpeed: 283.88T/s
GenerationTime: 23.962s
GenerationSpeed: 4.17T/s
TotalTime: 30.824s
Output:  1 1 1 1
-----

With FA:

Processing Prompt [BATCH] (1948 / 1948 tokens)
Generating (100 / 100 tokens)
[14:34:36] CtxLimit:2048/2048, Amt:100/100, Init:0.68s, Process:17.84s (109.17T/s), Generate:34.19s (2.92T/s), Total:52.04s
Benchmark Completed - v1.104 Results:
======
Flags: NoAVX2=False Threads=8 HighPriority=False Cuda_Args=None Tensor_Split=None BlasThreads=8 BatchSize=512 FlashAttention=True KvCache=0
Timestamp: 2026-01-21 13:34:36.301574+00:00
Backend: koboldcpp_vulkan.dll
Layers: 999
Model: openai_gpt-oss-20b-MXFP4
MaxCtx: 2048
GenAmount: 100
-----
ProcessingTime: 17.843s
ProcessingSpeed: 109.17T/s
GenerationTime: 34.194s
GenerationSpeed: 2.92T/s
TotalTime: 52.037s
Output:  1 1 1 1
-----

[pqnet]

These are on 1.102.3:
No FA:

Processing Prompt [BATCH] (1948 / 1948 tokens)
Generating (100 / 100 tokens)
[14:38:27] CtxLimit:2048/2048, Amt:100/100, Init:0.77s, Process:6.88s (282.98T/s), Generate:10.23s (9.78T/s), Total:17.11s
Benchmark Completed - v1.102.3 Results:
======
Flags: NoAVX2=False Threads=8 HighPriority=False Cuda_Args=None Tensor_Split=None BlasThreads=8 BatchSize=512 FlashAttention=False KvCache=0
Timestamp: 2026-01-21 13:38:27.888720+00:00
Backend: koboldcpp_vulkan.dll
Layers: 999
Model: openai_gpt-oss-20b-MXFP4
MaxCtx: 2048
GenAmount: 100
-----
ProcessingTime: 6.884s
ProcessingSpeed: 282.98T/s
GenerationTime: 10.229s
GenerationSpeed: 9.78T/s
TotalTime: 17.113s
Output:  1 1 1 1
-----

with FA i get a crash:

Processing Prompt [BATCH] (1948 / 1948 tokens)Traceback (most recent call last):
  File "koboldcpp.py", line 8378, in <module>
    main(launch_args=parser.parse_args(),default_args=parser.parse_args([]))
  File "koboldcpp.py", line 7358, in main
    kcpp_main_process(args,global_memory,using_gui_launcher)
  File "koboldcpp.py", line 8160, in kcpp_main_process
    genout = generate(genparams=genp)
  File "koboldcpp.py", line 1724, in generate
    ret = handle.generate(inputs)
OSError: [WinError -1073741569] Windows Error 0xc00000ff
[13636] Failed to execute script 'koboldcpp' due to unhandled exception!

seems to be a memory corruption thing quote from https://www.rubydoc.info/gems/windows_error/WindowsError/NTStatus: ```
STATUS_BAD_FUNCTION_TABLE =

(0xC00000FF) A malformed function table was encountered during an unwind operation.

WindowsError::ErrorCode.new("STATUS_BAD_FUNCTION_TABLE",0xC00000FF,"A malformed function table was encountered during an unwind operation.")

[pqnet]

These are benchmarks on llama.cpp llama-bench.exe on b7787:
(full command line: .\llama-bench.exe -ngl 99 -fa 1 --model openai_gpt-oss-20b-MXFP4.gguf)

model	size	params	backend	ngl	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	99	1	pp512	127.77 ± 0.55
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	99	1	tg128	13.43 ± 0.09

This is the same benchmark on b7757 (which is merged into 1.106.2)

model	size	params	backend	ngl	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	99	1	pp512	128.77 ± 2.03
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	99	1	tg128	12.00 ± 0.17

Suggesting it is not an upstream issue

[samschurter]

I've been experiencing this issue on a pair of Arc A770s since 1.101.1. I have not been able to test EVERY version since 101, several of them just crash on prompt, but each version that will run, runs slower than 1.101. Tested with multiple quants and models. Let me know if there's any tests or benchmarks I could run that would help diagnose.

System:
Windows 10
AMD Ryzen 1920X
32 GB RAM
Dual 16GB Arc A770
Intel Arc Graphics Driver 32.0.101.8331

[LostRuins]

Seems like an Intel GPU specific issue, i'll try to look into it

[LostRuins]

I don't have an Arc, tested on my intel igpu and it does seem to be marginally slower. Hoping someone can bisect to determine which commit caused it

[LostRuins]

@pqnet Could you try:

1. re-running your upstream benchmarks with -d 4096 to add 4096 ctx first
2. share the results of vulkaninfo --summary

[pqnet]

I don't have an Arc, tested on my intel igpu and it does seem to be marginally slower. Hoping someone can bisect to determine which commit caused it

this seems to be observable on AMD as well, (i have noticed it without benchmarks with 2 AMD cards as well), on slower cards it may be hard to observe because the bottleneck may not be memory speed anymore. The "ARC card" I'm testing on is basically a beefed up iGPU running on shared memory (the one integrated with intel ultra 9 first gen cpus) not the external one with its own memory (also branded arc)

[pqnet]

WARNING: [Loader Message] Code 0 : windows_read_data_files_in_registry: Registry lookup failed to get layer manifest files.
==========
VULKANINFO
==========

Vulkan Instance Version: 1.4.313


Instance Extensions: count = 13
-------------------------------
VK_EXT_debug_report                    : extension revision 10
VK_EXT_debug_utils                     : extension revision 2
VK_EXT_swapchain_colorspace            : extension revision 4
VK_KHR_device_group_creation           : extension revision 1
VK_KHR_external_fence_capabilities     : extension revision 1
VK_KHR_external_memory_capabilities    : extension revision 1
VK_KHR_external_semaphore_capabilities : extension revision 1
VK_KHR_get_physical_device_properties2 : extension revision 2
VK_KHR_get_surface_capabilities2       : extension revision 1
VK_KHR_portability_enumeration         : extension revision 1
VK_KHR_surface                         : extension revision 25
VK_KHR_win32_surface                   : extension revision 5
VK_LUNARG_direct_driver_loading        : extension revision 1

Instance Layers:
----------------

Devices:
========
GPU0:
        apiVersion         = 1.3.297
        driverVersion      = 101.6314
        vendorID           = 0x8086
        deviceID           = 0x7d55
        deviceType         = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
        deviceName         = Intel(R) Arc(TM) Graphics
        driverID           = DRIVER_ID_INTEL_PROPRIETARY_WINDOWS
        driverName         = Intel Corporation
        driverInfo         = 101.6314
        conformanceVersion = 1.3.6.3
        deviceUUID         = 8680557d-0800-0000-0002-000000000000
        driverUUID         = 33322e30-2e31-3031-2e36-333134000000

[pqnet]

llama.cpp version b7787:

model	size	params	backend	ngl	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	99	1	pp512 @ d4096	24.89 ± 0.02
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	99	1	tg128 @ d4096	11.47 ± 0.15

llama.cpp version b7757:

model	size	params	backend	ngl	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	99	1	pp512 @ d4096	25.23 ± 0.03
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	99	1	tg128 @ d4096	11.36 ± 0.11

I have no idea what -d does, documentation only says "depth"
prefill speed tanked, but decode speed is more or less unaffected