feat(server): expose swa_full_tokens_ratio as a startup CLI flag

[mkornreich]

What

Adds a --swa-full-tokens-ratio CLI flag that sets EngineConfig.swa_full_tokens_ratio at model load — the SWA radix cache's window/full pool ratio (and the DSV4 window-tier ratio). It could previously only be changed at runtime via POST /v1/cache/rebuild, so the window pool was always first sized at its 0.2 default and only resized afterward — which can't grow past a runtime rebuild's tighter double-buffer budget, and needs an out-of-band call after every start.

The flag mirrors --memory-ratio, validates to (0, 1] (matching the /v1/cache/rebuild check), and threads through unchanged via ServerArgs(**kwargs).

Why

On a sliding-window model, a shared prompt prefix longer than the window is reusable across requests only if its windowed KV is retained — which requires the window pool ≥ the prefix. When the pool is smaller than a fixed leading prompt, that prefix is never reused and every request re-prefills it. Sizing the window pool at startup (rather than a post-start rebuild) also keeps more full-pool capacity, since it isn't bounded by the rebuild's reallocation budget. The same ratio drives the DSV4 window tier, so this is not specific to SWA-radix models.

Tested on

Linux, NVIDIA RTX 5070 Laptop (8 GB), driver 595.84, CUDA 13.

Unit test (no GPU): tests/server/test_swa_full_tokens_ratio_flag.py — default resolves to the config value; an in-range value is accepted; out-of-range / non-numeric values are rejected at parse time (follows the parse_args mock pattern in test_parser_auto_selection.py).

Full-attention model on main — Qwen/Qwen3-0.6B (no window pool): the flag is accepted and applied (swa_full_tokens_ratio=0.3 in ServerArgs), the model loads and serves normally — a no-op, as expected.

ft serve --model-path Qwen3-0.6B --swa-full-tokens-ratio 0.3 --memory-ratio 0.82 \
         --max-running-requests 1 --cuda-graph-max-bs 1

SWA effect — gemma-4-E2B-it (q4_0 GGUF). The model support is my separate open PR #59, so this A/B is on main + #59 + this flag (the flag itself is model-family-independent). Request ≈ 55k tokens (a ~27k fixed leading prefix + ~28k body); a second, near-identical request measures prefix reuse:

swa_full_tokens_ratio	window pool	full pool	2nd-call cached tokens	2nd-call latency
0.2 (default = main behavior)	25,921	129,604	0	~39 s (full re-prefill)
0.3 (this flag, at load)	34,023	113,410	46,615	0.3 s

Cold first-call prefill is ~39 s either way; the flag changes whether the shared prefix is reused on later calls (the default pool is smaller than the 27k prefix, so it can't be retained).

ft serve --model-path <gemma-4-E2B q4_0 gguf> --swa-full-tokens-ratio 0.3 \
         --memory-ratio 0.82 --max-running-requests 1 --cuda-graph-max-bs 1

Notes

• One change, no behavior change unless the flag is passed (default is unchanged).
• Not a Roadmap item — happy to move to an issue / Slack first if the maintainers prefer.
• AI-assisted (disclosed in the commit trailer); authored, run, and verified on the hardware above.

[mkornreich]

This is a massive improvement for caching!