feat: add native Kimi-K3 inference support - #93
Open
earlvanze wants to merge 5 commits into
Open
Conversation
earlvanze
force-pushed
the
feat/kimi-k3-support
branch
from
August 24, 2026 02:38
0cb565f to
39320b5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #92.
What changed
latent-MoE routing, SiTU, and checkpoint weight mapping
inference-optimization/Kimi-K3-0.40B-MXFP4development checkpointtrust_remote_code=Trueand addsthe required
tiktokendependencycan support radix reuse safely
Validation
passed while CUDA was healthy
benchmark host lost GPU access
all seven routed-expert banks from revision
ef3902c318fb8e13c3507e26055656e687fdfe3832 simultaneous requests
Raw metrics, methodology, host/checkpoint metadata, and the disclosed host
failure are preserved in the
benchmark artifact.
The benchmark uses the 0.40B development checkpoint only as a functional test.
Its throughput is not presented as a proxy for the full 2.8T model. During a
sustained repetition, the live-swapped NVIDIA driver lost the passed-through
GPU; completed samples are retained, the interrupted sample is not reported,
and the failure is documented in the artifact.
Scope
snapshot support
This contribution was directed by the operator and implemented/tested with
Codex assistance. The implementation was manually checked against the official
configuration and checkpoint keys.
Full-checkpoint qualification evidence
The full
moonshotai/Kimi-K3checkpoint passed an end-to-end OpenAI-compatible qualification on 4× NVIDIA H200. The versioned record includes the exact server arguments, 21m30s cold start to first/v1/models, warmup and streaming measurements, and a 1/2/4/8/16-request concurrency sweep with 31/31 successful requests.docs/models.mdSupported Models entryThe branch was rebased on upstream
main(bd372b6). The Kimi, generic offload, GPT-OSS, and graph batch-size regression suite passes: 59 tests.