Native .NET LLM inference engine for GGUF models, including autoregressive LLMs and DiffusionGemma-style text-diffusion models. TensorSharp provides a console application, a web-based chatbot interface, and Ollama/OpenAI-compatible HTTP APIs for programmatic access.
Quick start in ~30 seconds on the verified native GGML fast path — Gemma 4 E4B. That covers copying and starting the commands; the model download and first .NET restore/build take longer and depend on your connection and machine.
Prerequisites: .NET 10 SDK, git, and curl, plus the toolchain for your selected GPU backend. Confirm that dotnet --version starts with 10.. See Prerequisites.
Gemma 4 E4B Q8_0 is the project's exercised dense multimodal Gemma 4 tier. On native GGML backends, its Per-Layer Embeddings (PLE) and shared-KV layout stay eligible for the fused whole-model prefill/verify and single-graph decode paths; the server also selects that fused per-sequence path automatically for a single active request. The recommended public file is gemma-4-E4B-it-Q8_0.gguf (8,031,240,160 bytes / 7.48 GiB). The text-only commands below exercise that fused path; the same model adds thinking, tools, and optional-projector image/video/audio support.
Run the block for your platform. Text-only inference does not need a projector.
git clone https://github.com/zhongkaifu/TensorSharp.git
Set-Location TensorSharp
New-Item -ItemType Directory -Force models | Out-Null
curl.exe -L --fail "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models\gemma-4-E4B-it-Q8_0.gguf
'Answer in one short sentence: what is TensorSharp?' | Set-Content prompt.txt
$env:TENSORSHARP_GGML_NATIVE_ENABLE_CUDA = 'ON'
dotnet run --project TensorSharp.Cli -c Release -p:TensorSharpSkipMlxNative=true -- --model models\gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 128 --backend ggml_cudagit clone https://github.com/zhongkaifu/TensorSharp.git
cd TensorSharp
mkdir -p models
curl -L --fail "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models/gemma-4-E4B-it-Q8_0.gguf
printf '%s\n' 'Answer in one short sentence: what is TensorSharp?' > prompt.txt
dotnet run --project TensorSharp.Cli -c Release -p:TensorSharpSkipMlxNative=true -- --model models/gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 128 --backend ggml_metalgit clone https://github.com/zhongkaifu/TensorSharp.git
cd TensorSharp
mkdir -p models
curl -L --fail "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models/gemma-4-E4B-it-Q8_0.gguf
printf '%s\n' 'Answer in one short sentence: what is TensorSharp?' > prompt.txt
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet run --project TensorSharp.Cli -c Release -p:TensorSharpSkipMlxNative=true -- --model models/gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 128 --backend ggml_cudaOn a supported Windows/Linux AMD, Intel, or NVIDIA Vulkan device, enable the native build with $env:TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN = 'ON' in PowerShell or by prefixing the Bash command with TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON, then use --backend ggml_vulkan; see Prerequisites for the required SDK/runtime. To host the same verified E4B path in the Web UI and APIs, run the command for your platform.
Windows + NVIDIA (PowerShell):
$env:TENSORSHARP_GGML_NATIVE_ENABLE_CUDA = 'ON'
dotnet run --project TensorSharp.Server -c Release -p:TensorSharpSkipMlxNative=true -- --model models\gemma-4-E4B-it-Q8_0.gguf --backend ggml_cuda --max-tokens 512macOS (Apple Silicon):
dotnet run --project TensorSharp.Server -c Release -p:TensorSharpSkipMlxNative=true -- --model models/gemma-4-E4B-it-Q8_0.gguf --backend ggml_metal --max-tokens 512Linux + NVIDIA:
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet run --project TensorSharp.Server -c Release -p:TensorSharpSkipMlxNative=true -- --model models/gemma-4-E4B-it-Q8_0.gguf --backend ggml_cuda --max-tokens 512Then open http://localhost:5000/index.html (GET / is the liveness response), or verify the hosted E4B model from a second terminal:
curl -s http://localhost:5000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"gemma-4-E4B-it-Q8_0.gguf","messages":[{"role":"user","content":"Say hello in five words."}],"max_tokens":32,"stream":false}'Windows PowerShell:
curl.exe -s http://localhost:5000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"gemma-4-E4B-it-Q8_0.gguf","messages":[{"role":"user","content":"Say hello in five words."}],"max_tokens":32,"stream":false}'The server binds 0.0.0.0:5000 and has no built-in authentication or TLS. Keep port 5000 behind a firewall, or use an authenticated HTTPS reverse proxy for remote access.
For image, video, or audio input, the recommended public companion is mmproj-gemma-4-E4B-it-Q8_0.gguf; download it and append --mmproj models/mmproj-gemma-4-E4B-it-Q8_0.gguf. The verified claim concerns the E4B Q8_0 architecture/tensor-layout path, while ggml-org supplies the recommended public artifacts. See the Gemma 4 architecture card for details and the engine comparison report for measured E4B Q8_0 runs.
The commands above run the normal GGML native build (with CUDA requested explicitly in the NVIDIA examples) and skip the unrelated MLX native build. You can instead configure the native backend once, run dotnet build TensorSharp.slnx -c Release, then invoke the built app from TensorSharp.Cli/bin/ or TensorSharp.Server/bin/. The CLI reads one-shot prompts from --input, samples greedily by default, and generates 100 tokens unless --max-tokens is set. Full options: CLI usage · Server usage.
Not sure which backend to use? Start here. Every backend falls back to CPU for any op it does not yet implement, so output stays correct on all of them.
| Your hardware | Recommended backend | Flag | Notes |
|---|---|---|---|
| Apple Silicon (Mac) | GGML Metal | --backend ggml_metal |
Default on macOS. --backend mlx is an alternative Apple-Silicon GPU path. |
| Windows / Linux + NVIDIA GPU | GGML CUDA | --backend ggml_cuda |
Most-tested NVIDIA path. --backend cuda is the direct PTX/cuBLAS backend for experimentation. |
| Windows / Linux + AMD / Intel / NVIDIA GPU | GGML Vulkan | --backend ggml_vulkan |
Vendor-neutral GPU path via ggml-vulkan (cooperative-matrix accelerated where the driver supports it). Built automatically when the machine has a Vulkan runtime; --no-vulkan opts out. |
| No GPU / portability / debugging | Pure C# CPU | --backend cpu |
No native dependencies. For faster CPU inference use --backend ggml_cpu (native kernels). |
See Compute Backends for the full description of every backend and what each one accelerates.
These architectures are implemented and exercised by the test/benchmark matrix. Pick a quantization that fits your hardware (e.g. Q4_K_M for low memory, Q8_0 for higher quality). More sizes and multimodal projector files are in Model Downloads.
| Family | Example model (GGUF) | Image / Video / Audio | Thinking | Tools | Card |
|---|---|---|---|---|---|
| Gemma 4 | gemma-4-E4B-it (also 31B, 26B-A4B MoE) | ✅ / ✅ / ✅ | ✅ | ✅ | gemma4.md |
| Qwen 3.5 / 3.6 | Qwen3.5-9B (also 35B-A3B MoE) | ✅ / — / — | ✅ | ✅ | qwen35.md |
| Qwen 3 | Qwen3-4B | — / — / — | ✅ | ✅ | qwen3.md |
| GPT OSS | gpt-oss-20b (MoE) | — / — / — | ✅ | ✅ | gptoss.md |
| Nemotron-H | Nemotron-H-8B (also 47B, Omni) | ✅ (Omni) / — / — | ✅ | ✅ | nemotron.md |
| Mistral 3 | Mistral-Small-3.1-24B | ✅ / — / — | — | — | mistral3.md |
| Gemma 3 | gemma-3-4b-it (the official google QAT repo is gated: HF login + Gemma license) | ✅ / — / — | — | — | gemma3.md |
| DiffusionGemma | diffusiongemma-26B-A4B-it | — / — / — | — | — | diffusiongemma.md |
| Qwen-Image-Edit | Qwen-Image-Edit-2511 (MMDiT + VAE + Qwen2.5-VL companions) | 🖼️ image→image | — | — | qwenimage.md |
- Trades wins with llama.cpp — from pure .NET — head-to-head on identical GGUF files and the same GPU, TensorSharp matches or beats the hand-tuned C++
llama.cppon the workloads that matter. In the project's GGML CUDA comparison run (reproducible viabenchmarks/engine_comparison): the Gemma 4 26B-A4B MoE prefills 1.32× faster and lands first tokens 1.30× sooner (geomean; up to 1.70× / 1.65× per scenario), Gemma 4 12B wins or ties every decode scenario (geomean 1.17×), streamed tool-call turns decode up to 2.37× faster, and structured-output (JSON) decode on Gemma 4 E4B streams 7.7× faster (405 vs 52 tok/s). → Head-to-head vs llama.cpp - Continuous batching & paged KV cache — vLLM-style paged KV pool with block-hash prefix sharing and an iteration-level scheduler, on by default in the server. → deep dive
- MTP / NextN speculative decoding — multi-token-prediction draft heads accelerate solo decode on Qwen 3.6 (NextN block embedded in the trunk GGUF) and Gemma 4 (separate
gemma4-assistantdraft GGUF). The draft proposes several tokens per step and the trunk verifies them in one batched forward, with the request's own sampler driving both. Opt in with the server's--mtp-specflag (+--mtp-draft-modelfor Gemma 4); the CLI has no MTP flags (use theTS_MTP_*env vars). → Speculative decoding - DiffusionGemma text diffusion — block-wise EntropyBound denoising over a Gemma-4-derived MoE backbone, with CLI generation flags and a Web UI denoising preview stream. → DiffusionGemma card
- Qwen-Image-Edit image editing — prompt + input image → edited image, driving the 60-block MMDiT diffusion transformer with a Qwen-Image VAE and a Qwen2.5-VL-7B text encoder. CUDA-graph-captured whole-DiT forward, FlowMatch-Euler true-CFG denoise, and live denoising previews in the Web UI. On the CUDA image-edit benchmark it beats stable-diffusion.cpp: a warm 4-step Lightning edit completes in 40.44 s vs 48.16 s (~1.19× faster). → Qwen-Image-Edit card
- Multimodal — image / video / audio inputs (Gemma 4); image inputs for Gemma 3, Qwen 3.5-family, Mistral 3, and Nemotron-H Omni. → Multimodal Support
- Tool calling / function calling — multi-turn tool calls across all three API styles, with architecture-agnostic output parsing. → Tool Calling
- Thinking / reasoning mode — structured chain-of-thought for Qwen 3, Qwen 3.5/3.6-family, Gemma 4, GPT OSS, and Nemotron-H. → Thinking Mode
- Ollama- & OpenAI-compatible APIs — drop-in endpoints for existing tooling, plus a browser chat UI. → HTTP APIs
- Native quantized compute — Q4_K_M / Q8_0 / MXFP4 / IQ2_XXS and more run in matmul without dequantizing to FP32.
Everything below is detailed reference. New here? The five sections above are all you need to get running.
| Start here | Use this when you want to... |
|---|---|
| Quick build and usage | Build the solution, compile the native GGML bridge, and run the CLI or server |
| Supported model architectures | Check which GGUF architecture keys, modalities, thinking mode, and tool calling paths are implemented |
| Compute backends | Choose between pure C# CPU, direct CUDA/cuBLAS, MLX Metal, GGML CPU, GGML Metal, GGML CUDA, and GGML Vulkan |
| HTTP APIs | Use the Ollama-compatible, OpenAI-compatible, or Web UI SSE endpoints |
| Per-model architecture cards | Read end-to-end documentation of one architecture (origin, forward graph, components, parameters, and how TensorSharp implements / optimizes prefill and decode) |
| Paged attention & continuous batching | Understand the vLLM-style paged KV cache, prefix sharing, and iteration-level scheduler |
| Environment variable feature matrix | See which high-impact runtime flags affect which models, backends, and prompt types |
| Test/benchmark matrix runner | Sweep model × backend × feature × env-var cells and generate regression reports |
| Server API examples | Copy complete curl and Python examples for the server surface |
| Server integration tests | Exercise the public API contract against a running server |
| Area | Status |
|---|---|
| Model families | Gemma 3/4, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family GGUFs (qwen35, qwen35moe, qwen3next), GPT OSS, Nemotron-H (incl. Nemotron 3 Nano Omni), and Mistral 3. Image editing via Qwen-Image-Edit (qwen_image MMDiT). |
| Inference hosts | CLI, interactive REPL, ASP.NET Core web UI, Ollama-style API, OpenAI Chat Completions-style API. DiffusionGemma currently uses the CLI diffusion run mode and Web UI denoising stream; Qwen-Image-Edit runs via the CLI image-edit mode and the Web UI image-edit flow. |
| Backends | Pure C# CPU, direct CUDA/cuBLAS (cuda), MLX Metal (mlx), GGML CPU, GGML Metal, GGML CUDA, GGML Vulkan |
| Multimodal | Gemma 4 image/video/audio; Gemma 3, Qwen 3.5-family, Mistral 3, and Nemotron-H Omni image input. PDF documents via Web UI upload and CLI --pdf: born-digital PDFs are text-extracted for any model, scanned PDFs become page images for vision-capable models |
| Continuous batching | vLLM-style paged KV cache, block-hash prefix sharing across requests, iteration-level scheduler (enabled by default; opt-out via --no-continuous-batching) |
| Speculative decoding | MTP / NextN draft heads for solo decode on Qwen 3.6 (embedded NextN) and Gemma 4 (separate gemma4-assistant draft GGUF); off by default, opt-in via the server's --mtp-spec (+ --mtp-draft-model for Gemma 4; CLI runs use the TS_MTP_* env vars). Profitable on ggml backends and the pure-C# cuda backend; CPU / MLX stay on standard decode. |
| Server model scope | One explicitly hosted GGUF via --model; optional explicit projector via --mmproj; no directory scanning |
| Observability | Structured per-turn logs, queue status, and KV-cache reuse metrics across Web UI, Ollama, and OpenAI response shapes |
| Test/eval harness | TensorSharp.TestMatrix sweeps supported hosts across model, backend, feature, and env-var cells, then compares against per-host baselines |
- Multi-architecture support -- Gemma 4, Gemma 3, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family, GPT OSS, Nemotron-H, Mistral 3, and Qwen-Image-Edit (image editing)
- Multimodal inference -- image, video, and audio inputs (Gemma 4); images for Gemma 3 / Qwen 3.5-family / Mistral 3 / Nemotron-H Omni
- Thinking / reasoning mode -- structured chain-of-thought output with
<think>/<|channel>thought/<|channel>analysistags (Qwen 3, Qwen 3.5/3.6-family, Gemma 4, GPT OSS, Nemotron-H) - Tool calling / function calling -- models can invoke user-defined tools; multi-turn tool-call conversations supported across all three API styles
- Quantized model support -- loads GGUF files with Q4_K_M, Q8_0, F16, MXFP4, and other quantization formats; performs native quantized matmul without dequantizing to FP32, including memory-efficient pure C# CPU loading for large GGUFs
- GPU-accelerated -- GGML Metal on macOS, GGML CUDA on Windows/Linux with NVIDIA GPUs, GGML Vulkan on Windows/Linux with AMD/Intel/NVIDIA GPUs, a direct CUDA/cuBLAS backend with PTX kernels, and an MLX backend for Apple Silicon (mlx-c / Metal), all with CPU fallbacks for unsupported ops
- Optimized pure C# CPU backend -- managed GEMM fast paths plus fused SIMD kernels for RMSNorm, RoPE, softmax, fused activations, and other inference hot paths
- Continuous batching & paged KV cache -- vLLM-style block-paged KV pool with block-hash prefix sharing across requests, iteration-level scheduler that admits / preempts sequences mid-batch, optional SSD-backed tier for very large KV working sets, and a native fused paged-attention kernel (
TSGgml_PagedAttentionForward) that drivesggml_flash_attn_exton Metal/CUDA/Vulkan. Enabled by default inTensorSharp.Server; opt-out with--no-continuous-batching. See docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING.md. - MTP / NextN speculative decoding -- multi-token-prediction draft heads accelerate solo (non-concurrent) decode. Qwen 3.6 ships its NextN block fused into the trunk GGUF; Gemma 4 loads a separate EAGLE-style
gemma4-assistantdraft GGUF via--mtp-draft-modelwhose draft layers attend the target's own KV cache. The draft proposes up to--mtp-drafttokens per step (kept while draft confidence ≥--mtp-pmin) and the trunk verifies them in a single batched forward; the request's own sampler — penalties included — drives both drafting and verification, so output is identical to standard decode. Opt in with the server's--mtp-specflag (off by default;TensorSharp.Clihas no MTP flags — set theTS_MTP_*env vars there). On ggml backends fused multi-token-verify / draft-step kernels make it a clear win; the pure-C#cudabackend runs a fully GPU-resident per-op verify/draft and is also a win. CPU / MLX stay on standard decode. Env:TS_MTP_*(shared) andTS_GMTP_*(Gemma 4 tuning). - Batched / parallel inference --
IBatchedPagedModel.ForwardBatchimplementations for Mistral 3, Gemma 4, GPT OSS, Qwen 3, Qwen 3.5/3.6-family, and Nemotron-H all run by default and pack N sequences into a single forward pass with paged K/V scatter and per-sequence attention via the native kernel. Gemma 4, Qwen 3.5/3.6, GPT OSS, and Nemotron-H expose a per-familyTS_<FAMILY>_BATCHED=0escape hatch (TS_GEMMA4_BATCHED=0,TS_QWEN35_BATCHED=0,TS_GPTOSS_BATCHED=0,TS_NEMOTRON_BATCHED=0) to fall back to the per-sequence KV-swap path for A/B comparison or regression isolation; Qwen 3 and Mistral 3 have no per-family switch — use the globalTS_SCHED_DISABLE_BATCHED=1. - Ollama & OpenAI API compatibility -- drop-in replacement endpoints for existing tooling
- Configurable sampling -- temperature, top-k, top-p, min-p, repetition/presence/frequency penalties, seed, stop sequences
- Chat templates -- auto-loaded from GGUF metadata (Jinja2), with hardcoded fallbacks per architecture
- Inference engine -- the new
InferenceEngine(worker-thread scheduler + paged block pool) replaces the legacy single-request FIFO queue insideTensorSharp.Server. The old queue object is now a compatibility shim for status/event shapes; the engine itself handles concurrency. - Batch processing -- JSONL input support in the console application, plus a built-in inference benchmark for prefill/decode throughput
- Streaming -- token-by-token output via SSE (web) or stdout (console), with abort/stop support for in-flight generations
- Text-diffusion generation -- DiffusionGemma uses an iterative EntropyBound denoising sampler instead of autoregressive
Forward(). The CLI exposes--diffusion-steps,--diffusion-seed, and--diffusion-blocks; the Web UI streams whole-messagereplaceevents for live denoising previews and batches concurrent diffusion requests throughDiffusionBatchScheduler. - Image editing (Qwen-Image-Edit) -- a prompt plus an input image produces an edited image. The loaded
qwen_imageGGUF is the MMDiT diffusion transformer; TensorSharp resolves two companion GGUFs alongside it — the Qwen-Image VAE (image ↔ 16-channel latent) and the Qwen2.5-VL-7B text encoder (prompt → 3584-dim conditioning, optional vision grounding via anmmproj). The pipeline VAE-encodes the reference, builds text (and optional image) conditioning, runs a FlowMatch-Euler true-CFG denoise loop with reference-latent concatenation, then VAE-decodes back to pixels. The whole 60-block DiT forward is CUDA-graph-captured (TSGgml_QwenImageForward), flash-attention is on by default, and the target area is auto-clamped to the device VRAM budget. An optional Lightning distillation LoRA (--qwen-image-lora/TS_QWEN_IMAGE_LORA,.safetensors) merges into the DiT weights at load time, cutting the denoise to the LoRA's step count (e.g. 4 or 8) and switching CFG to 1.0 (no negative pass). Driven from C# viaQwenImageModel.EditImage(prompt, RgbImage, QwenImageParams), from the CLI image-edit mode (--image,--prompt,--cfg,--diffusion-steps,--diffusion-seed), and from the Web UI with live denoising previews. → Qwen-Image-Edit card - Hybrid SSM-Transformer -- Nemotron-H mixes Mamba2 SSM layers, attention-only layers, and MoE FFN layers in a single model. The Mamba2 step has both a per-sequence native kernel and a batched native kernel (
TSGgml_NemotronMamba2BatchedStepF32, NEON SIMD + GCD parallelism) used by the batched path. - Hybrid Attention-Recurrent -- Qwen 3.5/3.6-family models mix full-attention layers with GatedDeltaNet recurrent layers; the batched path keeps recurrent running state in a per-slot recurrent-state pool
- Mixture of Experts -- Gemma 4 MoE variants (e.g. gemma-4-26B-A4B), GPT OSS MoE (e.g. gpt-oss-20b), Qwen 3.5/3.6-family MoE (
qwen35moe/qwen3nextvariants such as Qwen3.5-35B-A3B), and Nemotron-H MoE FFN layers - Batched GPU MoE -- a single fused GGML graph dispatch handles all selected experts (plus the optional shared expert and residual add) for Qwen 3.5/3.6-family and Nemotron-H decode, eliminating per-expert round-trips
- KV cache codecs -- pluggable codec interface (
IKvBlockCodec) with a built-in TurboQuant (2-bit affine / Q4 / Q8) compressed codec for paged blocks. The CLI accepts all four--paged-kv-quant-bits 0|2|4|8values; the server's legacy standalone flag accepts0|4|8, whileTS_KV_PAGED_QUANT_BITS=2selects the 2-bit codec directly. The 2-bit tier reaches ~10x compression on fp32 blocks for very long contexts. - Message editing -- edit or delete previous messages in the web chat UI and regenerate from that point
- Text/Image/Audio/Video/PDF uploads -- the web UI accepts file uploads up to 500 MB, with automatic token-budget-aware truncation for large text files. Born-digital PDFs have their text layer extracted and inlined into the prompt (token-budget truncated; cap pages with
TS_PDF_MAX_PAGES); scanned PDFs are rendered to page images for vision-capable models. The CLI accepts a PDF in one-shot mode via--pdf <file> - Per-turn observability -- structured logs capture the full user input and the full raw assistant output (both
<think>reasoning and the final result) plus the KV cache hit ratio. The same cache-hit stats are surfaced through every API:prompt_cache_hit_tokens/prompt_cache_hit_ratio(Ollama),usage.prompt_tokens_details.cached_tokens(OpenAI), andpromptTokens/kvReusedTokens/kvReusePercentin the Web UI SSEdoneevent
| Architecture | GGUF arch keys | Example Models | Multimodal | Thinking | Tool Calling | MTP spec | Card |
|---|---|---|---|---|---|---|---|
| Gemma 4 | gemma4 |
gemma-4-E4B, gemma-4-31B, gemma-4-26B-A4B (MoE) | Image, Video, Audio | Yes | Yes | Yes (separate gemma4-assistant draft GGUF) |
gemma4.md |
| Gemma 3 | gemma3 |
gemma-3-4b | Image | No | No | — | gemma3.md |
| Qwen 3 | qwen3 |
Qwen3-4B | Text only | Yes | Yes | — | qwen3.md |
| Qwen 3.5 / 3.6 family | qwen35, qwen35moe, qwen3next |
Qwen3.5-9B (hybrid Attn+Recurrent), Qwen3.5-35B-A3B / Qwen3.6-35B-A3B (MoE) | Image | Yes | Yes | Yes on Qwen 3.6 (embedded NextN block) | qwen35.md |
| GPT OSS | gptoss, gpt-oss |
gpt-oss-20b (MoE) | Text only | Yes (always) | Yes | — | gptoss.md |
| Nemotron-H | nemotron_h, nemotron_h_moe |
Nemotron-H-8B, Nemotron-H-47B (Hybrid SSM-Transformer, MoE), Nemotron 3 Nano Omni (image) | Image (Omni) | Yes | Yes | — | nemotron.md |
| Mistral 3 | mistral3 |
Mistral-Small-3.1-24B-Instruct | Image | No | No | — | mistral3.md |
| DiffusionGemma | diffusion-gemma, diffusion_gemma |
diffusion-gemma text-diffusion GGUFs | Text only | No | No | — | diffusiongemma.md |
| Qwen-Image-Edit | qwen_image |
qwen-image-edit MMDiT GGUFs (+ VAE & Qwen2.5-VL companions) | Image edit (image+text → image) | No | No | — | qwenimage.md |
See the per-model architecture cards for end-to-end documentation of each architecture (origin, forward graph, components, parameters, weight naming, and how TensorSharp implements / optimizes prefill and decode).
TensorSharp loads models in GGUF format. Below are verified Hugging Face repos for every supported architecture, including the multimodal-projector (mmproj) and MTP-draft companion files each family uses. Pick a quantization that fits your hardware (Q4_K_M / UD-Q4_K_XL for low memory, Q8_0 for higher quality, etc.).
| Architecture | Model | GGUF Download |
|---|---|---|
| Gemma 4 verified native tier | gemma-4-E4B-it Q8_0 | ggml-org/gemma-4-E4B-it-GGUF — recommended public artifact gemma-4-E4B-it-Q8_0.gguf; lower-memory Q4_K_M is also available; mmproj mmproj-gemma-4-E4B-it-Q8_0.gguf is in the same repo |
| Gemma 4 | gemma-4-12B-it (QAT) | unsloth/gemma-4-12B-it-qat-GGUF — mmproj mmproj-BF16.gguf and MTP draft mtp-gemma-4-12B-it.gguf in the same repo |
| Gemma 4 | gemma-4-26B-A4B-it (MoE, QAT) | unsloth/gemma-4-26B-A4B-it-qat-GGUF — mmproj mmproj-BF16.gguf and MTP draft mtp-gemma-4-26B-A4B-it.gguf in the same repo |
| Gemma 4 | gemma-4-26B-A4B-it (MoE) | ggml-org/gemma-4-26B-A4B-it-GGUF — mmproj files in the same repo |
| Gemma 4 | gemma-4-31B-it | ggml-org/gemma-4-31B-it-GGUF — mmproj files in the same repo |
| Gemma 4 | gemma4-assistant MTP drafts |
AtomicChat/gemma-4-E4B-it-assistant-GGUF (E4B) and AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF (26B-A4B) — load via the server's --mtp-spec --mtp-draft-model; pair each draft with its matching target size |
| Gemma 3 | gemma-3-4b-it | ggml-org/gemma-3-4b-it-GGUF — mmproj mmproj-model-f16.gguf in the same repo. The official QAT repo google/gemma-3-4b-it-qat-q4_0-gguf is gated (requires HF login + accepting Google's Gemma license) |
| Qwen 3 | Qwen3-4B | Qwen/Qwen3-4B-GGUF (text only — no companion files) |
| Qwen 3.5 / 3.6 family | Qwen3.5-9B | unsloth/Qwen3.5-9B-GGUF — mmproj mmproj-F16.gguf in the same repo |
| Qwen 3.5 / 3.6 family | Qwen3.5-35B-A3B (MoE) | ggml-org/Qwen3.5-35B-A3B-GGUF — mmproj mmproj-Qwen3.5-35B-A3B-Q8_0.gguf in the same repo |
| Qwen 3.5 / 3.6 family | Qwen3.6-35B-A3B (MoE, embedded NextN MTP) | unsloth/Qwen3.6-35B-A3B-MTP-GGUF — these GGUFs retain the NextN block for the server's --mtp-spec; mmproj mmproj-F16.gguf in the same repo. The base repo unsloth/Qwen3.6-35B-A3B-GGUF ships the same file names with NextN stripped — those load fine but silently fall back to standard decode |
| GPT OSS | gpt-oss-20b (MoE) | ggml-org/gpt-oss-20b-GGUF (gpt-oss-20b-mxfp4.gguf, text only) |
| Nemotron-H | Nemotron-H-8B-Reasoning-128K | bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF |
| Nemotron-H | Nemotron-H-47B-Reasoning-128K | bartowski/nvidia_Nemotron-H-47B-Reasoning-128K-GGUF |
| Nemotron-H | Nemotron 3 Nano Omni 30B-A3B (image-capable) | unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF — mmproj mmproj-BF16.gguf (same repo) is required for image input. Audio is preprocessed only: real audio inference needs a Parakeet audio mmproj these GGUFs do not ship |
| Mistral 3 | Mistral-Small-3.1-24B-Instruct-2503 | bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF — Pixtral mmproj mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf in the same repo |
| DiffusionGemma | diffusiongemma-26B-A4B-it | unsloth/diffusiongemma-26B-A4B-it-GGUF (general.architecture = diffusion-gemma) |
| Qwen-Image-Edit | MMDiT DiT (the --model GGUF) |
unsloth/Qwen-Image-Edit-2511-GGUF (e.g. qwen-image-edit-2511-Q4_K_M.gguf; general.architecture = qwen_image) |
| Qwen-Image-Edit | Qwen-Image VAE (required) | VAE/Qwen_Image-VAE.safetensors from QuantStack/Qwen-Image-Edit-GGUF — place next to the DiT or point --qwen-image-vae / TS_QWEN_IMAGE_VAE at it (the .safetensors VAE loads directly) |
| Qwen-Image-Edit | Qwen2.5-VL-7B text encoder (required) | unsloth/Qwen2.5-VL-7B-Instruct-GGUF — place next to the DiT or set --qwen-image-vl / TS_QWEN_IMAGE_TE |
| Qwen-Image-Edit | Vision mmproj (optional) | mmproj-BF16.gguf from unsloth/Qwen2.5-VL-7B-Instruct-GGUF — image-grounded conditioning via --qwen-image-mmproj / TS_QWEN_IMAGE_MMPROJ |
| Qwen-Image-Edit | Lightning LoRA (optional, 4/8-step) | lightx2v/Qwen-Image-Edit-2511-Lightning (Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors) — --qwen-image-lora / TS_QWEN_IMAGE_LORA; auto-switches to the LoRA's step count and CFG 1.0 |
The hf download commands below need the Hugging Face CLI (pip install -U huggingface_hub) and drop every file into ./models. Reminders that apply to all blocks: the CLI reads its one-shot prompt from a file via --input (--prompt is exclusively the Qwen-Image-Edit edit instruction), samples greedily by default, and generates only 100 tokens unless you raise --max-tokens; the server always listens on http://localhost:5000. Swap --backend ggml_cuda for the backend that fits your hardware (see Pick a Backend). Create a prompt file first:
echo "Give me three facts about the Moon." > prompt.txtGemma 4 — text + image/video/audio, thinking, tools, MTP (ggml-org/gemma-4-E4B-it-GGUF)
hf download ggml-org/gemma-4-E4B-it-GGUF gemma-4-E4B-it-Q8_0.gguf --local-dir models
hf download ggml-org/gemma-4-E4B-it-GGUF mmproj-gemma-4-E4B-it-Q8_0.gguf --local-dir models
hf download AtomicChat/gemma-4-E4B-it-assistant-GGUF gemma-4-E4B-it-assistant.Q8_0.gguf --local-dir models
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gemma-4-E4B-it-Q8_0.gguf --mmproj models/mmproj-gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gemma-4-E4B-it-Q8_0.gguf --mmproj models/mmproj-gemma-4-E4B-it-Q8_0.gguf --backend ggml_cuda --mtp-spec --mtp-draft-model models/gemma-4-E4B-it-assistant.Q8_0.gguf(The third download and the --mtp-spec --mtp-draft-model pair are optional — they enable MTP speculative decoding, a server-only feature.)
Gemma 3 — text + image (ggml-org/gemma-3-4b-it-GGUF; the official google/gemma-3-4b-it-qat-q4_0-gguf is gated: HF login + Gemma license)
hf download ggml-org/gemma-3-4b-it-GGUF gemma-3-4b-it-Q4_K_M.gguf --local-dir models
hf download ggml-org/gemma-3-4b-it-GGUF mmproj-model-f16.gguf --local-dir models
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --backend ggml_cudaQwen 3 — text, thinking, tools (Qwen/Qwen3-4B-GGUF)
hf download Qwen/Qwen3-4B-GGUF Qwen3-4B-Q4_K_M.gguf --local-dir models
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3-4B-Q4_K_M.gguf --input prompt.txt --max-tokens 300 --think --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3-4B-Q4_K_M.gguf --backend ggml_cudaQwen 3.5 / 3.6 family — text + image, thinking, tools, NextN MTP on 3.6 (unsloth/Qwen3.5-9B-GGUF)
hf download unsloth/Qwen3.5-9B-GGUF Qwen3.5-9B-UD-Q4_K_XL.gguf --local-dir models
hf download unsloth/Qwen3.5-9B-GGUF mmproj-F16.gguf --local-dir models
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf --backend ggml_cudaQwen 3.6 NextN speculative decoding (server-only; download from the -MTP- repo — base-repo GGUFs strip the NextN block and silently fall back to standard decode):
hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --local-dir models
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --backend ggml_cuda --mtp-specGPT OSS — text, thinking (always on), tools (ggml-org/gpt-oss-20b-GGUF)
hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-mxfp4.gguf --local-dir models
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gpt-oss-20b-mxfp4.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gpt-oss-20b-mxfp4.gguf --backend ggml_cudaNemotron-H — text, thinking, tools; image on the Omni distribution (bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF)
hf download bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --local-dir models
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --backend ggml_cudaFor image input use the Omni distribution instead: NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-UD-Q4_K_XL.gguf + mmproj-BF16.gguf from unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF. Audio is not functional (it needs a Parakeet audio mmproj the GGUFs do not ship).
Mistral 3 — text + image (Pixtral) (bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF)
hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf --local-dir models
hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --local-dir models
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --backend ggml_cudaDiffusionGemma — block text-diffusion (unsloth/diffusiongemma-26B-A4B-it-GGUF)
hf download unsloth/diffusiongemma-26B-A4B-it-GGUF diffusiongemma-26B-A4B-it-Q4_K_M.gguf --local-dir models
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --input prompt.txt --max-tokens 256 --diffusion-steps 48 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --backend ggml_cuda(The Web UI streams live denoising previews for DiffusionGemma; the compat APIs return the final text.)
Qwen-Image-Edit — image + prompt → edited image; needs the DiT + VAE + text encoder, Lightning LoRA optional (unsloth/Qwen-Image-Edit-2511-GGUF)
hf download unsloth/Qwen-Image-Edit-2511-GGUF qwen-image-edit-2511-Q4_K_M.gguf --local-dir models
hf download QuantStack/Qwen-Image-Edit-GGUF VAE/Qwen_Image-VAE.safetensors --local-dir models
hf download unsloth/Qwen2.5-VL-7B-Instruct-GGUF Qwen2.5-VL-7B-Instruct-UD-IQ2_XXS.gguf --local-dir models
hf download lightx2v/Qwen-Image-Edit-2511-Lightning Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors --local-dir models
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf --image input.png --prompt "Make the sky a dramatic sunset." --output edited.png --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-UD-IQ2_XXS.gguf --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-UD-IQ2_XXS.gguf --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors --backend ggml_cuda(In the Web UI, attach an image and type the edit instruction. The Lightning LoRA download and --qwen-image-lora flag are optional — they cut the denoise to 4 steps at CFG 1.0.)
| Backend | Flag | Best fit | Description |
|---|---|---|---|
| Direct CUDA/cuBLAS | --backend cuda |
NVIDIA inference and experimentation | Uses the CUDA Driver API, cuBLAS GEMM, PTX kernels for common float32 ops (fill, unary, binary, ternary, activations, RMSNorm, softmax, RoPE/RoPEEx, SDPA, GQA prefill/decode, causal mask, gather/concat), and native quantized matmul/get-rows for supported GGUF quant types. Unsupported ops route through CPU fallbacks while preserving tensor semantics. |
| MLX Metal | --backend mlx |
Apple Silicon (alternative to GGML Metal) | GPU-accelerated path built on mlx-c. Implements quantized ops (Q4_K_M, Q8_0, Q5_K, Q6_K, IQ2_XXS, IQ4_XS, IQ4_NL, MXFP4, etc.) without dequantizing to FP32, fused decode/prefill Metal kernels (fused QKV preprocess, fused gate+up+SiLUMul MoE, fused multi-dim KV write), compiled-graph kernels, async worker dispatch with periodic async_eval to overlap GPU/CPU work, batched MoE decode with stacked expert weight slabs, MoE expert offload, GGUF mmap pinned in physical RAM via mlock(2), host-derived allocator caps (TS_MLX_MEMORY_LIMIT_MB / TS_MLX_CACHE_LIMIT_MB / TS_MLX_WIRED_LIMIT_MB), and a CPU fallback for ops that aren't yet wired up. Requires libmlxc (built locally by TensorSharp.Backends.MLX/build-native-macos.sh or located via TENSORSHARP_MLX_LIBRARY / TENSORSHARP_MLX_LIBRARY_DIR). |
| GGML Metal | --backend ggml_metal |
Apple Silicon (default on macOS) | GPU-accelerated via Apple Metal. Quantized weights are mapped zero-copy from the GGUF file into Metal command buffers via host-pointer buffers, so the resident set stays close to the on-disk model size. |
| GGML CUDA | --backend ggml_cuda |
NVIDIA inference through ggml | GPU-accelerated via GGML CUDA on Windows or Linux. Quantized weights are uploaded to device memory once at load time and the host copy is released afterwards. |
| GGML Vulkan | --backend ggml_vulkan |
Vendor-neutral GPU inference through ggml | GPU-accelerated via GGML Vulkan on Windows or Linux — runs on AMD, Intel, and NVIDIA GPUs with a Vulkan 1.3 driver, using cooperative-matrix shaders (KHR coopmat / NV coopmat2) where the driver supports them. Weights are device-resident like GGML CUDA and the same fused whole-model decode/prefill graphs are used. Enabled automatically at native build time when the machine has a Vulkan runtime (loader installed); the build downloads a portable Vulkan toolchain (headers, glslc, SPIRV-Headers, and on Windows a loader import lib) via eng/fetch-vulkan-toolchain.ps1 / eng/fetch-vulkan-toolchain.sh when no Vulkan SDK or distro dev packages are installed. Opt out with --no-vulkan (or TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=OFF). |
| GGML CPU | --backend ggml_cpu |
Native CPU kernels | CPU inference using native GGML with optimized kernels. Quantized weights are mapped zero-copy from the GGUF file. |
| Pure C# CPU | --backend cpu |
Portability and debugging | Portable CPU inference with no native dependencies. |
TensorSharp/
├── TensorSharp.Core/ # Core tensor library (Tensor, Ops, memory, device abstraction, CPU SIMD/managed quantized kernels)
├── TensorSharp.Runtime/ # GGUF, tokenizers, templates, sampling, protocol parsing
│ ├── Paged/ # Paged KV cache primitives (BlockPool, BlockTable, KvBlock, BlockHashIndex, PagedKvStorage, PagedKvBatchOps, ManagedPagedAttention)
│ ├── Scheduling/ # Continuous batching engine (InferenceEngine, BatchExecutor, ContinuousBatchScheduler, SequenceState, SchedulerConfig/Output, InferenceRequestHandle) + MTP speculative decode core (MtpSpeculativeExecution)
│ ├── PagedKvCacheManager.cs # Per-session paged KV manager (block allocation, prefix reuse)
│ ├── PagedKvBlockStore.cs # On-disk / RAM-tiered paged block storage with optional SSD spillover
│ ├── SsdKvBlockTier.cs # SSD-backed cold tier for paged blocks
│ ├── TurboQuantKvCodec.cs # Quantized KV block codec (2-bit / Q4 / Q8) implementing IKvBlockCodec
│ ├── PrefillChunking.cs # Chunked-prefill helper used by SWA / very long prompts
│ ├── KvBlockHash.cs # Content-addressed block hash for prefix-cache sharing
│ └── Logging/ # JSON-line file logger + per-turn telemetry
├── TensorSharp.Models/ # Model architectures and multimodal encoders/injectors
│ ├── Models/<Family>/ # One folder per architecture (DiffusionGemma, Gemma3, Gemma4, GptOss, Mistral3, Nemotron, Qwen3, Qwen35, QwenImage)
│ │ ├── <Family>Model.cs # Legacy per-sequence ModelBase implementation
│ │ └── <Family>Model.BatchedForward.cs # IBatchedPagedModel.ForwardBatch — batched/paged path (Mistral3, Gemma4, GptOss, Qwen35, Nemotron, Qwen3)
│ ├── Paged/ # Tensor-side paged-attention helpers (TensorPagedAttention)
│ ├── KvBlockTransfer.cs # Helpers for extract/inject of KV blocks across sequences
│ ├── MtpSpeculativeDecoder.cs # MTP/NextN draft-verify-rollback driver shared by Qwen 3.6 and Gemma 4
│ └── ModelMultimodalInjector.cs # Vision / audio / video embedding injection
├── TensorSharp.Backends.GGML/ # GGML backend bindings (Metal/CUDA/Vulkan/CPU via native library)
├── TensorSharp.Backends.Cuda/ # Direct CUDA backend using CUDA Driver API, cuBLAS, and PTX kernels
├── TensorSharp.Backends.MLX/ # Apple Silicon MLX backend (mlx-c / Metal). Native bridge is built via `build-native-macos.sh`.
├── TensorSharp.GGML.Native/ # Native C++ bridge to ggml (builds libGgmlOps, split into focused source files)
│ ├── ggml_ops_core.cpp # Element-wise, reductions, basic shape ops
│ ├── ggml_ops_elementwise.cpp # Element-wise / activation fusions
│ ├── ggml_ops_matmul.cpp # GEMM / quantized matmul
│ ├── ggml_ops_fused.cpp # Cross-cutting fused per-layer kernels
│ ├── ggml_ops_norm_attn.cpp # Norm + attention fusions
│ ├── ggml_ops_transformer.cpp # Generic fused transformer layer/model decode + flash-attn decode
│ ├── ggml_ops_transformer_common.h # Shared transformer helpers + C# layer-descriptor structs
│ ├── ggml_ops_transformer_prefill.cpp # Fused layer prefill (Gemma 4, GPT-OSS, Qwen 3.5)
│ ├── ggml_ops_qwen35_decode.cpp # Qwen 3.5/3.6 fused decode (layer, whole-model, batched)
│ ├── ggml_ops_qwen35_verify.cpp # Qwen 3.5/3.6 fused multi-token verify
│ ├── ggml_ops_gemma4_decode.cpp # Gemma 4 dense whole-model decode (CUDA-graph persisted)
│ ├── ggml_ops_gemma4_batched.cpp # Gemma 4 dense + MoE token-batched decode
│ ├── ggml_ops_gemma4_verify.cpp # Gemma 4 dense verify + MTP draft step
│ ├── ggml_ops_gemma4_moe.cpp # Gemma 4 MoE layer/whole-model decode + verify
│ ├── ggml_ops_moe.cpp # Mixture-of-Experts forward / fused router
│ ├── ggml_ops_gated_delta_net.cpp # Qwen 3.5/3.6 GatedDeltaNet kernels (per-seq + batched)
│ ├── ggml_ops_mamba2.cpp # Nemotron Mamba2 kernels (per-seq + batched SIMD)
│ ├── ggml_ops_paged_attention.cpp # Paged-attention native kernel (drives ggml_flash_attn_ext + sinks variant)
│ ├── ggml_ops_diffusion.cpp # DiffusionGemma fused decode-layer / whole-model / lm-head kernels
│ ├── ggml_ops_qwen_image.cpp # Qwen-Image-Edit MMDiT whole-model forward (CUDA-graph-captured) + CFG-batched kernels
│ ├── ggml_ops_training.cpp # Training-only kernels (unused at runtime)
│ └── tests/ # Native unit + smoke tests
├── TensorSharp.Server/ # Web chatbot + API server (ASP.NET Core)
│ ├── Program.cs # Slim bootstrap: DI wiring, middleware, endpoint mapping, paged-KV + continuous-batching CLI translation
│ ├── ModelService.cs # Facade that keeps the public server inference API stable; owns the InferenceEngineHost
│ ├── ModelLifecycleService.cs # Model load/dispose and backend selection (CPU / CUDA / MLX / GGML CPU/Metal/CUDA/Vulkan)
│ ├── InferenceEngineHost.cs # DI-registered per-model InferenceEngine singleton (continuous batching entry point)
│ ├── ChatGenerationPipeline.cs # Prompt rendering, submits to InferenceEngine, streams tokens, stop handling
│ ├── InferenceTelemetry.cs # Prompt/eval timing, TTFT, tokens/sec, full input/output logs
│ ├── ChatHistoryPreparer.cs # History normalization, raw-token splice helpers, multimodal order helpers
│ ├── ChatSession.cs # Per-conversation tracked history + raw assistant tokens
│ ├── SessionManager.cs # Thread-safe session registry (default + per-tab sessions)
│ ├── InferenceQueue.cs # Backward-compatible queue-status surface (engine itself handles concurrency)
│ ├── BackendCatalog.cs # Discovery of available compute backends (CPU / CUDA / MLX / GGML*)
│ ├── TextUploadHelper.cs # Token-budget-aware text-file truncation
│ ├── WebUiChatPolicy.cs # Web UI chat request validation
│ ├── OpenAIResponseFormatParser.cs # OpenAI response_format (json_object / json_schema) parsing
│ ├── Hosting/ # Startup-time concerns: options builder (ServerOptionsBuilder), backend resolution, logging, web root, paged-KV / continuous-batching CLI translation
│ ├── RequestParsers/ # JSON request parsing (sampling, chat messages, tool functions)
│ ├── ResponseSerializers/ # Per-protocol response shape factories (Ollama, OpenAI, Web UI)
│ ├── StreamingWriters/ # SSE + NDJSON wire-format helpers
│ ├── ProtocolAdapters/ # Per-protocol request handlers (WebUiAdapter, OllamaAdapter, OpenAIChatAdapter)
│ ├── Endpoints/ # ASP.NET Core endpoint mapping (one extension method per protocol)
│ ├── Logging/ # Request logging middleware + low-noise path support
│ ├── wwwroot/index.html # Chat UI
│ ├── testdata/ # Integration test suites (bash + Python)
│ └── API_EXAMPLES.md # Detailed API documentation
├── TensorSharp.Cli/ # CLI application (one-shot generation, interactive REPL, batch JSONL, benchmarks)
├── TensorSharp.TestMatrix/ # Test / benchmark matrix runner, default prompts, env-var sweeps, and per-host baselines
├── InferenceWeb.Tests/ # xUnit unit tests covering ops, KV cache, paged scheduler, batched-model correctness, web/server helpers
├── AdvUtils/ # Utility library (logger)
├── docs/ # Developer reference
│ ├── models/ # Per-model architecture cards (one .md per model, EN + 中文)
│ ├── PAGED_ATTENTION_AND_CONTINUOUS_BATCHING.md # Paged KV cache, prefix sharing, scheduler, per-model batched-forward status
│ └── env_var_feature_matrix.md # Runtime flag × model/backend/feature coverage for TestMatrix
├── benchmarks/ # Reproducible benchmark harnesses
└── ExternalProjects/ # ggml/ is cloned from github.com/ggml-org/ggml at build time (not committed)
The repository is split along package boundaries so consumers can depend on only the layers they actually need. These are buildable package projects and IDs, but the current Runtime/Models/Backends/CLI/Server packages are not published on NuGet.org. Use project references from a source checkout for now; do not copy dotnet add package TensorSharp.Models examples until a matching version appears on NuGet.org.
| Project | NuGet package | Public namespace | Responsibility |
|---|---|---|---|
TensorSharp.Core |
TensorSharp.Core |
TensorSharp |
Tensor primitives, ops, allocators, storage, and device abstraction |
TensorSharp.Runtime |
TensorSharp.Runtime |
TensorSharp.Runtime |
GGUF parsing, tokenizers, prompt rendering, sampling, output protocol parsing, paged KV cache, continuous-batching scheduler |
TensorSharp.Models |
TensorSharp.Models |
TensorSharp.Models |
ModelBase, architecture implementations, multimodal encoders, batched / paged forward passes, and model-side execution helpers |
TensorSharp.Backends.GGML |
TensorSharp.Backends.GGML |
TensorSharp.GGML |
GGML-backed execution and native interop |
TensorSharp.Backends.Cuda |
TensorSharp.Backends.Cuda |
TensorSharp.Cuda |
Direct CUDA allocator, storage, cuBLAS GEMM, PTX kernels, and quantized CUDA ops |
TensorSharp.Backends.MLX |
TensorSharp.Backends.MLX |
TensorSharp.MLX |
Apple Silicon MLX backend (mlx-c / Metal) with quantized / fused / compiled kernels and MoE expert offload |
TensorSharp.Server |
TensorSharp.Server |
TensorSharp.Server |
ASP.NET Core server, OpenAI/Ollama adapters, inference engine host, web UI |
TensorSharp.Cli |
TensorSharp.Cli |
TensorSharp.Cli |
Console host and debugging / batch tooling |
This split keeps engine users off the web stack, keeps API-layer changes from leaking into core/runtime packages, and makes future benchmark or eval-harness projects easier to publish independently.
Validate package metadata and README dependency boundaries before publishing:
pwsh ./eng/verify-packages.ps1The verifier runs dotnet pack for the public packages above and fails if an internal dependency such as AdvUtils leaks into the .nuspec, or if a TensorSharp package depends on a layer outside this table.
The Publish NuGet workflow is configured to pack the public projects above on a version tag. A NuGet.org push occurs only when the repository has a valid NUGET_API_KEY; otherwise that step is skipped. This describes the release process, not current package availability:
git tag vX.Y.Z.W # the tag drives package version X.Y.Z.W
git push origin vX.Y.Z.W- The tag (with the leading
vstripped) overridesTensorSharpVersionfor every package, so all packages ship with a single coordinated version. You do not need to editDirectory.Build.propsfirst. - Packing is managed-only — the native GGML/CUDA/MLX libraries are not embedded in the packages — so the workflow runs on a stock runner with
eng/verify-packages.ps1 -SkipNativeBuild(which also setsTensorSharpSkipGgmlNative=true/TensorSharpSkipMlxNative=true). - Configure a
NUGET_API_KEYrepository secret for the NuGet.org push. If it is missing, the NuGet.org step is skipped with a warning and only the GitHub Packages push (which uses the built-inGITHUB_TOKEN) runs. - To rehearse without publishing, run the workflow manually (
workflow_dispatch) with aversioninput anddry_runchecked — it packs, verifies, and uploads the.nupkgfiles as a build artifact without pushing.
The Release Binaries workflow is intended to build self-contained archives of TensorSharp.Server and TensorSharp.Cli with the .NET 10 runtime and native libraries. However, the current latest release, v3.0.5.0, has no uploaded application archives (only GitHub's automatic source downloads), so users must currently build from source. Do not construct an archive URL from the names below without first confirming that the file is listed on the Releases page.
When a release workflow completes successfully, its intended archive matrix is:
| Archive suffix | Native backend(s) bundled | Format |
|---|---|---|
win-x64-cpu |
GGML CPU | .zip |
win-x64-cuda |
GGML CUDA + pure-C# CUDA (PTX) + CUDA 12.x runtime | .zip |
linux-x64-cpu |
GGML CPU | .tar.gz |
linux-x64-cuda |
GGML CUDA + pure-C# CUDA (PTX) + CUDA 12.x runtime | .tar.gz |
osx-arm64 |
GGML Metal + MLX | .tar.gz |
- Pushing a
v*tag triggers the archive and NuGet workflows; publication is conditional on every required job succeeding. - The
-cudaarchives bundle the CUDA runtime libraries (cudart/cublas/cublasLt) but still require an NVIDIA GPU and a compatible driver at run time; the-cpuarchives run anywhere. The macOS archive requires Apple Silicon. - To rehearse, run the workflow manually (
workflow_dispatch) with aversioninput — it builds every platform and creates a draft Release. Override the target GPU architectures for the CUDA build with thecuda_archinput.
- .NET 10 SDK
gitand network access: the GGML/CUDA native builds clone the ggml sources from github.com/ggml-org/ggml intoExternalProjects/ggml/on first build (seeeng/fetch-ggml.sh/eng/fetch-ggml.ps1). The clone tracks ggml's default branch (master); pin a different ref withTENSORSHARP_GGML_GIT_REF, or setTENSORSHARP_GGML_NO_UPDATE=1to skip the network update once cloned (offline rebuilds)- macOS (Metal backend): CMake 3.20+ and Xcode command-line tools for building the native GGML library; the MLX backend additionally builds
libmlxcfromTensorSharp.Backends.MLX/Native/viabash TensorSharp.Backends.MLX/build-native-macos.sh - Windows (GGML CPU / CUDA backends): CMake 3.20+ and Visual Studio 2022 C++ build tools; for
ggml_cudaorcuda, install an NVIDIA driver plus CUDA Toolkit 12.x or another compatible CUDA toolkit with cuBLAS - Linux (GGML CPU / CUDA backends): CMake 3.20+; for
ggml_cudaorcuda, install an NVIDIA driver plus CUDA Toolkit 12.x or another compatible CUDA toolkit with cuBLAS - Windows (GGML Vulkan backend): enabled automatically when the machine has a Vulkan runtime (
System32\vulkan-1.dll, shipped by every recent GPU driver). With a LunarG Vulkan SDK installed it is used directly; without one the build auto-provisions a portable toolchain (Vulkan-Headers, a vulkan-1 import library generated from the system loader, glslc, SPIRV-Headers) intoExternalProjects/vulkan-toolchain/viaeng/fetch-vulkan-toolchain.ps1. Opt out withbuild-windows.ps1 --no-vulkanorTENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=OFF. A GPU driver with Vulkan 1.3 support is required at runtime - Linux (GGML Vulkan backend): enabled automatically when a Vulkan loader (
libvulkan.so.1) is installed. Distro dev packages are used when present (apt install libvulkan-dev glslc spirv-headers); otherwise the build auto-provisions the missing pieces (Vulkan-Headers, glslc from the shaderc CI prebuilts, SPIRV-Headers) intoExternalProjects/vulkan-toolchain/viaeng/fetch-vulkan-toolchain.sh. Opt out withbuild-linux.sh --no-vulkanorTENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=OFF - GGUF model files (e.g., from Hugging Face)
dotnet build TensorSharp.slnx# Console application
dotnet build TensorSharp.Cli/TensorSharp.Cli.csproj
# Web application
dotnet build TensorSharp.Server/TensorSharp.Server.csprojThe native library is built automatically during the first dotnet build if it doesn't exist. To build it manually:
cd TensorSharp.GGML.NativemacOS:
bash build-macos.shLinux (CPU-only):
bash build-linux.shLinux (GGML_CUDA enabled):
bash build-linux.sh --cudaWindows (CPU-only):
.\build-windows.ps1 --no-cudaWindows (GGML_CUDA enabled):
.\build-windows.ps1 --cudaOn Windows and Linux, the native build script auto-detects the visible NVIDIA GPU compute capability and passes a narrow CMAKE_CUDA_ARCHITECTURES value to ggml-cuda (for example 86-real on an RTX 3080), which cuts CUDA build time substantially. The native build also runs in parallel by default with a conservative job cap so nvcc does not overwhelm typical developer machines.
If you want to override the auto-detected architecture list or the default build parallelism, use either environment variables or explicit build flags:
TENSORSHARP_GGML_NATIVE_CUDA_ARCHITECTURES='86-real;89-real' bash build-linux.sh --cuda
bash build-linux.sh --cuda --cuda-arch='86-real;89-real'
TENSORSHARP_GGML_NATIVE_BUILD_PARALLEL_LEVEL=2 bash build-linux.sh --cuda$env:TENSORSHARP_GGML_NATIVE_CUDA_ARCHITECTURES='86-real;89-real'; .\build-windows.ps1 --cuda
.\build-windows.ps1 --cuda --cuda-arch='86-real;89-real'
$env:TENSORSHARP_GGML_NATIVE_BUILD_PARALLEL_LEVEL=2; .\build-windows.ps1 --cudaYou can also request a CUDA-enabled native build from dotnet build:
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.Cli/TensorSharp.Cli.csproj -c Release$env:TENSORSHARP_GGML_NATIVE_ENABLE_CUDA='ON'; dotnet build TensorSharp.Cli/TensorSharp.Cli.csproj -c ReleaseOn macOS this compiles libGgmlOps.dylib with Metal GPU support. On Windows and Linux, the native scripts preserve an existing CUDA-enabled build and auto-enable GGML_CUDA when a CUDA toolchain is detected; build-windows.ps1 --cuda, build-linux.sh --cuda, and TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON force CUDA explicitly. The GGML Vulkan backend is auto-enabled the same way when the machine has a Vulkan runtime, downloading its build toolchain on first use; --vulkan / --no-vulkan or TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON/OFF force the choice explicitly, and an explicit choice sticks across rebuilds (see Prerequisites for the Vulkan toolchain the build auto-provisions). The build output is automatically copied to the application's output directory.
The direct cuda backend is built as managed C# plus PTX kernels. During dotnet build, TensorSharp.Backends.Cuda compiles native/kernels/*.cu to native/ptx/*.ptx when nvcc is available; if nvcc is missing, the build continues and PTX-backed ops use CPU fallbacks. cuBLAS-backed GEMM still requires the CUDA runtime libraries to be discoverable at run time.
The MLX backend depends on libmlxc (the C bindings for MLX). The repository pins a known-good tag of mlx-c in TensorSharp.Backends.MLX/Native/MLX_C_VERSION and a helper script fetches and builds it:
bash TensorSharp.Backends.MLX/build-native-macos.shThe script writes the resulting libraries (libmlxc.dylib, libmlx.dylib, and any backend deps) into TensorSharp.Backends.MLX/Native/dist/. At run time the backend probes the application directory first; you can also point it to a custom install with TENSORSHARP_MLX_LIBRARY=<path-to-libmlxc.dylib> or TENSORSHARP_MLX_LIBRARY_DIR=<dir-with-libmlxc>. If the library cannot be located the backend reports unavailable and --backend mlx is rejected at startup.
# Text inference
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --input prompt.txt --output result.txt \
--max-tokens 200 --backend ggml_metal
# Text inference on Windows/Linux + NVIDIA GPU
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --input prompt.txt --output result.txt \
--max-tokens 200 --backend ggml_cuda
# Interactive turn-by-turn chat (REPL) with KV cache reuse and slash commands
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --backend ggml_metal --interactive
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --backend ggml_metal -i \
--system "You are a terse assistant." --temperature 0.7 --top-p 0.9 --think
# Image inference (Gemma 3/4, Qwen 3.5-family)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --image photo.png --backend ggml_metal
# Video inference (Gemma 4)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --video clip.mp4 --backend ggml_metal
# Audio inference (Gemma 4)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --audio speech.wav --backend ggml_metal
# PDF document input: born-digital PDFs are text-extracted and inlined into the
# prompt; scanned PDFs become page images and need a vision model (--mmproj or a
# built-in vision encoder). --input provides the instruction over the document.
echo "Summarize the key findings of this paper." > question.txt
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --pdf paper.pdf --input question.txt \
--max-tokens 300 --backend ggml_metal
# DiffusionGemma text-diffusion generation
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <diffusion-gemma.gguf> --input prompt.txt --backend ggml_metal \
--max-tokens 256 --diffusion-steps 48 --diffusion-seed 0
# Qwen-Image-Edit image editing (prompt + input image -> edited image)
# The VAE + Qwen2.5-VL text-encoder companions are resolved next to the DiT GGUF
# (or set --qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj).
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <qwen-image-edit-DiT.gguf> --image input.png \
--prompt "Make the sky a dramatic sunset." --output edited.png \
--backend ggml_cuda --diffusion-steps 30 --cfg 2.5 --diffusion-seed 0
# Thinking / reasoning mode
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --input prompt.txt --backend ggml_metal --think
# Tool calling
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --input prompt.txt --backend ggml_metal \
--tools tools.json
# With sampling parameters
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --input prompt.txt --backend ggml_metal \
--temperature 0.7 --top-p 0.9 --top-k 40 --repeat-penalty 1.2 --seed 42
# Batch processing (JSONL)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --input-jsonl requests.jsonl \
--output results.txt --backend ggml_metal
# Multi-turn chat simulation with KV-cache reuse (mirrors the web UI behavior)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --multi-turn-jsonl chat.jsonl \
--backend ggml_metal --max-tokens 200
# Throughput benchmark: best-of-N prefill and decode timing
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --backend ggml_metal \
--benchmark --bench-prefill 256 --bench-decode 128 --bench-runs 3
# KV-cache reuse benchmark: measure prefill speedup across multiple chat turns
# (compares with-cache vs forced-reset prefill latency for an 8-turn conversation)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --backend ggml_metal \
--bench-kvcache --bench-kv-turns 4 --max-tokens 64
# Inspect the rendered prompt and tokenization without running inference
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --input prompt.txt --dump-prompt
# Compare hardcoded fallback templates against GGUF Jinja2 templates for every
# *.gguf file in a directory (useful when adding new architectures)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --test-templates ~/modelsCommand-line options:
| Option | Description |
|---|---|
--model <path> |
Path to a GGUF model file (required) |
--input <path> |
Text file containing the user prompt |
--input-jsonl <path> |
JSONL file with batch requests (one JSON per line) |
--multi-turn-jsonl <path> |
JSONL file for multi-turn chat simulation with KV cache reuse |
--output <path> |
Write generated text to this file |
--image <path> |
Image file for vision inference |
--video <path> |
Video file for video inference |
--audio <path> |
Audio file (WAV, MP3, OGG) for audio inference |
--pdf <path> |
PDF document input (one-shot mode). Born-digital PDFs have their text layer extracted and inlined into the prompt (token-budget truncated; page cap via TS_PDF_MAX_PAGES); scanned PDFs are rasterized to page images and require a vision model (--mmproj or a built-in vision encoder). --input text becomes the instruction over the document. |
--mmproj <path> |
Path to the multimodal projector GGUF file |
--max-tokens <N> |
Maximum tokens to generate (default: 100) |
--backend <type> |
Compute backend: cpu, cuda, mlx, ggml_cpu, ggml_metal, ggml_cuda, or ggml_vulkan |
--gpu-device <N> |
Vulkan device index for the ggml_vulkan backend on multi-GPU hosts (e.g. an integrated Intel GPU next to a discrete NVIDIA one). Defaults to device 0; use --list-gpus to see the indices. Also settable via the TS_GGML_VULKAN_DEVICE env var. |
--list-gpus |
List the Vulkan devices ggml-vulkan can see (index + adapter name) and exit |
--kv-cache-dtype <type> |
KV cache precision: f32, f16, q8_0, or q4_0 (default: auto — the backend/model pick; env KV_CACHE_DTYPE). Half-precision / quantized KV caches reduce memory at the cost of small numerical drift; q4_0 (~0.56 bytes/elem, ~1/7 of f32) is the most aggressive tier for very long (128K–256K) contexts where the KV cache dominates memory. Block-quantized caches (q8_0/q4_0) require the native GGML flash path. |
--interactive / -i |
Start an interactive REPL chat session (turn-by-turn input/output) with KV cache reuse, slash commands, hot-swappable model/backend/projector, file attachments (image, audio, video, text) and live sampling tuning. See the Interactive REPL commands section below for the full list. |
--system <text> |
System prompt to seed the interactive session (overridden inside the REPL by /system) |
--system-file <path> |
Read the initial system prompt from a UTF-8 text file (alternative to --system) |
--think |
Enable thinking/reasoning mode (chain-of-thought) |
--tools <path> |
JSON file with tool/function definitions |
--temperature <f> |
Sampling temperature (0 = greedy) |
--top-k <N> |
Top-K filtering (0 = disabled) |
--top-p <f> |
Nucleus sampling threshold (1.0 = disabled) |
--min-p <f> |
Minimum probability filtering (0 = disabled) |
--repeat-penalty <f> |
Repetition penalty (1.0 = none) |
--presence-penalty <f> |
Presence penalty (0 = disabled) |
--frequency-penalty <f> |
Frequency penalty (0 = disabled) |
--seed <N> |
Random seed (-1 = non-deterministic) |
--stop <string> |
Stop sequence (can be repeated) |
--dump-prompt |
Render the prompt + tokenization and exit (no generation) |
--benchmark |
Run a synthetic prefill/decode throughput benchmark |
--bench-prefill <N> |
Synthetic prefill length in tokens (default: 32) |
--bench-decode <N> |
Synthetic decode length in tokens (default: 64) |
--bench-runs <N> |
Number of benchmark runs; reports best and average (default: 1) |
--bench-kvcache |
Run a multi-turn KV-cache reuse benchmark (with-cache vs forced-reset prefill) |
--bench-kv-turns <N> |
Number of conversation turns for --bench-kvcache (default: 4, max: 8) |
--bench-chunked |
Run a chunked-prefill micro-benchmark (Gemma 4) |
--warmup-runs <N> |
Number of throw-away forward passes before timing real text / multimodal prompts (default: 0) |
--test-chunked-prefill |
Run the chunked-prefill correctness check (compares chunked vs non-chunked logits) |
--correct-prefill <N> |
Prompt length used by --test-chunked-prefill |
--correct-decode <N> |
Decode length used by --test-chunked-prefill |
--diffusion-steps <N> |
DiffusionGemma denoising steps per block (default: 48). For Qwen-Image-Edit, the FlowMatch-Euler step count — omit for auto (30, or the step count of a loaded Lightning LoRA). |
--diffusion-seed <N> |
DiffusionGemma deterministic sampler seed (default: 0) |
--diffusion-blocks <N> |
DiffusionGemma block-autoregressive canvas count. 0 derives the count from --max-tokens and the model canvas length. |
--image <path> |
Input image for Qwen-Image-Edit (also the image input for multimodal chat). Required to trigger image-edit mode on a qwen_image DiT GGUF. |
--prompt <text> |
Qwen-Image-Edit edit instruction (falls back to --input file contents if omitted). |
--output <path> |
Qwen-Image-Edit output PNG path (default: edited.png). |
--cfg <F> |
Qwen-Image-Edit true-CFG guidance scale (<= 1 disables the negative pass). Omit for auto: 2.5 (the Qwen-Image-Edit-2511 recommendation; 4.0 over-guides and distorts faces), or 1.0 when a Lightning LoRA is loaded. Shares --diffusion-steps / --diffusion-seed for step count and seed. |
--qwen-image-vae <path> |
Override the resolved Qwen-Image VAE companion (.gguf or .safetensors). |
--qwen-image-vl <path> |
Override the resolved Qwen2.5-VL-7B text-encoder GGUF. |
--qwen-image-mmproj <path> |
Override the resolved Qwen2.5-VL mmproj (vision grounding) GGUF. |
--qwen-image-lora <path> |
Qwen-Image-Edit Lightning distillation LoRA (.safetensors), merged into the DiT at load time. Auto-derives the step count (e.g. 4 or 8) and switches CFG to 1.0. Env: TS_QWEN_IMAGE_LORA. |
--test |
Run built-in tokenizer + Qwen3 chat-template + ollama-comparison tests |
--test-templates <dir> |
Validate hardcoded chat templates against GGUF Jinja2 templates for every *.gguf in <dir> |
--log-level <lvl> |
Console + file logger level: trace, debug, info, warning, error, critical, off |
--log-dir <path> |
Directory for the JSON-line file logger (default: <binDir>/logs) |
--log-file <0|1> |
Disable (0) or enable (1) the file logger (default: enabled) |
--log-console <0|1> |
Disable (0) or enable (1) the console logger (default: enabled) |
The CLI recognizes a small set of legacy projector filenames beside the model, but current repositories often use different names. Pass the downloaded file explicitly with --mmproj for reliable multimodal runs. TensorSharp.Server never auto-detects the projector.
JSONL input format:
Each line is a JSON object with messages, optional prompt, and optional sampling parameters:
{"id": "q1", "messages": [{"role": "user", "content": "What is 2+3?"}], "max_tokens": 50}
{"id": "q2", "messages": [{"role": "user", "content": "Write a haiku."}], "max_tokens": 100, "temperature": 0.8}Interactive REPL commands:
Once the CLI is launched with --interactive / -i, you can drive the running session with slash commands. Type /help (or /?) inside the REPL for the same list. Anything that does not start with / is treated as a user turn.
The prompt header summarizes the current state on every turn — model, backend, architecture, context length, projector, conversation depth, and any attachments queued for the next turn (e.g. [turn 3 (2 attachments pending)]> ). Press Ctrl+C while generating to interrupt the current reply; press Ctrl+C at the prompt to exit.
Conversation:
| Command | Description |
|---|---|
/help, /? |
Show all interactive commands |
/exit, /quit |
Leave the session |
/reset, /new |
Clear conversation history and KV cache |
/history |
Print the conversation history |
/save <file> |
Append the current transcript to a UTF-8 file |
/system <text> |
Set the system prompt (empty argument clears it). Resets KV cache. |
/think on|off |
Toggle thinking/reasoning mode for supported models |
/multiline on|off |
Toggle multi-line input (terminate the message with a single . on its own line) |
Model and runtime:
| Command | Description |
|---|---|
/info, /status |
Show the loaded model, backend, architecture, context/vocab size, projector, conversation depth, and pending attachments |
/model <path> |
Load a different .gguf model on the current backend (resets the session) |
/backend <name> |
Reload the current model on a different backend: cpu, cuda, mlx, ggml_cpu, ggml_metal, ggml_cuda, or ggml_vulkan |
/mmproj <path> |
Load (or replace) the multimodal projector for the current model. Aliases: /projector |
Sampling (live, persists across turns):
| Command | Description |
|---|---|
/sampling, /show |
Print the current sampling configuration |
/max <N> |
Maximum reply length in tokens |
/temp <float> |
Sampling temperature (0 = greedy) |
/topk <int> |
Top-K filtering (0 = disabled) |
/topp <float> |
Top-P / nucleus threshold (1.0 = disabled) |
/minp <float> |
Min-P filtering (0 = disabled) |
/repeat <float> |
Repetition penalty (1.0 = none) |
/presence <float> |
Presence penalty |
/frequency <float> |
Frequency penalty |
/seed <int> |
Random seed (-1 = non-deterministic) |
/stop <text> |
Add a stop sequence |
/clearstop |
Remove all stop sequences |
Uploads (queued for the next user turn, then auto-cleared after the turn):
| Command | Description |
|---|---|
/image <path>, /img <path> |
Attach an image (vision-capable models only) |
/audio <path> |
Attach an audio file (Gemma 4) |
/video <path>, /vid <path> |
Attach a video; frames are extracted automatically (Gemma 4) |
/text <path>, /file <path>, /txt <path> |
Inline a UTF-8 text/markdown/csv/code file into the next prompt (large files are token-budget truncated) |
/clearattach |
Drop any pending image/audio/video/text attachments without sending a turn |
Quoted paths (single or double quotes) are accepted, so drag-and-drop from a file manager works on macOS. Multimodal commands require a multimodal projector to be loaded — pass --mmproj at startup or use /mmproj <path> from the REPL.
Run these commands from the repository root after building:
# Start the server with the exact hosted model
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --backend ggml_metal
# Linux + NVIDIA GPU
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --backend ggml_cuda
# Multimodal models: host an explicit projector too
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --mmproj ./models/mmproj.gguf --backend ggml_cuda
# Configure server-wide default sampling parameters
# (used whenever a request does not override the value itself)
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --backend ggml_metal \
--temperature 0.7 --top-p 0.9 --top-k 40 --repeat-penalty 1.1 \
--presence-penalty 0.0 --frequency-penalty 0.0 --seed 42 \
--stop "</s>" --stop "<|endoftext|>"Open http://localhost:5000/index.html in your browser (GET / is the liveness endpoint). The web interface supports:
- Multi-turn chat conversations
- Per-tab chat sessions: each browser tab owns its own tracked conversation history; KV blocks are owned by the inference engine
- A single hosted GGUF selected explicitly with
--model - An explicit hosted multimodal projector via
--mmprojwhen needed - Image, video, and audio uploads for multimodal inference (up to 500 MB)
- Thinking/reasoning mode toggle
- Tool calling with function definitions
- Streaming token generation via Server-Sent Events
- DiffusionGemma denoising previews when a
diffusion-gemmaGGUF is hosted (the UI replaces the whole assistant message on each denoising step, then emits the final answer) - Backward-compatible queue-status events (the engine itself handles concurrency)
- Message editing and deletion with regeneration from any point in the conversation
- Free scrolling: scroll up to read earlier replies while new tokens stream in; the chat auto-scrolls again as soon as the user scrolls back to the bottom
Use --model to choose the hosted GGUF file and --mmproj to choose the hosted projector. TensorSharp.Server no longer scans a MODEL_DIR.
Server command-line options:
Running TensorSharp.Server with no arguments prints the full parameter reference (description, default, and an example per option) and exits; --help does the same. Pass --model at startup for inference. Other options can start a model-less status process, but /api/models/load cannot select a GGUF that was not supplied at startup.
| Option | Description |
|---|---|
--model <path> |
GGUF file to host (required for inference; when other options are passed without it, the server starts but /api/models/load will report no hosted model) |
--mmproj <path> |
Multimodal projector GGUF (resolved relative to the model directory when only a filename is given; pass none to disable). Requires --model. |
--backend <type> |
Default compute backend: cpu, cuda, mlx, ggml_cpu, ggml_metal, ggml_cuda, or ggml_vulkan |
--gpu-device <N> |
Vulkan device index for the ggml_vulkan backend on multi-GPU hosts (e.g. an integrated Intel GPU next to a discrete NVIDIA one). Defaults to device 0; use --list-gpus to see the indices. Also settable via the TS_GGML_VULKAN_DEVICE env var. |
--list-gpus |
List the Vulkan devices ggml-vulkan can see (index + adapter name) and exit |
--help |
Print the parameter reference (also shown when the server is started with no arguments) and exit |
--max-tokens <N> |
Default maximum tokens to generate when a request omits the limit (default: 20000) |
--temperature <f> |
Default sampling temperature when a request does not provide one (0 = greedy) |
--top-k <N> |
Default top-K filtering when a request does not provide one (0 = disabled) |
--top-p <f> |
Default nucleus sampling threshold when a request does not provide one (1.0 = disabled) |
--min-p <f> |
Default min-p filtering when a request does not provide one (0 = disabled) |
--repeat-penalty <f> |
Default repetition penalty when a request does not provide one (1.0 = none) |
--presence-penalty <f> |
Default presence penalty when a request does not provide one (0 = disabled) |
--frequency-penalty <f> |
Default frequency penalty when a request does not provide one (0 = disabled) |
--seed <N> |
Default random seed when a request does not provide one (-1 = non-deterministic) |
--stop <string> |
Default stop sequence (can be repeated). Per-request stop/stop_sequences fully replace the default list rather than merge with it. |
--kv-cache-dtype <type> |
KV cache precision for the hosted model: f32, f16, q8_0, or q4_0 (quantized caches trade small numerical drift for memory; see the CLI table above for the tier trade-offs). Default: auto — the backend/model pick. Env: KV_CACHE_DTYPE. |
--continuous-batching / --no-continuous-batching |
Enable (default) or disable iteration-level paged-batching. When enabled the server admits / preempts sequences mid-batch and packs them into one forward pass on models that implement IBatchedPagedModel. --no-continuous-batching falls back to per-sequence KV-swap for every model. Alias: --paged-batching / --no-paged-batching. |
--prefill-chunk-size <N> |
Chunked-prefill granularity under contention — the maximum prefill tokens scheduled per step while other requests are running, so parallel decodes get frequent turns at the GPU (default: 1024). Env: TS_SCHED_PREFILL_CHUNK. |
--mtp-spec / --no-mtp-spec |
Enable NextN/MTP speculative decoding (default off) on models that ship a multi-token-prediction draft head (Qwen 3.6's embedded NextN block, or a Gemma 4 gemma4-assistant draft loaded via --mtp-draft-model). Engages for solo (non-concurrent) sequences: the draft head proposes up to --mtp-draft tokens per step and the trunk verifies them in one batched forward, with the request's own sampler (penalties included) driving both drafting and verification, so output matches standard decode. Engaged automatically only where profitable (ggml backends and the pure-C# cuda backend); CPU / MLX serve standard decode. Env: TS_MTP_SPEC. |
--mtp-draft <N> |
Maximum tokens drafted per speculative step (default 8). Env: TS_MTP_DRAFT. |
--mtp-pmin <f> |
Minimum draft-head confidence in (0, 1] for a drafted token to be kept; drafting stops at the first low-confidence token (default 0.75). Env: TS_MTP_PMIN. |
--mtp-draft-model <path> |
Path to a separate MTP draft GGUF for architectures whose draft head ships as its own file (Gemma 4's gemma4-assistant). The draft's hidden size must match the target (e.g. pair the 12B target with its 12B draft, not the 26B-A4B draft); a mismatched or incomplete draft fails fast at startup with a remediation hint. Ignored for Qwen 3.6, which embeds its NextN block in the trunk GGUF. Env: TS_MTP_DRAFT_MODEL. |
--paged-kv / --no-paged-kv |
Legacy compatibility flags for the removed per-session paged-KV manager. Current server KV state is engine-owned; use continuous-batching / TS_SCHED_* knobs for the engine. Aliases: --paged-kv-cache / --no-paged-kv-cache. |
--paged-kv-block-size <N> |
Legacy standalone paged-KV block size. The current server engine uses TS_SCHED_BLOCK_SIZE. |
--paged-kv-ram-mb <N> |
Legacy standalone paged-KV RAM-tier cap. |
--paged-kv-ssd-dir <dir> |
Legacy standalone paged-KV SSD cold-tier directory. |
--paged-kv-ssd-mb <N> |
Legacy standalone paged-KV SSD cap. |
--paged-kv-quant-bits <0|4|8> |
Legacy standalone paged-KV block quantization accepted by the server (4/8 = symmetric). The runtime env var also accepts 2 for affine min+scale, and the CLI accepts 0|2|4|8. |
Per-request fields in the chat / generate JSON payloads (e.g. temperature,
top_p, top_k, min_p, repeat_penalty, presence_penalty,
frequency_penalty, seed, stop/stop_sequences) always win over these
server-wide defaults; the defaults only fill in fields the client omits.
Runtime environment variables:
| Variable | Description |
|---|---|
BACKEND |
Default compute backend (cpu, cuda, mlx, ggml_cpu, ggml_metal, ggml_cuda, or ggml_vulkan), used when --backend is not passed (default: ggml_metal on macOS, ggml_cpu elsewhere) |
MAX_TOKENS |
Default maximum generation length when neither --max-tokens nor a request-level limit is set (default: 20000) |
MAX_TEXT_FILE_CHARS |
Character cap used to truncate plain-text uploads when no tokenizer is available (default: 8000) |
VIDEO_SAMPLE_FPS |
Frames sampled per second of video for video prompts; time-based extraction (default: 1) |
VIDEO_MAX_FRAMES |
Optional upper bound on extracted video frames (evenly down-sampled); unset/0 means no cap (default: no cap) |
PORT / ASPNETCORE_URLS |
Currently overridden by the fixed http://0.0.0.0:5000 listener in Program.cs; Docker Space images rewrite that constant with APP_PORT at build time. |
TENSORSHARP_TEMPERATURE |
Default sampling temperature when neither --temperature nor the request body sets one |
TENSORSHARP_TOP_K |
Default top-K when neither --top-k nor the request body sets one |
TENSORSHARP_TOP_P |
Default top-P when neither --top-p nor the request body sets one |
TENSORSHARP_MIN_P |
Default min-P when neither --min-p nor the request body sets one |
TENSORSHARP_REPEAT_PENALTY |
Default repetition penalty when neither --repeat-penalty nor the request body sets one |
TENSORSHARP_PRESENCE_PENALTY |
Default presence penalty when neither --presence-penalty nor the request body sets one |
TENSORSHARP_FREQUENCY_PENALTY |
Default frequency penalty when neither --frequency-penalty nor the request body sets one |
TENSORSHARP_SEED |
Default random seed when neither --seed nor the request body sets one |
TENSORSHARP_LOG_LEVEL |
Minimum log level for both console and file loggers: Trace, Debug, Information, Warning, Error, Critical (default: Information). Also honored by TensorSharp.Cli. |
TENSORSHARP_LOG_DIR |
Directory the JSON-line file logger writes to (default: <binDir>/logs). Also honored by TensorSharp.Cli. |
TENSORSHARP_LOG_FILE |
Set to 0 to disable the file logger and keep only the console output (default: enabled). Also honored by TensorSharp.Cli. |
DIFFUSION_STEPS |
Server-side DiffusionGemma denoising steps per block (default: 48; CLI equivalent is --diffusion-steps) |
DIFFUSION_MAX_BATCH |
Maximum concurrent DiffusionGemma requests batched by the Web UI diffusion scheduler (default: 2) |
Paged KV cache & continuous-batching tunables (read at process / model start)
These can be set with either the --paged-kv* / --continuous-batching CLI flags (which translate to the env vars below) or directly via the environment:
| Variable | Description |
|---|---|
TS_KV_PAGED_CACHE |
Legacy compatibility switch for the standalone PagedKvCacheManager; current TensorSharp.Server request KV state is engine-owned. The CLI shortcuts are --paged-kv / --no-paged-kv. |
TS_KV_BLOCK_SIZE |
Legacy standalone paged-KV block size. The engine uses TS_SCHED_BLOCK_SIZE. |
TS_KV_CACHE_MAX_RAM_MB |
Legacy standalone paged-KV RAM-tier cap. |
TS_KV_CACHE_SSD_DIR |
Legacy standalone paged-KV SSD cold-tier directory. |
TS_KV_CACHE_MAX_SSD_MB |
Legacy standalone paged-KV SSD cap. |
TS_KV_PAGED_QUANT_BITS |
Legacy standalone paged-KV block quantization bits (0 = passthrough, 2 = affine, 4, or 8). |
TS_SCHED_DISABLE_BATCHED |
1 forces the per-sequence KV-swap fallback even when a model implements IBatchedPagedModel. The CLI shortcut is --no-continuous-batching. |
TS_SCHED_MAX_BATCHED_TOKENS |
Scheduler per-step token budget (default: 4096). |
TS_SCHED_MAX_RUNNING_SEQS |
Maximum in-flight sequences (default: 16). |
TS_SCHED_PREFILL_CHUNK |
Maximum prefill tokens per step when requests contend (default: 1024). |
TS_SCHED_SOLO_PREFILL_CHUNK |
Prefill chunk size for the fresh (start_pos = 0) part of a SOLO prompt — one uncontended request gets big fused-prefill chunks (default: 8192). |
TS_SCHED_NUM_BLOCKS |
Physical blocks in the engine block pool (default: 256). |
TS_SCHED_BLOCK_SIZE |
Tokens per block on the engine side (default: 256). |
TS_SCHED_PREFIX_CACHE |
0 disables block-hash prefix sharing across requests. |
TS_SCHED_DECODE_QUANTUM |
Tokens before a sequence-switch is allowed (default: block size). |
TS_QWEN35_BATCHED |
Set to 0 to force the Qwen 3.5/3.6 family onto the legacy per-sequence KV-swap path (default: batched/paged). Also implicitly disabled by --no-continuous-batching. |
TS_QWEN35_BATCHED_GDN_NATIVE |
Use the native batched GatedDeltaNet kernel inside Qwen 3.5/3.6 batched path. |
TS_GEMMA4_BATCHED |
Set to 0 to force Gemma 4 onto the legacy per-sequence KV-swap path (default: batched/paged). |
TS_GPTOSS_BATCHED |
Set to 0 to force GPT OSS onto the legacy per-sequence KV-swap path (default: batched/paged). |
TS_GPTOSS_PAGED_ATTN_MANAGED |
Use the managed (C#) paged-attention-with-sinks kernel inside GPT OSS batched path. |
TS_NEMOTRON_BATCHED |
Set to 0 to force Nemotron-H onto the legacy per-sequence KV-swap path (default: batched/paged). |
TS_NEMOTRON_MAMBA2_BATCHED_NATIVE |
Use the native Mamba2 batched step kernel inside Nemotron-H batched path. |
TS_PAGED_ATTN_KERNEL |
Paged-attention dispatch kernel for Mistral3Model.BatchedForward: native (default), tensor (C# Tensor-based), or managed (pure C# scalar). |
TS_MLX_PIPELINED_DECODE |
1 (default) enables pipelined greedy decode on the MLX backend when the request is greedy, has no stop sequences, and the model supports device-side argmax / next-embedding lookup. Set to 0 to disable. CLI only. |
TS_MLX_MLOCK_GGUF |
1 (default) pins the GGUF mmap region in physical RAM via mlock(2) so model weights stay resident between forward passes. Set to 0 to skip (use if the process memlock rlimit is too low or you want the OS to manage paging). MLX backend only. |
TS_MLX_FUSED_KV_WRITE |
1 (default) uses a single multi-dim slice_update to write the per-token KV block. Set to 0 to revert to the per-head loop (A/B testing / regression isolation). |
TS_MLX_BATCHED_MOE_DECODE |
1 (default) collapses K per-expert decode dispatches to one batched dispatch per (gate/up/down) kind for Qwen 3.5/3.6 MoE. Set to 0 on memory-constrained machines (saves ~weight-doubling overhead from the stacked weight slabs). |
TS_MLX_MOE_FUSED_GATE_UP_SILU |
1 (default) fuses gate matmul + up matmul + SiLUMul into one Metal kernel for batched MoE decode. Set to 0 to A/B against the legacy 3-dispatch path. |
TS_MLX_DEVICE_ROUTER |
1 (default) keeps MoE router top-K + softmax on device to skip ~60 host syncs/token on Qwen 3.6-35B-A3B. Set to 0 to disable; the code also falls back automatically when prerequisites are missing. |
TS_MLX_LOG_MEMORY_POLICY |
1 (default) prints once-per-load MLX memory-policy lines (wired limit, GGUF mlock status, allocator caps). Set to 0 to silence. |
TS_MLX_MEMORY_LIMIT_MB / TS_MLX_CACHE_LIMIT_MB / TS_MLX_WIRED_LIMIT_MB |
Override the MLX allocator hard cap / unused-buffer cache cap / wired-buffer residency cap (megabytes). Defaults are derived from the host's unified-memory capacity. |
TS_MLX_EVAL_EVERY_N_LAYERS / TS_MLX_GEMMA4_EVAL_EVERY_N_LAYERS |
Periodic mlx_async_eval cadence during decode to overlap GPU work with host queueing. Gemma 4 defaults to every 4 layers via TS_MLX_GEMMA4_EVAL_EVERY_N_LAYERS; Qwen 3 / Qwen 3.5 / Nemotron-H default to every 16 layers via TS_MLX_EVAL_EVERY_N_LAYERS. Set to 0 to disable where supported. |
TENSORSHARP_MLX_LIBRARY / TENSORSHARP_MLX_LIBRARY_DIR |
Override the search path for libmlxc when using --backend mlx. |
MTP / speculative-decoding tunables
These gate the optional multi-token-prediction speculative decode path (see MTP / NextN speculative decoding). TS_MTP_* are the shared knobs (also set by the --mtp-* CLI flags); TS_GMTP_* are Gemma 4 draft-path A/B switches.
| Variable | Description |
|---|---|
TS_MTP_SPEC |
1 enables MTP/NextN speculative decoding for solo sequences (default 0). CLI: --mtp-spec / --no-mtp-spec. |
TS_MTP_DRAFT |
Maximum tokens drafted per speculative step (default 8). CLI: --mtp-draft. |
TS_MTP_PMIN |
Minimum draft-head confidence in (0, 1] to keep a drafted token (default 0.75). CLI: --mtp-pmin. |
TS_MTP_DRAFT_MODEL |
Path to the separate Gemma 4 gemma4-assistant draft GGUF. CLI: --mtp-draft-model. Ignored by Qwen 3.6 (embedded NextN). |
TS_GMTP_NO_FUSED |
1 disables the Gemma 4 fused multi-token-verify / draft-step GGML kernels and falls back to the per-op path (A/B testing on ggml backends). |
TS_GMTP_NO_FAST_ROLLBACK |
1 restores the kept-prefix rollback path instead of the dense exact-match fast rollback used on partial draft acceptance. |
TS_GMTP_BATCHED_TRUNK |
1 opts the Gemma 4 verify trunk back into the batched paged path; the default runs the faster linear trunk for solo speculation. |
DiffusionGemma-specific tunables
| Variable | Description |
|---|---|
DIFFUSION_NO_SC |
Set to 1 to disable self-conditioning. Enabled by default. |
DIFFUSION_SC_TOPK |
Experimental self-conditioning top-K cutoff (default: 32). |
DIFFUSION_NO_PKV |
Set to 1 to disable prompt-KV caching on device-glue backends. Enabled by default where supported. |
DIFFUSION_NO_FUSED_DECODE |
Set to 1 to disable the GGML fused model decode path and fall back to per-op / per-layer diffusion decode. |
DIFFUSION_NO_FUSED_LMHEAD_TAIL |
Set to 1 to disable the fused output-norm + lm-head + softcap tail. |
DIFFUSION_BATCHED_FORWARD |
Set to 1 to use true batched DecodeCanvasBatched for active diffusion canvases; default time-slices the faster fused single-canvas path. |
DIFFUSION_LMHEAD_BATCH_CAP_MB |
Memory cap for batched diffusion lm-head logits before falling back to per-sequence lm-head (default: 300). |
DIFFUSION_PROFILE / DIFFUSION_STEPTIME / DIFFUSION_FUSED_DEBUG |
Development diagnostics for per-section diffusion timing and fused-kernel debug logging. |
Sampling parameter precedence (highest wins):
- Per-request JSON fields in the API call (e.g.
temperature,top_p,stop). - Server-wide CLI flags (e.g.
--temperature,--top-p,--stop). TENSORSHARP_*environment variables listed above.- Built-in
SamplingConfigdefaults (temperature=1.0,top_k=0,top_p=1.0,min_p=0,repeat_penalty=1.0, presence/frequency penalties0,seed=-1, no stop sequences).
Quick reference for which environment variables (and matching CLI flags) gate each major feature. Variables in bold are required to turn the feature on; everything else is a tunable for a feature that's already enabled by default.
| Feature | Default | Env vars | CLI equivalent |
|---|---|---|---|
Continuous-batching engine (InferenceEngine + scheduler) |
ON in TensorSharp.Server |
TS_SCHED_DISABLE_BATCHED=1 to force per-seq fallback |
--no-continuous-batching / --continuous-batching |
| Legacy per-session paged-KV manager | removed from Server request path | TS_KV_PAGED_CACHE (0 / 1), TS_KV_BLOCK_SIZE retained for compatibility / standalone tests |
--paged-kv / --no-paged-kv, --paged-kv-block-size N |
| Legacy paged-KV SSD spillover (standalone manager) | OFF | TS_KV_CACHE_MAX_RAM_MB, TS_KV_CACHE_SSD_DIR, TS_KV_CACHE_MAX_SSD_MB |
--paged-kv-ram-mb, --paged-kv-ssd-dir, --paged-kv-ssd-mb |
| Legacy paged-KV block quantization (standalone manager) | OFF (0 = passthrough) |
TS_KV_PAGED_QUANT_BITS (0 / 2 / 4 / 8) |
--paged-kv-quant-bits |
| Block-hash prefix sharing across requests | ON | TS_SCHED_PREFIX_CACHE=0 to disable |
— |
| Scheduler tunables (per-step token budget, max in-flight seqs, prefill chunks, block pool size, decode quantum) | engine defaults | TS_SCHED_MAX_BATCHED_TOKENS, TS_SCHED_MAX_RUNNING_SEQS, TS_SCHED_PREFILL_CHUNK, TS_SCHED_SOLO_PREFILL_CHUNK, TS_SCHED_NUM_BLOCKS, TS_SCHED_BLOCK_SIZE, TS_SCHED_DECODE_QUANTUM |
— |
| Model | Default state | Env var to flip default | Native-kernel sub-toggle |
|---|---|---|---|
| Mistral 3 | ON | — | TS_PAGED_ATTN_KERNEL = native (default) / tensor / managed |
| Gemma 4 | ON | TS_GEMMA4_BATCHED=0 to force legacy per-seq |
— |
| Qwen 3 | ON (reference port) | — | — |
| Qwen 3.5 / 3.6 family | ON | TS_QWEN35_BATCHED=0 to force legacy per-seq (or --no-continuous-batching) |
TS_QWEN35_BATCHED_GDN_NATIVE=1 enables native batched GDN kernel; FUSED_ATTN_LAYER_MIN_SEQ_LEN=N overrides fused-attention engage threshold (default 4096) |
| GPT OSS | ON | TS_GPTOSS_BATCHED=0 to force legacy per-seq |
TS_GPTOSS_PAGED_ATTN_MANAGED=1 forces the managed (C#) sinks softmax instead of the native paged-attention-with-sinks kernel |
| Nemotron-H | ON | TS_NEMOTRON_BATCHED=0 to force legacy per-seq |
TS_NEMOTRON_MAMBA2_BATCHED_NATIVE=1 enables the native batched Mamba2 step (NEON SIMD + GCD parallelism) |
| Gemma 3 | not implemented (per-seq fallback) | — | — |
| DiffusionGemma | Separate diffusion scheduler in the Web UI path; not an IBatchedPagedModel autoregressive path |
DIFFUSION_MAX_BATCH, DIFFUSION_STEPS |
DIFFUSION_BATCHED_FORWARD=1 enables true batched canvas decode; fused GGML decode is on by default unless disabled with DIFFUSION_NO_FUSED_DECODE=1 |
| Feature | Default | Env vars | CLI equivalent |
|---|---|---|---|
| Speculative decode engine (solo sequences) | OFF | TS_MTP_SPEC=1 |
--mtp-spec / --no-mtp-spec |
| Max tokens drafted per step | 8 |
TS_MTP_DRAFT |
--mtp-draft N |
| Min draft-head confidence to keep a token | 0.75 |
TS_MTP_PMIN |
--mtp-pmin X |
Gemma 4 separate draft GGUF (gemma4-assistant) |
none | TS_MTP_DRAFT_MODEL |
--mtp-draft-model <path> |
| Gemma 4 fused verify / draft kernels (ggml) | ON | TS_GMTP_NO_FUSED=1 falls back to per-op |
— |
| Gemma 4 dense fast rollback on partial accept | ON | TS_GMTP_NO_FAST_ROLLBACK=1 restores kept-prefix rollback |
— |
| Gemma 4 verify trunk path | linear (solo) | TS_GMTP_BATCHED_TRUNK=1 runs the batched paged trunk |
— |
| Feature | Default | Env vars | CLI equivalent |
|---|---|---|---|
| Default compute backend | ggml_metal (macOS), ggml_cpu (Windows/Linux) |
BACKEND |
--backend |
| MLX backend library lookup | probe app dir | TENSORSHARP_MLX_LIBRARY (full path to libmlxc), TENSORSHARP_MLX_LIBRARY_DIR (directory) |
— |
| MLX pipelined greedy decode (CLI only) | ON when eligible | TS_MLX_PIPELINED_DECODE=0 disables |
— |
MLX mlock(2) of GGUF mmap so weights stay resident |
ON | TS_MLX_MLOCK_GGUF=0 to disable |
— |
MLX fused multi-dim KV write (single slice_update per cache block) |
ON | TS_MLX_FUSED_KV_WRITE=0 to revert to per-head loop |
— |
| MLX batched MoE decode (Qwen 3.5/3.6 MoE) | ON | TS_MLX_BATCHED_MOE_DECODE=0 for legacy per-expert path |
— |
| MLX fused MoE gate+up+SiLUMul Metal kernel | ON | TS_MLX_MOE_FUSED_GATE_UP_SILU=0 for legacy 3-dispatch |
— |
| MLX on-device MoE router top-K + softmax | ON when prerequisites are met | TS_MLX_DEVICE_ROUTER=0 disables |
— |
MLX layer-boundary async_eval cadence |
Gemma 4: every 4 layers; Qwen / Nemotron: every 16 layers | TS_MLX_GEMMA4_EVAL_EVERY_N_LAYERS=N or TS_MLX_EVAL_EVERY_N_LAYERS=N (0 = disabled where supported) |
— |
| MLX allocator caps (memory / cache / wired buffer) | host-derived | TS_MLX_MEMORY_LIMIT_MB, TS_MLX_CACHE_LIMIT_MB, TS_MLX_WIRED_LIMIT_MB |
— |
| MLX one-line memory-policy banners at load | ON | TS_MLX_LOG_MEMORY_POLICY=0 to silence |
— |
These fill in fields the request body omits; per-request JSON always wins, CLI flags win over env vars.
| Sampling field | Env var | CLI equivalent |
|---|---|---|
temperature |
TENSORSHARP_TEMPERATURE |
--temperature |
top_k |
TENSORSHARP_TOP_K |
--top-k |
top_p |
TENSORSHARP_TOP_P |
--top-p |
min_p |
TENSORSHARP_MIN_P |
--min-p |
repeat_penalty |
TENSORSHARP_REPEAT_PENALTY |
--repeat-penalty |
presence_penalty |
TENSORSHARP_PRESENCE_PENALTY |
--presence-penalty |
frequency_penalty |
TENSORSHARP_FREQUENCY_PENALTY |
--frequency-penalty |
seed |
TENSORSHARP_SEED |
--seed |
| max tokens | MAX_TOKENS |
--max-tokens |
| stop sequences | — (CLI / per-request only) | --stop (repeatable) |
| Feature | Default | Env vars |
|---|---|---|
| ASP.NET Core listener | http://0.0.0.0:5000 |
Fixed in Program.cs; Docker Space images rewrite it with the APP_PORT build arg |
| Plain-text upload character cap (when no tokenizer available) | 8000 chars | MAX_TEXT_FILE_CHARS |
| Video-frame extraction | 1 fps (time-based, no cap) | VIDEO_SAMPLE_FPS, VIDEO_MAX_FRAMES |
| DiffusionGemma Web UI denoising | 48 steps, max batch 2 | DIFFUSION_STEPS, DIFFUSION_MAX_BATCH |
| Feature | Default | Env vars | CLI equivalent |
|---|---|---|---|
| Console + file log minimum level | Information |
TENSORSHARP_LOG_LEVEL |
--log-level |
| File logger output directory | <binDir>/logs |
TENSORSHARP_LOG_DIR |
--log-dir |
| File logger enabled | ON | TENSORSHARP_LOG_FILE=0 to disable |
--log-file 0|1 |
| Console logger enabled | ON | — | --log-console 0|1 (CLI only) |
These are read by build-linux.sh / build-windows.ps1 / the auto-build during dotnet build for TensorSharp.GGML.Native, not at run time.
| Feature | Default | Env vars | Build-script flag |
|---|---|---|---|
| Enable GGML CUDA in the native build | auto-detected from toolchain | TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON |
--cuda / --no-cuda |
| Enable GGML Vulkan in the native build | auto-detected from the installed Vulkan runtime; a portable toolchain (headers, glslc, SPIRV-Headers) is downloaded when no Vulkan SDK / dev packages are installed | TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON/OFF |
--vulkan / --no-vulkan |
Narrow CMAKE_CUDA_ARCHITECTURES list |
auto-detected from visible GPU | TENSORSHARP_GGML_NATIVE_CUDA_ARCHITECTURES |
--cuda-arch='86-real;89-real' |
| Native build parallelism cap | conservative auto-cap | TENSORSHARP_GGML_NATIVE_BUILD_PARALLEL_LEVEL |
— |
The server emits one structured Information-level entry at the start and end of every chat / generate turn, so a single grep over the log file reproduces the full request-response audit trail without replaying any traffic.
| Event id | Emitted on | Carries |
|---|---|---|
ChatStarted (1500) |
chat.start, generate.start, plus per-protocol request banners |
sampling config, message + attachment counts, userInput= (full latest user message), fullInput= (JSON-encoded array of EVERY message in the request: system prompts + all prior user/assistant turns + the new user message, with attachment counts), or the full prompt for /api/generate |
ChatCompleted (1502) |
chat.complete, generate.complete |
token counts, KV cache reuse (kvReused, kvReusePercent), TTFT, elapsed, throughput, finish reason, full raw assistant output (reasoning + result) |
ChatAborted (1503) |
client disconnected mid-stream | partial output, KV reuse fraction at the time of abort |
KvCacheReusePlan (1510) |
per-prefix-reuse decision | Debug-level fine-grained breakdown (exact match / partial / full reset) |
HttpRequestStarted/Completed (1100/1101) |
every HTTP request | method, path, remote IP, status, duration; /api/queue/status is demoted to Debug so high-frequency UI polling does not drown out the per-turn entries |
The raw assistant output captures <think>...</think>, <|channel|>analysis,
and any other inline framing the model emits, so the log line for a single turn
contains both reasoning and the user-visible result. Combined with the
fullInput= field on chat.start, every turn is fully reproducible from the
log file alone (request inputs + raw model output). Long uploads or long
reasoning traces can produce multi-kilobyte log lines; raise the log level
(TENSORSHARP_LOG_LEVEL=Warning) to suppress them while still keeping the start
banner and error logs.
Sample fullInput payload (formatted for readability; it is emitted as a
single line in the actual log):
[
{"role":"system","content":"You are a helpful assistant."},
{"role":"user","content":"What is the tallest mountain?"},
{"role":"assistant","content":"Mount Everest."},
{"role":"user","content":"How tall is it?","images":1}
]The same per-turn KV cache reuse stats are surfaced through every API:
- Web UI SSE (
POST /api/chat) - thedoneevent carriespromptTokens,kvReusedTokens, andkvReusePercent. - Ollama NDJSON (
POST /api/generate,POST /api/chat/ollama) - the final chunk and the non-streaming response carryprompt_cache_hit_tokens(int) andprompt_cache_hit_ratio(0..1). - OpenAI (
POST /v1/chat/completions) - theusageblock carriesprompt_tokens_details.cached_tokens, matching the OpenAI extension that existing SDKs already understand.
The Web UI footer line under each assistant message also surfaces the cache hit
inline (e.g. 187 tokens · 2.1s · 87.2 tok/s · KV 420/512 (82%)).
TensorSharp.Server exposes three API styles. See API_EXAMPLES.md for full documentation with curl and Python examples.
Ollama-compatible API:
# List models
curl http://localhost:5000/api/tags
# Generate text
curl -X POST http://localhost:5000/api/generate \
-H "Content-Type: application/json" \
-d '{"model": "gemma-4-E4B-it-Q8_0.gguf", "prompt": "Hello!", "stream": false}'
# Chat
curl -X POST http://localhost:5000/api/chat/ollama \
-H "Content-Type: application/json" \
-d '{"model": "gemma-4-E4B-it-Q8_0.gguf", "messages": [{"role": "user", "content": "Hi"}], "stream": false}'
# Chat with thinking mode
curl -X POST http://localhost:5000/api/chat/ollama \
-H "Content-Type: application/json" \
-d '{"model": "gemma-4-E4B-it-Q8_0.gguf", "messages": [{"role": "user", "content": "Solve 17*23"}], "think": true, "stream": false}'
# Chat with tool calling
curl -X POST http://localhost:5000/api/chat/ollama \
-H "Content-Type: application/json" \
-d '{"model": "gemma-4-E4B-it-Q8_0.gguf", "messages": [{"role": "user", "content": "What is the weather?"}], "tools": [{"function": {"name": "get_weather", "description": "Get current weather", "parameters": {"properties": {"city": {"type": "string"}}, "required": ["city"]}}}], "stream": false}'OpenAI-compatible API:
# Chat completions
curl -X POST http://localhost:5000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "gemma-4-E4B-it-Q8_0.gguf", "messages": [{"role": "user", "content": "Hi"}], "max_tokens": 50}'
# Structured outputs (OpenAI response_format)
curl -X POST http://localhost:5000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gemma-4-E4B-it-Q8_0.gguf",
"messages": [{"role": "user", "content": "Extract the city and country from: Paris, France."}],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "location_extraction",
"strict": true,
"schema": {
"type": "object",
"properties": {
"city": {"type": "string"},
"country": {"type": "string"},
"confidence": {"type": ["string", "null"]}
},
"required": ["city", "country", "confidence"],
"additionalProperties": false
}
}
}
}'OpenAI Python SDK:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:5000/v1", api_key="not-needed")
response = client.chat.completions.create(
model="gemma-4-E4B-it-Q8_0.gguf",
messages=[{"role": "user", "content": "What is 2+3?"}],
max_tokens=50
)
print(response.choices[0].message.content)Queue status:
curl http://localhost:5000/api/queue/status
# {"busy":false,"pending_requests":0,"total_processed":42}Models that support thinking mode (Qwen 3, Qwen 3.5/3.6-family, Gemma 4, GPT OSS, Nemotron-H) can produce structured chain-of-thought reasoning before generating the final answer. The thinking content is separated from the main response and can be displayed or hidden by the client.
- Qwen 3 / Qwen 3.5/3.6-family / Nemotron-H: uses
<think>...</think>tags - Gemma 4: uses
<|channel>thought\n...<channel|>tags - GPT OSS: uses Harmony format with
<|channel|>analysisfor thinking and<|channel|>finalfor the response
Enable via --think (console), "think": true (Ollama API), or the thinking toggle in the web UI.
Some architectures ship a multi-token-prediction (MTP / NextN) draft head that lets TensorSharp.Server run lossless speculative decoding for solo (non-concurrent) sequences. The draft proposes several future tokens cheaply, the trunk verifies all of them in one batched forward, and accepted tokens are committed in a single step. Because the request's own sampler — temperature, top-k/p, and repetition/presence/frequency penalties — drives both the draft and the verify, the output is identical to standard decode; speculation only changes how many forward passes it takes to produce it.
Speculative decoding is off by default. Enable it on the server with --mtp-spec (env TS_MTP_SPEC=1):
# Qwen 3.6 — use the -MTP- repository GGUF so the embedded NextN block is retained
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --backend ggml_cuda \
--mtp-spec --mtp-draft 8 --mtp-pmin 0.75
# Gemma 4 — load the separate gemma4-assistant draft GGUF that matches the target
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gemma-4-E4B-it-Q8_0.gguf --backend ggml_cuda \
--mtp-spec --mtp-draft-model models/gemma-4-E4B-it-assistant.Q8_0.ggufTwo draft-head shapes:
- Qwen 3.6 (embedded NextN) — the GGUF carries one extra decoder block past the main stack (
{arch}.nextn_predict_layers) plus the NextN projection/norm tensors. No separate file is required;--mtp-draft-modelis ignored. The recurrent trunk state (GatedDeltaNet) is snapshotted so a partially-rejected verify batch can be rolled back. - Gemma 4 (separate
gemma4-assistantGGUF) — an EAGLE-style recurrent drafter loaded with--mtp-draft-model. It holds no K/V of its own: every draft layer queries the target model's existing per-layer KV cache (last local + last global layer), so the drafter is stateless given(token, hidden). The draft's hidden size must match the target — pair the 12B target with its 12B draft, not the 26B-A4B draft. A mismatched, missing, or incomplete draft GGUF fails fast at startup with a remediation hint instead of silently disabling speculation.
Where it's profitable (engaged automatically; otherwise the engine serves standard decode):
| Backend | Qwen 3.6 | Gemma 4 |
|---|---|---|
| GGML CUDA / GGML Metal | ✅ fused multi-token-verify + draft-step kernels | ✅ fused multi-token-verify + draft-step kernels |
Direct CUDA (cuda, pure C#) |
✅ GPU-resident per-op verify/draft | ✅ GPU-resident per-op verify/draft |
| CPU / GGML CPU / MLX | standard decode (verify can't keep up) | standard decode |
Tuning: --mtp-draft (default 8) bounds tokens drafted per step; --mtp-pmin (default 0.75) is the minimum draft-head confidence to keep a token (drafting stops at the first low-confidence token). Gemma 4 draft-path A/B switches are the TS_GMTP_* env vars (see the MTP / speculative-decoding tunables table under Web Application). Per-architecture mechanics are in the Qwen 3.5/3.6 card and the Gemma 4 card.
Models can invoke user-defined tools and participate in multi-turn tool-call conversations. Define tools as JSON and pass them via --tools (console) or the tools parameter in the API.
Each architecture uses its own wire format for tool calls:
- Qwen 3 / Qwen 3.5/3.6-family / Nemotron-H:
<tool_call>{"name": "...", "arguments": {...}}</tool_call> - Gemma 4:
<|tool_call>call:function_name{args}<tool_call|> - GPT OSS (Harmony): tools are declared as a TypeScript namespace in the developer message, and calls are emitted on the commentary channel as
<|channel|>commentary to=functions.NAME <|constrain|>json<|message|>{args}<|call|>
The output parser (OutputParser.cs) automatically extracts tool calls from the model's raw output regardless of architecture.
Gemma 4 models support image, video, and audio inputs. For the E4B example above, pass the repository's mmproj-gemma-4-E4B-it-Q8_0.gguf explicitly with --mmproj (use the projector matching other target sizes).
- Images: PNG, JPEG, HEIC/HEIF
- Video: MP4 (time-based extraction at 1 fps using OpenCV; tune with
VIDEO_SAMPLE_FPS/VIDEO_MAX_FRAMES) - Audio: WAV (16kHz mono), MP3, OGG Vorbis
Gemma 3 supports PNG, JPEG, and HEIC/HEIF image inputs. The non-gated example above uses mmproj-model-f16.gguf; pass it explicitly with --mmproj.
All Qwen 3.5/3.6-family variants (qwen35, qwen35moe, and qwen3next) load through the same Qwen35Model implementation. Image inputs are supported via the dynamic-resolution Qwen35VisionEncoder; pass the selected repository's projector explicitly (for the 9B and Qwen 3.6 examples, mmproj-F16.gguf). The MoE variants (e.g. Qwen3.5-35B-A3B and Qwen3.6-35B-A3B GGUFs that report the same architecture keys) additionally enable a fused MoEExpertsSwiGLUResidual GGML kernel during decode that runs all selected experts, the optional shared expert, and the residual add in a single GPU graph dispatch.
Mistral 3 supports image inputs via the Pixtral vision encoder. The example repository uses mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf; pass it explicitly with --mmproj.
- Images: PNG, JPEG, HEIC/HEIF
The Nemotron Omni distribution adds a RADIO / v2_vl ViT image encoder. Pass the matching multimodal projector with --mmproj (e.g. nvidia_Nemotron-H-Omni-mmproj.gguf); the language-model GGUF stays the same. Image tokens are inserted at <image> placeholders and expanded into <img> + N tile tokens + </img> automatically by the multimodal injector.
- Images: PNG, JPEG, HEIC/HEIF
- Audio: the chat template emits
<so_embedding>per uploaded audio file and the CLI runs the Parakeet-style log-mel preprocessor for verification, but actual audio inference requires a Parakeet audio mmproj that the public GGUFs do not currently ship.
TensorSharp is structured as a layered system:
-
TensorSharp.Core provides the core
Tensortype, storage abstraction, and the extensible operation registry (Ops). CPU implementations useSystem.Numerics.Vectorsfor SIMD acceleration. -
TensorSharp.Runtime owns runtime-facing contracts and services: GGUF parsing, tokenization (SentencePiece / BPE), chat template rendering, configurable token sampling, output parsing, paged KV cache (
Runtime/Paged/*), the continuous-batching scheduler / engine (Runtime/Scheduling/*), theIKvBlockCodecinterface plus theTurboQuantKvCodec2-bit / Q4 / Q8 implementation, and reusable contracts such asIModelArchitecture,IBatchedPagedModel,IPromptRenderer,IOutputProtocolParser,IMultimodalInjector,IKVCachePolicy, andIBackendExecutionPlan. -
TensorSharp.Models implements
ModelBaseplus the concrete architectures and multimodal helpers (Gemma 3/4, DiffusionGemma, Qwen 3/3.5, GPT OSS, Nemotron-H, Mistral 3, Qwen-Image-Edit). Autoregressive architectures ship the legacy per-sequence forward, and most also expose anIBatchedPagedModel.ForwardBatchimplementation (<Family>Model.BatchedForward.cs) for continuous batching. DiffusionGemma is intentionally different:Forward()is unsupported, and generation goes throughDiffusionGemmaSamplerover fixed-length denoising canvases. Qwen-Image-Edit (QwenImageModel) is likewise not autoregressive —Forward()throws and image editing runs throughEditImage(), which orchestrates the MMDiT diffusion transformer, the Qwen-Image VAE, and the Qwen2.5-VL text encoder. Models are loaded viaModelBase.Create()which auto-detects the architecture from GGUF metadata. -
TensorSharp.Backends.GGML registers accelerated implementations of the same operations via a native C++ bridge (
libGgmlOps/GgmlOps.dll) that links against ggml. On macOS this provides Metal GPU compute, and on Windows/Linux it can expose GGML CUDA for NVIDIA GPUs. Operations include native quantized matmul (Q4_K_M, Q8_0, etc.) without dequantizing to FP32, plus paged-attention (TSGgml_PagedAttentionForward, with and without attention sinks) and architecture-specific batched kernels (Mamba2, GatedDeltaNet). -
TensorSharp.Backends.Cuda is the direct CUDA path. It uses the CUDA Driver API for device/context/storage management, cuBLAS for float32 GEMM, PTX kernels for hot scalar and transformer helper ops, and CPU fallbacks where native kernels are not implemented yet.
-
TensorSharp.Backends.MLX is the Apple Silicon MLX path. It wraps mlx-c (
libmlxc) with allocator, storage, async worker dispatch, quantized + fused + compiled kernels, MoE expert offload, and a CPU fallback layer for ops that aren't yet wired up. -
TensorSharp.Server is the HTTP/application layer. It provides Ollama-compatible and OpenAI-compatible REST APIs, the browser-based chat UI, upload handling, an
InferenceEngineHostthat owns the per-model continuous-batching engine for autoregressive models, aDiffusionBatchSchedulerfor DiffusionGemma Web UI turns, and a thin queue-status surface for backward compatibility. -
TensorSharp.Cli is the console/application layer for local prompts, multimodal experiments, prompt inspection, JSONL batch workflows, the interactive REPL, and the built-in prefill / decode benchmarks.
The list below is the cross-architecture summary; each per-model card under
docs/models/ walks through the same kernels in
context, with the exact GGML graph dispatched and the conditions under which
the fused path engages.
- Fused GPU decode (Gemma 4): all transformer layers are executed in a single GGML compute graph dispatch on Metal, reducing CPU-GPU round-trips from hundreds per token to one. This achieves ~2.6x speedup over per-operation dispatch.
- Fused GPU prefill (Gemma 4): for dense (non-MoE, non-shared, non-PLE/multimodal) layers,
Gemma4LayerPrefillruns the entire transformer block (RMSNorm + QKV + QK-norm + RoPE + attention + output projection + post-attn norm + GeGLU FFN + post-FFN norm + residual + layer scalar) as a single GGML graph dispatch per layer during prefill, extending the fused approach from decode to multi-token prefill. - Chunked prefill (Gemma 4): long prompts are split into bounded chunks (2x sliding window, max 2048 tokens) to avoid O(n^2) attention score tensors for SWA layers. Chunking is applied automatically when text-only (no multimodal embeddings) and keeps each chunk within the SWA window budget.
- Native whole-model decode (Qwen 3): all transformer layers run in one native call (
TransformerModelDecode) with pre-resolved per-layer weight pointers cached at load time, removing managed-loop overhead from the decode hot path. - Fused Qwen 3.5/3.6-family attention layer decode: a single GGML graph performs RMSNorm + fused QKV + Q/gate deinterleave + per-head QK norm + RoPE + KV cache append + flash attention + sigmoid-gated mix + output projection + residual add for each FullAttention layer. Replaces ~2 standalone GGML calls and ~6 small CPU/GPU sync points per attention layer. Engages once the cached sequence length exceeds 4096 tokens (override with
FUSED_ATTN_LAYER_MIN_SEQ_LEN=N). - Fused prefill attention (Qwen 3.5/3.6-family):
FusedPrefillAttentioncombines Q*K^T, causal mask, softmax, and *V into a single GGML graph dispatch during multi-token prefill, eliminating ~5 separate C#-to-GGML round-trips per attention layer. Handles both initial prefill and continuation with existing KV cache entries. - Fused output-projection + FFN (Qwen 3.5/3.6-family): for both FullAttention and GatedDeltaNet layers with dense FFN,
FusedOutProjFFNmerges the output projection, residual add, post-attention RMSNorm, and the full SwiGLU FFN (gate_up matmul + SiLU + down matmul + residual) into a single GGML graph dispatch, reducing two GPU round-trips to one per layer. - Fused output-projection + norm + router (Qwen 3.5/3.6-family MoE):
FusedOutProjNormRoutermerges the GatedDeltaNet output projection, residual add, post-attention RMSNorm, and MoE router projection into one dispatch. The pre-computed router logits are then consumed directly by the batched MoE kernel, eliminating a separate router dispatch per MoE layer. - Fused vision encoder (Qwen 3.5/3.6-family):
FusedVisionAttentionmerges LayerNorm + QKV + bias + 2D RoPE + scaled dot-product attention + output projection + bias + residual into one GGML graph dispatch (~8 ops → 1).FusedVisionMLPmerges LayerNorm + up + bias + GELU + down + bias + residual into one dispatch (7 ops → 1). Combined, these cut the per-block GPU round-trips from ~15 to 2. - Fused weight projections: Q/K/V projections are fused into a single QKV matmul; gate and up projections are fused into a single gate_up matmul.
- Native quantized compute: quantized weights (Q4_K_M, Q6_K, Q8_0, IQ2_XXS, MXFP4, etc.) are used directly in matmul without expanding to FP32, saving memory and bandwidth. A batched
AddmmQuantBatchkernel handles multiple sub-weight matmuls against a single quantized blob in one dispatch. - Direct CUDA kernels: the
cudabackend accelerates fill/copy, unary ops, activation fusions, RMSNorm, softmax, index select, causal masking, RoPE/RoPEEx, cuBLAS GEMM, and supported quantized matmul/get-rows while safely falling back for incomplete op coverage. - Batched GPU MoE:
MoEExpertsSwiGLUResidual(Qwen 3.5/3.6-family) andMoEExpertsForward(Nemotron-H) collapse all selected experts -- and, for Qwen 3.5/3.6-family, the optional shared expert and the residual add -- into a single GGML graph dispatch per MoE layer. - GEMM-based vision patch embedding (Qwen 3.5/3.6-family): the patch embedding step is reformulated as parallel im2col + matrix multiplication, replacing a single-threaded scalar quintuple-nested loop with a GPU-accelerated matmul.
- Parallelized Q/gate deinterleave (Qwen 3.5/3.6-family): the Q + sigmoid-gate deinterleave in FullAttention prefill is parallelized across tokens, scaling linearly with CPU core count for long prompts.
- Optimized pure C# CPU path: managed GEMM fast paths and contiguous float32 kernels accelerate decode, softmax, RMSNorm, RoPE, fused activations, and other hot paths while keeping quantized GGUF weights compressed during CPU loading.
- Circular KV cache: sliding-window attention layers use a fixed-size circular buffer, bounding memory usage regardless of sequence length.
- KV-cache prefix reuse: multi-turn conversations reuse the longest matching token prefix across turns. Truncation is automatically backed off by the sliding-window size for SWA models so the suffix can rebuild the SWA context.
- Paged KV cache & block-hash prefix sharing: the continuous-batching engine partitions KV into fixed-size blocks, content-hashes each full block, and shares them across concurrent and sequential requests. Models that have not implemented
IBatchedPagedModelstill use the engine's isolated per-sequence KV-swap fallback. - Native paged-attention kernel:
TSGgml_PagedAttentionForward(and theWithSinksvariant for GPT OSS) does a C++ gather of K/V from the paged buffer, builds a small GGML graph per sequence, and dispatchesggml_flash_attn_ext— the same fused GPU flash-attention kernel (Metal/CUDA/Vulkan) the legacy single-sequence path uses. On Ministral-3-14B long-context (4×~800 tokens) it is ~21 % faster than the legacy per-sequence GGML path. - Batched / paged forward passes: Mistral 3, Gemma 4, GPT OSS, Qwen 3.5/3.6 (incl. GatedDeltaNet recurrent state pool), and Nemotron-H (incl. Mamba2 recurrent state pool + native batched Mamba2 kernel) pack N sequences into a single
ForwardBatchcall with one batched linear-projection matmul per layer, paged K/V scatter viaslotMapping, and per-sequence attention via the native kernel. Gemma 4 batched path reaches 1.5× legacy throughput at batch=8 short prompts and 1.6× at 4×800-token prompts; Nemotron-H Mamba2 batched reaches 3.95× at batch=3 on Apple M4 Pro. See docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING.md. - MTP / NextN speculative decoding: solo sequences can run a multi-token-prediction draft head (Qwen 3.6 embedded NextN block; Gemma 4 separate
gemma4-assistantdraft GGUF). The draft proposes up to--mtp-drafttokens and the trunk verifies them in one batched forward, with the request's own sampler driving both, so decode is accelerated without changing the output. On ggml backends, fused single-graph multi-token-verify and draft-step kernels (NativeGemma4ModelVerify/TryFusedMoEModelVerify/NativeGemma4DraftStep, plus the Qwen 3.6 NextN graph) amortize the verify; the Gemma 4 path also adds gallocr verify scratch and a dense fast-rollback that avoids re-running the kept prefix on partial acceptance. The pure-C#cudabackend runs a fully GPU-resident per-op verify/draft (donor-cache attention, GQA decode kernel, GPU RoPE) so the verify layer loop issues zero host-sync stalls. Off by default;--mtp-spec. - DiffusionGemma prompt-KV caching and fused denoising: on GPU backends, the prompt side of
[prompt | canvas]is prefetched once per block and reused across denoising steps; GGML backends default to fused whole-model diffusion decode plus a fused lm-head tail. The Web UI batches concurrent diffusion requests at block boundaries throughDiffusionBatchScheduler. - Kernel warmup: both CLI and Server run a tiny forward pass at startup to pre-compile GPU kernels (Metal pipeline states, CUDA JIT) and warm the memory pool, avoiding cold-start latency on the first real inference request.
- Prefill caching (Gemma 4, Qwen 3.5/3.6-family): per-forward-pass SWA mask cache (Gemma 4), NeoX RoPE cos/sin lookup table cache across global layers (Gemma 4), and RoPE position tensor cache across layers (Gemma 4, Qwen 3.5/3.6-family) eliminate redundant recomputation during prefill.
- In-place QK RMSNorm (Qwen 3.5/3.6-family): per-head QK normalization is performed in-place using a
View, avoiding one tensor allocation and copy per Q/K per layer.
- Zero-copy file-mapped quantized weights (direct CUDA, GGML CUDA, GGML Metal, GGML CPU): the GGUF model file is memory-mapped and quantized tensors are bound directly into native ops via host-pointer buffers. This removes the per-tensor copy from disk into a freshly-allocated native heap buffer that previously roughly doubled the resident set on Apple Silicon for large quantized models. For example,
Qwen3.5-35B-A3B-IQ2_XXS(~10 GB GGUF) now runs with ~7 GB peak working memory under Metal instead of ~17 GB. The OS keeps the mapped file in its page cache and pages it out under memory pressure without any inference penalty on Apple Silicon (unified memory). - Best-fit memory pool: the GGML host allocator uses a best-fit search across pooled blocks instead of first-fit, which avoids handing out a large scratch block to satisfy a tiny intermediate-tensor request and keeps the working-set tightly bounded across long-running inference.
- Bounded pool retention: the integrated-GPU / CPU memory pool now caps individual retained blocks at 64 MB and the total pool at 32 blocks. Combined with mmap-backed weights, this keeps short-lived intermediate tensors recycled fast while bounding the peak resident set.
- Memory-efficient model loading: large tensors are streamed directly to native memory without intermediate managed allocations. F32 weights and norms still load on demand; quantized weights are mmap-backed when supported by the backend.
- Paged KV block pool with optional SSD spillover: paged KV blocks live in a per-engine
BlockPoolwith LRU eviction; thePagedKvBlockStorekeeps a configurable RAM cap (TS_KV_CACHE_MAX_RAM_MB) and spills cold blocks into an SSD tier (TS_KV_CACHE_SSD_DIR) up toTS_KV_CACHE_MAX_SSD_MB. Block content-hashes are kept in a global index so prefix matches are reused across sessions and requests without rematerialising the K/V. - KV block codecs: blocks can be optionally compressed in-place with
TurboQuantKvCodec(2-bit affine, Q4, or Q8) via--paged-kv-quant-bits, trading accuracy for a smaller per-block bandwidth and memory footprint — roughly half (Q8), a quarter (Q4), or a tenth (2-bit, fp32 blocks). The 2-bit tier uses an affine per-group min+scale (the block-min idea behind llama.cpp's Q2_K) so its four codes span the group's actual range; it is intended for long-context far-prefix reuse where attention weights dwarf the quantization noise. Recurrent-state models fall back to passthrough automatically.
A pure-.NET engine going toe-to-toe with the hand-tuned C++ llama.cpp on identical GGUF files, the same NVIDIA RTX 3080 Laptop GPU (16 GB), and one uniform OpenAI /v1/chat/completions surface — across short/long text, multi-turn, tool-calling, and structured-output (JSON) scenarios. The numbers below are the geomean speedup of TensorSharp over llama.cpp on the same GGML CUDA backend (single-stream, greedy, MTP off). > 1.0× means TensorSharp is faster (decode / prefill) or lower-latency (TTFT). The full per-scenario tables are in docs/engine_comparison_report.md.
| Model | decode | prefill | TTFT |
|---|---|---|---|
| Gemma 4 E4B it (Q8_0, dense multimodal) | 1.46× | 0.83× | 0.82× |
| Gemma 4 12B it (QAT UD-Q4_K_XL, dense) | 1.17× | 1.01× | 0.99× |
| Gemma 4 26B-A4B it (QAT UD-Q4_K_XL, MoE) | 0.96× | 1.32× | 1.30× |
| Qwen 3.6 35B-A3B (UD-IQ2_XXS, MoE) | 0.92× | 0.99× | 0.97× |
Where TensorSharp pulls clearly ahead:
- The MoE flagship owns prefill and first-token latency. On Gemma 4 26B-A4B, verify-based whole-model prefill beats llama.cpp in every scenario — up to 1.59× on short prompts and 1.70× in JSON mode — and the first token lands sooner in every scenario too, up to 1.65×. Geomean: 1.32× prefill / 1.30× TTFT.
- Dense 12B decode never loses. Gemma 4 12B wins or ties llama.cpp on decode in all five scenarios (geomean 1.17×), with streamed tool-call turns decoding 2.05× faster (81.0 vs 39.5 tok/s).
- Structured output (JSON mode) is a standout on E4B. Constrained-JSON decode streams 405 tok/s vs 52 — 7.73× — lifting E4B's overall decode geomean to 1.46×.
- Tool-calling turns fly on the MoE flagship. Function-call decode on 26B-A4B runs 2.37× faster (174.3 vs 73.4 tok/s), with a lower time-to-first-token on top.
- E4B out-prefills llama.cpp on every text scenario. Short prompts 1.22×, multi-turn chat 1.16×, tool-call 1.12×, long context 1.08× — with first tokens landing up to 1.20× sooner.
- Parity even at extreme quantization. The aggressively quantized IQ2_XXS Qwen 3.6 35B-A3B MoE holds decode within ~8% (0.92×) and prefill at parity (0.99×) — a pure-.NET engine keeping pace with hand-tuned C++ on 2-bit weights.
GGML Vulkan backend — the vendor-neutral GPU path was benchmarked the same way (same host, both engines on their Vulkan builds, Gemma 4 E4B + 12B). Text-scenario decode sits within a few percent of llama.cpp (0.91–1.02× per scenario, with 12B tool-call turns decoding 1.96× faster), and text prefill reaches parity on short prompts, JSON mode, and multi-turn chat (up to 1.20×). The sub-1.0× prefill geomeans (0.68× E4B / 0.70× 12B) are dominated by the image/audio prefill path, which is not yet optimized on Vulkan. Per-scenario tables are in the report's Vulkan section.
Outside these standout cells, decode sits within a few percent of llama.cpp, so the prefill, TTFT, tool-call, and structured-output wins come on top of competitive token generation. The remaining sub-1.0× cells — chiefly 26B-A4B plain-text decode and E4B JSON-mode prefill — are active optimization targets rather than finished results. Every cell, wins and gaps alike, is in the full report.
InferenceWeb.Tests exercises in-process behavior that doesn't require a running server: managed quantized ops, direct CUDA backend kernels when a CUDA device is available, MLX backend kernels when MLX is available, paged KV cache scheduling (ContinuousBatchSchedulerTests, PagedKvCacheTests, PagedKvCacheCodecTests), batched executor correctness (BatchedExecutorTests), per-model batched-forward correctness against the legacy path (Qwen35BatchedCorrectnessTests, Mistral3BatchedForwardTests, Gemma4BatchedForwardTests, GptOssBatchedCorrectnessTests, NemotronBatchedCorrectnessTests), MTP / NextN speculative-decoding correctness and opt-in end-to-end probes (MtpSpeculativeExecutionTests, Qwen36MtpTests, Gemma4MtpTests), DiffusionGemma denoising / prompt-KV / batched-generation probes (DiffusionGemmaTests), per-model batched perf microbenchmarks (*BatchedPerfBench.cs), TurboQuantKvCodec codec round-trips, prefill chunking, KV cache policies, KV-cache prompt rendering / multi-turn integration, chat-session and session-manager isolation, model service history plumbing, request-logging middleware and file-logger provider, image preprocessing, media helpers, structured-output validation, text-upload helpers, model-service upload logging, web UI chat policy, model context length parsing, backend catalog resolution, and the server CLI options builder (ServerOptionsBuilderTests).
dotnet test InferenceWeb.Tests/InferenceWeb.Tests.csprojIntegration tests for TensorSharp.Server are in TensorSharp.Server/testdata/. They cover all three API styles (Web UI SSE, Ollama, OpenAI), multi-turn conversations, thinking mode, tool calling, structured outputs, queue-status compatibility, concurrent requests, and abort support. Architecture-specific features (thinking, tool calling) are auto-detected and skipped when the active model does not support them.
# Start TensorSharp.Server, then run:
python3 TensorSharp.Server/testdata/test_multiturn.py
# or
bash TensorSharp.Server/testdata/test_multiturn.shSee TensorSharp.Server/testdata/README.md for the full test matrix.
TensorSharp.TestMatrix is the broader CLI-driven harness for long-running model/backend coverage. It discovers GGUF files, filters unavailable backends and unsupported prompt types, runs baseline plus env-var sweep cells, writes one JSON result per cell, emits an aggregate Markdown report, and compares results with per-host baselines when requested.
dotnet build TensorSharp.TestMatrix/TensorSharp.TestMatrix.csproj -c Release
dotnet run --project TensorSharp.TestMatrix -c Release -- --dry-runSee TensorSharp.TestMatrix/README.md and docs/env_var_feature_matrix.md for the current runner contract.
Zhongkai Fu
See LICENSE for details.