Kimi K3 Vendor Verifier: vLLM, SGLang and Provider Scores Explained

Moonshot’s Kimi Vendor Verifier (KVV) is an open-source test harness that checks whether a host serves Kimi K3 the way Moonshot does, and vLLM’s reference deployment passes it: Inferact’s vLLM row scores 0.891 on OCRBench, 0.818 on MMMU Pro Vision, 0.3188 on BEAM (1M) and 0.695 on DeepSWE, against Moonshot’s own 0.89 / 0.82 / 0.31 / 0.675. SGLang’s reference deployment (RadixArk) and seven commercial providers are on the same table, all inside a narrow band. The catch is that a KVV row certifies one specific deployment, not every endpoint that says “Kimi K3” on the label.

This guide covers the verifier as of September 25, 2026: what it tests, the full K3 scoreboard, the vLLM and SGLang recipes, why the same weights still behave differently between providers, and how to test the endpoint you pay for. For price and region, see the Kimi K3 API providers guide.

Why a vendor verifier exists at all

Open weights solve availability, not fidelity. After Kimi K2 shipped in 2025, Moonshot kept getting reports that K2 behaved worse on some APIs than on its own. The first response was the K2 Vendor Verifier (K2VV), which sent 4,000 requests to each provider and compared tool-call behavior against the official API.

The results were uncomfortable. In the November 15, 2025 run on K2-0905, Moonshot, DeepInfra, Fireworks, Infinigence and NovitaAI hit 100% tool-call schema accuracy, while Together scored 71.96%, Baseten (via OpenRouter) 72.49%, and Nebius posted 84.47% schema accuracy with only 50.60% trigger similarity. The open-source engines were near the bottom too: vLLM 76.00% and SGLang 73.13%.

Moonshot then broadened the idea. Its KVV announcement says benchmark complaints since K2 Thinking traced back largely to “misuse of Decoding parameters,” and that a gap between third-party and official APIs turned out to be widespread. Moonshot’s API started enforcing temperature 1.0 and top_p 0.95 in thinking mode, and the company open-sourced a broader suite with a public vendor leaderboard. Its summary line: “Weights are open. The knowledge to run them correctly must be too.”

The MIT-licensed Kimi-Vendor-Verifier repo went public in January 2026, per its commit history. K3 tests landed on July 27, 2026, release day.

What KVV tests for Kimi K3

KVV has two layers. The pre-flight suites are cheap pytest checks against raw chat-completions behavior, and the repo says formal benchmarks should run “only after all tests pass.” The scored benchmarks then measure whether capability survives the serving stack.

Check	What it does	What a failure looks like in production
tests/params/	Confirms the API constrains fixed parameters: temperature, top_p, presence_penalty, frequency_penalty, n	Answers drift from the official API; reasoning quality drops because sampling is wrong
tests/tool_call_json_schema/	Sends schemas as tools[].function.parameters, forces a call, validates returned arguments with jsonschema, once streaming and once not	Agents get malformed arguments or broken streamed tool calls
tests/k3_features/	Dynamic tools, response_format, tool_choice, thinking effort, malformed requests that should return HTTP 400	reasoning_effort silently ignored; structured output not enforced
tests/prompt_tokens/	Compares reported usage.prompt_tokens for text and vision cases against expected constants	You get billed for tokens that do not match the real prompt
OCRBench	OCR text recognition, the recommended first smoke test	Screenshot and document tasks fail
MMMU Pro Vision	Multimodal visual QA, up to 98,304 output tokens	Wrong image preprocessing; vision answers degrade
BEAM (1M)	Long-term memory over 1M-token conversations	”1M context” on the pricing page, much less in practice
DeepSWE	113 coding-agent tasks run through Kimi Code 0.23.6+ on the Pier platform	Agents lose the thread across multi-step tool loops

Two details matter for anyone reading scores. First, all K3 results are at thinking effort max, with thinking enabled and prior reasoning kept (keep: all), at temperature 1.0 and top_p 0.95. A provider that caps output tokens or strips reasoning will look worse, which is the point. Second, AIME 2025, the long-output stress test used for K2-era models, is marked “no longer used for K3.” BEAM and DeepSWE replace it.

The Kimi K3 scoreboard

Here is the full K3 table from the KVV README, listed there “in the order of submission time” (model kimi-k3, thinking effort max):

Provider	OCRBench	MMMU Pro Vision	BEAM (1M)	DeepSWE
Moonshot (official)	0.89	0.82	0.31	0.675
Fireworks	0.89	0.82	0.3037	0.664
Baseten	0.889	0.804	0.3219	0.693
Together	0.897	0.820	0.3160	0.678
DigitalOcean	0.89	0.816	TBD	TBD
Inferact (vLLM reference)	0.891	0.818	0.3188	0.695
Nebius	0.878	0.814	0.2913	0.673
Modal	0.887	0.817	0.322	0.658
RadixArk (SGLang reference)	0.895	0.820	0.3107	0.667

Source: MoonshotAI/Kimi-Vendor-Verifier, fetched September 25, 2026.

How to read it:

• Everyone listed is inside noise. OCRBench spans 0.878 to 0.897, MMMU Pro Vision 0.804 to 0.820. DeepSWE runs from 0.658 (Modal) to 0.695 (Inferact), but on 113 tasks that is roughly 74 versus 79 solved, a normal single-run gap. Several third parties beat Moonshot’s own row, so the variance is run-to-run, not a hidden advantage.
• There is no published pass mark for K3. The K2 verifier stated thresholds (tool-call F1 of at least 73% for K2 Thinking, 80% for K2-0905). The K3 table has no threshold and no pre-flight column. “Vendor-verified” in practice means “has a row on Moonshot’s table,” not a graded certificate.
• The table moves. Commit history shows corrections to Together and Inferact on August 3, Inferact’s BEAM and DeepSWE added August 6, Baseten updated August 11, Nebius revised August 13 and RadixArk added September 17. Check the live README, not vendor screenshots.
• DigitalOcean is half-done. Its long-context and agentic columns, the two that matter most for coding agents, still read TBD.

vLLM’s path: from the bottom row to the reference row

The headline “vLLM is vendor-verified” lands harder once you see the history in Moonshot’s own tables:

Engine	K2-0905 tool-call schema accuracy (Nov 2025)	K2 Thinking schema accuracy (Nov 2025)	Kimi K3 KVV (DeepSWE / OCRBench)
vLLM	76.00%	87.22%	0.695 / 0.891 (Inferact reference)
SGLang	73.13%	95.52%	0.667 / 0.895 (RadixArk reference)
Moonshot official	100%	100%	0.675 / 0.89

Ten months ago, the two main open-source engines were among the least accurate ways to serve K2’s tool calls. For K3 they are the reference implementations commercial hosts build on. That came from process. The vLLM preview post (July 22) describes pre-release access for “selected trusted partners, approved by both Moonshot AI and the vLLM/Inferact team,” with vendor verification inside the final validation loop. vLLM’s day-0 post separately reports 0.976 on GSM8K, 0.939 on GPQA-Diamond, 0.889 on OCRBench and 0.818 on MMMU Pro Vision at max effort, in line with the KVV row.

vLLM and SGLang support status for K3

Both engines had day-0 support on July 27. Here is what each ships, from their own posts.

	vLLM (Inferact)	SGLang (RadixArk)
Launch post	vLLM blog, Jul 27	LMSYS blog, Jul 27
Install path	Docker only for now: vllm/vllm-openai:kimi-k3 (CUDA 13, r580+ driver), vllm/vllm-openai_rocm:kimi-k3	Cookbook configurator at docs.sglang.io
Minimum hardware	8× B300 / GB300, or 8× MI355X; 16× B200 supported	GB300, B300, B200, H200, H20, MI350X, MI355X listed
Batch-1 decode	111 tok/s (TP8), 118 tok/s (TP16) on GB300 NVL72	~113 tok/s
With DSpark speculation	331 tok/s (TP8), 370 tok/s (TP16)	~423 tok/s
KVV reference row	Inferact	RadixArk (added Sep 17)

The vLLM quick start is short:

vllm serve moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

Three flags there decide whether you match the KVV row. --tool-call-parser kimi_k3 and --reasoning-parser kimi_k3 split K3’s output into separate reasoning, content and tool-call fields. --enable-prefix-caching must be passed explicitly, because vLLM leaves it off by default for K3 while the hybrid KDA cache design evolves. Leave it off and you lose the prefix-cache economics that make agent workloads affordable.

The vLLM recipe page (updated September 10) adds backend choices (deepep_v2 over RDMA, flashinfer_nvlink_one_sided over NVLink, deep_gemm_mega_moe for expert parallelism) and --prefix-match-unit 128 on Blackwell. It also links a community Ascend W4A8 checkpoint. That is a re-quantized build, not Moonshot’s native MXFP4, and it has no KVV row.

Speeds keep moving: vLLM’s September 15 DSpark post reports single-stream math decode rising from about 110 to about 435 tok/s per user. For sizing, see our VRAM requirements breakdown (roughly 1.7 to 1.84TB of accelerator memory).

Why the same weights serve differently

Every provider downloads the same ~1.56TB from moonshotai/Kimi-K3. The differences come from everything around the weights.

The frontend, not the math. Baseten’s day-0 engineering write-up is blunt: “While the popular narrative is that quantization is the root of all quality issues, in practice that is not true. Most quality issues, especially with tool calling and other structured model behaviors, come from the inference server frontend.” The frontend tokenizes, renders the chat template, parses reasoning and tool calls, and formats the response. Baseten treated passing KVV as an early milestone and re-ran it after later optimizations.

K3’s chat template is a program. Most models ship a Jinja template. According to vLLM, K3 instead “uses a Python program to build the prompt token sequence directly,” with exact control tokens for messages, images, tool definitions and tool results. vLLM rebuilt the renderer and streaming parser for it and wired K3’s format into XGrammar. An older build or a hand-rolled template can produce prompts that look right and still shift behavior.

Sampling and thinking parameters. KVV exists partly because providers ignored or remapped decoding settings. With K3 there is also thinking effort. On Moonshot’s API, KVV sends {"thinking": {"type": "enabled", "keep": "all", "effort": "max"}}. On an open-source deployment the same intent is {"chat_template_kwargs": {"thinking": true, "preserve_thinking": true, "thinking_effort": "max"}}. A gateway that translates one format to the other incorrectly quietly downgrades reasoning.

Tool calls, even when done right. vLLM’s own deployment tips say they have “occasionally seen K3 emit a tool-call format its own parser does not expect, yielding an empty tool_calls result.” They call it prompt- and run-dependent and recommend schema validation, retries or fallback, and strict or structured tool calling. This quirk affects correct deployments too, so build the retry into your agent loop no matter who hosts the model.

Long context and caching. K3 mixes KDA layers (a recurrent state that overwrites itself) with full-attention layers; SGLang counts 69 KDA layers interleaved with 24 MLA layers. Prefix caching over recurrent state is new engineering, which is why BEAM (1M) is on the verifier. A host that advertises 1M context but mishandles the hybrid cache will fail at depth, not on a short smoke test.

Quantization, where it applies. K3 ships natively in MXFP4, and Baseten says it used those weights directly rather than converting to NVFP4. Hopper GPUs dequantize at runtime. The quantization risk sits with re-quantized community builds and with hosts that do not say which checkpoint they serve.

The providers that are not on the table

KVV’s nine rows are a small slice of the market. OpenRouter’s endpoint API listed 20 Kimi K3 endpoints from 17 providers on September 25, far more than the single upstream it had at launch. Here is how part of that list lines up against KVV:

OpenRouter endpoint	Listed precision	Input / output per 1M	Tools param	KVV row?
Sail Research	fp4	$1.35 / $11.50	Yes	No
InferenceNet	fp4	$1.40 / $10.75	Yes	No
Relace	fp4	$2.10 / $10.50	Yes	No
DigitalOcean	not stated	$2.55 / $12.95	Yes	Partial (TBD columns)
DeepInfra	bf16	$2.85 / $14.25	Yes	No
Chutes	mxfp4	$3 / $15	Not listed	No
Moonshot AI	mxfp4	$3 / $15	Yes	Yes
Modal	mxfp4	$3 / $15	Yes	Yes
Baseten	fp8	$3 / $15	Yes	Yes
Fireworks (fast)	not stated	$4.50 / $22.50	Not listed	Base endpoint only

Source: OpenRouter’s public endpoints API for moonshotai/kimi-k3, queried September 25, 2026. Precision labels are self-reported. Together and Fireworks (both with KVV rows) list at $3 / $15; Parasail, Wafer, Phala, Morph, Makora and Alibaba serve K3 there without one.

Two cautions. The precision column is a label, not a measurement: Baseten describes native MXFP4 weights with MXFP8 activations, so its “fp8” tag is not a downgrade. And “no KVV row” does not mean a host is bad, only that nobody has published evidence either way. The burden shifts to you, most heavily at the cheap end.

OpenRouter does run its own quality layer. Auto Exacto, on by default since March 2026 for requests with tools, re-ranks providers about every five minutes using throughput, tool-call telemetry and benchmark scores. That helps in aggregate, but it can still route you to a host that has never run KVV.

How to test the provider you are paying for

You do not need a GPU cluster to run the pre-flight checks. They are ordinary HTTP tests.

1. Check the live table first. If your host has a KVV row, note which columns are filled and when the row last changed.
2. Clone and configure. Install with uv sync && uv pip install -e . and set KIMI_BASE_URL and KIMI_API_KEY to the provider, not to Moonshot.
3. Run the four pre-flight suites. tests/params, tests/tool_call_json_schema, tests/k3_features and tests/prompt_tokens. Use --think-mode kimi for Moonshot-style APIs and --think-mode opensource for vLLM, SGLang or KTransformers endpoints. The schema suite writes tool-call-schema-report.json with per-case results.
4. Smoke-test with OCRBench. The repo recommends it as the quick first benchmark. Anything well outside the published 0.878 to 0.897 band is a red flag.
5. Replay your own traffic. Take a few hundred real agent turns, send them to the candidate provider and to Moonshot’s API, and compare how often finish_reason is tool_calls, how often arguments fail your schema, and how often tool_calls comes back empty. This is the K2VV method, and it catches problems generic benchmarks miss.
6. Pin the provider when testing through a gateway. On OpenRouter, test one upstream at a time with provider routing. Otherwise you are averaging several hosts.
7. Check the bill. The prompt-token suite exists because misreported usage.prompt_tokens is a real failure mode. Compare reported usage to what Moonshot reports for the same request.

MMMU, BEAM and DeepSWE cost far more (Moonshot quoted about 15 hours on two 8-GPU H20 servers for a full K2-era run), so most teams can stop after step 5. If you run K3 through Kimi’s own coding CLI instead, see our Kimi Code K3 setup guide.

Which should you pick?

Your situation	What to do
Building coding agents that depend on tool calls	Stay on a host with a full KVV row (Moonshot, Fireworks, Baseten, Together, Modal, Nebius) and still validate tool-call schemas client-side
Self-hosting on Blackwell or MI355X	Start from the vLLM kimi-k3 Docker image or the SGLang cookbook, match the reference flags, run KVV pre-flight before opening traffic
Buying the cheapest endpoint	Run the pre-flight suites and a traffic replay first; budget endpoints mostly have no public KVV evidence
Long documents or 1M-token sessions	Favor rows with a BEAM score; DigitalOcean’s is still TBD
Using a gateway (OpenRouter, Vercel)	Pin a verified upstream for agent traffic, or accept that routing may land on an untested host
Running a community quantization (W4A8, GGUF)	Treat it as a different model until you have measured it; no KVV row covers it

Honest caveats

• KVV is Moonshot grading its own ecosystem. The repo does not say whether each row was run by Moonshot or submitted by the vendor. “Listed in the order of submission time” suggests vendors submit results, and the table shows no pre-flight pass/fail column.
• Point-in-time results. A provider that passed in launch week can ship a new engine build in September. Nothing in the table tells you when a row was last re-run.
• Max effort only. Scores are at effort: max. If you run at low for cost reasons, you are outside what the table measured.
• Single runs. DeepSWE differences of a few tasks, and BEAM differences in the third decimal, should not decide a vendor on their own.

FAQ

What is the Kimi Vendor Verifier? The Kimi Vendor Verifier (KVV) is Moonshot AI’s open-source, MIT-licensed test harness for checking whether a third-party API or a self-hosted deployment serves Kimi models correctly. For Kimi K3 it runs four pre-flight test suites (parameter constraints, tool-call JSON schema, K3 API features, prompt-token accounting) and four scored benchmarks: OCRBench, MMMU Pro Vision, BEAM at 1M tokens and the 113-task DeepSWE coding-agent benchmark.

Does vLLM being vendor-verified mean my vLLM deployment is correct? No. The Inferact (vLLM reference) row in the KVV table shows that vLLM, configured the way Inferact configured it, reproduces Moonshot’s own scores. Your deployment inherits that only if you use the same Docker image, flags and chat-template handling. A different vLLM build, a missing --tool-call-parser flag or a community quantization can all break it. Run the KVV pre-flight tests against your own endpoint.

Which providers have published Kimi K3 vendor verifier scores? As of September 25, 2026 the KVV K3 table lists nine rows: Moonshot, Fireworks, Baseten, Together, DigitalOcean, Inferact (vLLM reference), Nebius, Modal and RadixArk (SGLang reference). DigitalOcean’s BEAM and DeepSWE scores are still marked TBD. Many providers that sell K3, including most budget endpoints on OpenRouter, have no published KVV row.

Why does Kimi K3 quality differ between API providers? The weights are identical, but the serving stack around them is not. Most differences come from the inference server frontend: how the prompt is rendered into K3’s control tokens, how reasoning and tool calls are parsed out of the output, whether sampling parameters and thinking effort are honored, and how long context and prefix caching are handled. Quantization matters too, but Baseten’s engineering team says frontend bugs cause most tool-calling problems.

How do I test my own Kimi K3 provider? Clone MoonshotAI/Kimi-Vendor-Verifier, set KIMI_BASE_URL and KIMI_API_KEY to your provider, and run the pytest suites in tests/params, tests/tool_call_json_schema, tests/k3_features and tests/prompt_tokens. If they pass, run the OCRBench eval as a quick smoke test and compare against the published 0.878 to 0.897 band. Then replay a sample of your own agent traffic and count empty or schema-invalid tool calls.

What vLLM flags does Kimi K3 need for tool calling? vLLM’s day-0 quick start uses --enable-auto-tool-choice, --tool-call-parser kimi_k3 and --reasoning-parser kimi_k3, plus --trust-remote-code and --enable-prefix-caching (prefix caching is off by default for K3). vLLM still warns that K3 occasionally emits a tool-call format its parser does not expect, so production agents should validate against the schema and retry when tool_calls comes back empty.

Is quantization the main reason Kimi K3 providers underperform? Usually not. K3 ships natively in MXFP4, so most serious hosts serve the original weights without a further quantization step. Baseten reports that most tool-calling and structured-output problems come from the frontend code around the model. Quantization becomes a real risk with community re-quantizations such as W4A8 builds, so ask a provider which checkpoint it serves.

The bottom line

For Kimi K3, the download is the easy part. Moonshot’s verifier shows that when vLLM, SGLang and the major hosts wire up the chat renderer, parsers, thinking effort and hybrid cache correctly, they land right on top of Moonshot’s own scores. That is real progress from the K2 era, when both engines sat near the bottom of Moonshot’s tool-call table. But a KVV row covers one deployment on one date at max effort, and most of the market, including the cheapest endpoints, has no row at all. If your product depends on tool calls, pick a host with a full row, run the pre-flight suites yourself, and keep schema validation and retries in your agent loop.

Keep reading: the Kimi K3 API providers guide for pricing and regions, VRAM requirements before you self-host, the open weights breakdown, full specifications and the complete Kimi K3 guide.