Qwen3.6 27B
Dense 27B multimodal model tuned for flagship-level agentic coding, with a built-in MTP head for speculative decoding.
Result
Throughput across the 13 fixed test scenarios. Paddock, vLLM and SGLang all serve the byte-identical official FP8 checkpoint. llama.cpp has no FP8 weight type to serve it with, so an FP8 checkpoint has to be dequantised when it is converted (llama.cpp's own converter has a --fp8-as-q8 flag for exactly that). It is measured here against the Paddock row reading the same Q8_0 file it reads. Speculative decoding is off.
vs vLLM
11 of 13
scenarios faster, up to +9.0% - behind on 2 · FP8 weights
vs SGLang
9 of 13
scenarios faster, up to +7.3% - behind on 2, level on 2 · FP8 weights
vs llama.cpp
13 of 13
scenarios faster, 1.16x-17.1x · Q8_0 weights
Throughput as the Load Climbs
What the server delivers, from one client up to a full batch. Higher is better. The shaded region is what Paddock delivers above the strongest of the others at each step, and the figure above the line is its multiple there.
Balanced load
1,024 tokens in · 1,024 tokens out
Server throughput (tokens/s)
BESTWORSTThe 64-client step is left off: past a max batch of 32 every engine is queueing rather than serving. It is in the table.
Table view
| Clients | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 47.7 tok/s | 45.0 tok/s | 45.0 tok/s | 33.7 tok/s |
| 2 | 93.9 tok/s | 83.9 tok/s | 85.1 tok/s | 48.0 tok/s |
| 4 | 172.6 tok/s | 162.5 tok/s | 167.1 tok/s | 60.4 tok/s |
| 8 | 334.1 tok/s | 321.1 tok/s | 321.9 tok/s | 64.9 tok/s |
| 16 | 591.1 tok/s | 578.7 tok/s | 581.0 tok/s | 109.7 tok/s |
| 32 | 968.1 tok/s | 968.1 tok/s | 835.0 tok/s | 239.1 tok/s |
| 64 | 970.4 tok/s | 963.9 tok/s | 899.3 tok/s | 265.2 tok/s |
Agentic sessions
Shared-prefix sessions × 6 turns
Server throughput (tokens/s)
BESTWORSTSessions share a prefix, so this ladder also measures whether an engine’s prefix cache engages.
Table view
| Sessions | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 47.3 tok/s | 42.5 tok/s | 44.6 tok/s | 16.2 tok/s |
| 2 | 93.1 tok/s | 75.3 tok/s | 83.5 tok/s | 19.2 tok/s |
| 4 | 171.9 tok/s | 135.8 tok/s | 164.0 tok/s | 22.2 tok/s |
| 8 | 333.2 tok/s | 233.3 tok/s | 309.7 tok/s | 24.2 tok/s |
| 16 | 598.9 tok/s | 363.5 tok/s | 562.7 tok/s | 32.6 tok/s |
| 32 | 970.3 tok/s | 492.5 tok/s | 921.6 tok/s | 120.2 tok/s |
Total Throughput Against Per-User Speed
The trade every serving engine makes: pack in more clients and the server produces more tokens in total, but each client’s answer streams out slower. This is the curve the independent inference benchmarks publish - per-user speed across the bottom, total throughput up the side. Both axes are better when higher, so up and to the right wins. An engine is only genuinely better if it serves more tokens overall and streams faster to each client.
Balanced load
1,024 tokens in · 1,024 tokens out · each point is a client count
Server throughput (tokens/s)
BESTWORSTNumbers on the Paddock line are client counts. llama.cpp is dashed because it serves Q8_0 rather than FP8.
Table view
| Clients | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 47.7 tok/s · 47.9/client | 45.0 tok/s · 45.2/client | 45.0 tok/s · 44.9/client | 33.7 tok/s · 45.5/client |
| 2 | 93.9 tok/s · 47.4/client | 83.9 tok/s · 42.4/client | 85.1 tok/s · 43.1/client | 48.0 tok/s · 42.1/client |
| 4 | 172.6 tok/s · 43.9/client | 162.5 tok/s · 41.3/client | 167.1 tok/s · 42.2/client | 60.4 tok/s · 37.6/client |
| 8 | 334.1 tok/s · 42.7/client | 321.1 tok/s · 41.5/client | 321.9 tok/s · 41.2/client | 64.9 tok/s · 17.5/client |
| 16 | 591.1 tok/s · 37.8/client | 578.7 tok/s · 38.1/client | 581.0 tok/s · 37.7/client | 109.7 tok/s · 9.7/client |
| 32 | 968.1 tok/s · 31.0/client | 968.1 tok/s · 32.7/client | 835.0 tok/s · 32.8/client | 239.1 tok/s · 10.4/client |
| 64 | 970.4 tok/s · 30.8/client | 963.9 tok/s · 31.9/client | 899.3 tok/s · 32.6/client | 265.2 tok/s · 9.9/client |
Waiting for the First Token
How long a client waits before anything appears, as the load climbs. The line is the median wait; the 90th percentile is in the table beneath it. Lower is better, and the axis is logarithmic - each gridline is ten times the one below it, because no straight scale holds a tenth of a second and ten seconds in one frame.
Balanced load
1,024 tokens in · 1,024 tokens out
Time to first token, log scale
BESTWORSTTable view
| Clients | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 120 ms · p90 120 ms | 146 ms · p90 149 ms | 114 ms · p90 115 ms | 8.0 s · p90 8.3 s |
| 2 | 240 ms · p90 244 ms | 232 ms · p90 278 ms | 190 ms · p90 200 ms | 19 s · p90 19 s |
| 4 | 403 ms · p90 425 ms | 367 ms · p90 522 ms | 311 ms · p90 353 ms | 41 s · p90 41 s |
| 8 | 549 ms · p90 727 ms | 694 ms · p90 996 ms | 656 ms · p90 683 ms | 74 s · p90 75 s |
| 16 | 635 ms · p90 1.3 s | 1.5 s · p90 1.6 s | 1.1 s · p90 1.3 s | 24 s · p90 132 s |
| 32 | 697 ms · p90 2.3 s | 2.5 s · p90 3.1 s | 1.9 s · p90 2.5 s | 6.9 s · p90 146 s |
| 64 | 34 s · p90 35 s | 36 s · p90 37 s | 35 s · p90 36 s | 115 s · p90 239 s |
Agentic sessions
Shared-prefix sessions × 6 turns
Time to first token, log scale
BESTWORSTTable view
| Sessions | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 82 ms · p90 83 ms | 359 ms · p90 549 ms | 86 ms · p90 97 ms | 10 s · p90 10 s |
| 2 | 100 ms · p90 119 ms | 670 ms · p90 879 ms | 143 ms · p90 146 ms | 20 s · p90 21 s |
| 4 | 106 ms · p90 156 ms | 1.2 s · p90 1.6 s | 176 ms · p90 208 ms | 38 s · p90 41 s |
| 8 | 145 ms · p90 210 ms | 1.9 s · p90 2.3 s | 266 ms · p90 324 ms | 36 s · p90 78 s |
| 16 | 165 ms · p90 242 ms | 2.4 s · p90 3.2 s | 408 ms · p90 490 ms | 43 s · p90 138 s |
| 32 | 245 ms · p90 351 ms | 2.7 s · p90 3.9 s | 646 ms · p90 1.0 s | 5.4 s · p90 162 s |
Agentic sessions share a prefix on purpose - reuse is the thing being measured. Paddock reports a cache hit rate of 0.95-0.98 and llama.cpp 0.88-0.97. vLLM’s prefix cache reports 0.0 here, and the wait shows it - vLLM climbs to 2.7 s at the top of the ladder. SGLang does not report cache statistics by default, so its 0.0 is unreported rather than disproven.
Every Shape on the Board
The four prompt/answer shapes NVIDIA, vLLM and SGLang publish, at 1, 8 and 32 concurrent clients, plus a vision probe. Longer bars are better, and each panel is scaled to itself. The badge on an engine is how many times faster Paddock is on that scenario - a grey badge below 1.0x means that engine won it. Each is rated against the Paddock row in its own weight class: llama.cpp against the row reading the identical Q8_0 file, vLLM and SGLang against the FP8 row.
Short chat
128 tokens in · 128 tokens outShort turns, where per-request overhead shows up.Summarization
2,048 tokens in · 128 tokens outPrefill-bound: most of the work is reading the prompt.Long generation
128 tokens in · 2,048 tokens outDecode-bound: most of the work is writing the answer.Balanced
1,024 tokens in · 1,024 tokens outThe even mix, halfway between reading the prompt and writing the answer.Vision document Q&A
One page image + questionA controlled probe of the vision path, not an industry dataset.Where Paddock Wins and Where It Loses
The same scenarios as a margin against Paddock. The centre line is Paddock; a bar to the right means Paddock is ahead, a bar to the left means it is behind. Losses are labelled - they are the reason to read this chart.
Against vLLM and SGLang
Throughput per scenario, relative to Paddock on the byte-identical FP8 checkpoint
The scale snaps to a fixed ladder rather than fitting the data - ±20% here - so a small margin looks small. Every cell Paddock drops is at 32 clients at once, against vLLM and SGLang.
Against llama.cpp
Same Q8_0 GGUF file, both engines - how many times faster Paddock is
Requests That Actually Met the Target
Throughput counts tokens whether or not anyone is still waiting for them. Goodput counts only the requests that met both halves of a declared target - first token within 2.0 s and 50 ms per token after that. It is the number that decides how many clients a server can really take. Higher is better.
Goodput under load
1,024 tokens in · 1,024 tokens out, target 2.0 s / 50 ms
Requests meeting the SLO
BESTWORSTTable view
| Clients | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 100.0% | 100.0% | 100.0% | 0.0% |
| 2 | 100.0% | 100.0% | 100.0% | 0.0% |
| 4 | 100.0% | 100.0% | 100.0% | 0.0% |
| 8 | 100.0% | 100.0% | 100.0% | 0.0% |
| 16 | 100.0% | 100.0% | 100.0% | 0.0% |
| 32 | 87.9% | 25.0% | 52.3% | 0.0% |
| 64 | 7.0% | 4.3% | 8.2% | 0.0% |
One scenario costs us this target: summarization · 32 clients, where the admission wave blows the 2.0 s budget (Paddock 8.3%, vLLM 0.5%, SGLang 2.6%, llama.cpp 0.0%). Under the goodput lens the ranking can flip against raw throughput. We publish it because it points at a real thing to fix.
The Whole Board
Every scenario, every engine, one table - throughput with median time to first token beneath it.
| Scenario | Paddock (FP8) | Paddock (Q8_0) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|---|
| Short chat · 1 client | 47.3 · 61 ms | 47.2 · 61 ms | 43.9 · 98 ms | 44.1 · 76 ms | 40.7 · 348 ms |
| Short chat · 8 clients | 325.6 · 150 ms | 326.1 · 149 ms | 314.1 · 229 ms | 323.1 · 122 ms | 148.1 · 2.8 s |
| Short chat · 32 clients | 942.2 · 224 ms | 944.2 · 223 ms | 967.8 · 453 ms | 996.6 · 395 ms | 155.6 · 9.8 s |
| Summarization · 1 client | 44.9 · 196 ms | 45.0 · 195 ms | 42.3 · 267 ms | 42.6 · 190 ms | 11.9 · 8.0 s |
| Summarization · 8 clients | 232.3 · 659 ms | 232.5 · 659 ms | 213.2 · 1.5 s | 231.6 · 1.1 s | 13.6 · 34 s |
| Summarization · 32 clients | 424.2 · 730 ms | 425.1 · 729 ms | 417.5 · 2.4 s | 426.0 · 3.6 s | 64.8 · 4.5 s |
| Long generation · 1 client | 47.9 · 61 ms | 47.9 · 61 ms | 45.2 · 101 ms | 44.9 · 82 ms | 37.0 · 10 s |
| Long generation · 8 clients | 343.5 · 150 ms | 343.4 · 148 ms | 331.0 · 219 ms | 329.6 · 122 ms | 119.1 · 70 s |
| Long generation · 32 clients | 1,048.7 · 241 ms | 1,048.8 · 246 ms | 1,068.5 · 465 ms | 1,058.6 · 435 ms | 344.3 · 11 s |
| Balanced · 1 client | 47.7 · 120 ms | 47.7 · 120 ms | 45.0 · 146 ms | 44.9 · 114 ms | 32.9 · 8.7 s |
| Balanced · 8 clients | 334.3 · 530 ms | 334.2 · 528 ms | 319.9 · 956 ms | 322.5 · 653 ms | 71.4 · 73 s |
| Vision document Q&A · 1 client | 45.2 · 185 ms | 45.2 · 184 ms | 43.4 · 143 ms | 43.8 · 92 ms | 11.1 · 8.9 s |
| Vision document Q&A · 8 clients | 307.7 · 390 ms | 307.7 · 389 ms | 285.6 · 578 ms | 304.5 · 295 ms | 21.7 · 44 s |
How It Was Run
Machine and class
- GPU
- RTX PRO 6000 (sm_120, 96 GB, 600 W)
- Date
- 2026-08-08
- Class
- speculative decoding off on every engine, read back from each serve config, 8-bit KV everywhere (paddock/vLLM/SGLang fp8-e4m3; llama.cpp q8_0 - ggml has no fp8 KV), equal-length generations, closed-loop, one engine at a time, same client
- Target
- First token ≤ 2.0 s, 50 ms per token after
One engine on the GPU at a time, verified idle in between, ~40 s warmup before the first measurement, same client driving all 5. Speculative decoding is off on every engine, including Paddock, so nothing is compared against an engine running without it.
Memory reserved
- Paddock (FP8)
- 74.8 GB peak
- Paddock (Q8_0)
- 74.7 GB peak
- vLLM (FP8)
- 82.7 GB peak
- SGLang (FP8)
- 91.1 GB peak
- llama.cpp (Q8_0)
- 41.4 GB peak
Reserved, not consumed: each engine takes the share of the card its own memory setting tells it to.
Configuration
- Paddock (FP8)
- 7318490e2026-08-07FP8 byte-passthrough (byte-identical checkpoint to vLLM/SGLang) + fp8-e4m3 KV + EXACT f32 GDN state, --no-spec --max-batch 32 --max-ctx 8192
- Paddock (Q8_0)
- 7318490e2026-08-07Q8_0 GGUF default stack (same GGUF file llama.cpp serves) + fp8-e4m3 KV + EXACT f32 GDN state, --no-spec --max-batch 32 --max-ctx 8192
- vLLM (FP8)
- v0.26.0newest releaseofficial FP8 safetensors, --kv-cache-dtype fp8, --max-model-len 8192 --gpu-memory-utilization 0.85 --max-num-seqs 32 (speculative decoding off per its own log)
- SGLang (FP8)
- main@0ba46c88e2026-08-03; newer than release v0.5.16official FP8 safetensors, --kv-cache-dtype fp8_e4m3, --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.85 (speculative decoding off per its own log)
- llama.cpp (Q8_0)
- b10307fc3f10b38, newest release tagsame Q8_0 GGUF as the Paddock Q8_0 row + mmproj-BF16, -ngl 99 -fa on -c 262144 -np 32 -ctk q8_0 -ctv q8_0 --jinja
Check our numbers
Paddock is a plain OpenAI endpoint, so you can put these numbers to NVIDIA's aiperf on your own hardware, with a tool that is nobody's home advantage.