Qwen3.8 27B
The newest Qwen flagship - the best coding and agent quality in the catalog, reads images, with tunable reasoning depth.
Result
Throughput across the 13 fixed test scenarios. Paddock, vLLM and SGLang all serve the byte-identical official FP8 checkpoint. llama.cpp has no FP8 weight type to serve it with, so an FP8 checkpoint has to be dequantised when it is converted (llama.cpp's own converter has a --fp8-as-q8 flag for exactly that). It is measured here against the Paddock row reading the same Q8_0 file it reads. Speculative decoding is off.
vs vLLM
13 of 13
scenarios faster, 1.02x-1.19x · FP8 weights
vs SGLang
10 of 13
scenarios faster, up to +9.4% - behind on 2, level on 1 · FP8 weights
vs llama.cpp
13 of 13
scenarios faster, 1.5x-37.5x · Q8_0 weights
Throughput as the Load Climbs
What the server delivers, from one client up to a full batch. Higher is better. The shaded region is what Paddock delivers above the strongest of the others at each step, and the figure above the line is its multiple there.
Balanced load
1,024 tokens in · 1,024 tokens out
Server throughput (tokens/s)
BESTWORSTThe 64-client step is left off: past a max batch of 32 every engine is queueing rather than serving. It is in the table.
Table view
| Clients | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 48.4 tok/s | 45.0 tok/s | 45.1 tok/s | 26.5 tok/s |
| 2 | 93.3 tok/s | 83.5 tok/s | 86.7 tok/s | 35.3 tok/s |
| 4 | 174.6 tok/s | 162.0 tok/s | 169.7 tok/s | 43.8 tok/s |
| 8 | 340.0 tok/s | 319.9 tok/s | 327.5 tok/s | 51.9 tok/s |
| 16 | 621.6 tok/s | 576.5 tok/s | 590.6 tok/s | 70.4 tok/s |
| 32 | 1,062.3 tok/s | 958.1 tok/s | 844.2 tok/s | 193.0 tok/s |
| 64 | 1,062.6 tok/s | 956.4 tok/s | 915.9 tok/s | 230.9 tok/s |
Agentic sessions
Shared-prefix sessions × 6 turns
Server throughput (tokens/s)
BESTWORSTSessions share a prefix, so this ladder also measures whether an engine’s prefix cache engages.
Table view
| Sessions | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 48.0 tok/s | 44.2 tok/s | 44.5 tok/s | 11.2 tok/s |
| 2 | 93.2 tok/s | 81.0 tok/s | 84.2 tok/s | 13.0 tok/s |
| 4 | 175.9 tok/s | 152.5 tok/s | 166.2 tok/s | 15.0 tok/s |
| 8 | 345.0 tok/s | 284.1 tok/s | 319.0 tok/s | 17.3 tok/s |
| 16 | 642.4 tok/s | 467.5 tok/s | 573.5 tok/s | 23.6 tok/s |
| 32 | 1,136.3 tok/s | 677.6 tok/s | 914.6 tok/s | 100.6 tok/s |
Total Throughput Against Per-User Speed
The trade every serving engine makes: pack in more clients and the server produces more tokens in total, but each client’s answer streams out slower. This is the curve the independent inference benchmarks publish - per-user speed across the bottom, total throughput up the side. Both axes are better when higher, so up and to the right wins. An engine is only genuinely better if it serves more tokens overall and streams faster to each client.
Balanced load
1,024 tokens in · 1,024 tokens out · each point is a client count
Server throughput (tokens/s)
BESTWORSTNumbers on the Paddock line are client counts. llama.cpp is dashed because it serves Q8_0 rather than FP8.
Table view
| Clients | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 48.4 tok/s · 48.6/client | 45.0 tok/s · 45.3/client | 45.1 tok/s · 45.0/client | 26.5 tok/s · 44.4/client |
| 2 | 93.3 tok/s · 47.0/client | 83.5 tok/s · 42.2/client | 86.7 tok/s · 44.0/client | 35.3 tok/s · 41.4/client |
| 4 | 174.6 tok/s · 44.1/client | 162.0 tok/s · 41.2/client | 169.7 tok/s · 42.9/client | 43.8 tok/s · 36.7/client |
| 8 | 340.0 tok/s · 43.3/client | 319.9 tok/s · 41.3/client | 327.5 tok/s · 42.0/client | 51.9 tok/s · 29.2/client |
| 16 | 621.6 tok/s · 40.3/client | 576.5 tok/s · 38.0/client | 590.6 tok/s · 38.4/client | 70.4 tok/s · 7.6/client |
| 32 | 1,062.3 tok/s · 35.0/client | 958.1 tok/s · 32.5/client | 844.2 tok/s · 33.1/client | 193.0 tok/s · 10.1/client |
| 64 | 1,062.6 tok/s · 34.5/client | 956.4 tok/s · 31.6/client | 915.9 tok/s · 33.4/client | 230.9 tok/s · 9.7/client |
Waiting for the First Token
How long a client waits before anything appears, as the load climbs. The line is the median wait; the 90th percentile is in the table beneath it. Lower is better, and the axis is logarithmic - each gridline is ten times the one below it, because no straight scale holds a tenth of a second and ten seconds in one frame.
Balanced load
1,024 tokens in · 1,024 tokens out
Time to first token, log scale
BESTWORSTTable view
| Clients | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 123 ms · p90 123 ms | 152 ms · p90 156 ms | 112 ms · p90 114 ms | 16 s · p90 16 s |
| 2 | 180 ms · p90 310 ms | 232 ms · p90 281 ms | 187 ms · p90 187 ms | 34 s · p90 34 s |
| 4 | 274 ms · p90 464 ms | 389 ms · p90 547 ms | 343 ms · p90 353 ms | 66 s · p90 66 s |
| 8 | 428 ms · p90 836 ms | 717 ms · p90 1.1 s | 682 ms · p90 699 ms | 116 s · p90 122 s |
| 16 | 1.1 s · p90 1.6 s | 1.5 s · p90 1.7 s | 1.1 s · p90 1.3 s | 70 s · p90 208 s |
| 32 | 1.5 s · p90 2.9 s | 2.8 s · p90 3.5 s | 2.0 s · p90 2.6 s | 10 s · p90 266 s |
| 64 | 32 s · p90 33 s | 36 s · p90 37 s | 35 s · p90 35 s | 121 s · p90 357 s |
Agentic sessions
Shared-prefix sessions × 6 turns
Time to first token, log scale
BESTWORSTTable view
| Sessions | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 82 ms · p90 84 ms | 183 ms · p90 227 ms | 90 ms · p90 109 ms | 17 s · p90 17 s |
| 2 | 76 ms · p90 77 ms | 243 ms · p90 360 ms | 164 ms · p90 177 ms | 33 s · p90 34 s |
| 4 | 86 ms · p90 88 ms | 454 ms · p90 824 ms | 194 ms · p90 219 ms | 60 s · p90 64 s |
| 8 | 132 ms · p90 169 ms | 888 ms · p90 1.6 s | 298 ms · p90 345 ms | 55 s · p90 120 s |
| 16 | 149 ms · p90 171 ms | 1.6 s · p90 3.1 s | 442 ms · p90 641 ms | 46 s · p90 206 s |
| 32 | 189 ms · p90 204 ms | 3.0 s · p90 4.8 s | 886 ms · p90 1.6 s | 5.4 s · p90 251 s |
Agentic sessions share a prefix on purpose - reuse is the thing being measured. Paddock reports a cache hit rate of 0.98-1.00 and llama.cpp 0.88-0.97. vLLM’s prefix cache reports 0.0 here, and the wait shows it - vLLM climbs to 3.0 s at the top of the ladder. SGLang does not report cache statistics by default, so its 0.0 is unreported rather than disproven.
Every Shape on the Board
The four prompt/answer shapes NVIDIA, vLLM and SGLang publish, at 1, 8 and 32 concurrent clients, plus a vision probe. Longer bars are better, and each panel is scaled to itself. The badge on an engine is how many times faster Paddock is on that scenario - a grey badge below 1.0x means that engine won it. Each is rated against the Paddock row in its own weight class: llama.cpp against the row reading the identical Q8_0 file, vLLM and SGLang against the FP8 row.
Short chat
128 tokens in · 128 tokens outShort turns, where per-request overhead shows up.Summarization
2,048 tokens in · 128 tokens outPrefill-bound: most of the work is reading the prompt.Long generation
128 tokens in · 2,048 tokens outDecode-bound: most of the work is writing the answer.Balanced
1,024 tokens in · 1,024 tokens outThe even mix, halfway between reading the prompt and writing the answer.Vision document Q&A
One page image + questionA controlled probe of the vision path, not an industry dataset.Where Paddock Wins and Where It Loses
The same scenarios as a margin against Paddock. The centre line is Paddock; a bar to the right means Paddock is ahead, a bar to the left means it is behind. Losses are labelled - they are the reason to read this chart.
Against vLLM and SGLang
Throughput per scenario, relative to Paddock on the byte-identical FP8 checkpoint
The scale snaps to a fixed ladder rather than fitting the data - ±20% here - so a small margin looks small. Every cell Paddock drops is at 8 clients at once, against SGLang.
Against llama.cpp
Same Q8_0 GGUF file, both engines - how many times faster Paddock is
Requests That Actually Met the Target
Throughput counts tokens whether or not anyone is still waiting for them. Goodput counts only the requests that met both halves of a declared target - first token within 2.0 s and 50 ms per token after that. It is the number that decides how many clients a server can really take. Higher is better.
Goodput under load
1,024 tokens in · 1,024 tokens out, target 2.0 s / 50 ms
Requests meeting the SLO
BESTWORSTTable view
| Clients | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 100.0% | 100.0% | 100.0% | 0.0% |
| 2 | 100.0% | 100.0% | 100.0% | 0.0% |
| 4 | 100.0% | 100.0% | 100.0% | 0.0% |
| 8 | 100.0% | 100.0% | 100.0% | 0.0% |
| 16 | 100.0% | 90.6% | 100.0% | 0.0% |
| 32 | 82.6% | 34.4% | 64.1% | 0.0% |
| 64 | 3.5% | 3.6% | 7.8% | 0.0% |
One scenario costs us this target: summarization · 32 clients, where the admission wave blows the 2.0 s budget (Paddock 8.9%, vLLM 1.0%, SGLang 2.6%, llama.cpp 0.0%). Under the goodput lens the ranking can flip against raw throughput. We publish it because it points at a real thing to fix.
The Whole Board
Every scenario, every engine, one table - throughput with median time to first token beneath it.
| Scenario | Paddock (FP8) | Paddock (Q8_0) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|---|
| Short chat · 1 client | 48.0 · 56 ms | 47.9 · 56 ms | 44.2 · 95 ms | 44.3 · 79 ms | 29.6 · 1.4 s |
| Short chat · 8 clients | 317.5 · 164 ms | 318.4 · 165 ms | 310.4 · 265 ms | 322.2 · 170 ms | 106.8 · 5.5 s |
| Short chat · 32 clients | 1,034.9 · 582 ms | 984.9 · 576 ms | 921.2 · 834 ms | 967.1 · 552 ms | 49.7 · 51 s |
| Summarization · 1 client | 45.3 · 202 ms | 45.2 · 202 ms | 42.4 · 212 ms | 42.6 · 188 ms | 7.5 · 14 s |
| Summarization · 8 clients | 227.8 · 446 ms | 228.2 · 445 ms | 210.7 · 1.8 s | 232.6 · 1.1 s | 8.6 · 56 s |
| Summarization · 32 clients | 423.9 · 1.5 s | 423.9 · 1.7 s | 355.9 · 3.7 s | 422.7 · 3.6 s | 45.1 · 6.1 s |
| Long generation · 1 client | 48.6 · 57 ms | 48.5 · 56 ms | 45.2 · 99 ms | 45.2 · 81 ms | 32.3 · 17 s |
| Long generation · 8 clients | 350.7 · 158 ms | 350.7 · 158 ms | 329.8 · 239 ms | 335.7 · 169 ms | 87.0 · 120 s |
| Long generation · 32 clients | 1,175.0 · 750 ms | 1,168.5 · 745 ms | 1,065.2 · 564 ms | 1,073.7 · 518 ms | 255.1 · 64 s |
| Balanced · 1 client | 48.4 · 122 ms | 48.2 · 122 ms | 45.0 · 151 ms | 45.2 · 113 ms | 27.0 · 15 s |
| Balanced · 8 clients | 340.0 · 285 ms | 340.0 · 284 ms | 319.1 · 1.0 s | 328.0 · 680 ms | 52.0 · 119 s |
| Vision document Q&A · 1 client | 45.6 · 193 ms | 45.5 · 193 ms | 43.5 · 137 ms | 42.5 · 95 ms | 6.5 · 17 s |
| Vision document Q&A · 8 clients | 311.5 · 394 ms | 311.5 · 391 ms | 287.2 · 485 ms | 302.8 · 267 ms | 8.3 · 119 s |
How It Was Run
Machine and class
- GPU
- RTX PRO 6000 (sm_120, 96 GB, 600 W)
- Date
- 2026-08-22/23
- Class
- speculative decoding off on every engine, 8-bit weights everywhere; 8-bit KV everywhere (paddock/vLLM/SGLang fp8-e4m3; llama.cpp q8_0 - ggml has no fp8 KV), equal-length generations, closed-loop, one engine at a time, same client
- Target
- First token ≤ 2.0 s, 50 ms per token after
One engine on the GPU at a time, verified idle in between, ~40 s warmup before the first measurement, same client driving all 5. Speculative decoding is off on every engine, including Paddock, so nothing is compared against an engine running without it.
Memory reserved
- Paddock (FP8)
- 80.6 GB peak
- Paddock (Q8_0)
- 80.7 GB peak
- vLLM (FP8)
- 83.8 GB peak
- SGLang (FP8)
- 89.4 GB peak
- llama.cpp (Q8_0)
- 41.4 GB peak
Reserved, not consumed: each engine takes the share of the card its own memory setting tells it to.
Configuration
- Paddock (FP8)
- 47895b61d2026-08-22; incl. the entry-317 b=1 GEMV launch merge and #281 sampling electionsFP8 byte-passthrough (byte-identical checkpoint to vLLM/SGLang) + fp8-e4m3 KV + EXACT f32 GDN state, --spec off --max-batch 32 --max-ctx 8192 --mmproj
- Paddock (Q8_0)
- 47895b61d2026-08-22Q8_0 GGUF default stack (same GGUF file llama.cpp serves) + fp8-e4m3 KV + EXACT f32 GDN state, --spec off --max-batch 32 --max-ctx 8192 --mmproj
- vLLM (FP8)
- main@44fc57d7bprecompiled-CI-kernels venv; postdates release v0.27.1 and measured faster for this model - per-model election 2026-08-15official FP8 safetensors, --kv-cache-dtype fp8, --max-model-len 8192 --gpu-memory-utilization 0.85 --max-num-seqs 32
- SGLang (FP8)
- main@17313cf4b2026-08-20; contains newest release v0.5.18's fork point 0111b2903 - the 8 release-only cherry-picks do not touch this model; 2026-08-22official FP8 safetensors, --kv-cache-dtype fp8_e4m3, --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.85
- llama.cpp (Q8_0)
- b105812115b73d8, newest release tag at board time 2026-08-22 sm_120same Q8_0 GGUF as the Paddock Q8_0 row + mmproj-BF16, -ngl 99 -fa on -c 262144 -np 32 -ctk q8_0 -ctv q8_0 --jinja --top-k 20 --top-p 0.95 (the model's OWN generation_config values - fairness rule 5; llama.cpp's built-in defaults are its invention)
Check our numbers
Paddock is a plain OpenAI endpoint, so you can put these numbers to NVIDIA's aiperf on your own hardware, with a tool that is nobody's home advantage.