Gemma 4 31B
Google's strongest Gemma 4 model - a dense multimodal transformer with 5:1 sliding-window/global attention and a separate MTP drafter for speculative decoding.
Result
Throughput across the 13 fixed test scenarios. Paddock, vLLM and SGLang all serve the byte-identical official FP8 checkpoint. llama.cpp has no FP8 weight type to serve it with, so an FP8 checkpoint has to be dequantised when it is converted (llama.cpp's own converter has a --fp8-as-q8 flag for exactly that). It is measured here against the Paddock row reading the same Q8_0 file it reads. Speculative decoding is on.
vs vLLM
11 of 13
scenarios faster, up to +76.0% - behind on 2 · FP8 weights
vs SGLang
13 of 13
scenarios faster, 1.66x-2.91x · FP8 weights
vs llama.cpp
13 of 13
scenarios faster, 1.12x-7.2x · Q8_0 weights
Throughput as the Load Climbs
What the server delivers, from one client up to a full batch. Higher is better. The shaded region is what Paddock delivers above the strongest of the others at each step, and the figure above the line is its multiple there.
Balanced load
1,024 tokens in · 1,024 tokens out
Server throughput (tokens/s)
BESTWORSTThe 64-client step is left off: past a max batch of 32 every engine is queueing rather than serving. It is in the table.
Table view
| Clients | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 100.8 tok/s | 78.1 tok/s | 45.0 tok/s | 82.0 tok/s |
| 2 | 194.0 tok/s | 152.8 tok/s | 90.8 tok/s | 129.2 tok/s |
| 4 | 349.5 tok/s | 300.2 tok/s | 178.8 tok/s | 158.6 tok/s |
| 8 | 625.7 tok/s | 566.8 tok/s | 299.0 tok/s | 239.5 tok/s |
| 16 | 1,011.2 tok/s | 1,025.1 tok/s | 498.4 tok/s | 278.6 tok/s |
| 32 | 1,604.9 tok/s | 1,615.1 tok/s | 688.7 tok/s | 343.9 tok/s |
| 64 | 1,607.0 tok/s | 1,623.5 tok/s | 688.4 tok/s | 336.0 tok/s |
Agentic sessions
Shared-prefix sessions × 6 turns
Server throughput (tokens/s)
BESTWORSTSessions share a prefix, so this ladder also measures whether an engine’s prefix cache engages.
Table view
| Sessions | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 99.2 tok/s | 81.6 tok/s | 46.2 tok/s | 58.9 tok/s |
| 2 | 187.6 tok/s | 160.5 tok/s | 92.4 tok/s | 79.5 tok/s |
| 4 | 338.9 tok/s | 310.1 tok/s | 179.6 tok/s | 96.5 tok/s |
| 8 | 617.4 tok/s | 585.2 tok/s | 297.6 tok/s | 95.6 tok/s |
| 16 | 978.8 tok/s | 1,012.7 tok/s | 454.9 tok/s | 104.2 tok/s |
| 32 | 1,559.9 tok/s | 1,610.7 tok/s | 617.5 tok/s | 99.3 tok/s |
Total Throughput Against Per-User Speed
The trade every serving engine makes: pack in more clients and the server produces more tokens in total, but each client’s answer streams out slower. This is the curve the independent inference benchmarks publish - per-user speed across the bottom, total throughput up the side. Both axes are better when higher, so up and to the right wins. An engine is only genuinely better if it serves more tokens overall and streams faster to each client.
Balanced load
1,024 tokens in · 1,024 tokens out · each point is a client count
Server throughput (tokens/s)
BESTWORSTNumbers on the Paddock line are client counts. llama.cpp is dashed because it serves Q8_0 rather than FP8.
Table view
| Clients | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 100.8 tok/s · 105.3/client | 78.1 tok/s · 77.8/client | 45.0 tok/s · 44.7/client | 82.0 tok/s · 88.4/client |
| 2 | 194.0 tok/s · 99.2/client | 152.8 tok/s · 78.8/client | 90.8 tok/s · 46.5/client | 129.2 tok/s · 73.5/client |
| 4 | 349.5 tok/s · 89.9/client | 300.2 tok/s · 77.4/client | 178.8 tok/s · 46.0/client | 158.6 tok/s · 50.8/client |
| 8 | 625.7 tok/s · 80.9/client | 566.8 tok/s · 74.0/client | 299.0 tok/s · 39.5/client | 239.5 tok/s · 34.3/client |
| 16 | 1,011.2 tok/s · 66.4/client | 1,025.1 tok/s · 67.0/client | 498.4 tok/s · 32.4/client | 278.6 tok/s · 19.1/client |
| 32 | 1,604.9 tok/s · 53.1/client | 1,615.1 tok/s · 53.5/client | 688.7 tok/s · 22.4/client | 343.9 tok/s · 12.0/client |
| 64 | 1,607.0 tok/s · 51.7/client | 1,623.5 tok/s · 52.2/client | 688.4 tok/s · 21.9/client | 336.0 tok/s · 11.1/client |
Waiting for the First Token
How long a client waits before anything appears, as the load climbs. The line is the median wait; the 90th percentile is in the table beneath it. Lower is better, and the axis is logarithmic - each gridline is ten times the one below it, because no straight scale holds a tenth of a second and ten seconds in one frame.
Balanced load
1,024 tokens in · 1,024 tokens out
Time to first token, log scale
BESTWORSTTable view
| Clients | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 104 ms · p90 104 ms | 109 ms · p90 132 ms | 176 ms · p90 176 ms | 878 ms · p90 900 ms |
| 2 | 142 ms · p90 206 ms | 162 ms · p90 210 ms | 174 ms · p90 249 ms | 1.0 s · p90 1.6 s |
| 4 | 165 ms · p90 362 ms | 163 ms · p90 354 ms | 174 ms · p90 446 ms | 1.3 s · p90 3.2 s |
| 8 | 171 ms · p90 472 ms | 167 ms · p90 673 ms | 182 ms · p90 893 ms | 1.9 s · p90 7.2 s |
| 16 | 217 ms · p90 957 ms | 240 ms · p90 945 ms | 195 ms · p90 1.7 s | 3.2 s · p90 13 s |
| 32 | 257 ms · p90 2.0 s | 271 ms · p90 1.8 s | 413 ms · p90 3.1 s | 4.8 s · p90 22 s |
| 64 | 20 s · p90 21 s | 20 s · p90 21 s | 47 s · p90 49 s | 100 s · p90 104 s |
Agentic sessions
Shared-prefix sessions × 6 turns
Time to first token, log scale
BESTWORSTTable view
| Sessions | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 45 ms · p90 226 ms | 79 ms · p90 96 ms | 130 ms · p90 159 ms | 1.6 s · p90 1.6 s |
| 2 | 73 ms · p90 226 ms | 125 ms · p90 128 ms | 129 ms · p90 158 ms | 2.1 s · p90 3.1 s |
| 4 | 117 ms · p90 278 ms | 129 ms · p90 192 ms | 128 ms · p90 170 ms | 1.9 s · p90 5.2 s |
| 8 | 85 ms · p90 370 ms | 140 ms · p90 198 ms | 167 ms · p90 215 ms | 4.8 s · p90 9.6 s |
| 16 | 163 ms · p90 535 ms | 203 ms · p90 283 ms | 232 ms · p90 326 ms | 5.7 s · p90 12 s |
| 32 | 177 ms · p90 800 ms | 307 ms · p90 378 ms | 355 ms · p90 508 ms | 10.0 s · p90 67 s |
Agentic sessions share a prefix on purpose - reuse is the thing being measured. Paddock reports a cache hit rate of 0.82-0.99 and llama.cpp 0.90-0.99. vLLM’s prefix cache reports 0.0 here, though on this architecture it costs little - its wait stays level with ours. SGLang does not report cache statistics by default, so its 0.0 is unreported rather than disproven.
Every Shape on the Board
The four prompt/answer shapes NVIDIA, vLLM and SGLang publish, at 1, 8 and 32 concurrent clients, plus a vision probe. Longer bars are better, and each panel is scaled to itself. The badge on an engine is how many times faster Paddock is on that scenario - a grey badge below 1.0x means that engine won it. Each is rated against the Paddock row in its own weight class: llama.cpp against the row reading the identical Q8_0 file, vLLM and SGLang against the FP8 row.
Short chat
128 tokens in · 128 tokens outShort turns, where per-request overhead shows up.Summarization
2,048 tokens in · 128 tokens outPrefill-bound: most of the work is reading the prompt.Long generation
128 tokens in · 2,048 tokens outDecode-bound: most of the work is writing the answer.Balanced
1,024 tokens in · 1,024 tokens outThe even mix, halfway between reading the prompt and writing the answer.Vision document Q&A
One page image + questionA controlled probe of the vision path, not an industry dataset.Where Paddock Wins and Where It Loses
The same scenarios as a margin against Paddock. The centre line is Paddock; a bar to the right means Paddock is ahead, a bar to the left means it is behind. Losses are labelled - they are the reason to read this chart.
Against vLLM and SGLang
Throughput per scenario, relative to Paddock on the byte-identical FP8 checkpoint
The scale snaps to a fixed ladder rather than fitting the data - ±200% here - so a small margin looks small. Every cell Paddock drops is at 32 clients at once, against vLLM.
Against llama.cpp
Same Q8_0 GGUF file, both engines - how many times faster Paddock is
Requests That Actually Met the Target
Throughput counts tokens whether or not anyone is still waiting for them. Goodput counts only the requests that met both halves of a declared target - first token within 2.0 s and 50 ms per token after that. It is the number that decides how many clients a server can really take. Higher is better.
Goodput under load
1,024 tokens in · 1,024 tokens out, target 2.0 s / 50 ms
Requests meeting the SLO
BESTWORSTTable view
| Clients | Paddock (FP8) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|
| 1 | 100.0% | 100.0% | 100.0% | 100.0% |
| 2 | 100.0% | 100.0% | 100.0% | 100.0% |
| 4 | 100.0% | 100.0% | 100.0% | 75.0% |
| 8 | 100.0% | 100.0% | 100.0% | 56.2% |
| 16 | 100.0% | 100.0% | 100.0% | 1.6% |
| 32 | 89.6% | 91.4% | 80.5% | 0.0% |
| 64 | 7.2% | 8.2% | 2.3% | 0.0% |
One scenario costs us this target: summarization · 32 clients, where the admission wave blows the 2.0 s budget (Paddock 15.6%, vLLM 37.8%, SGLang 3.1%, llama.cpp 0.0%). Under the goodput lens the ranking can flip against raw throughput. We publish it because it points at a real thing to fix.
The Whole Board
Every scenario, every engine, one table - throughput with median time to first token beneath it.
| Scenario | Paddock (FP8) | Paddock (Q8_0) | vLLM (FP8) | SGLang (FP8) | llama.cpp (Q8_0) |
|---|---|---|---|---|---|
| Short chat · 1 client | 113.2 · 35 ms | 112.7 · 36 ms | 82.9 · 62 ms | 51.0 · 97 ms | 87.4 · 200 ms |
| Short chat · 8 clients | 702.0 · 85 ms | 693.6 · 88 ms | 598.9 · 104 ms | 353.2 · 99 ms | 186.1 · 1.1 s |
| Short chat · 32 clients | 1,959.8 · 132 ms | 1,890.7 · 143 ms | 1,857.9 · 149 ms | 960.0 · 199 ms | 282.0 · 2.3 s |
| Summarization · 1 client | 103.2 · 184 ms | 96.2 · 204 ms | 68.2 · 190 ms | 41.2 · 288 ms | 46.9 · 1.5 s |
| Summarization · 8 clients | 344.8 · 247 ms | 344.7 · 239 ms | 332.8 · 401 ms | 198.0 · 368 ms | 49.3 · 6.8 s |
| Summarization · 32 clients | 518.0 · 437 ms | 487.4 · 707 ms | 527.7 · 1.1 s | 311.6 · 1.2 s | 67.7 · 10 s |
| Long generation · 1 client | 97.6 · 35 ms | 93.4 · 36 ms | 79.6 · 65 ms | 47.0 · 109 ms | 83.7 · 204 ms |
| Long generation · 8 clients | 664.1 · 73 ms | 638.0 · 75 ms | 601.6 · 106 ms | 321.1 · 110 ms | 342.2 · 489 ms |
| Long generation · 32 clients | 1,856.5 · 106 ms | 1,813.5 · 110 ms | 1,888.1 · 147 ms | 787.3 · 260 ms | 478.5 · 1.4 s |
| Balanced · 1 client | 102.6 · 103 ms | 95.4 · 114 ms | 77.6 · 132 ms | 46.1 · 176 ms | 81.5 · 825 ms |
| Balanced · 8 clients | 624.2 · 169 ms | 616.7 · 150 ms | 571.6 · 166 ms | 305.5 · 182 ms | 91.6 · 2.2 s |
| Vision document Q&A · 1 client | 158.9 · 51 ms | 144.6 · 52 ms | 90.3 · 123 ms | 54.6 · 112 ms | 81.4 · 448 ms |
| Vision document Q&A · 8 clients | 657.1 · 119 ms | 649.5 · 258 ms | 493.8 · 325 ms | 357.4 · 201 ms | 90.6 · 2.9 s |
How It Was Run
Machine and class
- GPU
- RTX PRO 6000 (sm_120, 96 GB, 600 W)
- Date
- 2026-08-03
- Class
- 8-bit weight class + speculative decoding on every engine; fp8-e4m3 KV on paddock/vLLM/SGLang, f16 KV on llama.cpp (its MTP path requires it); equal-length generations, closed-loop, one engine at a time, 40s warmup, same client, 2 measured rounds per scenario, best taken; every other engine upgraded to its newest release the same day
- Target
- First token ≤ 2.0 s, 50 ms per token after
One engine on the GPU at a time, verified idle in between, ~40 s warmup before the first measurement, same client driving all 5. Speculative decoding is on for every engine at the same draft depth, so no engine is measured without the thing Paddock is using.
Memory reserved
- Paddock (FP8)
- 88.6 GB peak
- Paddock (Q8_0)
- 88.5 GB peak
- vLLM (FP8)
- 83.7 GB peak
- SGLang (FP8)
- 82.7 GB peak
- llama.cpp (Q8_0)
- 90.6 GB peak
Reserved, not consumed: each engine takes the share of the card its own memory setting tells it to.
Configuration
- Paddock (FP8)
- a43c04d2026-08-03FP8 (official bf16 checkpoint via PADDOCK_FP8_NATIVE) + prefix cache + speculative decoding + fp8 KV, --max-batch 32
- Paddock (Q8_0)
- a43c04d2026-08-03Q8_0 GGUF default stack (same GGUF file llama.cpp serves) + speculative decoding + fp8 KV, --max-batch 32
- vLLM (FP8)
- v0.26.0release tag, source build, measured 2026-07-31fp8 online quant + gemma4_mtp k3 spec + fp8 KV, --max-num-seqs 32
- SGLang (FP8)
- v0.5.16release tag, source build, measured 2026-07-31fp8 + FROZEN_KV_MTP k3 spec + fp8_e4m3 KV, --max-running-requests 32
- llama.cpp (Q8_0)
- b10206release build, measured 2026-07-31Q8_0 GGUF + MTP (--spec-type draft-mtp n-max 3) + f16 KV + vision + --cache-ram 0, -np 32
Check our numbers
Paddock is a plain OpenAI endpoint, so you can put these numbers to NVIDIA's aiperf on your own hardware, with a tool that is nobody's home advantage.