Paddock Benchmarks
We measure Paddock against the engines people actually deploy, on the prompt shapes and metrics the industry publishes, and we publish the losses alongside the wins.
How We Test
Benchmarks are easy to get wrong in a flattering direction. These are the rules we hold ourselves to, and the reasons each one exists.
The Same Bytes on Every Engine
An engine serving a smaller version of a model will always look faster. So each one is measured against the Paddock configuration reading the identical weight file - and where no shared file exists, that comparison is run separately and labelled, never blended into one table.
The Shapes Everyone Publishes
Short chat, summarization, long generation and a balanced mix - the prompt/answer sizes NVIDIA, vLLM and SGLang publish - each at 1, 8 and 32 concurrent clients, plus a vision probe. Our numbers sit next to published ones instead of on a shape we chose.
Goodput, Not Just Throughput
Tokens per second counts tokens whether or not anyone is still waiting for them. We also report the share of requests that met a stated latency target, which is what decides how many clients a server can take - including where that ranks us second.
Nothing Cached, Nothing Shortened
Every prompt is unique, so no cache can serve a request twice. Every engine generates the same number of tokens - a short answer inflates throughput and has produced false numbers before. One engine on the GPU at a time, verified idle in between.
Every Request Recorded
Each request's own latency and token counts are kept, not just the round average, so every figure on a board comes from the individual measurements behind it rather than a summary we cannot re-check.
The Losses Too
Each board names the cells where another engine beat us, by how much, and what we think causes it. A benchmark that never loses is a benchmark that was designed to win.
Boards
Run your own numbers
Paddock is a plain OpenAI endpoint, so you can check any of this with NVIDIA's aiperf instead of a tool of ours. The benchmarking guide has the scenarios we run and the settings that decide whether a comparison means anything.