Benchmarking

Paddock is an OpenAI-compatible server, so it measures with the tools everyone else already uses. We hold every claim to NVIDIA's aiperf as well as to our own sweep, and this page is how to run aiperf against Paddock yourself, including the settings that decide whether the resulting number means anything.

Why aiperf

A benchmark tool we wrote and shipped ourselves would be worth very little to you: you would be checking our engine with our client under our rules. aiperf is NVIDIA's, it is what people reach for when they compare inference servers, and it treats Paddock as one more OpenAI endpoint. If a claim only holds under our own measurement, it is not a claim worth making, so aiperf is a second scoreboard rather than a formality. It has cost us cells: it prices the admission path into throughput where a steady-state sweep under-weights it, and cells we win on one have gone against us on the other.

That is also why the server carries a Prometheus endpoint: aiperf scrapes /metrics during a run, and a server that does not answer it loses its column on someone else's scoreboard.

Install aiperf

python3 -m venv aiperf-venv
./aiperf-venv/bin/pip install aiperf
./aiperf-venv/bin/aiperf --version

Serve The Model

Serve one model, on one GPU, with nothing else running on the card. Give the server a batch width at least as wide as the concurrency you intend to drive, and a context window that fits the shape you are measuring:

paddock serve qwen3.6-27b --max-batch 32 --max-ctx 8192

aiperf sends no authorization header at all, which is why loopback callers never need a key. Benchmark from the machine running the server, or the run stops at the first request.

Run A Measurement

Put the run in a config file rather than on the command line. A scenario file is reviewable, diffable and rerunnable months later. This is a fixed-shape run of 128 input tokens and 128 output tokens at 32 concurrent clients, and it is the shape of the file we use ourselves:

schemaVersion: "2.0"
randomSeed: 7

benchmark:
  model: qwen3.6-27b
  endpoint:
    url: http://localhost:11540
    type: chat
    streaming: true
    timeout: 600.0
    extra:
      ignore_eos: true
      temperature: 0.7
      seed: 7
  tokenizer:
    name: Qwen/Qwen3.6-27B
  dataset:
    type: synthetic
    entries: 256
    prompts:
      isl: {mean: 128, stddev: 0}
      osl: {mean: 128, stddev: 0}
      corpus: coding
  warmup:
    type: concurrency
    requests: 32
    concurrency: 32
  profiling:
    type: concurrency
    requests: 192
    concurrency: 32
  artifacts:
    dir: ./artifacts/syn_128x128_c32
    summary: [json]
    records: [jsonl]

aiperf config validate --config syn_128x128_c32.yaml
aiperf profile --config syn_128x128_c32.yaml --ui simple

You get time to first token, inter-token latency, time per output token and output tokens per second, each as a full distribution rather than one average, plus a per-request record file the distributions can be re-derived from at any percentile.

The Settings That Decide Whether It Means Anything

Most benchmark disagreements are not disagreements about engines. They are two people measuring different things. These are the ones that have actually bitten us:

Server-Side Token Counts

aiperf defaults to client-side token counting and ignores what the server reports. Paddock sends a terminal usage chunk, so use_server_token_count: true gives exact counts if you want them. We leave the default alone in our own runs, because most people run aiperf bare and our published numbers should be the ones they will see.

Shapes Worth Measuring

Fixed input/output shapes, swept over concurrency, are what the field publishes, so a number measured this way sits next to other people's:

Sweep concurrency in powers of two from 1 to just past the server's batch width. Throughput and latency trade against each other, and a single concurrency figure hides which end of that trade an engine is good at.

Methodology

The published comparisons follow a fixed set of rules:

• Identical weights per engine. Apples to apples means every engine reads the same bytes. Each one is measured against the Paddock configuration serving the identical weight file; where no shared file exists - llama.cpp has no FP8 weight type, so an FP8 checkpoint can only be converted by dequantising it - that engine is read against the Paddock row on the weights it can serve, and the format is stated on the chart rather than left implied.
• The shapes the industry publishes. Fixed cells at 128×128, 2048×128, 128×2048 and 1024×1024 tokens, each at 1, 8 and 32 concurrent clients, plus concurrency ladders for the latency-throughput curves. These are the pairs NVIDIA, vLLM and SGLang publish, so the numbers sit next to published ones rather than on a shape we chose.
• Vanilla defaults. Every engine runs as it ships. We do not tune another engine down, and we do not tune ourselves up with switches a customer does not get; where a cell goes against us, the answer is an engine fix rather than a flag.
• Same machine, one engine at a time. Every engine runs on the same machine and GPU, never at the same time, driven by the same client code, with the GPU verified idle between engines.
• Unique inputs. Every benchmark input is globally unique. Repeated inputs let prompt and prefix caches re-serve earlier work and make an engine look faster than it is; inputs are salted to rule that out. Shared-prefix agentic sessions are the deliberate exception, because there the reuse is the measurement.
• Equal-length generations. Every engine generates the same number of tokens. An engine that stops early posts a higher tokens-per-second for doing less work, and this has produced false numbers before.
• One speculative-decoding class per board. Speculative decoding on and off are separate boards. Mixing them - ours with it, another without - produces a number that means nothing.
• Warmup, then the best timed round. Each fixed cell gets a warmup first, then the best of the timed rounds is reported. Curve points run once, trading best-of for breadth.
• Correctness before timing. Output quality is verified before a configuration is timed; the speed of something broken is not a result.
• Latency and goodput, not just throughput. Time to first token and time per output token are reported as full distributions, alongside the share of requests that met a stated latency target - including where that ranks Paddock second.
• Per-request records, not round averages. Each request's own latencies and token counts are retained, so every published figure comes from the individual measurements behind it and can be re-derived at any percentile.
• Two scoreboards, not one. Our own sweep measures steady state; aiperf prices the admission path, cold-burst prefill and time to first token, into throughput. The two disagree on some cells by design, and a claim has to survive both.

Published Results

The boards comparing Paddock with other inference engines, one model at a time with the setup and the losses included, are published at truespar.com/paddock/benchmarks.

Publish yours too. The license grants the right to measure, benchmark and publish Paddock's performance and behaviour expressly, so you need no permission from us, and a result that goes against us is still a result worth having.

Measuring in the Studio

For quick, informal measurement, every chat turn in the Studio records its own time to first token, tokens per second, and peak GPU load in the turn's run details, and the telemetry dock shows live engine throughput and GPU state while a reply streams. That is a sanity check rather than a benchmark: one turn, one client, no controlled shape.