Benchmarking

Paddock ships a benchmark harness, paddock-bench, that measures throughput and time to first token against the engine directly or against any OpenAI-compatible server, so Paddock and other engines can be compared with the same client under the same rules.

paddock-bench

paddock-bench is a separate executable included with the release.

It has two measurement modes, plus a concurrency mode on top of the HTTP path:

Mode	What it measures
In-process	The Paddock engine directly, no HTTP in the way; needs a GGUF and, on CUDA, a kernel pack
HTTP	Any OpenAI-compatible server; point it at Paddock's own server and at other engines for a like-for-like comparison
Concurrency	Many streaming sessions fired at each endpoint at once, reporting aggregate throughput and latency distributions

In-Process Mode

paddock-bench --model ~/paddock/models/your-model.gguf \
  --pack pd-cuda-sm120.so --device cuda \
  --prompt-tokens 128 --decode-tokens 64

A synthetic prompt of --prompt-tokens tokens is prefilled, then --decode-tokens tokens are decoded and timed. The first --warmup decode tokens (default 8) are discarded so one-time setup cost does not pollute the steady-state number.

HTTP Mode

paddock-bench \
  --endpoint http://127.0.0.1:11540/v1 --name paddock \
  --endpoint http://127.0.0.1:8080/v1  --name other-engine \
  --endpoint-model your-model --decode-tokens 128

--endpoint is repeatable, so several servers can be compared in one run; --name labels align with the endpoints by position. Results print as one row per runner, and with more than one result the tool also prints the decode tokens-per-second ratio against the first.

Concurrency Mode

paddock-bench --endpoint http://127.0.0.1:11540/v1 \
  --concurrency 8 --decode-tokens 256 --unique-prefix

Flag	Description	Default
`--concurrency <N>`	Fire this many streaming sessions at once at each endpoint; 0 keeps the single-shot modes	`0`
`--shared-prefix-words <N>`	Length of the shared prompt prefix in words	`400`
`--stagger-ms <N>`	Stagger session arrivals so later sessions join while earlier ones decode, stressing chunked prefill	`0`
`--unique-prefix`	Give each session a distinct prefix: cold prefill, no cache sharing	off

Sweep Clients

The head-to-head reports are produced by Python sweep clients in the repository's docs/bench/ directory, which drive every engine with identical request patterns: a generation sweep over streaming chat completions across several concurrency and prompt-length configurations, and an embedding and reranking sweep over batch and thread configurations. Both salt every input so it is globally unique, and a companion script assembles the raw per-round output into the published report files.

Methodology

The published comparisons follow a fixed set of rules:

Same machine, one engine at a time. Every engine runs on the same machine and GPU, never at the same time, driven by the same client code.
Unique inputs. Every benchmark input is globally unique. Repeated inputs let prompt and prefix caches re-serve earlier work and make an engine look faster than it is; inputs are salted to rule that out.
Warmup, then the best timed round. Each configuration gets a warmup request first, then the best of the timed rounds is reported.
Correctness before timing. Output quality is verified before a configuration is timed; the speed of something broken is not a result.
Current engine versions. Competitor versions are checked at benchmark time and recorded in the report data.
Raw data published. The per-round measurements behind every chart are published alongside the reports.

Published Results

The head-to-head reports comparing Paddock with other inference engines, model by model with the setup and raw data included, are published at truespar.com/paddock/benchmarks.

Measuring in the Studio

For quick, informal measurement, every chat turn in the Studio records its own time to first token, tokens per second, and peak GPU load in the turn's run details, and the telemetry dock shows live engine throughput and GPU state while a reply streams.