We're releasing Paddock - a high-performance inference server that runs open AI models on NVIDIA GPUs. Built in Rust, it gives accelerated performance on NVIDIA Blackwell, and your existing tools connect to it the same way they connect to OpenAI and Anthropic.
What it is
An AI model is a large file of numbers. To get answers out of it, something has to load that file onto a graphics card and do the calculations. That something is called an inference server, and Paddock is a high-performance inference server.
Inference servers can use various ways to optimize performance, such as batching, caching and optimizing what are called CUDA kernels, which are compiled and run on the GPU. Better optimized CUDA kernels turn into higher efficiency, faster tokens per second and lower time to first token. The more load you can handle, the higher the efficiency and the lower the cost.
The card costs the same whether it is busy or idle. So what you pay per answer comes down to how many answers you get out of it per hour: twice the throughput is half the cost, or twice as many people served before you buy a second card.
Why we built it
We are users of the popular Ollama, which packages the llama.cpp engine and supports older generations of GPUs as well as CPU-only inference. It was made for developers to test and use local models, it does that job well, and it is still our preferred method for wide support of GPUs and CPUs. What it was not built for is load, or the kind of optimization that cuts costs in production. On the same model file, one client at a time, Paddock is between 1.2 and 3.8 times faster than llama.cpp on tested models, and at 32 clients at once between 3 and 6.6 times faster.
On the other side are vLLM and SGLang, high-throughput inference servers that came out of academic research labs and are built on top of PyTorch, an open-source deep learning library. Both require significant knowledge to set up and run efficiently in production. They are overall the best out there. Against both, Paddock is faster in most of our tests, up to two or three times on some models.
llama.cpp, vLLM and SGLang all ship CUDA kernels of their own, and all three also build on shared libraries that the whole field uses: cuBLAS in llama.cpp's case, and CUTLASS, FlashAttention, FlashInfer and Triton across vLLM and SGLang, with PyTorch underneath those two. That is a reasonable choice, and it is what lets them cover so much hardware for so much less effort. The cost is that the fast path is shared. It is tuned for the general case across many models and many GPUs, rather than cut to fit the one model on the one card in front of you.
We went the other way. Paddock has no Python, no tensor framework, no cuBLAS, no CUTLASS and no Triton. Every kernel is ours, written per model and per GPU generation, and that is where the speed comes from. To demonstrate this we packaged the first version as two programs in one folder: the manager, which you can skip entirely if you want to run headless, and the runner. The runner carries everything it needs to serve a model, kernels included, so running headless means pointing it at a model file you already have.
Paddock was built for the new generation of cards, the ones with 32 GB to 192 GB of memory that hold one large model or several smaller ones. Our focus during 2026 is the GeForce RTX 5090, the RTX PRO 4500, 5000 and 6000 Blackwell line, and B200.
For older generations, Ada and Hopper cards, the kernels are in the build and they will very likely run, and we have run on them ourselves, but we have not finished measuring them, so Paddock refuses them when it starts rather than serve numbers nobody has checked. There is a switch to force it for testing.
How fast
Our two most recent boards are Qwen 3.6 27B and Gemma 4 31B, both on an RTX PRO 6000. On the Qwen board, against vLLM, Paddock was faster in 11 of 13 scenarios, by up to 9%, and slower in 2. Against SGLang, faster in 9 of 13, by up to 7.3%, and slower in 2. Against llama.cpp, faster in all 13, by between 1.16 and 17.1 times. The Gemma board reads much the same, including two more scenarios where vLLM beat us.
NVIDIA's aiperf is the tool people reach for when they compare inference servers. You can run it against Paddock yourself, and we would like you to. Our benchmarking guide has the scenarios and, more importantly, the handful of settings that decide whether the number you get means anything at all. Test from cold, with nothing else on the GPU, and tell us what you find.
Your existing tools already work
Paddock is compatible with OpenAI and Anthropic APIs. For most software, pointing it at Paddock is one line: change the address it calls.
Documents, pictures and speech
Models are text or vision machines. Real work arrives as a contract, a spreadsheet, a scan, a recording. Paddock does that conversion on the way in, and tries to hand over the things a model cannot work out for itself. Vision models analyze pixels.
- Word documents arrive with tracked changes resolved, so the model reads the current text rather than a muddle of both versions, and is told that edits were present.
- Spreadsheets arrive as tables of computed values. A model shown a formula cannot evaluate it; a model shown the result can use it.
- PDFs and scans are turned into page images for a model that can see, or read into text for one that cannot.
- Photos bring what the picture cannot show: when it was taken, what took it, and where, if the camera recorded that.
- Recordings become text, from a file or a live microphone, in several languages, with per-word timings where you need them.
None of this needs extra software installed, and none of it is a separate product. We call it context enrichment: giving the model the things a person reading the same file would notice and a model on its own would not.
The Studio
Start Paddock and open your browser. Download models, run them, chat with them, give them web search and other tools, and read what the graphics card is doing while it works.
The feature we use most is asking the same question of two models at once and reading both answers side by side, with the time each took shown underneath. You can put a model running on your own machine next to a hosted one on your own account, which turns "is the local one good enough for this" from an opinion into something you can look at.
Early release
This is an early release. Expect frequent updates, more models, and each model getting faster than it is today. There is a feedback button in the Studio that reaches our development team directly, and we read all of it.
It runs on Windows and Linux, both 64-bit.
What it costs
Paddock is free for individuals and companies, with no usage limits. Paid plans exist for organisations that need a support agreement or the source code for audit purposes.
Getting started
# start the manager and Studio
paddockThe banner will list the URL. Open it in your browser and pick a model.
Download Paddock, read the benchmarks, or get in touch if you want us to measure a card you care about.