Automatically Optimizing Inference with Standard Kernel and CoreWeave

Standard Kernel and CoreWeave logos over a step chart of performance rising with search time

At Standard Kernel, we’ve been building autonomous systems to accelerate chip performance, with an approach designed to work across silicon architectures. In collaboration with CoreWeave, we're demonstrating this approach on NVIDIA GPUs for one of the most compute-intensive use cases today: LLM inference. In this blog post, we’ll highlight how we achieved up to 63% more users per node for GLM-5.3, 2.0x per-user throughput for Qwen3.8-27B, and 10% higher throughput for DeepSeek V4.1 Flash compared to stock vLLM and SGLang.

Overview

The best implementation for an LLM inference deployment depends on the model, the hardware, and the workload it serves. Standard inference engines like vLLM and SGLang offer decent performance out of the box, but they fall short of achieving maximal possible performance given the capabilities of the underlying hardware. Producing optimal performance requires careful work from performance engineers, and this work needs to be repeated as new models, hardware, and common use cases arrive.

There are a number of ways to improve performance on a given setup. For example, you could do something as simple as changing configuration flags to enable prefix caching. Or, you could go all the way down to the kernel level and rewrite the kernels to be more efficient for the given workload. And even if you produce speedups, faster individual components don't necessarily translate into faster end-to-end inference, and speedups are often short-lived as the market moves its focus to new hardware and models. That's all to say, optimizing inference is a difficult, nuanced, and expensive process.

At Standard Kernel, we've built systems to automate that process.

Our System

Our system starts with a model, a representative workload, and a performance goal for a given deployment. Given these inputs, it makes changes to the deployment’s serving stack and measures whether each change meaningfully helps achieve the performance goal. Rather than simply giving a coding agent a performance target, we've built efficient infrastructure to correctly target optimizations, explore optimizations in parallel, rapidly test correctness and performance, and accumulate knowledge from previous experiments. This lets the system search more possibilities and improve over time.

The optimization system: application source code and a benchmark go in; an orchestrator of baseline, analyst, parallel implementer and merger agents iterates; source code that scores better on the benchmark comes out

These changes range from adjusting existing engine settings to modifying the engine source code itself and writing new GPU kernels. We broadly categorize our changes into three categories:

Three categories of changes: engine configuration, engine source code, and GPU kernels, each with examples

Through this autonomous system, we achieve a custom configuration for each client, tuned to their model and their particular performance goals. Each run of our system needs minimal human involvement. We review the results afterward to make sure the improvements are real, rather than coming from reduced output quality or optimizations that work on the benchmark but not on actual customer traffic.

Our collaboration with CoreWeave brings together automated inference optimization and the GPU infrastructure needed to evaluate changes at scale. Even small improvements in inference efficiency can help teams get more from their GPU resources. CoreWeave’s infrastructure supports the experimentation needed to identify and validate those gains. Our system evaluates candidate changes through benchmarks and correctness checks, running multiple experiments in parallel across different GPU configurations. We use CoreWeave Kubernetes Service (CKS) to provision GPU resources on demand, allowing us to scale experimentation up or down as needed and test changes in realistic deployment environments.

Select Results

We showcase three model specific implementations with different optimization objectives.

Results relative to baseline: GLM-5.3 1.63x traffic served vs SGLang; Qwen3.8-27B NVFP4 2.00x output tokens/s per user vs vLLM; DeepSeek V4.1 Flash 1.10x coding-agent throughput vs vLLM

GLM-5.3 for 63% more traffic

We optimized GLM-5.3 using an hour of replayed production traffic from a customer. Compared to baseline SGLang, our optimized inference server supported 63% more traffic and 30% more simultaneous users per node under our customer’s SLA, while reducing median time per output token by about 25%. Accuracy was unchanged on MMLU-Pro, IFEval, and LongBench.

Qwen3.8-27B for 2x TPS

We optimized serving for the NVFP4 version of Qwen3.8-27B. The benchmark simulates multi-turn sessions whose context grows over time, and it measures throughput per user as the number of concurrent users increases. At both low and high concurrency, the optimized server reached 2.0x more output tokens per second per user compared to baseline vLLM. These figures include speculative decoding, which we enable adaptively at lower concurrency. Engine and kernel changes alone added 13% to 65% on top of a vLLM configuration we had already tuned for low and high concurrency respectively. Accuracy on GSM8K was comparable to stock vLLM.

DeepSeek V4.1 Flash for 10% more coding agent throughput

For this model, we optimized around a customer provided agentic coding benchmark, which uses long inputs of 70,000 to 90,000 tokens with heavy prefix reuse (95%+ cache hit rate), at an average of 16 concurrent requests. In this setting, our system achieved 10% higher throughput than vLLM with the recommended flags enabled. We validated output quality on customer-provided quality benchmarks, along with GSM8K, DeepSWE, and several long-horizon benchmarks, to confirm that no performance gain came at the cost of accuracy on either short-form reasoning or extended multi-step tasks.

Faster components don't always mean a faster server

During the runs we sometimes encounter changes that are faster in isolated tests and pass all of their correctness tests, but do not improve the overall performance. We saw an example of this with the DeepSeek V4.1 Flash model where the throughput was reduced by about 16% in a full serving setting. For this reason, the proper testing setup is very important. We observed that our system rejected 40% of changes that turned out to be false positives on DeepSeek V4.1 Flash.

Next Steps

We are continuing to apply the system to new models and workloads as they appear. Our approach is multi-silicon, and we will share more about our work on other chips soon.

If you serve open-weight models and would like to improve their performance on your workload, contact us at hi@standardkernel.com.