Baseten on the inference frontier: 200,000-token routing, 4–6× speedups, and self-optimizing AI

This Latent Space podcast interview has host swyx asking Baseten’s Philip Kiely and Ali Taha how inference turns an open model into a product: request routing, quantization, GPU kernels, model parallelism, and AI video. It gets properly weird near the end, when GLM-5.2 helps rewrite its own serving code and continual learning starts to blur the line between training and inference.

A long request goes to an available prefill worker and, when possible, a replica that already holds some of its context. Separate decode workers can then generate and stream the response.

Watch the related Philip Kiely clip on X

• A workload-specific speculative model drafts tokens for the main model to accept or reject, raising decode speed when acceptance is high.
• Dedicated deployments give high-volume users more control over reliability, precision, batching, parallelism and traffic-specific speculation.
• Post-training and quantization must preserve valid tool-call JSON; bad training examples can make a model invent the result.
• Baseten uses a state machine or grammar to force the required output format, but the model can still call the wrong tool.

Producing one token from a new open model is easier than serving it through a reliable API. The remaining work spans runtimes, calibration, model retrofits and failures outside the weights.

• Providers may still need to add architecture support, quantize and calibrate the model, and train a speculator.
• One engineer joined a Kimi vision encoder to GLM-5.2 by training a small projector while freezing both models.
• The team replaced an inefficient attention layer and trained the retrofit to recover its speculative-token acceptance rate.
• Live traffic revealed repeated-token collapse, prompting the endpoint to stop or retry after at least four repetitions.
• Identical weights can behave differently across inference engines, kernels, hardware and cluster interconnects.

Quantization trades precision for speed, so the practical goal is to keep the optimized model close to the original. Several optimizations can produce 4–6× gains; 10× needs aggressive assumptions.

Watch the related Baseten inference optimization clip on X

• Baseten wants users to be unable to distinguish its optimized API from the model maker’s API.
• For image models, the team avoids quantizing modulation layers and output projections because both directly affect results.
• Choosing layers whose errors cancel reportedly enabled 20% more quantization and 20% more throughput than another provider.
• Speed comparisons change with hardware, load, prompt shape, response length and the chosen latency or throughput measure.
• With hardware and GPU count fixed, the speakers put combined optimization gains nearer 2–4×.

NVIDIA Dynamo coordinates state such as KV caches across inference hardware; installing it does not automatically make a model faster. Local and data-centre systems also start from different limits.

• Dynamo spans frameworks and hardware, moves cluster state and offers replaceable defaults, but deployment choices determine its value.
• Users can often reuse published quantized checkpoints and speculative decoders instead of building their own.
• Running a small speculative model beside the main model consumes shared hardware, and each extra recursive speculator adds more work.
• Local techniques can help data centres, but an optimization for laptop memory may slow a B200 when kernel overhead outweighs bandwidth savings.

Model parallelism trades memory capacity against communication, latency and throughput. Production teams must test configurations and kernels against real traffic instead of trusting a universal optimum.

• Tensor parallelism shards each step across GPUs, needs a fast interconnect and usually targets lower latency.
• Expert parallelism keeps experts whole and replicates the small router, reducing communication and potentially raising throughput.
• Pipeline parallelism splits layers across hardware and is mainly needed when one node cannot hold the model.
• The speakers recommend shadowing real traffic and sweeping kernel parameters to find the best measured setup.
• A fused mega kernel still cannot remove cross-GPU communication and may lose to easier-to-tune modular kernels.

Rubin systems must move KV caches and data across CPUs and GPUs, not merely run fast kernels. A 2.8-trillion-parameter example shows how quickly model weights consume rack memory.

• More low-precision compute and bandwidth make cache offloading, cache-aware routing and disaggregation more important.
• The speakers said AI-specific GPUs may reduce the need for ASICs, though they disagree and cannot yet predict which hardware will win.
• They estimate that 2.8 trillion NVFP4 parameters occupy about 1.4 terabytes, while each GB300 has 288 GB.
• Eight GB300s can hold those weights, but long-context KV caches reduce the remaining headroom.

Long video faces quadratic attention costs and consistency drift as visual tokens accumulate. Autoregressive generation could extend and stream output, perhaps paired with diffusion-style refinement, but today’s open systems remain weak.

• Media companies still choose stronger closed video models even when optimized open inference could cost much less.
• Five seconds of 480p video at 16 frames per second can yield about 35,000 latent tokens.
• Autoregressive video could generate images sequentially and reuse language-model optimizations, but current open results are worse.
• Extending a clip from its final image can darken or degrade it, while autoregression cannot revisit earlier images.
• The proposed autoregressive and diffusion combination is a qualified possibility, not an established design.

Training and inference increasingly depend on each other through faster rollouts, quantization-aware post-training and production feedback. GLM-5.2 has already profiled its serving path and rewritten bottleneck kernels, though serious limits remain.

Watch the related Ali Taha and GLM-5.2 clip on X

• Slow inference can bottleneck training, while low-precision models may need post-training or distillation to recover quality.
• The speakers expect production traces to feed continual post-training, testing and product improvement.
• GLM-5.2 identified SGLang bottlenecks and wrote replacement kernels, but models still show poor judgment and reward hacking.
• One speaker estimates direct node-to-node KV-cache transfers could approach 100× speedups, while acknowledging major uncertainty.
• Continual learning might retain updates through compacted KV caches or lightweight adapters instead of repeatedly fetching full weights.