Back to deep dives

The 100,000 Sandbox Problem — Akshat Bubna, Modal CTO

The 100,000 Sandbox Problem — Akshat Bubna, Modal CTOInside Cognition's inference stack: RL, speculative decoding & DFlash
Watch full clip
  • AI Engineering, Software, And Developer Tooling
  • AI Infrastructure, Compute, Chips, And Energy
Thumbnail for The 100,000 Sandbox Problem — Akshat Bubna, Modal CTO
Image: The 100,000 Sandbox Problem — Akshat Bubna, Modal CTO

Audio deep dive

Listen to this deep dive

IntroductionSection 01

This Recap covers a *Latent Space* podcast interview and a follow-up technical conversation published by Modal. Hosts Shawn Wang and Vibhu Norby talk with Modal CTO Akshat Bubna about cloud systems for bursty inference, training jobs, and agent sandboxes. Modal inference lead Gani then talks with Cognition research lead Silas about RL execution, speculative decoding, DFlash, and automating inference optimization. One RL workload can demand 100,000 isolated sandboxes while the serving stack balances speed, cost, memory, and compute.

Modal rebuilt cloud infrastructure for bursty AI work and coding agentsSource4:55

That was really important because we really didn't want people to spend so much time writing YAML and it seemed like you could really condense the surface area of what you're doing: put it in code so you can actually operate on it just like you can operate in other code like build stuff that's more expressive and dynamic. And so, yeah, that was always a very important part. Yeah, we never really got push back for that because the nice thing about modal is you can bring whatever code you have and sure the DSL is that the sort of configuration layer for what hardware you're using, how you're scaling things up, but you still own the code.

Modal built a serverless runtime for specialized workloads that scale faster than conventional web services. Its code-first configuration now serves agents as well as developers.

  • Kubernetes was designed for slower-scaling web servers, while Modal’s customers needed accelerators, varied images, and sharp bursts.
  • Users keep hardware and scaling configuration beside their application code instead of maintaining extensive YAML.
  • Developers use a few decorators to expose that configuration through a live, self-provisioning runtime.
  • Agents can inspect problems through the CLI; people still interpret results and make judgment calls.

GPU snapshots help Modal launch as many as 100,000 sandboxesSource15:14

That's why you need a lot of burstiness for inference. But then people also do a lot of on demand training like for RL stuff; your rollouts are bursty, as you said. People also do a lot of batch jobs. So we'll see a lot of companies before they have a training run, they'll need thousands of GPUs to run encoding or something like that. And I think those things are much more bursty. Sandboxes are except when you're doing RL. RL is insanely bursty. Like when you're doing rollouts you sometimes need 100,000 sandboxes.

GPU snapshots shorten cold starts as Modal scales inference, training, and RL jobs. RL rollouts can create bursts of 100,000 sandboxes.

  • Custom audio, video, robotics, and biology models gave Modal its first product-market fit.
  • Many models across regions make autoscaling harder because demand peaks at different times.
  • A snapshot can preserve compiled PyTorch state and make the next GPU cold start faster.
  • Training preparation may need thousands of GPUs, while RL rollouts may need 100,000 sandboxes.

DFlash trades draft speed, acceptance, and concurrency across the inference frontierSource7:20

Generally with the ablations that we've done training different models with different techniques, we've made some improvements to the acceptance length. That is the number of accepted tokens each time that you draft a bunch of tokens. We've made the acceptance length a little bit higher than Eagle 3. And then the second thing that I would say is probably more dominant is the actual runtime of this draft model is significantly faster using the D flash method. That comes down to a couple of things. One of them is the architecture of D flash is diffusion based. It's a single step that produces the tokens that are drafted rather than the older techniques where those draft tokens are produced one by one over time. I would say the second part of that argument is as we've been able to increase the accept lengths, it matters more and more that the draft stage is this constant run time rather than something that gets longer the better your speculation is.

DFlash drafts tokens in parallel while the larger model verifies them. Its design shifts the balance among latency, throughput, cost, memory traffic, compute, and concurrency.

  • DFlash uses a small model to draft several tokens, then the large model verifies them in one batch.
  • Bubna reports a two-to-four-times speedup without reducing the verified model’s output quality.
  • A constant-time draft stage gains value as the model produces longer accepted spans.
  • DFlash keeps drafting efficient as concurrency or the number of speculative steps increases.
  • More verification work per model load can improve both throughput and latency.

Production inference depends on utilization, autoscaling, and request reliabilitySource4:03

Yeah, obviously everybody's talking about throughput per GPU, and I think it matters, but what we found is like a really important thing is also the utilization across the 24 hours of the day, cuz it's very cyclical, right? So, like your daily peak might be multiple times higher than the daily low. So, a 10 to 20% difference on throughput per GPU might be dominated by the 24/7 utilization. So, I think auto scaling has become a really key factor for us. I think in the end you probably need a combination of both, but I think auto scaling, especially for bursts and launches and like daily peaks, I think it's very important, and then obviously if you have a fixed cluster, you can also find ways to have other types of workloads you can schedule into the off-peak cycles.

Production inference must handle traffic cycles as well as model speed. Utilization, elasticity, tail latency, delivery guarantees, and exact benchmarks all affect the result.

  • Scale-to-zero and burst capacity can matter more than simply finding a GPU and serving a model.
  • Production systems must control tail latency and deliver each request at least once.
  • Daily demand can swing several times above its low point.
  • Fixed clusters can run other jobs during off-peak hours.
  • Autoscaling reduces the need to keep rearranging fixed capacity when traffic changes.

Production agents need colocated compute, private networking, and hard boundariesSource24:46

In AI workloads or machine learning workloads, it used to be like 8:1 GPU to CPU and now it's more like 1:1, which is interesting because of how much agents basically are blocked or call out to CPU-heavy stuff. The actual like limiting factor like swings back and forth from GPU to CPU a lot more than it used to be all GPU and then occasional CPU, GPU, CPU. And now it's like just constantly and you just have to collocate everything.

Agents move repeatedly among GPU inference, CPU tools, storage, and sandbox execution. Modal combines local components and private networking with enforceable security boundaries.

  • Production agents use sandboxes, persistent storage, and supporting services in one coordinated system.
  • One host says agent workloads shifted the GPU-to-CPU ratio from roughly 8:1 toward 1:1.
  • Private IPv6 addresses let workspace containers communicate without exposing the network to others.
  • Bubna rejects model-granted sandbox permissions when data exfiltration requires hard boundaries.
  • Teams building production agents need control over files, snapshots, networking, isolation, and GPU access.

Modal pools 17 cloud providers and plans capacity for cheaper batch workSource25:29

We've built this capacity pool that spans 17 cloud providers. So we're very good at running on various kinds of cloud capacity across the world. We see our differentiator in the software layer and being capital light and focusing on the software helps us move really fast. So far it's worked out well because there are so many other people building data centers that we're able to work effectively with them and again focus on what makes us special.

A software and reliability layer joins capacity from 17 cloud providers. Hardware commitments and flexible scheduling support cheaper batch work.

  • The global pool combines 17 providers and several kinds of cloud capacity, including NeoClouds.
  • Working with data-center builders keeps the company capital-light and focused on software.
  • Its reliability layer shields workloads from failures such as a GPU falling off the bus.
  • Capacity planners model reservation terms, GPU choices, regions, forecasts, and supply-chain changes.
  • A planned batch tier offers lower prices when customers can wait about 24 hours.

Agents now run reinforcement-learning and inference experiments inside ModalSource17:46

I think agents are really critical for research actually for us internally as well. We're using a lot of agents to actually iterate on inference optimizations themselves. We have this internal dashboard and database called auto inference where you actually have agents registering different serving configurations and running experiments on them through that schema. It's been really cool watching them run overnight even without human input. Something we did actually just last week is we realized there was something wrong with one of the benchmarks, so we wanted to update it. And we got Devin to go in and find all the relevant things to run again. And it basically ran a very large experiment sweep all by itself.

Cognition uses Modal’s elastic GPUs for RL rollouts, distributed training, and automated inference experiments. Agents run sweeps and monitor jobs, while researchers still supply ideas and diagnose failures.

  • Training infrastructure, rollouts, sandboxes, and inference must work together across thousands of GPUs.
  • Silas estimates 20 researchers can now match a former team of about 100 by running agents.
  • Modal’s harness lets agents profile alternatives, tune configurations, and compare H200 with B200 GPUs.
  • Devin runs its inference experiments inside Modal GPU sandboxes.
  • After a benchmark failed, Devin found the affected work and reran a large sweep.

Open models and media agents push infrastructure below a fixed APISource42:09

We are building primitives that make our users lives much easier. So I think for example with LM inference, thousands more companies are going to post train their own models and deploy open source models for inference. So we're thinking a lot about what is the best product shape for that and that involves everything from our training gym to then endpoints that get frontier level performance. Without having to talk to anyone it looks somewhat different on other verticals. We're also seeing a lot of real-time audio-video stuff in there which is why we're working on things like regional routing with fallbacks so you can get sort of GPUs that are as close to users as possible, so you get like low latency for video streaming.

Open models and real-time media require more control than a fixed API provides. Teams may need custom backends, regional GPU routing, and agents that orchestrate several models and tools.

  • Bubna expects thousands more companies to post-train and deploy open models.
  • Regional routing with fallbacks places GPUs near audio and video users to cut latency.
  • Starter code lets teams modify and run a complete backend instead of calling one endpoint.
  • One customer needs code-level changes because its model architecture is fully custom.
  • Wang describes video agents that use code, tools, and models to make work longer than one clip.

Coding agents are reshaping CI, SDKs, CLIs, and infrastructure interfacesSource53:40

There's a lot of wasted time in CI on preparing your artifacts and preparing your dependencies and whatnot. Obviously, build systems help with that. But if you have primitives that are like memory snapshot and restore, can you just run CI more efficiently?

Coding agents will create more CI work and need direct access to infrastructure. The company is adapting its SDKs, CLI, startup path, and benchmark around those workflows.

  • Memory snapshot-and-restore primitives could reduce time spent preparing CI artifacts and dependencies.
  • Python, Go, and TypeScript SDKs are available; agents favor TypeScript while model work remains Python-heavy.
  • Faster service startup and production deployment are part of the agent experience.
  • The Modal Bench records agent failures that may justify new features or CLI access to logs and metrics.

Tags

  • Open Source AI
  • Inference Infrastructure
Back to deep dives