Concurrency Limits: The Hidden Tax in Voice AI (and how we solved it!)

Introduction

At Phonic, we're building a platform that lets anyone deploy voice agents that are as good as humans. Naturally, you might hear this and think about our audio model — how natural it sounds, whether it handles interruptions well, whether it can reason through a hard conversation. Of course, we obsess over every detail there. But there's a less obvious dimension that matters too: how do we actually serve millions of voice agents at once?

Voice agents free teams from human constraints — in theory, you can parallelize millions of user research interviews, hiring screens, or customer support calls all at the same time.

In practice, most voice AI platforms enforce concurrency limits, which are the number of calls their customers can run in parallel.

Every voice AI provider purports to have high concurrency limits. Not many of them get into what happens when you actually hit those limits mid-spike. The answer is that new GPUs take minutes to come online so you end up with failed calls and error messages. If cold starts take 3-6 minutes, voice AI providers can either predict spikes and pre-scale which is basically guesswork, or keep idle GPUs running 24/7 which is expensive — if they can even get the GPUs in the first place.

Most providers do some mix of both and hope their customers don't all spike at the same time. At Phonic, we start new voice containers absurdly quickly, at a fraction of the time of everyone else in the industry?— this enables us to pass the benefits to our users: Phonic can support higher concurrency limits than many other providers, while consuming fewer resources to do so.

How did we get to 26 second cold starts?

Before we could make cold starts faster, we needed to know what was making them slow.

This is pretty easy to analyze — we ran a quick profile of cold start time on a fresh container. Just like everyone assumed, it was dominated by compiling the model — a process that improves the model’s latency at the expense of just-in-time compiling the model at initialization.

So the obvious question was could we make compilation faster?

Step 1: Caching the compilation artifacts

The first thing we reached for was caching. If torch.compile regenerates the same kernels on every boot, why not persist them? We persisted the CUDA, Triton, and PyTorch’s Inductor cache across an object store so new containers could pull them instead of recompiling from scratch. It worked — cold start dropped from 5.6 minutes to 2.5 minutes.

Step 2: Reducing recompilations

Caching cut cold starts by more than half — but 2.5 minutes still wasn't fast enough for when we need a new container ready mid-spike. When we dug into where the remaining time was going, we found that a significant chunk wasn't compilation that we hadn't cached. It was compilation we'd already done, happening again. The cache was being invalidated and recompiled more often than it needed to be.

If we set TORCH_LOGS=recompiles, the problem became obvious enough:

torch/_dynamo/guards.py:2791] [0/1] [__recompiles] Recompiling function _embed_and_mask_input in /root/echo/models/models.py:180
torch/_dynamo/guards.py:2791] [0/1] [__recompiles]     triggered by the following guard failure(s):
torch/_dynamo/guards.py:2791] [0/1] [__recompiles]     - 0/0: tensor 'L['tokens']' size mismatch at index 1. expected 1, actual 64

In PyTorch, guards are assumptions on shapes that the model will encounter during inference. By placing assumptions, it can focus on a serving a smaller set of shapes via specialized kernels.

When the guards are invalidated, the model must recompile all relevant kernels for an updated set of shapes — substantially increasing the compilation time.

Thankfully, we can give PyTorch a compiler hint that the tokens tensor will have a dimension greater than one with a one-liner of code: torch._dynamo.decorators.mark_unbacked(tokens, 1)

We set up compiler hints for every tensor that invalidates guards and re-ran the benchmark:

Step 3: Zooming out

After addressing some low-hanging fruit, we were still at 2 minutes of compilation time. Given that our goal was to start the entire container within seconds, it seemed like optimizing compile times was going to be fundamentally asymptotic: we might be able to reduce by another factor of 2, but some amount of compilation would still exist — which was unacceptable.

If we zoomed out, compiling even a bit might be a bad idea. But what if we got rid of compilation during runtime entirely?

torch.compile is a just-in-time compiler, which translates PyTorch code into Triton (which is then compiled into PTX) at runtime. This is the default path to speed up PyTorch code!

Instead of compiling just-in-time, what if we compiled the models ahead-of-time, and just ran the compiled binary? This would eliminate the entirety of torch.compile warmup and get us to that final goal of being able to initialize a new container in seconds.

Thankfully, torch.export does exactly this. It’s an ahead-of-time compiler that pre-compiles PyTorch modules into portable self-contained binaries designed to be executed anywhere.

It's not a free lunch and imposes several constraints:

1. Our PyTorch module should be entirely self-contained in PyTorch — it cannot defer control flow to Python or require interacting with the CPU during inference.
2. All control flow must be predetermined, it can’t depend on data (e.g. if x.shape[0] > 2  or x.item() > 5 are not allowed).
3. Our PyTorch module, which is traditionally a class, is now a function. As a result, our module cannot hold any state.

These tradeoffs were acceptable to us, so we make the necessary changes:

1. Previously, the KV caches used to be an attribute of the model (self.k_cache) . Instead, we refactor them into a service.
2. We remove any CPU ↔ GPU communication during model inference
3. We set up GitHub Actions runners to create compiled binaries for various deployment targets (Intel, AMD, etc).

The changes were well worth it — our model cold start time went from 5.6 minutes to 26 seconds, which easily lets us scale up during spiking traffic.

At 26 seconds, auto scaling stops being theoretical. When traffic spikes, new containers are ready before the spike is over.

So why doesn’t everyone do this? torch.export is an opinionated interface: it  only runs certain shapes for certain modules and needs to be precompiled for certain instruction sets. vLLM or SGLang have to build an interface that is compatible for the entire community and their needs; Phonic’s customized, latency-built serving stack only has to serve our customer’s needs. By focusing, we built a solution that handles cold starts >10x faster.

Obsessively Built for Voice Agents

By eliminating compilation entirely, we can pass the benefits to the you: Phonic’s serving architecture can structurally support more traffic than most other companies in voice.

This is why companies like Rho and Assort Health are choosing Phonic’s vertically integrated voice stack: we’re incredibly obsessive over the voice agent experience, from background noise handling, natural voices, an intuitive SDK, end-to-end observability, and even diving into the details of cold starts so we can support customers whose businesses 10x in traffic overnight.

Are you building in voice? We’d love to give you a product tour of the platform built for voice agents as good as humans. Schedule a demo with us here and we can chat more!