Why a Finished Answer Still Has to Wait

What is continuous batching?

A scheduling technique for LLM inference that checks after every generation step which requests are done, frees their slots, and immediately admits new requests, instead of waiting for the whole batch to finish.

Eight chat requests start at the same time on the same server. One of them is a simple yes-or-no question. It is done after three tokens. Another is a long summary. It needs two hundred tokens in total.

The short answer still does not arrive any sooner. It waits until the long one is done. The short request’s slot simply sits unused until then.

A Slot That Frees Up, But Stays Empty

A server processes several requests together in a batch. That is meant to keep the GPU busy. At every generation step, it computes one more token for every request in the batch, at the same time.

If the batch runs as a fixed group, it only ends once every request is finished. If the short request reaches its end after three steps, the remaining 197 steps for the long request still keep running. The short request’s slot computes nothing more during those steps. It is free. It stays empty anyway, because the server only reassigns slots once the whole group ends.

This fixed group has a name: static batching.

The alternative refills a freed slot immediately with a waiting request, before the next step even begins. This mid-run refilling has its own name: continuous batching, sometimes called in-flight batching.

Here is where this term sits in the series:

Under the Hood

A static scheduler makes its decision once per batch: which requests start together. A continuous-batching scheduler makes it once per generation step. After every step, it checks which sequences have produced an end token or hit their length limit, removes them from the batch, and pulls waiting requests from the queue into the freed slots. The next step then runs with a partly different composition than the one before it.

Making that decision at the level of each individual step, instead of the whole request, is called iteration-level scheduling. The term comes from the Orca paper (OSDI 2022), which introduced it together with a second technique, selective batching: requests of different lengths can still be batched together for the linear compute steps, where every token is treated the same, but not for the attention computation, which reads a separate KV cache per sequence. Selective batching splits the two kinds of computation apart instead of padding every request to a common length. Under the same latency target, the paper measured up to 36.9x the throughput of FasterTransformer, a system using static batching at the time. That number compares two serving systems under the same latency constraint, not “batching versus no batching.”

This term builds on the KV cache: Why a Model Gets More Expensive the Longer It Talks, and on Prefill and Decode, the two phases each step gets reassigned for.

When the Serving Software Cannot Refill Slots

Continuous batching is a property of the serving software, not the GPU hardware. A simple integration that calls the model once per incoming request and waits for its complete output does not have a batch at all. It processes requests strictly one after another, no matter how many arrive in parallel. The GPU then computes for only a single request at a time and stays far below its capacity, regardless of how powerful the card is.

For a Mittelstand company self-hosting an open model instead of renting one, this exact property of the serving software decides whether the economics in that comparison hold at all. Cost per token drops where the machine actually runs several requests at once, not automatically with more compute. Anyone checking whether their own hardware is worth it should first check whether the serving software in use (vLLM, TGI, or a comparable engine) supports continuous batching, before thinking about bigger GPUs. And even with the right software, the effect stays tied to utilization: an internal tool with few concurrent users rarely has a queue that a freed slot could be refilled from.

Continuous batching never leaves a freed slot idle. The moment one request finishes, the next waiting one slides in. It does not wait for the slowest to end.

Next term: paged attention. How a server manages the KV cache of many concurrent requests without losing memory to fragmentation.

Frequently Asked Questions

What is continuous batching? A scheduling technique where the server checks after every generation step which requests are done, frees their slots, and immediately admits new waiting requests.

What is the difference between continuous batching and static batching? Static batching starts a fixed group together and only ends once every request is finished. Continuous batching refills every freed slot immediately.

What is iteration-level scheduling? Scheduling decisions at the level of each individual generation step instead of the whole request. The term comes from the Orca paper (OSDI 2022).

Does more GPU power help if the serving software cannot do continuous batching? No. An integration that processes requests one after another computes for only a single request at a time and never fully uses the GPU, regardless of how powerful the card is.

Part of the series AI Engineering Explained. Related: Why a Model Gets More Expensive the Longer It Talks (KV cache), Why the Same AI Request Has Two Different Speeds (prefill/decode), and Hosting Your Own AI Models or Renting? The Honest Math