Why GPU Memory Is Already Full Before It Has To Be

What is paged attention?

Paged attention splits the KV cache into equally sized blocks. Each request receives the blocks needed for its tokens so far. This leaves less memory unused.

A short answer can tie up more GPU memory than it needs. This happens when an engine reserves space for each request in advance. It includes tokens that do not exist yet. Often they never will: the answer ends earlier. Until then, other requests cannot use that space.

The way memory is allocated therefore affects how many requests a server can handle. Adding memory is one way out. Using the existing space better is another.

Memory Arrives One Block at a Time

The engine splits the KV cache into small blocks. A request initially receives enough space for the tokens in its prompt. Once that space fills up, the engine adds a free block. It does not have to be next to the previous one.

A table records which blocks belong to each request. The engine can then find the right content even when the blocks are scattered across memory. When a request ends, its blocks can become available again.

This technique has a name: paged attention.

How the Engine Finds Each Block

vLLM introduced the technique in 2023. Its general configuration default is 16 tokens per block. A logical block sequence describes the order within a request. The block table maps it to physical memory blocks.

A worked example: a KV cache holding 19 tokens needs two blocks of this size. That allocates 32 slots, leaving 13 unused. All of them are in the last block. As the cache grows, the request fills those slots first.

In 2023, the vLLM team reported 60 to 80 percent unused KV cache memory in the older systems it studied. With paged attention, that share fell below 4 percent in its experiments. The paper also measured two to four times the throughput of FasterTransformer and Orca at comparable latency. These figures describe those comparisons at the time.

Block size remains a trade-off. Small blocks leave little unused space but can make poorer use of GPU parallelism. Large blocks increase the unused space at the end of a request.

This assumes the KV cache and continuous batching. The cache retains earlier computation results. Continuous batching fills available slots with new requests.

Identical Starts Can Share Memory

Several requests often begin with the same system prompt. With prefix caching enabled, the engine can reuse matching blocks from that start. It then does not need a separate copy for every request. The benefit depends on the length and frequency of the shared start.

Appending tokens must not overwrite data still shared with other requests. Writable shared blocks use copy-on-write: the request making a change receives its own copy. This prevents its changes from affecting other requests.

A system serving multiple users must also define who may share a cache. vLLM supports separate groups through cache_salt. This also limits inferences about other requests based on differences in response time.

What to Check Before Choosing Hardware

A larger GPU can accommodate more requests even without paged attention. It does not remove the cause of unused reservations. Planning a self-hosted assistant therefore involves both questions: How much memory is available? How does the engine allocate it?

vLLM and TGI support paged attention. Testing the intended prompt and answer lengths shows whether the setup handles enough concurrent requests. A short shared system prompt alone does not establish a large saving.

With paged attention, a request receives more memory blocks as its token count grows.

Next term: quantization. How fewer bits reduce the space needed for model weights and can change the answers.

Frequently Asked Questions

What is paged attention? Paged attention splits the KV cache into equally sized blocks. Each request receives the blocks needed for its tokens so far. This leaves less memory unused.

Why can a short answer tie up so much memory? If an engine reserves space for the maximum sequence length in advance, some of it remains unused. Other requests cannot use that space in the meantime. Paged attention allocates additional blocks when they are needed.

What is prefix sharing in the KV cache? With prefix caching enabled, requests can reuse matching cache blocks from an identical start. The benefit depends on how long that shared start is and how often it occurs. Appending tokens must not modify data still shared with other requests.

Does a larger GPU help without paged attention? Yes. More memory can accommodate additional requests. A larger card does not remove the waste from preallocated regions, though. Both memory capacity and memory management belong in the plan.

Sources

• Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention”, SOSP 2023. Paper: block tables, copy-on-write, block sizes, and the throughput comparison at the time.
• vLLM team, June 2023. Introducing PagedAttention: measured memory waste in the systems compared.
• vLLM: CacheConfig for the general block-size default; prefix caching for reuse and cache isolation.
• Hugging Face: PagedAttention in TGI.

Part of the series AI Engineering Explained. Related: KV Cache, Continuous Batching, and Hosting Your Own AI Models or Renting?.