Why a Model Gets More Expensive the Longer It Talks

What is the KV cache?

The KV cache is a buffer in GPU memory. The model stores the computed results of all previous tokens there so it can read them on the next step instead of recomputing everything. It grows with every token and is usually the first thing to run out.

Same model, same question. Yet every answer gets slower and more expensive the longer the conversation runs. Ten messages exchanged, and the difference is measurable.

The model is not getting worse. The reason lies in what it holds onto. The model is the brilliant, fast, occasionally-lying intern. And like anyone who works fast: it preps things and keeps them within reach.

The Prep on the Counter

A cook cuts the onions once at the start and lays everything within reach on the counter. Every new dish reuses that prep. Only what is new gets cut. But the counter is finite. The longer the evening, the more prep stacks up. At some point there is no room left. The evening does not end because cooking gets harder. It ends because space runs out.

The model behaves the same way. It does not recompute the entire conversation history for each new answer. It stores the intermediate results of all previous tokens on the “counter” and reads them directly on the next step. Every new token piles on top.

This picture has a name: KV cache.

Here is where this term sits in the series:

Under the Hood

Without the cache, the model would recompute attention over the entire sequence so far for every new token. Instead, it stores the key and value tensors of all processed tokens in VRAM and reads them on the next step.

The cache size works out roughly as:

2 × layers × kv_heads × head_dim × seq_len × batch × bytes_per_value

For a modern 8B model with grouped query attention (8 KV heads, 32 layers), a single conversation at 100,000 tokens reaches around 13 GB of cache while the model weights in FP16 occupy about 16 GB. Sixteen parallel conversations at 4,000 tokens each come to around 8 GB. Older architectures without GQA run up to four times higher. That is why VRAM fills up at the cache first, not at compute. A larger context window moves this limit up, but does not make it disappear.

This term assumes familiarity with tokens: What Is a Token?

Where It Breaks

When VRAM fills up, the system has to evict cache entries. If the wrong prep gets thrown out, the model recomputes those tokens on the next call. Time and compute cost doubled.

When too many long conversations hit a system at once, throughput drops. Not because the model computes slower, but because memory cannot fit more requests.

For organizations working with long contexts, this lands directly. Contract analyses, multi-hour customer conversations, large knowledge documents: the longer the document, the larger the cache, the sooner the limit. Estimating AI costs from short test documents underestimates real production costs considerably.

The defenses are fixed-size cache pages that reduce fragmentation (paged attention), and shared starting blocks for requests with the same opening prefix (prefix reuse). Both come in the next part.

The KV cache is the prep that gets kept so the past never has to be cooked twice. And it is the first thing to run out.

Next term: prefill and decode. Why the same request runs at two different speeds, and which one the user actually feels.

Frequently Asked Questions

What is the KV cache? The KV cache (key-value cache) is a buffer in GPU memory. The model stores the computed results of all processed tokens there and reads them on the next step instead of recomputing everything. It grows with every token and for longer conversations is often larger than the model weights themselves.

Why does an AI conversation get more expensive over time? With every new message the KV cache grows. The model needs more GPU memory, and on busy systems throughput drops because fewer parallel requests fit in memory.

What is paged attention? Paged attention manages the KV cache in fixed-size pages, similar to how an operating system divides virtual memory. This reduces memory fragmentation and lets requests that share the same conversation start reuse the same cache block.

Why does VRAM usage spike for long texts? The KV cache grows linearly with sequence length and the number of parallel requests. For long documents or conversations it can exceed the size of the model weights themselves. That is usually the bottleneck that determines how many requests can run concurrently.

Part of the series AI Engineering Explained. Related: What Is a Token? and What Does an AI Project Cost?