|
On-call Best Practices for SREs (Sponsored)
 |
On-call shouldn’t feel like constant firefighting. This guide from Datadog breaks down how high-performing SRE teams reduce alert fatigue, streamline incident response, and design rotations that don’t burn engineers out.
You’ll learn how to:
Cut alert noise by tying signals to real user impact
Improve response with clear roles and smarter escalation paths
Turn incidents into feedback loops that improve system reliability
Why does sending a model a 100K-word prompt cost so much more than sending it a short one, even when the model and the hardware stay the same?
A key part of the answer lies in a block of working memory called the KV cache.
This memory is built up while the model generates a response. It is separate from the knowledge stored in the model’s weights, and it holds the key and value vectors computed for every token of the input. The cache grows with every token, and in a long context, it can take up significant space on the GPU.
For example, for a 70-billion-parameter model at a context of 128,000 tokens, it comes to roughly 40 gigabytes, a serious amount of GPU memory that grows with every user you add.
 |
The above chart raises an obvious question, which is why a cache exists at all and why it grows the way it does. In this article, we will learn how LLMs use memory, how it gets expensive, and how to fix it.
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
Recomputation
Let us start with the work a model does to produce one token.
To choose the next word, it runs an attention step, where the newest token compares itself against every token that came before it. This comparison uses two vectors for each earlier token, a key and a value, which are simply the numerical summaries the model computes for that token inside each layer. A model that rebuilt the key and value for every earlier token at every step would watch the work per token climb as the input grows. Such a repetition is pure waste, because those keys and values stay the same once a token has been processed.
The KV cache removes the waste by storing those key and value vectors the first time they are computed. On the next step, the model computes the key and value for only the new token and reads the rest straight from the cache.
 |
Overall, this is a great solution. However, caching fixes the speed problem while creating a new one. The cache now has to be read on every step, and this turns out to be the real source of increasing cost.
One detail worth understanding here is that the cache holds vectors rather than the original text. It also explains out-of-memory errors that look puzzling when the model itself fits with room to spare.
Decoding
Generation of tokens runs in two phases, and they stress the hardware in different ways:
The first phase is prefill, where the model reads the entire input at once. It processes all of the input tokens in parallel and builds their key and value vectors into the cache in a single pass. Prefill keeps the GPU’s math units busy, so we call it compute-bound, meaning the limit is how fast the chip can do arithmetic.
The second phase is decoding, where the model produces the output one token at a time. Each new token runs an attention step against the whole cache, which means the model reads every stored key and value out of GPU memory before it can emit the next token. It repeats that read for every token it produces. The limit here is how fast the cache can move from memory into the compute units, so we call decoding memory-bound.
 |