KV caching

KV caching is a method in which a large language model, while it produces a sentence one token at a time, keeps the attention keys (Key) and values (Value) it has already calculated in memory. Each time a new token is produced, it does not have to recalculate the earlier part from the beginning, so duplicated computation shrinks.

This technique became necessary because, as the context grows longer, the same calculations repeat, which makes responses slower and costs higher. To handle long inputs in practice, a design was needed that keeps earlier results and builds them up instead of throwing them away.

The model calculates the key and value used in attention for each input token. In the generation step, it stacks these values up in a storage space called the cache. When a new token comes in, it calculates only its own query (Query) and compares it with the stored values.

Thanks to the cache, the amount of calculation per step drops a lot, but the storage space grows in proportion to the context length. So the longer the request, the more GPU memory it takes, and compression techniques to reduce this are also being studied.

When a chatbot continues a conversation, the server keeps a cache so that it does not recalculate the earlier conversation each time. In services where the same system prompt repeats, reusing the cache for the shared opening part can also shorten the time to the first response.

The cache also speeds up repeated predictions when you ask several questions about the same context, as with a tabular prediction model. However, how long to keep the cache has to be decided between memory cost and response speed.

KV caching is not a technique that makes the model's answers smarter. The result for the same input stays the same, and it is most accurate to understand it as an optimization that reuses computation to improve speed and cost.

The cache takes up a large share of GPU memory in a long context and limits the number of requests that can be handled at once. A bigger cache can even cause delays, so it is a mistake to assume it is always faster.

It is like spreading out your notes from earlier problems on the desk during an exam, and using them right away when you work on the next problem.

Knowing the principle of cutting repeated computation helps you understand why AI services use a lot of memory with long contexts and how response speed is decided.