What the model can hold, and what it costs.
Everything a model works with has to fit in its context window, and every token in that window takes memory while it writes. The KV cache is why generation is fast, and why long context is expensive.
- Say what has to fit in a context window
- Explain prefill, decode and the KV cache
- Work out a KV cache’s memory from a model’s config
You need: Lesson 3, Attention.
-
The context window
Predict first
Qwen 3 (8B) handles 32,768 tokens on its own. What has to fit in that?
Just show me
All of it: the system prompt, the conversation, retrieved text and the answer being written.
The context window is the model’s working memory, counted in tokens. Its model card gives 32,768 tokens natively and up to 1,31,072 with a stretching method (YaRN). Whatever doesn’t fit, the model never sees.
Quick check
A long chat starts forgetting its first messages. Why?
Apps trim or summarise old turns to stay inside the window.
-
One token at a time
Predict first
To write a 500-token answer, how many times does the model run?
Just show me
About 500: one pass reads the whole prompt (prefill), then one pass per new token (decode).
Prefill reads the prompt in one parallel pass, so a long prompt mostly costs time before the first word appears. Decode then adds one token per pass, so a long answer costs time per word. That’s why services quote both time to first token and tokens per second.
Quick check
A user pastes a 30-page document. Which phase gets slower?
The prompt is read in prefill; its length sets the wait before the first token.
-
Keep the keys and values
Predict first
Writing token 501, does the model need token 1’s key and value again?
Just show me
Yes: every new token attends to all the earlier ones, in every layer.
Without a cache, each step would recompute every earlier token’s keys and values in every layer. The KV cache stores them once and reuses them, so each step only computes the new token’s. The price is memory: the cache grows with every token.
Quick check
What does the KV cache trade?
It spends memory to avoid recomputing the past.
-
How big is the cache?
Predict first
At its full 32,768 tokens, which is bigger for Qwen 3 (0.6B): its weights or its KV cache?
Just show me
Its KV cache: 3.8 GB against about 1.2 GB of weights in 16-bit.
Per token, the cache holds a key and a value (2) for every layer, every key-value head and every number in a head, at 2 bytes each in 16-bit. From Qwen 3 (8B)’s config: 2 × 36 layers × 8 heads × 128 × 2 bytes = 1,47,456 bytes (about 147 KB) per token, or 4.8 GB for 32,768 tokens, and that’s for one conversation. Serve many users at once and the caches, not the weights, fill the GPU.
The model
Tokens in the conversation
Conversations at once
4.8 GBof KV cache for 1 conversation of 32,768 tokens
KV cache4.8 GB
Weights, 16-bit16.4 GB
2 × 36 layers × 8 KV heads × 128 × 2 bytes = 1,47,456 bytes a token
The KV cache, symbol by symbol
Quick check
Eight users each hold a 32,768-token chat with the 8B model. Cache memory?
Each conversation keeps its own cache.
-
Sharing keys and values
Predict first
Qwen 3 (8B) has 32 query heads but only 8 key-value heads. What does that save?
Just show me
4 times: 4.8 GB at 32,768 tokens instead of 19.3 GB.
Grouped-query attention lets several query heads share one key-value head. Quality stays close to full multi-head attention while the cache shrinks, which is why most recent open models use it.
Quick check
Grouped-query attention mainly helps with what?
It shrinks the KV cache, the memory that grows with every token served.
-
Longer isn’t always better
Predict first
The answer sits in the middle of a very long context. Will the model always find it?
Just show me
Not always. Researchers found models use information at the start and end of a long context better than in the middle.
Long context costs memory, time and money, and models don’t read it evenly. Retrieval that hands over only the relevant passages, summaries of older turns and putting key instructions first or last usually beat stuffing everything in.
See what retrieval hands a model in RAG chunking
Quick check
With a huge context window, do you still need RAG?
Sending only what matters beats paying for, and diluting, a giant context.
Lesson 5 done.
Next: Hallucination. A model can write a confident, well-formed answer that is simply wrong.
In the interview: context and the KV cache.
EasyWhat is a context window, and does a bigger one solve everything?
HardHow does the KV cache speed up generation, and what limits it?
HardEstimate the KV cache memory for one 8,192-token request to Llama 3 8B.
HardWhat are the prefill and decode phases of LLM inference, and what limits each?
HardWhat are multi-query and grouped-query attention, and why do they matter when serving a model?
EasyModels now accept very long contexts. Do we still need RAG?
Ask Prism about this lesson
Sources
- Pope and others, “Efficiently Scaling Transformer Inference”, 2022: the KV cache’s memory when serving.
- Ainslie and others, “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints”, EMNLP 2023.
- Liu and others, “Lost in the Middle: How Language Models Use Long Contexts”, TACL 2024.
- Qwen, the Qwen 3 models (0.6B, 8B, 32B), Apache 2.0: their config files and model cards, pinned.