KV Cache Calculator
Work out how much GPU memory a transformer's key-value cache takes at your context length, batch size and precision. See the cost per token, a per-layer breakdown, GQA vs MHA vs MQA, and how it grows with context.
KV cache calculator
num_key_value_heads
num_attention_heads, used for the MHA comparison
Per-layer breakdown
| Part | Size |
|---|---|
| One layer, K cache | 16 MiB |
| One layer, V cache | 16 MiB |
| One layer, one KV head (K+V) | 4 MiB |
| One layer, one token (K+V) | 4 KiB |
| All 32 layers (3.13% each) | 1 GiB |
Standard transformers cache the same amount in every layer. Models with sliding-window or shared-KV layers (some Gemma, Mistral and Cohere variants) cache less in those layers, so this is an upper bound for them.
GQA vs MHA vs MQA for the same model
| Attention type | KV heads | Total cache | vs configured |
|---|---|---|---|
| MHA (one KV head per query head) | 32 | 4 GiB | 4.00× |
| GQA (as configured) | 8 | 1 GiB | 1.00× |
| MQA (single KV head) | 1 | 128 MiB | 0.13× |
MHA gives every query head its own K/V; GQA shares K/V across groups of query heads; MQA shares one K/V pair across all of them. Fewer KV heads means a smaller cache, usually with a small quality cost.
KV cache vs context length (batch 1, FP16 / BF16)
KV cache = 2 (K and V) × layers × KV heads × head dim × tokens × batch × bytes per value. Model weights and runtime overhead are extra.
How to calculate KV cache size
- Pick a model preset, or choose Custom and enter the layers, KV heads and head dimension from the model's
config.json(num_hidden_layers,num_key_value_heads,head_dimor hidden size ÷ attention heads). - Enter the context length and how many sequences you serve at once (batch size).
- Choose the cache precision: FP16 is the default, FP8 halves it and INT4 quarters it.
Formula
KV cache bytes = 2 × layers × KV heads × head dimension × bytes per value × tokens × batch. The 2 is for the keys and the values. Only the KV heads count: with grouped-query attention (GQA) a model with 32 query heads but 8 KV heads caches a quarter of what full multi-head attention would.
Worked examples
- Llama 3.1 8B (32 layers, 8 KV heads, head dim 128) at FP16: 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token, so 8,192 tokens take exactly 1 GiB.
- Llama 3.1 70B (80 layers, 8 KV heads, head dim 128) with an FP8 cache, 32,768 tokens and batch 4: 163,840 bytes per token × 32,768 × 4 = 20 GiB.
The KV cache is only part of the memory a model needs. To add model weights and overhead and see which GPUs fit, use the LLM RAM and VRAM calculator. To estimate throughput use the tokens per second calculator.
Get this free widget for your site
Estimate LLM KV cache memory from model, context length, batch size and precision. Free to use; the snippet includes a "Powered by MyFreeToolkit" link back to this tool.
See all free widgets.
See also
- Need the full memory budget with weights, overhead and GPU fit? Use the LLM RAM and VRAM calculator.
- Estimate generation speed with the tokens per second calculator.
- Convert a context length into pages with the tokens to words converter.
Frequently Asked Questions (FAQ)
During generation a transformer stores the key and value vectors of every earlier token in every layer so it does not recompute them. That store is the KV cache. It grows linearly with context length and batch size and can be larger than the model weights for long contexts.
Multiply 2 (keys and values) by layers, KV heads, head dimension, bytes per value, tokens and batch size. Llama 3.1 8B at FP16 uses 2 × 32 × 8 × 128 × 2 = 131,072 bytes (128 KiB) per token, so an 8k context takes 1 GiB.
Grouped-query attention lets several query heads share one key/value head. Llama 3.1 8B has 32 query heads but only 8 KV heads, so its cache is a quarter of the multi-head equivalent. Multi-query attention (MQA) shares a single KV head and shrinks it further.
FP8 and INT8 caches usually cost very little accuracy and halve the memory. INT4 saves more but loses more on long or precise tasks, so test it on your workload. Runtimes such as vLLM and llama.cpp offer cache quantization separately from weight quantization.
This is the raw tensor size. Serving engines add paging overhead, padding and fragmentation, and models with sliding-window or shared-KV layers can use less. Add model weights and activations separately.
Every sequence being served has its own cache, so memory scales linearly with batch size. Serving 16 users with 8k contexts on Llama 3.1 8B needs 16 GiB of cache at FP16.
In the model's config.json on Hugging Face: num_hidden_layers, num_key_value_heads and head_dim (or hidden_size divided by num_attention_heads). The presets use these published values.
Related Tools
- LLM Tokens per Second CalculatorEstimate local LLM generation speed from GPU memory bandwidth, model size and quantization.
- Self-Host vs API Cost Calculator (LLM Break-Even)Compare monthly LLM API cost with self-hosting on a rented or owned GPU and find the break-even token volume. All assumptions shown.
- GPU Rental Cost CalculatorEstimate cloud GPU cost per job and per month from hourly rate, GPU count, job and idle hours, storage and egress. Example rates are dated and editable.
- LLM API Cost CalculatorCompare GPT, Claude and Gemini API rates and estimate monthly token costs.
- RAG Chunking and Embedding Cost CalculatorEstimate RAG chunk count, one-off embedding (indexing) cost and monthly query cost from document size, chunk size, overlap, embedding price and retrieval settings. Prices are dated and editable.
- AI Token CounterCount tokens, words and characters in your prompts for GPT, Claude, Gemini and Llama.