KV Cache Calculator

Work out how much GPU memory a transformer's key-value cache takes at your context length, batch size and precision. See the cost per token, a per-layer breakdown, GQA vs MHA vs MQA, and how it grows with context.

KV cache calculator

num_key_value_heads

num_attention_heads, used for the MHA comparison

Memory per token
128 KiB
131,072 bytes, all 32 layers
Total KV cache
1 GiB
8,192 tokens × 1 sequence
Per layer
32 MiB
K and V, whole context and batch

Per-layer breakdown

PartSize
One layer, K cache16 MiB
One layer, V cache16 MiB
One layer, one KV head (K+V)4 MiB
One layer, one token (K+V)4 KiB
All 32 layers (3.13% each)1 GiB

Standard transformers cache the same amount in every layer. Models with sliding-window or shared-KV layers (some Gemma, Mistral and Cohere variants) cache less in those layers, so this is an upper bound for them.

GQA vs MHA vs MQA for the same model

Attention typeKV headsTotal cachevs configured
MHA (one KV head per query head)324 GiB4.00×
GQA (as configured)81 GiB1.00×
MQA (single KV head)1128 MiB0.13×

MHA gives every query head its own K/V; GQA shares K/V across groups of query heads; MQA shares one K/V pair across all of them. Fewer KV heads means a smaller cache, usually with a small quality cost.

KV cache vs context length (batch 1, FP16 / BF16)

KV cache size by context length0 B16 GiB32 GiB2k256 MiB4k512 MiB8k1 GiB16k2 GiB32k4 GiB64k8 GiB128k16 GiB256k32 GiB

KV cache = 2 (K and V) × layers × KV heads × head dim × tokens × batch × bytes per value. Model weights and runtime overhead are extra.

How to calculate KV cache size

  1. Pick a model preset, or choose Custom and enter the layers, KV heads and head dimension from the model's config.json (num_hidden_layers, num_key_value_heads, head_dim or hidden size ÷ attention heads).
  2. Enter the context length and how many sequences you serve at once (batch size).
  3. Choose the cache precision: FP16 is the default, FP8 halves it and INT4 quarters it.

Formula

KV cache bytes = 2 × layers × KV heads × head dimension × bytes per value × tokens × batch. The 2 is for the keys and the values. Only the KV heads count: with grouped-query attention (GQA) a model with 32 query heads but 8 KV heads caches a quarter of what full multi-head attention would.

Worked examples

  • Llama 3.1 8B (32 layers, 8 KV heads, head dim 128) at FP16: 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token, so 8,192 tokens take exactly 1 GiB.
  • Llama 3.1 70B (80 layers, 8 KV heads, head dim 128) with an FP8 cache, 32,768 tokens and batch 4: 163,840 bytes per token × 32,768 × 4 = 20 GiB.

The KV cache is only part of the memory a model needs. To add model weights and overhead and see which GPUs fit, use the LLM RAM and VRAM calculator. To estimate throughput use the tokens per second calculator.

Get this free widget for your site

Estimate LLM KV cache memory from model, context length, batch size and precision. Free to use; the snippet includes a "Powered by MyFreeToolkit" link back to this tool.

See all free widgets.

See also

Frequently Asked Questions (FAQ)

What is the KV cache?

During generation a transformer stores the key and value vectors of every earlier token in every layer so it does not recompute them. That store is the KV cache. It grows linearly with context length and batch size and can be larger than the model weights for long contexts.

How do I calculate KV cache size?

Multiply 2 (keys and values) by layers, KV heads, head dimension, bytes per value, tokens and batch size. Llama 3.1 8B at FP16 uses 2 × 32 × 8 × 128 × 2 = 131,072 bytes (128 KiB) per token, so an 8k context takes 1 GiB.

What is GQA and how does it reduce the cache?

Grouped-query attention lets several query heads share one key/value head. Llama 3.1 8B has 32 query heads but only 8 KV heads, so its cache is a quarter of the multi-head equivalent. Multi-query attention (MQA) shares a single KV head and shrinks it further.

Does quantizing the KV cache hurt quality?

FP8 and INT8 caches usually cost very little accuracy and halve the memory. INT4 saves more but loses more on long or precise tasks, so test it on your workload. Runtimes such as vLLM and llama.cpp offer cache quantization separately from weight quantization.

Why is the real memory use higher than the calculator says?

This is the raw tensor size. Serving engines add paging overhead, padding and fragmentation, and models with sliding-window or shared-KV layers can use less. Add model weights and activations separately.

How does batch size affect the KV cache?

Every sequence being served has its own cache, so memory scales linearly with batch size. Serving 16 users with 8k contexts on Llama 3.1 8B needs 16 GiB of cache at FP16.

Where do I find the layers, KV heads and head dimension?

In the model's config.json on Hugging Face: num_hidden_layers, num_key_value_heads and head_dim (or hidden_size divided by num_attention_heads). The presets use these published values.