LLM Tokens per Second Calculator

Estimate local language model generation speed from GPU memory bandwidth and model quantization.

Estimated decode speed: about 163.8 tokens/second

How to use

  1. Select a GPU or enter its published memory bandwidth from the dropdown.
  2. Enter the model parameter count and quantization.
  3. Adjust efficiency to reflect real-world utilization; lower values give more conservative estimates.

Formula and example

Estimated tokens/second ≈ memory bandwidth (GB/s) × efficiency ÷ (parameters × bytes per parameter). This bandwidth-bound estimate assumes each generated token reads the model weights once.

Example: RTX 4090 at 1,008 GB/s, 8B model in INT4 (0.5 byte/parameter) and 65% efficiency: 1,008 × 0.65 ÷ (8 × 0.5) ≈ 164 tokens/second.

Static memory bandwidth dataset, reviewed November 18, 2026. Actual speed depends on kernels, KV cache, context, batching, thermals and software.

FAQ

Is this a benchmark?
No. It is a theoretical estimate, especially useful for comparing hardware and quantization choices.

Why does actual output speed differ?
Attention, context length, memory overhead, multi-GPU communication and runtime optimizations affect token generation.

Will the model fit in VRAM?
Check the LLM VRAM Calculator; memory capacity and memory bandwidth are separate constraints.

Related: LLM VRAM Calculator · LLM Cost Calculator.