LLM Inference Memory Calculator (weights + KV cache)

Understanding the LLM inference memory calculator — concept and principle

Before running a large language model on a GPU, you need to know whether it fits. The two main consumers of memory are the model weights, which are fixed, and the key-value (KV) cache, which grows with every token of context and every request in the batch. This calculator estimates both from the model's size and architecture to help you choose a GPU, quantisation level or context length.

How it is calculated

• Weights (GB) = parameters (billions) × bytes per parameter. FP16/BF16 uses 2 bytes, INT8 1 byte, 4-bit 0.5 byte.
• KV cache (GB) = 2 × layers × KV heads × head dimension × context tokens × batch size × 2 bytes ÷ 10⁹.
• Total = weights + KV cache.

The first factor 2 counts one key and one value vector; the last factor 2 is the bytes per FP16 value. GB means 10⁹ bytes.

The principle

During generation a transformer attends to every earlier token. Instead of recomputing the keys and values of past tokens at each step, it stores them, in every layer and for every KV head. That store grows linearly with context length and batch size. Many recent models use grouped-query attention, in which several query heads share one KV head; this shrinks the cache in proportion to the number of KV heads, which is why that input matters more than the total number of attention heads.

Worked example

1. Inputs: 8 billion parameters at FP16 (2 bytes), 32 layers, 8 KV heads, head dimension 128, 8,192 tokens, batch 1.
2. Weights = 8 × 2 = 16 GB.
3. KV cache per token = 2 × 32 × 8 × 128 × 2 = 131,072 bytes (128 KiB).
4. × 8,192 tokens = 1,073,741,824 bytes = 1.0737 GB (exactly 1 GiB).
5. Total = 16 + 1.0737 = 17.07 GB.

How to read the result

Add roughly 10–20% for activations, the CUDA context and framework overhead, which puts this example near 19–20 GB: too much for a 16 GB card at FP16, but comfortable after 8-bit or 4-bit quantisation (8 GB or 4 GB of weights). The KV cache scales directly with batch size, so serving 16 parallel 8K-token requests would need about 17 GB of cache alone.

Common mistakes

• Entering the number of attention (query) heads instead of KV heads for a grouped-query model.
• Forgetting that the KV cache is counted per concurrent request.
• Mixing GB (10⁹ bytes) with GiB (2³⁰ bytes).
• Assuming 4-bit weights also shrink the KV cache; it is FP16 here unless quantised separately.

Frequently asked questions

Where do I find layers, KV heads and head dimension?

In the model's configuration file, often named num_hidden_layers, num_key_value_heads and head_dim (or hidden size ÷ attention heads).

Does a longer context slow generation as well?

Yes, each new token must attend to all cached tokens, so attention cost per token grows with context length.

Can the KV cache itself be quantised?

Some inference engines support 8-bit KV caches, which roughly halves cache memory at a small accuracy cost.

Sources: A. Vaswani et al., Attention Is All You Need, NeurIPS 2017 · J. Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, EMNLP 2023