LLM VRAM & GPU Count Calculator (weights + KV cache for concurrent users)

How it works

Weights (GB) = parameters (billions) × bytes per parameter. KV cache per token (bytes) = 2 × layers × KV heads × head dimension × bytes per value — the 2 is for keys and values. KV cache = per-token size × context length × concurrent requests. Total = (weights + KV cache) × (1 + allowance), and GPUs = total ÷ (memory per GPU × usable share), rounded up. 1 GB = 10⁹ bytes. Splitting a model across GPUs adds some memory per GPU, and serving software with paged KV caches can fit more requests, so treat this as a first sizing step.

⚠ Estimates for information only — not financial advice.

FAQ

Where do I find layers, KV heads and head dimension?

In the model’s config.json on its model card: num_hidden_layers, num_key_value_heads and hidden_size ÷ num_attention_heads. The defaults match a typical 70-billion-parameter model with grouped-query attention.

Why does the KV cache matter so much?

It grows with every token of context and every simultaneous user. With long contexts and many users it can need more memory than the model weights themselves.

Checked 10 Nov 2026. NVIDIA Technical Blog, “Mastering LLM Techniques: Inference Optimization” (S. Verma & N. Vaidya, 17 Nov 2023): weights ≈ parameters × bytes per parameter; KV cache per token = 2 × layers × (heads × head dimension) × bytes, using the KV heads for grouped-query attention.