LLM Inference Throughput → Servers Needed Calculator (peak load, redundancy, cost)

How it works

Peak demand (tokens/s) = requests per day × busiest-hour share × output tokens ÷ 3,600. Servers = ⌈peak ÷ (throughput per server × target utilisation)⌉ + spare servers. Running below 100% leaves headroom for bursts and keeps latency stable. Cost per 1M output tokens = monthly server cost ÷ output tokens per month (30 days). Throughput depends heavily on model, hardware, batch size and prompt length, so measure it with your own load test.

⚠ Estimates for information only — not financial advice.

FAQ

Why plan for the busiest hour, not the daily average?

Users do not arrive evenly. If you size for the average, replies slow down or fail at peak times. Check your traffic logs for the busiest hour’s share.

What about input tokens?

Long prompts take GPU time too (the prefill step). Run your load test with realistic prompt lengths so the measured tokens per second already includes that cost.

Checked 10 Nov 2026. Capacity-planning arithmetic: peak demand ÷ usable capacity per server, rounded up, plus N+1 style spare capacity (see Google, Site Reliability Engineering, O’Reilly 2016, chapters on capacity planning and load balancing).