Hong Life CalculatorsUnderstanding the model training compute and time estimator — concept and principle
Training a large neural network is measured in floating-point operations (FLOPs). This estimator uses the widely cited rule that training a dense transformer costs about 6 FLOPs per parameter per training token, then divides that total by the throughput your GPUs actually achieve to estimate wall-clock days and total GPU-hours. It is useful for budgeting and sizing a cluster.
How it is calculated
- Total compute C ≈ 6 × N × D, where N is the number of parameters and D is the number of training tokens.
- Effective throughput = peak TFLOPS × 10¹² × number of GPUs × MFU ÷ 100 (FLOP per second).
- Days = C ÷ effective throughput ÷ 86,400.
- GPU-hours = C ÷ (peak TFLOPS × 10¹² × MFU ÷ 100) ÷ 3,600, which equals days × 24 × number of GPUs.
MFU (model FLOPs utilisation) is the share of the GPU's theoretical peak that goes into useful model computation.
The principle
In a forward pass, each parameter takes part in roughly one multiplication and one addition per token, about 2 FLOPs. The backward pass computes gradients with respect to both activations and weights and costs about twice the forward pass, roughly 4 FLOPs. Together that gives about 6N FLOPs per token. Kaplan et al. (2020) used it in their scaling-law study. It ignores the attention computation that grows with sequence length, which is small for typical context lengths in large models.
Worked example
- N = 7 × 10⁹ parameters, D = 1,000 × 10⁹ = 10¹² tokens.
- C = 6 × 7 × 10⁹ × 10¹² = 4.2 × 10²² FLOPs.
- Effective throughput = 989 × 10¹² × 256 × 0.40 = 1.0127 × 10¹⁷ FLOP/s.
- Time = 4.2 × 10²² ÷ 1.0127 × 10¹⁷ = 414,718 s ÷ 86,400 = 4.80 days.
- GPU-hours = 4.80 days × 24 × 256 = 29,491 GPU-hours.
| Parameters (billions) | Total compute (FLOPs) |
|---|---|
| 1.75 | 1.05e+22 |
| 3.5 | 2.1e+22 |
| 5.25 | 3.15e+22 |
| 7 | 4.2e+22 |
| 8.75 | 5.25e+22 |
| 10.5 | 6.3e+22 |
| 14 | 8.4e+22 |
How to read the result
The result is ideal compute time. Real runs also lose time to evaluation, checkpointing, restarts after hardware failures and debugging, so plan a margin. MFU is the most uncertain input: the PaLM paper, which introduced the term, reported about 46%, and poorly optimised setups can be much lower. Multiply GPU-hours by the hourly price for cost.
Common mistakes
- Using a sparse or FP8 peak while training in BF16.
- Confusing tokens with samples or epochs; D is the total number of tokens processed.
- Applying 6ND to mixture-of-experts models with total rather than active parameters.
Frequently asked questions
Why 6 and not 2?
2 FLOPs per parameter per token is inference (forward only); training adds the backward pass, about 4 more.
How many tokens should a model be trained on?
Hoffmann et al. (2022) found about 20 tokens per parameter compute-optimal, but many models are trained far longer to be cheaper at inference.
Does fine-tuning use the same formula?
Yes for full fine-tuning, with D as the fine-tuning tokens.
Sources: J. Kaplan et al., Scaling Laws for Neural Language Models, arXiv:2001.08361, 2020 · J. Hoffmann et al., Training Compute-Optimal Large Language Models, arXiv:2203.15556, 2022 · A. Chowdhery et al., PaLM: Scaling Language Modeling with Pathways, arXiv:2204.02311, 2022