Learning Rate Schedule Calculator (warmup + cosine)

Understanding the learning rate schedule calculator — concept and principle

This calculator gives the learning rate at any training step for the widely used "linear warmup + cosine decay" schedule. Use it to check what your optimizer is actually doing at a given step, to plan a resume from a checkpoint, or to compare schedules before launching a long training run.

How it is calculated

• Warmup (step < warmup steps): LR = peak × step ÷ warmup. The rate rises in a straight line from 0 to the peak.
• Cosine decay (step ≥ warmup): progress p = (step − warmup) ÷ (total − warmup), then LR = peak × [m + (1 − m) × 0.5 × (1 + cos(π × p))].

Here m is the minimum learning rate as a fraction of the peak (0.1 means the schedule ends at 10% of the peak). Steps are optimizer updates, not epochs. The learning rate has no unit; it scales the gradient step.

The principle

At the start of training the weights are random and adaptive optimizers such as Adam have poor estimates of gradient statistics, so a full-size step can push the model into an unstable region. Warmup lets those statistics settle while steps are still small. After that, a large rate explores quickly, and a gradually smaller rate lets the model settle into a good minimum. The term 0.5 × (1 + cos(πp)) falls smoothly from 1 to 0 as p goes from 0 to 1: it decays slowly at first, fastest in the middle, and gently again near the end. The factor m lifts the floor so the rate never reaches exactly zero. The shape comes from SGDR, without restarts.

Worked example

1. Inputs: peak 3e-4, warmup 2,000, total 100,000, current step 50,000, minimum fraction 0.1.
2. Step 50,000 is past warmup, so use the cosine branch.
3. p = (50,000 − 2,000) ÷ (100,000 − 2,000) = 48,000 ÷ 98,000 ≈ 0.48980.
4. cos(π × 0.48980) ≈ 0.03205, so 0.5 × (1 + 0.03205) ≈ 0.51603.
5. Multiplier = 0.1 + 0.9 × 0.51603 ≈ 0.56442.
6. LR = 3e-4 × 0.56442 ≈ 0.000169327 (about 1.69e-4).

How to read the result

The multiplier tells you where you are: 1.0 at the end of warmup, about 0.55 halfway through the decay when m = 0.1, and m at the final step. Step 50,000 sits just before the midpoint of the decay phase, so the value is slightly above 0.55. Warmup is often 1–5% of total steps, and a floor of 0 to 0.1 of the peak is common.

Common mistakes

• Entering epochs instead of optimizer steps, or forgetting gradient accumulation (one step = one update).
• Changing total steps when resuming, which shifts the entire curve.

Frequently asked questions

Why does the learning rate start at 0?

At step 0 the formula gives peak × 0 ÷ warmup = 0. Many libraries do the same, so the first update is tiny; some start at a small fraction instead.

What happens after the total steps are reached?

In this formula p exceeds 1 and the cosine climbs again. Real schedulers usually clamp at the minimum, so do not train past the total you entered.

Sources: I. Loshchilov and F. Hutter, "SGDR: Stochastic Gradient Descent with Warm Restarts", ICLR 2017 · P. Goyal et al., "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour", arXiv:1706.02677 (2017)