Softmax & Sigmoid Calculator

Understanding the softmax and sigmoid calculator — concept and principle

Neural networks produce raw scores called logits, which can be any real numbers. Softmax turns a list of logits into probabilities that are positive and add up to 1, and the sigmoid turns a single logit into a probability between 0 and 1. This calculator applies both, lets you adjust the softmax temperature, and reports the entropy of the result, which shows how spread out or confident the distribution is.

How it is calculated

• Softmax: pᵢ = e^((zᵢ − m)/T) ÷ Σⱼ e^((zⱼ − m)/T), where m is the largest logit and T is the temperature.
• Sigmoid of each logit: σ(z) = 1 ÷ (1 + e^(−z)), computed independently for each value.
• Entropy H = −Σ pᵢ log₂ pᵢ, in bits.

Subtracting the maximum m does not change the probabilities, because the factor e^(−m/T) cancels between numerator and denominator, but it prevents numerical overflow when logits are large.

The principle

Exponentiating makes every score positive and turns differences into ratios: two logits that differ by 1 give probabilities in the ratio e ≈ 2.718 to 1 at T = 1. Dividing by the total makes the values sum to 1. Temperature rescales the logits first: T below 1 sharpens the distribution toward the top class, T above 1 flattens it toward uniform. The sigmoid is the two-class special case of softmax: softmax of (z, 0) gives exactly σ(z) for the first class. The sigmoid values are not meant to sum to 1; each one treats its logit as a separate yes/no question.

Worked example

1. Logits 2.0, 1.0, 0.1 with T = 1. Subtract the maximum 2.0: 0, −1, −1.9.
2. Exponentials: 1, 0.36788, 0.14957. Sum = 1.51745.
3. Softmax = 1 ÷ 1.51745, 0.36788 ÷ 1.51745, 0.14957 ÷ 1.51745 = 0.659001, 0.242433, 0.0985659.
4. Sigmoids: σ(2) = 0.880797, σ(1) = 0.731059, σ(0.1) = 0.524979.
5. Entropy = −(0.659 log₂ 0.659 + 0.2424 log₂ 0.2424 + 0.0986 log₂ 0.0986) = 1.2216 bits.

How to read the result

For three classes entropy ranges from 0 (all probability on one class) to log₂ 3 ≈ 1.585 bits (all equal). Here 1.22 bits means the model favours the first class but is far from certain. Raising T increases entropy; lowering it decreases entropy.

Common mistakes

• Adding sigmoid outputs and expecting 1; use softmax for mutually exclusive classes.
• Reading softmax outputs as calibrated probabilities without checking calibration.
• Setting T = 0, which divides by zero; very small T approaches picking the largest logit.

Frequently asked questions

Why is temperature used in language models?

Sampling from a softmax with T above 1 gives more varied text; below 1 gives more predictable text.

What does adding a constant to all logits do?

Nothing. Softmax depends only on differences between logits.

When should I use sigmoid instead of softmax?

For binary outputs or multi-label tasks, where several labels can be true at once.

Sources: I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, chapter 6 · G. Hinton, O. Vinyals and J. Dean, Distilling the Knowledge in a Neural Network, arXiv:1503.02531, 2015