KL & Jensen-Shannon Divergence Calculator

Understanding the KL and Jensen-Shannon divergence calculator — concept and principle

Kullback-Leibler (KL) divergence measures how different one probability distribution is from another, and Jensen-Shannon (JS) divergence is a symmetric, bounded version of it. They are used to compare a model's predicted distribution with observed data, to detect data drift between training and production, to train generative models and in information theory generally. Enter two lists of probabilities (or counts) over the same categories in the same order.

How it is calculated

• Both lists are first normalised: each value is divided by its list's total so that it sums to 1.
• KL(P‖Q) = Σ pᵢ log₂(pᵢ ÷ qᵢ), skipping terms where pᵢ = 0.
• KL(Q‖P) = Σ qᵢ log₂(qᵢ ÷ pᵢ).
• JS(P, Q) = ½ KL(P‖M) + ½ KL(Q‖M), where M = (P + Q) ÷ 2 is the average distribution.

Using log base 2 gives results in bits.

The principle

KL(P‖Q) is the expected number of extra bits you need to encode data that really follows P if you use a code optimised for Q. It is never negative (Gibbs' inequality) and equals 0 only when the two distributions are identical. It is not a true distance: it is not symmetric, and it becomes infinite when Q gives zero probability to something P says can happen. JS fixes both problems by comparing each distribution with their average, which is never zero where either one is positive. In bits, JS always lies between 0 and 1, and its square root is a proper metric.

Worked example

1. P = (0.4, 0.4, 0.2) and Q = (0.3, 0.5, 0.2) already sum to 1.
2. KL(P‖Q) = 0.4 log₂(4/3) + 0.4 log₂(0.8) + 0.2 log₂(1) = 0.16601 − 0.12877 + 0 = 0.03724 bits.
3. KL(Q‖P) = 0.3 log₂(0.75) + 0.5 log₂(1.25) + 0 = 0.03645 bits; slightly different, showing the asymmetry.
4. M = (0.35, 0.45, 0.2). JS = ½ KL(P‖M) + ½ KL(Q‖M) = 0.00919 bits.

How to read the result

Values near 0 mean the distributions are almost the same. There is no universal threshold; compare against values you see between batches that you know are healthy. KL depends on direction: KL(P‖Q) heavily penalises places where P is large but Q is small. If you need one symmetric number on a fixed 0–1 scale, report JS.

Common mistakes

• Listing categories in different orders in P and Q.
• Using KL when Q contains zeros where P does not, which makes it infinite.
• Mixing bits (log₂) with nats (natural log); 1 nat ≈ 1.4427 bits.

Frequently asked questions

Is KL divergence a distance?

No. It is asymmetric and does not satisfy the triangle inequality. The square root of JS divergence does.

What is the relation to cross-entropy?

Cross-entropy H(P, Q) = H(P) + KL(P‖Q), so minimising cross-entropy with P fixed minimises KL.

Can I enter raw counts?

Yes. The calculator normalises each list, so counts are turned into proportions automatically.

Sources: S. Kullback and R. A. Leibler, On information and sufficiency, Annals of Mathematical Statistics, 1951 · J. Lin, Divergence measures based on the Shannon entropy, IEEE Transactions on Information Theory, 1991 · T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006