Regression Error Metrics (MAE, RMSE, MAPE, R²)

Understanding the regression error metrics calculator — concept and principle

When a model predicts a number, such as sales, a price, a temperature or demand, you judge it by how far the predictions fall from the actual values. This calculator takes a list of actual values and a matching list of predictions and returns the most widely used error measures: MAE, MSE, RMSE, MAPE, sMAPE and the coefficient of determination R².

How it is calculated

Each error is eᵢ = predictionᵢ − actualᵢ, and n is the number of pairs.

• MAE = (1/n) Σ |eᵢ|, in the same unit as the data.
• MSE = (1/n) Σ eᵢ², in squared units.
• RMSE = √MSE, back in the original unit.
• MAPE = (1/n) Σ |eᵢ ÷ actualᵢ| × 100%.
• sMAPE = (1/n) Σ 2|eᵢ| ÷ (|actualᵢ| + |predictionᵢ|) × 100%.
• R² = 1 − Σ eᵢ² ÷ Σ (actualᵢ − mean of actual)².

The principle

MAE treats every unit of error equally, so it reflects the typical miss. Squaring in MSE and RMSE gives large errors far more weight, so RMSE is always at least as large as MAE, and the gap grows when a few predictions are badly off. Percentage measures remove the unit so you can compare series of different scales, but MAPE divides by the actual value and therefore blows up near zero; sMAPE divides by the average size of actual and prediction to soften that. R² compares your squared error with that of the simplest possible model, which always predicts the mean.

Worked example

1. Actual 100, 150, 200, 250; predictions 110, 140, 210, 240. Errors: +10, −10, +10, −10.
2. MAE = 40 ÷ 4 = 10; MSE = 400 ÷ 4 = 100; RMSE = √100 = 10.
3. MAPE = (10% + 6.67% + 5% + 4%) ÷ 4 = 6.42%.
4. sMAPE = (20/210 + 20/290 + 20/410 + 20/490) ÷ 4 × 100 = 6.35%.
5. Mean of actual = 175; Σ(actual − 175)² = 12,500. R² = 1 − 400 ÷ 12,500 = 0.968.

How to read the result

Every error here is 10, which is why MAE equals RMSE. R² = 0.968 means the model removes 96.8% of the squared variation around the mean. R² = 0 is no better than predicting the mean, and R² can be negative when the model is worse than that. Judge MAE and RMSE against the scale of the data and against a naive forecast such as "same as last period".

Common mistakes

• Using MAPE when some actual values are zero or near zero.
• Reading a high R² as proof that predictions are accurate enough for the business need.
• Comparing RMSE across datasets measured in different units.

Frequently asked questions

Should I use MAE or RMSE?

Use RMSE when large errors are especially costly; use MAE when you want a robust measure of the typical error.

Why does MAPE treat over- and under-forecasts differently?

With non-negative forecasts, an under-forecast can cost at most 100%, while an over-forecast has no upper limit, so minimising MAPE tends to favour forecasts that are too low.

Is R² here the same as the squared correlation?

Only for a least-squares fit with an intercept on the same data; for arbitrary predictions the two can differ.

Sources: R. J. Hyndman and A. B. Koehler, Another look at measures of forecast accuracy, International Journal of Forecasting, 2006 · R. J. Hyndman and G. Athanasopoulos, Forecasting: Principles and Practice, OTexts