MAPE, MAD and Bias: How to Measure Forecast Error (Formulas and Example)

Why measure forecast error

A forecast that is not measured does not improve. And not only that: without measuring, you do not know which part of the process adds value. Does the statistical model get it right and the sales adjustments spoil it? Or the reverse? Does the error come from three families or is it spread out? Do you miss upward (inventory) or downward (stock-outs)?

Forecast error metrics answer those questions. They are few and simple, but each has a trap, so it is worth understanding what each one measures before putting it on a dashboard.

The example we will use

Three items, one month, forecast against actual demand (in units):

Convention: error is forecast minus actual, so positive means over-forecast. Every tool uses its own; what matters is to fix it and document it.

MAD: mean absolute deviation

The average of absolute errors, in units. In the example: 175 ÷ 3 = 58.3 units.

It is easy to understand and is used to size safety stock and for the tracking signal. Its limit: it is in units, so it does not allow comparing items of different volume or aggregating into an accuracy percentage.

MAPE: mean absolute percentage error

The average of each row\'s percentage error. In the example: (100/1000 + 60/200 + 15/10) ÷ 3 = (0.10 + 0.30 + 1.50) ÷ 3 = 63 %.

And here the problem appears. Item 3, which weighs less than 1 % of demand, has a 150 % error and drags the average to 63 %, when the big item was forecast with a 10 % error. Also, if an item has zero demand, the division is undefined. That is why MAPE:

• Works per item, with continuous demand, for a conversation about that series.
• Is not averaged across items to get the catalog error.
• Does not work for intermittent demand.

WAPE: the weighted version

The sum of absolute errors divided by the sum of demand. In the example: 175 ÷ 1,210 = 14.5 %.

That number does describe what happened: the catalog was forecast with an error of 14.5 % of volume. Each item weighs what it sells, it is not undefined with zeros and it aggregates correctly from item to family to total (always summing errors and demand before dividing, never averaging WAPEs). It is the accuracy metric we recommend for any dashboard. Some companies publish its complement as "forecast accuracy": 100 % − 14.5 % = 85.5 %.

Bias: the signed error

The sum of signed errors divided by demand. In the example: −25 ÷ 1,210 = −2.1 %: overall there was a slight under-forecast.

Bias says something WAPE cannot: which way you miss. Two catalogs with 15 % WAPE are very different if one has zero bias (errors cancel out) and the other has +14 % (everything over-forecast). The first needs safety stock; the second has a systematic problem, almost always in commercial adjustments, and it is inventory piling up.

Bias is examined by family and by who adjusts, because the total usually cancels out: item 1 at −10 % and item 2 at +30 % give a small total bias that hides two opposite problems.

Tracking signal

The cumulative sum of (signed) errors divided by the MAD. If the forecast has no bias, errors cancel out and the signal hovers near zero. When it leaves the ±4 range (some use ±3 or ±6), the model has stopped working for that item: demand changed and the model has not noticed.

Its use is as an alarm: in a catalog of a thousand items nobody looks at a thousand series every month, but they can review the twenty whose tracking signal spiked.

MASE: comparing against a naive method

The forecast\'s mean absolute error divided by what a naive method would have had (for example, "next month sells the same as this one", or the average of the last three months). A MASE of 0.7 means the forecast reduces error by 30 % compared with doing nothing sophisticated; a MASE above 1 means the naive method does better.

It is the metric that answers the uncomfortable question: is the model worth it? It works with intermittent demand (it does not divide by actuals), it is comparable across items and it was the basis of the M5 competition (with its RMSSE variant). Its only drawback is that it is less intuitive than a percentage for leadership; that is why it usually coexists with WAPE.

Which one to use in each case

How to organize them in a dashboard, with the other inventory and service indicators, is in demand planning KPIs.

Mistakes when computing them

1. Averaging MAPEs. The example shows it: 63 % average MAPE against 14.5 % WAPE for the same month. Sum errors and demand before dividing.
2. Measuring against sales when there were stock-outs. The forecast seems to over-forecast; demand was there, product was missing.
3. No defined lag. The "current" forecast was made with almost all the information. Measure the one made when the decision was taken.
4. Mixing levels. A 12 % at family level is not comparable with a 30 % at item level. Publish each with its level.
5. Changing the sign convention. If one month positive bias is over-forecast and another it is under-forecast, the series is useless. Fix it in writing.
6. Measuring the adjusted forecast without measuring the statistical one. Keep both and compare them: it is the only way to know whether adjustments add value.
7. No naive reference method. A 20 % WAPE can be excellent or terrible depending on what the three-month average gives.