"Accuracy" is not a forecast metric. It is a word people use when they want you to stop asking questions. It carries the self-evident authority of a round number \u2014 92% sounds rigorous, sounds measured, sounds like someone ran a calculation and got a good result. But it could mean anything. It could mean the percentage of periods where the forecast was within 10% of actual. It could mean the percentage of SKUs where the direction was correct. It could mean the person subtracted their MAPE from 100 and called that "accuracy."<\/p>\n
Every one of those definitions will give you a different number. Every one of them can be gamed. And every one of them will look fantastic on a vendor slide until the day your warehouse is stuffed with stock nobody ordered and empty of everything customers actually want.<\/p>\n
Good forecasting requires a scorecard. Not a single number, not a marketing figure, but a set of diagnostic tools that each answer a specific question about forecast quality. The problem is that most supply chain professionals were never taught which tools exist, what they measure, or when each one lies to you.<\/p>\n
That is what this post fixes.<\/p>\n
We will cover:<\/p>\n
- \n
- The benchmark every forecast must beat before any metric matters<\/li>\n
- Four metrics \u2014 MAE, RMSE, MAPE, MASE \u2014 and the specific situations where each one misleads you<\/li>\n
- A decision matrix for picking the right metric for your situation<\/li>\n
- Residual diagnostics: the three charts that reveal whether your model is hiding something<\/li>\n
- The train\/test split: why in-sample accuracy is almost always a lie<\/li>\n<\/ul>\n
By the end, you will be able to look at any forecast \u2014 your own, your team’s, your vendor’s \u2014 and ask the questions that actually matter. Let’s build the toolbox.<\/p>\n
The Benchmark That Catches Liars: Naive and Seasonal Naive<\/h2>\n
Before we talk about metrics, we need to talk about yardsticks.<\/p>\n
A metric measures how far off your forecast was. A benchmark tells you whether that distance is impressive or embarrassing. You need both. Without a benchmark, any metric is just a number with no frame of reference.<\/p>\n
The most important benchmark in supply chain forecasting is the seasonal naive forecast<\/strong>. It is exactly what it sounds like: take last year’s value for the same period, and use it as the forecast. October’s forecast is last October’s actual. The forecast for week 23 of this year is the actual from week 23 of last year.<\/p>\n
That’s it. No model. No parameters. No training. An intern could build it in Excel in twenty minutes.<\/p>\n
And here is the embarrassing truth: on real supply chain data, it beats sophisticated models more often than anyone likes to admit.<\/p>\n
![\"A](\"https:\/\/inphronesys.com\/wp-content\/uploads\/2026\/04\/toolbox_snaive_beats_ets-2.png\")
<\/p>\nIn this chart, both models were trained on the same historical data. The ETS model is doing something genuinely clever \u2014 it’s estimating trend, level, and seasonal components simultaneously and updating them as errors arrive. The Seasonal Naive model is doing nothing except copying last year. On the training data, ETS wins comfortably. On the hold-out test period \u2014 the only accuracy that matters \u2014 Seasonal Naive edges ahead.<\/p>\n
This is not unusual. It is, in fact, the Hyndman & Athanasopoulos curriculum’s way of making a blunt point: a complex model that overfits the training data is not better than a simple model that generalises. The benchmark is the tribunal that makes this visible.<\/p>\n
The rule:<\/strong> before you report any accuracy metric, always compute the same metric against a naive or seasonal naive baseline. If your model doesn’t beat the baseline, you don’t have a model \u2014 you have a more expensive version of doing nothing. The
fpp3<\/code> package makes this effortless:SNAIVE()<\/code> andNAIVE()<\/code> are first-class model types infable<\/code>, fit and evaluated exactly like ETS or ARIMA.<\/p>\nThe naive benchmark is not the ceiling you aspire to. It is the floor you must clear. Once you’ve cleared it, the metrics below tell you by how much.<\/p>\n
Four Metrics, Four Personalities<\/h2>\n
Forecast accuracy metrics are not interchangeable. Each one measures something different, emphasises different kinds of errors, and breaks down under different conditions. Here are the four metrics that matter most in supply chain \u2014 what each one does, what it hides, and when it will lie to you.<\/p>\n
MAE \u2014 The Honest Average Miss<\/h3>\n
MAE stands for Mean Absolute Error. In plain English: take every forecast error (actual minus forecast), ignore whether it’s positive or negative, and average them. The result is in the same units as your demand \u2014 pieces, pallets, euros, kilograms.<\/p>\n
What it tells you:<\/strong> "On average, how many units did I miss by?" If your MAE is 45 units and you produce in batches of 200, you are probably fine. If your MAE is 45 units and your batch size is 50, you have a serious problem. The units-level interpretation is MAE’s greatest strength \u2014 it is immediately operationally meaningful.<\/p>\n
What it hides:<\/strong> The averaging is symmetric and linear. A forecast that misses by 100 units in January and nails February and March perfectly has the same MAE as one that misses by 33 units every single month. These are very different demand patterns, but MAE treats them identically. It also hides the direction of bias \u2014 you can’t tell from MAE alone whether you are systematically over-forecasting, under-forecasting, or just noisy. For that, you would look at ME (Mean Error) separately, or examine the residual plot (more on that below).<\/p>\n
When to use it:<\/strong> MAE is the right default for high-volume, stable SKUs where demand is well above zero and you care primarily about average operational impact. It is the most intuitive metric for operations teams and S&OP conversations \u2014 straightforward to explain, hard to misinterpret, impossible to game through rescaling. It is the reference metric that MASE is defined against, which is a strong vote of confidence from the people who invented the alternatives.<\/p>\n
RMSE \u2014 The Metric That Hates Outliers<\/h3>\n
RMSE stands for Root Mean Squared Error. The "squared" part is where the action is: instead of taking the absolute value of each error, you square it before averaging, then take the square root at the end. This has one important consequence: large errors get punished much more heavily than small ones.<\/p>\n
![\"Two](\"https:\/\/inphronesys.com\/wp-content\/uploads\/2026\/04\/toolbox_mae_vs_rmse-2.png\")
<\/p>\nIn this chart, both Forecast A and Forecast B have exactly the same MAE. Their average miss in absolute terms is identical. But look at the error distribution: Forecast A has consistently moderate errors spread across the period. Forecast B is accurate most of the time \u2014 and then catastrophically wrong in one period, with a miss several times larger than anything Forecast A produces.<\/p>\n
If you evaluate both forecasts using MAE, they look equally good. If you evaluate them using RMSE, Forecast B looks significantly worse, because that one enormous miss gets squared before averaging. RMSE "sees" the outlier. MAE does not.<\/p>\n
When does this matter?<\/strong> Think about the cost structure of your operation. For perishables<\/strong>, a single large over-forecast means spoilage \u2014 not proportionally worse than a moderate miss, but catastrophically worse. For critical spare parts<\/strong> in capital equipment, a single large under-forecast means a production line going down. In both cases, the penalty for a big miss is non-linear \u2014 one huge error costs far more than many small ones of the same total size. RMSE models that cost structure. MAE does not.<\/p>\n
What RMSE hides:<\/strong> Because outlier periods dominate the RMSE calculation, the metric can be misleading if your historical data contains a few anomalous periods (promotions, pandemic disruptions, one-time customer orders). Those periods inflate RMSE in ways that don’t reflect typical operating conditions. If you compare models on RMSE and the dominant signal is one or two extraordinary months, you may end up selecting for robustness to those specific months rather than general forecasting skill.<\/p>\n
Use RMSE when big misses are genuinely more costly than their size suggests, and when your data is clean of one-off outliers that you don’t need your model to handle.<\/p>\n
MAPE \u2014 The Percentage Everyone Quotes, the Metric That Lies on Slow Movers<\/h3>\n
MAPE stands for Mean Absolute Percentage Error. It takes the absolute error for each period and expresses it as a percentage of the actual demand for that period, then averages those percentages. This is why supply chain managers love it: "Our MAPE is 12%" communicates something intuitive, scale-free, and easy to put in a board presentation.<\/p>\n
It is also the most dangerous metric to use on slow-moving and intermittent demand.<\/p>\n
![\"MAPE](\"https:\/\/inphronesys.com\/wp-content\/uploads\/2026\/04\/toolbox_mape_explodes-2.png\")
<\/p>\nHere’s what goes wrong. The MAPE calculation divides the error by the actual demand. When actual demand is 100 units and you missed by 10, the percentage error is 10% \u2014 sensible. When actual demand is 2 units and you missed by 2, the percentage error is 100% \u2014 painful but interpretable. When actual demand is 1 unit and you missed by 2 (a very common scenario for slow movers), the percentage error is 200%. When actual demand is 0 and any forecast at all is non-zero, you are dividing by zero \u2014 and MAPE becomes undefined or, in some implementations, infinity.<\/p>\n
Dividing by something close to zero makes the percentage explode. A single period of near-zero demand can dominate the entire MAPE calculation, making an otherwise reasonable forecast look catastrophic. Worse, because you’re dividing by the actual (not the forecast), over-forecasting and under-forecasting are penalised asymmetrically: a forecast of 100 when actual is 1 gives 9,900% error, while a forecast of 0 when actual is 100 gives 100% error. The same absolute miss creates wildly different MAPE values depending on direction.<\/p>\n
What MAPE is genuinely good for:<\/strong> High-volume, well-above-zero demand with no intermittent periods. Items like fast-moving consumer goods, core catalogue items with stable high demand, or categories where zero-demand periods simply don’t occur. In those settings, MAPE is intuitive, scale-free, and easy to communicate. The number means what it sounds like.<\/p>\n
What MAPE hides on everything else:<\/strong> Slow movers, spare parts, seasonal items with deep troughs, anything with intermittent demand \u2014 MAPE will misrepresent every one of them. If your portfolio contains a mix of fast and slow movers, averaging MAPE across the portfolio is mathematically meaningless, because the slow-mover percentages will swamp the fast-mover ones regardless of which model performs better overall.<\/p>\n
You will see MAPE in almost every vendor dashboard and S&OP report. That does not make it right. It makes it familiar.<\/p>\n
MASE \u2014 The Scale-Free One Rob Hyndman Wishes You’d Use<\/h3>\n
MASE stands for Mean Absolute Scaled Error. It was introduced by Rob Hyndman and Anne Koehler in 2006 precisely because MAPE was \u2014 their words \u2014 "widely used, but can be infinite or undefined, and it is not meaningful if actual values are close to zero."<\/p>\n
MASE solves this by changing the denominator. Instead of dividing each error by the actual demand (which causes the explosion on slow movers), MASE divides by the average error that a seasonal naive forecast would have made on the training data. The result is a ratio: above 1 means your model made larger errors than seasonal naive, below 1 means your model beat it.<\/p>\n
In plain English: MASE below 1 means you beat naive. MASE above 1 means you should fire the model and use naive instead.<\/strong><\/p>\n
![\"Same](\"https:\/\/inphronesys.com\/wp-content\/uploads\/2026\/04\/toolbox_mase_scalefree-2.png\")
<\/p>\nThis chart shows the same forecast model applied to three SKUs with very different demand volumes: a slow mover averaging 10 units\/month, a medium-volume item at 1,000 units\/month, and a fast mover at 100,000 units\/month. MAPE produces wildly different numbers for each \u2014 not because the model performs differently, but because MAPE is sensitive to the volume scale. MASE produces similar values across all three, because it is always measuring relative to seasonal naive on that specific SKU’s own scale. You can meaningfully average MASE across a mixed-volume portfolio. You cannot do that with MAPE.<\/p>\n
What makes MASE special:<\/strong> It handles zero-demand periods gracefully (no division by zero problem), it is scale-free (meaningful across SKUs of any volume), it has a built-in interpretation threshold (1.0), and it automatically adjusts for seasonality. The denominator is calculated using the seasonal period of your data, so a monthly model uses a 12-period lag, a weekly model uses a 52-period lag \u2014 no configuration required.<\/p>\n
The honest caveat:<\/strong> MASE is harder to explain to stakeholders who’ve only ever seen MAPE. "Our MASE is 0.78" doesn’t communicate intuitively to a category manager the way "12% MAPE" does. This is a communication problem, not a statistical one. Use MASE for model selection and portfolio benchmarking. Keep MAPE for high-volume SKU dashboards where it’s honest. Explain the distinction.<\/p>\n
Which Metric When<\/h2>\n
The four metrics each shine in different situations. Here’s the framework:<\/p>\n
![\"Visual](\"https:\/\/inphronesys.com\/wp-content\/uploads\/2026\/04\/toolbox_metric_decision_matrix-2.png\")
<\/p>\nA few patterns worth noting:<\/p>\n
The mixed-portfolio trap.<\/strong> If you have 500 SKUs ranging from slow movers to fast movers and you want to know which forecast method performs best across the portfolio, MASE is the only valid aggregation. Every other metric will be dominated by volume-outlier SKUs or, in MAPE’s case, by slow movers with near-zero demand.<\/p>\n
The stakeholder communication problem.<\/strong> MASE is hard to explain. For a C-suite dashboard, MAPE or MAE are more intuitive \u2014 but only compute them on the SKUs where they’re valid. Never show MAPE averaged across a mixed portfolio. Consider showing MAE in units alongside MASE as the "honest" metric, so stakeholders get both interpretability and rigour.<\/p>\n
The perishables exception.<\/strong> RMSE is the right instinct when your cost function is non-linear \u2014 when one large miss costs much more than many small ones. But RMSE inflates when historical data contains event-driven outliers. If you use RMSE for perishables model selection, clean the anomalous periods from your training data first.<\/p>\n
The default.<\/strong> When in doubt, compute all four. The fpp3
accuracy()<\/code> function returns MAE, RMSE, MAPE, and MASE simultaneously \u2014 there is no reason to compute only one metric when you can compute all four in a single line.<\/p>\nResidual Diagnostics \u2014 Reading the Model’s Conscience<\/h2>\n
Accuracy metrics tell you how far off the forecast was. Residual diagnostics tell you why<\/em> \u2014 or more precisely, whether the model has anything left to learn from the data.<\/p>\n
Residuals<\/strong> are the difference between what the model predicted and what actually happened, calculated on the training data where the model had access to each observation. If a model has learned everything there is to learn from the historical pattern, the residuals should look like random noise: no structure, no drift, nothing your eye can grab onto. If there is visible structure in the residuals, that structure is signal the model missed \u2014 and it probably means the forecasts will be systematically wrong in predictable ways.<\/p>\n
The three-panel residual diagnostic plot is your standard tool for this check.<\/p>\n
![\"Three-panel](\"https:\/\/inphronesys.com\/wp-content\/uploads\/2026\/04\/toolbox_residual_diagnostics-2.png\")
<\/p>\nPanel 1 \u2014 Residuals over time.<\/strong> This is simply the residuals plotted as a time series. What you’re looking for is nothing \u2014 a flat cloud of points with no visible trend, cycle, or seasonal pattern. If you can see a pattern (drifting upward, oscillating, or with one distinct shift), the model has missed some structure. Common violations in supply chain data: residuals that drift after a structural break (a product launch, a logistics change, a major new customer), or residuals that show seasonal structure (which tells you the model’s seasonal component is wrong).<\/p>\n
Panel 2 \u2014 ACF (Autocorrelation Function).<\/strong> The ACF plot shows whether current residuals are correlated with past residuals at various time lags. In a well-specified model, residuals at lag 1, lag 2, lag 3… should all be approximately zero \u2014 knowing what happened last month shouldn’t help you predict this month’s residual. White noise<\/strong> is the technical term for this ideal state. In the ACF plot, white noise looks like bars all staying inside the blue confidence bands. If bars stick out above the bands at any lag, that lag is significant \u2014 there is autocorrelation in the residuals, which means there is exploitable signal the model hasn’t captured. This is the forecasting equivalent of leaving money on the table.<\/p>\n
Panel 3 \u2014 Histogram.<\/strong> The histogram shows the distribution of residuals. You’re looking for something roughly bell-shaped and centred near zero. A distribution centred near zero means the model is unbiased<\/strong> \u2014 it’s not systematically over- or under-forecasting. A distribution skewed to the right means the model regularly over-forecasts. Skewed left means systematic under-forecasting. A well-shaped distribution suggests the residuals are plausibly random; a heavily skewed one tells you the forecast will be wrong in a predictable direction.<\/p>\n
The Ljung-Box test<\/strong> is the statistical judge that formalises what your eye sees in the ACF. It tests whether the residuals are legally<\/em> allowed to be called noise \u2014 that is, whether the observed autocorrelation pattern could plausibly have arisen by chance. A high p-value (> 0.05) means you cannot reject the "this is noise" hypothesis. A low p-value means there is statistically significant autocorrelation \u2014 the residuals are not noise, the model has missed something. In fpp3,
gg_tsresiduals()<\/code> produces all three panels in one call, and you can run the formal test withBox.test()<\/code> on the residuals vector or withfeatures(.resid, ljung_box, lag = 12)<\/code> on the augmented tibble.<\/p>\nResidual diagnostics are not pass\/fail in practice. Almost no real-world model produces perfectly white noise residuals on messy supply chain data. The question is: how much structure is left? Is it economically meaningful? A slight first-lag autocorrelation on monthly demand might be statistically significant but operationally negligible. A strong seasonal pattern in the residuals of what was supposed to be a seasonal model is never negligible.<\/p>\n
Look at the residuals. Always.<\/p>\n
The Train\/Test Split \u2014 The Only Honest Accuracy Test<\/h2>\n
Here is a statistic that should make you uncomfortable: every forecasting model ever fitted to historical data will look better on that same historical data than on any future data it has never seen.<\/p>\n
This is not a flaw of bad models. It is a mathematical certainty for all models. When a model is fitted to a dataset, it adjusts its parameters to explain the observed history as well as possible \u2014 and it will always do this at least a little too well, capturing some of the random noise in that specific history as if it were real signal. Apply the model to new data, and those "signals" are just noise \u2014 and the model is wrong in ways the training accuracy didn’t predict.<\/p>\n
In-sample accuracy is not accuracy. It is a description of the past.<\/strong><\/p>\n
The only honest test of forecast skill is holding out data the model has never seen \u2014 fitting the model on an earlier period, then measuring performance on a later period that was deliberately excluded from fitting.<\/p>\n
![\"Time](\"https:\/\/inphronesys.com\/wp-content\/uploads\/2026\/04\/toolbox_train_test_split-2.png\")
<\/p>\nThe mechanics are simple. Take your full time series and split it at a cutoff date. Everything before the cutoff becomes the training set<\/strong> \u2014 the model sees this data and learns from it. Everything after the cutoff is the test set<\/strong> (sometimes called the hold-out set) \u2014 the model is evaluated here, but these observations were never available during fitting. The accuracy metrics you compute on the test set are what you report. The accuracy on the training set is, at best, a sanity check.<\/p>\n
How much data to hold out? The standard guidance is to use a test set at least as long as the forecast horizon you care about. If you want 12-month forecasts, hold out at least 12 months. Longer is generally better \u2014 a hold-out of 6 periods gives you a noisier accuracy estimate than a hold-out of 24 periods. But your training set also needs enough data to fit the model reliably, particularly for seasonal models that need to observe several full seasonal cycles.<\/p>\n
In fpp3,
filter_index()<\/code> makes this split clean and readable in a single pipe \u2014 see the collapsible R code at the end of the post for the full pipeline.<\/p>\nA preview of what’s coming.<\/strong> The single train\/test split is the honest test. But it has a weakness: the result depends on where you draw the line. A model that happened to perform well during 2023 might have performed very differently if you’d held out 2021 instead. Time series cross-validation<\/strong> \u2014 running the honest test many times, at many cutoff points, and averaging the results \u2014 is the rigorous solution to this problem. That is the technique we’ll use in the next post to pit six models against each other and pick a winner.<\/p>\n
Interactive Dashboard<\/h2>\n
The four metrics only become intuitive once you’ve felt<\/em> what they do on different demand profiles. You can read about MAPE exploding on slow movers \u2014 or you can turn a slider and watch it happen.<\/p>\n