Why Forecasting Models Degrade (and How to Build Ones That Last)

A forecast that was accurate last year is proof it was accurate last year. Nothing more.

Every demand, revenue, and staffing forecast is a bet that the future will behave like its history. Then pricing changes, a competitor shows up, or a data feed quietly breaks, and the number your team stocks, hires, and budgets against drifts from reality. Nobody gets an alert. The misses just grow until someone in the operating meeting stops trusting the model.

That decay is normal. Building forecasts that survive it is a design choice.

The world moves and the model does not

A forecasting model learns a relationship: when the inputs looked like this, the outcome looked like that. Then it assumes the relationship holds.

When it stops holding, researchers call it concept drift. A widely cited survey in ACM Computing Surveys defines it as the relation between the input data and the target variable changing over time.^[1] In plain words, the same signals start meaning something different.

It helps to split this in two.

One is a change in the mix. You start selling to smaller accounts, so the inputs look different even though each type of customer behaves as it always did. A model that has seen enough of each type can often absorb this.

The other is a change in behavior. A pipeline stage that used to convert steadily now converts less because buyers are slower to sign. Same inputs, different outcome. The survey notes this kind of drift is not visible from the input data alone; catching it typically relies on feedback about how the predictions are actually performing.^[1] Translation: you find it by comparing forecast to actuals, not by staring at the inputs.

Four patterns, four responses

The same survey describes drift arriving as sudden, incremental, gradual, or recurring. It adds a point worth remembering: a one-off outlier is not drift, and needs no adaptation.^[1]

Sudden. The world switches overnight. MIT Technology Review reported in May 2020, citing the consultancy Pactera Edge, that a supplier of sauces and condiments to retailers in India needed help after bulk pandemic orders broke its automated inventory system; the sales forecasts it used to reorder stock no longer matched what was selling.^[2] History before a break like that is partly the wrong teacher. Override, shorten the training window, and retrain once the new pattern has enough data.

Incremental. The change creeps. Deal cycles lengthen a few days a quarter. Order sizes shrink. No single month looks broken. This is what scheduled retraining, weighted toward recent history, is for.

Gradual. The new pattern comes and goes before it sticks, like customers moving between old and new product lines. Expect wider error bands for a while.

Recurring. Seasonality and year-end buying. If the model does not know about Q4, it is not degrading in Q4. It was never built for it. Put the calendar into the model.

The outlier, one enormous order or one freak storm week, gets flagged and explained, not learned.

A forecast is not a monument. It is a bet that has to be re-checked every cycle.

Some decay is plumbing, not the market

Sometimes customers did not change. The data did.

Google's published rules for machine learning engineering describe a joined table that stops updating while the system keeps running, decaying gradually instead of crashing. Google's Play team once found a table stale for six months, and refreshing it alone gave a 2% boost in install rate.^[3]

The mid-market version: an ERP migration changes a unit of measure, a renamed CRM field fills the feed with blanks, one region stops sending data. The forecast throws no error. It just gets worse. That is Bad Data = Bad Results, playing out after launch. A 2015 NeurIPS paper from Google engineers makes the broader point: the external world is rarely stable, and live monitoring is critical for long-term reliability.^[4]

Complexity is not durability

When a forecast breaks, the instinct is to buy a fancier model. The public evidence says be careful.

The M4 forecasting competition tested 61 methods on 100,000 time series.^[5] Of the 17 most accurate methods, 12 were combinations of mostly statistical approaches. The six pure machine learning methods performed poorly, none beating the combination benchmark.^[6] The winner was a hybrid of statistical and machine learning ideas, so the lesson is not "never use ML." It is that durable forecasts come from sound structure, combined approaches, and honest testing. M4 also scored prediction ranges, and the two most accurate methods did well at getting their 95% ranges right.^[6]

How to build forecasts that last

Test on periods the model never saw. The forecasting textbook by Rob Hyndman and George Athanasopoulos describes evaluation on a rolling forecasting origin: train only on data before a point in time, forecast forward, score it, move the point, repeat.^[7] Pretend it is last January, forecast February, check, then do March. Include a dumb baseline like "same month last year." If the model cannot beat it, do not pay to maintain it. (Why leaked future data makes backtests look brilliant: Feature Engineering for Tabular Business Data.)

Ship a range, not a number. The same textbook says point forecasts can be of almost no value without prediction intervals, and those intervals usually widen the further out you forecast.^[8] A range changes the decision. Stock high-margin, cheap-to-hold items toward the top of the band. Staff to the middle. Plan cash against the bottom.

Compare forecast to actual every cycle. The NeurIPS paper suggests a simple check: predictions should roughly match what actually happened in aggregate, and a change in that gap often signals the world shifted.^[4] Set a tripwire. If actuals land outside the 80% range several periods running, someone reviews the model before the next planning meeting.

Decide how stale is too stale. Google's rules suggest asking how much performance drops if the model is a day, a week, or a quarter old.^[3] A weekly replenishment forecast and an annual budget model need different retraining schedules.

Keep the human override, and log it. When you know something the model does not, like next month's price increase, adjust, write down why, and check later whether it helped.

What this changes on Monday

Picture a distributor that reorders weekly off a demand forecast. The ask is not predicting a pandemic. It is smaller: catch a supplier feed that went quiet before it causes a stockout, tell a seasonal spike from a real shift, and give buyers a range to order against with their judgment on top.

That is a durable forecast. Tested on periods it never saw, watched every cycle, retrained as fast as your market moves. It is the forecasting piece of Using Your Data, built to keep working past the first quarter.

If you want forecasts fitted to your order, pipeline, and capacity history and built to be monitored, talk to Gamify Data. We start with the data you already collect and build forecasts your planners can keep using.

Sources

1. João Gama, Indrė Žliobaitė, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia, "A Survey on Concept Drift Adaptation," ACM Computing Surveys, vol. 46, no. 4, March 2014. https://doi.org/10.1145/2523813
2. Will Douglas Heaven, "Our weird behavior during the pandemic is messing with AI models," MIT Technology Review, May 11, 2020. https://www.technologyreview.com/2020/05/11/1001563/covid-pandemic-broken-ai-machine-learning-amazon-retail-fraud-humans-in-the-loop/
3. Martin Zinkevich, "Rules of Machine Learning: Best Practices for ML Engineering" (Rules #8 and #10), Google for Developers, last updated August 25, 2025. https://developers.google.com/machine-learning/guides/rules-of-ml
4. D. Sculley et al., "Hidden Technical Debt in Machine Learning Systems," Advances in Neural Information Processing Systems 28 (NeurIPS 2015), December 2015. https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf
5. Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos, "The M4 Competition: 100,000 time series and 61 forecasting methods," International Journal of Forecasting, vol. 36, no. 1, January 2020. https://doi.org/10.1016/j.ijforecast.2019.04.014
6. Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos, "The M4 Competition: Results, findings, conclusion and way forward," International Journal of Forecasting, vol. 34, no. 4, October 2018. https://doi.org/10.1016/j.ijforecast.2018.06.001
7. Rob J Hyndman and George Athanasopoulos, "Time series cross-validation," Forecasting: Principles and Practice, 3rd edition, section 5.10, OTexts, 2021. https://otexts.com/fpp3/tscv.html
8. Rob J Hyndman and George Athanasopoulos, "Distributional forecasts and prediction intervals," Forecasting: Principles and Practice, 3rd edition, section 5.5, OTexts, 2021. https://otexts.com/fpp3/prediction-intervals.html