Forecasting: What a Model Is Entitled to Claim

A forecast says: given these assumptions and given that the past resembles the future, this is the expected range. For an applied example of a labour-cost calculation, this resource explains a wage percentage calculator.

By the time it reaches the meeting it has usually become a single number on a slide, with the assumptions and the range removed. Everything that goes wrong afterwards follows from that removal.

What a forecast is entitled to say

Within the range of conditions it has seen. A model fitted on spend between £50k and £150k a month can say something about £120k. It cannot say anything reliable about £400k, and it will produce a number anyway.

Extrapolation beyond the observed range is the most common failure, and it is invisible in the output. The model does not report that it is guessing.

Assuming the structure holds. If a competitor exits, a platform changes its algorithm, or the product changes, the relationships the model learned may no longer apply.

As a range, not a point. Any forecast worth having comes with an interval, and the interval widens with horizon.

For a stated horizon. A model that predicts next month well may be useless at twelve months, and the error compounds.

The three things to check

How does it do out of sample? Fit on the historical data is trivial to achieve and means little. Hold back the most recent months, predict them, compare. This is the only meaningful validation and it is frequently skipped.

How wide is the interval, in money? If the range spans an amount that would change the decision either way, the forecast does not support that decision.

What happens if a key assumption moves? Re-run with different assumptions about the biggest drivers. If the conclusion flips under a plausible variation, the conclusion belongs to the assumption.

Simple beats complex more often than expected

A well-established result in forecasting, and it runs against instinct.

Simple methods — a seasonal naive baseline, an exponential smoothing model — frequently outperform elaborate ones, particularly at short horizons and with limited data.

Always compute a naive baseline. Last year's same period, adjusted for trend. If the sophisticated model does not beat it materially out of sample, the sophistication is not earning anything.

This is worth doing before building anything, because it sometimes ends the project favourably.

Where marketing forecasts go wrong specifically

Assuming linear returns to spend. Doubling spend does not double results. Saturation is real, and a forecast built on a linear relationship will be badly wrong at the top of the range — which is exactly where budget decisions are made. See mix modelling.

Ignoring the feedback loop. Budgets respond to performance, so historical spend and performance are entangled. A forecast that treats spend as independent is modelling a relationship that does not exist as stated. See correlation and causation.

Forecasting the metric rather than the business outcome. Sessions, clicks and impressions can all be forecast accurately while revenue does something else.

Not accounting for known future changes. A price rise, a product launch, a channel being switched off. The model extrapolates the past; the future has scheduled events in it, and adding them is manual work someone must remember.

Point forecasts becoming targets. Once a forecast number becomes a commitment, the incentive shifts from accuracy to defensibility, and forecasts stop being honest. This is an organisational failure rather than a modelling one, and it is the most damaging on this list.

Presenting one

Lead with the range, in money.

State the horizon and show that the range widens with it. A fan chart makes this obvious without any explanation.

List the assumptions on the same page, not in an appendix. Three or four that drive the answer.

Give scenarios rather than one number where the decision is significant. Pessimistic, central, optimistic, each with what would have to be true.

Say what the model has not seen. If spend is planned above the historical range, say that the forecast is extrapolating there.

Show the previous forecast against what happened. The single most credibility-building thing available, and almost nobody does it.

Tracking accuracy

The discipline that makes forecasts improve rather than repeat.

Record every forecast with its date, horizon and interval.

Compare against actuals when the period closes.

Check calibration, not just error. If your 80% intervals contain the outcome roughly 80% of the time, the uncertainty is honest. If they contain it 40% of the time, the intervals are too narrow — which is the usual direction.

A forecast log is a small file and it is the strongest argument you will ever have in a conversation about uncertainty. See explaining uncertainty.

When not to forecast

When the honest interval is wider than the decision needs. Say so rather than producing a number that will be treated as more certain than it is.

When there is no history, or the history is from a different business.

When something structural is changing, and the model has no way to represent it.

In these cases, scenarios beat forecasts. "If the channel performs as it did last quarter, this; if it performs like the launch period, that." Two clearly-labelled possibilities are more honest and more useful than one number with hidden assumptions.

The summary

A forecast is conditional, and the conditions get stripped off on the way to the meeting. Keep them attached.

Validate out of sample, and always against a naive baseline — which wins more often than people expect.

Extrapolating beyond observed conditions is the most common failure and the model will not warn you.

Track calibration over time. Intervals that contain the outcome as often as they claim are worth more than any single accurate prediction. For a technical overview of forecasting methods, see NIST forecasting guidance.