Metrics That Mislead: Rates, Averages and Survivorship

Most misleading numbers in marketing reports are not wrong. They are correctly calculated answers to a question nobody asked, and three patterns account for most of them. Even apparently simple denominators need definitions; this explanation shows why annual work-hour totals vary by assumptions.

Rates: the denominator moves too

A conversion rate is a ratio, and it changes when either part changes. Reports treat it as a measure of performance, which it is only if the denominator is stable.

The classic case: conversion rate improved, sales fell. A campaign that stopped bringing in low-intent traffic raises the rate and lowers the total. The rate reports an improvement in something nobody wanted.

And the reverse. A successful awareness campaign brings a flood of early-stage visitors. Conversion rate drops. The campaign worked.

Watch the numerator and denominator separately. Always, and it takes no extra work — plot both. Most rate confusion disappears the moment the components are visible.

And beware rates whose denominator is a measurement artefact. If your denominator is "tracked sessions," a consent banner change moves the rate without anything happening in the business. See consent.

Simpson's paradox

The specific case of denominator movement that reverses conclusions, and it is worth knowing by name because it appears constantly in channel reporting.

A trend present in every subgroup can reverse when the subgroups are combined, if the mix changes.

Concretely: channel A converts better than channel B on mobile, and better on desktop. Combined, channel B looks better — because channel A's traffic is mostly mobile, where everyone converts less.

Both facts are correct. The aggregate is not a summary of the parts; it is a summary of the parts weighted by their sizes.

The practical instruction: when an aggregate surprises you, break it apart. And when the mix has changed between two periods, the aggregate comparison is not valid without accounting for it.

Averages describe nobody

An average is the right summary for a symmetric distribution. Almost nothing in marketing is symmetric.

Order value, session duration, customer lifetime value, revenue per customer — all heavily skewed. A small number of large values pulls the mean above nearly every observation.

Mean order value of £85 is consistent with most orders being £40 and a handful being £3,000.

Report the median alongside the mean. If they differ substantially, the mean is not a useful summary and the distribution needs looking at.

Report percentiles for anything skewed. The 25th, 50th and 75th tell you the shape in three numbers.

And watch the average of averages, which is wrong whenever the groups differ in size. The mean of five regions' conversion rates is not the overall conversion rate unless the regions are identical in traffic. This appears in dashboards constantly.

Survivorship

Your data is the data that survived to be recorded. What did not survive is invisible, and it is frequently the interesting part.

Customers who churned are absent from a satisfaction survey of current customers.

Sessions that failed — a JavaScript error, a timeout, a page that never finished loading — never fire the tracking event. Your analytics describes successful sessions.

Users who blocked tracking are absent by definition.

Campaigns that were cancelled early are absent from the analysis of campaign performance, and they were cancelled because they were failing.

The test archive that records only wins produces the same distortion within an organisation. See A/B tests.

The check: what would be missing from this dataset, and would its absence change the conclusion? Usually the answer is that the missing cases are the negative ones, which means every summary is optimistic by construction.

Three more that recur

Vanity metrics. Impressions, reach, followers, sessions. Real numbers, and they move independently of anything commercial. The test: if this number doubled and nothing else changed, would the business be better off? For most vanity metrics the answer is no.

Ratios where both parts are estimates. Cost per acquisition combines a spend figure you know precisely with a conversion count that is partly modelled and partly missing. The precision of the numerator lends unearned credibility to the ratio.

Percentage change from a small base. A 300% increase from three to twelve. Correct, and it belongs with the absolute numbers beside it.

Making a report harder to misread

Show absolute numbers next to every rate. The single most effective change.

Show the historical range, not just the current value and the previous one. A number outside its normal range is news; a number inside it is not, and twelve weeks of context makes that visible without any commentary.

Label the denominator. "Conversion rate (tracked sessions)" versus "conversion rate (all sessions, estimated)" are different metrics, and dashboards routinely present one as the other.

State whether a figure is observed or modelled, in the report rather than in a footnote.

Round honestly. Two decimal places on an estimate with a six-point interval is precision theatre.

Put the definition next to the number, or link it. Half the arguments about metrics are two people using one word for two calculations. See documenting metric definitions.

The questions to ask of any number in a report

• What is the denominator, and did it move?
• Mean or median, and how skewed is the distribution?
• What is missing from this dataset, and why?
• Observed or modelled?
• Is this outside its normal range, or inside it?
• If this number doubled, would anything actually be better?

Six questions, and they take a minute. Most misleading numbers fail at least one.

The summary

A rate moves when either part moves. Show both.

Averages describe skewed data badly. Show the median, and the shape.

Your data is what survived, and what did not is usually the negative half.

And the fastest improvement available to any report is putting the absolute numbers next to the rates, with twelve weeks of history behind them. The UK Statistics Authority publishes the Code of Practice for Statistics for trustworthy use and presentation of statistics.