Reading a Vendor's Case Study

"Client X increased conversions by 47% after implementing our platform." Lists of high-performing employers should be read with the same scrutiny; this example provides a concrete remote-company comparison.

The number is probably accurate. The implied claim — that the platform caused the increase — is usually unsupported, and the case study is constructed so that the gap is not visible.

Six questions expose it, and they take two minutes.

1. Compared with what?

The universal question. Against the previous period, against a control group, or against nothing?

Before-and-after is the weakest possible comparison. Everything else also changed: seasonality, other campaigns, the market, the client's own effort during a high-attention implementation period.

If there is no control group, the case study describes a period, not an effect. See correlation and causation.

2. What else changed at the same time?

Implementing a new platform is never the only change. It comes with a project team, renewed attention, a tracking rebuild, and usually a strategy review.

A "47% uplift" that coincides with a site redesign, a fixed tracking bug and three months of senior attention is not attributable to the software.

The case study will not list these. Their absence is not evidence they did not happen.

3. How was the metric defined, and did the definition change?

A platform migration frequently changes measurement. New tracking captures conversions the old setup missed. The number rises; the business does not.

This is extremely common and it is the most likely explanation for a large improvement, particularly where the previous setup was poor — which it usually was, since that is why they switched.

Ask whether the before and after were measured the same way. They rarely were.

4. Which client, out of how many?

One case study is a selected data point. The vendor has many customers, and the ones featured are the ones with good numbers.

This is survivorship in its purest form. The customers where nothing improved, or where the implementation failed, do not appear. See metrics that mislead.

The question worth asking directly: what is the median result across your customers, and how many churned in the first year? A vendor that can answer is telling you something; one that cannot, or will not, is also telling you something.

5. Is the base rate stated?

"47% increase" from what? A rise from 0.3% to 0.44% and a rise from 4% to 5.9% are both 47%, and they are different businesses.

Percentage change without the absolute numbers is not interpretable, and its use is a choice about what to make salient.

6. Is this business like yours?

Sector, scale, price point, purchase cycle, channel mix, market. A result from a business with a £15 impulse product does not transfer to one with a six-month sales cycle.

And the reverse is worth noting: if the case study client is much smaller than you, the implementation effort will not be comparable either.

The formats worth extra suspicion

"Up to" figures. The best result achieved by anyone, presented as the expected result.

Aggregate customer statistics — "our customers see an average 30% improvement" — with no methodology, no sample, and no definition of the comparison.

Award submissions. Written to win a category, judged on presentation rather than on rigour.

Results with no time period stated.

Testimonials in place of numbers, which is frequently what happens when the numbers were unremarkable.

What a credible case study looks like

They exist, and the differences are visible.

A control group or holdout, described.

Absolute numbers alongside percentages.

The time period, and what else changed in it, acknowledged.

A statement of what did not improve. A case study reporting a mixed result is far more credible than one reporting universal success, and vendors who publish these are worth taking more seriously.

Named methodology, and ideally a named person who ran it.

And the strongest signal: an incrementality test rather than a before-and-after. If a vendor has run a holdout with a client and published the result, that is a different class of evidence. See incrementality.

The questions to ask in the sales conversation

Since you will usually be reading these with a salesperson present.

"Was there a control group?"

"What else changed during that period?"

"Was the measurement the same before and after?"

"What is the median result across your customers?"

"Can I speak to a customer you have not chosen for me?"

"Do you have a case where it did not work, and what happened?"

The last two do most of the work. A vendor comfortable with both is in a different category from one who deflects.

The symmetric point

The same scrutiny applies to your own reporting.

Most internal case studies have the same weaknesses: before-and-after comparison, definition changes, selected examples, percentage without base.

If you would not accept a claim from a vendor, do not make it internally. The person receiving your report is in the same position you are in when reading theirs. See answering "did the campaign work" honestly.

The summary

The numbers are usually true; the causal claim usually is not.

Compared with what, and what else changed — two questions that resolve most of it.

A platform migration changes measurement, which is the most likely explanation for a large improvement.

Ask for the median result and an unselected reference. The response is more informative than the case study. For standards on supporting marketing claims, see FTC guidance on advertising substantiation.