Answering \\"Did the Campaign Work?\\" Honestly

The campaign ran. Sales went up. Someone wants to know whether it worked. Before turning an observed event into a personnel conclusion, this overview shows how broad lists of fireable offences can hide the need for evidence and policy context.

Without a comparison group, that question has no clean answer, and the two available responses — a confident yes and a lecture about causal inference — are both bad. One is wrong; the other loses you the room.

There is a third response, and it is the useful skill.

Why sales going up is not the answer

Sales move for many reasons at once. During the campaign there was also seasonality, a competitor's price change, the weather, a public holiday, a product review, and whatever else was running.

Attribution does not resolve this. It records which touchpoints appeared before conversions; it does not construct the world where the campaign did not run. See what attribution actually measures.

The uncomfortable core: to say the campaign worked, you need to know what sales would have been without it, and nobody observed that.

What to say when there is no comparison group

Not "we cannot know." That is true and useless, and it makes you the person who never has an answer.

Say what you can see, at what confidence, and what would settle it.

Something like:

"Sales were 14% above the previous four weeks. Normal week-to-week variation is around 8%, so this is above the usual range but not far outside it. Two other things changed in the same period — the price promotion and the seasonal peak — and I cannot separate their contributions from the campaign's.

What I can say: nothing suggests the campaign hurt, and the movement is consistent with it having helped. What I cannot say is how much.

If we hold back the next campaign in a few matched regions, we get an actual number in three weeks."

This is a complete answer. It reports what was observed, states the uncertainty, names the confounders, and ends with a route to a real answer. It also takes about forty seconds.

Making the observation as useful as it can be

Since a descriptive answer is often all you have:

Show the metric against its own history, with the normal range visible. A single before-and-after comparison invites over-reading; twelve weeks of context makes the size of the movement obvious without any argument.

Name every other thing that changed in the period. Price changes, promotions, seasonality, product launches, PR, competitor activity, site changes, and — frequently forgotten — tracking or consent changes. See consent.

Check whether the shape is plausible. If the campaign ran for four days and the increase began two weeks earlier, that is informative.

Look at unexposed segments if any exist. Regions the campaign did not reach, or channels it did not touch, are an accidental control group. Frequently they move too, which is the finding.

Look at the intermediate steps. If the campaign was meant to drive traffic and traffic did not move, the sales increase came from somewhere else regardless of what attribution says.

Setting it up so next time is different

The real fix is upstream of the question.

Agree the measurement before the campaign runs, not after. A conversation at planning about how success will be judged takes twenty minutes and changes the entire nature of the reporting.

Hold something back. Regions, an audience segment, a period. The cost is the forgone spend in the holdout; the benefit is knowing whether the rest of the budget did anything. See incrementality.

Write down what result leads to what decision, in advance. A test with no pre-agreed decision rule produces a discussion; one with a rule produces a decision.

Stagger changes. If the promotion and the campaign launch together, neither can be measured. Separating them by a fortnight costs nothing and makes both interpretable.

Keep a log of everything that changed, on one timeline. This single artefact answers more questions than most dashboards, and almost nobody keeps one.

Handling the pressure to be certain

The person asking usually needs to justify a decision to someone else, and a hedged answer is inconvenient for them. That is a real problem and it deserves respect rather than a lecture.

Give them the strongest honest sentence. There usually is one: "the evidence is consistent with it working, and we cannot size the effect." That is defensible, and it is something they can repeat.

Offer certainty on something adjacent that is measurable. Cost per acquisition within the channel, reach, the fact that the creative was delivered — these are observable and they are not nothing.

Do not let "we cannot be certain" become "we have no information." Distinguish clearly between not knowing the size of an effect and having no evidence at all.

And commit to a date for the real answer. "The next campaign will have a holdout and we will have a number by the end of the month" turns your caveat into a plan rather than an obstacle.

What not to do

Do not present the attribution number as the answer. It is a description of observed touchpoints, and presenting it as causal is how the wrong budget decision gets made.

Do not compare against the immediately preceding period only. Two weeks against two weeks ignores seasonality and normal variance.

Do not quietly pick the flattering window. Choosing the comparison period after seeing the data is a form of the multiple comparisons problem, and it is more common than anyone admits.

Do not answer with a number you would not defend under questioning. The follow-up question — "how do you know?" — arrives eventually, and it is much better to have been hedged from the start.

The summary

Without a comparison group, "did it work" cannot be answered cleanly, and pretending otherwise is how bad budget decisions get made confidently.

Give the descriptive answer with its uncertainty, name the confounders, and propose the test. Forty seconds, and it is both honest and useful.

Agree the measurement before the campaign, hold something back, and keep one timeline of everything that changed.

The question is always the same one: compared with what? For evaluation design and causal questions, consult the Magenta Book.