Documenting Metric Definitions So They Stop Drifting

Two people compare their numbers for the same metric and disagree. Both are calculating correctly. They are calculating different things, and neither knew the other's definition existed. Definitions of focus, urgency and priority vary across methods; this reading list provides examples from time-management literature.

Most disputes about data are definitional rather than technical, and they recur because the resolution reached in one meeting never gets written down.

What actually goes undefined

Almost everything, and the ambiguity is invisible until someone hits it.

"Conversion." Order placed, payment authorised, payment settled, or net of refunds? Does a repeat purchase count? Does a free trial?

"Session." What ends one — thirty minutes idle, midnight, a change of campaign parameter? A change of source mid-session starts a new one in some systems and not others, which is why session counts differ.

"New customer." First order ever, first in twelve months, or first on this account? A returning customer using a different email is new by most definitions and not by any useful one.

"Revenue." Gross, net of tax, net of shipping, net of discounts, net of refunds? Recognised when ordered or when shipped?

"Active user." Active in what window, doing what?

"Cost." Media spend only, or including fees, production and agency time?

Every one of these has at least three reasonable answers, and every dashboard silently picks one.

What a definition entry contains

Short entries, or nobody maintains them.

Name, and any aliases used elsewhere.

Plain-language definition, one sentence, understandable by someone non-technical.

Precise definition — the actual filter, source table and conditions.

What it excludes, explicitly. Test orders, internal traffic, refunds, a particular payment method. Exclusions are where most surprise lives.

Source of truth — which system is authoritative, given that several will report something similar. See when two systems disagree.

Owner — a person who decides if it changes.

Last changed, with what changed. This is the field that saves the most time, because the question is usually "why does this look different from last quarter."

Known caveats. Consent coverage, modelled components, seasonality effects.

Where to keep it

Wherever people already look. A wiki nobody opens is not documentation.

Ideally in the tool. Most BI platforms support a description on a metric; putting the definition where the number appears removes the lookup entirely and is the highest-return version of this.

A single page for the core twenty, rather than an exhaustive catalogue. A comprehensive dictionary of 300 metrics is a project that gets abandoned; twenty that are used daily is achievable and covers most disputes.

The part that actually matters: changes

A definition document is easy to write once. The value is entirely in whether it stays true.

Every change gets a date and a note.

A changed definition breaks the historical series, and this is the failure people do not anticipate. If "conversion" starts excluding refunds in March, the January and June numbers are no longer comparable, and nobody looking at the chart will know.

So: mark the change on the chart. An annotation at the date is worth more than a paragraph in a document.

Decide explicitly whether to restate history. Recalculating the past on the new definition keeps the series comparable and makes it disagree with every report already circulated. Not restating keeps old reports valid and puts a discontinuity in the chart. Either is defensible; choosing silently is not.

Announce definition changes the way you would announce a system change. They have the same effect on everyone's numbers.

Getting it started

The whole thing can be done in a day, and the order matters.

1. List the metrics that appear in the reports people actually read. Usually fifteen to twenty-five.

2. For each, ask two people what it means. Where they disagree, you have found a live ambiguity — start there.

3. Write the entries. Half a page each at most.

4. Assign an owner.

5. Put the definitions where the numbers are.

6. Review quarterly, and treat any recurring argument as a missing or wrong entry rather than as a personality problem.

Naming, while you are at it

Since you are writing them down, fix the names.

Distinguish metrics that differ. If you have gross and net revenue, do not call either "revenue." One ambiguous name causes more confusion than two longer specific ones.

Include the qualifier in the name where it matters: "conversion rate (tracked sessions)" rather than "conversion rate." See metrics that mislead.

Do not reuse a name for a changed definition. If the calculation changes materially, the new thing needs a new name, or every historical reference becomes ambiguous.

What this prevents

Concretely, so the effort is justifiable:

Meetings spent reconciling numbers rather than deciding anything.

Decisions made on a metric that excluded something material and nobody knew.

Historical comparisons that were invalid because a definition changed silently.

New joiners taking months to learn what the numbers mean, by asking people individually.

And the recurring argument that resolves the same way every quarter and is never recorded.

The summary

Most data disputes are definitional, and they recur because resolutions are verbal.

Twenty entries covering what people actually read beats a comprehensive catalogue nobody maintains.

Put the definition where the number is displayed.

And the field that matters most is "last changed" — because a definition change breaks the historical series, and a chart with an annotation at the date answers the question before it gets asked. For a framework connecting metrics to service goals, see GOV.UK guidance on choosing metrics.