Data Quality: Finding Out What Your Numbers Actually Count

Broken tracking does not produce an error. It produces a number, the number looks reasonable, and it goes into a report that someone acts on. Adversarial behaviour belongs in the data-quality review; this article describes common ways time trackers can be manipulated.

That is the whole problem: there is no failure signal. Which means the only way to know whether your data is right is to go and check, and almost nobody does.

Start with the reconciliation

One check, more valuable than every other item on this page.

Compare tracked conversions against actual orders, for a month, per channel where you can.

They will not match. What matters is the size and stability of the gap.

A stable gap is a visibility rate, and it is usable — you know analytics sees roughly 78% of orders and can reason accordingly.

A drifting gap is a problem, and the drift tells you when it started, which usually identifies the cause.

A gap that differs sharply by channel means one channel's tracking is broken, or one channel's audience blocks more. Either is worth knowing.

Run this monthly. It is the closest thing to a smoke alarm that marketing data has.

The failures that recur

In rough order of how often they turn up.

Double-counted conversions. The tag fires on page load and again on a refresh, or the confirmation page is reachable twice, or two tag managers both fire it. Symptom: tracked conversions exceed orders.

Missing conversions on a payment method or device. One flow does not reach the confirmation page — a redirect that skips it, an app that returns elsewhere, a payment provider that lands users on its own page. Symptom: a channel or device with implausibly poor conversion.

Self-referral. A payment provider or authentication step sends the user away and back, and the return visit is attributed to the payment provider rather than to the original source. This silently reassigns a large share of conversions and it is one of the most common misconfigurations there is.

Internal traffic counted. Staff, agency and QA activity in the numbers. Small sites can be substantially distorted by their own team.

Bot traffic. Both malicious and benign — monitoring services, previews, scrapers. Symptom: sessions with zero duration, implausible geography, traffic spikes with no conversions.

Test transactions in production data.

Currency and tax handled inconsistently across systems, so revenue does not match between analytics and finance for reasons that have nothing to do with tracking.

Refunds and cancellations never reflected. Analytics counts the order; the business counts the money. On a category with high returns these are very different numbers.

Timezone mismatches, so daily totals differ between systems and nobody can reconcile a specific day.

The afternoon audit

Concrete, and it finds most of the above.

1. Reconcile a month. Tracked conversions against orders, in total and by channel.

2. Check for duplicates. Look for the same transaction identifier appearing more than once. If transaction identifiers are not being sent, that is the first fix.

3. Complete a purchase yourself, on each payment method and each device, and watch what fires. This finds the missing-flow problems that no amount of dashboard staring will.

4. Check the referral exclusion list. Payment providers, authentication services, and your own domains should be on it. This one setting fixes the self-referral problem.

5. Look at the traffic sources list, sorted by sessions, and look for anything you do not recognise. Bots and misconfigured UTMs both surface here.

6. Check session duration and bounce distribution for implausible spikes at zero.

7. Compare revenue against finance, for one month, and find out where the definitions differ. They will.

8. Check your own team is excluded.

9. Look at a day in each system and confirm the totals refer to the same 24 hours.

The UTM problem

Campaign tagging is the most common source of unusable channel data, and it decays without maintenance.

Case sensitivity. Email, email and EMAIL are three sources.

Inconsistent naming. paid-social, paidsocial, social-paid across three agencies.

Auto-tagging and manual tagging fighting, so the same click is labelled two ways.

Internal links tagged with UTMs, which restarts the session and destroys the original attribution. This one is common and quietly devastating.

The fixes are unglamorous: a written taxonomy, lowercase enforced, a validation step before launch, and a periodic look at the source list for values that should not exist. See documenting metric definitions.

Monitoring rather than auditing

An audit is a snapshot. Tracking breaks after deployments, and the gap between breaking and noticing is where the damage happens.

Alert on conversion volume dropping below a threshold, per channel. The most valuable alert in the stack.

Alert on the reconciliation ratio moving, not just on volume.

Alert on a source disappearing. A channel going to zero usually means a tag broke, not that the channel stopped.

Check tracking after every site deployment. An automated test that completes a purchase in a test environment and asserts the events fired is a day of work and it catches the most expensive class of failure.

Log site and consent changes on the same timeline as campaign changes. Half of unexplained metric movements are explained by something on this list. See consent.

What to do with data you now know is imperfect

Not throw it out. Use it with its limits stated.

Use transactional data for absolute numbers. Orders, revenue, refunds. It is what actually happened.

Use analytics for relative comparison within a stable period. Which channels moved, which pages perform differently.

Do not compare across periods where the tracking changed, and know when it changed.

Publish the visibility rate. If analytics sees 78% of orders, saying so in the report is more useful than a footnote and it pre-empts the argument about why two systems disagree. See when two systems disagree.

The summary

Broken tracking produces plausible numbers, so absence of errors is not evidence of correctness.

Reconcile tracked conversions against actual orders monthly. One check, and it catches most things.

Complete a purchase yourself on every payment method, because that finds what dashboards cannot.

Alert on volume and on the reconciliation ratio, and check tracking after every deployment.

The goal is not perfect data. It is knowing precisely how imperfect it is, which is achievable in an afternoon and is worth more than any dashboard. For a formal set of data-quality dimensions, see the Government Data Quality Framework.