Our metric layer now flags a total that cannot be true

The situation

Our metric layer passes a dashboard vendor’s numbers to our analyses. Figures that add up, such as spend, it builds from daily rows. Unique users, reach and frequency cannot be summed, so those three took the vendor’s total with no second opinion.

What it looked like

The vendor answered "success" and returned a number.

What was actually true

A period’s unique count must lie between its largest part and the sum of its parts. On one client’s social-ads source, a month’s reach was below that of one week inside it. The "total" was the first week repeated, for each figure we read there, understating reach by at least a fifth, possibly four fifths. The source had lost its permissions; the vendor said so in an "unauthorised" flag inside its success answer, and nothing in our code looked at it.

How we found it

Before writing code we measured live data from two clients: of fourteen decisive readings, ten passed, four failed. Two proposed checks died. Comparing totals across granularities was empty; the vendor returns the same total at each. "Reach times frequency equals impressions" held to eight decimal places, suspiciously exact: the vendor derives frequency from the other two, so it balanced on three wrong numbers. Of eleven faults we planted in the new code, the tests caught ten; the eleventh sat in a guard that could never change an answer, so the guard went.

What changed

The fit check, with two warnings on the figure: total out of range, and total not backed by its parts. The vendor’s flag is now read, so a flagged source no longer reports as fine. An unused file was removed once an audit found nothing importing it.

Where it ended

Merged and confirmed on the main code. Reconnecting the client’s source is a separate human task, still open; a change note comes first, because it will move figures clients see.

What we did not claim

That the removed file had always been unused. We could not see its full history.

Why it matters beyond this case

A number that arrives every morning stops being questioned. ---

All our work →