Product Metrics Definition Document
A metric is defined when you can name the events that compute it, the filters that run, and the platforms those events actually reach.
Free download · No account needed
Two dashboards publish Weekly Active Teams for the same week and return 4,812 and 5,376. The usual response is a meeting about what active means, and it never works, because both teams already agree on the sentence. What they disagree on is which events count. This pack settles that the only way it can be settled, by writing every metric as an expression in event names rather than as a description, and then going and checking what those events actually do.
Every row in the register carries two columns a metrics document never has. Coverage is the share of your users on which every event a metric needs actually fires, so a metric whose event set skips Android is not undercounting Android, it cannot see it. Comparable From is the date the last of those events landed everywhere inside that coverage set. Before that date the line on the chart contains a rollout, and a rollout looks exactly like growth.
Send the dashboards, the weekly report and the board pack, plus whatever describes your instrumentation and roughly how your sessions split across platforms. Back come a register, an event-to-platform map and a log of every pair of surfaces that disagree, each gap decomposed to the one clause causing it. The pack assumes your events exist and mostly work. If you are not sure they do, audit the events first, because this reads instrumentation rather than repairing it.
What is in the pack
Metric Register
One row per published metric: the aggregation, the event expression, the filter clauses, the platforms every required event fires on, coverage, and the date it became comparable. The expression is written in event names, so it can be checked and it can be wrong.
Instrumentation Coverage
One row per event, not per metric, with first-fired dates by platform and the metrics that depend on it. Even a vendor's own built-in events are annotated by platform, which is why this has to be a real map rather than an assumption.
Known Discrepancy Log
Every pair of surfaces publishing different numbers for the same metric, with the gap decomposed to the clause causing it and the arithmetic closing. Closed rows stay, because a closed row is the reason a metric agrees with itself today.
Definition Standard
The shape a definition has to take before it counts as one: expression, filters, window, population, coverage and comparable-from date. Enough that a second team implementing it independently gets the same number, which is the only test that matters.
Change Policy
What happens the next time an event is renamed, a filter is added or a platform ships late: who is told, which series get restated, and how a comparable-from date moves. Written before the next change rather than argued about after it.
How a Metric Is Traced to Its Events
The method, plus ten prompts that run it: read every surface publishing a number, trace, map events to platforms, compute coverage, recompute the metrics that appear twice, and answer the which number is right question when it arrives.
How it works
- 1
Send every surface that publishes a number
Dashboard exports, the weekly report, the board pack, alerting thresholds, lifecycle audiences. The metric list comes off the surfaces people actually read, not out of the document that claims to define them, because those two lists are never the same.
- 2
Each metric becomes an expression
River rewrites every metric as an event expression with its filter clauses, window and population made explicit. Where a metric cannot be traced, it is recorded as untraced rather than guessed at, because a guessed expression is a wrong answer wearing a right one's clothes.
- 3
Events get mapped to platforms and dated
Every required event is mapped to the clients it fires on and the date it started. Coverage is the intersection expressed against your session split; comparable-from is the last of those dates. Both land in the register beside the metric they qualify.
- 4
The disagreements get decomposed
Any metric appearing on two surfaces is recomputed both ways and the difference is split into named parts that add up. You get the standard and the change policy alongside, so the same argument does not restart the next time an event moves.
Frequently asked questions
How is this different from a KPI dictionary?
A dictionary holds a sentence per metric: name, owner, plain-language definition, formula, source, refresh cadence. Two teams can implement the same agreed sentence faithfully and still publish different numbers, because the sentence never named the events. This register holds expressions in event names, and adds the two columns that decide whether an expression is even computable.
What does Coverage actually measure?
The share of the population a metric claims to describe on which every one of its required events fires. Any missing event removes the whole platform, because a team there cannot satisfy the expression no matter what they do. It is measured against the claimed population, so a deliberately mobile-only metric sits at 100%.
Why date a metric from when it became comparable?
Because instrumentation ships gradually and charts do not. Apple releases an update to 1% of users on day one and 100% on day seven, so an event added in a mobile release lands over a week rather than at once. Before the last platform is in, the series mixes behaviour with rollout, and a rollout looks like growth.
Our metrics doc already lists formulas. Is that enough?
Check whether the formulas name events. A formula written as activated divided by signups is a shape, not a definition, and Amberline's two dashboards both implemented it correctly to get 38.4% and 44.1%. The numerator was identical at 3,164 accounts. The 1,064 unverified signups in one denominator were the entire gap.
Will this fix our broken events?
No, and it should not pretend to. This reads instrumentation and tells you what your current numbers can and cannot support. Fixing the events themselves is a separate pass over volumes, property completeness and duplicates, which is what the instrumentation audit does.
What if a metric turns out to be uncomputable?
It gets recorded as untraced with the specific event that would fix it, rather than filled in with something plausible. In the worked example one event, shipping on two mobile clients, would have taken six partial metrics to full coverage. That is a roadmap item with a number attached, not a caveat.
Where does this sit next to our OKRs and experiments?
Upstream of both. An OKR written on a metric with 63.7% coverage is measuring two thirds of the company, and an experiment readout on a metric that changed definition mid-flight is reading a rollout. Trace first, then set targets against something that holds still.
Find out how many of your metrics trace clean
Send the dashboards and whatever describes your instrumentation. The first thing back is the count, and the metric with the worst coverage.
Trace my metrics