Software & TechnicalFree
Monitoring Coverage Gap Analysis Report
River counts the services with a monitor that can fire when the service stops emitting, then checks that against how your incidents were actually found.
River reads the alert rules, the service inventory and the record of how your last incidents were actually found. The first number back is not a coverage percentage. It is the count of services with no monitor that can fire when the service stops emitting, which is a different and much smaller set. Then the grid, one row per service and one column per failure mode. Then the services that would fail undetected, ranked by how long that class of failure runs before a human notices.
Every coverage report counts the same thing, which is how many services have a monitor attached. A dashboard can compute that from configuration alone, which is why every tool ships it. It is also a self-graded score, and it says nothing about whether those monitors would fire. Sandbourne Retail, a 240-store grocer, scored 94% on that measure across 84 services. Counted per failure mode the same estate scored 62%, and against the incident record, where an alert caught 16 of 41 incidents, 39%.
Built for the reliability lead who has just been asked why nobody was paged, and for the platform engineer who inherited an estate they did not instrument. Run it in the week after an incident a monitor should have caught. The alert noise review is the other half of the same problem, and the inventory this needs is the one the cloud bill analysis already reads. Where a service has no owner at all, that absence is a finding with a name against fixing it.
The rule that cannot fire
Take an ordinary error-rate rule, something over a threshold for five minutes. It compares a number against a line, so it needs the number. Prometheus documents what happens when the number stops: a series that is no longer exported is marked stale, and once it is stale no value is returned for it. The comparison has nothing on its left-hand side, the rule produces no result, and the alert stays resolved. The service is gone and the monitor watching it reads green.
Managed monitoring does the same thing by default rather than by accident. AWS lists four ways an alarm can treat missing data and states that the default behavior is missing. Under it, an alarm whose data points have all gone transitions to INSUFFICIENT_DATA rather than to ALARM. Check what is subscribed to that state before counting the alarm as cover. Both stacks ship an antidote and both make you ask: Prometheus has absent_over_time, for alerting when no series exists.
The received failure-mode list does not rescue it either. Google's SRE book puts the four golden signals at latency, traffic, errors and saturation, and each one is a measurement a running service emits. A grid built on those four grades a service on how well it reports while it is up. Sandbourne's price feed scored well on it. Its one rule counted rejected rows, so the night the job never started there were no rows, then no series, and 240 stores ran yesterday's prices for 61 hours.
How it works
Send the rules
The alert rules, alarms and routing config, in whatever form your monitoring stack exports them.
Add the inventory
Every running service with its tier and owner, marking which of them are jobs or queue workers.
Name the detections
How your recent incidents were really found, whether an alert paged somebody or a customer wrote in.
Read the ranking
The services that would fail with nothing firing, worst first, each with the monitor to add.
What you get
- Every rule sorted into those that fire on a bad number and those that fire on none
- Coverage per failure mode, so a service graded green on one column stops reading green
- The services with no monitor that can fire when the service stops emitting at all
- Your configured coverage set against how many real incidents an alert actually caught
- Each gap ranked by how long that class of failure runs before a human notices it
- The specific monitor to add per gap, written against the stack you already run
Common questions
Our dashboard already says we are at 90-something percent.
That number counts services with a monitor attached, which is a fact about your configuration. It cannot tell you whether the monitor fires when the service stops emitting. Sandbourne's estate scored 94% on that measure and 39% against its own incident record. Both are correct, and only the second one predicts the next outage.
We don't record how incidents were detected.
Then the run does the configuration half and says so. You still get the breach and absence split, the failure-mode grid and the ranked gaps. The detection reconciliation is the part that needs history, so it comes back marked unavailable rather than estimated. Ten incidents recalled from memory, with who noticed, is enough to start.
Is this the same as an alert fatigue review?
Opposite direction. A noise review scores alerts that fire too often and names the ones to delete. This one finds the failures nothing fires for. Most teams have both at once, because the tuning pass that quietens a rotation tends to remove the only monitor on a service quiet enough that nobody defended it.
We run Datadog, not Prometheus or CloudWatch.
The mechanism travels, because it falls out of how threshold evaluation works. A monitor comparing a metric to a value needs the metric, so every platform carries a separate setting for what to do when data stops arriving. What the run checks is which of your monitors have that setting chosen deliberately rather than left at whatever the default is.
How does it rank the gaps?
Tier weight multiplied by how long that class of failure actually took to reach a person, taken from your own incident record rather than from a rule of thumb. At Sandbourne a P1 hourly import outranked two P0 services, because its class ran four days before anyone noticed while theirs ran 31 hours and six.
Does it tell us what the thresholds should be?
No, and that is a separate job. A threshold is derived from what you promised, so it belongs with the SLO and error budget pack. This run answers the question underneath it, which is whether anything fires at all. A perfectly tuned threshold on a series that stopped arriving still fires nothing. Whether it holds at peak is a load test.
What do we do with the output?
The sheet goes to whoever owns each service, one row per gap with the monitor to add. The document goes to the reliability review. After the next incident the postmortem can point its detection line at a named row, and the runbook picks up the new alert.
Monitoring Coverage Gap Analysis Report
Fill in the form and your workspace opens with the work already underway.