Software & TechnicalFree
Alert Fatigue Review With Actionability Rate
Every alert scored on how often it led to an action, with the ones to delete, retune or turn into a ticket named.
River's alert review reads the firing history and what happened after each page. Out comes a sheet with one row per alert: how often it fired, how many distinct incidents it actually opened, and what share of its pages led to a human action. The same row carries how many arrived overnight and how long they sat unacknowledged. Then a document naming the alerts to delete, the ones to retune, the ones that belong in a ticket queue, and the reasoning for each.
The advice for this problem is consistent and it is all advice. Consolidate your tools, tune your thresholds, use severity levels, review your alerts regularly. None of it tells you which alert to delete on Monday, because that decision needs a number. And the number lives in two places at once: the firing history in the monitoring tool, the outcome in the incident tool. Joining them is the whole analysis, and nobody does it by hand.
Built for the reliability lead who has to defend deleting a monitor, the engineering manager whose rotation is burning out, and the on-call who stopped reading pages three weeks ago. Run it quarterly, or in the week after a rotation everybody complained about. What survives the cut deserves a runbook built from its own resolution history, and the pages that did open a real incident feed the postmortem built from the alert record.
An alert is not an incident
Start with the definition, because it decides the count. Google's SRE book defines an incident as a sequence of events and alerts related to the same root cause, the kind that would be discussed in one postmortem. So a flapping monitor that pages eleven times in an hour is one incident and eleven pages. Any noise number computed before that grouping is wrong, and it is wrong in the direction that makes the loudest alert look like the biggest problem.
The same chapter publishes the budget nobody quotes. It puts the average cost of an on-call incident at six hours, counting root cause analysis, remediation and the follow-up work, and derives a ceiling of two incidents per twelve hour shift. It goes further: the distribution of paging events should be flat, with a likely median of zero per day. That turns a complaint into an arithmetic problem, and the answer is usually that the incidents are inside budget while the pages are nowhere near it.
Two of the findings are mechanical. Prometheus's alerting best practices say to page on latency at one point in a stack, so a second latency alert one layer down is noise by construction. For batch jobs it says the threshold should allow at least two full runs, and works the example: a job running every four hours and taking one hour justifies ten. A page at ninety minutes is caused by the threshold, not by the job, and tuning it is a config change.
How it works
Paste the history
Drop in the alert export, the firing history and the incident records that followed.
River groups the pages
Alerts sharing a root cause collapse into one incident, so the counts stop double-counting flaps.
Read the register
One row per alert with its actionability rate, and the verdict document behind it.
Cut the noise
Take the deletions, retune what is worth keeping, and rerun after the next rotation.
What you get
- Every alert with its fire count beside the number of distinct incidents it actually opened
- Actionability rate per alert, measured against what a human did after each page
- The overnight split, so a quiet weekday average stops hiding the 2am pattern
- Alerts that never opened an incident of their own, ranked by how loud they are
- Batch and threshold pages separated from real failures by the schedule arithmetic
- A verdict per alert: keep, retune, convert to a ticket, or delete, with the reason
Common questions
What does it need from me?
The alert definitions and their firing history, plus whatever record exists of what happened after each page. An incident export is the strongest second input. Resolution notes, acknowledgement times and the on-call channel all get read. Ninety days is enough to rank every alert; thirty is enough to find the worst offender.
How is actionability actually measured?
By what a human did, not by severity. Each firing is classed as an action that changed the outcome, an escalation, a self-resolution nobody touched, a duplicate inside an incident another alert already opened, or an acknowledgement with no follow-up. The rate is the first class over all firings, and every class is shown so you can argue with any row.
Will it tell me to delete something that catches a real outage?
No, and grouping first is why. An alert reaches the delete list only when it opened no distinct incident of its own across the whole window, and the report names which surviving alert caught each incident instead, with the lead time given up. Coverage is checked before anything is cut. Where two tools disagree about which fired first, the incident timeline reconstruction settles the order.
What about alerts that fire rarely but matter?
They score well, because the measure is the share of firings that led to an action rather than the count. An alert that fired four times and produced four actions is the healthiest row in the sheet. Volume only ever appears beside actionability, which is what keeps a rare and important alert safe.
Where do the alerts that should not page go?
Into a ticket queue or onto a dashboard, and the report says which. An alert with thirty days of notice is work rather than an interruption, and a disk warning that cleared itself twenty-nine times out of thirty is a capacity trend. Both keep their signal and stop consuming a night.
Can it tell me whether the rotation is sustainable?
It computes that. Incidents per twelve hour shift against the published ceiling of two, the median pages per day, the overnight share, and the engineering hours implied at six hours per incident. That last number is the one that moves a planning conversation, because it is denominated in the same units as feature work.
What happens after the cut?
The alerts that survive deserve documentation, and now there are few enough for that to be realistic. Feed each into the runbook built from its resolution history, then rerun this review after the next rotation to see whether the retuned thresholds actually held. The thresholds themselves come out of a budget, which is what the SLO and error budget template sets.
Alert Fatigue Review With Actionability Rate
Fill in the form and your workspace opens with the work already underway.