On-Call Handbook and Runbook Coverage
Six documents and four sheets that compute runbook coverage from your alert definitions and rank the writing queue by night pages.
Free download · No account needed
Every on-call handbook template hands you the same four headings: severity levels, an escalation ladder, a handoff format, and a page about staying calm. The headings were never the problem. What went wrong on the last bad night was that an alert fired, the responder opened the runbook link, and found a document written for a threshold that had moved in April. Nobody knew, because runbook coverage is self-reported, and self-reporting is always wrong in the same direction.
This pack computes coverage instead of asking for it. Prometheus documents the annotations clause on an alerting rule as the place to keep alert descriptions or runbook links, so the join between an alert and its documentation already lives in version control. Runbook Coverage by Alert reads that field for every rule, then adds two dates: when the expression last changed, and when somebody last followed the runbook. Their order splits linked runbooks into current and stale.
Then it changes the denominator. Coverage by rule flatters you, because the undocumented rules barely fire. Coverage by night page is the number that predicts a bad night, and the two routinely disagree by twenty points. Because every finding costs somebody time, Rotation Feasibility checks whether the rotation has any. Two people on-call at once on week-long shifts puts each engineer in a team of six on a third of the calendar, which is a capacity finding rather than a documentation one.
What you get
Runbook Coverage by Alert
One row per alert rule with pages, night pages, the runbook link read from the rule's own annotations, and the two dates that decide whether it counts. Three states rather than two: covered, stale, uncovered. Stale is the class no alert index tracks and the one that reads as coverage.
Service and Owner Register
One row per service with its tier, whether a customer can see it fail, the rotation covering it, and a count of its rules that are uncovered or stale. A tier 1 service with alert rules and no rotation at all is the row this sheet exists to surface.
Rotation Feasibility
Models the share of the calendar each engineer carries at several rotation sizes against the operational load ceiling, so a proposal to grow the rotation goes to a manager as arithmetic rather than as a complaint.
Handoff Log
Six fields per shift, and two of them are the ones a status report never has: what the outgoing engineer deliberately left alone, and the runbook edits owed from pages taken. That second field is how the coverage sheet stays current without anybody scheduling maintenance.
On-call Guide
Written for the engineer who did not build the system. What the shift is and is not responsible for, what authority a responder has without asking, the first fifteen minutes, and when to stop investigating and escalate.
Severity Definitions
One page, defined by customer-visible symptom rather than by cause, because a responder can observe a symptom immediately and cannot establish a cause for twenty minutes. Each level names who gets paged, the communication obligation, and who is allowed to declare it.
Escalation Policy
The ladder, what each rung is actually for, and the parallel tracks for communications and security that do not wait their turn. Includes the third-party path, which is the one nobody can find at 03:00.
How Coverage Is Counted
The reference behind the arithmetic: the three denominators, the three states, and why an undated runbook counts with the gap rather than with the coverage. Send it to anybody who disputes their number.
How it works
- 1
Send the alert rules
The rule definitions themselves, in any format, rather than the wiki page that indexes them. River reads the runbook annotation off each rule and builds one row per rule. A wiki index records what somebody meant to document; the rule records what a responder is actually handed.
- 2
Add a paging export
A quarter of notification history with timestamps. This is what turns a rule count into a night-page count, and the night-page figure is the whole point. Without timestamps River says so rather than reporting the flattering number alone.
- 3
Get the two dates joined
River dates each linked runbook against the last change to its alert expression, reading version control on the rule block rather than the file so a formatting commit does not read as a threshold move. Every stale row gets a sentence naming what moved.
- 4
Take the short queue and the rotation number
Four or five named documents with the coverage they buy, the rules that should be automated or deleted instead of documented, and the share of the calendar your rotation currently costs each engineer.
Frequently asked questions
Why count night pages instead of alerts?
Because they measure different costs. A page at 15:00 with no runbook costs twenty minutes of asking in a channel. The same page at 03:00 is the incident. Coverage by rule is flattering because undocumented rules barely fire, and the two figures routinely disagree by twenty points.
What makes a linked runbook stale rather than covered?
The runbook was last verified before its alert expression last changed. A threshold moved, a duration changed, a broker count grew, and the document did not follow. That is worse than no runbook, because a missing runbook tells the responder they are on their own and a stale one does not.
Does this write the runbooks?
This space decides which ones are worth writing and in what order. Generating the content from how the alert was actually resolved is a separate job, and the runbook from incident history tool does that one. Coverage first, because the list of twenty-one is what stops anybody starting.
What if an alert should not exist at all?
Then deleting it improves coverage arithmetically and improves the nights in fact, and the register marks it rather than queueing a document. Scoring every alert on how often it led to an action is its own exercise, handled by the alert quality review.
Where does the 25 percent ceiling come from?
Google's SRE book, which states that of the time left after engineering work, no more than 25% can be spent on-call. The same chapter derives eight as the single-site minimum for a primary and secondary rotation on week-long shifts, which is where a six-person team's arithmetic falls over.
How does this relate to our SLOs?
Targets and coverage answer different questions. What number to promise and how to enforce it belongs to the SLO and error budget pack; what a responder is handed when the budget starts burning belongs here. What the whole rotation costs in engineer-days is the monthly reliability review.
We have no runbook dates. Is the sheet useless?
No, but it reports unknown rather than guessing, and unknown counts with the gap. An undated document is not evidence of currency. In practice teams find the edit date in version control and label the column honestly as an edit date instead of quietly treating the two as equivalent.
Find out which pages land on nothing
Send your alert rule definitions and a quarter of paging history. River reports coverage three ways, names the stale runbooks, and ranks the queue by night pages.
Build my handbook