Software & TechnicalFree
Incident Runbook Template From History
Every triage step ordered by how often it actually ended this alert, with the checks that wasted time last quarter marked as such.
River's runbook tool reads the alert definition and every record of what happened when it fired. The monitor expression, the past incidents, the resolution notes and the channel threads go in. Out comes the runbook: what the alert means, the first check, the checks in order after it, and the named escalation. Each triage step carries how many of the last firings it actually resolved, so the on-call reads them in the order the evidence puts them rather than the order someone imagined.
Search the query and you get a blank runbook. The good ones are well structured and every field is still yours to fill: what the alert means, how to confirm it, what to try first. Filling that in from how the system is supposed to work produces a document that describes intent. The on-call needs the opposite, which is what has actually ended this alert before, and that answer is scattered across a quarter of incident records nobody reads together.
Built for the on-call engineer who inherits the pager, the reliability lead who owns coverage, and the team that keeps getting woken by the same monitor. Reach for it after the second or third firing, once there is a record to read. The incident postmortem built from the alert record is what you write after the bad night; this is what stops the next one. For a single fresh incident with no history behind it, the production incident runbook is the faster path.
A runbook decays at the rate production changes
Google's own guidance is blunt about the maintenance problem. The SRE Workbook's on-call chapter states that details in playbooks go out of date at the same rate as the production environment changes, so a team that releases daily is looking at playbook edits on any given day. A runbook written once from intent is wrong within a release or two. One assembled from the resolution record gets reassembled from the newer record, which is a maintenance problem you can actually win.
The same chapter calls the shape of a playbook a contentious topic. Some Google SREs keep entries general so they change slowly, with one entry covering every RPC errors high alert. Others write step-by-step procedures to reduce human variability. The chapter settles it from the other end: where a playbook is a deterministic list of commands the on-call runs every time, it recommends implementing automation instead. Resolution history is what tells you which of your alerts is in that state. The alert quality review settles the prior question of whether it should page.
There is a sharper reason to record what was wrong last time. The SRE book's chapter on being on-call gives the example directly. The same alert pages for the fourth time in a week, and the previous three were caused by an external system. Confirmation bias then makes it extremely tempting to attribute the fourth firing to the same cause, and a runbook that describes what the system is supposed to do offers nothing against that. One that records the firing where the attribution was wrong, and the check that settled it, does.
How it works
Paste the alert
Drop in the monitor definition, the past incidents it created and the resolution notes.
River reads the history
Every firing matched to what ended it, including the ones resolved by nothing at all.
Read the runbook
The document, the symptom-to-action decision table, and the steps worth automating instead of documenting.
Correct it once
Tell it where the record is misleading, and the ordering and the checks rebuild around that.
What you get
- The alert explained in one paragraph: what fires it, what a user feels, what it is not
- A first check chosen because it separated the last firings from each other, not by convention
- Triage steps in order, each carrying how many recent firings it actually resolved
- A decision table mapping symptom to action, so the 3am read is one row
- The steps identical every single time, flagged for automation rather than left in prose
- The escalation point with the condition that triggers it, read off past handoffs
Common questions
What does it need from me?
The alert definition and whatever record exists of it firing. An incident export with resolution notes is the strongest input. The channel threads are usually more useful, because the reasoning lives there and the notes say resolved. Two firings are enough to start; five give you an ordering you can defend. Where the records themselves disagree about times, the incident timeline reconstruction settles that first.
What if the history is thin, or the alert is new?
Then you get a shorter runbook that says so. Steps with no firing behind them are marked unverified rather than presented as tested, and the document names what the next firing should record to close the gap. That is a runbook you can trust the shape of, which a full page of invented procedure is not. The failures you have not had yet are a failure mode review instead.
Does it cover the actions that made things worse?
Yes, in their own section, with the firing they came from. A restart that re-delivered in-flight messages and a scale-up that exhausted the connection pool are the two most useful lines in any runbook, and they only exist in the record. They are written as what happened rather than who did it, on the same blameless incident report discipline.
Is a runbook the same thing as a playbook?
Same artifact, different house style. Google's SRE material calls it a playbook and defines it as high-level instructions for responding to an automated alert, covering severity, impact, debugging suggestions and the actions that mitigate it. Most teams say runbook. This produces that document, keyed to one alert rather than to a whole service.
How does it stay current?
You rerun it after the next few firings and the ordering moves. A step that resolved three of the last five drops down when the newer record stops confirming it. Because the document is derived rather than hand-written, an update costs one paste instead of an afternoon of remembering what changed.
Where does this sit next to a postmortem?
A postmortem explains one incident to the people who were not there. A runbook tells the next person what to do at 3am. They read overlapping material, so if the outage is already written up, feed that in: the postmortem built from the alert record produces exactly the timeline this wants.
Can it tell me which alerts to automate away?
It flags them. Where the last five firings all ran the same commands in the same order, the runbook says so, because Google's own guidance is to implement automation rather than document a deterministic procedure. The output names the sequence, the guard you need in front of it, and the one firing that would have broken it.
Incident Runbook Template From History
Fill in the form and your workspace opens with the work already underway.