Software & TechnicalFree
Failure Mode Analysis for a Service
River derives each failure mode's detection from your alert rules and dependency map, then ranks by how far the symptom lands from the cause.
River's failure mode review reads the architecture and the alert rules together. Out comes a register with one row per failure mode: what fails, what the symptom looks like, and where in the system that symptom first becomes visible to something you own. The detection column is derived rather than scored, so every row names either the component whose rule fires or the number of hops between the failure and the first thing that notices it. A diagram marks the propagation paths behind those hops.
Every published method for this ranks by a risk priority number: severity times occurrence times detection, each scored one to ten. AWS's own guidance defines detection as how hard the failure is to catch before customers are affected, and tells you to validate that score through controlled experiments or historical analysis. Almost nobody does, because detection is the one column with no artifact behind it. It is also the only one of the three that is computable from files you already have.
Built for the reliability engineer writing a register nobody will read, the tech lead who inherited a service and wants to know how it breaks, and the on-call who spent an outage debugging the wrong service. Run it before a chaos experiment, so the experiment has a hypothesis worth testing. What it finds about missing rules feeds the coverage review, and the rules that survive deserve a runbook built from their own history.
Detection is the column nobody can score
The reason that score gets guessed is that the failures it describes are the invisible ones. A study of 101 fail-slow hardware incidents gathered from production clusters at twelve institutions found 1 percent were pinpointed within minutes, while 41 percent took days, weeks or months. Time to detect was unknown in another 45 percent of the reports, which is its own kind of answer. Nobody sitting in a workshop scores their own system a nine.
The register makes it worse by holding only the failures that are easy to picture. A room enumerates the component being down, because down is a sentence somebody can write on a card. The same component running at a quarter of its speed never gets a row, and that is the row where the health check passes, the dashboard stays green, and the queue behind it grows for six hours. Here every fail-stop mode gets its degraded twin generated alongside it.
Then there is where the symptom lands. A component with no rule on it fails quietly, and the first thing to fire is whatever calls it, one or two hops away. The page names the caller. Your on-call opens the caller's dashboard, finds nothing wrong with the caller, and the clock runs. That distance between the failure and the alert is most of the difference between a short incident and a long one, and it is measurable from the graph rather than arguable.
How it works
Paste the map
Drop in the architecture, the dependency edges and every alert rule you currently have.
River walks it
Each failure mode is traced to the nearest component whose rule can fire on that symptom.
Read the register
One row per mode with its detection state, its hop distance and its blast radius.
Close the blind spots
Add the rules the report names, then rerun to confirm the silent rows have moved.
What you get
- One row per failure mode with the component whose alert would actually fire first
- A degraded twin generated for every fail-stop mode that has a meaningful slow form
- Diagnostic distance in hops, so you can see which pages will name the wrong service
- The failure modes nothing anywhere detects, kept separate from the ones detected late
- Blast radius per shared component, so correlated rows stop counting as independent risks
- The rules that would close the register's severe gaps, with the count before and after
Common questions
What does it need from me?
The architecture in whatever form exists, plus every alert rule you have and what each one is attached to. Terraform, Kubernetes manifests, a service catalog or a plain list of which service calls which all work. Past incidents sharpen the occurrence column, but the detection work runs without them.
How is detection derived rather than scored?
By asking what would fire. Each failure mode's symptom is tested against every rule you have, and the row lands in one of three states: caught by a rule on the component that failed, caught only by something downstream, or caught by nothing. Downstream rows carry the hop count and name the component that would page you instead.
Does this replace the risk priority number?
It fills in the column that number depends on. Severity and occurrence stay yours, because they are judgements about your business. Detection stops being one. At Wexcombe the two orderings shared nothing: the top five by risk priority number and the top five by severity against diagnostic distance had no row in common.
Why generate a slow version of every failure mode?
Because those are the rows with the long tail. A room writes down the component being unavailable and stops there, but a component at a quarter speed passes its health check. Twenty-one of Wexcombe's thirty-four rows had a meaningful slow form, and two of those were detected by nothing at all. The threshold for slow comes out of your error budget.
Is this the same thing as chaos engineering?
It is the step before it. An experiment needs a hypothesis, and the register is where one comes from: take the row saying nothing would fire, break that component inside a controlled window, and watch whether anything fires. Running experiments without the register tests the failures you already handle. For the rows where the component does not come back, the recovery pack measures whether it can.
What about failures only a customer would notice?
They get their own state, and they are the point of the exercise. Four of Wexcombe's fifty-five rows had no detector anywhere, and two were message consumers that had simply stopped. Nothing calls a consumer, so no caller notices it is gone, and the timeline afterwards starts hours late.
We already have plenty of alerts.
Then the question is whether they sit where the failures are. Wexcombe had eleven rules across six of twenty-three components, and three of the ten components needing work already had one. It was the wrong kind, because a rate rule cannot fire on a dependency that only got slower. The alert review works from the other end.
Failure Mode Analysis for a Service
Fill in the form and your workspace opens with the work already underway.