Software & TechnicalFree

Failure Mode Analysis for a Service

A written list of every way your service can break, how you would find out about each one, and which failures nothing would catch.

River's failure mode review reads how your services connect and what your alerts watch, together. Out comes a register with one row per way something can fail: what fails, what it looks like, and where that problem first becomes visible to something you own. Detection is worked out rather than guessed, so every row names the alert that would go off, or how many steps away from the fault it would go off. A diagram shows the paths a failure travels.

Every published method for this ranks by a risk priority number: severity times occurrence times detection, each scored one to ten. AWS's own guidance defines detection as how hard the failure is to catch before customers are affected. In practice that score gets guessed in a workshop, because detection is the one column with no document behind it. It is also the only one of the three you can work out from files you already have: how your services connect and what your alerts watch.

Built for the reliability engineer writing a register nobody will read, the tech lead who inherited a service and wants to know how it breaks, and the on-call who spent an outage debugging the wrong service. Run it before a planned failure test, so the test has a hypothesis worth checking. Services with no monitoring at all come out as their own list, and the alerts that survive deserve a runbook built from their own history.

Detection is the column nobody can score

The reason that score gets guessed is that the failures it describes are the invisible ones. A study of 101 fail-slow hardware incidents gathered from production clusters at twelve institutions found 1 percent were pinpointed within minutes, while 41 percent took days, weeks or months. Time to detect was unknown in another 45 percent of the reports, which is its own kind of answer. Nobody sitting in a workshop scores their own system a nine.

The register makes it worse by holding only the failures that are easy to picture. A room enumerates the component being down, because down is a sentence somebody can write on a card. The same component running at a quarter of its speed never gets a row, and that is the row where the health check passes, the dashboard stays green, and the queue behind it grows for six hours. Here every outage row gets a slow version added alongside it.

Then there is where the symptom lands. A component with no rule on it fails quietly, and the first thing to fire is whatever calls it, one or two hops away. The page names the caller. Your on-call opens the caller's dashboard, finds nothing wrong with the caller, and the clock runs. That distance between the failure and the alert is most of the difference between a short incident and a long one, and it is measurable from the graph rather than arguable.

How it works

Paste the map
Paste or attach how your services connect, plus the alerts you have today, in any form.
River walks it
Each failure is traced to the nearest part of the system with an alert that would go off.
Read the register
One row per failure with whether it gets caught, by what, and how much else it takes down.
Close the blind spots
Add the rules the report names, then rerun to confirm the silent rows have moved.

What you get

One row per way the service can fail, with the alert that would actually go off first
A slow version of every outage added, because slowness passes the health checks outages fail
How far each alert lands from the real fault, so misleading alerts are named in advance
The failure modes nothing anywhere detects, kept separate from the ones detected late
Shared parts that would take several services down at once, named instead of counted separately
Services with no monitoring at all, and the few alerts that would close the worst gaps

Common questions

What does it need from me?

How your services connect, in whatever form exists, plus the alerts you have and what each one watches. A diagram, a service catalogue, infrastructure files or a plain note of which service calls which all work. Past incidents sharpen how likely each failure is, but working out detection runs without them.

How is detection derived rather than scored?

By asking what would go off. Each failure's symptom is checked against every alert you have, and the row lands in one of three states: caught by an alert on the part that failed, caught only by something that depends on it, or caught by nothing. Indirect rows say how many steps away the alert fires and which service it would name.

Does this replace the risk priority number?

It fills in the column that number depends on. Severity and occurrence stay yours, because they are judgements about your business. Detection stops being one. In an illustrative example, a company we will call Wexcombe, the two orderings shared nothing: the top five by risk priority number and the top five by severity against distance from the alert had no row in common.

Why generate a slow version of every failure mode?

Because those are the rows with the long tail. A room writes down the component being unavailable and stops there, but a component at a quarter speed passes its health check. In the Wexcombe example, twenty-one of thirty-four rows had a meaningful slow form, and two of those were detected by nothing at all. The threshold for slow comes out of your error budget.

Is this the same thing as chaos engineering?

It is the step before it. An experiment needs a hypothesis, and the register is where one comes from: take the row saying nothing would fire, break that component inside a controlled window, and watch whether anything fires. Running experiments without the register tests the failures you already handle. For the rows where the component does not come back, the recovery pack measures whether it can.

What about failures only a customer would notice?

They get their own state, and they are the point of the exercise. In the Wexcombe example, four of fifty-five rows had no detector anywhere, and two were background workers that had simply stopped. Nothing calls a background worker, so nothing notices it is gone, and the postmortem afterwards starts hours late.

We already have plenty of alerts.

Then the question is whether they sit where the failures are. In the Wexcombe example there were eleven alerts across six of twenty-three components, and three of the ten components needing work already had one. It was the wrong kind, because an error-count alert cannot fire on a dependency that only got slower. The alert review works from the other end.

Sources

Built by

Chandler Supple

Co-founder & CEO, River

Chandler has spent the last six years working in AI, building tools for millions of users and Fortune 100 companies. In 2025, he co-founded River to build a new kind of operating system for AI agents.

LinkedIn GitHub X

Reviewed by

Brian Joseff, Co-founder, River

LinkedIn X