River
Y CombinatorBacked by Y Combinator
FREE TEMPLATE

Monthly Reliability Review and Toil Estimate

Six documents and four sheets that price your operational load in engineer-days and project it to the quarter it breaches the engineering floor.

Free download  ·  No account needed

A monthly reliability review reports uptime, incident count and mean time to resolution. All three describe the month accurately and change nothing, because none of them is denominated in anything a planning conversation can spend. The review gets read, the numbers get nodded at, and the roadmap that was set before the meeting is the roadmap after it. Nothing in that sequence is anybody's fault. The review simply had no currency to bid with, so it lost every argument it was in before it started.

This pack produces one. Toil Estimate holds occurrences and minutes in separate columns and derives engineer-days, which is the unit a quarter's roadmap, a hiring request and a proposed automation are all already priced in. Google's SRE book defines toil as work that is manual, repetitive, automatable and scales linearly as a service grows, and publishes its own measured average of about a third of engineering time. So the level, on its own, is reassuring.

The trajectory is not, and it is already in that definition. Toil Projection tags each line with the thing that drives it and applies your measured growth rather than the plan. Headcount is held flat, because this is the do-nothing case rather than a forecast. In the worked example a six-engineer team sits at 29 percent today and 53.2 percent four quarters out, before on-call is counted against the remaining half. That date is the finding, and it is the version of the argument that gets work scheduled before the floor is breached.

The level that reads fine, and the trajectory that does not

Every figure in Toil Projection recomputes from Toil Estimate and the measured growth rates. Nothing on it was typed in by a person.

Toil Estimate

Illustrative rows for a fictional B2B scheduling platform, Bellwater. Six engineers, one quarter.

ActivityOccurrencesMins eachEng-daysScales withBuildPayback
Alert response and triage5123436.3Trafficnot automatablen/a
QA environment rebuilds3813510.7Fixed141.31
Customer data-fix requests786210.1Customersproduct changen/a
Manual capacity resizing441059.6Traffic60.62
Failed nightly export reruns96459.0Traffic121.33
Access provisioning and removal104388.2Headcount80.97
Report requests answered by hand84457.9Customers253.17
Release runbook execution44857.8Fixed182.31
Stuck queue drains58526.3Traffic91.43
Certificate and secret rotation26703.8Fixed51.32

Total 109.7 engineer-days against a capacity of 378, so 29.0%, which is 1.74 of six engineers doing nothing but running the service. That figure sits inside the range published for organisations that take this seriously, which is exactly why it cannot be the finding.

Toil Projection

Traffic +23% and customer count +14% per quarter, both measured from the last four quarters. Headcount held flat at six.

QuarterTraffic linkedCustomer linkedFixedTotalShareWith four automations
26Q161.218.030.5109.729.0%20.5%
26Q275.320.530.5126.333.4%24.3%
26Q392.623.430.5146.538.8%28.9%
26Q4113.926.730.5171.145.3%34.5%
27Q1140.130.430.5201.053.2%41.4%
27Q2172.334.730.5237.562.8%49.7%

Four quarters of the same growth with no automation and no hiring puts operational load past half the team, and it gets there before on-call, meetings or any other non-project work is counted. The four shortest-payback automations recover 32.3 engineer-days a quarter for 33 days of build, a payback of 1.02 quarters, which buys five quarters rather than solving the problem. That is the honest way to write the ask.

Incident Register

26 incidents, 1,741 minutes of customer impact, counted twice.

Cause categoryIncidentsShareImpact minutesShareMinutes each
Dependency726.9%70240.3%100
Capacity519.2%32818.8%66
Data415.4%37621.6%94
Code defect415.4%1337.6%33
Config change311.5%1176.7%39
Infrastructure27.7%613.5%30
Human process13.8%241.4%24

Code defects and data problems are both 15.4% of incidents. One is 7.6% of the damage and the other 21.6%, so a count-based priority list puts them in the same place and a minutes-based one does not. Below the category: four of the seven dependency incidents are the same partner calendar API returning stale tokens, at 146, 118, 91 and 214 minutes. That is 15.4% of incidents and 32.7% of all customer impact, from a mechanism that had four competent closed postmortems.

Detected byIncidentsImpact minutesMinutes each
Alert201,00550
Customer6736123

All six customer-detected incidents were in reporting, calendar sync or a data correctness problem, which is a named list of three detection gaps rather than an observation that monitoring could be better.

What you get

01

Toil Estimate

One row per recurring operational activity, with occurrences from a system and minutes from timed instances kept in separate visible columns. Engineer-days is derived rather than entered, so a disagreement lands on one cell instead of on the whole finding.

02

Toil Projection

Projects each line by its own driver at your measured growth rate with headcount held flat, and reports two totals side by side: the do-nothing case, and the same case with the recommended automations shipped.

03

Incident Register

Every incident counted twice, by number and by customer-impact minutes, with both share columns beside each other. The second denominator is what finds the category that is cheap and frequent and the one that is expensive and equally frequent.

04

SLO Attainment

Three months per row, so direction is readable without computing it. Sorted by direction rather than by pass or fail, because a passing objective that has declined for three months is where intervention is still cheap.

05

Investment Recommendation

One page, ranked by build days divided by days recovered per quarter rather than by size, with the three ways to decline it written out. Priced in the same unit as the feature work it competes with.

06

Trend Analysis

The four cross-cuts a per-incident review structurally cannot run: count against impact, repeat failure modes below the cause category, detection source, and objectives that are passing while declining.

07

Review Procedure

An hour, with the agenda in minutes and the assembly done a day ahead. Includes who has to be in the room for the recommendation to have somewhere to go, and what not to spend time on.

08

How Toil Is Costed

The reference behind the arithmetic: what counts, what people wrongly include, the four decisions, and why the projection is presented as the do-nothing case rather than as a forecast.

How it works

  1. 1

    Send the incidents

    A ticket export of what closed last quarter. River builds the register, categorises by cause, and runs the count-against-impact comparison before it looks at a single dashboard.

  2. 2

    Add the operational counts

    Pages taken, tickets by label, failed job reruns, access requests, manual deploys. These live in four systems and any two are enough to start. Occurrences come from a system, never from memory.

  3. 3

    Get the trajectory

    River tags each line with what drives it, applies your measured growth from the last four quarters rather than the plan, holds headcount flat, and names the quarter operational load crosses the engineering floor.

  4. 4

    Take the one-page ask

    Build days against days recovered per quarter, a payback figure, and every line automation will not fix named with the reason. Plus the cheaper subset, in case the answer is no.

Frequently asked questions

Why engineer-days instead of a percentage?

Because a percentage is a comparison and engineer-days is a currency. A quarter's roadmap, a hiring request and a proposed automation are all already priced in engineer-days, so putting operational load in the same unit lets the four be compared in one sentence instead of four meetings.

Our toil is about 30 percent. Is that bad?

No, and that is the trap. Google's own quarterly surveys put its SREs at about a third against a stated 50 percent ceiling, so 30 percent is unremarkable. The projection is what matters: 30 percent with 23 percent quarterly traffic growth and flat headcount is past the floor in four quarters.

Does this run monthly or quarterly?

The review runs monthly. The toil estimate refreshes quarterly, because a single month of operational load is too noisy to project from and a monthly re-estimate turns a useful number into a chore nobody does. The register and attainment sheet are monthly.

What if the biggest toil line cannot be automated?

Then the recommendation says so explicitly, which is the point. Alert response is a third of the total in the worked example and automating a response to an alert that should not have fired is the wrong work. That belongs to an alert quality review instead.

How is this different from doing postmortems?

A postmortem explains one incident. This finds the mechanism that produced four of them, each with a competent closed postmortem, and prices it. Tracking whether those postmortems' actions were ever finished is its own job, handled by the action follow-through tool.

Do we need SLOs before this works?

No. The incident register and the toil estimate work with no objectives at all. Attainment is the one sheet that needs them, and setting a target you can actually enforce is a separate exercise in the SLO and error budget pack.

What if our runbook coverage is the real problem?

It often is, and it shows up here as an inflated alert response line. Computing which alerts have no current documented response, weighted by the pages that landed at night, is what the on-call handbook pack does. Run that first if your nights are the complaint.

Find the quarter your team stops doing engineering

Send an incident export and your operational counts. River prices the load in engineer-days, projects it at your measured growth, and ranks the automation queue by payback.

Build my review