Integration Failure Runbook Template
Three documents and three sheets that reconcile the destination against the source to name every record a sync failure actually dropped.
Free download · No account needed
Error Register
Brindlewood Outfitters, one quarter of logs
| Error type | Signal | Backfill |
|---|---|---|
| Auth token expiry | Loud, total | 194 records |
| Bundle-SKU partial sync | Silent, recurring | Field fix, no resend |
| Rate-limit queuing | Delay, not loss | None |
The bundle-SKU row reports success on every record. It surfaced only when a warehouse picker found 227 orders missing their component SKUs, not from an alert.
Every integration troubleshooting guide ranked for this query teaches detection: checksums, correlation IDs, dead-letter queues, drift alerts. That is real advice, and this pack assumes a team already runs some version of it. None of them ship what a team needs once a gap is confirmed: the specific records that fell into it, priced and dated, ready to hand to whoever runs the backfill. Shopify's own webhook documentation is direct about the limit: after eight failed delivery attempts over four hours it stops trying, and recovering what never arrived is left to the merchant.
On the worked example already in the sheets, Brindlewood Outfitters lost warehouse sync for six hours and 56 minutes overnight when an API credential expired with no renewal alert set. Of 214 orders placed in that window, 8 self-healed on a pending retry and 12 were already re-keyed by a support agent chasing calls. Reconciling the destination against the source found the remaining 194, missing from the warehouse system: $18,624 in exposure, 21 orders overdue on a two-day shipping promise, and 9 that had oversold by the time anyone checked.
This space starts once an error is suspected, not before. Choosing and building the integration is a separate CRM implementation job. A one-time cutover between two systems fails differently and gets validated before go-live rather than reconciled after, which is what a systems migration plan covers. What is left, the backfill a live sync needs once it drops records, and the runbook that gives the next failure a known response instead of a fresh investigation, is what this pack closes.
What's in the pack
Failure Runbook
By error type rather than one generic response: loud and total, silent and recurring, or a delay that clears on its own, each gets the response that type actually needs.
Backfill Procedure
Dependency order, a resend-time state check against current inventory and pricing, and expedited orders sequenced first into the earliest batch.
Stakeholder Comms
Four separate messages for four audiences who need different information, not one status update with a name swapped in.
Error Register
Every error type your logs actually show, classified loud or silent, ranked by real dollar and labor exposure for the period.
Records Missed During Outage
The reconciliation itself: destination counted against source for the exact window, with an order value on every missing row.
Backfill Log
Every resend attempt with its own outcome, closed only once every row carries backfilled clean, resolved, or withdrawn. Retiring the source system for good afterward is a separate dependency check.
How to use it
- 1
Open in River, or take it blank
Open the pack in River and hand it your logs, or download the Word documents and CSV sheets and fill them in yourself.
- 2
Send your logs and both counts
The integration logs or error export for the window in question, plus a record count from the source system and the destination system.
- 3
Get the register and the reconciliation
Every error type classified loud or silent, and the specific records missing from the destination named, dated, and priced.
- 4
Order the backfill, keep the runbook
Dependency order, a resend-time state check, and a runbook the next failure of the same type can reuse instead of starting over.
Frequently asked questions
Is this template free?
Yes. The zip is Word documents and CSV sheets, with no account and no card needed. Edit with AI is the optional half: River reads your actual logs and error exports, builds the register and reconciliation from them, and orders the backfill as you go.
What format are the downloaded files?
Three Word documents and three CSV sheets, zipped together. Excel, Numbers and Google Sheets read the sheets with no reformatting, and Word, Pages or Google Docs open the runbook and comms directly. Nothing needs converting first.
We already have alerting, retries and a dead-letter queue. What does this add?
Detection, which is real work and this pack assumes a team already has some version of it running. What monitoring does not produce is the artifact needed once a gap is confirmed: the specific records that fell into it, priced and dated, reconciled against the destination system rather than estimated from an alert.
How do you avoid creating a duplicate order when resending?
Every backfill candidate gets a resend-time state check first: whether it is already present under a different ID, and whether price or inventory changed since the record was dropped. Stripe's own idempotent-request documentation exists for exactly this risk; a blind resend can perform the same operation twice instead of fixing it.
How is this different from a migration cutover communication plan?
Cutover comms cover a planned migration weekend, sent on a schedule set in advance, including a fallback message you hope not to send. This pack's Stakeholder Comms respond to an unplanned failure discovered after the fact, addressed to whichever specific customers the actual outage affected.
Does this tell me whether the integration itself was worth what we paid for it?
No, that comparison is a post-implementation review, run against the original business case while a renewal is still negotiable. This pack starts once a specific sync has already failed and stays focused on naming what it dropped and closing that one incident.
What if my logs don't separate loud failures from silent ones?
Send what you have. A connector's own dashboard usually shows the loud failures directly: a rejected call, a stream of 401s. The silent ones typically surface somewhere else entirely, a support ticket, a picker's exception note, so mention those too and the register gets built from both sources.
Name exactly what the outage dropped
Send your logs and a record count from both systems. The first thing back is the gap between them, not a guess that it's probably fine now.
Edit with AI