Data Pipeline Documentation Template
A contract that records every threshold your project declares beside what actually happens when it is violated, plus a blueprint per pipeline.
Free download · No account needed
Pipeline Blueprint
One document per pipeline that produces a number somebody reads
What it reads, including the inputs nothing declares
Every input at the grain it arrives in. The seed file nobody has edited in nineteen months, the mapping table somebody maintains by hand, the list of plan codes hardcoded inside the transformation. None of those appear in a generated lineage graph, and they are where the numbers actually go wrong.
The transformations that are not obvious
Not a restatement of the SQL. Which rows are excluded and why, how a record arriving three days late is attributed, which timezone a day boundary uses, what the deduplication keys on, and every hardcoded value listed individually.
What is not checked at all
The explicit list: volume, freshness on the output rather than the input, referential integrity across a join, plausibility of a total. Naming a gap is documentation. Leaving it out is a promise nobody made, and a blueprint that says the pipeline is tested is worse than one that says nothing, because it stops the reader looking.
How to tell whether today's number is right
The numbered procedure, cheapest check first, naming which passing checks mean nothing for this pipeline specifically. Did the run happen at all, did the output rebuild or only the input arrive, is the volume plausible for this weekday, do the parts add up to the whole. This is the section people actually open.
Who to tell when it is wrong
The consumer list, with the undeclared consumers marked. Whoever reads the number from a scheduled export or a workbook connection will never see an incident banner, and they are usually the ones presenting it.
An on-call analyst never asks how a pipeline works. They ask whether the number is right, and every indicator being green answers less than it looks. A data test's severity defaults to error, and setting it to warn makes the tool skip the error condition entirely. The test can then return ten thousand null revenue rows, print a warning, and let every model downstream build on top of them. The threshold is declared. Nothing is enforced.
Three more mechanisms do the same thing. An expectation takes a tolerance defaulting to one, and its own worked example sets it to 0.66, so a column documented as not null passes while a third of it is empty. Freshness is declared on sources, so a stale mart on a fresh source is green everywhere. And task-level service level agreements were removed outright in Airflow 3, leaving the parameter sitting in the code raising nothing.
So the data contract carries two threshold columns. One transcribes what the project literally says, parameters and severity intact; the other records what happens on violation, and a third names the mechanism where they differ. Filter it to anything but agrees and you have the numbers nobody can stand behind. It is the same shape as the register that reconciles scanner severities, and where the transformations themselves are undocumented, reading them out of the repository comes first. Where the pipeline is being replaced rather than documented, the warehouse migration pack covers that comparison instead.
What's in the pack
Data Contract
One row per check, carrying what the project declares beside what happens on violation, and a reconciliation column naming the mechanism where those two disagree.
Dependency and Consumer Register
Who reads each number, filled from query history and from asking people rather than from lineage alone, marking which consumers can never see an incident notice.
Failure Runbook Index
Keyed on the symptom rather than the pipeline, because nobody on call is handed the name of a failing model. They are handed a screenshot and a question.
Reading a Declared Threshold
The four mechanisms that let a threshold exist and enforce nothing, each read off the tool that causes it, and the fixed vocabulary the contract uses instead.
Pipeline Blueprint
The template, annotated inline, including the transformations that are not obvious, the explicit list of what is not checked, and the procedure for judging today's number.
Ownership and Response Times
The eight questions a team answers once: who owns what, when the clock starts, and what a consumer who cannot see an incident channel is entitled to.
Error Handling Guide
Triage in order of cost, separating the loud failure from the silent one, because everything is green is a different incident from everything is red.
Worked Pipeline subscription_revenue
One pipeline end to end: fourteen checks, four of them enforcing nothing, and the two-day stale dashboard its contract predicted three weeks in advance.
How to use it
- 1
Open in River, or take it blank
Claim the pack in River and give it the project, or take the blank documents and registers and fill them in by hand.
- 2
Answer the ownership questions
Who owns what, when the response clock starts, and what happens to a threshold that enforces nothing. Every register column checks against those answers.
- 3
Send the project
A dbt project or just its schema files, a DAG directory, a warehouse schema dump, an expectation suite, or a screenshot of the dashboard nobody trusts.
- 4
Read the rows that do not agree
The numbers standing on an unenforced threshold, ordered by who reads them. The weekly sweep reports any threshold that gets quietly weakened.
Frequently asked questions
Is this template free?
Yes. Download it and nothing asks for a card, an email or an account. Edit with AI is optional and it is where the work happens: the agent reads whatever project you have and fills the contract, including rows for checks nobody ever wrote. Other packs sit in the template library.
What format are the downloaded files?
Everything arrives as one zip of standard files: the documents in .docx, the three registers in .csv. There is no importer, no proprietary format and no account needed to read any of it. The illustrative rows stay in the sheets so each column shows what it expects.
We already have generated docs from our transformation tool. Why this?
Generated documentation describes what the code says. It cannot tell you that a test set to warn enforces nothing, that a freshness rule covers a source rather than the model anybody queries, or that a hardcoded mapping stopped matching reality nineteen months ago. Those are absences, and nothing generates an absence. A test on the wrong column is the gap the analytics standards pack flags.
Does it work if we do not use dbt?
Yes. The mechanisms differ by tool and the shape does not: every stack has checks that fail a run, checks that only warn, thresholds enforced in the warehouse rather than the project, and numbers with no check at all. Send whatever configuration and schema files you have.
Can it tell me who read a number that turned out to be wrong?
That is what the consumer register is for, and it reads query history rather than trusting declared lineage, because scheduled exports and spreadsheet connections appear in no graph. If it becomes an incident, the postmortem starts from that list. What each of those readers thinks the number means belongs in the field and metric dictionary.
How does it handle the pipelines nobody documented?
By reading the transformation code for the decisions somebody would otherwise reverse-engineer: which rows are excluded, how late-arriving data is attributed, which timezone a day boundary uses, and every hardcoded value listed individually. Where it cannot determine something, it writes down who would know. Those exclusions are also what an analysis write-up has to state before it quotes a number.
Our warehouse schema cannot leave our network. Can we still use this?
It does. The pack runs unchanged in a private AI workspace, which keeps schemas, query history and consumer names inside your own tenancy. Teams under a data residency obligation take that route, and the contract, the registers and the drift sweep behave exactly as they do here.
Find out which numbers nobody can stand behind
Start from the blank documents and registers, or let River read your project and say which numbers rest on a threshold that enforces nothing.
Edit with AI