Product & DesignFree
Usability Test Report With Severity Ratings
Paste your session transcripts and tasks. Get findings ranked by how many people each problem blocked, an issue sheet, and a readout deck.
River reads the sessions rather than asking you to grade them. Transcripts, notes and the task log all go in, and every finding comes back attached to the task it broke, the participants who hit it, and the point in the session where it happened. Severity is computed before it is judged: how many people attempted the task, how many failed it, and how many recovered without help. The researcher's own rating sits beside that count as a separate column rather than replacing it.
The 0 to 4 severity scale every report template copies came from Jakob Nielsen. The same page that defines it warns that severity ratings from a single evaluator are too unreliable to be trusted, and recommends averaging three. Almost no team has three evaluators, so the number that decides which problems get scheduled is one person's estimate formatted as data. Most report templates hand you that column and leave you to fill it.
Built for the researcher writing up a moderated round on a deadline, and for the designer who ran the sessions because there is no researcher. Reach for it when the last session has finished and the findings document has not started. Interviews without tasks are a different analysis, and that is research synthesis. Unsolicited complaints from a support queue are customer feedback triage. Once a fix ships, whether it worked is an experiment readout.
What a severity rating is actually measuring
Task completion is the one usability measure that does not require an opinion. A participant either finished the task or did not, so the result is a 1 or a 0 and the rate is arithmetic. Jeff Sauro's analysis of 1,189 tasks across 115 usability tests puts the median completion rate at 78 percent, which is the benchmark a finding gets measured against. A task your participants completed 40 percent of the time is not a matter of interpretation.
Six participants, one checkout flow, four findings. The label everyone squinted at got rated a major problem because it was the thing the researcher noticed most; it cost four people about ten seconds each and all six finished. The billing address field that silently rejected a valid postcode was rated minor, and it stopped five of six from completing at all. Ranked by rating, the label leads. Ranked by who failed the task, it is last.
The other half of getting a fix scheduled is making the finding watchable. A row saying users were confused by the billing step is an argument. A row carrying the participant, the timestamp and the forty seconds where they typed the postcode three times is not. Every finding here points at its moment in the session you supplied, so the person who has to fix it can watch it fail rather than take your word for it.
How it works
Hand over the sessions
Transcripts, notes and the task list, in whatever form your research tool exported them.
Score the tasks
Each participant marked complete or failed per task, with the ones who recovered without help noted.
Derive the severity
Findings ranked by who they actually blocked, with your own rating alongside and the gaps called out.
Write the report
Findings ranked with the moment to watch for each, an issue sheet, and a readout deck.
What you get
- One row per finding, carrying the task it broke and the participants who hit it
- Severity derived from attempts, failures and unaided recoveries before any rating is applied
- Your own severity rating kept as a separate column, with the disagreements flagged
- Every finding pointing at the participant and timestamp where it happened in the session
- Task completion rate per task, against the 78 percent median from published benchmarks
- A readout deck built from the same rows as the findings document and issue sheet
Common questions
How is severity worked out if I only had one researcher?
From the sessions rather than from you. Each finding carries how many participants attempted the task, how many failed it, and how many got through only with help. That produces a rank before anyone rates anything. Your own rating still goes in its own column, and where the two disagree the report says so.
Is five participants enough to report on?
Five is enough to find problems and not enough to measure rates precisely, and the report treats those as different claims. A finding that blocked four of five participants is reported as four of five, never as 80 percent of users. Nielsen's own five-user recommendation is about discovery, not about producing a completion rate you can quote.
Can it work from transcripts alone?
Yes. Transcripts are the main input and notes are a convenience. Where the task log is missing, completion is read from the transcript and each judgement call is flagged as inferred rather than recorded. Sessions where the participant went off-script are marked instead of being scored as a clean pass or fail. If you only have video, export a transcript from your recording tool first.
What about unmoderated recordings from Hotjar or Clarity?
Those are a different job. Without set tasks and a moderator there is no pass or fail to score, so this report does not fit them. The session recording review works from those exports instead: it compares what people who left clicked with what people who stayed clicked, and picks the recordings worth watching.
What does the engineering team actually receive?
An issue register, one row per finding, with the task, the failure count, the derived severity, the screen element and the timestamp to watch. It drops into a backlog without rewriting. When a finding needs a requirement written into a spec, the PRD generator picks it up from there.
Does this replace my research repository?
No, it starts where the repository stops. Dovetail and Lookback store and tag sessions; neither produces the ranked findings document. Export one session at a time if that is all yours offers, and hand over the whole set. The coded output is a sheet you keep, not a view inside another tool. Filing the finished report by the questions it answered is what the research repository pack does.
Sources
Built by
Chandler Supple
Co-founder & CEO, River
Chandler has spent the last six years working in AI, building tools for millions of users and Fortune 100 companies. In 2025, he co-founded River to build a new kind of operating system for AI agents.
Usability Test Report With Severity Ratings
Describe what you need, add any files you have, and your workspace opens with the work already underway.