Product & DesignFree
NPS Survey Analysis and Score Decomposition
Your respondent mix moved more than your customers did. This splits the reported change into who answered and what they think.
River takes the raw export with every column still attached, and runs the same quarter twice. Once as reported. Once with this quarter's scores applied to last quarter's respondent mix. The difference between those two numbers is how much of your movement was customers changing their minds, and how much was simply a different set of people picking up the survey. At the fictional Kellsworth the reported score rose 6.8 points and the mix-held version rose 0.1, computed from the same 1,600 rows.
That split is only possible if the metadata survives the export. Plan tier, seats, account age, renewal date, region and who the survey was sent to are what turn 1,600 rows into segments. Qualtrics writes an embedded field into the file only if it was saved to the survey flow, so a column nobody declared is not sparse, it is absent. The dashboard shows a headline and a wall of verbatims, and the columns that would have explained the headline sit unread in the download.
Then the comments get read inside score bands and inside segments, never as one pile. Detractors comment at more than twice the rate of passives, so a theme table built over everything silently over-weights them. Send the export, the previous period if you have it, and whatever you know about how the survey went out. If the same complaints are arriving through tickets and public reviews, say so, and if the instrumentation is the real problem, audit the events first.
A frequency table cannot tell you the number moved
Every guide to this job proposes the same sequence. Split into promoters, passives and detractors, tag the comments into six to twelve themes, cross-tab theme frequency by band, then sort by frequency and fix the top three. It is a reasonable sequence and it cannot answer the only question anyone actually asks, which is whether the number moving means anything. Frequency tables describe the people who answered. They say nothing about whether those are the same people who answered last time, and they are usually not.
Kellsworth ran the same survey two quarters running to 14,270 accounts and got 1,600 responses both times, an 11.2% response rate in each. Between them the survey moved from an email to account admins to an in-product prompt. Starter accounts log in least, so their share of respondents fell from 40.0% to 26.9% while Business went from 18.8% to 26.2% and Enterprise from 8.8% to 15.6%. Nobody touched the questions. The reported score went from 28.4 to 35.2.
Hold the mix at last quarter's shape and Q4 comes out at 28.5, so 6.7 of the 6.8 points are who answered. Weight each tier by accounts rather than by answers and the score falls 1.8 points instead. Starter is the only tier that got worse, down 4.1 points, and it is 66% of the account base. Three defensible numbers come off the same export, computed the same way both quarters, and they disagree about the direction.
How it works
Send the export
The full download with every column, not the summary. The previous period too, if you have it.
Segments get rebuilt
River reads the metadata columns back into tiers, cohorts and send methods, and reports what is missing.
The change gets split
Scores are recomputed on last period's respondent mix, so mix and sentiment separate and still sum to the headline.
Comments read by band
Themes are counted inside each score band and inside each segment, with the comment rate stated beside every count.
What you get
- The reported change split into a mix effect and a sentiment effect that add back to the headline
- Every metadata column carried through, so score by tier, cohort, region and send method all survive
- An account-weighted score beside the response-weighted one, and the gap between them named
- Comment rate per band, so a theme frequency table is read against the base that produced it
- Detractor themes broken out by segment, not just by band, and ranked inside the segment that moved
- The passive band costed in points, including what converting a tenth of them would be worth
Common questions
What does holding the mix constant actually do?
It recomputes this period using each segment's current score but last period's share of respondents. If the answer matches the reported number, your mix was stable and the movement is real. At Kellsworth it came out at 28.5 against a reported 35.2, which says almost the entire quarter's gain was who picked up the survey.
Our export has no segment columns. Is this useless?
It loses the mix decomposition, which is the sharpest part. What still works is comment coverage by band, theme frequency read against the right base, and the passive arithmetic. You also get a specific list of the fields to attach before the next send, which is a cheaper fix than it sounds and pays back the first time the score moves.
Why weight by accounts as well as by responses?
Because response rates are not equal across tiers and never have been. Kellsworth's Enterprise accounts answered at 40.3% and Starter at 4.6%, so a response-weighted score is mostly a large-account score. Weighting each tier by its account count gave 19.0 against a reported 35.2, and moved in the opposite direction across the quarter.
Why do the passives get their own treatment?
Because the score is the promoter share minus the detractor share, nines and tens minus zeroes through sixes, and sevens and eights sit in the denominator only. Moving one passive up is worth exactly what moving one detractor up is worth. Kellsworth's 334 passives left 103 comments, so they are the quietest band and the cheapest points.
Does theming all the comments together not work?
It works, it just answers a different question than people think. Detractors commented at 74.1% and passives at 30.8%, so detractors were 21.9% of respondents and 33.2% of comments. Every theme count over the whole pile carries that 1.5x tilt, which is fine if you know it is there and misleading if you do not.
How does this change what we do next?
Kellsworth's top detractor theme overall was a slow report builder. Inside Starter, the only tier whose score fell, the top theme was renewal pricing at 40.9% of its comments, and 73% of every renewal-price comment came from that tier. It ranks third in the table everyone reads, first in the segment that moved, and it routes to pricing rather than the backlog.
What about tracking this over time?
The mix-held series is the one worth trending, because it is the only version where two quarters are comparable. Put it beside the reported number rather than replacing it. If you are already treating survey scores as an outcome measure, keep the definitions traced so the same question does not come back next quarter.
NPS Survey Analysis and Score Decomposition
Fill in the form and your workspace opens with the work already underway.