Research & PolicyFree
Open Ended Survey Response Coding Tool
Send the free-text export, get coded responses with theme frequencies by segment and a written record of what the scheme missed.
River builds a coding scheme from a random subsample of the responses, writes it down with an inclusion rule and an example per code, then applies it to every response and counts. What comes back is a coded row per respondent joined to your segment variables, frequencies on a base that is named rather than assumed, and a count of the responses no code fit. That residual is the finding most reports leave out, and it is the one that tells you whether the scheme describes the data.
Search this and page one is survey vendors explaining that you should read your responses and look for themes, plus text-analytics products that return a word cloud and a sentiment score. Neither produces a scheme anyone can audit. A word cloud cannot be checked, and a theme found by reading cannot be applied consistently by a second person. Both quietly turn the free-text box into a place to shop for quotes that agree with the closed items.
For the analyst writing up an optional free-text item, the author answering a reviewer who called the quotes cherry-picked, and anyone designing next year's closed items from this year's text. It expects a cleaned export, which splitting metadata from responses and logging every exclusion produces, and its counts feed a results report that carries each subgroup's base. Where the item wording is the problem, the instrument pack that documents each item's purpose is the place to fix it.
Twenty five responses no code fit
A membership survey with 2,180 respondents and one optional free-text item. 1,347 left something in the box, an item response rate of 61.8 per cent. 118 of those are non-substantive, mostly 'n/a', 'none' and single characters, which leaves 1,229 codeable. The federal survey standards ask agencies to plan for a nonresponse bias analysis if the expected item response rate is below 70 percent for any item used in a report. This item is under it, before any coding happens.
The scheme is drafted on a random 200 of the 1,229 and comes to 11 codes. Applied to all 1,229 it produces 1,586 code applications, 1.29 per response, because people raise more than one thing. Ninety six responses fit no code, 7.8 per cent. Reading only those ninety six yields two more codes covering 71 of them, and the remaining 25 stay unclassifiable, 2.0 per cent of codeable responses. The scheme has 13 codes and a stated residual, and the residual is what makes the other 13 believable.
Then the base. The cost theme was applied to 412 responses. That is 33.5 per cent of codeable responses, 30.6 per cent of everyone who wrote something, and 18.9 per cent of all respondents. Same 412, three defensible denominators, a 14.6 point spread, and no convention that picks one for you. By tenure the reportable range is 31.9 to 37.3 per cent, a 5.4 point spread. The undisclosed-tenure segment has a base of 22 and is suppressed rather than reported.
How it works
Read a subsample
A random subset drafts the scheme, so the codes are not fitted to the responses you noticed.
Write the scheme
An inclusion rule, an exclusion boundary and one real example for each code.
Code everything
Every response, multiple codes where earned, and the ones nothing fits held aside.
Count and disclose
Frequencies by segment on a stated base, plus the residual and what it contained.
What you get
- A written coding scheme with an inclusion rule and a real example for every code
- Every response coded and joined to your segment variables, one row per respondent
- The count of responses no code fit, reported rather than absorbed into an other bucket
- Frequencies on a named base, with every alternative denominator stated alongside it
- Segments below a base floor suppressed instead of reported as a percentage
- Representative verbatims chosen by fit to the code, with the code's frequency next to them
Common questions
How is this different from the theme summary my survey tool already gives me?
A vendor theme summary is unauditable. It reports themes without a rule for what belongs in one, so nobody can check a borderline response or apply the same scheme next year. This writes the rule down first, then applies it, then reports what the rule failed to cover. The scheme is the deliverable and the counts are downstream of it.
Why does the residual matter so much?
Because a scheme with no residual is either genuinely complete or quietly forcing responses into codes that do not fit, and from the outside those look identical. A stated residual lets a reader judge. It is also the most useful raw material you have: the responses nothing fit are usually where next year's closed items come from.
Which denominator should I report?
Whichever one your claim needs, named in the sentence that makes the claim. A share of those who answered describes people who chose to write. A share of all respondents describes the population and will always look smaller. Both are honest, mixing them within one report is not, and a percentage with no stated base is the default failure here.
Can it compare against last year's coding?
Yes, and it will say where the comparison breaks. Prior codes get carried forward where the definitions still hold and flagged where they had to change, because a theme that moved from ten per cent to eighteen means nothing if its inclusion rule widened. The comparison table shows which codes are like for like and which are not.
How does it pick the quotes?
By fit to the code rather than by how quotable they are, and each verbatim is printed next to the frequency of the code it represents. That pairing is the point. A vivid quote standing alone implies a prevalence it may not have, and a quote labelled as one of 44 responses in a segment of 118 tells the reader exactly how much weight to give it.
What if I only have a few hundred responses?
It works, with smaller segment cells and more suppression. Below roughly a hundred codeable responses the scheme is drafted on all of them rather than a subsample, and frequencies get reported as counts rather than percentages. It will tell you which segment breakdowns your base cannot carry rather than printing a percentage off nine people.
What comes back?
A Sheet with one row per respondent, the codes applied, the segment variables, and a frequency table by segment carrying every base. A Doc holding the coding scheme, the themes with representative verbatims and their frequencies, the residual and what was in it, and the item response rate with what it means for the claims.
Open Ended Survey Response Coding Tool
Fill in the form and your workspace opens with the work already underway.