Survey Question Types: How Question Format Affects Survey Analysis

The mathematical answer is clear. Candidate B is the winner with 25 points and the next nearest candidate, C, is a whopping 8 points behind, only 4 points ahead of Candidate D. But look at the spatial map. Is B that much superior? 4 of the 5 committee members thought B & C were really a dead heat. Member 5 didn’t like C for some reason — and liked D more than anyone else.

Are C & D closer in preference than B & C? That’s what the (bogus) math shows, yet the spatial map paints a different picture. B may be preferred to C, but it’s close enough to spark a reasoned debate. That debate in fact, is what happened in our deliberations.

What Wrong Data Analysis Did the Consultants Do?

The consultants had us rank order the candidates, generating ordinal data. They then treated the data as interval data and added the scores.

• Ordinal data means that the answers are in some order, but says nothing about the distance between the ordered items.
• Interval data means there’s an equal distance — cognitively — between the points on the scoring scale.

So, imagine instead that we had been asked to rate the candidates on a 1 to 10 scale. Done properly, that would have generated interval data — if we all viewed the difference a 10 and a 9 as the same as between a 9 and an 8, and so on.

In another article, I make the point that interval scales are not perfect. In fact, they are lousy for measuring importance among a set of factors — or in this case the preference for one candidate. If every member rated every candidate a 10, then no differentiation would result. That’s why the consultants wanted rank orders; it forced us to choose one over the other. But adding the rank scores was wrong mathematically. Near ties and huge gaps were treated the same in the math.

What Data Analysis Would Have Been Appropriate?

They could have developed cumulative frequency distributions that could be derived from the frequency distribution table below for the data displayed in the spatial map:

Look at this table. Candidates B is still the clear leading choice, but compare the analysis of the summed ranked scores with what this table shows. Are Candidates C & D closer than Candidates B & C as the ranked sums indicated? No. Candidate C clearly is second, but D is certainly more distant. This analysis, which is mathematically correct for ordinal data, does show B & C as being close contenders. But still missed is the fact that four of the five members felt those two candidates were almost the same.

Another alternative would have been to use a fixed sum (also known as fixed allocation) question format. In this case, we would have been told to allocate 100 points among the five candidates based upon our preferences. If we felt that all five were equal, then we would have given each 20 points. But if we felt one candidate was better we should allocate more than 20 points to that candidate. However, then some other candidate(s) would have to get lower scores. Our allocations must add to the fixed sum of 100 points.

Based on the spatial map above, the scores might have something like:

These scores are shown in the table below with the averages for each candidate.

Notice what this analysis would show. Candidates B & C are neck and neck. (I used round numbers to simplify the display so maybe not quite so neck and neck.) Since the fixed sum question format captures relative distinctions and has interval properties, the data can be added and averaged. They better reflect the true underlying relationships.