(Being a surveyor in many ways is like being an accountant. Tell me what numbers you want and I can deliver them to you!)
If all the scores are clustered on the high end because of some quirk of questionnaire design, how can you identify where to focus improvement efforts? Those with background in quality management know the goal of Pareto Analysis: separate the critical few from the trivial many. With no dispersion, the critical few are hidden.
Survey Scale Design Experiment
So what scale designs lead to dispersion? We experimented with several different scale designs measuring the same attributes. The different questionnaires were posed randomly to 10,000 respondents on a telephone survey.
Here’s another scale I’ve used that has good dispersion properties. I call it a Met Expectations scale.
If the perceptions matched the expectations, then the response should fall in the center of the scale. The numbering scheme, while still a 5-point scale, reinforces the midpoint as an average position. However, this scale is a challenge to present to management. I recall telling my anxious client when I ran my first numbers on the survey data set that the company scored a 1.1 on the overall satisfaction question! It doesn’t sound anywhere near as impressive as a 4.1 on a 1-to-5 scale.
Summary of Survey Scale Design Lessons
Scale design has a huge impact upon how respondents will answer a question. Thus, a performance measurement system will be greatly affected by the questionnaire design. Goals have to be set cognizant of the questionnaire.
Also, the goals of a performance measurement system using surveys — where higher scores are wanted — can compromise the role of that same survey program in identifying areas for quality improvement where we want a dispersion of responses to separate out the “critical few” on which to focus attention.