I’m anxious about your reaction to this article.
Unclear what I mean by that? Exactly the point. When designing survey questions and response scales for interval rating questions, clarity of meaning and lack of ambiguity are critical to capturing valid, useful data, data that don’t suffer from instrumentation bias.
Why Examine the Survey Response Scale Design?
If you review the actual survey instruments used by pollsters, you will see questions that should give you cause to pause before accepting the findings. We all should do this because the reporters won’t — or can’t. This election cycle we again see the Washington Post ABC News poll using an anxious survey response
scale. (See image at top of article.)
Ambiguity in the Survey Response Scale Design
What’s the problem here? “Anxious” has multiple meanings. I am anxious to hear what you feel about this article, but I’m also anxious about what you’re going to say. I explained the distinctions in an article from the 2012 election cycle this way: In high school,
- I was anxious for the school year to end, but
- I was also anxious about final exams.
One meaning addresses timing – I can’t wait! The other addresses concern – Ugh, I’m scared!
This year’s version of the Washington Post poll may be better compared to the 2012 version. (See nearby screenshot.) The 2012 scale ran from Not Anxious At All to Very Anxious. This year the response scale runs from Very Anxious to Very Comfortable, attempting to establish a contrast between comfort and anxiety. (Let’s just hope they don’t try to trend the data from 2012 to 2016. Reporters no doubt will.)
However, the new question wording could still lead respondents to misunderstand the question. If you analyze the question, it’s clear the survey designer wanted the respondent to focus on the degree of concern they had or have for a certain person serving as president.
But remember that the respondent doesn’t have time to analyze the question, especially since the survey is delivered by telephone. They can ask for the question to be repeated, which would entail some negative stigma, but they can’t just reread it as on a paper or webform survey. The interviewers’ intonations can also lead the respondent toward an interpretation — an interviewer bias.
Some number of respondents are going to hear the question and think, “Yes, I am anxious for Hillary (or Donald or Gary or Jill) to become president.” In other words, they may not just be comfortable with that person as president, but they can’t wait for the person to take office. Anxious now isn’t the polar opposite of comfortable connoting anxiety; it’s farther out on the same end of the scale!
Proper Survey Response Scale Design
A properly worded question doesn’t require the respondent to parse the language of the question and think, “Is this what they mean?” If so, then the question is unclear and ambiguous, compromising what researchers call validity. The survey question wording coupled with the response scale wording should allow the respondent immediately to be thinking about the appropriate response option — not trying to figure out the question.
How can we tell if our survey question wording needs improvement? Just like any product, a questionnaire should be tested internally and externally. Internal (alpha) testing involves asking colleagues to review and critique the questions. External (beta or pilot) testing involves asking people from the target audience to take the survey with an interviewer posing probing questions to identify potential ambiguities. For example, ask the respondent, “Why did you give that response?” will elicit the respondent’s interpretation of the question.
Perhaps the best testimony to the problem with this survey question and survey response scale design comes from the pollsters. Why did they redesign the question so dramatically?
Clearly they were anxious about the findings from the original question design.
~ ~ ~
Questionnaire design, question wording, and all elements in survey project design and execution are key parts of the training delivered in our Survey Workshop Series.