Posted: February 20, 2018
Surveys can help us learn how a group of people feels about something, be it customer or employee satisfaction, future needs, attitudes towards an organization, views of some policies, or a whole host of other things. To truly learn the data collected must be valid, that is, they must measure what you’re intending to measure.
How can it not? Well, there’s numerous ways we can corrupt our survey process, but the single biggest mistake is ambiguous wording in our survey questions. That happens when the meaning of key words or phrases is unclear. People could have different interpretations of the question.
How then can we interpret the data in different people were, in essence, asked different questions?
Examples of ambiguous questions abound, but let’s study a survey question asked in US since the 2016 presidential election: Russian meddling in the presidential election.
Numerous public opinion surveys have sought to measure the public’s view on this, and they typically use phrasing similar to:
“Do you think Russia interfered in the 2016 election?”
The question might be posed as a binary Yes/No question or on an Agreement interval rating scale.
Seems straightforward, right?
Yet, two people with identical underlying viewpoints could give diametrically opposite responses. How? The ambiguity in the question phrasing.
That quandary exemplifies the validity issue created by ambiguous questions.
An ambiguous question in a survey results when various people interpret some word or phrase in different ways. The above election-meddling question contains four words that form how you interpret what the question is asking. (Researchers would say these words “operationalize the construct.”)
Let’s take them in order.
At this writing, significant evidence shows “Russia” fed misinformation and fomented disruption during the campaign through Facebook and other social media platforms. Some evidence exists that local election systems were probed for possible hacks, but none were successful apparently.
So, a respondent with a tight definition of “election” as the voting and vote tallying process could answer No to the question, while someone else with a looser definition of “election” to include the campaign could answer Yes.
Both would be right. And both could believe that Russia engaged in nefarious “electioneering.”
You might think that an interval rating scale would solve the problem, but it wouldn’t. If using an Likert-type Agreement scale, you would ask the level of agreement with the statement, “Russia interfered in the 2016 election.” If you interpreted “election” as the voting process, then your Agreement rating would be low. Someone with a broader definition of “election” would give a high Agreement rating. Ambiguity still is an issue.
Of course, this all assumes there’s no response bias in play. Many people would answer this question not based upon their view of Russian interference in the election, but only based upon their like or dislike of President Trump — or the sponsors of the polling. In this case, their response is driven by an ulterior motive and not an honest statement of how they feel about the issue at hand.
I’ve had people challenge me on my concern for ambiguous questions. Data are data they would argue. My pushback to them is simple: “How can you interpret the results? More importantly, what action would you prescribe?” Hmmm…
Let’s assume some percentage of the respondents interpreted “interfered in the 2016 election” as meaning the voting process and some interpreted it as feeding misinformation into the campaign. The implications for the action differ depending upon which interpretation was predominant. And you wouldn’t know which meaning was most commonly assumed!
Running your own survey programs? Learn how to do it right from us — the premier worldwide trainers for survey design & analysis.
Featured Classes:
In reality, many (most?) survey designers will be blind to their ambiguous questions and will simply assume that all respondents interpreted the question the way they, the designer, intended for them to interpret it. This assumption can lead to erroneous findings and recommendations for action.
And that’s why ambiguous questions are the biggest problem in survey research.
The unskilled researcher is likely blind to the validity problem.
For the unscrupulous researcher, ambiguous questions are wonderful. They would use a broad phrasing that might lead many to agreement with a statement even though their true feelings are more narrow. The researcher can claim a mandate for action where none in reality exists.
If we’re going to take action, that action should be based upon real knowledge, not interpretations of questionable research.
In my survey workshop training, I preach the need for survey instrument testing, both internal and external, before going live. Here are three recommendations.
The above suggestions will reduce ambiguity and should eliminate the really egregious errors. But your survey questionnaire will never be 100% clean. Yes, that’s the goal, but you’re unlikely to achieve it. Some respondents due to some life circumstances will have some bizarre interpretation of some question(s). That’s reality.
That doesn’t mean we should just accept ambiguity in our questions. We shouldn’t, and we should strive to eliminate ambiguity. Just know that perfection is unattainable.