What Makes a Survey Question Ambiguous?
An ambiguous question in a survey results when various people interpret some word or phrase in different ways. The above election-meddling question contains four words that form how you interpret what the question is asking. (Researchers would say these words “operationalize the construct.”)
Let’s take them in order.
- Think. Various respondents might require different levels of proof to assert that they “think” or “believe” something in reality happened. Imagine how differently some would respond if the question asked: “Are you certain…”
- Russia. What does “Russia” mean in this context? The Russian people? Russian oligarchs? The Russian government? My guess is that most everyone would interpret this as Vladimir Putin or his operatives, but some would certainly use one of those other options.
- Interfered. This word is open to interpretation and needs to be viewed in the context of the following prepositional phrase “in the 2016 election”. How you interpret “election” affects how you interpret “interfered” as we’ll see.
- Election. Here we have real ambiguity. For some “election” means the actual voting process where we cast ballots and tally the results. Others may view “election” as including the months-long campaign. Those are quite different. So “interfered in the 2016 election” could be interpreted as:
1) To influence people’s views or
2) To change voting results.
At this writing, significant evidence shows “Russia” fed misinformation and fomented disruption during the campaign through Facebook and other social media platforms. Some evidence exists that local election systems were probed for possible hacks, but none were successful apparently.
So, a respondent with a tight definition of “election” as the voting and vote tallying process could answer No to the question, while someone else with a looser definition of “election” to include the campaign could answer Yes.
Both would be right. And both could believe that Russia engaged in nefarious “electioneering.”
Maybe Use an Agreement Scale Format?
You might think that an interval rating scale would solve the problem, but it wouldn’t. If using an Likert-type Agreement scale, you would ask the level of agreement with the statement, “Russia interfered in the 2016 election.” If you interpreted “election” as the voting process, then your Agreement rating would be low. Someone with a broader definition of “election” would give a high Agreement rating. Ambiguity still is an issue.
Of course, this all assumes there’s no response bias in play. Many people would answer this question not based upon their view of Russian interference in the election, but only based upon their like or dislike of President Trump — or the sponsors of the polling. In this case, their response is driven by an ulterior motive and not an honest statement of how they feel about the issue at hand.