Survey Training Classes
Running your own survey programs? Learn how to do it right from us — the premier worldwide trainers for survey design & analysis.
Featured Classes:
Have a look at these results from an April 20, 2020 CBS News Poll on Covid. (Detailed breakdown here. Well, partial details…)
- If you had to choose, what should be the top priority for the country right now?
| Try to get the economy going by allowing businesses to reopen with appropriate safety practices, including social distancing in the workplace, even if means more people might be exposed to the coronavirus | 30% |
| Try to slow the spread of the coronavirus by keeping people home and social distancing, even if this does near- to long-term damage to economy with many permanent businesses closings and lost jobs. | 70% |
Now I have to be honest. I just lied to you. As a researcher, I strongly believe in ethical research, but for this research the deceit had a purpose. Here’s the actual CBS News poll results:
- If you had to choose, what should be the top priority for the country right now?
| Try to get the economy going by sending people back to work, even if it means more people might be exposed to coronavirus | 30% |
| Try to slow the spread of coronavirus by keeping people home and social distancing, even if the economy is hurt in the short term | 70% |
Do you think the response percentages would be different if the first set of response options were given? I’d make a large wager on that.
Binary Choice Question Type: Why the Media Loves Them
Surveyors would call the above question type a multiple choice, single response . More specifically, it’s a binary choice question type. The respondent has to pick one or the other.
Accordingly, a journalist knows he can write: “A majority of people said…”
Pretty slick, eh?
The question is also called a forced choice question. In research forcing respondents to engage in trade-off helps get us to truly understand viewpoints. If those two options were each posed on an agreement scale, we’d probably find a majority of people agreeing with both statements.
Both questions could draw high agreement
While one option may get a higher agreement percentage, the contrast isn’t as stark. Agreement scales allow the respondent to “have his cake and eat it too.” Binary choice questions make you cast your fate with one of them.
Binary Choice Questions: Design Considerations
Because the findings are so stark, question design here is critical. The two response options should be:
- Diametrically and equally opposite. That is, each should be the same “distance” from a neutral position
- Cleanly stated without any loaded language
- Rotated so the order varies across respondents to avoid a primacy or recency effect.
One reason “researchers” like this question type is that it’s easy to “manufacture” a desired response – and thus a desired headline. I will suggest that’s what happened here.
How Does This Binary Choice Question Stack Up Against Good Design Criteria
Let’s examine the first option.
Consider the loaded wording used: “by sending people back to work”. Maybe we send our kids to school or send soldiers off to battle. And in totalitarian authoritarian regimes people may be sent back to work, but in a free market economy employers induce people to work for them. If the writers of this question think their wording is appropriate and not loaded, it speaks volumes about their view of our socio-economy.
Thankfully, the writers didn’t go off the deep end in the 2nd half of the sentence. They wrote that “more people might be exposed to coronavirus,” not “will contract coronavirus.”
Let’s turn to the second option.
Events have happened quickly over a short time span with this virus, but even in mid-April when this poll was conducted, did anyone believe that the economic damage of the lockdown would be “short term”? That wording minimizes the negative consequences of the lockdown policy. Unless you believe that “short term” means 3 to 5 years.
In mid-April we were a full month into the lockdowns and the longer term implications were evident. Virtually every day we had heard news of companies – restaurants for example – that didn’t intend to ever reopen. (Restaurant churn is normal in a dynamic economy, but normally a restauranteur could sell her business. Not now.) Major retailers were considering bankruptcy protection, which has happened. The economic writing on the wall was clear.
Further, the damage is more than economic; it’s societal. Other health issues are exacerbated by delayed “non-essential” medical procedures, and as we saw during the Great Recession, mental health and drug abuse issues grew. And that was without confinement.
To summarize, the first option is for the cold and heartless while the second option comes from a fantasy world where the lockdown policy impact is minor.
Is it surprising that the results were 70% to 30%? That’s only rational. If those were the choices in reality, I would pick the 2nd option too! The options used loaded language and were not diametrically equal opposite.
Were the options rotated? I have no idea. The methodology description is sparse – and that’s being kind. My educated guess: no rotation.
How Does The Alternative Binary Choice Question Stack Up Against Good Design Criteria
In the response choices I wrote, notice the difference in the operative phrases:
- “allowing businesses to reopen with appropriate safety practices”
- “near-to long-term damage to economy with many permanent businesses closings and lost jobs”
I will argue that those statements more accurately reflect the reality and choices at hand — and they’re more balanced opposites. The first option reflects the idea of the “new normal.” The CBS version presented a return to the “old normal,” thus driving people to the lockdown choice.
Let me be clear. I am not saying these are easy decisions as I write this on May 15, 2020. Heavens no! Whatever public policy decisions are made will be wrong in some way. Zero risk options do not exist!
I do feel that the downsides of extended lockdowns have been downplayed –- for whatever reasons. We’re looking for the right balance with proper health protocols.
We hear that policy decisions are (or should be) “informed by science.” Polling data such as these data are not scientific; they are manipulated, manufactured data that is being presented as truth.
Convenient Restating of the Question Options
But have the headlines presented the nuances of the question? Here are the headlines from the CBS News website:
And here’s the graph that summarized that question:
Are the description true reflections of the questions?
Notice what’s missing? No “send people back to work” where they might get exposed. And no “short term” economic impact. Those conditional statements are critical to understanding what drove respondents to their selections!
Look at the headlines and the chart. Now re-read Question 9 above. Is it an honest presentation of the findings? If you can say that with a straight face, I don’t want to play poker with you.
Here’s a legitimate summary of the findings:
Americans Prioritize Staying Home Over Sending People Back To Work If The Economic Damage Is Short Term
Rephrasing a survey question for the click-bait headline is common practice. And it’s intellectually dishonest. Again, we’re supposed to have decisions guided by science. This practice is the antithesis of science.
Possible Sequencing Effects
To be fair, the survey had Question 14 that again showed that most people were concerned that “Things will open up too fast.”
However, is there a sequencing effect in play? Question 9 conditioned respondents to view the lockdown more positively — or at least less onerous. (After all, it’s only “short term” damage.)
Do note that Questions 9 & 14 measure the same attitude, but there was a 7% swing. Not overwhelming, but likely to be statistically significant. Could that be because Question 14 used the more loaded pejorative term “lockdown”?
Selective Reporting of Results
Here’s a screenshot from the so-called “Topline” results of the poll for these questions.
What happened to Questions 10 to 12?
Now, a perceptive reader might be asking why I didn’t include the results for Questions 10 to 12. (Gold star to you folks!) I didn’t because I couldn’t. The report on the survey excluded the results from those questions, as well as from Questions 20-23, 25-26, 28, 40 to 43.
So, results from 13 of the 43 questions were omitted. Almost a third! Were they ever released as promised at the end of the report? If so, I couldn’t find them.
Why not disclose results from every question? Did the findings prove contrary to the desires of the pollsters? I know this makes me sound like some conspiracy theory nut, but the authors foster this view with their omissions. If someone hid something from you, how much credibility would you give them for their other claims?
Digesting Polling Data: Some Advice
James Carville, Bill Clinton’s pollical advisor, is famous for stating in the 1992 election that “It’s the economy, stupid.”
When it comes to polling data:
“It’s the methodology, stupid.”
Never trust the findings from any poll without looking at the methodology:
- Instrumentation bias:
- Question wording
- Sequencing of questions
- Survey invitations & introductions
- Response bias
- Administration bias & accuracy
- Sample size
- Sample bias: How the sample was drawn
Legit cave.
However, if you have some point you want to prove with survey data, binary choice questions are a good option.
