Survey Question Design: Headlines or Meaningful Information?

in

Summary: Surveys can be designed to generate meaningful information or to manufacture a compelling “headline”. This article examines the forced-choice, binary-option survey question format and shows how an improper design can potentially misrepresent respondents’ views.

~ ~ ~

The Wall Street Journal and NBC news conduct periodic telephone surveys done by a collaboration of professional survey organizations. The poll results released on August 6, 2014 included a central question on the “Views of America” regarding individual opportunity. I say “central” since only a few of the 29 survey questions were reported in the paper. Here’s what they reported regarding one question:

A majority of those polled agreed with the statement that growing income inequality between the wealthy and everyone else “is undermining the idea that every American has the opportunity to move up to a better standard of living.”

That’s a pretty startling finding, and this is a bit of a dangerous article to write since it uses a public policy survey as the example. My purpose here is not to argue a point of view, but to show the impact of good question design — and bad. I’ll show how a mistake in survey question design can distort what your respondents actually feel. Or, to approach it from the opposite angle, the article shows how a survey can be used to generate a false headline with seemingly valid data.

~ ~ ~

Let me walk you through the progression of the survey so that you can experience properly this particular survey question. Question 14 used a 5-point scale ranging from Very Satisfied to Very Dissatisfied, “I’d like to get your opinion about how things are going in some areas of our society today,” specifically,

  • “State of the US economy” — 64% were either Somewhat or Very Dissatisfied
  • “America’s role in the world” — 62% were either Somewhat or Very Dissatisfied
  • “The political system” — 79% were either Somewhat or Very Dissatisfied

The next two questions were each asked of half the respondents. “And, thinking about some other issues…”

Q15 Do you think America is in a state of decline, or do you feel that this is not the case?

— 60% believe the US is in a state of decline

Q16 Do you feel confident or not confident that life for our children’s generation will be better than it has been for us?

 — Only 21% “feel confident,” significantly lower than the trend line shown in the report

The next question presents two statements about America with the statements rotated as to which one is presented first to the respondents.

Q17 Which of the following statements about America comes closer to your view? (ROTATE)

Statement A: The United States is a country where anyone, regardless of their background, can work hard, succeed and be comfortable financially.

Let me pause here and ask you to think about that statement. It’s a motherhood-and-apple-pie statement about America. If you disagree with any of that statement, you are likely predisposed to agree with the subsequent statement. And note that the previous questions have elicited quite strong negative opinions. I’ll come back to that effect.

To continue with Question 17…

…or…

Statement B: The widening gap between the incomes of the wealthy and everyone else is undermining the idea that every American has the opportunity to move up to a better standard of living.

Results from Question 17 on Views of America. 54% agree that “Widening gap undermining opportunity”.  (That’s how the pollsters described it in their PDF summary.) 44% agree that “Anyone can succeed”. 1% of respondents volunteered that both statements were true, and 1% volunteered that neither were true. Those options were not offered overtly. For those readers with good addition skills, 2% of the respondents are unaccounted. Perhaps that’s rounding, but the pollsters don’t say.

Sequence Effects upon Question 17. Unfortunately, the pollsters do not report the splits depending upon which statement is presented first. That split could be very enlightening about the sequence effect in that question’s design. That is, are people more likely to chose Statement B if it’s asked first or second?

We also have a sequence effect in play from the previous questions. Would the results be different if Question 17 had been asked before Question 14? I suspect so since the previous three questions put respondents into a response mode to answer negatively. Rotating the order of the questions might have made sense here if the goal is to get responses that truly reflect the views of the respondent. They also don’t report the splits for Question 17 based upon whether Question 15 or 16 was asked of the respondent immediately prior.

wsj-nbc-news-poll

The Design of Survey Question 17. This question is an example of a binary or dichotomous choice question.  You present to the respondents contrasting statements and ask which one best matches their views. This format is also call a forced choice question.

The power of the binary choice question lies in, well, the forcing of choice A or B. The surveyor is virtually guaranteed a majority viewpoint for one of the two choices!! Pretty slick, eh? Look again at what the Wall Street Journal reported:

A majority of those polled agreed with the statement that growing income inequality between the wealthy and everyone else “is undermining the idea that every American has the opportunity to move up to a better standard of living.”

What an impressive finding! Did they tell the reader that respondents were given only two choices? No. Not presenting the findings in the context of the question structure is at best sloppy reporting, at worst disingenuous, distortive reporting. In a moment we’ll consider other ways to measure opportunity in America in a survey, but first let’s look at the improper design of this binary question.

Proper Design of Binary Choice Questions. When using a binary choice question the two options should be polar opposites of the phenomenon being measured. This is a critically important design consideration. In this question, the phenomenon is — supposedly — the state of opportunity in American today.

But note the difference in construction of the two statements.

  • Statement A says the American Dream is alive and vibrant.
  • Statement B says the American Dream has been “undermined” and presents a cause for the decline.

So, if you feel that opportunity has been lessened, you’re likely to choose Statement B even if you don’t feel income inequality is the cause. In the end you’re agreeing not only that opportunity has been reduced, but also to the cause of it.

The astute reader may say the question doesn’t directly assert a cause-and-effect relationship between income inequality and personal opportunity. It says “the idea” of equal opportunity has been “undermined.” This question wording is purposeful. It softens the assertion of a cause-and-effect relationship, making it easier to agree with the statement. Will those reading the findings, including the media, see that nuance? No. The fine distinction in the actual question will get lost in the headline. Just look at the shorthand description that the pollsters used: “Widening gap undermining opportunity”.

Alternative Survey Question Designs to Research Opportunity in America. The issue could have been researched differently. The pollsters could have posed each statement on a 1-to-5 or 1-to-10 Agreement scale and then compared the average scores. Those findings arguably would have been more meaningful since respondents wouldn’t be forced to choose just one. But would the findings have had the same pizzazz as saying “A majority of those polled…”?

Another design choice would be to present more than two options from which to choose. What if the statement options had been:

  • Statement A: The United States is a country where anyone, regardless of their background, can work hard, succeed and be comfortable financially.
  • Statement B: The widening gap between the incomes of the wealthy and everyone else is undermining the idea that every American has the opportunity to move up to a better standard of living.
  • Statement C: The country’s continued economic weakness is denying opportunity for a better standard of living even to those who apply themselves and work hard.

Without much trouble, we could develop even more statements that present equally valid and divergent views of opportunity in America. However, presenting more choices does cause a problem in survey administration.  For telephone surveys we are asking a lot, cognitively, of our respondents. They must listen to, memorize, and select from multiple statements. Each additional option increases the respondent burden, but could the telephone respondents have handled three choices? Probably, if the choices truly represent very different opinions as just presented.

A key advantage of paper or webform surveys is that the visual presentation of the questions allows for multiple options to be presented to the respondent without undue burden and with less likelihood of a primacy or recency effect — the tendency to choose the first or last option.

Something else happens when we present more than 2 choices. We’re much less likely to get a compelling headline that “A majority of those polled agreed…”

The Importance of Clear Research Objectives. Any survey project should start with a clear understanding of its research objectives. For this particular question the research objectives could be:

  • To see if people feel opportunity in America is weakening.
  • To identify the cause of weakening opportunity.

For this survey, that question’s research objective is really both — and neither.

Income inequality may be a legitimate topic for discussion, but is it the primary cause of the loss of opportunity to the extent that no other possible cause would be offered to respondents? I think you’d have to be a died-in-the-wool Occupy-Wall-Streeter to view income inequality as THE cause of any loss of individual opportunity. In fact, the Wall Street Journal‘s liberal columnist, William Galston, discussed “secular stagnation” in an article on August 26, 2014 never mentioning income inequality as a cause of our sideways economy.

How much different — and more useful — would the poll findings have been if the question had been presented this way?

Q17 Which of the following statements about America comes closer to your view? (ROTATE)

Statement A: The United States is a country where anyone, regardless of their background, can work hard, succeed and be comfortable financially.

…or…

Statement B: The opportunity for every American, regardless of their background, to move up to a better standard of living through their own hard work has weakened over the years.

(If respondent chooses Statement B. Check appropriate statement from list below.)
What do you see as the two primary reasons for the drop in individual opportunity?

Income inequality
Weak economy means fewer jobs available
Weak economy means fewer career advancement opportunities
Increase in part-time jobs
Poor educational system
Good jobs are outsourced overseas
Government regulations deters business growth and thus job growth
Can’t get business loans to start or expand a business
Opportunity goes to those with the right connections
Discrimination in career advancement whether racial, gender…
Etc.

(Note that in a telephone survey, we generally do not read a checklist to the respondent. Instead, we pose an open-ended question, and the interviewer then checks the voiced items. Reading a comprehensive list would be tiresome. A webform survey could present a checklist, but the list must be comprehensive, balanced, and unbiased.)

Do you thing a majority — or even a plurality — of respondents who chose Statement B would say income inequality was a primal cause? I very much doubt it.

Wouldn’t this proposed question structure present a better picture of what Americans see as limiting opportunity, more fully answering the research objectives listed above?

But would the headline be as compelling? Pardon my cynicism, but the headline is probably the real research objective behind the question design.

~ ~ ~

Flawed Question Design in Organizational Surveys. Now you know why the forced choice, binary format is liked by those who want to generate data to argue a point of view. Is this restricted to political polls? Of course not. In an organizational survey measuring views of customers, employees, members or suppliers, we could accidentally — or purposely — word the choices to direct the respondent to one choice.

For example, imagine this binary-choice question for an internal survey about a company’s degree of customer focus.

Statement A: All our employees treat our customers in a manner to create and foster greater loyalty.

Statement B: Recent cuts in staff mean our customers are no longer treated by our employees in a way that will increase their loyalty.

Or a survey on member preferences for some group

Statement A: The association’s programs provide real value to our organization.

Statement B: The content offered in the associations programs doesn’t meet our requirements.

Notice how the question structure here parallels the question in the poll. A novice in survey design could stumble into the error in the above questions’ design. Or the person could want to “prove” that staff cuts are endangering customer loyalty or that content is the problem for members’ disaffection.

Any survey designer worth his salt can prove any point you want through a “professionally” designed and executed survey. And someone designing surveys for the first time needs to be aware of the importance of proper survey question design to generating meaningful, useful data.