An Example of the Impact of Question Sequencing
Summary: The New York Times and CBS News released a nationwide poll on July 19, 2012 that conveniently ignores the impact of question sequencing and presents questionable interpretations of the data. The example shows why consumers of survey data should always examine the methodology of the survey, especially the design of the survey instrument.
~ ~ ~
In a related article I looked at some polling done by the New York Times, CBS News, and Quinnipiac University. In this article, I’ll turn to a nationwide poll that the Times and CBS News released on July 19, 2012. It shares many of the questions that the state-focused polls do, and it’s a horribly long survey at about 100 questions. My focus here is on the impact of question sequencing and how the reporters summarized and presented the findings. Again we see why you should always examine the survey instrument and the methodology of the surveyor before believing the survey’s findings — especially as presented.
About two thirds of the way through this long survey after a series of issue questions, Question 41 asked:
41. Looking back, how much do you think the economic policies of George W. Bush contributed to the nation’s economic downturn — a lot, some, not much, or not at all?
I ask you, the reader, to think about your “mental frame” as you consider that question. In other words, what are you thinking about? To achieve a valid questionnaire, every respondent should have the same interpretation of the survey questions. So, for this question to be valid we should all have similar interpretations — and the person who summarizes the results should also share that interpretation.
I think it’s fair to say that most people would be thinking about how much they blame the recession of 2008-09 on the Bush policies. That’s when the “economic downturn” occurred, and the authors of the survey have asked you, the respondent, to “look back.”
The results of that question were:
a lot — 48%
some — 33%
not much — 12%
not at all — 6%
don’t know — 2%
Here is how those results were presented in the New York Times article, which was the closing thought for the article.
Nearly half of voters say the current economic plight stems from the policies of Mr. Obama’s predecessor, George W. Bush, which most voters expect Mr. Romney would return to. (emphasis added)
Question 41 did not ask about the “current economic plight.” When you read “the nation’s economic downturn” in question 41 were you thinking of the “current economic plight?” I doubt it. (Economic growth is miserably anemic as I write this in August 2012, and the economic tea leaves are not pointing up, but currently available data do not have us in a “downturn.”) Granted, the question does not have a specific timeframe, so the authors can get away with this interpretation. I guess.
Question 42 repeated the previous question but asked about President Obama.
42. Looking back, how much do you think the economic policies of Barack Obama contributed to the nation’s economic downturn — a lot, some, not much, or not at all?
The results of that question were:
a lot — 34%
some — 30%
not much — 23%
not at all — 12%
don’t know — 1%
The reporters didn’t see fit to report these results in the article. More interesting to me as a survey designer is that Questions 41 and 42 were rotated. I would love to see the results broken out based upon which question was asked first, but the Times does not provide that detail.
Clearly, there is a sequencing effect in play.
If you were asked first about Obama’s impact on the “economic downturn,” you are certainly thinking more near term. It is doubtful that people were considering the 2008-09 recession as Obama’s fault (except maybe for those real political wonks who know of Senator Obama’s votes protecting Fannie Mae and Freddie Mac from proposed deeper regulatory oversight but even then the impact would be minimal).
So hearing the question about Obama’s impact on the “economic downturn” has set a more near-term mental frame. Now you are asked about Bush’s impact on the “economic downturn.” Are you thinking about the 2008-09 recession? Certainly not as much as if the Bush question were asked first. I think it’s fair to say that people blame Bush far less for today’s economy than the economy of 2008-09.
To summarize, I am sure the scores for questions 41 and 42 varied significantly depending upon which one was asked first. If we were only told the splits…
The proper, unbiased phrasing for the question would be,
Thinking about the current state of the economy, to what extent do you consider [Bush/Obama] to blame for the economic problems our country currently faces?
That in fact is how the writers of the article in the Times present the question, but that’s not the question that was asked. Far from it.
~ ~ ~
Now let’s look at the last phrase of the Times summary.
Nearly half of voters say the current economic plight stems from the policies of Mr. Obama’s predecessor, George W. Bush, which most voters expect Mr. Romney would return to.
According to the polling data, do “most voters expect Mr. Romney would return to” President Bush’s policies? This finding is based on question 57:
57. If elected, how closely do you think Mitt Romney would follow the economic policies of George W. Bush — very closely, somewhat closely, or not too closely or not at all closely?
The results were:
very closely — 19%
somewhat closely — 46%
not too closely — 18%
not at all closely? — 7%
don’t know — %
We can debate until the cows come home and the keg runs dry about the interpretation of “somewhat closely.” But perhaps more importantly, the survey treats “economic policies” with one broad brush. Some of those policies led to the “economic downturn,” but other policies most assuredly did not.
Further, some of the respondents who believe Mr. Romney “would return to” Bush policies may not have responded in Question 41 that they thought those policies “contributed to the economic downturn.” You cannot legitimately make the statement that the authors did linking the results of Questions 41 and 57 without segmenting the data and analyzing it properly. But they did.
~ ~ ~
Bottom line. The closing statement of the New York Times article distorts what the survey data actually said, due to sequencing effects and a convenient reinterpretation of the question. The Times is making it sound as if the polling supports the contention that voters still hold Bush responsible for the current weak economy. That may be true, but these polling data, properly analyzed, do not support that contention.