What Pollsters Can Teach Us — About Survey Practices
Summary: Political pollsters have a lot at stake in their polling predictions — their reputations. Many of their challenges are the same ones we confront in surveying our customers or employees or members. Recent newspaper articles discuss some of the challenges pollsters are confronting, and we confront similar changes. This articles presents some of these challenges to survey design and administration.
~ ~ ~
Deep into silly season with the US Fall 2010 elections right around the corner, we’re all sick of the ads and the polls. At least we’ll get a respite on November 2 from all this, but you can be sure the 2012 election starts in earnest on January 2.
That said, what can we learn professionally from all the political polls? We certainly see differences across polls. The differences can result from many factors, factors that are also in play when we design surveys to measure feelings of our customers, employees, members etc. The Wall Street Journal (WSJ) has had a couple of research articles examining the differences, which are enlightening. – and you don’t see this kind of public discussion about organizational surveying practices.
WSJ’s “The Numbers Guy,” Carl Bialik, had a recent article on the question rating presidential approval. Gallup presents the respondent with binary options, approve or disapprove. In contrast Rasmussen, presents a four-point scale adding “somewhat approve” and “somewhat disapprove” options. The Rasmussen poll finds much lower net approval ratings – those indicating some level of approval minus those indicating some level of disapproval. The difference is significant, about 5 % to 15%. Scott Rasmussen posits that those with mild disapproval are hesitant to say “disapprove” when presented with only the hard, binary options.
I’ve see binary options in many surveys. For example, hotel surveys may ask, “Will you stay at our hotel the next time traveling to our city? Yes or no?” Sorry, the world is not that clear cut. Many factors will be in play. If I see this on a paper survey, I add in a “maybe” option. If it’s a web form survey, I usually will skip the question or say, “no” and explain why in a comment box, hoping someone will read it.
Bialik’s column also points out that very complex questions will lead to more people choosing “Don’t know.” Political polls are typically done by telephone, which makes paramount the need for simple phrasing.
Of course, the rubber meets the road on election day. Which polling method better predicts the election? Rasmussen in a recent radio interview stated that after the election, his staff does a full debrief to see what they got right and what they got wrong. This will guide them in refining their polling procedures and in their statistical adjustment models. The key challenge is identifying those likely to vote.
In our organizational surveys we too should debrief after every survey to see what we have learned about the surveying process to refine future procedures. Even the pollster’s challenge of identifying those likely to vote has parallels for us. We have ongoing debates about what question(s) best measure the attitude customers feel toward companies, e.g., the Net Promoter Score®. Ours is a different domain but the same challenge.
In an earlier interview in the WSJ, Scott Rasmussen suggests a real flaw in many pollsters’ approach. Most pollsters fall into what Rasmussen would call the “Political Class” whom he feels view the world differently than the “Mainstream Public.” He differentiates the two groups through a series of questions. Pollsters tend to live in the power centers of the country and are disconnected from the Mainstream Public.
When pollsters write questions, they apply their world view, which may not be shared by most respondents. The response options offered in the survey will reflect the pollsters’ biases, and many respondents won’t know how to respond since none of the options reflect how they feel, even if they do understand the question. Thus, those who do provide a response are more likely to be ones who share the pollsters’ world view. We see this vibrantly in the non-scientific polls on advocacy media, but it is also present in the “scientific polls.”
When your organization designs its surveys, do you leave your safe confines and actually talk to those in your group of interest to find out what’s of concern to them or are you certain you know how they view their relationship with your organization? Is that confidence really justified? I see this practice frequently in surveys where the mental frame of the survey designer doesn’t align with the mental frame of the respondent group.
A Bloomberg National Poll, released on October 12, 2012 — but no longer available online, displays some other common survey mistakes, as well as some good practices. First, they open with the summary attitudinal question of right direction vs. wrong track. As it is the first question, the response is unbiased except by whatever opening the interviewer used. But I’ve always been intrigued by the wording. “In general, do you think things in the nation are headed in the right direction, or have they gotten off on the wrong track?” (Emphasis in original.)
Why isn’t the choice “right direction” versus “wrong direction” or “right track” versus “wrong track”? The wording makes it sound like an unintentional accident that we “got off on the wrong track” rather than the leaders’ choices that have taken us in the wrong direction. I can’t speak to the impact of the uneven wording, but I am sure it has an impact. Results reported on October 12: Right Direction = 31%, Wrong Track = 64%.
Also in the Bloomberg poll, questions that present a list of options, such as what are the most critical issues, are rotated in the order presented to responses to eliminate any bias toward choosing the first option. This is a practice that makes sense in all surveys, but it requires a fairly sophisticated survey tool.
While the poll had many interesting practices to analyze, probably most telling was the question about what changes to taxation and government programs should be adopted to reduce the deficit. When you look at the answers, you find an “I want my cake and eat it too” message. We want low taxes, except for the rich who should be taxed to pay for all the entitlements and programs we like. What a surprise!
If it was only that simple, but the fact is that all public policy decisions involve trade-offs. Binary choice and interval-rating questions for each item don’t force respondents to consider the range of trade-offs. Fixed-sum questions force consideration of trade-offs, and conjoint surveys also test trade-offs better. However, those approaches are very difficult to deliver through telephone surveys. So, we see the impact of the administration technique used upon the validity and value of the information garnered.
In our organizational surveys, we are also confronted with this challenge. The administration method constrains what we can do in a survey. But if your research objectives are to measure the trade-offs our “constituents” consider, then we have to use a method that provides valid results. Unless of course, the results that don’t force consideration of the trade-offs are the results you really want to argue some position.
Finally, we see another shortcoming in play in either the administration or analysis. According to the poll, Democrats will retain their majority in Congress, 42% to 40%. Mind you, the question was presented immediately after a question on the positive elements of the health care law, so a sequencing effect could well have been in play. I am writing this on October 15. Their opening “right direction” question makes one question the results of this generic ballot question, and every other poll shows Republicans leading the generic ballot by high single digits or more. Why does this one poll show the opposite?
Maybe they’re right, but it could also be due to very elements of sampling bias. Bloomberg polled 721 “likely voters,” giving it a margin of error or 3.7%, according to the authors. They contacted randomly selected landline and cell phone telephone numbers. The poll says they weighted the scores by gender and race to reflect the recent Census data, which means we can assume they did not weight the scores to reflect political background, which most polls do. They also do not indicate the randomly selected calls were made to insure a geographic spread nor that any weighting done to reflect geographic distinctions. We also do not know how they assessed those 721 people were likely to vote.
Perhaps these administrative and analytical shortcomings explain the difference that this poll shows in that summary question versus other contemporaneous polls. We, too, in our surveys have to be aware of biases we introduce into our data set from question sequencing, word choices, and administrative methods. We may not be predicting elections, but we may be basing significant organizational decisions upon the results.