Caveat Survey Dolor: “Show Me the Questionnaire”

Summary: “Show me the car fax” is one those lines from a TV ad that frankly gets annoying after a while. My version of it is “Show me the survey instrument.” I annoy some organizations when I ask to see the survey instrument before I’ll even contemplate the findings derived from the survey. To most people, examining the instrument would seem an unnecessary annoyance. In this article I will show you why you should always go to the source and verify the validity of the data generated by the survey instrument.

In fact, I had a long string of emails with a local-to-me company that published some survey findings that got national attention. I wanted to see how they presented certain terminology to respondents that I suspected would bias how people took the survey. They declined to show me the instrument with a very lame excuse. I even told them I would help them with future survey projects in exchange for the publicity. But I guess their reasoning is: why let sound research get in the way of a good headline.

~ ~ ~

We’re in the political silly season in this summer of 2012 with polls coming out almost daily. Should you believe the summaries presented by newscasters or newspaper writers are true to the data collected? Should you believe the data collected are accurate? We see major differences across polls, so these are legitimate questions. While we can’t do a full audit of the polling processes, we can look, perhaps, at the survey instruments used.

In this article I examine a poll conducted by the New York Times, CBS News, and Quinnipiac University. Let me state right up front that I am pointing out the shortcomings of a survey done by two liberal news outlets. (Yes, my dear Pollyanna, the New York Times has a liberal bias. Shocking, I know.) I suspect if I dug into a conservative news outlet’s survey, I would find questionable distortions, though in ones I have examined, I have not seen validity issues with questions like the ones below.

On August 1 and 8, 2012 the New York Times published polls of six battleground states for the November election: Florida, Ohio, Pennsylvania, Virginia, Colorado, and Wisconsin. To their credit, the paper does provide access to the actual survey script used for the telephone survey and summary results by question. Most of the major polls make their survey language available. Those that don’t are probably hiding sloppy instrument designs — or worse.

The survey scripts appear identical for questions posed for the national level. However, they did change their definition of relevant population or sampling frame from the first batch of surveys to the second batch. For Florida, Ohio, and Pennsylvania they only report results for “likely voters;” whereas, the Virginia, Colorado, and Wisconsin surveys reported results for some questions that included registered but not likely voters and some that included non-registered respondents. See why it can be hard to do comparisons across surveys — and these surveys were done by the same organizations!

Much has been made of the fact that these pollsters oversampled democrats. (That is, the self-reported affiliation of respondents as republicans, democrats, and independents had democrats in greater proportions than in the registered voter base.) We can also look at the sequencing of questions and ask whether it creates a predisposition to answer subsequent questions a certain way. But here I want to focus on two questions that clearly show how the pollsters’ world views affected the questions they asked.

Question 19 reads as follows:

19. From what you have read or heard, does Mitt Romney have the right kind of business experience to get the economy creating jobs again or is Romney’s kind of business experience too focused on making profits?

The pollsters present the false dichotomy of business experience as focusing on either jobs or profits — a favorite theme of some. Businesses do not choose either jobs or profits. Jobs result from profitably run businesses. The question displays an incredible lack of understanding of how businesses function — or perhaps it was purposeful. In a similar vein, we have heard that corporations are sitting on a pile of cash and are “refusing” to hire people.

~ ~ ~

The next question in the survey is:

20. Which comes closest to your view of Barack Obama’s economic policies:

   1. They are improving the economy now, and will probably continue to do so,
2. They have not improved the economy yet, but will if given more time, OR
3. They are not improving the economy and probably never will.

Notice what’s missing? 1% of respondents in Florida and Colorado did. The pollsters didn’t offer choice 4. “Obama’s economic policies are hurting the economy.” 1% in Florida and Colorado apparently took the initiative to voice that option, and to the pollsters’ credit they captured it.

Isn’t it legitimate for some people to believe that the president’s economic policies are hurting the economy? Apparently, not to these pollsters. They only think that Obama’s economic policies can help the economy or be benign. Yet, rational people can certainly feel that regulations, promised tax policies, and the uncertainty of Obama’s temporary fiscal and economic policies are hurting the economy.

The pollsters only provided neutral to positive response options with no negative options. A basic requirement of a well-designed question is that it provides the respondent a reasonably balanced set of response options. This is not a mistake a seasoned survey designer would make.

Another problem with the question is that “economic policies” covers a very broad area that is open to multiple interpretations by respondents — and manipulation by the writer of the findings. The pollsters would have generated more valuable, interesting, and valid data if they had structured their question as:

Consider each of the following areas of Barack Obama’s economic policies. What impact do you feel each has had upon the economy now and in the future? Greatly helped, Helped somewhat, No impact yet but will, No impact now or in the future, Hurt somewhat, Greatly hurt.

— Policy 1
— Policy 2, etc.

~ ~ ~

Is the purpose of the polls performed by major news organizations

  1. to understand the feelings of the populace
  2. to drive those opinions or
  3. to generate data that certain, preferred candidates can use to their advantage in the campaign?

Looking at these two questions — as well as phrasing in a July 19 poll — it’s hard to say the former, which should be the goal of responsible, unbiased researchers.

In summary, these two questions show that these pollsters bring bias to their polling. Always look at the survey instrument to sense if there’s bias in the wording and fairness in interpreting the data before accepting the findings. This caveat applies to political polls as well as organizational surveys.

~ ~ ~

So why does a business hire (or layoff) someone?

A business hires someone if they feel the long-run value delivered to the organization will exceed the fully loaded cost of employing the person. It’s really that fundamental. While it’s unlikely a company can measure the direct value to the bottom line of a single employee or even a group — except perhaps for the sales force — that is what companies decide in the budgeting process. If the cost of employment exceeds the benefit, bottom line profit decreases. Why would a company hire people if the value they bring doesn’t exceed their cost?

The counterargument may be made that companies fire people to increase profits. It is true that laying off people may increase bottom-line profit, at least in the short run. (Google, not a politically conservative company at all, laid off many at its Motorola Mobile acquisition.) If the people being laid off had costs that exceeded their benefit, yes, profit will increase. But keeping people on the payroll just for the sake of “employment” can hurt those who deliver positive value to the company.

I worked for Digital Equipment Corporation in the 1980s. The company was on top of the world in 1987 when it employed more than 120,000 people worldwide. When senior management missed the changes in the competitive market, the company still resisted layoffs until the financial health of the company was threatened. In a decade Digital no longer existed with tens of thousands of job losses that greatly affected the Boston technology beltway for years to come.

More recently, look at the US car companies that employed people who literally did nothing in their “job banks.” Did that lack of focus on profit advance the bankruptcies? Most certainly.

No one’s business experience is focused on creating jobs. Entrepreneurs and business people want to build sustainable businesses by creating products and services people chose to buy. Jobs are a by-product, albeit a very important by-product, of a successful, profitable company.

Go to a thousand company websites and read their mission statements, preferably small growing companies that may not yet be profitable but are our job-creation engines. How many companies say their primary mission is to create jobs? I doubt you’ll find one.

Here’s the empirical proof that we all know. Ever heard of an established, unprofitable company that is hiring lots of people?

I recognize that was a bit of a rant, but as a business school professor this idea of “jobs versus profits” needs to be challenged for the misrepresentation that it is, and it is disturbing to find it in a survey done by a professional organization.