Summary: Sometimes even well-honed survey questions don’t measure the underlying attribute that the survey designers want. A recent poll by the Boston Globe about Boston’s Mayor Menino’s popularity shows a clear example of this. The question attempted to objectively measure how many people the Mayor has met, but the results — when thought through to the logical conclusions — show that the question was really measuring something else. If a question doesn’t work, it’s time to rethink the question.

~ ~ ~

Sometimes reported survey results really show the errors inherent in poorly conceived or executed surveys. A past example of a poorly designed survey (survey no longer available online unfortunately) was a survey done by an online survey vendor that conducted an internet-based survey and found that most people want to be surveyed online rather than by phone, etc. It’s a self-fulfilled prophesy. Of course people who completed a web-based survey prefer to be surveyed via the web. Yet, many took the findings as gospel.

But this week (late March 2013) the Boston Globe reported a poll about Mayor Menino who is finishing his fifth — yes, 5th — 4-year term. He’s been a good mayor, and his malaprops are absolutely legendary — or should I say he had “ionic” moments on the “jumbletron.” He’s decided not to run for a sixth term. The press will miss him. Here are more famous quotes, some of which you have to be a Bostonian to fully appreciate.

The poll found his popularity was still sky high at 74% and part of the reason may be that “almost half said they had met Menino personally, an extraordinary feat for a big city mayor.” This was “down eight points from four years ago” according to the Globe article.

Boston is a city of 625,000 of which 83% are over 18, the sampling frame for the poll. That’s 520,000 people, indicating that Menino has “met” 260,000 people — if we believe the poll. (To keep NRN — nice round numbers — I rounded down the population a bit but rounded up the percentage met from 49% to 50%.) Remember, this would be 260,000 unique people he has met; meeting someone twice doesn’t count as 2.

Let’s assume that people interpreted the question as having met Menino while he was mayor in the past 20 years. That would mean he would “have met” 13,000 different people per year or 250 per week or 36 per day — in a 7 day week. (Mayors don’t get days off!) In a 60-hour work week that would be 4 people per hour. That is quite an “extraordinary feat.” According to the poll taken four years ago, he had met 356 people per week! (More on that curious fact below…)

Part of the Mayor’s hallmark has been his touch with real, everyday people. He spends considerable time in the various Boston neighborhoods, which is why he is so popular, never having received less than 57% of the vote in mayoral elections.

But… even if we include the 50 years of Menino’s life before he became Mayor, the numbers are still too fantastic.  (Over 70 years he would have had to meet 71 people per week to have met 260,000 people.) The numbers don’t pass face validity. What’s in the mix?

  • The wording of the question. “Have you, personally, ever met Mayor Menino?” is how the question was worded. Note it’s not “met with” which implies a real meeting with discussion on some point. To their credit, the survey designers did include the work “personally” to try to exclude people who might think they “met” the Mayor by seeing him on TV or by being in a crowd at some rally with 1,000 other people where the Mayor spoke. That phrasing implies to me that you’ve shaken his hand and at least exchanged a few words, mumbled though they might be. Clearly, the attempts at precision wording failed. Undoubtedly, some people have a very loose definition of “met.”It would have been interesting to have asked a follow up question about where the respondent met the Mayor to validate the previous response. If the person said, “I don’t recall,” that would cast doubt about the validity of the response. I think we can all recall when we’ve met someone famous. So, not remember would lessen the credibility of the respondent saying he had met Menino. (I can recall clearly my encounters with former Senators Brown, Kerry, and Brooke and sports figures Reggie Jackson, Bobby Orr, Jerry Remy, and Carl Yastrzemski. And some of those encounters were many decades ago.)
  • Response bias. Don’t you want to say to some interviewer that you have met the Mayor? It sounds good. It makes you feel a bit important. It’s a good thing. Response bias is caused by predispositions of the respondent that they bring to the survey experience. My guess is this is what’s in play. In fact, the added word “personally” may have accentuated the response bias rather than increased the question’s accuracy.Within the response bias domain are several specific effects that could be the explanation:
    • Acquiescence where the respondent says what they think the interviewer wants to hear.
    • Conformity where the answer makes the respondent part of the crowd.
    • Social desirability where your answer is the “right” answer following societal norms.
  • Self-selection bias. To some extent people choose whether to be surveyed. While telephone surveys, which this was, suffer less from this self-selection bias, I suspect it is in play. If you were asked to participate in a poll about your mayor, wouldn’t you be more inclined to be polled if you had met the person?

Here’s how I know the question has validity issues. The 49% percent who met the Mayor was down 8% from four years ago, far greater than the 4.7% margin of error for the survey. Note that the question said have you “ever met” Menino, not have you met the Mayor “in the past four years.” If it was a 4-year window, take those absurd numbers above and multiple by 5!!

Doesn’t “ever met” mean the number should only go up? If you asked me how many students I have “ever taught,” that number goes up every year.

How could 8% fewer now say they’ve met Menino? That’s 41,600 people who had met Menino as of 4 years ago but who now say they haven’t. Apparently, people who have met Menino move out of the city — or die — in large numbers!

My point is that this question is really measuring a subjective feeling about the likeability of the Mayor, and my educated guess is that the surveyors at the Survey Center at the University of New Hampshire know this. I suspect there’s a strong correlation between this number and other measures of sentiment about the Mayor. But the reporters who wrote up the story about the poll purport it to be an objective measurement of the visibility of the Mayor.

Lessons from this survey…

  • Know what information you are getting from your surveys.
  • Especially consider what underlying attribute or construct you are truly measuring.
  • If some results do not look right, be inquisitive. Challenge the results with common sense. Find out why. Examine the survey practices.
  • If a question doesn’t work after a few tries at different wording, resign yourself to the reality of the surveying situation and take a different approach to measuring the construct.
  • Whoever designs the survey questionnaire should also do the data analysis to keep a tight feedback loop on survey instrument design issues. The larger the surveying operation, the more likely that the instrument designer is divorced from the data analysis, which is not good.
  • When reporting the results, don’t misrepresent what the data are saying.
  • When reporting the results, actually think about the data. (Is that too much to ask of a reporter?)
  • And recognize there’s a lot of imprecision in surveys.