Reverse Engineer a Statement of Survey Research Objectives

Summary: A successful survey program starts with a firm foundation. Without a clearly articulated Statement of Research Objectives the survey project is likely to meander with disagreements among the players during the project especially about the questions to create for the survey instrument. Failure is more likely. This article outlines the critical elements that should be part of this first step in a survey project and how to reverse engineer the research statement from a currently used survey questionnaire.

~ ~ ~

Perhaps the most common and important mistake a surveyor (or researcher) can make lies at the very onset of the project: good project planning. A customer feedback program is composed of a series of projects, and good project management skills should be exerted. I won’t get into budgeting and scheduling here, but instead focus on what should be the first section of any project plan: a statement of project objectives.

What’s in a good statement of research objectives? I like to use the Who, What, When, Where, How, and Why metaphor, sometimes called the 5 Ws (and an H) in the journalism profession.

Who:

Is our group — and subgroups — of interest?

What:

Are we trying to understand about those groups’ views?
Will we do with the results?
That is, Why are we doing the survey?

When:

Are we going to send out the survey invitations and reminders?
Are we closing the administration period and start the data analysis?

How (& Where):

Will we develop the questionnaire?
Will we administer the survey? (What method, sampling procedure, incentive…)

The when, how, and where are mostly logistical planning issues. The who (aside from being a bunch of aging rock and rollers) is critical. It makes us think about exactly who is the target audience for our research. That sounds simple on its face, but it can actually be an area of discussion and disagreement on the project team. Is our target population all customers, customers who have purchased from us in the past year, customers who have purchased a certain set of products, customers who have contacted us for service, etc., and who in the customer organization should be completing the survey?

The what and why strike to the heart of the matter.  If you can’t articulate what you’re going to do with the findings, then how can you justify wasting some customers’ time filling out this poorly conceived survey? One of my workshop alumni runs the customer survey program office for a major technology company, and if someone who wants to survey customers can’t answer the what and why, then she won’t execute the survey. Period. A good discussion with the sponsors of the survey research may bring out the hidden agenda for the goals of the research. As an outsider doing a project for a client, I always need to know those agenda so that I can properly position the actual findings, but even as an insider doing a survey for your own company, those agenda need to be known.

Recognize that I’m not saying that this statement of research objectives is locked in and cannot be changed. It can be and very well will be as you move through the survey project. But with a well-thought-out statement of research objectives you’ll make changes having thought through the implications.

What happens if you skip this step and just start writing survey questions? The survey instrument may wander and meander across multiple topics that likely need to be answered by different respondents. I see this all too frequently in surveys brought to my workshops by attendees. Frequently, it’s because multiple players have been pushing their agenda for the surveys, wordsmithing the survey questions to achieve the results they want. There’s an old adage that a camel is a horse built by committee. I’ve seen many “camel surveys”.

It is also likely that when you get the data from the survey and then — finally — start thinking about what research questions you want to answer, you will find you do not have the data you need or the data aren’t structured as you need it for some analysis.

When someone asks me to review a survey instrument, I like to “reverse engineer” the statement of research objectives. That is, I look at the survey and determine what I see as the implicit research objectives. It’s a good exercise that you can practice with any survey you encounter.

Let’s look at an example. The nearby survey was one I got from the company that installed a new heating and cooling (HVAC) system in my 200-year-old house. What are the firm’s research objectives? What does the sponsor of this survey seem to think drives customer satisfaction with the purchase, design, and installation of a heating and cooling system? Now, put yourself in the position of the person who has hired a company to do the work. (It doesn’t have to be an HVAC system, but any major work done in your home or business.) What is it you’re buying? What are the products and services you have bought? Let me add that this survey was left in my house about half way through the installation process, which took about a week. That’s the where and how.

Even a cursory review of the questionnaire reveals that the company thinks the behavior of their technicians in my home was all that really mattered. The techs could be great, but what if the system doesn’t work? Where are the questions about the design of the system and whether it works as promised? Guess who would be measured by those questions? Exactly, the head of the company who performed the design work. Will the owner of the firm learn about all the things that truly drive customer satisfaction? I don’t think so. (There are several specific shortcomings with the wording of questions as well.)

You might argue that the survey was to capture my feedback about what was happening right then — my interaction with this swarm of techs in my house. This isn’t the proper time to ask about the quality of the system design; I needed to experience it first. You’re right; I would agree with you. However, I did not receive any follow up research after the system had been functioning for a week or a month to find out how I liked the performance of the system.

Notice I keep using the term, “research objectives.” You might have thought, “Hey, I’m doing a customer survey. I’m not doing research.” Yes, you are. Legitimate research needs to be done following a certain level of rigor to truly  answer the business questions. The above discussion brings out this point. To fully understand the customer’s satisfaction with the HVAC system you would need a multi-pronged research approach. One survey about the techs in my house cannot answer all the relevant research questions. A program of research, which may include surveys and/or interviews of different people at different times, may be needed.

If I may make a bad pun on a common business phrase, you need to think outside the survey research box.