How to Select a Web Survey Tool

This is a continuation of our article about Online Survey Tools. The previous page addresses the Benefits of Survey Software Tools, and the Dangers of Blind Reliance on Web Survey Tools.

Types of Survey Software Tools

Two major versions of online survey tools exist:

PC survey software. With these tools you buy desktop or server-based software that resides on your computers.  The system will generate an HTML web form that you then post to a website. When a respondent submits a completed survey, typically the data arrives to a designated email box. The application will process these emails and load the results in a database.

The key advantages of this approach are 1) you have a one-time purchase cost – though you may need to buy upgrades and support; 2) you’re working on your own computer or within your company’s network so the software will function faster; 3) you can post the survey to your own company’s web server.  Perseus is among the oldest and best known of these applications.

Hosted survey software. With this approach, you rent use of an application at the vendor’s web site. SurveyMonkey is the best known of these tools, but there are literally hundreds of similar products. Comparison shopping is advised. You create a survey interactively on the web site and the HTML survey form is posted on the hosting company’s website.

The key advantages of this approach are 1) no large up-front purchase cost and easier budgeting; 2) the hosted application handles all the IT issues, except for your having a working browser; 3) upgrades to the survey functionality occur continuously; 4) quick implementation — just sign up and get surveying. With the PC-based tools, you become responsible for posting the form and its associated scripting files. This can be troublesome.

The hosted approach is simpler, but in the very long run, it may cost more. Though, when you include all the costs associated with the PC-based survey tool approach, hosted surveys could be less expensive. Some companies for security and privacy reasons may require the survey to run on its own servers. In this case, the hosted applications lose out.

Key Features of Online Survey Tools

The potential feature list for survey tools is quite extensive. Many of these may be unimportant to you. But many may be important to you — and you won’t know it until you create a survey and find out the survey tool you selected doesn’t support certain features. Or they may become more important as you become more sophisticated in your survey practices. All the survey tools will support the basic features.

Comprehensiveness of Survey Question Types. Look at the survey question types supported. (Note: the survey software tools may use different names for these question types.) All will support multiple choice, interval rating, and open-ended comment boxes. But are there limits on the number of points on interval scales? How large a text buffer can you have for comment boxes?

You may also want to use semantic differential, forced ranking, or fixed sum survey question types. Are these available? Does the tool provide a running sum as the respondent completes the fixed sum question?

Advanced users may want to randomize the question order to control for question sequencing interaction, randomize the order of the answers within a question, reverse code selected questions, or control the number of responses in a multiple choice question a respondent can select.

Design Flexibility. Can you customize the formatting of the output that the respondent sees? Shading of alternate questions is a nice feature. Can you add a logo and custom HTML? Can you require an answer for selected questions only? Does the vendor force you to have their “ad” at the bottom of the screen?

If you’re setting up a matrix or table for a series of interval rating questions, can you control the width of the table with the radio button response options?  If you are putting anchors — those words that describe the scale — only over the endpoints of the scale, can you control the distance between response options?  This is vital to maintaining some semblance of interval properties to the question.

Conditional branching is a very useful feature to pass respondents by a series of questions that are not applicable.  Is this supported? Can you loop through a series of questions for multiple occurrences of some event, for example, feedback on a number of different classes taken? Is text piping supported in the looping?

Frequently, we want to branch based upon the response a person gives. For example, for a low score we may want to branch to an open-ended comment box. Is this supported?

Progress indicators are a nice feature as is the option for respondents to postpone completing the survey should they get interrupted — saving the previous responses of course.  How simple is it to add section headers or other descriptive text that is not part of a question?  Upon completion of the survey, can you redirect the respondent to a specific web page?

Ease of Use. How easy is it to create a question or apply a scale to a series of questions? Can you create your own library of questions or scale types that you can readily call upon for future surveys?

Respondent Friendliness. Check for the experience of the respondent. If the respondent makes an error in entering an answer — a particular problem with forced ranking and fixed sum question types — what are the error messages? I’ve seen some tools that scold the respondent. Not good for response rates! Does the system support halo text or some other means of providing further explanation of a survey question?

Integration with Transactional System. If you’re performing event or transactional surveys, does the system integrate with your transaction system to provide a seamless transfer of contract information?  Are there administrative controls to control for repeated surveying of an individual?

Administration Support. Will the survey tool create a random sample from a list of contacts? A stratified random sample? Will it tell you an approximate sample size you need for a certain level of statistical accuracy? Can you restrict access to the survey only to those to whom you’ve sent an invitation? Can you send out reminders to those who have not completed the survey?

As you can see, the capabilities of survey software tools can be quite extensive – and the above is not even a comprehensive list!  Do your homework and think about your true needs.

The Decision-Making Process for Selecting an Internet Survey Software Tool

Finally, how should you go about selecting a web survey tool?

Determine Your Needs. This should seem obvious, but it’s not. I’ve worked with companies where senior managers have become enamored with SurveyMonkey, Zoomerang, or some other product because they had experienced a survey as a respondent from some tool.  It shows great marketing! But when I ask them if they know if SurveyMonkey will meet their requirements for their survey, the tone of the response told me to back off. The basic tools, including SurveyMonkey, have limitations.

How to determine your needs? Develop your survey questionnaire in a rigorous fashion, discussing the role of the survey in your organizational process, and being sure that the survey properly captures the true concerns of the respondent group and not what management thinks are the concerns of the respondent group. The questionnaire along with various administration requirements will become your requirements statement.

Cost Comparison. Cost comparisons can be difficult since each survey software company may use its own pricing algorithm. Pricing may be driven by the number of survey projects, the number of responses to a specific survey, the number of invitations sent for a survey, or the number of questions in a survey. Also, the features you want may drive you to a higher-level package or customization. Create your expected scenario of how you will use the survey software and look at the price.

Take a Test Drive. All of the tools — or all that you should consider — will allow you to download or use the software. Some will let you execute a survey project, limiting the number of respondents’ data in the database or some other limitation. Definitely take a test drive. Two products may compare well on price and features, but check out their ease of use. You’ll be driving this car. Wouldn’t you like to know how comfortable the seats are for a long trip?

In summary, online survey software tools can make the job of conducting survey research much easier and less expensive. When selecting a particular tool, you have many factors to consider. The most important factor is your level of knowledge of good survey research practices. Without that, you may well be driving blindly without knowing you’re blind. (I’ll give a final plug for my Survey Design and Survey Data Analysis Workshops.)