Summary: When deciding on an online web survey tool what are the key features to consider and how should you go about the selection process?

This is a continuation of our article about Online Survey Tools, which addressed the Benefits of Survey Software Tools and the Dangers of Blind Reliance on Web Survey Tools.  And see our related article on the impact of mobile surveys.

~ ~ ~

Types of Online Survey Software Tools

Two major flavors of online survey tools exist:

  • PC survey software. With these tools you buy desktop or server-based software that resides on your computers.  The system will generate an HTML web form that you then post to a website. When a respondent submits a completed survey, typically the data arrives to a designated email box. The application will process these emails and load the results in a database.

The key advantages of this approach are:

  1. One-time purchase cost – though you may need to buy upgrades and support
  2. Working on your own computer or within your company’s network means you don’t have the lag time with cloud based apps;
  3. Security of having the webform residing within your organization’s firewall.
Tips for a Successful Survey

Request Chapter 1 of the new edition of our Survey Guidebook for key points in a more effective survey program.

Fill out my online form.
  • Hosted survey software. With this approach, you rent use of a cloud-based application. SurveyMonkey is the best known of these tools, but there are literally hundreds of similar products. Comparison shopping is advised. You create a survey interactively on the web site and the HTML survey form is posted on the hosting company’s website.

The key advantages of this approach are

  1. No large up-front purchase cost and easier budgeting
  2. Hosted application handles all the IT issues — no need to know about CGI scripting.
  3. Upgrades to the survey functionality occur continuously (which can actually be a disadvantage if you’re about to launch a survey)
  4. Quick implementation — just sign up and get surveying.

The hosted approach is simpler, but the issue of where the survey webform resides could be the deal breaker.

Key Features of a Web Survey Tool

The potential feature list for survey tools is quite extensive. Many may seem unimportant to you, but they may be important to you — and you won’t know it until you create a survey and find out the survey tool you selected doesn’t support certain features. Or they may become more important as you become more sophisticated in your survey research practices. Most all the survey tools will support the basic features.

  • Comprehensiveness of Survey Question Types. Look at the survey question types supported. (Note: the survey software tools do use different names for these question types.) All will support multiple choice multiple choice, interval rating, and open-ended comment boxes. But are there limits on the number of points on interval scales? How large a text buffer can you have for comment boxes?

You may also want to use semantic differential, forced ranking, or fixed sum survey question types. Are these available? Does the tool provide a running sum as the respondent completes the fixed sum question? How are forced ranking questions handled from the respondent’s point of view?

  • Comprehensiveness of Design Features. Can you customize the formatting of the output that the respondent sees? Shading of alternate questions is a nice feature. Can you add a logo and custom HTML? Can you require an answer for selected questions only? Does the vendor force you to have their “ad” at the bottom of the screen?

If you’re setting up a matrix or table for a series of interval rating questions, can you control the width of the table with the radio button response options?  If you are putting anchors — those words that describe the scale — only over the endpoints of the scale, can you control the distance between response options?  This is vital to maintaining some semblance of interval properties to the question.

Conditional branching is a very useful feature to pass respondents by a series of questions that are not applicable.  Is this supported? Can you loop through a series of questions for multiple occurrences of some event, for example, feedback on a number of different classes taken? Is text piping supported in the looping?

Frequently, we want to branch based upon the response a person gives. For example, for a low score we may want to branch to an open-ended comment box. Is this supported?

Progress indicators are a nice feature as is the option for respondents to postpone completing the survey should they get interrupted — saving the previous responses of course.  How simple is it to add section headers or other descriptive text that is not part of a question?  Upon completion of the survey, can you redirect the respondent to a specific web page?

Advanced users may want to randomize the question order to control for question sequencing effects, randomize the order of the response set within a question, reverse code selected questions, or control the number of responses in a multiple choice question a respondent can select.

  • Ease of Use. How easy is it to create a question or apply a scale to a series of questions? Can you create your own library of questions or scale types that you can readily call upon for future surveys?
  • Respondent Friendliness. Check for the experience of the respondent. If the respondent makes an error in entering an answer — a particular problem with forced ranking and fixed sum question types — what are the error messages? I’ve seen some tools that scold the respondent. Not good for response rates! Does the system support halo text or some other means of providing further explanation of a survey question?
  • Integration with Transactional System. If you’re performing event or transactional surveys, does the system integrate with your transaction system to provide a seamless transfer of contract information?  Are there administrative controls to control for repeated surveying of an individual?
Survey Sample Size Calculator

Get our Excel-based calculator. It can also be used to gauge statistical accuracy after the survey has been completed.

Fill out my online form.
  • Administration Support. Will the survey tool create a random sample from a list of contacts? A stratified random sample? Will it tell you an approximate sample size you need for a certain level of statistical accuracy? Can you restrict access to the survey only to those to whom you’ve sent an invitation? Can you send out reminders to those who have not completed the survey?

As you can see, the capabilities of survey software tools can be quite extensive – and the above is not even a comprehensive list!  Do your homework and think about your true long term needs.

How to Select aWeb Survey Tool

Finally, how should you go about selecting a web survey tool?

  • Determine Your Needs. This should seem obvious, but it’s not. I’ve worked with companies where senior managers have become enamored with SurveyMonkey, QuestionPro or some other product because they had taken a survey as a respondent from that tool.  It shows great marketing! But when I ask them if they know if Survey Tool A will meet their requirements for their survey, the tone of the response told me to back off. The basic tools, including SurveyMonkey, have limitations.

How to determine your needs?

  1. Develop your survey questionnaire in a rigorous fashion
  2. Discuss the role of the survey program in your organization long term
  3. Be sure that the survey captures the true concerns of the respondent group and not what management thinks are the concerns .

The questionnaire, along with various administration requirements, will become your requirements statement.

Conversely, if your organization has already decided on some survey tool platform, then the capabilities of that tool become constraints on your questionnaire design.

  • Cost Comparison. Cost comparisons can be difficult since there are various pricing algorithms. Pricing may be driven by the number of:Survey projects in a period of time
  1. Responses to a specific survey
  2. Invitations sent for a survey
  3. Questions in a survey
  4. User accounts with access.

Also, advanced features may drive you to a higher-level package — or customization. Create your expected scenario of how you will use the survey software and then calculate the price for the various tools.

  • Take a Test Drive. All of the tools — or all that you should consider — will allow you to download or use the software in some limited fashion. Check out their ease of use. You’ll be driving this car. Wouldn’t you like to know how comfortable the seats are for a long trip?

~ ~ ~

In summary, online survey software tools can make the job of conducting survey research much easier and less expensive. When selecting a particular tool, you have many factors to consider. The most important factor is your level of knowledge of good survey research practices. Without that, you may well be driving blindly without knowing you’re blind.