Knowing what the client wants is the key factor to success in any type of business. News media, government agencies and political candidates need to know what the public thinks. Associations need to know what their members want. Large companies need to measure the attitudes of their employees. The best way to find this information is to conduct a survey.

Conducting surveys can be done very simply, or it can be very complicated, depending on how much you want to ask on the survey and the number of people to whom it is administered. Here some information for small/medium scale surveys. These are certainly not a strict prescription and they should be clearly adapted to your needs, especially if you go to larger/multiple target groups.

This article provides an overview of the key points related to surveys. It is divided into 12 web pages with a table of content...


What are Surveys?

A survey is a way of collecting information that you hope represents the views of the whole community or group in which you are interested.

There are three main ways of going about this:

  • Case study surveys: to collect information from a part of a group or community, without trying to choose them for overall representation of the larger population. Several of these informaiton may be needed before you get a sense of how the larger community would respond to your survey. Case study surveys only provide specific information about the community studied.
  • Sampled surveys: ask a sample portion of a group to answer your questions. If done well, the results for the sample will reflect the results you would have gotten by surveying the entire group. For the sample to accurately represent the larger group, it must be carefully chosen.
  • Census surveys, in which you give your survey questionnaire to every member of the population you want to learn about. This will give you the most accurate information about the group, but it may not be very practical for large groups. A census is best done with smaller groups -- all of the clients of a particular agency, for example, as opposed to all of the citizens of a city.

Surveys are usually written, although sometimes the surveyor reads the questions aloud and writes down the answers for another person; they can be distributed by mail, fax, e-mail, through a web page, or the questions can be asked over the phone or in person (in the street, on a fair, at a seminar...).

Surveys collect information in as uniform a manner as possible -- asking each respondent the same questions in the same way so as to insure that the answers are most influenced by the respondents' experiences, not due to how the interviewer words the questions.


Why conducting a survey?

You can collect information about the behaviors, needs, and opinions using surveys.
Surveys can be used to find out attitudes and reactions, to measure client satisfaction, to gauge opinions about various issues, and to add credibility to your research.
Surveys are a primary source of information -- that is, you directly ask someone for a response to a question, rather than using any secondary sources like written records.

You can use surveys to measure ideas or opinions about community issues related to your initiative.
For example, you may want to know how many people use your services, what users think about your services, what new users expect from your services, and whether users are satisfied with what you provide.

Deciding whether to conduct a survey

There are advantages in doing surveys, but you should consider whether a survey will be the best way of obtaining the information you need. 
Even though surveys are a useful method of gathering information, they are not the only way. 
You need to decide whether a survey will produce the information you need. 
The information you need may be obtained through other means, such as informal unstructured conversation that takes place in the course of another activity; census figures; meeting with people in the community; interviews; or observation.
Don't forget to clearly identify the goal and purpose of your survey (see page 4).


When conducting a survey?

A survey may be your best choice when:

  • You need a quick and efficient way of getting information
  • You need to reach a large number of people
  • You need statistically valid information about a large number of people
  • The information you need isn't readily available through other means

Written surveys: Pros and Cons

Advantages of a written survey  Disadvantages of a written survey
  • Large numbers of people can give their input
  • Low cost
  • People can respond at their convenience
  • Avoids interviewer bias
  • Provides a written record
  • Easy to list or tabulate responses
  • Wide range of respondents
  • No training needed as with interviewing
  • Often has low return rate
  • Limited alternative expression of respondent's reaction
  • Depends on the selected sample
  • May not truly represent of the whole group
  • Respondent may skip sections (depending on the technology) 

If you have decided that what you need is a large-scale, formal survey, make sure you will have the resources and capacity to exploit the informaiton and then the results you get. Hiring someone to do it for you or working with local colleagues or a nearby university may be your best bet. If you're going to do it on your own, keep in mind that some people you present your report to may not give much credit to a survey you did on your own.


Get prepared with your survey

Identify goals and purpose of the survey.

If you have decided to do a survey, you must be sure exactly why you're doing it. The first step in any survey is deciding what you want to learn.
What questions do you want to answer? Is it to get a general idea of the demographics of your area? To find out what people think about a particular issue or idea? Or is there another reason you're considering a survey?

The goals of the project determine whom you will survey and what you will ask them. If your goals are unclear, the results will probably be unclear. Some typical goals include learning more about:

  • The potential market for a new product or service
  • Ratings of current products or services
  • Employee attitudes
  • Customer/patient satisfaction levels
  • Reader/viewer/listener opinions
  • Association member opinions
  • Opinions about political candidates or issues
  • Corporate images

These are only general areas. The more specific you can make your goals, the easier it will be to get usable answers.

In any case, you will need to keep the purpose of the survey in mind throughout the process, as it will influence the choice of questions, the survey population, and even the way the survey is delivered (e.g., a computer-savvy population can be surveyed over the Internet; a population that is largely illiterate shouldn't be asked to take a written survey, and so forth).

Identify whom you will survey

The next step is finding out who has the answers to your question or questions. In other words, it's time for you to determine your audience -- the people who can best answer the questions your initiative needs to ask. Who will you survey? Is it the general public? The current program beneficiaries? People in a specific neighborhood or segment of the community? Potential members?

There are two main components in determining whom you will interview.

  • The first is deciding what kind of people to interview. Researchers often call this group the target population. If you conduct an employee attitude survey or an association membership survey, the population is obvious. If you are trying to determine the likely success of a product, the target population may be less obvious. Correctly determining the target population is critical. If you do not interview the right kinds of people, you will not successfully meet your goals.
  • The next thing to decide is how many people you need to interview. Statisticians know that a small, representative sample will reflect the group from which it is drawn. The larger the sample, the more precisely it reflects the target group. However, the rate of improvement in the precision decreases as your sample size increases. For example, to increase a sample from 250 to 1,000 only doubles the precision. You must make a decision about your sample size based on factors such as: time available, budget and necessary degree of precision.

Sampling
Almost all surveys rely on sampling, i.e.: identifying a section of your population that satisfies the characteristics you try to survey, rather than doing a census.
For a truly representative sample, you must be sure that every member of the group you want to survey has an equal chance of being in the sample, and/or you must have a fairly large sample. It's important to make sure that the sample size you choose is adequate and not excessively large or small. If too large, it may be impossible to survey everybody effectively and within your budget; if too small, your credibility may suffer. A general rule to keep in mind is that the larger the sample size, the more accurate a reflection of the whole it will be.

You can figure out how big your sample should be by using a sample size calculator, such as:

  • ResearchInfo's Sample Size Calculator allows you to decide whether you want to calculate for 95% or 99% confidence level (the statistical term for the amount of certainty you have about the accuracy of your results).
  • UCLA's Sample Size Calculator from the online statistics textbook is a bit more advanced.

You might also need to give some thought to the design of your sample, especially if you are hoping to get representative responses from two or more groups; you need to come up with separate population counts for each of these groups and then select a sample from each. The samples should be large enough to represent the group it is drawn from, but the sample sizes should be proportional to the groups they represent.

Potential sampling pitfalls
Sampling is a challenge to conducting good surveys, but there are other pitfalls. For example, when people volunteer to respond to a survey, we say they are self-selected. These people may have a special interest in answering your survey, so their answers may not be truly representative of the group you're interested in. There are ways of dealing with self-selected audiences, such as only using a random selection of their surveys when only self-selection is involved. For example, if you get back 300 completed surveys, you might decide to only use every third one in order to randomize the results.

Avoiding a Biased Sample
A biased sample will produce biased results. Totally excluding all bias is almost impossible; however, if you recognize bias exists you can intuitively discount some of the answers. The following list shows some examples of biased samples.

 Example of Biased Sample 
Sample  Probable bias  Reason 
Your Customers  :-) favorable If they were unhappy, they wouldn't be your clients. The important point is to know why they are happy. 
Your Ex-customers  :-( unfavorable  If they were happy, they would still be your clients. The important point is to know why they drop you. 
Phone-in :-$ Extreme view Only people with a strong concern (For or Against) are likely to phone in. In addition, they may phone several time to load the votes/poll.
Daytime :-? Non-working Most of the people at home during the days do not work. Their interview/opinion at that occasion may not reflect the working people.
Internet :-@ Atypical people Not the majority of people do have/are used to internet. These users are representative fo the general population, even more when matching sampling on gender, age... 
Another issue, respondents may complete several forms to bias results (except if the software can prevent that).

The consequences of a source of bias depend on the nature of the survey. For example, a survey for a product aimed at retirees will not be as biased by daytime interviews as will a general public opinion survey. A survey about Internet products can safely ignore people who do not use the Internet.

 Quotas

A Quota is a sample size for a sub-group. It is sometimes useful to establish quotas to ensure that your sample accurately reflects relevant sub-groups in your target population. For example, men and women have somewhat different opinions in many areas. If you want your survey to accurately reflect the general population's opinions, you will want to ensure that the percentage of men and women in your sample reflect their percentages of the general population.

If you are interviewing users of a particular type of product, you probably want to ensure that users of the different current brands are represented in proportions that approximate the current market share. Alternatively, you may want to ensure that you have enough users of each brand to be able to analyze the users of each brand as a separate group. If you are doing telephone or Web page interviewing, The Survey System's optional Sample Management or Internet Module can help you enforce quotas. They let you create automatically enforced quotas and/or monitor your sample during interviewing sessions.


How to collect your survey data?

Will your survey be written or oral? Is there going to be a number where people can call to register their results? Are you going to have a post office box to which completed surveys should be mailed? You need to decide whether it's going to be administered by people known to the audience and whether it will be done in person, by phone, or by mail. Remember that the more personal you make it, the higher the return rate will be. Surveys that are delivered cold have a return rate of only two to three percent, unless they're on a very hot topic for the community you're surveying.

Keep in mind whom you want to survey. Does your public feel more comfortable writing or speaking? Will it be efficient to leave surveys somewhere for people to pick up at their will, or should you do something to make sure they get one? If your survey is to be administered orally, will people feel honored or annoyed about being asked for their opinions?

Personal Interviews
An interview is called personal when the Interviewer asks the questions face-to-face with the Interviewee. Personal interviews can take place in the home, at a shopping mall, on the street, outside a movie theater or polling place, and so on.

 Advantages  Disadvantages
  •  The ability to let the Interviewee see, feel and/or taste a product.
  • The ability to find the target population. For example, you can find people who have seen a film much more easily outside a theater in which it is playing than by calling phone numbers at random.
  • Longer interviews are sometimes tolerated. Particularly with in-home interviews that have been arranged in advance. People may be willing to talk longer face-to-face than to someone on the phone.
  • Personal interviews usually cost more per interview than other methods. This is particularly true of in-home interviews, where travel time is a major factor.
  • Each mall has its own characteristics. It draws its clientele from a specific geographic area surrounding it, and its shop profile also influences the type of client. These characteristics may differ from the target population and create a non-representative sample.

Telephone Surveys 
Surveying by telephone is a popular interviewing method. This is made possible by nearly universal phone coverage in many countries.

Advantages  Disadvantages 
  • People can usually be contacted faster over the telephone than with other methods. If the Interviewers are using CATI (computer-assisted telephone interviewing), the results can be available minutes after completing the last interview.
  • You can dial random telephone numbers when you do not have the actual telephone numbers of potential respondents.
  • CATI software, such as The Survey System, makes complex questionnaires practical by offering many logic options. It can automatically skip questions, perform calculations and modify questions based on the answers to earlier questions. It can check the logical consistency of answers and can present questions or answers choices in a random order (the last two are sometimes important for reasons described later).
  • Skilled interviewers can often elicit longer or more complete answers than people will give on their own to mail, email surveys (though some people will give longer answers to Web page surveys). Interviewers can also ask for clarification of unclear responses.
  • Some software, such as The Survey System, can combine survey answers with pre-existing information you have about the people being interviewed.
  • Many telemarketers have given legitimate research a bad name by claiming to be doing research when they start a sales call. Consequently, many people are reluctant to answer phone interviews and use their answering machines to screen calls.
  • The growing number of working women often means that no one is home during the day. This limits calling time to a "window" of about 6-9 p.m. or on week-ends (when you can be sure to interrupt dinner or a favorite TV program).
  • You cannot show or sample products by phone. 

Mail Surveys
(or using postal/distribution channels for a paper format).

Advantages  Disadvantages 
  • Mail surveys are among the least expensive (depending on country)
  • This is the only kind of survey you can do if you have the names and addresses of the target population, but not their telephone numbers.
  • The questionnaire can include pictures - something that is not possible over the phone.
  • Mail surveys allow the respondent to answer at their leisure, rather than at the often inconvenient moment they are contacted for a phone or personal interview. For this reason, they are not considered as intrusive as other kinds of interviews.
  • Time! Mail surveys take longer than other kinds. You will need to wait several weeks after mailing out questionnaires before you can be sure that you have gotten most of the responses.
  • In populations of lower educational and literacy levels, response rates to mail surveys are often too small to be useful. This, in effect, eliminates many immigrant populations that form substantial markets in many areas. Even in well-educated populations, response rates vary from as low as 3% up to 90%. As a rule of thumb, the best response levels are achieved from highly-educated people and people with a particular interest in the subject (which, depending on your target population, could lead to a biased sample).
  • One way of improving response rates to mail surveys is to mail a postcard telling your sample to watch for a questionnaire in the next week or two. Another is to follow up a questionnaire mailing after a couple of weeks with a card asking people to return the questionnaire. The downside is that this doubles or triples your mailing cost. If you have purchased a mailing list from a supplier, you may also have to pay a second (and third) use fee - you often cannot buy the list once and re-use it. 

Another way to increase responses to mail surveys is to use an incentive. One possibility is to send a bill (money)/stamps along with the survey (or offer to donate it to a charity specified by the respondent). If you do so, be sure to say that the reward is a way of saying "thanks," rather than payment for their time. Many people will consider their time worth more than a few stamps. Another possibility is to include the people who return completed surveys in a drawing for a prize. A third is to offer a copy of the (non-confidential) result highlights to those who complete the questionnaire. Any of these techniques will increase the response rates.

Remember that if you want a sample of 1,000 people, and you estimate a 10% response level, you need to mail 10,000 questionnaires. You may want to check with your local post office about bulk mail rates - you can save on postage using this mailing method. However, most researchers do not use bulk mail, because many people associate "bulk" with "junk" and will throw it out without opening the envelope, lowering your response rate. Also bulk mail moves slowly, increasing the time needed to complete your project.

Computer Direct Interviews
These are interviews in which the Interviewees enter their own answers directly into a computer. They can be used at malls, trade shows, offices, and so on. The Survey System's optional Interviewing Module and Interview Stations can easily create computer-direct interviews. Some researchers set up a Web page survey for this purpose.

 Advantages  Disadvantages
  • The virtual elimination of data entry and editing costs.
  • You will get more accurate answers to sensitive questions. Recent studies of potential blood donors have shown respondents were more likely to reveal HIV-related risk factors to a computer screen than to either human interviewers or paper questionnaires. The National Institute of Justice has also found that computer-aided surveys among drug users get better results than personal interviews. Employees are also more often willing to give more honest answers to a computer than to a person or paper questionnaire.
  • The elimination of interviewer bias. Different interviewers can ask questions in different ways, leading to different results. The computer asks the questions the same way every time.
  • Ensuring skip patterns are accurately followed. The Survey System can ensure people are not asked questions they should skip based on their earlier answers. These automatic skips are more accurate than relying on an Interviewer reading a paper questionnaire.
  • Response rates are usually higher. Computer-aided interviewing is still novel enough that some people will answer a computer interview when they would not have completed another kind of interview.
  • The Interviewees must have access to a computer or one must be provided for them.
  • As with mail surveys, computer direct interviews may have serious response rate problems in populations of lower educational and literacy levels. This method may grow in importance as computer use increases. 

Email Surveys
Email surveys are both very economical and very fast. More people have email than have full Internet access. This makes email a better choice than a Web page survey for some populations. On the other hand, email surveys are limited to simple questionnaires, whereas Web page surveys can include complex logic.

 Advantages  Disadvantages
  • Speed. An email questionnaire can gather several thousand responses within a day or two.
  • There is practically no cost involved once the set up has been completed.
  • You can attach pictures and sound files.
  • The novelty element of an email survey often stimulates higher response levels than ordinary "snail" mail surveys. 
  • You must possess (or purchase) a list of email addresses.
  • Some people will respond several times or pass questionnaires along to friends to answer. Many programs have no check to eliminate people responding multiple times to bias the results. The Survey System's Email Module will only accept one reply from each address sent the questionnaire. It eliminates duplicate and pass along questionnaires and checks to ensure that respondents have not ignored instructions (e.g., giving 2 answers to a question requesting only one).
  • Many people dislike unsolicited email even more than unsolicited regular mail. You may want to send email questionnaires only to people who expect to get email from you.
  • You cannot use email surveys to generalize findings to the whole populations. People who have email are different from those who do not, even when matched on demographic characteristics, such as age and gender.
  • Email surveys cannot automatically skip questions or randomize question or answer choice order or use other automatic techniques that can enhance surveys the way Web page surveys can.
  • Many email programs are limited to plain ASCII text questionnaires and cannot show pictures. Email questionnaires from The Survey System can attach graphic or sound files. Although use of email is growing very rapidly, it is not universal - and is even less so outside the USA (three-quarters of the world's email traffic takes place within the USA). Many "average" citizens still do not possess email facilities, especially older people and those in lower income and education groups. So email surveys do not reflect the population as a whole. At this stage they are probably best used in a corporate environment where email is common or when most members of the target population are known to have email.
 

Internet/Intranet (Web Page) Surveys
Web surveys are rapidly gaining popularity. They have major speed, cost, and flexibility advantages, but also significant sampling limitations. These limitations make software selection especially important and restrict the groups you can study using this technique.

Advantages  Disadvantages 
  • Web page surveys are extremely fast. A questionnaire posted on a popular Web site can gather several thousand responses within a few hours. Many people who will respond to an email invitation to take a Web survey will do so the first day, and most will do so within a few days.
  • There is practically no cost involved once the set up has been completed. Large samples do not cost more than smaller ones (except for any cost to acquire the sample).
  • You can show pictures. Some Web survey software can also show video and play sound.
  • Web page questionnaires can use complex question skipping logic, randomizations and other features not possible with paper questionnaires or most email surveys. These features can assure better data.
  • Web page questionnaires can use colors, fonts and other formatting options not possible in most email surveys.
  • A significant number of people will give more honest answers to questions about sensitive topics, such as drug use or sex, when giving their answers to a computer, instead of to a person or on paper.
  • On average, people give longer answers to open-ended questions on Web page questionnaires than they do on other kinds of self-administered surveys.
  • Some Web survey software, such as The Survey System, can combine the survey answers with pre-existing information you have about individuals taking a survey.
  • While growing every year, Internet use is not universal. Internet surveys do not reflect the population as a whole. This is true even if a sample of Internet users is selected to match the general population in terms of age, gender and other demographics.
  • People can easily quit in the middle of a questionnaire. They are not as likely to complete a long questionnaire on the Web as they would be if talking with a good interviewer.
  • If your survey pops up on a web page, you often have no control over who replies - anyone from Antartica to Zanzibar, cruising that web page may answer.
  • Depending on your software, there is often no control over people responding multiple times to bias the results.
  • At this stage we recommend using the Internet for surveys mainly when your target population consists entirely or almost entirely of Internet users. Business-to-business research and employee attitude surveys can often meet this requirement. Surveys of the general population usually will not. That said, Internet surveys did about as well, and in some cases better, than other methods in predicting the outcome of the 2012 U.S. presidential election.

 

 

Even when Internet users may not closely match your target population, a Web page survey may be your best choice if you want to show video or both sound and graphics. A Web page survey may be the only practical way to have many people view and react to a video.

In any case, be sure your survey software prevents people from completing more than one questionnaire. You may also want to restrict access by requiring a password (good software allows this option) or by putting the survey on a page that can only be accessed directly (i.e., there are no links to it from other pages).


Questionnaire designKeep It Simple Stupid and short

General Considerations

The first rule is to design the questionnaire to fit the medium. Phone interviews cannot show pictures. People responding to mail or Web surveys cannot easily ask "What exactly do you mean by that?" if they do not understand a question. Intimate, personal questions are sometimes best handled by mail or computer, where anonymity is most assured.

KISS - keep it short and simple. If you present a 20-page questionnaire most potential respondents will give up in horror before even starting. Ask yourself what you will do with the information from each question. If you cannot give yourself a satisfactory answer, leave it out. Avoid the temptation to add a few more questions just because you are doing a questionnaire anyway. If necessary, place your questions into three groups: must know, useful to know and nice to know. Discard the last group, unless the previous two groups are very short.

Start with an introduction or welcome message. In the case of mail or Web questionnaires, this message can be in a cover page or on the questionnaire form itself. If you are sending emails that ask people to take a Web page survey, put your main introduction or welcome message in the email. When practical, state who you are and why you want the information in the survey. A good introduction or welcome message will encourage people to complete your questionnaire.

Allow a "Don't Know" or "Not Applicable" response to all questions, except to those in which you are certain that all respondents will have a clear answer. In most cases, these are wasted answers as far as the researcher is concerned, but are necessary alternatives to avoid frustrated respondents. Sometimes "Don't Know" or "Not Applicable" will really represent some respondents' most honest answers to some of your questions. Respondents who feel they are being coerced into giving an answer they do not want to give often do not complete the questionnaire. For example, many people will abandon a questionnaire that asks them to specify their income, without offering a "decline to state" choice.

For the same reason, include "Other" or "None" whenever either of these is a logically possible answer. When the answer choices are a list of possible opinions, preferences, or behaviors, you should usually allow these answers.

On paper, computer direct and Internet surveys these four choices should appear as appropriate. You may want to combine two or more of them into one choice, if you have no interest in distinguishing between them. You will rarely want to include "Don't Know," "Not Applicable," "Other" or "None" in a list of choices being read over the telephone or in person, but you should allow the interviewer the ability to accept them when given by respondents.

Question Types
Researchers use three basic types of questions: multiple choice, numeric open end and text open end (sometimes called "verbatims"). 
Rating Scales and Agreement Scales are two also common types of questions that some researchers treat as multiple choice questions and others treat as numeric open end questions.
Questions can take many forms:

  • Open-ended: Designed to prompt the respondent to provide you with more than just one or two word responses. These are often "how" or "why" questions. For example: "Why is it important to use condoms?" These questions are used when you want to find out what leads people to specific behaviors, what their attitudes are towards different things, or how much they know about a given topic; they provide good anecdotal evidence. The drawback to using open-ended questions is that it's hard to compile their results.
  • Closed-ended (also sometimes referred to as forced choice questions): Specific questions that prompt yes or no answers. For example: "Do you use condoms?" These are used when the information you need is fairly clear-cut, i.e., if you need to know whether people use a particular service or have ever heard of a specific local resource.
  • Multiple choice: Allow the respondent to select one answer from a few possible choices. For example: "When I have sex, I use condoms... a) every time, b) most times, c) sometimes, d) rarely, e) never." These allow you to find out more detailed information than closed-ended questions, and the results can be compiled more easily than open-ended questions.
  • Likert scale: Each respondent is asked to rate items on a response scale. For instance, they could rate each item on a 1-to-5 response scale where:
    • 1 = strongly disagree
    • 2 = disagree
    • 3 = undecided
    • 4 = agree
    • 5 = strongly agree
    • If you want to weed out neutral and undecided responses you can use an even-numbered scale with no middle "neutral" or "undecided" choice. In this situation, the respondent is forced to decide whether he or she leans more towards the "agree" or "disagree" end of the scale for each item. The final score for the respondent on the scale might be the sum of his or her ratings for all of the items.

Question and Answer Choice Order
Two broad issues to keep in mind:

  • How the question and answer choice order can encourage people to complete your survey.
  • How the order of questions or the order of answer choices could affect the results of your survey.

Ideally, the early questions in a survey should be easy and pleasant to answer. These kinds of questions encourage people to continue the survey. In telephone or personal interviews they help build rapport with the interviewer. Grouping together questions on the same topic also makes the questionnaire easier to answer.

Whenever possible leave difficult or sensitive questions until near the end of your survey. Any rapport that has been built up will make it more likely people will answer these questions. If people quit at that point anyway, at least they will have answered most of your questions.

Answer choice order can make individual questions easier or more difficult to answer. Whenever there is a logical or natural order to answer choices, use it. Always present agree-disagree choices in that order. Presenting them in disagree-agree order will seem odd. For the same reason, positive to negative and excellent to poor scales should be presented in those orders. When using numeric rating scales higher numbers should mean a more positive or more agreeing answer.

Question order can affect the results in two ways:

  • One is that mentioning something (an idea, an issue, a brand) in one question can make people think of it while they answer a later question, when they might not have thought of it if it had not been previously mentioned. In some cases you may be able to reduce this problem by randomizing the order of related questions. Separating related questions with unrelated ones can also reduce this problem, though neither technique will eliminate it.
  • The other way question order can affect results is habituation. This problem applies to a series of questions that all have the same answer choices. It means that some people will usually start giving the same answer, without really considering it, after being asked a series of similar questions. People tend to think more when asked the earlier questions in the series and so give more accurate answers to them.

If you are using telephone, computer direct or Internet interviewing, good software can help with this problem. Software should allow you to present a series of questions in a random order in each interview. This technique will not eliminate habituation, but will ensure that it applies equally to all questions in a series, not just to particular questions near the end of a series.

Another way to reduce this problem is to ask only a short series of similar questions at a particular point in the questionnaire. Then ask one or more different kinds of questions, and then another short series if needed.

A third way to reduce habituation is to change the "positive" answer. This applies mainly to level-of-agreement questions. You can word some statements so that a high level of agreement means satisfaction (e.g., "My supervisor gives me positive feedback") and others so that a high level of agreement means dissatisfaction (e.g., "My supervisor usually ignores my suggestions"). This technique forces the respondent to think more about each question. One negative aspect of this technique is that you may have to modify some of the data after the results are entered, because having the higher levels of agreement always mean a positive (or negative) answer makes the analysis much easier. However, the few minutes extra work may be a worthwhile price to pay to get more accurate data.

The order in which the answer choices are presented can also affect the answers given. People tend to pick the choices nearest the start of a list when they read the list themselves on paper or a computer screen. People tend to pick the most recent answer when they hear a list of choices read to them.

As mentioned previously, sometimes answer choices have a natural order (e.g., Yes, followed by No; or Excellent - Good - Fair - Poor). If so, you should use that order. At other times, questions have answers that are obvious to the person that is answering them (e.g., "Which brands of car do you own?"). In these cases, the order in which the answer choices are presented is not likely to affect the answers given. However, there are kinds of questions, particularly questions about preference or recall or questions with relatively long answer choices that express an idea or opinion, in which the answer choice order is more likely to affect which choice is picked. If you are using telephone, computer direct, or Web page interviewing, have your software present these kinds of answer choices in a random order.


Collecting the surveys

Soon after the surveys are distributed, some of them will begin to arrive at the sponsoring organization.
Here are the steps you should take to collect your surveys:

  • Gather incoming surveys collected at participating sites. A representative of your organization should collect incoming surveys as they arrive in the mail or your drop box. He or she should also call or stop by collection sites from time to time to pick up any surveys that have been dropped off.
  • Review returned surveys, checking for any that are incomplete. If any surveys were returned for having an improper mailing address, try to find the correct address and mail it out again, if you can.
  • If you have raw data/paper version of the data, start encoding the data as early as possible to finalise the test of your own data treatment system and get the whole chain of people ready for when the mass of reply will arrive.
  • Secure a larger return, if necessary. This may mean distributing surveys again, or expanding your sample size, or also  sending reminders to your sample list (especially if you are able to identify who replied or who din't in order not to upset people who already responded).

How do you analyse and compile the results of your survey?

Now that you've gathered the completed surveys, you'll need to figure out the results. Sometimes all you have to do is tabulate the results -- that is, add them up and display in a table.
In fact, you should already now and have tested the way you will process your data at the very early stage once you have defined the goal/purposes and target groups of your survey. (Don't wait to have the data to figure out how you will deal with them !).

For instance, if 100 questionnaires were returned in a survey about problems in the neighborhood, you just need to count the answers. Let's say that there was a question asking what people felt was the biggest challenge facing the neighborhood; 70 people mentioned law enforcement, 10 cited transportation, 15 marked potholes, and 5 said noise. The result in cases like this is clear.

However, analysis can be far more complicated than that. If you're looking, for instance, at how people feel about a service or problem, you may end up with a lot of answers to open-ended questions that are apparently unrelated. In this case, you will need to try to find patterns.

Once you've done that, what do these numbers mean? Well, you will need to look at the overall survey to see how each percentage compares to the others. For example, what questions had the highest proportions of similar responses?

We suggest that you write up a brief report -- one page is sufficient -- summarizing the results of the survey. In your report, look for any patterns -- do people in a particular part of town feel more strongly about a particular issue than those in other areas?

Share this information with your staff. Get their feedback and discuss whether any further surveying needs to be done before completing.

Now that you've figured out what the results mean, you need to decide what to do with them. To whom are you going to communicate them, and how? In case of a community initiative, the results should be made public as soon as possible so that members in the community and community leaders can be made aware of a problem or potential problem and start working to solve it. If other similar surveys have done in the same area, you may want to compare your results with the other surveys' results.
This aspect should also be anticipated very clearly. You should already have scenarii at the definition stage on what you want to do with the data; how you get prepared to exploit the results and to be ready to react as well, in a multi-actors cooperation approach, to the findings).

An organization conducting a survey about its' services might want to use results to provide a better service or to change a current policy to a more efficient one. In a situation where funding is at stake, the results would need to go to the funder to convince the funder of the need for new or continued support. The results could also be used by the organization itself to determine where and what kinds of services are needed.


General Tips

  1. Keep the questionnaire as short as possible. We mentioned this principle before, but it is so important it is worth repeating. More people will complete a shorter questionnaire, regardless of the interviewing method. If a question is not necessary, do not include it.
  2. Start with a Title (e.g., Leisure Activities Survey). Always include a short introduction - who you are and why you are doing the survey. If you are asking about different brands, it is often a good idea to give the name of the research company rather than the client (e.g., XYZ Research Agency rather than the manufacturer of the product/ service being surveyed). Many firms create a separate research company name (even if it is only a direct phone line to the research department) to disguise themselves. This is to avoid possible bias, since people rarely like to criticize someone to their face and are much more open to a third party.
  3. In some cases, though, it may help to mention the client. If you are surveying members of an organization, the members may be more likely to respond if they think the organization is asking their opinions on how it can best meet their needs. The same could be true when you are surveying users of a particular service.
  4. Preserve anonymity. Reassure your respondent that his or her responses will not be revealed to your client, but only combined with many others to learn about overall attitudes.
    If you hand out questionnaires on your premises, you obviously cannot remain anonymous, but keep the bias problem in mind when you consider the answers.
  5. Include a cover letter with all mail surveys. A good cover letter or invitation to take a Web page survey will increase the response rate. A bad one, or none at all, will reduce the response rate. Include the information in the preceding two paragraphs and mention the incentive (if any). Describe how to return the questionnaire. Include the name and telephone number of someone the respondent can call if they have any questions. Include instructions on how to complete the survey itself.
    The most effective cover letters and invitations include the following elements: Ask the recipient to take the survey. Explain why taking it will improve some aspect of the recipient's life (it will help improve a product, make an organization better meet their needs, make their opinions heard). Appeal to the recipient's sense of altruism ("please help"). Ask the recipient again to take the survey.
  6. Number mail questionnaires on each page and include the return address on the questionnaire itself, because pages and envelopes can be separated from each other. Envelopes should have return postage prepaid. Using a postage stamp often increases response rates, but is expensive, since you must stamp every envelope - not just the returned ones.
  7. You may want to leave a space for the respondent to add their name and title. Some people will put in their names, making it possible for you to recontact them for clarification or follow-up questions. Indicate that filling in their name is optional. If the questions are sensitive in nature, do not have a space for a name. Some people would become suspicious and not complete the survey.
  8. If the survey contains commercially sensitive material, ask a "security" question up front to find whether the respondent or any member of his family, household or any close friend works in the industry being surveyed. If so, terminate the interview immediately. They (or family or friends) may work for the company that commissioned the survey - or for a competitor. In either case, they are not representative and should be eliminated. If they work for a competitor, the nature of the questions may betray valuable secrets. The best way to ask security questions is in reverse (i.e., if you are surveying for a pharmaceutical product, phrase the question as "We want to interview people in certain industries - do you or any member of your household work in the pharmaceutical industry?). If the answer is "Yes" thank the respondent and terminate the interview. Similarly, it is best to eliminate people working in the advertising, market research or media industries, since they may work with competing companies.
  9. After the security question, start with general questions. If you want to limit the survey to users of a particular product, you may want to disguise the qualifying product. As a rule, start from general attitudes to the class of products, through brand awareness, purchase patterns, specific product usage to questions on specific problems (i.e., work from "What types of coffee have you bought in the last three months" to "Do you recall seeing a special offer on your last purchase of Brand X coffee?"). If possible put the most important questions into the first half of the survey. If a person gives up half way through, at least you have the most important information.
  10. Make sure you include all the relevant alternatives as answer choices. Leaving out a choice can give misleading results. The need to include all relevant alternatives is not limited to political polls. You can get misleading data anytime you leave out alternatives.
  11. Do not put two questions into one. Avoid questions such as "Do you buy frozen meat and frozen fish?" A "Yes" answer can mean the respondent buys meat or fish or both. Similarly with a question such as "Have you ever bought Product X and, if so, did you like it?" A "No" answer can mean "never bought" or "bought and disliked." Be as specific as possible. "Do you ever buy pasta?" can include someone who once bought some in 1990. It does not tell you whether the pasta was dried, frozen or canned and may include someone who had pasta in a restaurant. Few people can remember what they bought more than three months ago unless it was a major purchase such as an automobile or appliance.
  12. The overriding consideration in questionnaire design is to make sure your questions can accurately tell you what you want to learn. The way you phrase a question can change the answers you get. Try to make sure the wording does not favor one answer choice over another.
  13. Avoid emotionally charged words or leading questions that point towards a certain answer. You will get different answers from asking "What do you think of the XYZ proposal?" than from "What do you think of the Republican XYZ proposal?" The word "Republican" in the second question would cause some people to favor or oppose the proposal based on their feelings about Republicans, rather than about the proposal itself. It is very easy to create bias in a questionnaire. This is another good reason to test it before going ahead.
  14. If you are comparing different products to find preferences, "give each one a neutral name or reference. Do not call one "A" and the second one "B". This immediately brings images of A grades and B grades to mind, with the former being seen as superior to the latter. It is better to give each a "neutral" reference such "M" or "N" that do not have as strong a quality difference image.
  15. Avoid technical terms and acronyms, unless you are absolutely sure that respondents know they mean. If you must use an acronym, spell it out the first time it is used.
  16. Make sure your questions accept all the possible answers. A question like "Do you use regular or premium gas in your car?" does not cover all possible answers. The owner may alternate between both types. The question also ignores the possibility of diesel or electric-powered cars, and even doesn't cover the case if the person doesn't have a car.
  17. If you want only one answer from each person, ensure that the options are mutually exclusive (no intersection between the choices).
  18. Score or rating scale questions (e.g., "If '5' means very good and '1' means very poor how would rate this product?") are a particular problem. Researchers are very divided on this issue. Many surveys use a ten-point scale, but there is considerable evidence to suggest that anything over a five point scale is irrelevant. This depends partially on education. Among university graduates a ten point scale will work well. Among people with less than a high school education five points is sufficient. In some populations, a three-point scale (good/acceptable/bad) may be all some respondents can understand. Even consider not giving the possibility of a neutral position (with a scale of 4, see §20 below). Don't forget to allow the N/A (non applicable) choice if relevant.
  19. Giving a verbal or written label to each point on a scale, instead of just the endpoints, will usually yield higher-quality data, though this may not be practical when there are more than five points on the scale.
  20. Another issue on which researchers differ is whether to use a scale with an odd or even number of points. Some like to force people to give an answer that is clearly positive or negative. This can make the analysis easier. Others feel it is important to offer a neutral, middle option. Your interviewing mode can make a difference here. A good interviewer can often get an answer, but in a self-administered interview, such as a Web page survey, a person who is frustrated by being unable to give a middle answer may leave a question blank or quit the survey altogether.
  21. Be sure any rating scale labels are meaningful. For example: "best", "average" or ""worst" will force most answers into the middle category, resulting in very little usable information.
  22. If you have used a particular scale before and need to compare results, use the same scale. Four on a five-point scale is not equivalent to eight on a ten-point scale. Someone who rates an item "4" on a five-point scale might rate that item anywhere between "6" and "9" on a ten-point scale.
  23. Do not use negative numbers when asking for ratings. Some people do not like to give negative numbers as answers. A scale of -2 to +2 is mathematically equivalent to a scale of 1 to 5, but in practice you will get fewer people picking -2 or -1 than would pick 1 or 2. If you want 0 to be the midpoint of a scale when you produce reports, you can weight the answers after data collection to get that result.
  24. Always discount "favorable" answers by a significant factor. Unfortunately, there is no hard and fast rule on how much to do this. It depends on the situation. Some people tend to give answers they think will please an interviewer or even a survey company or survey sponsor, and in some cultures it might be considered impolite to give negative answers. One factor to consider is the survey mode. People tend to pick the most positive answer on a scale more often when answering telephone or in-person surveys than other types of surveys, regardless of the details of the question.
    The desire to please translates into a tendency to pick agreeing answers on agreement scales. While logically the percentage that strongly agrees that "X is good" should exactly equal the percentage that strongly disagrees that "X is bad," in the real world, this is unlikely to be true. Experiments have shown that more people will agree than disagree. One way to eliminate this problem is to ask half your respondents if they agree that "X is good" and the other half if they agree that "X is bad." You could then reverse the answers given by the second group. This is extra work, but it may be worth it if it is important to get the most accurate percentage of people who really agree with something.
  25. People sometimes give answers they feel will reflect well on them. This is a constant problem for pre-election polls. More people say they will vote than actually will vote. More people say they go to museums or libraries than actually do. This problem is most significant when your respondents are talking directly to a person. People give more honest answers when answering questions on a computer. Mail surveys are in-between.
  26. Because people like to think of themselves as normal or average, the range of answer choices you give when asking for a quantity or a frequency can affect the results. for a same value, you may make it feel more extreme if you put a s the maximum of the possible anwsers compared to another possible set of choices where that value would appear in the medium or mediu-top range.The first list of choices makes the value sound extreme, while the second list of choices makes it seem typical.
  27. In personal interviews it is vital for the Interviewer to have empathy with the Interviewee. In general, Interviewers should try to "blend" with respondents in terms of race, language, sex, age, etc. Choose your Interviewers according to the likely respondents.
  28. Leave your demographic questions (age, gender, income, education, etc.) until the end of the questionnaire. By then the interviewer should have built a rapport with the interviewee that will allow honest responses to such personal questions. Mail and Internet questionnaires should do the same, although the rapport must be built by good question design, rather than personality. Exceptions to this rule are any demographic questions that qualify someone to be included in the survey. For example, many researchers limit some surveys to people in certain age groups. These questions must come near the beginning.
  29. Do not have an interviewer ask a respondent's gender, unless they really have no idea. Have the interviewer fill in the answer themselves.
  30. Paper questionnaires requiring text answers, should always leave sufficient space for handwritten answers. Lines should be about half-an-inch (one cm.) apart. The number of lines you should have depends on the question. Three to five lines are average.
  31. Leave a space at the end of a questionnaire entitled "Other Comments". Sometimes respondents offer casual remarks that are worth their weight in gold and cover some area you did not think of, but which respondents consider critical. Many products have a wide range of secondary uses that the manufacturer knows nothing about but which could provide a valuable source of extra sales if approached properly. In one third world market, a major factor in the sale of candles was the ability to use the spent wax as floor polish - but the manufacturer only discovered this by a chance remark.
  32. Always consider the layout of your questionnaire. This is especially important on paper, computer direct and Internet surveys. You want to make it attractive, easy to understand and easy to complete. If you are creating a paper survey, you also want to make it easy for your data entry personnel.
    Try to keep your answer spaces in a straight line, either horizontally or vertically. A single answer choice on each line is best. Eye tracking studies show the best place to use for answer spaces is the right hand edge of the page. It is much easier for a field worker or respondent to follow a logical flow across or down a page. Using the right edge is also easiest for data entry.
  33. The choose a survey system which let you create a Questionnaire Form with the answer choices in two columns. Creating the form that way can save a lot of paper or screen space, but you should recognize doing so makes the questionnaire a little harder to complete. It also slows the data entry process when working with paper questionnaires.
  34. Questions and answer choice grids. They can look attractive and save paper, or computer screen space. They also can avoid a long series of very repetitive question and answer choice lists. Unfortunately, they also are a bit harder than the repeated lists for some people to understand. As always, consider whom you are studying when you create your questionnaire.
  35. When using a grid, the way you lay out the choices will affect the results. It is not clear whether it is always best to make the leftmost column the most positive response or the most negative response, but there is a tendency for people to pick the left side of the grid more than the right side, regardless of whether the left side is positive or negative. So any time you use a grid you should discount the left side responses to some degree. This tendency does not interfere with comparing the answers in different rows in the grid, but does affect absolute statements of a line of the grid.

Additional Tips for Web Surveys

  1. One principle is to consider good Web page design when creating your survey pages. Do not use too many colors or fonts. They are distracting. On the other hand, bolding, italicizing, and changing the colors of key words, used appropriately, can make your questions easier to understand. Using color and/or a smaller font size to make instructions distinct from question text can make your questionnaire easier to follow.
  2. Always specify a background color, even if it is white (usually a good choice). Some browsers may show a background color you do not expect, if you do not specify one. Background images usually make text harder to read, even when they make a page more attractive at first glance.
  3. Use graphics sparingly. Some home Internet users still connect via modems, and graphics slow download times. Remember that showing a large graphic at a small size on a Web page does not reduce the time needed to download the graphic. Create or modify the graphic to a file size that is no bigger than you need. If your sample consists of people at work, you may use more graphics, since those people usually have faster connections, but even they appreciate faster downloads.
  4. Use video only if that is what you are testing (e.g., a commercial). Make sure you do not require people to scroll horizontally to view part of the survey page. Most people find horizontal scrolling annoying. Question text wraps to fit the available space, but you can make a grid that is wider than some screens. As of January 2013 about 9% of people still use 1024x768 screen resolution. You may want to design your pages to be up to 980 pixels wide (leaving room for the browser edges and a scrollbar). In any case, you should not ask opinions on any graphic wider than that, since some people will have to scroll to see it.
  5. Use "Responsive Layout" format when possible. Smartphones are a special consideration. Many newer ones have screens with 1024x768 or higher resolutions, but their screen sizes are so small that font sizes that are perfectly readable on a computer-sized screen may be unreadable without zooming in on a phone. If you think that many of your respondents will be taking your survey on a smartphone or smaller tablet, you may want to use large fonts for both labels and text input.
  6. Include an introduction or welcome page. Explain the reason for the survey (as far as you can without compromising the survey). Put instructions at the point they are needed, instead of grouping them on the first page.
  7. Make sure your page and question layout are consistent. Do not put answer choices on the right for some questions and on the left for others. Use color consistently. For example, always use the same color to represent an instruction, which is not part of a question per se. Use a different color (or bolding) any time you want to highlight words within questions.
  8. Recognize that requiring that questions be answered will likely increase the number of people who drop out of a survey in the middle. If you do require answers, consider doing so only on key questions. Whenever you require an answer make sure the available options include all possible answers, including "don't know," "decline to state," or "not applicable," if there is any chance that these may represent some people's answers.
  9. Consider your sample when designing the pages. Using answer grids and presenting answer choices in two or more columns can look attractive, save space and help avoid vertical scrolling. Unfortunately, these formats are a bit harder for some people to understand than a simple vertical list of answer choices. If you think your target population may have some trouble understanding how to fill out the survey, use these formats sparingly.
  10. Allow space for long replies to comment type questions. Some people will type in longer answers on a Web page than they would write on a paper questionnaire or say to an interviewer. Drop-down lists save space on the screen, but be careful using them. Lists that require scrolling to see some choices can bias the results. Use them only if there is only one possible choice a person can make. One example is state of primary residence. If you present a list of choices that people have to think about, and only some of the choices are initially visible, there will be a bias in favor of those initially visible choices.
  11. Researchers have been looking into the issue of whether it is better to present a survey in one or more long scrolling pages or in a series of separate pages that do not need scrolling. Research has not yet provided a clear answer. There is some evidence that grouping several similar questions on a page may result in answers that are more similar than if the questions were on different pages. Some people may dislike scrolling down a long page, while others may dislike the brief wait between questions when each is on a different page. Having your questionnaire split into multiple pages has the advantage that if someone quits partway through, at least you have the answers they have already given. You will also need to split your survey into multiple pages, if you want some people to not see certain questions, or if you want the answers given for some questions to affect those shown for later questions.
  12. Sometimes researchers like to announce the start of a new section in a survey. Using separate section break pages that announce the start of a section, but do not include any questions will likely lead to some people quitting the survey at one of those pages. If you want to announce a new section, include at least one question on the page. The one exception to this rule is if the section start includes some instructions, and you have decided to randomize the order of the questions in the section. In that case, the section instructions would have to be one their own page, since you do not know which question would follow.
  13. Some researchers like to show progress bars in a web survey. A fast moving progress bar (20%, 40%, 60%...) might encourage people, but a slow moving one (2%, 4%, 6%) will discourage people and likely lead to people quitting the survey.
  14. When you have finished creating the survey and have it up on your Web site, test it thoroughly. Make sure that all the pages look as you wish and that all skips, randomizations and other logic work as you intend. A test on your own PC or a paper copy of the questionnaire does not guarantee that the copy on the Web will look and act the same.
  15. We also recommend you monitor the live results as your survey progresses (good Web survey software allows this). Doing so can help you spot any problems that did not appear during your testing.

 


Sources: