An Introduction to Surveys

Surveys are used to collect information about populations of people that cannot be easily observed, such as attitudes, concepts, and behavior. Due to the un-observable nature of the information that surveys collect, survey designers must take steps to ensure surveys collect information that is accurate, reliable, and representative of the target population. The survey development process requires five key steps:

1. Designing the Survey Process
2. Developing the Survey Questions
3. Testing the Survey Questions
4. Collecting Data
5. Analyzing Data

Developing surveys using this five step process is both an art and a science. This field guide will provide you with an overview of best practices for developing high quality surveys.

Designing The Survey Process

Surveys can be administered through a variety of means including in-person interviews, through the mail, and, via an electronic medium such as a computer. The type of survey you choose to administer will depend on your goals as a researcher, the timing of the survey, and your target population. As you decide what type of survey to administer, ask yourself the following questions:

• Goals: What kind of information do I want to collect about my target population?
• Timing: When do I need the data?
• Target population: Who am I researching? How large is the target population and how easy are they to contact?

Your answers to these questions will help you determine when and how you administer your survey.

Developing and Testing the Survey Questions

In order for survey results to be meaningful, all survey questions must be both reliable and valid. Reliable questions mean the exact same thing to everyone. This means that the same respondent will answer the question in the exact same way when they see it more than once. Valid questions measure the traits they are intended to measure.

Ultimately, survey questions need to be easy to understand and answer, and it is helpful to keep the  respondent’s point of view in mind while you are writing questions in order to achieve this aim. Also, bear in mind that some questions—particularly those referring to a specific period of time—are more taxing on respondents than others. This is because these questions require respondents to sort through large amounts of information from the past in order to produce meaningful responses.

Before the survey is administered, its questions should be field tested to ensure their reliability and validity. Survey design is an iterative process, and questions may need to be rearranged and/ or rewritten.

Collecting and Analyzing Data

All of the decisions you have made about your survey up to this point will affect your response rate. Although there is no agreed upon value for a “good” response rate, in general, every effort should be made to incentivize participants to complete their surveys. Common incentives include raffles, small cash incentives, and other prizes.

Selecting a Sample

Depending on the size and geographic spread of your target population you may need to use a representative sample of the survey population instead. The representative sample of survey participants must be carefully and randomly selected. Otherwise, bias is introduced into the results, which limits your ability to make inferences about the larger population based on the sample. In other words, selection bias can be introduced because the respondents differ in some fundamental way from the overall population of interest, like they’re more motivated to answer a survey to express their like or dislike of a program relative to others in the target population.

As a result, opt-in surveys and convenience samples that are selected based on membership in or organization or proximity to a specific location should be avoided, as they share a common weakness: there is no direct theoretical support for using them to describe the characteristics of the larger target population.

Determining Sample Size

Once you’ve decided to use a sample to analyze your target population, you will need to decide how many survey responses you will need. In order for your survey data to yield meaningful results, your sample should neither be too large nor too small, and the specific number will depend on several different variables. As you determine what your sample size will be, take the following into consideration:

• Population size: What is the size of my target population? Typically, larger target populations require larger sample sizes.
• Confidence Interval (Margin of Error): How much error am I willing to allow? In light of the fact that no sample will perfectly
  represent the larger population, the researcher must decide how much higher or lower than the actual population average they are willing to let the average within their sample fall. The margin of error determines the maximum and minimum amounts by which the sample results differ from the actual population and is usually expressed as plus or minus some percentage.
• Confidence Level: How confident do I want to be that the average for the total population falls within my confidence interval?The confidence level tells you how sure you can be that the true average for your total target population lies within the confidence interval. Confidence levels are typically expressed as percentages, with 90%, 95%, and 99% being the most common.Below is a table that shows what different sample sizes might look like based on the margin of error for a survey with a confidence level of 99%.

Interpreting Data

Remember that correlation does not equal causation. Survey data can reveal relationships between things, but they cannot say that one thing caused another thing to happen.

Survey data can, however, reveal a great deal about a population of people’s attitudes, wants and needs.

Sources

The Following Works Were Consulted During the Creation of This Field Guide:

• Groves, R., Fowler, F., Couper, M., Lepkowski, J., Singer, E., & Tourangeau, R. (2009). Survey Methodology Second Edition.
• Holman, M. (n.d.). Survey Methodology (Duke University School of Law). Retrieved November 2015.
• Smith, S. (n.d.). Determining Sample Size: How to Ensure You Get the Correct Sample Size. Retrieved November 2015.
• Thayer-Hart, N., Dykema, J., Elver, K., Schaeffer, N., & Stevenson, J. (2012). Survey Fundamentals: A Guide to Designing and Implementing Surveys (University of Wisconsin). Retrieved November 2015.
• Van Dessel, G. (2013). How to Determine Population and Survey Sample Size. Retrieved November 2015.

Click here to download a print-able version of this field guide.