Bias is pernicious: it creeps into all kinds of research, including surveys and social network analysis! The more bias you introduce into your survey, the less reliable and accurate your data will be. This is why it’s important to check your survey for bias, at all stages of implementation, including survey question design, sampling and response collecting. Let’s review the ways bias creeps into surveys at each of these stages – that’s the best way to ensure you create bias-free surveys.

What is Survey Bias?

First, what exactly do we mean by bias? A bias is a tendency, inclination, or prejudice toward or against something or someone. In the context of survey research, bias allows outside factors to influence data, reducing its validity – how well it measures what you intend it to measure. For example, if we ask people whether they hate their mother, many people may answer untruthfully because they fear judgement. To ensure survey results are valid, evaluators and researchers need to create bias-free surveys. While it’s impossible to get rid of all bias, there are ways to help!

Question and Response Order Matters

You likely already know that your question wording matters. However, the way you order questions can also introduce bias. The question you ask first creates a context that influences everything that comes after. For example, if you ask a very specific question about satisfaction which people rank highly, they are more likely to give a high satisfaction response to the following questions. This is called the “priming effect,” and reflects our brain’s attempt to identify patterns to help make sense of complex ideas. Make sure that you take into account your question order, and how your first questions frame later question before you send it out.

Use the Right Sampling to Create Bias-Free Surveys

Once your survey is ready, you need to consider how choosing your respondents can introduce bias. Sampling is one of the most common places for bias to occur in surveys. First, consider undercoverage: whether any groups you wish to survey are underrepresented. In the context of networks, consider whether you have included unlikely partners that matter in your network but may not be on a formal list of partners. Second, think about non-response bias: When a subgroup of respondents that share a common trait don’t answer the survey. A common issue in network surveys is language: If you don’t make your survey available in multiple languages, you may see those who speak a common language underrepresented. Don’t finalize your respondent list without considering these factors. 

Consider How Respondent Bias Affects Their Answers

Lastly, people completing your survey may respond to your survey with bias themselves, seeking to be agreeable. Known as Acquiescence bias, it can cloud your results and inflate how successful your network is, or how value one another. Avoid it by wording your questions to avoid yes and no questions. A similar issue, desirability bias, leads people to give the response they believe is socially desirable. If they believe that high-trust is socially desired in their network, they may inflate how much they trust one another. Framing your survey to respondents so they see value in accurate data can help ground their answers, though it’s hard to totally combat this bias.

Review: Three Pitfalls to Avoid to Create Bias-Free Surveys

First, make sure you order your question in a way that limits bias. If you’re not sure how to do this, consider the PARTNER default survey, which we’ve ordered for you! Second, think about how your sampling may include under representation or non-response bias. This is why it’s critical to take time to bound your network before sending out a survey. Third, think about questions from the respondent’s perspective to minimize acquiescence and desirability bias. With these tips, you’re well on your way to create bias-free surveys!