Sample quality in market research has always been an industry concern. Not that long ago, survey fraud was conducted largely by “professional” respondents, real people eager to earn extra incentives and rewards for completing a survey, providing inaccurate answers that impacted the integrity of the results.
This manual method of survey fraud seems quaint in comparison to the technology that’s available to fraudsters that want to cheat the system. AI powered bots and other technologies have the potential to scale survey fraud exponentially in an environment where its already estimated that 15% to 50% of online survey responses are fraudulent.
The pervasiveness and potential growth of survey fraud demands proper steps be taken by the industry to address the problem. The issue has led to the creation of the Global Data Quality Initiative, led by six prominent organizations in the global research and analytics industry to address data quality risks.
Meantime, Phase 5 will not stand idly by. We understand the importance of confronting this issue head on, which is why we (special thanks to my colleague Christine Sorensen) have developed our own set of best practices designed to mitigate the risk and ensure our clients continue to get market research results and insights that are valid, accurate and therefore actionable.
Enhancing Sample Quality: Phase 5's Best Practices
In the realm of survey research, maintaining high data quality is paramount. At Phase 5, we’ve developed a comprehensive set of best practices to monitor, manage, and improve sample quality throughout the project lifecycle. Our approach focuses on selecting reliable panel providers, incorporating robust quality checks in survey design, and implementing stringent processes during and after fieldwork to detect and prevent fraud.
Key Considerations for Data Quality
At the heart of our methodology is the understanding that ensuring data quality is a shared responsibility. While panel providers play a crucial role, their ability to contribute depends on their access to the data—a factor that isn’t always within their control. Therefore, effective collaboration requires transparency about risks and an agreement on the level of effort needed to manage those risks.
Quality management strategies must be tailored to each project’s unique context, considering factors such as sample source, study topic, target audience, and survey type. We recognize that reaching certain audiences, like B2B respondents or those in low-incidence groups, requires more effort and resources due to the higher risk of fraud. As such, it’s important to balance quality efforts with budget constraints, as perfection isn’t always achievable or realistic.
Selecting the Right Panel Provider
Choosing the right panel provider is critical to ensuring sample quality. At Phase 5, we prioritize providers with a proven track record of delivering high-quality samples consistently. This includes a willingness to collaborate and resolve issues promptly, as well as transparency in their processes and techniques.
We also value providers who adhere to industry best practices, such as using legitimate and diverse onboarding sources, implementing double opt-in processes, and employing advanced techniques like digital fingerprinting and proxy server detection. Innovation in quality control methods is another key criterion for selecting panel providers.
Phase 5's Techniques for Quality Assurance
Before a project begins, we meticulously evaluate vendors, maintain detailed records of past experiences, and engage in ongoing discussions to stay informed on the latest industry best practices. We also ensure that our staff is regularly trained and updated on these practices.
During the survey design phase, we follow best practices to minimize respondent fatigue and disengagement. This includes selecting key indicators for quality assessment, such as speeders, IP addresses, and open-ended questions. We also employ AI tools to enhance the depth of our insights and identify low-quality or fraudulent responses.
Throughout fieldwork, we conduct regular quality checks to ensure timelines are met and high-quality respondents are retained. We use a combination of indicators to assess data quality, applying a 3-strike rule for ambiguous cases to balance effort and objectivity. In high-risk cases, we involve clients in reviewing questionable data, ensuring transparency and collaboration.
Commitment to High-Quality Data
At Phase 5, our commitment to high-quality data is unwavering. By adhering to these best practices, we help our clients achieve reliable, actionable insights while balancing efficiency and cost-effectiveness. Whether working on a large-scale survey or a niche study, our goal is to deliver the highest quality data possible, ensuring that our clients can make informed decisions with confidence.
If you would like to discuss sample quality or survey fraud, please contact us.
