
Quantitative market research relies fully on the accuracy and reliability of survey data. And yet, on any given day, while scrolling through LinkedIn posts, there’s a plethora of conversations about sample quality. In fact, in its 8th year of reporting on trending “buzz” topics, Greenbook’s latest 2024 GRIT Insights Practice Report* categorizes ‘data quality’ as a buzz phrase, and notes that “Especially due to the opportunities and challenges posed by developments in AI, automation, methodologies, and analytics, challenges in data quality are top-of-mind for more people than ever, it seems, and more meaningful”.
As insights professionals, those of us here at Talk Shoppe are well aware of the threat of fraudulent survey activity and infiltration of bad actors and have seen first-hand its impact on project timelines and response quality. According to the 2024 GRIT Report, 40% of full-service research professionals and 45% of buyer-side insights professionals interviewed reported that they had to address doubts about sample-related research quality more than once in the past 6 months.

At Talk Shoppe, we’ve had some good laughs reading the open-ended responses of some 🤖 respondents. Here are a few of our favorites in response to specific questions about the experience using a particular app:
“It’s absolutely fantastic and very interesting very very interesting and very interesting and very good.”
“The first time you have a chance I want a pet please can someone please adopt us and then I’ll be your friend.”
In another, when asked to provide a summary of a marketing-related survey just taken that had nothing to do with blockchain integration or transaction security, one respondent wrote:
“Blockchain integration is revolutionizing transaction security”
You shall not pass. Removed!

The Rising Concern of Survey Fraud
Survey fraud is certainly not a new issue, but its complexity and scale have grown with advancements in technology. Fraudulent responses can come from various sources, including:
Automated Bots and AI: These are programmed to complete surveys with fake data, often using sophisticated algorithms to mimic human behavior.
Click Farms: Groups of low-paid workers are employed to fill out surveys quickly, without providing genuine or thoughtful responses.
Panel Fraud: Some respondents join multiple panels under different identities to maximize their rewards, skewing the data.
Mitigating Survey Fraud
To combat these concerns, Talk Shoppe employs a multi-faceted, holistic approach to respondent data quality.
We like to think of our approach as falling into two distinct buckets: ensuring the SAMPLE is composed of the actual people we want to hear from and that the SURVEY is measuring what we need it to measure.
First, let’s talk about the SAMPLE…
The first relates to some of the tactics that Talk Shoppe and our trusted partners take to filter out the cheaters, speeders, straightliners, bots, and fraudsters 🕵️ 🤖 🥸:
1 | Screening techniques:
CAPTCHA: Implementing CAPTCHAs can help differentiate between human and automated responses.
IP address tracking: Monitoring IP addresses to identify and block responses from suspicious sources.
Digital fingerprinting: This technique identifies unique characteristics of devices and browsers, helping to detect multiple entries from the same source.
2 | Data quality checks:
Attention: Including questions that require specific responses to ensure participants are paying attention.
Speeding: Identifying respondents who complete surveys unrealistically fast, suggesting they are not providing genuine answers.
Consistency: Comparing answers within the survey to identify contradictory responses.
Quality: Reading through open-ended questions carefully to scan for unique patterns in syntax, vocabulary, capitalization, and looking for coherent and relevant responses to questions.
3 | Panel management:
Panel profiling: Maintaining detailed profiles of panel members can help identify unusual patterns and potential fraud. Cross-referencing member information with third-party databases (e.g., LinkedIn) goes that extra step in verifying identity and ensuring human responses.
Regular audits: Conducting periodic audits of panel members and their responses to ensure compliance with data quality standards.
Geo-location verification: Ensuring that respondents are located in the target geographical area as stated in their profiles.
4 | Incentive Structures:
Fair compensation: Providing appropriate compensation for genuine responses can reduce the temptation to engage in fraudulent activities.
5 | Leveraging Technology:
AI and machine learning: Using these technologies to detect anomalies in response patterns that may indicate fraud.
Real-time monitoring: Implementing systems that monitor survey responses in real-time to identify and address suspicious activity immediately.
Behavioral analysis: Analyzing respondent behavior, such as mouse movements and keystroke patterns, to differentiate between genuine and fraudulent responses.
6 | Post-survey analysis:
Data validation: Cross-referencing survey data with known benchmarks and external databases to validate responses.
Response patterns: Identifying unusual patterns, such as straight-lining (selecting the same answer for multiple questions), which may indicate fraudulent activity.
Text analytics: Utilizing text analytics tools to assess the quality and relevance of open-ended responses.
The Impact of Survey Design
The second piece of tackling these issues is where we as Talk Shoppe researchers have the utmost responsibility in being fair to the very real humans taking our surveys - surveys must be engaging if we expect good quality feedback! Poor survey experiences are an important contributor to poor quality data. The following are some survey design best practices that the Talk Shoppe team regularly puts into action to engage our respondents:
Survey length: It has been argued that the average human attention span is at an all-time low. One professor of informatics at UC Irvine reported that her research showed that in recent years, people using their devices in their natural environments spend around 50 seconds before switching to another screen. (Is anyone still here reading this article? Congrats, you’re an anomaly!) There are published studies out there on optimal survey length, but we’ll cut to the TL;DR - shorter is better!
Question relevance: There’s nothing worse than being asked to answer questions about an experience you’ve not had. Using effective routing and survey logic, we make sure respondents only get questions meant for them, based on their previous answers.
Question clarity: In a nutshell; the easier a question is to understand and answer from a respondent’s perspective, the better we feel about the quality of the results.
Device agnostic: These days, our respondents are taking surveys in airplanes, on trains, on hikes, really anywhere! And using any device imaginable. It’s paramount that we ensure our surveys look good and are easy to take on all different types of devices. Testing before the survey launches is key!
Language and culture matter: Just as it’s important for our insights to be representative of the broader population surveyed, we must do our due diligence to ensure that translations are accurate, and that we’re asking questions and providing response options in a way that’s culturally and personally relevant for the audience taking our surveys.
Build with empathy: When we write survey questionnaires and test survey links, we have to put ourselves in our respondents’ shoes. If it’s not integral to the analysis, if it’s uncomfortable to answer, or if it feels repetitive, it’s time for some edits. Our survey respondents are giving us valuable time and feedback, and we must respect theirs in return.
So, to our clients out there - when we push back on survey length and sneak in those “quality check” questions we all know and love, or have an extra day built into the timeline for “quality checks”, this is why! The more the collective “we”, as an insights industry, have conversations around data quality, the more we’ll be able to continue pushing the needle forward on ways to combat it.
Comments