If you found yourself nodding in agreement to last week's post -- about the differences between good online respondents and bad ones -- you're probably looking for some tips about how to get more of the good. We've identified three potential problem areas, and how to take measures to improve the effectiveness in each one.
1. Consider your sources
Where are your respondents being recruited from? Broadly speaking, the options are to recruit from
- Lists: your company's own list, a purchased list, a custom-created list
- Panels: pre-screened respondents who have agreed to participate in research, usually for some form of compensation, and about whom certain demographic information is known (age, occupation, income level, etc.)
If budget and criteria allow, we prefer to recruit respondents from lists, ideally with a phone conversation being part of the recruiting process.
Sometimes panels are necessary, for example when rapid survey turnaround is needed or sometimes for cost reasons. But the hard truth is that panel members are often motivated primarily by incentive rather than being interested in the brand or the topic at hand. This leads to poor engagement and requires careful recruiting and management.
To mitigate the issues of panels, here are some rules of thumb:
- Use panels only when necessary
- Ensure quality panelists are selected using the screener methods outlined below
- Communicate clearly to panelists that your online study is a qualitative study that last a longer period of time, so that they have appropriate expectations
- Screen your panel company with some of the questions outlined by ESOMAR -- the quality of panels is HIGHLY variable
2. Scrutinize participants
The problems at the participant-level were outlined last week. To summarize,
- Some respondents who are recruited have poor written communication skills
- Some are not overly engaged / do not care about topic / brand
- Some are not very tech savvy and cannot take screenshots, copy and paste URL links, or share photos through the community
A comprehensive online screener needs to include the following:
- "Test" of written language abilities. Implement an open-ended response that requires a paragraph answer -- not to pick the most eloquent writers but simply to make sure their responses will be understandable.
- Questions about technical abilities. Ask respondents if they’re comfortable taking screenshots, copying and pasting through their phone, and sharing photos. Inform respondents that they will be asked to do so.
- Assessment of likelihood to be engaged. Ask respondents how interested they are in the topic at hand, the extent to which they like to share their opinions (e.g., post reviews of products/services online), whether they consider themselves creative and willing to share ideas, and the extent to which they’ may be a ‘lead user’ (those who face problems with a product ahead of the market and want to find solutions)
3. Incentive -- not too high!
If you only listen to the companies that are paid to recruit respondents, you might start to believe that "you get what you pay for." They'll tell you the way to get better information is to pay the respondent more money.
There is some truth in that. If a respondent feels like they are spending more time on our online study than they thought they would have to spend, and the compensation is low, you're likely to get cursory responses and little followup.
At the same time, though, there's an interesting paradox. We have found that if incentives are too high it also creates a problem. A high incentive can attract the least desirable study respondent, the type who will say anything to get into the study just so that they can do as little as possible to get the compensation.
The sweet spot, then, is a dollar amount where participants feel adequately respected for the time they are spending to participate in the study, but is not so high as to tempt in the riffraff. What that dollar amount actually is varies widely. A study of lower-income homeowners will not need to pay anything near what is needed for a study of corporate tax lawyers. But in either case, that balance must be found.
