Why participant recruiting is non-trivial, and why nobody is talking about this

Summary: Recruiting UX participants requires careful consideration of the users you want to involve in your study. While recruiting is commonly performed and is necessary for gaining fast access to your users, be careful of the quality of your recruits. Poor recruits mean poor insights. Aim for high quality recruit targets to keep your data sparkling clean. 

I’m recruiting users, what’s the big deal?

That’s great if you are already running user studies and recruiting UX participants. It’s a big step in itself. The problem is that over the past 15 years or so, the market has become flooded with user research testing tools and recruiting services. That means there are thousands of users who do this as a side-job. Some do it just for their $10+ incentive. This situation attracts a lot of “actor” style users who will adjust their role as “users” based on the site or app that wants their feedback. You want to avoid these ready-made users, mainly because they lack relevancy. 

In addition to the problem with the potentially sloppy UX habit of grabbing only 5 users (yep, 5 users might not be enough to test with), ad-hoc recruiting can lead to lost or skewed insights. That’s major. Why waste your time, or if you only have that small handful of users, why risk your insights because of poor recruit tactics?

Why nobody seems to be discussing this…

Data contamination is actually a big deal in UX research. Poor data captured from users can leave you empty-handed or not fully informed. It is important to understand why participant recruiting is not often discussed so you can avoid data contamination in your efforts. 

First, many UX people or groups do not do their own recruiting, so they don’t see the issue first-hand. Market research groups typically recruit for UX teams (that’s what we did for ten years), and the quality of the targets they bring you are trusted. Bottom line: You might want to scrutinize your recruits. 

Next, quality user recruiting requires lots of experience and trial and error. At Experience Dynamics, we stopped using outside parties and took recruiting in-house. We were unsatisfied with mediocre user recruits. Today, we have recruited hundreds of participants for our user research and continue to refine our recruiting tactics for even better results: few to no “no-shows”; the “right” users, rich insights, and quality data. 

Finally, gurus like Jakob Nielsen evangelized rapid user testing (the 5-user mantra) for years, emphasizing low cost and informality. As great as that was to popularize user testing, it left the quality of user issues to the sidelines*. Many start-ups and even enterprise teams (like Google) do riskily recruited “cafeteria testing” or “down the hall” testing with users. The idea is that a “user is a user is a user”. Worse, Lean UX, and authors of the book with that title– Jeff Gothelf and Josh Seiden–among others, repeat Nielsen’s advice, as does Jake Knapp (Sprint author) and Google Ventures in popularizing Design Sprints (day 5 is a test with 5 users!). Neither of these books discusses or emphasizes recruiting users who are actually doing the thing you are studying.

Note on representative users

Nielsen Norman Group does not advocate sloppy recruiting by any means. In fact, they provide a free guide with recruiting tips (PDF) that emphasizes recruiting representative users. Because many teams new to UX research don’t do this, we need to address this in user research studies ASAP.  

Lean or Agile UX is about taking calculated risks, not being risky. 

Add these quality-assurance steps to recruiting UX participants

1. Use a screener; don’t over-screen but also don’t under-screen. 

It is important to filter out the “survey takers”, or people that just do this for extra income. Note: Not all those people are low-data, but if you want to boost quality, you’re looking for organic users, not the panel people. One new technique we have successfully leveraged is to present screening question options that are “false positives”– essentially answers that are deliberately irrelevant, so that survey takers don’t know what the study is truly about. This helps filter irrelevant participants.  

2. Conduct a “fit-test” pre-interview with the user.

Pre-screeners (a screener before the official screener) as well as a screening interview can help safeguard against getting the wrong users.  Alison Gavine, Global User Research Director at Experience Dynamics, says: “Trust your instincts with users. Don’t just recruit someone who says ‘Yes’ to all your screener questions. Firstly, most good user targets are mildly interested in participating, so that’s a warning…next, when you talk to them, you ought to be able to clarify that they actually do the thing you are studying…not just telling you they are in order to join the study”. 

3. Don’t promise users a spot until they are approved by a supervisor. 

This is a good way to allow a double vetting process. It also let’s users know there is a level of scrutiny involved, and that you may have to disqualify them.  

4. Vet your sources of users. 

Is your Market Research firm using a recycled pool of users? Is the respondent desperate to participate? Are you pulling from a Participant Database or consumer panel that has lots of “full-time” user participants? All of these are red flags and mean you need to expand your reach or get creative about where you source your leads. 

5. Get creative about how and where you recruit. 

If your recruiting is turning up a mixed bag, go to where you users are:

“One of our clients spent more than nine months recruiting just 3 users and countless hours getting no results. The users were technicians who work in a food processing factory environment. To recruit them, we made up a flyer with the study details and waited for them to leave work, then approached them and asked if they wanted to help. Within 2 weeks we had recruited 10 quality users for the study, just in time to meet the tight deadline.” (Alison Gavine, Global User Research Director at Experience Dynamics).

Conclusion

I used to think recruiting UX participants was trivial. It took me many years to appreciate the nuance of getting recruiting right. I hope these tips help you improve your recruitment tactics. Teams that want the best insights from their user research ought to pay attention to this issue when conducting or managing a recruiting partner. The benefits of getting the right users with relevant exposure and experience to your UI/ UX are noticeable and will be deeply appreciated by those directly using the data to improve your organization’s products or services. 

Leave a Reply

Your email address will not be published. Required fields are marked *


The reCAPTCHA verification period has expired. Please reload the page.

Search

Search

Recent Posts

Scroll to top

Get a quote or discuss your project

Tell us about your project

Arrange a 30 min call

Project in mind?

logoblack

Fight for the rights of your users. We'll show you how.

Read more articles like this for exclusive insights into the best ways to approach UX and Service Design challenges. Find out when events occur first. Privacy protected, no exceptions.

Subscribing indicates your consent to our Privacy Policy

Should we add you to our email list?

Privacy protected-You can unsubscribe at any time.

Download the Better UX kit