Summary: UX efforts should improve their quality of data and informed decision-making by making sure feedback is coming from the right users.
Let's talk about fake users
Fake users are the wrong people who will provide you biased, tainted or innacurate and imprecise feedback. Precision is everything if you want to be successful with UX. Getting the right users, priceless.
At Experience Dynamics we learned this from doing it the other way (Hint: Your UX Agency does it this way most likely, ask them). Like many, we used to have a Market Research firm recruit for us, now we do it all meticulously in-house.
Market research firms provide recruiting (focus groups are happening all the time). But it's also very common to have user no-shows, luke-warm relevant feedback or someone that feels like a badly hired actor. We have not experienced any of that for many years-- that's when you know you are doing it right.
So about 10 years ago, we stopped using our Market Research firm. We didn't like the quality of users they were giving us (as one of the best firms in the market).
Authenticity is critical in getting clean data. It's that simple. We are excited when our clients get the best data possible.
Who are fake users?
Anyone you use as a substitute for whatever reason to represent your real user.
- Your CEO.
- Your employees.
- Your agency who brings you the wrong panelists.
- Consumers who are only on a panel to make some money.
- Sloppily recruited respondents from your market research firm.
- Your mother.
- The guy down the hallway, in the cube next door, on floor 3.
Why are they dangerous?
1. The wrong people give you dirty UX data. Fake users tamper with your data providing unclear or foggy findings. Note: You can bias your own findings by doing this, which is why a full dedicated UX agenda, team or expert can make a quality difference.
2. Motivation, intent and emotion add big insights to your UX research. Users recruited who are not "in the act" you are observing are never as meaningful as people motivated organically by the thing they are doing-- relevant to your product. Make sure you recruit people doing what you are studying!
Experience Dynamics is often brought in to fix the damage of a brand/ interactive or marketing agency (sometimes another UX agency). We've seen your vendors do a focus group involving the CEO; we've been only allowed permission to study 'neutral' employees on a consumer product; we've been told "we cannot talk to our users" or that the advice from Steve Krug and Jakob Nielsen was to "grab someone from down the hallway".
Yes, sorry about that. In the rush to evangelize Usability and Usability Testing, a very bad habit emerged that is still tolerated in many organizations: using fake users to get insights that should come from bona-fide users.
How to avoid fake users:
- Make sure you, or your agency are recruiting users who are doing the thing (task, problem, domain) you are studying.
- Professionally screen potential users.
- Double or triple screen users and make sure they feel right for the study.
- Re-recruit or have extra people on standby to recover from a "low data" user.
- Do decent sized samples of users, none of this 5-user stuff... at minimum test 10 users, do chair-side visits with 15 users.
A Note on Sample Sizes: At Experience Dynamics our user tests are typically 10-15 users and field studies typically 20-40 users. We have found this to yield the correct amount of insights, empathy and empirical validation of business, product and UI concepts. More on that in another post. Subscribe to our Newsletter to Stay in the Loop.