Summary: Surveys are often over-used or misplaced as a research technique that many teams rely on to figure out usability and user experience problems. Understand the limits and opportunities of the tool and instead either don’t lean on it too heavily or use it as an adjunct to reach a wider audience–as part of a qualitative user research project (user interviews).
Why surveys fail you for UX insights
Surveys are great, except when they are not. Surveys are often used as a way to gather usability feedback. This makes sense; after all, questions like: “Do users use your site or app? Do they find it valuable? What do they want?”…are perfectly reasonable questions to ask your users. So, what is the problem with surveys for user experience questions?
First, be clear that surveys are a market research tool. That means they are not entirely appropriate as a usability or user research tool.
The problem with surveys?
1) They are quant-itative, and most UX research is qual-itative or insight, not numbers driven.
2) They do not reveal context. UX is about understanding tasks and context of experience: What?; Why?; How?; When?; Where? etc.
3) Surveys are used as a replacement for user behavioral observation and empathy gathering. This is perhaps the biggest issue with surveys.
We’ve worked with many teams that often use surveys exclusively as a way to probe around the iceberg of usability issues. You know who you are. We’ve all done it, and some of us still do it (including UX Designers and UX Consultants!). When you are using just a survey for your usability requirements, it’s not going to help you solve your real UX/UI issues.
Wait, are surveys good for anything in UX?
Oh yes, surveys are fine IF, and only if, you are using them in conjunction with qual-itative user research. They provide a nice quant-itative contrast to contextual field data usually motivated by user emotions. At Experience Dynamics, we discovered years ago that pairing online surveys with user research provides a rounded insight— and it does not cost that much. Plus, tools today like SurveyMonkey offer logic-branching, which allows for conversational-style surveying and richer data. You can ask those “whys” more quickly, though you have to be careful not to over-ask, and of course, you might get back short, snippy answers– instead of stories.
So surveys are story-less. If you blend user research with surveys (good idea), you start to notice this phenomenon, and you start realizing that surveys are almost dangerous on their own.
Oh, what else are surveys good for? Getting an initial idea of how bad things are on your site or app. Except, sometimes they don’t reveal the actual pain– but they might give you clues. So they are a starting point. Just don’t stop with surveys.
How to get better UX insights from surveys
If you’ve been reading above, we already gave you the answer: blend them with user interviews! UX evangelists are known for educating us that stories and story-telling are critical to UX. They are how we learn, how we bring back the empathy, and how we transmit data from the field– and from user’s drawers, closets, cabinets, desktops, app screens, and of course, thoughts, feelings, needs and habits.
If you are using surveys in your team and still scratching your head or happy with “we asked our users”, stop and start planning chairside or home visits with users. An extra tip is to start with a survey and use it to recruit people for your one-to-one visits.
Be clear that your survey data and your in-person contextual interviews are different types of data that can help each other. In particular, use survey data to reach out to a wider audience, thousands instead of dozens of users.
Conclusion:
Surveys are excellent for collecting quantitative data from a large sample, but they may lack depth in understanding user emotions and motivations. By combining surveys with other research methods like interviews and the power of Ethnography, you can achieve a more comprehensive view of the user experience.
Go Deeper:
Check out this Masterclass: Using Surveys for UX like a pro