Summary: Surveys are often over-used or mis-placed as a research technique that many teams rely on to figure out usability and user experience problems. Understand the limits and opportunities of the tool and instead either don’t lean on it too heavily or use it as an adjunct to reach a wider audience–as part of a qualitative user research project (user interviews).
When Surveys Fail You
Surveys are great, except when they are not. Surveys are often used as a way to gather usability feedback. This makes sense, after all questions like: Do users use your site or app? Do they find it valuable? What do they want? are perfectly reasonable questions to ask your users. So what is the problem with surveys for user experience questions?
First, be clear that surveys are a market research tool. That means they are not entirely appropriate as a usability or user research tool. Why?
1) They are quant-itative, and most UX research is qual-itative or insight, not numbers driven.
2) They do not reveal context. UX is about understanding context of experience: What?; Why?; How?; When?; Where? etc.
3) Surveys are used as a replacement for user behavioral observation and empathy gathering. This is perhaps the biggest issue with surveys.
We’ve worked with many teams that often use surveys exclusively as a way to probe around the iceberg of usability issues. You know who you are. We’ve all do it, some of us still do it (including UX Designers and UX Consultants!). When you are using just a survey for your usability requirements, it’s not going to help you solve your real UX/UI issues.
Wait, are surveys good for anything?
Oh yes, surveys are fine IF, and only if, you are using them in conjunction with qual-itative user research. They provide a nice quant-itative contrast to contextual field data that is usually motivated by user emotions. At Experience Dynamics, we discovered years ago, that pairing online surveys with user research provides a rounded insight— and it does not cost that much. Plus tools today like SurveyMonkey offer logic-branching, which allows for conversational-style surveying, and richer data. You can ask those “why’s” more easily, though you have to be careful not to over-ask, and of course you might get back short, snippy answers– instead of stories.
So surveys are story-less. If you blend user research with surveys (good idea), you start to notice this phenomenon and you start realizing that surveys are almost dangerous on their own.
Oh, what else are surveys good for? Getting an initial idea of how bad things are on your site or app. Except, sometimes they don’t reveal the actual pain– but they might give you clues. So they are a starting point. Just don’t stop with surveys.
What to do with surveys instead
If you’ve been reading above, we already gave you the answer: blend them with user interviews! UX evangelists are known for educating us that stories and story-telling are critical to UX. They are how we learn, how we bring back the empathy and how we transmit data from the field– and from user’s drawers, closets, cabinets, desktops, app screens and of course thoughts, feelings, needs and habits.
If you are using surveys in your team and still scratching your head, or happy with “we asked our users”, stop and start planning chairside or home visits with users. An extra tip is to start with a survey and use it to recruit people for your one-to-one visits.
Be clear that your survey data and your in-person contextual interviews are different types of data that can help each other. In particular use survey data to reach out to a wider audience, thousands instead of dozens of users.
Now, would you like to take a survey?
