Summary: UX Practitioners, and more importantly stakeholders must avoid making the mistake of over-generalizing or incorrectly observing user test sessions. Since the purpose of usability testing is to create informed design decisions, this is not a trivial issue. Poor (biased) observations are rampant. Let's fix this now!
Avoiding These User Testing Observation Traps...
Observing a usability test seems straight forward: watch and form an opinion. It is that simple, but if you are not careful of the way your mind works, you can end up with the wrong set of take-aways-- a potential disaster to what otherwise could be a rich learning experience. Here are the main things you need to avoid and be aware of, to get the most from your observations:
Summary: Emotion is not to be underestimated in human decision-making, as recent US elections and Twitter's analysis of election tweets show. In the same way candidates with the strongest emotional appeals get voters attention, so do designs. Emotion is important to confront in a design, but also should be measured and used as a tool to gain deeper user engagement.
As this analysis from Twitter shows, Trump won emotion. Politically he utilized all emotion (good and bad) to attract attention to his "fix America" message. His use of emotion attracted many voters, unpredicted by US media polls. So what does this mean for your user experience?
Summary: Hiring the wrong UX designer or UX agency can cost you, not only in your delivery but also in your confidence to benefit from the latest in UX leadership and ROI. Making the right hire is critical, but first you have to be able to spot weakness which is difficult if you are new to the subject of UX hiring.
Characteristics of the Wrong Hire
In our experience of working with clients who complain of being burned by low quality UX firms, we've noticed some patterns over the past 15 years.
Summary: Doing regular, phased or iterative usability testing gives you the benefit to gain additional validation on design direction. It's primary function is to protect usability quality as designs progress or get distorted by internal constraints. Regular testing can help you establish maturity or UX culture and can teach you the important UX skill of user advocacy.