6 Challenges in Usability Testing and Ingenious Ways to Fix Them

Question: How many usability testers does it take to screw in a light bulb?

Answer: Are we sure the users need a new light bulb? How do they feel about their experience in the dark? Can they show us how they go about turning on the light?

And with that, we’ve set the stage to ensure how chaotic, complex, and (sometimes) thankless usability testing can be. Besides their razor-sharp observation skills, people in this profession need to develop a robust sense of humor to keep their sanity in check and their focus unwavering.

Here are 6 challenges that expectedly or surprisingly pop up during most UT sprints. We’re sharing ways to ensure you’re always on top of them.

6 Usability testing challenges & their fixes

Getting the right user sample

The success of any usability testing sprint hinges on the composition of the user sample. However, there is a lot of noise surrounding this particular step. Do you need a large sample size? Can a 5 user sample suffice? Does it differ when testing enterprise and commercial workflows?

Well, larger isn’t always better in case of sample sizes for usability testing as much as choosing the right type of users.

In larger companies, it may happen that you’re given a random bunch of people (who were “available”) to participate in the test. However, usability testing can only provide the right insights if it’s run with the users meant to use the application. The wrong sample can cause the results to be fudged up and derail the design directions from there on.

Here’s the fix –

• Provide clear requirements about the type of users required and the ones to be avoided.
• Come to an agreement with the stakeholders to define the right user sample.

Picking the right task to test

“Under testing is a sin, over-testing is a crime”
There is a method for choosing which elements are to be tested. Framing your test is critical; for example, the requirements for testing a new feature differ from testing an updated feature. Every task to be tested calls for a different set of templates, questions, and approaches.

Running a usability testing sprint without defined objectives ruins the whole point of the exercise and provides no value. It can result in the wrong workflow/feature being adopted, which can do no good for the product experience.

Here’s the fix –

• Define the overall goal of your usability test (why are you conducting the test – what questions are you trying to answer?).
• Define the ideal outcome and its consequences on the product experience (what impact is the test expected to have – what are your priorities?).
• Lay down guidelines to assess the changes that can be realistically implemented.

Picking the right task to test

Successfully crafting, conducting, and analyzing usability tests calls for careful planning, and there are several challenges that you may stumble across. For example, kicking off a UT project with hazy objectives guarantees you will not get value from the testing. It can spiral into a long line of wrong decisions being taken at each stage – the incorrect test approach being adopted, the wrong questions being asked, and the wrong test sample being chosen – all of which can result in meaningless outcomes.

Usability testing is not a method employed to jazz up the look & feel of the application; it is to zero in on the part where the user gets frustrated, either with design or with functionality.

Here’s the fix –

• Let the user lead the testing. Let them convey what matters to them and allow them to explore naturally.

Embracing the pressure to deliver

Deadlines, KPIs, close deals – everyone in usability testing is familiar with these pressures. To say that UT can be chaotic and stressful is an understatement. There is always a rush – a rush to push prototypes into demos, the rush from PMs looking for validation – and it can be overwhelming.

Realistically, this seems to be a staple of the process, and try as one might, it’s impossible to imagine conducting UT in an easy-breezy manner. There are some things worth trying, though.

Here’s the fix –

• Set the right expectations for stakeholders, and provide a realistic view of the outcomes, be negative or positive.
• Organize a roadmap to temporally focus on behind-the-scenes features and then actually conduct lightweight user testing to build a backlog of validated features ready for development. This way, the UT process kicks off without causing development delays.

Dealing with false confidence

Conducting UT in an enterprise setting can be a double-edged sword. On one hand, you may encounter stakeholders who moonlight as subject matter experts (SMEs), positioning themselves as end-users. On the other hand, we felt too comfortable making decisions on behalf of our users thinking we knew what they wanted. While their inputs can help clarify the complexities of the workflows, it does not replace the need to validate ideas through user testing.

Here’s the fix –

• Demonstrate how stakeholders are different from end-users.
• Let stakeholders/SMEs sit in as note takers so they can view the experience an actual user has with the product. Nothing drives home the need for UT more than seeing someone struggle with something you originally thought was obvious and easy to use.

Getting organizational buy-in

Saving the best (read: most complex) challenge for last. Usability testing has its roots in human-centric design. It requires an innate sense of empathy and understanding of user needs and behavior – two things that can be difficult to articulate and easy to dismiss.

Any sort of user test loses value in the face of skepticism from the decision-makers. Test results and design directions will be disregarded and implementing change will be an uphill battle.

While this has more to do with the level of UX maturity of the company in question, it is possible to make headway if an opportunity presents itself.

Here’s the fix –

• Engage key stakeholders right from the start. Ensure their participation and input to build testing objectives.
• Involve stakeholders to observe/participate in UT sessions. Observing users struggle is undeniable evidence of usability issues.
• Ensure their participation in discussing the test findings. Each observer has their own set of takeaways. Getting them to discuss the outcomes while the evidence is still fresh is a good way to get them on board.

And as we conclude,
Question: How many usability testers does it take to change a light bulb?

Answer: None. Testers do not fix problems; they just find them.

Got any usability testing challenges you overcame? Do share your insights in the comments!