Quantitative and qualitative testing

Collecting data to inform and improve the user experience design is a never-ending need, for companies from tiny start-ups to large enterprises. There’s much healthy debate on what UX research or usability testing methods should be used, and when.

When it comes to quantitative versus qualitative research, best practices suggest a mix of qualitative and quantitative approaches to capture a holistic perspective. The challenge, however, is how to combine both in a cost-effective, productive way to obtain data to drive user-centered design.

What’s the difference?

The difference between quantitative and qualitative approaches is often explained using contrasting terminology like hard vs soft, numeric vs descriptive, statistics vs insights, measure vs explore, what vs why. These quick contrasts are useful to highlight the strengths and limitations of each approach if used alone without any of the other.

Qualitative tells stories.

At the core of user experience is the subjective, emotion-based response of the individual user – the way a website makes visitors feel. These feelings can range from delighted, impressed, or hooked, to confused, frustrated, and angry. All these welling emotions, and the ones in between, have one thing in common: they won’t show up in the numbers.

Quantitative data can tell you which pages people visited, and how long they stayed, and where they came from and where they went to, but the story itself is missing; the feelings aren’t there.

The user that clicked to your registration page and then left without signing up – what kept them from continuing on? The users staying on the homepage for so long, was it because their interest was captured by great content, or because they were fruitlessly searching around for an About section? When someone clicks to a new page, did they move a step closer to their goal or did they find that they had mistaken the page for something it didn’t provide?

Listening to a user narrate their journey, hearing their reactions as they navigate a site, fills in those blanks. The ups and downs, the irritations and confusions, the aha moments, the satisfaction of a task completed, all come together to tell a story, and the best and worst things about your design stand out like warm bodies through infrared goggles. Qualitative feedback tells “why” at a level that quantitative data cannot reveal.

Quantitative gives context.

Quantitative feedback allows you to understand your site’s usability in the context of a much bigger picture.

Unlike qualitative information, it can be used to make easy, reliable comparisons. User testing tools like SUS (System Usability Scale), the SEQ (Single-Ease Question), and task durations and completion rates that measure and quantify usability can map the individual’s user experience, chart usability increases and decreases over time, and show how your website performs compares to other sites.

For this reason, quantitative data has a unique capacity to persuade. It shows stakeholders and decision-makers what’s working and what’s not, and demonstrates, with numbers, undeniable disparities in performance. That gives it the capacity to strengthen and justify qualitative findings. Then it can be used to set predictable objectives for a new design sprint.

Combining qualitative and quantitative in your research

The challenge with getting quantitative data is that it requires scale for the numbers to mean anything. But user testing is a research method that relies on video data, and video analysis is hard to scale up.

The key is to read between the lines of your quantitative data. Video may not scale well, but if you know how to use all the usability testing data you’ve collected, it’s a lot easier to handle, and can be dissected efficiently even at a large scale.

Data like task duration or task completion are more than just statistics or benchmarks to show higher-ups. Which task took the longest to complete? Which could not be completed? Was it longer than you expected? Look for users whose times/completion scoring stand out from the rest; an unusually long task duration could indicate a user who struggled with the task.

Both of these statistics even provide a shortcut timestamp to launch user videos immediately on the task timestamp so researchers can quickly get to the bottom of what went wrong.

Now you have a starting point to tackle any mass of video data: instead of randomly picking one of your user videos and watching it beginning to end, pick the user that struggled most, and skip to the task they had trouble with.