I was recently discussing usability testing and our product with a friend, who asked what one can realistically glean from a few individual usability tests. After all, a small sample set is not significant, and there’s no assurance that those individuals are representative.
He indicated that he would only be comfortable with a large sample set, which could be difficult to assess using a tool such as TryMyUI. While Nielsen et al have found that in many ways the optimal number of users is 5 or so, I tried to provide a more quotidian response.
Traveling outside of your mind
Imagine you visit a foreign country, say China or Mongolia or the Middle East, and find yourself in conversation with one or two local individuals. Can you learn anything from these conversations? After all, who’s to say they are representative of prevailing attitudes?
Of course, the answer is yes, you can learn something, because the exposure to a different perspective is often educational, and may contain many “aha” moments. This is particularly true politically, because we approach global political issues with a lens shaped by our media, political environment, and so forth, which is quite different from that of other countries. That is, our mental model is quite different from that of others. Similarly, albeit usually not to the same degree, we design websites with our mental models, which are not necessarily the same as that of our target audience. After all, we are experts both in the domain and in the technology.
Even a few narrated user testing videos can be instructive in showing us how well our mental models/information architecture map to what is expected by our audience. It may not be scientifically “valid” to make judgments based upon a couple of data points, but what we’re doing is not “scientific,” it’s softer and more creative and deductive.
Check out a user testing example to see what you can learn from a usability study.
Trymata
Hi Farhad, Good article, I did an article comparing usability testing techniques (eye trackinv vs mouse tracking) and one of the greates things favouring mouse tracking is the unbound sample set (basically all your real users). You may find it interesting:
http://www.picnet.com.au/blogs/Guido/post/2010/02/03/Mouse-Tracking-vs-Eye-Tracking.aspx
This is more ‘creative development’ than quantitative testing and should be viewed in that manner.
I was a bit skeptical about the idea of testing just 5 users but, i have to admit, the analogy with meeting foreigners and being able to learn something from them is very convincing, thank you! And, surely, there are tools, like open card sorting where more than 10 participants truly is a crowd .