I was recently discussing usability testing and our product with a friend, who asked what one can realistically glean from a few individual usability tests. After all, a small sample set is not significant, and there’s no assurance that those individuals are representative.
He indicated that he would only be comfortable with a large sample set, which could be difficult to assess using a tool such as TryMyUI. While Nielsen et al have found that in many ways the optimal number of users is 5 or so, I tried to provide a more quotidian response.
Traveling outside of your mind
Imagine you visit a foreign country, say China or Mongolia or the Middle East, and find yourself in conversation with one or two local individuals. Can you learn anything from these conversations? After all, who’s to say they are representative of prevailing attitudes?
Of course, the answer is yes, you can learn something, because the exposure to a different perspective is often educational, and may contain many “aha” moments. This is particularly true politically, because we approach global political issues with a lens shaped by our media, political environment, and so forth, which is quite different from that of other countries. That is, our mental model is quite different from that of others. Similarly, albeit usually not to the same degree, we design websites with our mental models, which are not necessarily the same as that of our target audience. After all, we are experts both in the domain and in the technology.
Even a few narrated user testing videos can be instructive in showing us how well our mental models/information architecture map to what is expected by our audience. It may not be scientifically “valid” to make judgments based upon a couple of data points, but what we’re doing is not “scientific,” it’s softer and more creative and deductive.
Check out a user testing example to see what you can learn from a usability study.