Have you ever wondered how many people you need for a UX research study? It’s important to know the right sample size, because it can make all the difference in reaching accurate conclusions.
In this blog post, we will discuss what factors influence the number of participants needed and how to calculate the right amount.
What is user experience research?
Why do companies conduct user experience research?
Parameters influencing sample size
Suggestions for the right sample size
Final thoughts
What is user experience research?
User experience (UX) research is a way of understanding how people feel about a product or service. It’s the process of collecting and analyzing data about users, with an emphasis on measuring usability: how easy it is to use something for its intended purpose (a website, app, or any other interactive product or system).
UX research includes:
- Generating ideas and brainstorming solutions
- Conducting market studies and competitive audits
- Evaluating user satisfaction through surveys and interviews
Why do companies conduct user research?
- To validate their assumptions before investing in development. This helps minimize wasted time and money due to incorrect decisions.
- To get early feedback from real users so they can iterate based on actual needs rather than imagined ones. This speeds up the design cycle by up to 50%, which reduces costs.
- To conduct market research on their product or potential new products within the same category (features, user groups, and more) so that they can improve and optimize what is already there before releasing something brand-new that may not meet expectations.
One of the most common questions people ask when it comes to UX research is: “How big should my sample be?”
Read more: Top 6 UX research trends impacting retail businesses
Parameters influencing sample size
Many factors influence this decision, but one thing remains true regardless. If you want meaningful results with statistical validity & significance, there are well-defined minimum sample size thresholds that you need to meet.
These thresholds are different depending on the contents and aims of the study, as well as the research approach. But whatever the study, there is usually something like a “right answer” to the question of sample size, found and proven by research professionals.
For example, a basic survey for collecting quick, mainly quantitative feedback generally requires a minimum of 100 participants. That’s quite a bit more than what the minimum threshold would be for a qualitative usability test on an iteration of a product design that’s already been on the market (which would usually be 5+ participants).
And then, of course, there are several other minimum thresholds in between these examples. While it’s often repeated as a rule that 5 users is sufficient for any user testing study, it’s not actually quite that simple. Sometimes that number might be 12, or 25, or something else!
Read more: How many users for my user test?
If you’re new to research and feeling daunted at recruiting 5, 12, 25, or 100 people, consider partnering with an experienced agency or provider that can guide you through the process.
You may also consider using online tools such as Trymata (a cloud-based platform for remote usability testing). It allows researchers to run tests with any sample size of their choosing – a trial account comes with 5 testers to run your first study. In addition to handling recruiting for you, it also includes analytical features that will help simplify your research efforts while saving time & money!
Our purpose is to provide an affordable solution that unlocks the full potential of UX research by allowing companies to perform studies without breaking their budgets or becoming data scientists.
The consequences of sample size
By declaring only a certain amount of people out of the total population as test subjects, we are limiting the number of people who can use your system. As a result, you miss out on valuable information and/or data points that could lead to greater insights.
When examining two or more innovative solutions, it’s common to compare based on behavioral characteristics like accuracy, speed, and ease of use.
It’s difficult to establish who the champion was without a full understanding of what success involves. The sample size you use is determined by the magnitude of the variation you want to discover.
This is just a sample size calculator. It helps you determine the number of test subjects you need to achieve statistically significant results. It can help you find out how many participants you require for your particular situation.
The ‘right’ answer depends on several factors including budget, resources (time), type of product being tested (fast vs robust), and what success looks like. The more complex and variable the design space becomes, the more participants you need to test.
In this case, statistical significance is very important. Because we want results that will have a large impact on how our product evolves in the future.
Failure to meet these criteria will result in a weak signal that can be misleading when making decisions about design changes.
Better safe than sorry.
Suggestions for the right sample size
The sample size depends on various factors as we discussed above. Here are a few steps to determine the sample size for your UX research study.
The first step is to clearly define the problem or research objective.
The second step is to determine what data points you need for your study, and how you will use them in formulating a solution.
The third step is to decide on the statistical significance of results that meets your budget constraints.
In addition, it also helps if you have participants from various geographical locations as well as demographic backgrounds. This will improve generalization of results across users, resulting in better insights.
Sample size depends on many factors. Here are some commonly cited rules of thumb:
- “For quantitative studies collect at least 30 responses per variable (independent/dependent)” – rule proposed by Jacob Cohen.
- “For qualitative studies aim for at least five participants per each interview” – suggested by Miles and Huberman.
- “For usability studies aim for around fifteen users to get enough data points (30 is ideal)” – suggested by Rubin, Chisnell & Jared Spool.
- More participants often lead to better data.
- Including more people who are representative of your target users is always a good idea. It’s important to keep in mind that the best number for you will depend on certain criteria. As in, what you’re trying to accomplish with user research, how much time and money you have available, and more.
There isn’t one perfect sample size, but ultimately there need to be enough participants so that results can be generalized across all your customers/users.
Usually, 30 responses per variable should provide adequate generalization (in other words the minimum needed to make this assumption).
But it depends.
Conditions, caveats, and clarifications
Sometimes qualitative studies need fewer participants than quantitative, because certain types of insights require more in-depth understanding.
If you need to test a hypothesis, then you can use power analysis to determine how many participants you require.
One of the most important factors in determining sample size is what your resources allow for. If you don’t have enough money or time, then testing with fewer people will have to suffice.
Doing this may introduce more risk into testing, and it’s unlikely that results will generalize, which could limit future studies.
This would mean using other methods like triangulation (testing multiple solutions against each other) instead of A/B tests.
Testing smaller samples also means running fewer tests overall because there isn’t enough data to get statistically significant findings. You should work out all these numbers yourself along with everything else before you start testing.
The risk of doing UX research work with smaller samples is that results may not be generalizable. This means it might not apply to other people and the same types of problems in different contexts.
It would mean using triangulation instead which involves testing multiple solutions against each other, as opposed to just A/B tests for example where only one solution needs to be tested at a time.
Testing smaller sample sizes also means running fewer studies overall because there isn’t enough data to get statistically significant findings. Hence, think about everything involved very carefully beforehand.
To address concerns with usability testing, the sample size needed would be less, usually around 30-40 participants for a good experience.
For remote usability testing, where you have multiple locations being tested at once, test with 15-20 users per location to ensure that the data is representative of their behavior and feedback.
You should be able to get statistically significant results from smaller sample sizes. However, it would take longer so think about your research aims carefully before proceeding.
Additionally, these tests run over one day rather than two which can affect what happens during them if people are tired they might not give useful feedback, and so on.
Therefore, again just bear this in mind when running small studies like this or consider testing with more users instead.
As always, let’s remember that no matter how many users we talk to there is never 100% certainty about what we find out. If you want to be confident in your results then running a larger study is your best option.
Read more: How to perform UX research like a pro
Final thoughts
The right sample size for a UX research study is one that provides the right amount of data to answer your research question.
With the help of this article, you will now be able to determine the right size for your next UX research study.
Niraj is co-founder and Head of Engineering at Chisel Labs, a premiere agile product management software company that brings together roadmapping, team alignment, and customer connection. Niraj is passionate about building scalable infrastructure and systems and he also happens to be a huge fan of Cricket!
