The following is a guest post by Niraj Rajput, cofounder and Head of Engineering at Chisel Labs
Have you ever wondered how many people you need for a UX research study? It is important to know the right sample size, because it can make all the difference in reaching accurate conclusions.
In this blog post, we will discuss what factors influence the number of participants needed and how to calculate the right amount.
What is user experience research?
Why do companies conduct user experience research?
Parameters influencing sample size
Suggestions for the right sample size
Final thoughts
What is user experience research?
User experience (UX) research is a way of understanding how people feel about a product or service. It’s the process of collecting and analyzing data about users, with an emphasis on measuring usability, how easy it is to use something for its intended purpose (a website, app, and so on).
UX research includes:
- Generating ideas and brainstorming solutions
- Conducting market studies and competitive audits
- Evaluating user satisfaction through surveys and interviews
Why do companies conduct user research?
- To validate their assumptions before investing in development. This helps minimize wasted time and money due to incorrect decisions.
- To get early feedback from real users so they can iterate based on actual needs rather than imagined ones. This speeds up the design cycle by up to 50%, which reduces costs.
- To conduct market research on their product or potential new products within the same category (features, user groups, and more) so that they can improve and optimize what is already there before releasing something brand-new that may not meet expectations.
One of the most common questions people ask when it comes to UX research is: “how big should my sample be?”
Read More: Top 6 UX Research Trends That Is Impacting Retail Businesses
Parameters influencing sample size
Many factors influence this decision, but one thing remains true regardless. If you want results with statistical validity & significance then your study needs at least 100 respondents! It sounds like a lot but keep in mind that these numbers come from real-world studies. These studies are conducted by professionals who know what they’re doing.
Do you mean at least 100?
No way.
I can only work with a handful of users right now. I don’t need that many for my study.
That’s okay, but keep in mind you’re sacrificing valuable insights by not including enough respondents to reach statistical significance (95% confidence level).
If you don’t have that many people on your team or you’re new to research, then consider partnering with an experienced agency that can guide you through the process.
You may also consider using the product, Loop11 (a cloud-based platform for visual usability testing). It allows researchers to test with just five respondents. Also, it contains features that will help simplify your research while saving time & money!
Our purpose is to provide an affordable solution that unlocks the full potential of UX research by allowing companies to perform studies without breaking their budgets or becoming data scientists.
By declaring only a certain amount of people out of the total population as test subjects, we are limiting the number of people who can use your system. As a result, you miss out on valuable information and/or data points that could lead to greater insights.
When examining two or more innovative solutions, it’s common to compare based on behavioral characteristics like accuracy, speed, and ease of use.
It’s difficult to establish who the champion was without a full understanding of what success involves. The sample size you use is determined by the magnitude of the variation you want to discover.
This is just a sample size calculator. It helps you determine the number of test subjects you need to achieve statistically significant results. It can help you find out how many participants you require for your particular situation.
The ‘right’ answer depends on several factors including budget, resources (time), type of product being tested (fast vs robust), and what success looks like. The more complex and variable the design space becomes, the more participants you need to test.
In this case, statistical significance is very important. Because we want results that will have a large impact on how our product evolves in the future.
Failure to meet these criteria will result in a weak signal that can be misleading when making decisions about design changes.
Better safe than sorry.
Suggestions for the right sample size
The sample size depends on various factors as we discussed above. Here are a few steps to determine the sample size for your UX research study.
The first step is to clearly define the problem or research objective.
The second step is to determine what data points you need for your study and how you will use them in formulating a solution.
The third step is to decide on the statistical significance of results that meets your budget constraints.
In addition, it also helps if you have participants from various geographical locations as well as demographic backgrounds. This will improve generalization of results across users, resulting in better insights.
Sample size depends on many factors. Here are some commonly cited rules of thumb:
- “For quantitative studies collect at least 30 responses per variable (independent/dependent)” – rule proposed by Jacob Cohen.
- “For qualitative studies aim for at least five participants per each interview” – suggested by Miles and Huberman.
- “For usability studies aim for around fifteen users to get enough data points (30 is ideal)” – suggested by Rubin, Chisnell & Jared Spool.
- More participants often lead to better data.
- Including more people who are representative of your target users is always a good idea. It’s important to keep in mind that the best number for you will depend on certain criteria. As in, what you’re trying to accomplish with user research, how much time and money you have available, and more.
There isn’t one perfect sample size, but ultimately there need to be enough participants so that results can be generalized across all your customers/users.
Usually, 30 responses per variable should provide adequate generalization (in other words the minimum needed to make this assumption).
But it depends.
Sometimes qualitative studies need fewer participants than quantitative, because certain types of insights require more in-depth understanding.
If you need to test a hypothesis, then you can use power analysis to determine how many participants you require.
One of the most important factors in determining sample size is what your resources allow for. If you don’t have enough money or time, then testing with fewer people will have to suffice.
Doing this may introduce more risk into testing, and it’s unlikely that results will generalize, which could limit future studies.
This would mean using other methods like triangulation (testing multiple solutions against each other) instead of A/B tests.
Testing smaller samples also means running fewer tests overall because there isn’t enough data to get statistically significant findings. You should work out all these numbers yourself along with everything else before you start testing.
The risk of doing UX research work with smaller samples is that results won’t be generalizable.
This means it might not apply to other people and the same types of problems in different contexts.
It would mean using triangulation instead which involves testing multiple solutions against each other, as opposed to just A/B tests for example where only one solution needs to be tested at a time.
Testing smaller sample sizes also means running fewer studies overall because there isn’t enough data to get statistically significant findings. Hence, think about everything involved very carefully beforehand.
To address concerns with usability testing, the sample size needed would be less, usually around 30-40 participants for a good experience.
For remote usability testing, where you have multiple locations being tested at once, test with 15-20 users per location to ensure that the data is representative of their behavior and feedback.
You should be able to get statistically significant results from smaller sample sizes. However, it would take longer so think about your research aims carefully before proceeding.
Additionally, these tests run over one day rather than two which can affect what happens during them if people are tired they might not give useful feedback, and so on.
Therefore, again just bear this in mind when running small studies like this or consider testing with more users instead.
As always, let’s remember that no matter how many users we talk to there is never 100% certainty about what we find out. If you want to be confident in your results then running a larger study is your best option.
Read More: How To Perform UX Research Like A Pro?
Final thoughts
The right sample size for a UX research study is one that provides the right amount of data to answer your research question.
With the help of this article, you will now be able to determine the right size for your next UX research study.
Niraj is co-founder and Head of Engineering at Chisel Labs, a premiere agile product management software company that brings together roadmapping, team alignment, and customer connection. Niraj is passionate about building scalable infrastructure and systems and he also happens to be a huge fan of Cricket!
