There’s something of a cap on usability testing: as the number of tests you run increases, eventually you’ll have more footage than you have time to watch. Especially if you don’t have a UX department dedicated solely to doing usability research and watching user testing videos, that cap can be pretty low. You know usability testing is important, but you just don’t have the time to do it at scale.
But what if you didn’t have to watch all the videos to get the same information out of them?
In recent years, the power of the crowd has been harnessed to solve problems in more and more fields. Now, we’re bringing that same power to bear on usability research, letting the crowd do the bulk of the work so that you can scale up the number of tests you run and get useful, manageable, quality feedback.
The UXCrowd
We’re currently building the UXCrowd, a tool for using the crowd to build a UX roadmap:
At the end of every test, the user will be prompted to name 3 things they liked about the site, 3 things they disliked about the site, and up to 3 ideas they have for improving it. After submitting their answers, they will be shown the submissions of other testers, which they will vote up or down depending on their level of agreement, and will be able to add comments to others’ answers if they choose.
What you’re left with after running a batch of user tests, then, is a complete, condensed, and prioritized list of all of the relevant and useful content of your videos. The ideas with the most votes are right at the top for you, so you can quickly see where people had the most issues, or what people would most like to see on your site, as well as a vote count showing how controversial each submission was, and any additional comments or insights other users had.
Ideas descend from most votes to least votes – not only will you see the greatest problems (and strengths) of your site, but you’ll also see which complaints aren’t really an issue.We run lots of usability tests ourselves on a number of different websites, and almost always there’s a range of small problems that only one user complains about – sometimes even the same thing that another user was happy with. The vote count will make it easy to tell if these are real issues that need to be fixed, or if most users disagreed that it was a problem.
From there, it’s easy to build a concrete plan for improving your website’s UX and fixing usability issues. You already have a complete list of big problems, medium problems, small problems, and non-problems, plus interactions to definitely keep and interactions that need a little extra work. You may even get some new ideas for features or interactions that you had never thought of before. The crowd has a knack for innovation.
Also, the submitter of each idea is listed by name, so you can go back to their particular video and watch them run into that issue or see the interaction they suggested an improvement for. The same goes for comments and commenters. The videos are still very important for getting a genuine, personal view of your user experience, but now you can use them in a more targeted, precise way.
Scaling up your usability research
The UXCrowd will make scaling up your usability testing efforts feasible and even increase the returns of doing so. Larger numbers of users will improve the accuracy of the voting system and differentiate more between the different ideas submitted, making it a lot easier to build your UX roadmap.
Not only that, but scaling up your testing immensely improves the reliability of quantitative measures such as the SUS score and measurements like time taken per task, allowing you to more accurately track your site’s usability over time and see how it ranks against other websites.
All of these qualitative and quantitative tools in combination make it easy for you to run usability tests at scale and leverage the different kinds of feedback you receive to easily and effectively build a roadmap for improving your UX.
To learn more about the UXCrowd, join the webinar December 9th with former Zynga user research director Rob Aseron and IdeaScale CEO Rob Hoehn: https://trymyui.com/webinar/uxcrowd-crowdsourcing-quality-ux-feedback
Trymata
Sounds fantastic! One question: say you have 10 videos. The 9th video will submit their answers. But there’s only one more person (the 10th video) who will vote on the answers from the 9th video. So the most PLUS votes that the 9th video can get are 1. But the 9th video may have had the best idea. But you won’t know it from the voting. I’d love to understand this better.
Thanks for your question Michael. That’s a conceptual issue we’ve struggled with, because obviously it would have a significant impact on the value and accuracy of the results.
Our solution is to have testers submit their responses right after taking the test, and then when all of the responses have been collected, bringing them back to do the voting (their test video will play in a module next to the voting to keep it fresh on their minds).
[…] your UX roadmap with the UXCrowd. The UXCrowd is a voting-based survey at the end of a usability test that allows […]