The SUS Webinar: Q&A

We talked to Jeff Sauro of MeasuringU October 9th in an hour-long webinar on the System Usability Scale. Below is a recap of the Q&A segment.

View the full webinar on YouTube

Q: Can SUS be used as a before-and-after analytic to measure responses to major changes or overhauls to sites?

A: Yes. There are two ways to do this: with a between-subject study, some testers can be assigned an old version of your website, and some the new version, and then the SUS scores for each version compared. The other method, a within-subject study, has each tester use both the new and the old version themselves, in a random order, and again the SUS scores for each version are compared. Within-subject studies are equivalent to having roughly 4 times the sample size.

Both methods are a good way to empirically measure changes in usability in regards to major site updates. At TryMyUI, we make this process easy by offering user testing not only for live sites, but also for prototypes and wireframes.

Q: Can SUS measure the usability of multiple interactions?

A: SUS, as Jeff said, is not diagnostic – while it can reliably measure website usability, it can’t tell you which features need improvement. Of course, it is conceivable to pose the 10 items of SUS to testers after the completion of each individual task, but this quickly becomes overkill – for a simple 5-task test, this amounts to 50 questions for one tester plus the ordinary 10 for the final evaluation.

A better option is something called the “Single Ease Question” (SEQ): just one question posed to the tester after each task, that targets the ease or difficulty of performing that task. Certain questions designed to such a purpose have been found to be pretty reliable in comparison to longer post-task questionnaires, and also correlate with the overall SUS score.

Q: How useful is SUS when employed independently from usability testing?

A: The allure of numbers is powerful for persuading third parties of a problem’s existence, compared to subjective comments and viewpoints that are sometimes open to interpretation. Numbers transcend opinion and become fact; they can make discussions data-driven and shortcomings undeniable. What’s measurable, after all, is manageable.

Nonetheless, without open-ended feedback on your website’s processes and features, it is difficult to put SUS numbers into action. There is a certain power in being able to see in video the exact moment when the user stumbles; witnessing first-hand your site’s failings and confounding bits adds an invaluable layer of self-knowledge. Together, SUS and user testing combine to present a well-rounded and multifaceted picture of web usability.

More good news: SUS is reliable with small sample sizes, so it doesn’t take a lot of testers to get solid data on your usability.

Q: What other scales are there to measure usability, and how are they different from SUS?

A: As a usability metric, SUS is technology agnostic – it can be used to measure not only websites, but any system that users interact with and form an opinion or attitude about. The downside of this otherwise useful attribute is that it doesn’t provide information that is specifically and exclusively relevant to websites (or any other kind of system).

MeasuringU has developed an alternative scale called SUPR-Q (Standardized User experience Percentile Ranking Questionnaire) which is based on elements of the System Usability Scale but incorporates inquiries into appearance, navigation, credibility, and more in a web-oriented context.

Read more: Comparing SUS and PSSUQ, two usability quantification metrics

Other usability measurement scales include the SUMI (Software Usability Measurement Inventory) and WAMMI (Website Analysis and Measurement Inventory) surveys. These questionnaires are comprised of several dozen items each, in contrast to the “quick and dirty” 10 questions of SUS.

Nonetheless, SUS scores have been found to correlate well with the results of these more extensive questionnaires; as Jeff said, SUS could well be characterized as “quick and clean” instead.

Q: Can users implement their own preferred questionnaire models on their TryMyUI tests? Are there plans for other options?

A: We are currently working on providing the ability to choose whatever questionnaire model you bring with you, either as an alternative or in addition to SUS. We are also looking to grow the number of measurement tools we have on offer to give you the most expansive and multi-dimensional analysis of your website’s usability.

Additionally, we will soon be implementing SUS in Portuguese, Spanish, French, and German translations to accommodate usability studies in either language.

Q: Are there any usability scales for mobile?

A: As yet, no industry standard exists for testing the usability of apps; those few mobile questionnaires that have been developed pre-date the iPhone, and are overlong and outmoded. MeasuringU, however, has used their SUPR-Q questionnaire to test mobile websites and compare their scores to other mobile websites. Beyond this, mobile usability is a field in which there is still much opportunity ahead.

Ready to measure the usability of your website or app? Get started with a free trial: