On December 8th we were joined by Wendi Chiong, Senior Design Researcher at Motivate Design, in a discussion about strategies for employing usability data to impact product direction.
Wendi spotlighted a 3-step strategy called What? So what? Now what? in which the researcher first identifies a What (the critical or relevant information contained within the data), then communicates a So what (why those insights matter), and finally formulates a Now what (a course of action for improving the product based on the data).
The talk ended with a Q&A session in which Wendi and host Ritvij Gautam, our CEO, responded to questions from attendees. Below is a transcript of the Q&A segment.
Q: What’s your favorite method of data storytelling?
Wendi: It can happen in so many ways. So often people think of very specific tools that they have to use to tell a story – “I have to use a journey map” or “I have to use personas” – and I wouldn’t necessarily say that’s the case.
Oftentimes I think what’s more important than tools or particular methods is realizing that a story has to make sense. Being able to tell an arc – you can use a journey map, or personas, or any of these things that will help, but honestly the most basic tool I will often start out with is this framework of What, So what, Now what because it forces me to move beyond the findings.
Everyone has seen a usability report before that’s like, “Half the users liked this version, half the users liked that version” and that’s where it stops.
What we’re really trying to tell you today is to think beyond that. What does it mean? What’s the impact on the user? What’s the impact on the business? How can we work around this pain point?
Being able to think about the broader implications of the findings, whether they relate to the business or the overall experience, that’s what’s important. And then if you want to use other tools along the way, its not one-size-fits-all.
Rit: My favorite method of data storytelling is trying to make sure I can link my findings to some metric that’s being tracked by the company already. That gives immediate value because now I’m no longer telling them this abstract story about design principles they’re violating, it’s more like: Hey! If you don’t change this, your bottom line will decrease; or, This is a big blocker for you to get conversions on this page.
Once you start talking about those kinds of user testing metrics, the story starts making sense. And you’re talking in lingo that the people you’re talking to also understand. That really changes it up in a big way.
Q: How do you present your data in a way that compels your leadership to take action?
Rit: I would say the same thing: your leadership cares about this kind of thing. They care about the bottom line, they care about metrics, they care about conversion rates; and those are things that, if you can prove a link between your findings and one of those metrics, you have yourself a very compelling data story that people are going to take notice of.
Wendi: I feel like among those of us who work in UX, that can be a common pain point and challenge that I hear from a lot of colleagues.
For any of you guys who are signed onto this webinar, you understand the importance of having a good user experience. You’re fully bought into that and you realize how valuable and important it is for your company, and it’s really frustrating if you’re working with colleagues or counterparts that don’t see the value in investing in UX.
That’s, I would say, one of the most common things I hear: How do we get people to care?
View: See the slide deck from the webinar
I think what Rit highlighted – understanding how this ties back to metrics that they are trying to move the needle on – the slides that we showed today had some examples like conversions, or sales, even something as simple as who was able to get through the flow or complete the flow. Having metrics like that can be really powerful, particularly for folks who are more quantitatively oriented.
Q: How does So What relate to ROI?
Rit: The So what relates to ROI based on what your company does.
You have some definition of what it means to convert, or some kind of core objective. And that’s what you’ve set out to achieve. It might be getting more signups, it might be having more checkouts, it might be getting views, or comments, or shares on a certain post. So your So what is very dependent on what your core objective as a site is.
TryMyUI will be able to give you the data points, and it will give you the data point that affected the most users. Where the So what comes in is where you need to bridge the gap. You need to say, given this data point, and given my core purpose, how does that map over.
And that’s kind of privileged knowledge to you as a designer at your company or for your product, how you can bridge that gap.
So what TryMyUI can do for you is bring you to that point where you can clearly see all of the different Whats and you can see which of them is the most critical or affected the most users. Then you can make a judgment call from there saying, well, my goal is to increase the number of signups. So if I’m trying to increase the number of signups, maybe I should put “Sign up” in a more obvious place.
TryMyUI can help you make a case for that change by helping you pick that data and catalog your evidence for suggesting a certain change – a certain Now what. TryMyUI makes the So what very clear.
Q: How do you format your Now whats for an abstract problem, or problems that have hundreds of possible solutions?
Wendi: That’s a really good question. I’m going to try and interpret it a little bit. I’m assuming you’re talking about a pretty open, abstract question – not a classical design question of, you know, move this button here or add this to the checkout flow so users understand it better.
This is where the Now what and the So what maybe get a little more ambiguous, because there’s not an immediate design step that’s obvious as the right decision to go forward with. This might be when you’re doing explorative data, and I’ve talked about personas before, maybe you’re doing really early-stage generative work where you’re just trying to understand who are our different types of users, what are different types of use cases for our product, things like that.
They’re probably going to give you moreso frameworks of how to start thinking about your products and different user groups, as opposed to very very specific tactical design changes. So instead of like, a make this call-to-action more clear, or add this page – stuff that’s very tactical – you’re probably thinking about much more broad, open-ended data.
Watch: Data is Useless! 3 Questions to Make it Matter (Full video recording)
I think in that case you’re probably talking about general persona types or use case types, you’re talking about really broad, overall insights about the experience and how a user might be interacting with a product or experiencing a product; and that’s where I think just building out frameworks for your product team to understand can be valuable.
They might not be able to know exactly that your product needs to have feature X or widget Y, but they’ll know that, ok, we need to consider this type of use case. Using the e-commerce example, here’s somebody who’s specifically looking for a gift, whose main goal is to get her brother a gift for the holidays; she just needs to find something that will kind of fit his needs; versus me, I’m shopping around for myself and I just want to explore and browse what this site has.
While you might not have very tactical design recommendations yet, just even knowing that there is that switch in use cases, or in how people are thinking about or using that site, can be really valuable for your product team and can encourage them to start thinking about Now what? How do we start to design for use case 1, how do we start to design for use case 2?
So those are two very high-level examples, but when you’re talking about dealing with some of that open, early-stage work, I think just building out frameworks for people to understand the different ways that people will experience your product is really valuable.
Q: How can you convince someone who’s very visually oriented and judges the site and improvement on looks and doesnt care so much about numbers? Any advice on influencing and convincing these profiles?
Rit: This is actually very interesting because we’ve been talking about how to pitch this stuff to higher-ups, how to make the data palatable to them and going from “Hey this is good UX” to “Hey this is good UX AND it also helps you with these conversion metrics.” This is the flipside of that question, how do you convince someone who is not metrics-driven about these changes.
I think a great example is something like Craigslist. It’s far and away one of the ugliest sites I’ve ever used, but that’s because its core purpose, its objective, has nothing to do with being pretty.
We have done usability testing on Craigslist in the past and the overwhelming feedback has been, all the information is really accessible but its kind of plain. But they don’t care about it being plain, they just care about the information being accessible, and that’s their main driver. So every time they hit up against a What that says “That looks kind of plain,” their So what is just So….nothing. And that’s totally fine if a user’s opinion doesn’t map with your core goals.
Whether you’re metrics oriented or visually oriented, this is something that needs to be made abundantly clear. Because there’s a deeper issue if someone just wants to make something that looks really pretty – all this is really functional, it has to work, it has to meet its objectives for the design to mean anything, for the website to mean anything and achieve its core purpose.
So maybe don’t bring up the numbers because that might turn them off, but you’ve got to talk about the core purpose of the site. “Hey look, we’ve designed this site to achieve (blank) and it is achieving (blank).” You can even run a quick AB test on it to see if there’s a huge sway on the core metrics you’re tracking, and use that data to help you say So what.
Learn more: user testing better products and user testing new products
Wendi: Rit, I would just add to that, I think that’s where the So what question can help. This question is a tricky one but I think you hit the nail on the head: sure a site is pretty, but is it useful? Do people know how to use it, can they get through it?
If the answer is no, then that’s some pretty powerful data right there, that most people should be able to listen to – especially if you can draw that So what implication that those people wouldn’t use this product.
So your data might say users liked the look and feel of the site, or users didn’t like the look and feel of the site, and the So what can be, well it didn’t really matter because users still didn’t understand how to use the site. Your So what is, users wouldn’t be able to get through, they wouldn’t be able to accomplish the key tasks they need to accomplish on this site; which means our site has not succeeded in doing its task.
If you can get them to see that users are still struggling for some reason, that hopefully can be the impetus for change.
