If you’ve been following our social media, then you’ve probably seen the UX playoffs polls we’ve been running for the past several weeks. It started as a fun little campaign that we started in celebration of the NBA playoffs. As folks engaged with and voted on their favorite food delivery apps, we were reminded of a time when we used to share UX Wars blog posts and how those resonated with everyone. So, here we are! Soft relaunching our old campaign. Don’t worry, friends. The name change was temporary and apropos only to the NBA playoffs situation.Now, let’s get into it, shall we?
Overview
For those who haven’t been following the social media saga, our UX playoffs campaign pitted several food delivery apps against one another in 7 rounds of voting. Each round, we posed the question “Which app has the better UX?” The winner of each round moved on to the next stage of the bracket, until we narrowed down the panel to two contestant, who faced off in the UX finals.
In the finals, we decided to make things more interesting by conducting usability testing on both apps through our platform. Similar to our UX Wars posts – which we will be resurrecting in the near future – we set up usability tests for each platform with a series of similar tasks for users to complete. From there, we assessed the overall results from the UX Diagnostics suite, as well as taking a closer look at the responses to each individual task.
Here is a closer look at how things panned out with the polls and the usability tests.
UX playoffs poll results
Across all social media platforms, we had an overwhelming response in favor of Instacart as the superior food delivery app. Votes came in on Twitter and Instagram, leaving Deliveroo in the dust, 3-0. We cant be sure of the factors influencing these results, but we have some thoughts. While our social media audience isn’t all American, it is heavily skewed in that direction. Since Deliveroo is an international app not available in the US, it’s likely that it got no votes for that reason.
This is another good reason for conducting usability testing. Because we can specifically target users that live in areas where these apps are accessible, we can get a more accurate read on how they feel about the app’s user experience. Which brings us to the usability testing results.
UX playoffs usability testing results
Upon conducting parallel, or A/B usability testing, we saw a much different outcome. A first glance at the data shows that folks felt more positively about Deliveroo’s user experience, compared to Instacart. We took a closer look at the results and what users had to say about their experiences. Below, we will dive into that closer look, how we set up the tests and what the reflections and results were.
Demographics
When we initially set up usability tests for both mobile apps, we kept the demographic filtering simple and similar for both tests. We only set parameters for age group and country. For both apps, we set the age range at 18-34 and 35-54. For Instacart, which is primarily used in the. United States, we set the country as US. For Deliveroo, an international app, we set the country for the United Kingdom
UX playoffs usability testing scenarios
In light of the recent NBA Playoffs and Finals, and since we have been running with the UX Playoffs theme on social media, we decided to center the task scenario on watching a basketball game. Specifically, we instructed users to put themselves in the following mindset:
You are watching the NBA Finals with a group of your friends. Everyone is a little hungry, so you want to order some snacks and drinks for you and your friends.
For the Deliveroo, we adjusted the scenario a bit, because basketball is not as popular in the UK. Therefore, we catered the scenario to a football audience:
You are watching a football match with a group of your friends. Everyone is a little hungry, so you want to order some snacks and drinks for you and your friends.
With the scenarios clearly outlined, we could move forward with launching our usability tests. Surprisingly, we got results back very quickly, with very definitive results and feedback from both groups. Here are the key points we noted from each test.
Instacart UX playoffs usability testing
The tasks
For Instacart, we outlined a very clear set of tasks with specific steps. They are as follows:
- Log into the Instacart app. If you don’t have an account already, use Google to sign up. From there, navigate to your preferred grocery store.
- Update/confirm your delivery zip code and address. Does your preferred store deliver to your area? If not, find another store that does deliver to your area.
- Scroll to “Snacks” and use the arrows to find any Tostitos chips of your choosing. Add 2 bags to the cart
- Use the search bar to find salsa for the chips. Select your favorite brand and put two jars in the cart. If you don’t have a favorite brand, select the cheapest option and put two jars in the cart.
- Use the categories function to navigate to “Beverages.” Scroll until you find Dr. Pepper. Add two 2L bottles or two 12 packs to your cart.
- Select the cart icon to verify what you have added. Continue to checkout.
- Under “choose a delivery time” pick any free delivery slot ahead of your current time.
- Under delivery instructions. check “leave at my door if I am not around” and hit continue.
- Add and save your correct contact information. Skip the option to share contact details with the store.
- Change and save the delivery tip amount.
- Navigate back to the cart and remove all items.
The outcome
As you can see here, we utilized the Adoption Likelihood Factors Questionnaire, or ALFQ for Mobile Apps, Instacart had an overall score of 94.69%. This score is determined by 4 sub-scores, Usability, Usefulness, Credibility, and Desirability. Below is a closer look at how each user answered the individual questions.
As you can observe, most users had a generally positive response to the app’s user experience. However, there is one person who had a slightly less favorable response. Let’s get into what possibly went wrong.
The issues
A deeper dive into the quantitative data shows that there are clearly some pain points with Instacart’s UX. When you look at the Net Promoter Score, one of the users is an active detractor for the platform, the same user with the lower score.
What does that mean? Let’s see if we can find out in other portions of our data reporting, before looking at what things the users had to say about the overall experience.
Single ease question (SEQ) – task usability
With this graph, we can clearly see that at least one user found difficulties with task 1, 5, 9, and 10. Our outlier, Shatara T, seemed to have difficulties with the 1st task and nothing else.
Task completion
As far as task completion goes, both task 9 and 10 had one user who could not complete. However, neither of these users was the outlier/detractor.
Task duration
With most users, duration time is pretty low for each task. However, there were a few minor jumps in task duration, as well as one major outlier in task 5 that definitely begs for us to investigate further.
Key reflections
When we looked at the videos and specifically honed in on key areas in the quantitative data that pointed to issues, we were able to get feedback from our users on what went wrong. We started with our overall outlier, Shatara, to see what she had to say about her experience with the app.
Surprisingly, she didn’t have any major issues with the app itself. Her concerns were more for users who are new to the app and how easily they might be able to navigate things. This is something she brought up right away, during task 1, as she was logging in. She noted that while she had no problem launching the app and getting in as a repeat user, she felt that it might be confusing for new users.
Upon gaining that valuable insight, we moved to analyzing the qualitative feedback for task 5, another obvious pain point in the app. we found that every user had an issue with this task. As we watched the videos, we realized that the issue was with the accessibility of the product, Dr. Pepper. No matter what store each user chose, we found that every single one had an issue finding Dr. Pepper using our instructions.
A lot of the users ended up searching for the beverage within the app, instead of using the categories function and looking under “Beverages.” Part of the issue was a wording choice. It appears that the app uses “Aisles” instead of “Categories,” which confused some users. In other instances, Dr. Pepper was specifically not within the beverage category. Sometimes, it was in promotional areas within the selected grocery stores. Because the soda selection wasn’t readily available within the category, this created issues for users.
Deliveroo UX playoffs usability testing
The tasks
For Deliveroo, we kept most of the same tasks that we wrote for Instacart. However, since the layout and flow of the app are a little different, we had to reword some tasks and take a few out. Here is the task list we created for the Deliveroo usability test:
- Log into the Deliveroo app. If you don’t have an account already, Use Google to sign up. From there, navigate to your preferred grocery store.
- Update/confirm your delivery post code and address. Does your preferred store deliver to your area? If not, find another store that does deliver to your area.
- Scroll to “Snacks” and use the arrows to find any Walkers crisps of your choosing. Add 2 bags to the cart
- Use the search bar to find salsa for the crisps. Select your favorite brand and put two jars in the cart. If you don’t have a favorite brand, select the cheapest option and put two jars in the cart.
- Use the categories function to navigate to “Beverages.” Scroll until you find Dr. Pepper. Add two 2L bottles or two 12 packs to your cart, based on what the store offers.
- Click continue to checkout. Verify your items before fully continuing.
- Add and save your correct contact information.
- Navigate back to the cart and remove all items.
The outcome
Per the above screenshot, Deliveroo’s ALFQ score was 96.56%, just beating out Instacart ever so slightly. The average sub-scores tended to trend higher than Instacart’s as well.
A closer look at the scoring for the questionnaire shows that mostly everyone had largely positive thoughts about Deliveroo’s UX. And even the one with negative deviation scored within the 90th percentile, demonstrating that the overall view of the Deliveroo interface and user experience is positive.
The issues
When we look at the Net Promoter Score, immediately we see that there are no detractors. There are, however, a couple of passives, which was not the case with Instacart. There ends up being a pretty close split between promoters and passives here.
Single ease question (SEQ) – task usability
For the Single Ease Question (SEQ), there is one obvious outlier for task usability with number 5. While most of the tasks tend to show easy use, there are a few points where users struggled a bit with a task.
Task completion
Consistent with the data from our SEQ chart, task 5 presented a problem for some of the users. Here, in the task completion bar chart, we see that two users were unable to finish the task. This means, we will have to examine how users specifically navigated that task in the videos.
Task duration
The graph here shows that it didn’t take users very long to complete the tasks assigned. However, there was some variance with task number 5, which is consistent with what we’ve seen in previous data points.
Key reflections
When we take a look at the qualitative feedback, it becomes very easy to see why the quantitative data looks like it does. First, we looked at Damilola, who had the most outlying data points in our report. A quick look at her video shows us that she struggled with inventory at the store she selected. Instead of finding a suitable substitute, like most other users did, she would go to another store entirely to find what she was tasked to look for. This created difficulties with her completion time, as well as other data points. And because store inventory is outside of Deliveroo’s control, that does not reflect on their UX overall.
Moving on, we were able to have a more general look at what user’s thought of the navigation and accessibility of the app. Specifically, when we look at what areas users struggled most with, we found some common thoughts. Most of the issues were with inventory at users’ selected stores. This is not something Deliveroo is responsible for, so it does not impact the conversation about the app’s UX.
One minor detail that was brought up by Shikha S. in task 7 and Farwaj C. in task 2 is that adding, saving, and confirming contact information within the app is not easy or intuitive. Both commented that it was difficult to tell if their contact information was saved and confirmed within the app.
Another minor issue brought up by Farwaj C. is how difficult he found it to navigate back to the cart and remove the items. He suggested that Deliveroo offer a better option for opening the cart to see the items.
Overall, users found navigating the tasks required very easy and Deliveroo’s interface simple enough to navigate.
Conclusion
In conclusion, social media results for the UX playoffs are definitely skewed by the reach of our audience. While Instacart may have won in the public poll, more objective results seem to favor Deliveroo more. The user interface, navigation, and other aspects of its user experience were easier to work with, according to user feedback. At the end of the day, this shows that conducting UX research yields more accurate and unbiased results.
Related reading: