Jared Spool is acting as master of ceremonies this morning at Bolt Peter’s User Research Friday. Here is a summary of the talk…

Four Stories about Five Studies

Jared’s talk was in the form of 4 real stories of user research for customers.

1) Abandonment

This case was about a company that had a problem at the last step of the purchase process.

OBSERVATION: 65% abandonment after entering credit card data.

Here are the inferences the customer thought for sure was causing the problem and what happened when they addressed them…

• Adding secure checkout. -> No change!
• Adding messaging about secure checkout -> No change!

UIE ran compelled shopping test, a special kind of usability testing where the testers are chosen because they actually want to purchase something the site offers in reality and are given the money to buy it.

In this case Jared tells a story of a tester who was in the market for a big printer. The customer loved the site!  It had the product he was looking for at a great price! He was ready to make the purchase! Got all the way to check out…

and… stopped.

Why?

ACTUAL CAUSE: It turns out the site did not give shipping cost before  entering a credit card. Shipping cost can sometimes be a big cost for a large item. Customers were afraid to purchase with out that information.

MORAL: Its easy to assume the wrong inference from an observation.

2) The Blank

Jared showed us a front page screen for Wells Fargo. The screen had..

• A login to the left
• A search box on top
• A bunch of information in the middle.

OBSERVATION: According to the log files, the number one search entry was blank – no text. This accounted for 43% of searches!

Why?

Jared asked the audience what they thought was causing this problem. The funny thing was that what the audience said was exactly what was on his next slide and in the same order!

• Focus was broken on the page and hitting return after entering log in was causing a search event with nothing in the search box.
• Users were actually clicking search without a query for some reason.
• There is a bug in the logging system.

MORAL: There usually plenty of suspected causes for an observation.

2.5) OIORD!

Jared’s central point is that:

In user research you start with a basic observation and need to get to a decision.

Basic Observation -> Design Decision.

But there are steps in between that are necessary…

Observation -> Inference -> Opinion/ Theory -> Reccomendation -> Design Decision

MORAL: Where most user researchers stumble is between Observation and Inference.

* By the way, I spoke to Jared after the talk and asked him about what is the difference between Inference and Theory. My understanding from his explanation is that is a subtle distinction. Inference is the assumed proximate reason for an observation. Theory is a generalized hypothesis concerning the inference.

Note to self: Should have asked him for some examples. Jared if you read this you might want to leave a comment with some examples.

3) The $300,000,000 Solution

OBSERVATION: Only 5% of folks who had put items in their shopping cart made it to checkout.

CLIENT INITIAL INFERENCE: This was due to some problem in the shopping cart.

UIE ran a ran a compelled shopping study with 24 customers…

RESEARCHED INFERENCE: They found that the actual reason users abandoned is that the site asked them to login in the shopping cart checkout process 45% of the users had forgotten their login or password.

RECOMMENDATION: Guest checkout.

Result 45% increase in checkouts accounting for $300,000,000 in increased revenue!

MORAL: The initial inference you have may not even be in the correct part of the process. In this case the problem was in login rather than in shopping cart where the user assumed it was.

4) The True Experience

This was a huge study. The remuneration alone for participants was $94,000 for 94 participants. The client worked with both UIE and Netraker.

Netraker – based study

METHODOLOGY:

• User got a $10 gift certificate.
• Were not actually in the market for a product.
• Looked at click stream data to see how long users would take

RESULT:

• Find assigned product -> 1m 18sec
• Find your dream computer -> 1m 42sec

Compelled Shopping Study

METHODOLOGY:

• Detailed screening to make sure users are truly in the market to buy a laptop.
• Give the users 2 weeks advanced notice to pick what they want to buy
• Give them the money to make the purchase. They keep the money whether they make the purchase or not.

RESULT:

• Find your desired product -> 18 Minutes!

MORAL:

A 20x difference in the result as a consequence of 2 different methodologies!

• Are we exploring the likely inferences from our observations?
• What happens when we are using non direct observation techniques?
• Are we taking the time to verify what we are seeing?
• How can we use tools like analytics effectively?
• How do we help teams that have conflicting data?

Dana Chisnell at B|P User Research Friday – Research in a Social Context