While user testing on trymyui is easy, it’s even easier for us to test our own design because each test that comes back reflects whether we’ve done a good job with how we display our instructions. In a trymyui test, the test creator establishes a “frame-of-mind” for the tester, such as, “Imagine you’re going on a trip to italy with your family of 6. Use this website to find accommodations”. We presented this text prior to launching the recording windows, and also on the recording window, and yet we found that some testers (who have already been through a qualification round) would often not read it carefully. Why? Presumably because the “frame-of-mind” section was on a page with other instructions, and the testers were missing it because the other instructions don’t change from test to test.
A remedy that we recently implemented was to add a first test task asking testers to read the “frame-of-mind” section out loud. Not necessarily pretty, but quite effective.
Tim Rotolo
Trymata