A primary rule of web management is: ‘Do as I do, not as I say’. So why do we ask people to say what they’re doing during usability tests?
About a year ago a university professor explain concisely how he liked to use a particular website. “Now when I have this type of problem to solve, I would always go to this section of the website.” And he clicked on a link. “See how well it’s organized to solve this sort of problem,” he stated enthusiastically.
A couple of minutes later we gave him a task that was an example of the very problem he had being talking about. He clicked on a different link to the one he had said he would click on.
In the 1980s, Consumer Reports got experts to rate 45 jams. Sometime later Timothy Wilson at the University of Virginia asked a group of students to rate the jams. There was strong agreement between the experts and the students. Then Wilson asked another group of students to analyze and write about their decisions as they made them. The jam that was rated worst by the first group of students was now rated best by the students who had to explain and justify their decisions as they went along.
Some tasks need to be thought through in order to be completed successfully. These tend to be tasks that you have not done before much and whose solution runs contrary common to sense (as the solutions to a great many complex tasks do).
A bunch of fire fighters were running from a forest fire that had suddenly spiraled out of control. Their leader, Wag Dodge, abruptly stopped and shouted at everyone else to stop. He realized that they could not outrun the fire. He took out his matches and lit some ground in front of him. Everyone thought he was crazy and kept running. Dodge lay down on the ground he had burned. Dodge survived. Almost everyone else died.
This story comes from the book The Decisive Moment by Jonah Lehrer. When professional golfers start thinking about their swing, they’re in trouble, Lehrer argues. They start second guessing themselves and talking inside their heads and they lose the instinctive feel they have developed through years of practice.
When people are on the Web, they are instinctive, impatient, impulsive, in a hurry. They click on the first link that looks in any way right. They search with the minimum number of words possible. Only when they get bad search results do they add more words to the search string. I have seen tests where people preferred to get poor results quickly than invest a little more time and effort and get much better ones.
We need to become as invisible as possible and observe people as they try to complete tasks. Patterns of behavior will always emerge. We’ll see where people get stuck, where they give up, where they get puzzled. But if we ask them to tell us why or how, they may end up telling us why they think the worst tasting jam is actually the best.