Self reporting techniques, such as surveys, interviews, and focus groups have long been used by marketing reps for collecting customer data, for good reason. They provide excellent information on demographics and user reaction to a new technology. But this information is not necessarily helpful in all circumstances. For example, they are often misused when undertaking User-Centred Design (UCD).

Good design should empower users to succeed at their key tasks. A designer’s job, therefore, is to predict the future use of the product, which should include some kind of user research to be successful. Unfortunately, many UCD practitioners mistakenly apply self-reporting techniques without realising that surveys are an ineffective design tool. They can lead to excellent systems that do entirely the wrong thing. Instead, user observations are the key for designers. They are as effective at identifying design data as surveys are at collecting demographic data.

Recently, a company asked us to design them a Knowledge Management (KM) system. Its purpose was to facilitate information sharing across departments, and improve the consistency of reporting procedures. But ethnographic observations quickly determined that a KM system would have been doomed to failure, as it would not have met the essential needs of the users. They were already busy, and typically stored data in several repositories. For example, all users had important documents stored in their inboxes. It was generally agreed that these documents should be saved to their hard drives and/or the shared network drive, but only one had actually done so. Moreover, each repository had a different organisational structure, and all had stored incomplete or minor artifacts, as well as complete reports.

There were two major problems for a KM system here. Firstly, it would have required users to move all the items out of their individual repositories, and then to categorise and meta-tag each data object. All this would have taken time and behavioural changes on the part of the users, which decreases the likelihood of acceptance.

Secondly, a KM system depends on a general understanding of the organisation and categorisation of the data and few KM systems contain incomplete or minor artifacts.

Therefore, what the organisation actually needed was a better search mechanism. This system did not depend on any additional user effort, worked across all data repositories, considered both minor and incomplete artifacts as well as major and complete ones, and actually provided the users with better suggestions of information sources.

Here is a clear example where self-reporting techniques would have provided an entirely inappropriate solution. It might have been very good, but it would not have fixed the problem. This is because users are much better at recognising or reacting to a design solution, than they are at describing the problem. Added to this, they typically know less about the potential technologies than the designer, so their input is limited. As Henry Ford once said, "if I had asked my customers what they wanted, they would have said a faster horse."

Users also tend to describe their tasks differently from how they actually perform them. Often this is because they blame themselves for any mistakes they make, rather than the technology. In one observation, a hospital lab manager told us that her organisation’s current tool was a good system, but then committed eight errors within a 30 second task. Fortunately, because the consequences could have been life-threatening, she recovered from all of them. Later, when asked about the errors, she mentioned that it was her fault that she made the mistakes and that she just wasn’t smart enough to use the tool correctly. Was this a case of a ’stupid’ employee, or was the system unnecessarily complicated?

Too many products force the users to change their tasks to fit their perception of how to use the product. Your objective is to optimize the users’ tasks and design a product that fits the optimized tasks, not to improve how users use the product. The product should be transparent to the task. Users want to write a letter, not operate a word processor. By observing users using your product you will likely end up merely automating their current frustrations.

For instance, we were once asked to design a website for an on-line florist. Instead of watching how flowers were bought on-line, we went out and observed people buying flowers in a shop. Have you ever seen shopping carts in a florist shop? Of course not, so then why should the site have one? Eliminating the ubiquitous online shopping cart saved the client thousands of dollars, and they could still up sell associated items, such as vases and teddy bears. This and other key innovations derived from our user observations have made this one of the world’s most successful and profitable e-commerce sites.

Finally, self-reporting methods require lots of respondents, which takes time and effort, while you only need to observe half a dozen representative users to get a good insight into their needs. Surveys are good at getting user reactions to solutions, but observations are much better at defining the actual problem. So instead of relying on self-reporting methods in your user research, try to conduct user observations. You’ll get better insights and you’ll actually take less time.