In June, three of us from the Optimal family attended the Usability Professionals’ Association conference in Portland, Oregon. We learnt a lot, including how "strategic dismemberment" relates to game usability (watch the violent video here). We thought we’d dedicate this newsletter to sharing a few gems from the conference. (Forgive us if we get a bit geeky with the terminology). Enjoy!

 
Expert Reviews

• According to the latest research, the optimal process for an expert review is to conduct evaluations in pairs using domain-specific heuristics. It’s kind of like pair programming, with the domain-specific heuristics acting as inspiration.
• Expert reviews with highly experienced practitioners are comparable to user testing in terms of the number and types of issues found. Also interesting - the method was originally designed to help beginners prevent usability errors, not help experts identify them after the fact.
• Rolf Molich only reports back his top 40 findings to clients when doing an expert review. Steve Krug has told me that he only reports back 10 issues, and tells clients to come back to him when those 10 are fixed.
• Jakob Nielsen is famous for his 10 Heuristics. Well, now his company have amassed a total 2,666 guidelines for websites, intranets and email across 9,356 pages of reports. Not so usable.

User Testing

• Users tend to report high satisfaction if they can complete a task in a user test (independent of time taken or the number errors they encounter). In other words, if you are lazy you can just count the number of times people complete a task, rather than asking them to rate their subjective post-task satisfaction.
• The latest version of Morae can use a wireless Wii remote (which costs US$40) to create markers of interesting events during a user test. Handy!
• Some companies are using Facebook to advertise for user testing participants. It allows them to narrow down by some very specific demographics, and can be quite successful.
• Phonetag.com captures voicemail and automagically transcribes it. This sounds great for longitudinal studies for only US$30/month.

Persuasion
 
One of the hot areas in usability is in the psychology of persuasion. It’s fascinating stuff. One workshop that I attended (hi Kas!) had some great points on the "Science of Compliance". Here are 5 ways you can persuade people:

• Persuade via Scarcity: People want things that are rare, hard to get or in decreasing supply. For example, we could let people know when there are "only 3 places left in our Introduction to User Testing course".
• Persuade via Authority: People who are perceived to be more credible are more persuasive. For example, we could build credibility by running training, writing whitepapers, and talking about number of clients we have.
• Persuade via Social Proof: In other words get people thinking "If lots of other people are doing it, I probably want to do it too". For example, with our recent government special we talked about the fact that we’d done 28 expert reviews for other government departments.
• Persuade via Liking: You are more persuasive to people who like you. One (non obvious?) way to get people to like you is to work on things together.
• Persuade via Reciprocity: People feel the need to pay back favours. For example if you invite people to free events that you put on, or give people free content, they feel that they owe you.

Mobile

• There are five times more phones than computers in the world.
• Interestingly though, only 2% of US mobile phones are iPhones. They are still getting a lot more press than usage.

 
One final suggestion for you to try at your work place: apparently, if you put a mirror in front of you while you work, you are more productive and make fewer mistakes. Go on, try it. We dare you!
 

Back in 2004 we were doing a card sorting project for a banking client, and really needed to be able to slice-and-dice the data in ways that existing tools didn’t support. We didn’t find anything that could do what we wanted, so we ended up building our own Flash-based card sorting tool over a weekend.

Three hectic years later, we finally ended up releasing a public beta of our card sorting tool to the world. It had gone through a lot more testing and revisions by that stage, and several thousand participants had completed card sorts for our clients. We thought it was pretty useful but we weren’t sure if anyone else would think so.

Given that we’ve had the tool out in the wild for nearly 18 months we thought it would be fun to some analysis into who was using Optimal Sort, and what they were doing. (We first talked about this stuff at OZ-IA 2008, and you can get the presentation online).

So, our first fact is that information architects are very multinational. The 2,552 Optimal Sort customers come from 121 countries, including some unusual locations like Mozambique, Ghana, Bolivia, Serbia, the Bahamas, Georgia, Malta and the Cayman Islands. (As a general rule, Asian countries are vastly under-represented in our customers, possibly because our tool doesn’t support non-English languages.)

People using Optimal Sort tend to work for an internal team (45% of users), which is typically small in size (48% work in a team of less than 5 people).

The companies that they work for though, are massive. Our customers come from organisations that have an average (!) of 5,169 staff.

All those information architects have definitely been busy. Since we went live with the public version of Optimal Sort, 3,875 different card sort studies have been run. A staggering 117,932 people have completed a card sort.

What’s interesting is to see that 65% of the card sorts are what we called open sorts. (Open sorts are when you let participants choose their own category names, whereas in a closed card sort you force people to drop the cards into pre-defined categories.)

The reason there are more open sorts than closed sorts is that closed card sorts aren’t very useful. They are one way of validating a proposed information architecture, but there are much smarter, more robust ways of doing this. (Stay tuned for an announcement very soon of an exciting new tool to give you much more confidence that you have the structures and labels right.)

What is probably the clearest statistic of all is that information architecture as a field is growing in prominence and it’s growing fast.  According to the IA institute annual report there were 750 members in 2007. In 2008 there were 1,500 members. Its membership doubled in 12 short months. Doubled.

It’s a great time to be in the field, and we owe a depth of gratitude to all the pioneers who have used Optimal Sort and helped us to learn what it takes to make a useful card sorting tool. Thank you all!

On the 8^th of November we celebrated World Usability Day 2007. The theme was Healthcare so we’ve invited Suzanne Currie to write this month’s Thought Leader article. Suzanne is a good friend of ours based in Minneapolis, and for the last sixteen months she has been working as a principal Human Factors Specialist at Medtronic, designing medical products. In this article, she will share her insights on applying user-centred design to medical devices.

It’s been said that Medtronic is to medical products what Microsoft is to computers. Medtronic’s 43,000 employees design and manufacture a staggering variety of medical products that are used all over the world. I joined Medtronic for a number of reasons, but the biggest draw card was working on products that contributed directly to human well-being. These are some of my thoughts as I’ve made the transition.

Firstly, it is impossible not to be emotionally affected by what one sees when observing patients and doctors using the technologies we build. For example, it took me three days to process my feelings after conducting my first observation of defibrillator patients.

An elderly couple had come in to the clinic for their quarterly check-ups and graciously allowed me to observe the procedure. After asking them some questions, the nurse investigated the implant sites and data from the defibrillators was downloaded. The tests revealed that the wife was actually very ill and was immediately scheduled for an EKG. She was warned that there was a chance she would need emergency surgery.

It was so hard to tell what this lady was thinking. Initially, she and her husband were both very quiet and did not make any eye contact with the nurse or each other. However, as the nurse continued to focus on analysing the data she began to talk to me.

She carefully told me about her medications, her alternative health practices and how much she and her husband used to love dancing when they were young. She told me how they loved travelling to different states to compete in square dancing, the friends they had way back when, and other descriptions of her life. All I could do to help her was listen, so I put my notebook down, and gave her my full attention.

This was my first ever medical observation. This experience and several others on the first day quickly introduced me to the issues of conducting observation in a medical context. Since that day, I’ve developed a personal style when observing patients, nurses, and physicians at work, taking into account the physically cramped environment, the continual interruptions, and the intensely personal and emotional nature of patient health.

As well as the emotional effects, there are a number of other challenges worth mentioning:

• Access to clinicians is very limited - if you get an hour you’re very fortunate. Getting in-depth, uninterrupted time must be done over dinner or another social context.
• Work spaces are often very tight; you may not be able to spread out your paper prototypes.
• Multiple interruptions may occur during one usability inspection, so you need to know how you will resume the inspection after an interruption.
• Sometimes a test will need to be abandoned due to a patient emergency, so you’ll want to book additional tests to account for this.
• Quantitative tests may be less useful than qualitative tests; task timings are especially challenging given the environmental issues.
• Typically a sales or support representative "owns" the relationship with the clinician-customer, and may pressure you to leave the user with a good impression.
• Always pair in-depth interviews with direct observation in the domain of use. Workflow can only be observed in the dynamic environment of the clinic.
• Terminology is critical to get right, and you should look to test the behaviour that the user associates with the term to ensure accuracy and ease of use.

In terms of design thinking, the medical products industry is 10-15 years behind other industries, such as consumer electronics. MP3 players receive so much more attention than medical products which sustain and improve life for thousands of people. There are untold user experience improvements to be made in medical products. Perhaps at some point in your career as a usability engineer, user experience researcher, or user interface designer, you will take a close look at the rich variety of work available to you in the medical products industry.

If you have any comments or questions, or to continue this discussion, please email me at stcurrie@gmail.com.

In the second of In the second of our Thought Leader series Sam Ng catches up with Donna Maurer from Australian company MaadMob. Donna is an acknowledged leader in the field of information architecture and is currently writing a book on card sorting. Here’s what they discussed:

Sam: So Donna, many of our readers would have heard of you before but, for those who don’t know you, how would you describe what you do and how you’ve come to achieve celebrity status with regard to card sorting?

Donna: I work as a freelance information architect and interaction designer. I usually work on big websites, intranets and business application - either organising all the content (that’s the information architect bit) or designing the pages, interfaces and way things work (that’s the interaction designer bit).

When I was learning IA, I heard about card sorting and I got the impression that it was the ultimate (and easy) technique for creating an IA. So I tried it and had all sorts of problems. I started blogging about my problems & questioning the value of the technique.

But as I did more IA work and gained a better understanding of classification and categorisation theory I started to think about how to use card sorting more effectively. I realized that card sorting is a good support technique for the category development process but is not a replacement for it. Now I don’t rely on card sorting as the only approach, but one method among many to help me organise things.

Sam: One of the biggest challenges with card sorting is analysis and making sense of results. Where do you suggest people start?

Donna: At the beginning ;-

Seriously, analysis starts before the card sort starts. You have to know what you want to learn and run the card sort in the right way - involving an appropriate number of participants and cards. It is wasteful to run a detailed, remote card sort with hundreds of participants if you just want to get some initial insights into how people think. And it’s just as silly to involve 6 people and a small set of cards if you want to explore all the ways content can be grouped.

If you plan up front, analysis does become easier. In the first example (when you want some initial insights), run an in-person team-based sort and don’t collect more data than you need. It then is fairly easy to do some qualitative analysis and identify the main insights. For the second example (exploring content groupings in detail), you need to get organised up front and store your data in a way that lets you do more in-depth analysis - either with a statistical tool or something like my card sorting spreadsheet. That way you aren’t overwhelmed with so much data that you can’t identify patterns.

Sam: Do you ‘eyeball’ the data? I’m sure this can mean different things to different people (and I bet experience counts for a lot). Any practical tips and guidelines on how you ‘eyeball’ data? What do you look for?

Donna: Some of the ‘eyeballing’ process (where you mainly look at what you have collected) is to spot the ‘interesting’ things - the things that you didn’t expect to see or that surprise you. It is also to spot the things that support your ideas. It isn’t about coming up with a structure, but finding insights and patterns.

You’re right - it becomes easier with experience - you have a better idea of what to expect and can notice the ‘interesting’ things more easily. But even without experience, a card sort will help you learn things you otherwise wouldn’t have.

Sam: Many people expect that card sorts produce site maps. What is your response to that? And what are the outputs from one of your card sort projects?

Donna: My response to that is that you should never let any technique do the work for you. A technique is a tool to help you learn things, not create answers.

Sure, a card sort can produce a sitemap - you can collect data, run the outputs through a statistical tool and make a dendogram - a hierarchical site map. But you shouldn’t just use that as the sitemap for your website. The card sort doesn’t consider the business goals of the site, what users need to achieve and how they are likely to work. It just shows you some ideas for grouping content - this is hugely valuable, just not the only thing you need to consider. You have to put that together with many other things to create a good structure for a site.

Sam: There are a lot of people who love the idea of a card sort but find it hard to convince the powers to be to let them run a card sort. Any tips on how to do this?

Donna:I’m a bit cheeky sometimes. I tell my clients that they can let me make it up and take the risk that I get it wrong and have to start over. Or they can let me get some user input, be more likely to get it right the first time and be able to demonstrate why it is right. That goes for any type of user research, not just card sorting.

Self reporting techniques, such as surveys, interviews, and focus groups have long been used by marketing reps for collecting customer data, for good reason. They provide excellent information on demographics and user reaction to a new technology. But this information is not necessarily helpful in all circumstances. For example, they are often misused when undertaking User-Centred Design (UCD).

Good design should empower users to succeed at their key tasks. A designer’s job, therefore, is to predict the future use of the product, which should include some kind of user research to be successful. Unfortunately, many UCD practitioners mistakenly apply self-reporting techniques without realising that surveys are an ineffective design tool. They can lead to excellent systems that do entirely the wrong thing. Instead, user observations are the key for designers. They are as effective at identifying design data as surveys are at collecting demographic data.

Recently, a company asked us to design them a Knowledge Management (KM) system. Its purpose was to facilitate information sharing across departments, and improve the consistency of reporting procedures. But ethnographic observations quickly determined that a KM system would have been doomed to failure, as it would not have met the essential needs of the users. They were already busy, and typically stored data in several repositories. For example, all users had important documents stored in their inboxes. It was generally agreed that these documents should be saved to their hard drives and/or the shared network drive, but only one had actually done so. Moreover, each repository had a different organisational structure, and all had stored incomplete or minor artifacts, as well as complete reports.

There were two major problems for a KM system here. Firstly, it would have required users to move all the items out of their individual repositories, and then to categorise and meta-tag each data object. All this would have taken time and behavioural changes on the part of the users, which decreases the likelihood of acceptance.

Secondly, a KM system depends on a general understanding of the organisation and categorisation of the data and few KM systems contain incomplete or minor artifacts.

Therefore, what the organisation actually needed was a better search mechanism. This system did not depend on any additional user effort, worked across all data repositories, considered both minor and incomplete artifacts as well as major and complete ones, and actually provided the users with better suggestions of information sources.

Here is a clear example where self-reporting techniques would have provided an entirely inappropriate solution. It might have been very good, but it would not have fixed the problem. This is because users are much better at recognising or reacting to a design solution, than they are at describing the problem. Added to this, they typically know less about the potential technologies than the designer, so their input is limited. As Henry Ford once said, "if I had asked my customers what they wanted, they would have said a faster horse."

Users also tend to describe their tasks differently from how they actually perform them. Often this is because they blame themselves for any mistakes they make, rather than the technology. In one observation, a hospital lab manager told us that her organisation’s current tool was a good system, but then committed eight errors within a 30 second task. Fortunately, because the consequences could have been life-threatening, she recovered from all of them. Later, when asked about the errors, she mentioned that it was her fault that she made the mistakes and that she just wasn’t smart enough to use the tool correctly. Was this a case of a ’stupid’ employee, or was the system unnecessarily complicated?

Too many products force the users to change their tasks to fit their perception of how to use the product. Your objective is to optimize the users’ tasks and design a product that fits the optimized tasks, not to improve how users use the product. The product should be transparent to the task. Users want to write a letter, not operate a word processor. By observing users using your product you will likely end up merely automating their current frustrations.

For instance, we were once asked to design a website for an on-line florist. Instead of watching how flowers were bought on-line, we went out and observed people buying flowers in a shop. Have you ever seen shopping carts in a florist shop? Of course not, so then why should the site have one? Eliminating the ubiquitous online shopping cart saved the client thousands of dollars, and they could still up sell associated items, such as vases and teddy bears. This and other key innovations derived from our user observations have made this one of the world’s most successful and profitable e-commerce sites.

Finally, self-reporting methods require lots of respondents, which takes time and effort, while you only need to observe half a dozen representative users to get a good insight into their needs. Surveys are good at getting user reactions to solutions, but observations are much better at defining the actual problem. So instead of relying on self-reporting methods in your user research, try to conduct user observations. You’ll get better insights and you’ll actually take less time.

Creating something that people love to use is easier said than done. Doing it with limited time and resources is even harder. But this is the task we set ourselves 12 weeks ago.

As consultants we provide expert advice, but we seldom get the opportunity to be fully involved in the implementation process. We are aware that it is easier to give advice than to act on it.

So, in order to walk the talk, we decided to design, build and release a new online card sort tool. We set ourselves a small budget, set a 10 week time frame and formed a team of two to make it happen. These constraints proved to be one of our most valuable assets. The constraints forced focus, focus helped reduce unnecessary complexity and required our decision making to centre around our users.

This exercise taught us a lot about what it means to juggle conflicting needs and objectives. Most importantly it also confirmed to us the importance of goal-directed design and the need to stay ‘intimate’ with users.

Some insights we gained include:

• The value of ‘checking in’ every week.User testing twice a week meant that we had few surprises left when we launched. With the simple (and free!) web tools Gotomeeting.com and Skype, we conducted user testing with people from all around the world. The investment in time was quite literally 2 or 3 hours a week but the insight and peace of mind was invaluable.
• Quick iterations save time.The sooner we learnt we should not be spending time on something, the better. With the feedback we got from our user tests, we were constantly tweaking and refining. I’m sure it drove our developer a little crazy at times, but it meant we were able to keep to our budget and release on time.
• Low tech can be just fine. A lot of time can be wasted in "pretend planning", or cleverly disguised procrastination. Often the most effective solutions are remarkably low tech. For us, four large sheets of paper and Post-It notes worked perfectly well for tracking our development progress. It took 10 minutes to set up and cost virtually nothing.
• Small can be good. A bigger team and larger budget is not necessarily the most effective (or only) way to deliver on projects. A small but well organised team can be surprisingly effective. Sitting next to each other, a shared calendar and regular structured communication were our biggest assets in our team.

The finished product, OptimalSort, was released in June at the Usability Professionals’ Association conference in Austin, Texas.

We have been pleasantly surprised by the response. Hundreds of users have signed up and thousands of participants have completed card sorts in just two short weeks. It seems that creating something people love using is well worth the effort.

Take a look and let us know what you think - http://www.optimalsort.com

The project is about to go into launch.  Weeks, months, if not years, of planning, designing, development and testing has gone into it.  Project managers, business analysts, developers, designers, testers and contractors have all been putting blood sweat and tears to make sure things go smoothly.  Of course, it’s seldom that everything is perfect – there is some smoke and mirrors (but only a few!).  But it’s version 1.0.  Version 1.0 always has problems we know about – we’ll fix them in version 2.0.

Then it happens.  Someone finds a problem because she had her friend use it when she was showing off her work – and he couldn’t for the life of him figure out how to work it.  He didn’t even notice the fancy new functions that took the team spent a lot of time arguing over.  Worse still, he is using one of the basic functions for something completely different to what the project team had intended.  There is a slight moment of panic.  Perhaps he’s an exception.  How could he not know how to work it?  It seems blindingly obvious.  A few more ‘corridor tests’ reveal the same worrying problem – no one seems to know to use it properly!

Just to be safe, the project manager gets someone to do some usability testing.  That should fix it.  They will come up with some ‘quick fixes’ and everything will now be fine.

Most readers on this newsletter will be familiar with similar stories, or have experienced it themselves.  While this narrative is somewhat dramatized, it’s not uncommon.  Say hello to the usability band aid.

In real life, band aids are messy.  They are seldom big enough to cover the wound.  It’s sticky, and it hurts to take it off.  They can increase the risk of infection and can even slow healing.   Unfortunately, the usability band aid is very similar to its real life counterpart.

Fortunately, it is possible to prevent the need for a band-aid altogether.  We’ve been fortunate enough to work with some clients who employ usability not as a messy band-aid near the end of the project, but more like vitamin supplements.  Vitamin styled usability is not only a good preventative; it can also serve as a powerful competitive advantage.

Good usability demonstrates respect for users.  It shows that customers’ needs have been considered from the start rather than added on as an afterthought.  While this simple truth is nothing new, it is surprising how seldom we see it exercised with websites, intranets, software and the plethora of consumer electronic devices.

Just like real vitamins, proactive usability needs to be taken in daily doses and takes commitment.  But that doesn’t make it hard.  Besides, consider the benefits.  In the end, the products that standout all share common design traits, including good usability that has been planned for.  Google, iPod, Nokia mobiles, Tivo, Amazon, Flickr - all products that are easy to use and enjoyable.

What about what you’re working on now?  How can you start taking usability vitamin supplements?