How to put the ‘real’ in ‘realistic’ user testing – 10 tips

Posted by Sharon Cartwright in user testing on May 4, 2012 | 0 comments

The opportunity doesn’t arise for every user testing project, but occasionally the perfect project emerges that is a great candidate for more realistic user testing.

Generally user testing participants are pretty good at suspending disbelief and engaging with ‘make believe’ type tasks in a session, but seeing participants in a more realistic situation definitely returns richer insights. If the opportunity arises to make your testing more ‘real’ and less ‘make believe’ you should take it.

So how do you make your test more realistic? If you want to understand how users buy things from your website – get them to do it with real money. If you want to find out the pain points when users sign up for a landline and broadband service – find people who genuinely want to do this and are prepared to do this in a session.

Here are 10 tips to help you get started:

1. Right people at the right time

It’s always important to test with the right people and even more so when participants are conducting a real task. Make sure you have participants that are genuinely motivated to complete the task and that the need is present. For example, if you want someone to sign-up to a mobile contact for 12 months during the session, it is important to recruit participants that aren’t locked into another mobile contract already.

2. Be generous

Participants are likely to feel more pressure in a more realistic test so you’ll want to beef up the value of the incentive. Make sure the gain equates to the pain. If participants are well rewarded they will be more accommodating and accepting of doing genuine tasks with an observer. At a minimum double your usual incentive. If you are getting someone to sign up to a paid service consider paying for that service for a limited time e.g. the first year.

3. Don’t give it away

You don’t want your participants ‘preparing’ for the session beforehand, so avoid being too explicit during recruitment about what will happen in the session.

4. Incorporate the incentive into the session

Testing realistically for e-commerce websites means participants are going to have to part with some real cash during the session. This is where a generous incentive comes in handy – get the participants to spend it during the session. But make sure they bring their credit card in case they want to spend more than the incentive!

5. Be prepared

Make sure you allow the preparation time needed for realistic testing. You will need to think through all the possible scenarios. For example how does the participant get remunerated if they were to spend their incentive during the session and they can’t find anything to buy? What if the participant sees something they want and it’s more than the incentive? Prepare for these situations which do arise.

6. Allow more time

When people are genuinely engaged in a task it takes longer. Participants spend more time reading content, reviewing details, comparing products and making decisions. We watched one participant spend 5 minutes comparing the same mobile phone in both pink and silver. Extend a standard 1 hour user test to at least 1 and a half hours.

7. No pressure

Although you might have research objectives you are eager to meet, ethically you need to ensure your participant is comfortable and not pressured during the session. AT the beginning of the session highlight any standard rights they may have when purchasing from a website e.g. ‘Change of Mind’ guarantees, and ‘Rights of return’. If a participant doesn’t find anything to buy in the session you will still get really useful results by asking them to pretend, and a happy participant at the end of the session.

8. Have help at hand

Be prepared for participants to get stuck in a session and have a plan for how you will deal with this. You will want to see how they would try and resolve their issue themselves, but if they reach a point when they would give up, you’ll need to be able to provide them with the information they’ll need to proceed with confidence. You might know the product or service well enough to help them, but if you don’t, consider having a subject matter expert on hand. You may want to position them down the hall with a phone and allow the participant to call them as they would usually call customer support.

9. Have a back-up plan

Not all participants will be comfortable with what you have in store for them, so make sure you have a back-up plan if a participant doesn’t want to play ball. In the test introduction run through how you want the session to run. “I’d like you to try and spend the $200 incentive in the session today.” See if they are comfortable with what you are proposing; if they aren’t, use a scenarioed script (you have prepared earlier).

10. Follow-up

Consider follow-up research with the participants to understand the wider user experience. If they made a purchase during the test, find out how the rest of purchase process went . Did the product turn up? Was it delivered in an acceptable timeframe? Was it what they expected?

If you aren’t already looking for the opportunity to inject some realism into your next round of user testing, start now.

May 4, 2012. Posted by in user testing.

Sign up to our email newsletter

Sign up to receive regular updates on our latest projects, research, and other news in the world of usability.

Email addresses are not sold or given to anyone.

Twitter updates