User Testing

Why do user testing?

In a user test, the user interface to a product or service is tested by real users carrying out real tasks in realistic conditions.

User testing is the most effective way to discover any barriers that users will face.

A user test will identify aspects of a design that cause users difficulty, confusion, or misunderstandings. These may lead to errors, delays, or in extreme cases inability to complete the tasks for which the product or service is designed. For example, a user test can answer the following questions about the elements of a user interface and the contents of the service:

  • Do users know it is there?
  • Do they understand it?
  • Can they understand how to operate it?
  • Can they use it successfully to complete the tasks it is designed for?

The evidence is objective and realistic, not based on personal opinion or speculation.

User tests also provide insight into users' preferences. The information that is gathered enables the designers to improve the interface in the next design iteration, so that it better fits users' requirements.

For designers and service providers, user testing has the following benefits:

  • It gives confidence, by verifying that real users can use it successfully, or identifying what prevents them from doing so
  • It helps designers understand users and see things from their perspective, so that they are more likely to design something that works for users first time
  • Video evidence and user test reports can be used to communicate issues to other people involved in the development or implementation process

Is it expensive?

The expense of user testing is generally far less than the cost of getting it wrong. Even if a design seems to be accessible, or is theoretically accessible, if it is not tested with real users and real tasks, there is a risk that it may not be accessible in practice.

The expense of user testing is generally far less than the cost of getting it wrong. Even if a design seems to be accessible, or is theoretically accessible, if it is not tested with real users and real tasks, there is a risk that it may not be accessible in practice.

If it cannot be used for its intended purpose, it will be condemned by customers, may lead to extra support and training requirements and will ultimately have to be redesigned. This may be particularly costly since, in general, changes made late in the development process cost far more than changes made early on.

What happens in a user test?

A user test has three essential ingredients:

  • Representative users
  • Real tasks
  • A controlled environment

The actual testing is just one part of a methodical procedure that involves the following stages:

  • Planning the test
  • Recruiting users
  • Scripting tasks and scenarios
  • Running and observing user sessions
  • Collecting data
  • Analysing the observations and data
  • Reporting issues and recommendations

Typically, users are asked to "think out loud" while they are carrying out a task. What they say reveals their assumptions or beliefs about the product or service, their goals or intentions, and any confusion or misunderstandings. Combined with observations of what the users actually do, this allows the testers to determine why they are having difficulties, making errors or getting stuck. Testers usually remain silent, but may sometimes ask the user questions, either during or after the task, to clarify issues or gather further information.

In addition to these qualitative observational tests which reveal the existence and causes of difficulties, testers may also carry out separate quantitative performance tests. They may measure the time required to complete specific tasks or the error rates at various points in the design. This data can then be compared with benchmarks established during requirements gathering, to determine whether accessibility criteria have been met, or to compare two or more alternative designs.

How does user testing compare with other types of testing?

User testing is very different from market research techniques such as focus groups and questionnaires which are designed to find out people's opinions and attitudes about a product. They rely on people reporting their experiences after consideration and may not employ a realistic use scenario. They are not useful for determining behaviour and experience as it happens during task performance.

It is also different from functional testing such as bug testing, load testing or stress testing. These need not involve real users at all.

User testing is also different from other usability or accessibility investigation methods, such as heuristic evaluation, expert evaluation, compliance testing or cognitive walkthrough. However, it is often useful to combine user testing with these methods at different stages in the design process

When should you do user testing?

User testing can be carried out at any stage of a design, from the initial drawings, through successive prototypes, to the final product. For example, the information structure and interaction style can be tested with a non-working prototype by asking representative users to indicate how they would carry out a task given a particular design.

In general, the most efficient way of carrying out user testing is to do it as early as possible in the design process, so that fundamental problems are eradicated before further design work is carried out. However, this may be difficult where users use assistive technologies, since these devices may require an actual working product to interface with. In these cases, testing may have to wait until a working prototype has been built.

The best general approach to testing is to start early and test often, using an iterative development process of design, test, modify, design, test, modify, etc.

Who should do user testing?

Planning, scripting, running and analysing a user test is a skilled activity. There are many ways in which untrained testers can introduce bias, fail to target representative users and tasks or misinterpret user behaviour. If this happens, the results will be at best useless and at worst misleading. If possible, it is best to engage specialist external consultancies to carry out user tests, or at least to advise and train internal staff.

Wherever possible, internal staff such as developers and project managers should arrange to attend user tests as observers. This can help enormously with promoting understanding and creating a culture of accessibility.