In 2000 Jakob Nielsen suggested that a sample size of five was adequate for a web usability test. This has given much comfort to web and intranet managers struggling to find the time and resources to carry out usability tests. This ‘magic number’ was not seriously challenged until 2010 when Hwang and Salvendy published a paper in the May 2010 issue of Communications of the ACM (Association for Computing Machinery) which analysed a number of research papers on usability testing and suggested that the optimum number of users was 10 +/- 2. To be fair to Jakob Nielsen he highlights the fact that ideally 15 users would be required to find all the possible usability areas, suggesting that 5 would find 85% of usability problems. However that caveat is usually ignored by those promoting the benefits of small test sizes.
Now Martin Schmettow has written a paper in the April 2012 issue of Communications of the ACM in which he reviewes all the literature on test sizes for usability testing. His conclusion is that not only is 10 +/- 2 too low, but that there is no ‘magic number’ for usability tests. If that is the bad news the good news is that there are approaches which can indicate the number of undiscovered usability problems from the initial outcomes of testing. This may seem improbable but there is a well-established precedent in the capture-recapture methodologies for quality management in software development.
Schmettow’s paper is a very detailed analysis of the problems in arriving at usability test sizes and rather than try to summarise it in this blog post I would recommend you download it from Schmettow’s site on Mendeley. It is well worth reading. The main message is that we should not be complacent about undertaking small-scale usability tests and assuming that from these we have found all usability problems. Nielsen indicates that a sample size of 5 may only find 85% of usability issues, and that raises the issue about whether or not the 15% that are not identified are in fact more important to identify and solve than the 85% which have been found.
Martin White