Many people take the view that the amount of time taken carrying out a search is a good measure of the quality of the search experience, and that the focus should be on reducing the time to gain a measurable productivity improvement. In my view any change made in an information system with the aim of improving productivity is totally misguided. In 2008 James Robertson wrote an excellent blog setting out 25 reasons why saving time on an intranet is a bad metric. Others take the view that if a user spends more time on a search as a result of a change to the search application then that is also an indicator of better quality search because the users have not become frustrated with the search experience, given up and phoned a friend.
At the Fourth Information Interaction In Context Symposium, held in Nijmegen earlier this year Pia Borlund (Royal School of Information Science, Aalborg, Denmark), Sabine Dreier (Aalborg University Library) and Katrina Bystrom (Swedish School of Library and Information Science, Boras, Sweden) presented a paper on research they had carried out to try to determine what the period of time spent on a search actually indicates. They used two different research approaches, a laboratory-based study and a more naturalistic study. After the searches had been completed the participants were asked if the time spent on each search was an indicator of whether or not the search was interesting, whether or not there was a lot of information to search through, and whether or not the topic was interesting. The conference paper also reported on a parallel enterprise search project undertaken in a large engineering company using electronic diaries to capture the time spent on a range of tasks.
The conclusion from a review of the outcomes of all three studies was that there is no direct link between the time spent on the search and any one measure of search quality or search performance. The range of responses to the questions set out above was wide and no analysis of them came up with a single answer. Search is a complex process. The amount of prior knowledge of a topic, the performance of the search application, the amount of information retrieved and a number of other factors should all be taken into account when assessing search satisfaction and search performance. This was quite a small scale study and the authors point out that more work needs to be carried out in this area. Nevertheless the main outcome for me is that any measure of the time taken to conduct a search has little value without taking a lot of other factors in to account.
A signficant amount of research has been and is being conducted into information retrieval performance, and the need to take great care in the design and interpretation of tests to assess search satisfaction. One of the reasons why enterprise search needs a support team is that there needs to be someone on the team that has the time and expertise to monitor the information retrieval research literature. There is a significant amount of this research which can be used both to design test procedures to evaluate search performance and provide ideas for how how search performance can be enhanced.
Martin White