During the search process, searchers need to decide when they should abandon the current query (and perhaps issue a new query after examining the current results list), and when to curtail their search by stopping the search session altogether. Knowing when to stop is considered a fundamental aspect of human behaviour.  Stop too early, and important information may be missed. Stop too late, and time and effort is wasted. Worse still, the examination of fruitless result lists will mean not having time to examine other lists which may potentially contain greater yields for the searcher. David Maxwell and Leif Azzopardi (School of Computing Science, University of Glasgow, UK) and Kalervo Järvelin and Heikki Keskustalo (School of Information Sciences, University of Tampere,Finland) have published a fascinating research paper on this topic. The four authors are in the very top echelon of IR research so what they have to say should be taken very seriously.

Two of the earliest stopping rules proposed were devised by W.S.Cooper in 1973 (search goes back a long way!) who proposed:

• the frustration point rule, where a searcher stops after examining a certain number of non-relevant documents; and
• the satisfaction stopping rule, where searchers would stop only when a certain number of relevant documents were found.examine other lists which may potentially contain greater yields for the searcher.

The frustration point rule is especially interesting. The authors define it as counting the number of non-relevant documents seen in the ranked list at position k. If the total number of non-relevant documents exceeded
a given threshold, the searcher would then stop. So if we have a personal rule that if we have got to the third page of results (say k = 30) and found few if any relevant results then we give up in frustration. Our time is too precious and we may, or may not, start again. I will do the authors a great disservice by jumping to the end of their paper but the main outcomes are that the two most common stopping strategies are

• Fixed Depth. Under this stopping strategy, the simulated searcher will stop once they have observed a self-defined number of results snippets, regardless of their relevance to the given topic.
• Contiguous Non-Relevant. The searcher will stop once they have observed (say) 5 non-relevant snippets in a row (contiguously).

The authors caution that a great deal more work needs to be done to understand these behaviours. However the research indicates that we may need to rethink our approaches to evaluating search success and search failure, at least taking into account search users strategies which may be internalised and pragmatic rather than just a function of relevance. It  would seem to put a priority on precision over recall but that might be taking the research too far. It would also have an impact on session time. Indeed it might be interesting to look at the variance of session times for search users and see if there are any patterns.  I should note that is paper was given at the ACM CIKM’15 Conference held in Melbourne, Australia,  in October 19 – 23, 2015 and so is only available to ACM members unless you are willing to pay an access fee.

Martin White