The paper on enterprise search satisfaction factors by Dr Paul Cleverley and Professor Simon Burnett, referred to in Part 1, runs to 24 pages, and it is impossible to do just to the depth of analysis on a single blog post. Nevertheless, there are some outcomes from the research project that stand out as being especially worthy of close attention by the enterprise search community. In commenting on them in this post I run the danger of taking them out of context of the research and analysis so I would encourage you to read the full paper.

In the paper the three factors identified that predominately influenced satisfaction were technology, information quality and information literacy and task utility. The technology factors include search tool reliability, search ranking and query syntax handling. In total these factors were the largest single group (38%) and that could be used as a justification for investing further in search technology. However together information factors (36%) and literacy factors ((26%) accounted for 62% of the reasons for dissatisfaction and to me that indicates that technology investment on its own is not going to make a significant difference to search satisfaction.

Moving on to search-level metrics, the search application was used by around 70,000 staff each month and generated over 450,000 search queries. The average query length was 1.89 words and the top 30 most frequent searches fell from 14% of all search queries at the start of the project to just 8% at the end of the project two years later when of course users had gained substantially more experience with the application. This confirms my anecdotal evidence that the tail of low frequency queries is very long in the enterprise environment. In my view this has significant implications for ‘cognitive search’ because there will be such low levels of use data from the majority of the queries to be able to predict optimal results. The percentage of results with ‘no results’ decreased from 0.4% to 0.3% over the same period. These metrics are the baseline that search managers have been seeking for years without success.

Throughout the section in which verbatim comments are included it is very clear that resolving these comments requires a knowledge of the technology, the content and the use case. This requires a skilled search support team, but from the outcomes of the research this team also has an important role to play in overcoming the problems of a lack of search literacy. The literacy factors include the selection of a query term, results scanning and familiarity with the search application. As over 25% of all dissatisfaction events were attributed to poor search literacy there has to be a major question mark over the view that when technology and information content are optimised search outcomes will take care of themselves.

My final comment is so well stated by the authors that I will reproduce it from the paper.

“The importance of configuration in enterprise search was evident, where an unintentional change biasing documents over web pages let to sub-optimal results. With an average query length of approximately two words, made by users to dynamic growing corpus sizes, it is unlikely that many information needs will be met without constant configuration, promotion of authoritative (trusted) corporate information and monitoring of performance”

With all research projects there is always the danger that the outcomes are not scalable and extensible to other organisations. Reading this paper so many of the comments aligned with the experiences I have had with probably 100 or more enterprise search-related projects that I have every confidence that the outcomes will translate (with due care and attention) to any organisation that depends for its future success on being able to offer complete and effective access to global information repositories.

No matter how small or large your organisation, if you have responsibility for search management you should be taking this remarkable paper, marking it up para by para, and then using it to benchmark your approach to achieving the levels of search satisfaction that your employees expect. I have to applaud the authors and the company that supported the research project for commitment to what was clearly a very challenging project. This research will change the way that the enterprise search community (and that includes search software vendors) consider the opportunities and challenges of effective enterprise search management. I do hope that this will encourage other research teams to build on the outcomes.

Martin White