The concept of ‘relevance’ lies at the core of enterprise search development and assessment. Many (too many!) vendors focus on the promise of presenting the most relevant results on the first search results page (SERP) using AI. The concept dates back to development work at Ramo-Woolridge Corp on search applications by Melvin Maron who in 1958 recognized that a binary Yes/No assessment of the value of a document was far too limited and that there was a gradation across a document set. Maron’s initial idea was then developed further by his colleague Larry Kuhns who evolved the concept of a vector metric of similarity that remains a fundamental element of most commercial enterprise search applications.
A paper published in 1996 by Stefano Mizarro reviewed the various approaches to defining relevance. This paper runs to 42 pages with citations to over 250 research papers. Many more have been published since then. Google Scholar currently lists over 3000, with almost 1000 of these having been published in the last four years. Relevance continues to be a very significant challenge.
In the case of enterprise search relevance is not the whole story. Under test conditions it is of course entirely possible for an employee to assess the ranking of the notional relevance of a set of documents against the query. Repeating that test with a number of users will probably result in a broadly similar ranking. Based on those assessments all sorts of complex mathematical performance metrics can be calculated.
Job done? No!
Some years ago David Hawking suggested that ‘utility’ was a better measure of the immediate value of a document to an employee. In the enterprise employees tend to build up collections of documents for two reasons. The first is that they get sent a substantial number of documents by a range of other enterprise applications, which include email. They may well rank documents as relevant to the query but as they already have them the items are, paradoxically, also irrelevant.
The second is that in the majority of organisations the level of trust in the search application is so low that they feel they cannot trust search to deliver the information they need to make business (and indeed career) critical decisions.
Research by Paul Cleverley and Simon Burnett indicated that the factors that affect enterprise search satisfaction are technology, content and training. Enterprise content is not curated in the way that web and e-commerce is managed. Add to that the assumption that search is intuitive and no training is required and you account for two out of the three factors!
It is also important to note that there has been no published research on the modes of search inside an enterprise. There is a limited amount of research on professional searchers (employees whose role requires intensive use of search applications) which indicates that (for example) lawyers search in very different ways to patent agents but this does not scale across an organisation. I count myself fortunate that over the last 15 years I have gained a substantial amount of experience about the use of enterprise search applications but have never got around to publishing the outcomes. There is certainly a range of objectives from users from a high precision result to a high recall set of results. Understanding the intent behind the query is very difficult when any single employee has multiple roles, responsibilities and objectives and may well be searching for information on behalf of their team, not themselves.
If you are attracted by one of the many vendors offering solutions to enterprise search it might be worth asking how they gained the knowledge to develop applications that meet the very diverse personas and intents of enterprise search use. They should have a track record of satisfied customers dating back a number of years and not be proposing to use your organisation as a beta release opportunity. That should quickly give you a short list of vendors to contact. Note that experience with web search, or even e-commerce applications, does not translate to the enterprise environment because the issues around content curation referred to above.
Martin White