Attivio, Coveo and Sinequa are just three vendors that are positioning themselves as being in the insight/cognitive search sector, playing down the ‘enterprise search’ label. Gartner has endorsed this approach in its latest Magic Quadrant on Insight Engines. Of the various comments I have seen Paul Nelson (Search Technologies) has posted the most balanced perspective on this trend. I especially valued his comments on the Gartner attitude to open source applications. If the Gartner definitions exclude a consideration of these applications then it is time for the definitions to change! His post also helpfully provides links to the Magic Quadrant itself. If I take an information retrieval perspective on this trend then I see many very interesting research papers which look at the possibilities of natural language processing, neural networks and topic identification in supporting complex discovery tasks. Exploratory search presents some significant challenges in user/application dialogue management that are way beyond the bot-level “Find me people who speak Spanish” known-object type of search.

From an enterprise search development perspective I see two major barriers to the immediate widespread adoption of these technologies. The first is that there is no consideration of another aspect of cognition, namely cognitive barriers to search. Professor Reijo Savolainen, at the University of Tampere, has been studying these barriers for many years. His paper on Cognitive Barriers to Information Seeking should be essential reading for search managers and vendor sales teams. These barriers cannot be overcome by making applications ‘intuitive’. The issues are far deeper.

The second, and much more intractable barrier, is that there is a very significant shortfall in people with the skills, training and experience to both develop and implement these applications. It is a topic that I considered in some depth in a report co-authored with Stavri Nikolov for the European Commission in 2013. In my new report Making Search Work I bring together research from AIIM and Findwise which highlights the fact that around 50% of organisations have at most a single person working on optimising the search experience and trying to gain an acceptable level of user satisfaction. The scale of the requirement is massive. Anyone running a SharePoint installation will by now have begun to realise that search needs a full-time search manager to get the best from the search functionality. So that makes 250,000 SharePoint licensees looking for search management skills. Add in other commercial and open source installations and that might well bring the total to over 300,000. You may regard these numbers as over-the-top. So assume that only 1 in 100 of these organisations needs a full time search manager. That is still 3000 vacancies and there are not 3000 people out there fully trained up in both the technology of SharePoint search and the wider search management skills and looking for jobs.

I’ve commented before on the value of some form of certification for search managers. As far as I aware (and I’d love to be proved wrong!) there are no vendor-neutral training courses for search managers nor are there any university courses. There is a MOOC on Text Retrieval and Search Engines which has been developed by Professor Chengxiang Zhai, and this forms the basis for his excellent book (with Sean Massung) on Text Data Management and Analysis. The very fact that this book runs to over 500 pages is an indication of the depth of the topic.

I am certain that there will be many organisations that would benefit from the next generation of insight engines. A few will have the in-house resources to implement these applications. Many more will either fail to invest in, or be unable to find, employees with the skills they need. When the application fails to deliver significant value the blame will be on the technology, and that will not be good news for the vendor community or for the organisation as it seeks to create competitive advantage from the technology investment. What worries me most of all is that the majority of organisations still do not appreciate the value of information discovery. After all, if you can’t find information you can’t use it or share it.

Martin White