This is the third in a series of posts sparked by the recent publication by Gartner of a Magic Quadrant for Insight Engines and the Forrester report on Cognitive Search, in both cases making assessments of search vendors on their capabilities to deliver cognitive search. In Part 1 I looked at the industry analyst business and in Part 2 at the long history of natural language processing which is a foundation for cognitive search. It is time to move on to these reports.

My first observation is that industry analysts have a fundamental duty to cut through vendor sales language and present a reality check on what is being promised. The introduction to the Gartner report reads “Insight engines apply relevancy methods to describe, discover, organize and analyze data. This allows existing or synthesized information to be delivered proactively or interactively, and in the context of digital workers, customers or constituents at timely business moments. Insight engines gather content from a broad spectrum of sources, including those relevant to the immediate needs of digital workers. They extend beyond enterprise search by providing the capability to engage with content and extract insights without touching the source of that content. Natural language and rich context are used in such interactions, which will come to rely less on a search box and a button and more on proactivity over time”  A button?  I’m trying hard to work out the connection between a search box and a button. If you want clarity on cognitive applications then read Seth Early’s analysis. Informed and balanced. Everything an industry analyst should be delivering. Then I try to understand the section about extracting insights without touching the source of the content. What does that actually mean and what is the implication for an IT professional? Search me!

Forrester states that it has “observed a significant change in the technologies that search technology vendors use, such as natural language processing and machine learning, to dramatically improve the effectiveness of these solutions.”  The company goes on to say “Search is about finding answers, content, and documents; adding useful context to apps; and augmenting human intelligence. The most important requirements are the relevancy and completeness of the returned results.”  I find myself asking ‘So what?’ I cannot understand what “the completeness of the returned results” means. In particular I am intrigued by the statement that cognitive search ” can dramatically improve the effectiveness of these solutions.”  Here is a very important statement presented with no evidence at all, and this is from a vendor that sets out scores accurate to three significant figures based on subjective opinions on search vendors. It would be of immense value to the search business if Forrester included a summary of many A/B  case studies they must have undertaken to be able to justify this statement.

Then I come to this statement. “But it’s also a significant opportunity to dramatically increase employee productivity if search technology can return highly relevant results. Keyword-based enterprise search engines of the past are obsolete.”  There are around 250,000 SharePoint installations in the world all using keyword-based search. Forrester has declared them obsolete. Inside SharePoint, as indeed with most search applications, is a term (i.e.keyword) ranking formula known as TF-IDF. This stands for Term Frequency – Inverse Document Frequency. Increasingly it is incorporated into the BM25 ranking model and is the core ranking model inside Lucene. So Forrester have not only written off SharePoint search as obsolete but any implementation of Lucene. A very irresponsible comment from an industry analyst.

I’m currently advising a global company with 120,000 employees that uses SharePoint 2013 to deliver very high quality search results in response to keyword searches. It works well as a result of having the right set of skills and a clear understanding of user requirements. At a project meeting next week I have to tell the company that it is using obsolete technology. According to Forrester “Enterprise search has evolved Into something delightfully different”.  I’m sure I will have delightful fun explaining that statement to my client. Notice also that Forrester state that ‘search has evolved’, not ‘is evolving’. I must have blinked and missed that transition.

Forrester actually excludes Microsoft from its survey on the basis it does not market search as a stand-alone package. That totally misses the point.  IT managers will be interested in a view from Forrester on whether Microsoft technology can, or will be able, to deliver cognitive search so that they can decide whether to move to an alternate search application. If you want insights on Microsoft Search Jeff Fried provides the clarity I would expect to see from an industry analyst.

I don’t want readers of this blog to think for one moment that I am not enthusiastic about the potential opportunities for ‘cognitive technologies’. Anyone who has heard me speak or read my book will know that I’m passionate about supporting anything that helps organisations to be certain that their employees can find all the relevant information they have collected over the last few years. But when industry analysts with the brand strength of Gartner and Forrester go about inadvertently creating the Fear, Uncertainty and Doubt that Charlie Hull refers to they are doing a disservice to the vendors they profile, the IT professionals they seek to serve and employees who need to find information.

Search lies at the heart of a digital workplace, be it keyword or cognitive. It’s time that industry analysts recognised this and paid serious attention to informing and encouraging the adoption of search technology. My question to search vendors is this. The position you have on the Gartner and Forrester charts makes for good promotional copy, but do you buy in to all the comments in the reports? My recommendation is that you highlight your success stories and make your own news.

Martin White