I’ve just finished work on a report on Big Data for a client that required me to work through the two dozen or so reports that I have collected on the subject and take a detailed look at the market opportunity. When the October 2012 issue of Harvard Business Review includes three papers on Big Data then it must be main-stream and it must be true that the future of the world depends on companies investing in Big Data solutions. But is it? A cliché of Big Data is that it is a function of the volume, velocity of change, variety and value of data collections. That may be the case but how many companies have reliable measures of the first three of these and a dependable algorithm for determining the value of data integration? There are certainly companies operating in the financial services, retail, manufacturing and exploration sectors that will be able to make use of Big Data applications.
However there are 20 million businesses in the EU and 99.8% of these have 250 or fewer employees. That leaves 40,000 companies with more than 250 employees, and a significant number of these have less than 1000 employees. By the time you strip out companies who by the nature of their business are never going to fulfil the volume, velocity and variety criteria the conclusion I come to is “Big Data – Small Market”. Now it is certainly the case that companies in that Small Market will make substantial investments in data management but in reality many have already done so. The Vestas wind turbine company in Denmark already uses a supercomputer to decide where to erect turbines.
Part of the folk law of corporate information is that 80% is unstructured and 20% is structured. No one (least of all Autonomy) has ever been able to justify this statement but then no one has been able to prove it is incorrect. If we assume that in fact the balance is 50-50 there is still a substantial requirement to be able to search unstructured data. A number of recent surveys (notably from Findwise) show that companies are coping very badly with finding information. The Findwise survey indicates that 95% of respondents cannot be sure of finding the information they are looking for because of the poor performance of their search application.
This inability to find unstructured information will also have an impact on data analysis. There is an excellent survey from AIIM (Big Data – Extracting value from your digital landfill) that looks at the role of both structured and unstructured data in business decision making. Over 60% of respondents would find it very useful if they could correlate text-based data with transactional data, but only 2% are able to do so at present.
At a recent European Commission meeting on search computing a senior executive of a major search vendor told the audience that enterprise search is dead and that data analysis is where the action is. I see a different picture, with companies starting to realise at long last the importance of being able to search unstructured information. This is driven partially by a growing inability to find information in SharePoint 2010 implementations because of a lack of understanding of how to make search work and a total lack of investment in a search support team. If you talk to search vendors and search integration companies in Europe they have more work than they can cope with – their challenge is finding employees with the right set of skills and experience.
Enterprise search is not dead – Big Data is just going to increase market demand. My view is that ‘enterprise search’ is providing a managed framework of search applications which enable employees to find the data, information and knowledge that they have the right to access in order to make business decisions, and that includes Big Data applications.
Martin White