I expect that few readers of this blog will have read the 2011 edition of Modern Information Retrieval edited by Ricardo Baeza-Yates and Berthier Riberio-Neto. It runs to almost 900 pages. Chapter 15 is a brilliant essay by David Hawking which sums up everything you should know about enterprise search in 40 pages.
The Commonwealth Scientific and Industrial Research Organisation is the national research organisation in Australia. David joined CSIRO in 1998 as a research scientist but had been working on the technical issues of ‘enterprise search’ since 1991. When Google arrived on the search scene CSIRO saw an opportunity to address the problems of enterprise search by commercializing some ongoing research into text retrieval and from it created P@ANOPTIC. This very capable search application had a number of very neat technical elements (read the book and you will be amazed) and quite quickly gained a collection of highly satisfied users from operations in Australia, the USA, the UK and Poland. The company was spun off in 2005 as Funnelback Pty Ltd and was sold to Squiz in 2009.
P@ANOPTIC was a team effort in which the vision and expertise of David Hawking played a crucial role. I first met David in 2012 when working on a project about the European Enterprise Search market for the European Commission. David was on the Steering Committee and I am forever grateful for the way he steered the discussions and also generously supported the work I was undertaking, which at the time was not going down well with the project sponsors. A couple of years later I worked on an implementation of Funnelback at University College London, and was very impressed with its functionality and the very high level of customer support.
David retired in 2018 and has now written the story of his involvement in the development and exploitation of P@ANOPTIC, and what a story it is. Reading ‘Funnelback and Me: Celebrating 30 Years of Funnelback Technology 1991-2021’ will give you a very clear understanding of the depth of technology behind the search box and the development effort involved.
To quote from the Readers Guide that David has included in the book:
“This book tells the story of Funnelback and its precursors in some detail. My goal is to be “entertaining and accurate” rather than “comprehensive.” As noted in the Acknowledgements, the story is told from my own perspective, though I’ve interviewed and/or consulted many key players. Large parts of the story are scientific or technical in nature. I have tried to write up these parts as clearly and simply as possible, but I understand if you want to skip them. I have tried to record the scientific questions we investigated, the software and datasets we built, and when we did those things. I certainly don’t intend to claim scientific priority for everything we did. We were the first to do some things, but in many cases we were following up ideas due to others, or applying them in new areas.”
You can read this book at so many levels; the commercialization of academic research, the importance of having the right team and the right culture, the complexity of the technology and the determination of one man to build a better mouse trap. I can recall no other autobiographical account of life inside enterprise IT. Perhaps only Tracy Kidder’s book The Soul of a New Machine comes close.
I don’t have to encourage you to buy this book as it is open access at https://openresearch-repository.anu.edu.au/handle/1885/265519. Interestingly you can download it at normal (30MB) or high (104MB) resolution!
If you have any interest in information retrieval and enterprise search you will gain much from this book, which runs to 265pp and includes many photographs of the CSIRO team. You can jump over the technology if you wish and just enjoy the story of David’s journey of discovery over a period of 30 years. David has long been one of my guiding stars in the world of enterprise search and I just wish I could have accomplished even a very small fraction of what he has contributed in the pursuit of enterprise search perfection. Thank you David!
Martin White