A history of enterprise search 1980 – 1989
In the early 1980s there was a great deal of interest in the UK around the use of text retrieval software running on mini-computers. The STATUS User Group in particular was very active. Because none of the vendors were interested in the commercial success of their products users of these products (they are listed in the previous post) were very willing to share their experience. As far as the technical development of enterprise search was concerned probably the most important advance was the release of the Snowball English language stemmer developed by Dr. Martin Porter. To be pedantic it was first released in 1979 but was not more publicly promoted until 1980. Martin Porter tells the story from a 2001 perspective on his website where his original stemming code and many more algorithms for various languages are available as open source. According to the Wikipedia entry the name Snowball was chosen as a tribute to the SNOBOL programming language, with which it shares the concept of string patterns delivering signals that are used to control the flow of the program.
Martin Porter also developed the Muscat (MUSeum CATalogue) search application while at Cambridge University. Released in 1984 it sought to take advantage of the work of Stephen Robertson and others on a probabilistic approach to information retrieval. Muscat Ltd. became a successful company with clients, including Fujitsu. Muscat was eventually rewritten and released as the open source Xapian library which survived the eventual acquisition of Muscat Ltd. by a short-lived dotcom era company and remains in use today. There is a good summary of Muscat on the Flax website.
By the mid-1980s the IBM STAIRS full-text search application was setting the standard for enterprise search. In 1985 a wide-ranging research study was carried out by Blair and Maron of the retrieval performance of STAIRS, which at that time was being promoted as a litigation support tool. The results were far from impressive. The study is too detailed to try to summarise here, and it is better to read the paper by Blair published as a retrospect in 1995 than the original paper. This study remains the most comprehensive of its type, with nothing approaching it having been published in the last thirty years. It had commercial implications for the legal sector as this was the time when there started to be a number of major anti-trust cases brought by the US Department of Justice where reliable access to millions of corporate documents was of great importance. It should also be born in mind that the IBM PC had been launched in 1981 and the 1980s were the time when documents started to be created on personal computers rather than being transcribed onto word processors.
I would suggest that the first enterprise search application was developed by Fulcrum Technologies, established in Ottawa in 1983. This was a client-server application, rather than mainframe. It was most visible for the rest of the decade as a provider of search software for CD-ROM applications. From 1983 to 1988 Fulcrum pretty much had the search market to itself but failed to make much headway. The arrival of Verity, born in the entrepreneurial climate of California, marked a gradual decline of Fulcrum as a business. A succession of owners over the 1990s led eventually to being purchased by Hummingbird in 1997, which itself was then acquired by OpenText in 2006.
In 1985 Advanced Decision Systems was set up in San Jose, California with the objective of developing expert system and artificial intelligence applications. In 1986 David Glazer and Philip Nelson developed an innovative search application called Topic which was beta tested with success by the US Strategic Air Command. Topic made use of a probabilistic search ranking engine which offered significantly better management of ranking than the Boolean operators that had been used prior to the release of Topic, though STAIRS also used this model. This early success led to the spin-out of what was to become Verity from ADS, led by Michael Pliner with a technical team led by David Glazer and Philip Nelson. The full story of the success of Verity is a story for the next decade but there can be no doubt that Verity was the proto-typical enterprise search application as unlike IBM STAIRS it was platform agnostic. At launch a multi-user licence cost $39,500, quite a substantial licence fee in the late 1990s. Both Glazer and Nelson now hold senior positions at Google.
Two other search software companies started out towards the end of the 1980s. David Thede set up dtSearch in 1988, initially offering a desktop search application. dtSearch remains one of the very few search software vendors to have been in the same ownership from start-up to the present day. Also in 1988 but across the other side of the world in Australia Ian Davies was developing the Isys software suite. This ended up being acquired by Lexmark in 2012. Several others were on the drawing board but did not emerge until the early 1990s.
Martin White