Evaluating search performance is a challenging task, and there is even a book largely devoted to the subject. The mathematics can get quite complicated. When I start an enterprise search consulting project I need some fairly instant metrics of search performance that enable me to have interesting discussions with senior managers about their unwillingness to invest not only in search technology but more importantly in a search support team.
Over the last couple of years I have found FYOR to be a useful metric. It stands for Find Your Own Report. Early on in a research interview I ask the interviewee to find a report that they had written perhaps 12-18 months ago using whatever search application(s) they have available to them. The only rule is that they cannot use any of the words in the title of the document as others looking for information that is contained in the report may not know the correct title. It is interesting to see how they go about creating a search query, often using just a single word. The results are always fascinating viewing.
Very rarely does the report come on the first page of the results, and the interviewee is initially surprised to find out just how much information is available on the subject of the report they wrote! You can see a look of great disappointment cross their face when the report is found about five pages down in the search results as they know themselves that it has entered the Zone of Invisibility. Usually the ranking will be higher when I ask them to search on words in the title, and then they begin to realise how important it is to ensure documents have meaningful titles and ideally some consistent metadata.
Of course there are risks with the FYOR approach but I have a store of prepared comments to cover most eventualities! This approach enables me to get to the heart of search performance problems within a few minutes of starting the interview. For example the question “Who would you contact to find out why your report does not appear on the first page?” usually identifies that the search team are as invisible as the report, assuming of course that a team or even a single search manager is in post. The FYOR approach can easily be extended to finding people. presentations and other information items. I do ask interviewees not to tell their colleagues about the FYOR challenge after finding that a couple of managers had clearly been practising prior to me meeting them!
Martin White