Search technology can fail to deliver in so many ways that understanding the modes of failure is not at all easy. There is a superb diagram from Sam Marshall (ClearBox Consulting) which summarises the challenges. At a search-by-search and day-to-day level you may well have only limited evidence as to how well your search applications are working. It is very easy to become complacent, especially if there is a work-around in asking co-located and remote colleagues if they have the information you are seeking.
For this reason I have evolved a set of ten search stress tests (based on failed searches in client engagements) that will give you some sense of whether you can trust your enterprise (and/or intranet search) to meet your personal requirements. All of the tests are based on common enterprise enquiries and often feature in ‘moan-lists’ about search not working. Each test should only take 2 minutes to complete. If it takes you longer then you have failed the stress test even if eventually you track down the information. All ten tests should not take more than around 20-25 minutes but of course there is no reason to do them in a single session.
A good search application supported by a search team should deliver satisfactory results for all these tests. It is not just about the technology; these stress tests also check out the quality of the information management in your organisation.
Test 1 Specific document
Think of a document you wrote in 2018 that is stored as a pdf file. Then search for it without using any of the words in the document title. The reason for this is that probably only you can remember exactly what the title was, and it might be along the lines of ‘Q4 report on Project Solon’. You can use as many query terms as you like. The idea that people only use single query terms is erroneous. The choice of a pdf is to assess the performance of the indexing of pdf documents which present significant challenges.
Test 2 Version discovery
Select a policy or procedure document that has been through a number of (perhaps annual) revisions and then search for all the previous versions, justified by wanting to know when a particular section was added or deleted.
Test 3 Confidentiality
Search for the term [confidential] and reassure yourself that you do have permission to see all the results returned. It should also bring back the protective marking scheme for the organisation.
Test 4 Expertise discoverability
You do of course have some specific expertise that justifies why your organisation hired you. Using no more than two query terms see how high up you come in the list of potentially relevant experts. No-one is going to look beyond p1! The query term number is limited to 2 because in expertise search people often do not have the expertise to define the expertise they are seeking. Strange but true. If you do not appear on page 1 of the results (and so are potentially invisible) how many query terms does it take for you to be positioned on page 1? Then consider whether anyone will in fact use all these terms to track you down.
Test 5 Project files
Think of any project, either by name or objective, and find the definitive (signed off) final report of the project.
Test 6 Data search
Almost certainly you will be adding information into an Excel spreadsheet on a regular basis. Can you find the last version published in FY2019 (not calendar year) just to check that 2019 information has been correctly included in a document. Not the latest – that is too easy.
Test 7 Presentation retrieval
Recall a PowerPoint presentation you gave in 2019 and search for it using query terms taken from the agenda slide. Does it turn up in the top 10 results?
Test 8 Given name search
The range of family names is always greater than given names. Imagine that someone in a Zoom/Teams meeting suggests that ‘John is the person to talk to’. The assumption is that you know which John (or is it Jon or is his name actually Jonathan?) and what their role and (virtual) location is. Search for a colleague just using their first name and see how long it takes you to track down their profile, role and location.
Test 9 Phonetic name search
You will almost certainly have a colleague whose given name and/or their family name is not spelled the way that it is pronounced. Search for their profile based only on how their name sounds, not how it is spelled.
Test 10 Application support
Select a corporate IT application that you rarely (if at all) use and search for information about how to use the application, including on-line training and comments on social media regarding tips, tricks and work-arounds. Do you feel you have enough to open up the application and use it successfully?