From many years of helping clients create new intranets or install a new search application I know beyond reasonable doubt that there are two processes where the project timeline will diverge from even the most detailed of forecasts. The first of these is in forecasting the time to undertake a proof of concept exercise and the second is in migrating content to the new application. In this column I want to consider the management of a proof of concept (PoC)  exercise with special reference to enterprise search.

A PoC has to have a number of attributes

• It has to be realistic (is this how users will use it every day?)
• It has to be scalable (it works with 1000 items, but what about 100 million?)
• It has to be extensible (it works for the IT Department, but what about R&D?)
• It has to be able to be evaluated (because the proof has to be unambiguous)

Designing a PoC to meet these four attributes is not easy.

Developing the PoC scenarios

The PoC is, in principle, a comfort factor that the vendor selection is optimal for the organisation. As such it is something that should be scoped out at the start of the procurement exercise as the elements of the PoC should ideally be tied into the core objectives of the implementation. Just as examples the PoC might focus on searching a digital asset management system, searching a product lifecycle management application and coping with an archive of scanned project documents from a company that was acquired a few years previously.

As the process of defining requirements and making the business case proceed the scope of the PoC may change, and that is a good outcome. What should be avoided at all costs is allowing the vendor to turn the PoC into a demonstration of the advanced functionality they are offering you. It is also of the greatest importance that your lead search manager is able to devote all their time to supporting the PoCs. There will be many incidental lessons to be learned about the software and about the way the vendor manages a potential customer.

Concurrent or sequential?

The next issue is whether the PoC is going to be used to select one from two or three applications, or just as a due diligence on the preferred vendor. There is a practical consideration. Running PoCs concurrently takes a very considerable amount of staff time and scheduling and will almost certainly involve two project managers. Setting a date for PoC tests in the RFP is not a solution. It might even result in a no-bid from a potentially attractive vendor because you look like a client that has no sense of reality.  The scheduling also requires coordination with the vendor as they will have staff specifically trained to undertake PoC tests.  Ideally potential systems integrators (which may be different for each of the shortlisted applications!) should be involved in the PoC as there could be important lessons to be learned about the final specification and implementation plan.

The other option is to run the PoCs sequentially. The danger here is that if one of the early PoCs overruns it will impact the schedule for subsequent PoCs. Another common problem is that the first PoC throws up some fundamental issues that the client needs to address (security management is usually on the list) and the vendor then asks if they can run the PoC again as their competitors will benefit from the discoveries and remedies.  

It is likely that Procurement will have imposed a requirement to score the vendor evaluations. Before doing so do read Martin Tate’s excellent book that provides invaluable insight into scoring methodologies.  If you have scored the initial proposals you will then not only have to score the PoC but also decide what weight to give the PoC score against the proposal evaluation score.

In either case there will need to be some detailed considerations of the systems configuration needed to create a PoC environment, and in particular how secure access will be provided to the vendor. 

The other option is only to run PoC tests on the preferred application from the initial evaluation. This in theory reduces the amount of work involved but if the PoC fails then you will have to talk nicely to the next in line and start the process all over again!

OOTB or customized?

Another important issue with search is the extent to which the vendor is expected to undertake customization of the application and/or the user interface ahead of the PoC. Vendors are naturally not very keen on doing this unless there is a payment for the development effort. However OOTB search applications are very basic, as any user of Microsoft O365 will appreciate. It is not worth running a PoC on a base configuration when both you and the vendor know that it is the ability to customize the application that is the reason for considering it. A major issue here is how you will be able to assess the performance of AI applications on a small test set with only a partial index when the sales pitch is all about how well AI is going to identify user intents and personalize results.

Are you still sure you want to do PoC tests?

From this brief summary I hope you are getting a sense of the challenges of scheduling and complexity of PoC assessments. My experience suggests that many of the questions about an application that the search procurement team hope to resolve in a PoC can be addressed in meetings with current clients of the vendor. There may be vendor pushback on this because they are worried that commercial information about (for example) pricing and implementation challenges will be exchanged. I’m talking about serious two-hour meetings around a defined agenda, not a 30 minute chat over Zoom. Long virtual meetings are not a comfortable experience so need to be planned very carefully.

The key point is that a failure to specify requirements clearly in the RFP cannot be remedied by PoCs. The RFP is the corner-stone of a search procurement and it has to be grounded in a search strategy. The more effort you put into the RFP the less the requirement for large-scale PoCs and the greater the chance of staying on schedule, meeting requirements and staying sane.

Martin White