Even before the arrival of GPT-3/ChatBot and now GPT-4 there were many companies offering enterprise search solutions that claim to solve all the known challenges of enterprise search. I am sure that more will join them. Deciding on whether these solutions meet your functional and non-functional requirements is going to be increasingly difficult when so much of the marketing material on the websites of vendors suggests that they offer the ultimate solution so just send the cheque.
In this series of posts I will be highlighting some of the areas that you need to ensure you have complete transparency about the capabilities of the software and the vendor. I’m not suggesting that the applications on offer will not be able to meet the rather unusual requirements of an enterprise search application. Just remember the old adage of caveat emptor aka let the buyer beware.
Enterprise search is unique in the level of granularity that needs to be managed to ensure that employees can indeed see all the information that the enterprise holds – but in practice it is all of just the information they are allowed to see. That immediately brings up the question of who in the organisation sets the policies, issues the guidelines and governs the implementation of information confidentiality. I have had many discussions over the years with managers who assert that their information confidentiality is managed under ISO 27001-2013 until I ask to see the documentation for this process. This is not a feature of the standard.
There are two underlying approaches which are widely used in many IT applications. The first of these is early binding, where the tagging on the record only permits the employee to search through the information they have permission to look at. The second is late binding, with the employee searching the complete index but then the application only delivers approved content to the user interface. One of the downsides is the potential latency in delivering all the permissible results. It is also possible to have a hybrid (sometimes referred to as dispid) approach. Just how the ‘tag’ is created and managed by the back-end is quite an interesting challenge in its own right.
Where things get interesting is where the security needs to be applied at a sectional level in a document, so that (for example) employees can read a summary but do not have access to further information that could be in subsequent sections of the document.
Applying security tags to internally generated content can be managed (at least in principle) through whatever CMS/EDMS you are using. Managing third-party content that has come into (or for that matter across!) the organisation through an email or social media raises all sorts of issues, largely depending on where the employee will store the information. Perhaps in their own DropBox account?
So far so good. But you will be migrating content from your current search application to the new application. The question that you need to ask is how the migration of the security tagging is going to be managed, and what tests will you be able to run to ensure there are no black holes in the application. When I start work with a client on their enterprise search application I always search for [confidential] and make sure I am looking at their faces when the results appear given that I only have guest access! Will the incoming vendor warrant under the terms of the contract that they will be liable for any breach of confidentiality caused by a blip in the tagging migration? And as a follow up, what support will they provide when you acquire a new business entity that has a totally different information security model.
In short, the requirement for secure search is so important that if it is not addressed in the initial marketing pitch of the vendor you should ask yourself, and then the vendor, why they seen to have no transparency on this topic. I might add that this is in fact not just an enterprise issue – many B2B e-commerce companies tag products so that employees can only purchase from a limited range of products that have been selected for them by the purchasing team.
See here for Part 2
Martin White, Principal Analyst 16 March 2023