I’ve not blogged for several weeks, a combination of completing the text of my forthcoming book on workarounds and watching for the inevitable arrival of multiple AI-Generated Content (AIGC) applications. There was no way that OpenAI was going to dominate the market. Now we are back into a situation I am very familiar with in working with clients to select an enterprise search application from a short short list of perhaps six candidates. Today I took part in a very lively IntraTeam member meeting chaired by Kurt Kragh Sorensen, with Frank Giroux (Bayer) providing an enterprise perspective based on his experience in chatbot implementation for his company. My contribution was specifically about the intersections between AIGC search applications and current technology applications. If you want to get a sense of the scale of release of AIGC models take a look at the HELMS database. As for a list of APIs – that’s a very different scale of problem but the listing at GitHub is at least a start.
I developed a set of questions for the IntraTeam meeting related to the potential requirement to select an AIGC-based search application in parallel with, or to replace, an enterprise search application. This is my initial list of 10 and I’m sure I will be adding to it over the next few months.
Q1 Who is going to own AIGC applications in the enterprise and lead the process of writing the specification, deciding on the selection criteria and negotiating the contract?
Q2 An AIGC application are going to cost money, not just for a license fee but for computer resources, the investment in the core implementation team and the support of cross-enterprise training. Will that budget come from an increase in IT spend or will it require other investments to be put on hold.
Q3 How much detail do you have about your current content scale and usage? In the enterprise the decay on content relevance can be significant and the last thing you want is out-of-date information being used to train your particular AIGC.
Q4 How large is your content collection compared to the LLM you are working with? Will it be large enough that the LLM does not completely, or even partially, overwhelm your organisation’s content
Q5 How will content confidentiality be maintained, and what does the AIGC vendor offer that would mimic the traditional early, late and hybrid binding approaches that are currently in use.
Q6 How compliant will the AIGC be to GDPR conformance, especially when dealing with the transfer of personal information to a third country, notably the USA?
Q7 How are you going to benchmark the performance of the AIGCs you are considering at the selection stage, or will you take the easy way out and use the AIGC offered by your primary cloud provider to ease the contract negotiation process?
Q8 Do you have a strong information governance framework under which you can establish policies on the attribution of AIGC content and guidance on good practice?
Q9 Have you discussed with your risk management team the potential implications of using AIGC content within the risk appetite of your organisation and any indemnity insurance you have regarding issues over contract terms, abuse of copyright and disclosure of trade secrets?
Q10 How will you be presenting the business case to the Board? If it based on productivity will you aim to balance the books by committing to higher outputs or by reducing head count?
Below is a list of some recent papers (all open access pre-prints) which provide a research perspective on some of these issues.
- Evaluating Verifiability in Generative Search Engines https://arxiv.org/abs/2304.09848
- Can ChatGPT-like Generative Models Guarantee Factual Accuracy? On the Mistakes of New Generation Search Engines https://arxiv.org/abs/2304.11076
- Why Does ChatGPT Fall Short in Answering Questions Faithfully? https://arxiv.org/abs/2304.10513
- Perspectives on Large Language Models for Relevance Judgment https://arxiv.org/abs/2304.09161
- Just Tell Me: Prompt Engineering in Business Process Management https://arxiv.org/abs/2304.07183
- Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understanding https://arxiv.org/abs/2304.05368
- The ROOTS Search Tool: Data Transparency for LLMs https://arxiv.org/abs/2302.14035
- Using GPT for Market Research https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4395751
Martin White
24 April 2023