The Search Interfaces: From Text Boxes to Conversational AI

Conversational AI is an exciting revolution in artificial intelligence. It is powered by large language models (LLM) tools and can answer questions by summarizing information from the Internet. In this article, we’ll be sharing our thoughts on conversational search based on three main points:

• What is it, and why do we need it?
• How does it work?
• What are its limitations?

Now, let’s get into it!

What is conversational AI, and why do we need it?

Would you want to answer questions based on a knowledge base that you control? Do you have the documents that answer questions in your domain? Is it impossible to specify all your questions so you can answer them from your knowledge base? If you answered “yes” to all these questions, Conversational Search might be for you.

Conversational Search empowers users with the ability to pose arbitrary questions and receive answers from their own documents. It leverages your documents to reveal relevant content for a question and employs large language models to generate text from that content. The question and the generated answer may not be explicitly present in your knowledge base, but the answer will always be rooted in your knowledge base. This user-centric approach, where users can ask conversational questions and receive answers grounded in a knowledge-based search, is what we call Conversational Search.

This approach is known as “Retrieval Augmented Generation.” The system will conduct the following: 

1. Retrieve context from a search query. 
2. Answers the question by augmenting the search results with generated text.

LLMs are known to create unjustified yet confident answers. They are usually “trained on the Internet.” Would you trust an average snippet from the internet? Conversational Search diminishes these hallucinations by retrieving a significant portion of the answer sourced directly from the knowledge base and responding with “I don’t know” if it can’t use the knowledge base. 

Using generative AI for document collection is easier than scripting an FAQ or chatbot over countable questions. It generated answers instead of passages, making it more reliable for users. Finally, it also offers sources for generated answers, making the responses more trustworthy. 

How does it work?

Traditionally, the user flows from “user query” > “knowledge base search” > “return relevant pages.”

In conversational search, the flow is “user query” > “knowledge base search + LLM” > “return answer with evidence.”

We’ve observed how pure “knowledge base search” works. Pure LLM sends a query to an interface like ChatGPT, where it answers your questions based on its knowledge base (training data) – instead of your knowledge base. The magic happens when a combination of “knowledge base search plus LLM” is combined.

Conversational search begins with traditional search but feeds search results into a greater language model. The LLM uses an optimally engineered prompt with context from the knowledge base. This prompt tells the LLM to generate an answer constrained to the knowledge base. The shown prompt is illustrative, and your chat platform can add additional instructions, such as telling LMM when it should say “I don’t know.”

Most of the work of conversational search falls on the chat platform provider. The chat solution builder must need to:

• Offer relevant documents (or a link to the knowledge base).
• Find a connection to the LLM.

On the other hand, the chat platform provider does more of the heavy lifting:

• Hosts and manages the LLM.
• Build the most effective prompt. LLMs have a strict limit on prompt size that leads to tradeoffs. How many search results should be included? And how detailed should the prompt instructions be?
• Tuning the knowledge base API calls with prompt engineering to receive concise and relevant packages.

What are the limitations and variations of conversational AI?

Conversational search is currently in its infant stage. There are plenty of challenges to wide-scale implementation, along with several variations on its basic theme.

1st Challenge: It’s expensive

The large language models behind conversational search are relatively expensive and run on specialized hardware, which requires specific skills to tune. Large AI players host most of the LLMs; we expect most companies to buy instead of build. API calls to LLMS are usually significantly more expensive than most knowledge-based search APIs.

One option is using conversational search as a fallback. Instead, build a traditional Q&A chatbot for the most common questions and use conversational search for any questions that aren’t pre-programmed (the “long tail”). Think of the traditional Q&A as a cache. In this pattern, you can use the LLM to create answers for the most common questions once and store them in the traditional chatbot.

2nd Challenge: Who hosts the model?

As we mentioned, models are expensive. Many model providers reduce the cost by keeping the requests sent to their model for future training data. Are you comfortable sending your questions and database to a third party? We expect businesses to demand options for using LLM that prioritize the protection of confidential data.

3rd challenge: This is limited to Q&A and is not transactional (yet)

Conversational search can generate exceptional answers to almost any question. We’ve seen excellent answers to “How to open an account,” or “How can I reset my password,” and more. However, wouldn’t it be nice if it also helps open the account or reset the user’s password? This is the obvious next step coming from platforms and providers. Companies will seek a solution to integrate with APIs and complete a task.

Conclusion 

Conversational search is exciting as every company and enterprise has a knowledge base, and it is easy to unlock it! We expect that conversational search will be the next baseline for knowledge bases in upcoming years as users will love answers supported by evidence, and knowledge base owners will adore the increased value received from their content.

Ultimately, the rise of conversational search marks a significant shift in how we interact with digital content. By leveraging advanced AI to provide detailed, contextually relevant answers, businesses can offer a superior user experience, drive engagement, and unlock new opportunities for growth and innovation.