SITUATION​

Our client sought to elevate the quality of responses in their internal chatbot for Technology Acceptance Model (TAM) and general roles. They aimed to refine the structure of documents in the knowledge base of their Retrieval-Augmented Generation (RAG) system as part of their Learning and Skills Development Inhouse Training Platform. The existing system was not meeting the desired accuracy and relevancy standards, which impacted the overall user experience. The company needed a robust solution to streamline their knowledge base and improve the performance of their chatbot.​

Everforth Apex stepped in with a comprehensive approach. We meticulously refined the knowledge base documents, which served as the underpinning files the chatbot drew on. We also evaluated multiple types of Large Language Models (LLM), and provided additional related support for the project, such as model training, LLM pipeline development, and product ownership. Our solution involved properly segmenting knowledge base documents, refining the RAG-based system to accommodate document styles, and clearly defining evaluation criteria for responses based on team requirements. This holistic approach ensured that the system was optimized for better performance and user satisfaction.​

RESULT​

The impact was transformative. Everforth Apex streamlined the RAG Architecture into a centralized structure supporting over 600 internal employees. We tested and evaluated six different LLMs, selected the highest performing model to create a solution that significantly boosted the accuracy and relevancy of chatbot responses from 49% to 84%. Additionally, the knowledge base was refined into 24 clearly defined documents, enhancing the evaluation criteria of responses and overall performance of the RAG Agent. This project not only improved the chatbot's efficiency but also contributed to a better user experience and higher satisfaction rates among employees.