Integrating AI into an existing SaaS

Our objective was to integrate an AI assistant into an existing SaaS product, giving it permission-based access to internal documents so it could answer questions using the knowledge base. However, much of the user experience depended on AI behaviour that we could not fully predict.

We had neither the time nor access to the client's external users. However, the company used its own platform internally in the same way its clients did. While defining the exact requirements with stakeholders, we also discussed their pain points. I pushed to speak with internal users and conducted six interviews with people in key roles. Stakeholders saw the AI chat as another way to search internal documents, and the interviews confirmed that finding specific documents was both frustrating and necessary several times a day.

An AI assistant is only as good as its answers. Left to its own devices, the model rambled, drifted in tone, and would confidently offer answers it never found in the knowledge base. Before the interface could earn anyone's trust, the AI's behaviour had to become predictable enough to design around.

In a typical project, I define every interaction, from where users click to the output they see. With an AI chatbot, however, much of the user experience depends on how the AI communicates. Whether users continue using an AI feature largely depends on how the conversation feels, yet generative output is difficult to control by nature.

The project arrived at an opportune time. Having recently completed the IBM AI Developer course, I saw a chance to address this part of the user experience too. While I waited for stakeholder feedback and the developers familiarized themselves with the system, I built a Python evaluation system to improve the AI's responses.

The script uses LLM-as-a-judge and few-shot prompting. It asks the assistant model (Llama 3) questions from a database in sequence and provides semantically relevant articles it can use to craft each response. A separate judge model (Claude) then compares each answer with the question and source articles, scoring it against an evaluation framework. The assistant is rated on conciseness, factuality, professionalism, and retrieval accuracy, for a maximum of 40 points.

After the initial round, the five best and worst responses are stored as examples and passed to the assistant in subsequent loops. Once all 20 questions have been answered, the script checks whether the average score is at least 35. If not, the judge model updates the assistant's prompt and starts a new evaluation round.

One evaluation loop using my local LLM took about 15 minutes. After eight loops, the system reached the score threshold. The improvement was clear even without comparing the scores: responses became concise rather than rambling, the tone stayed consistently professional, and the assistant usually acknowledged when the database did not contain an answer. It occasionally hallucinated even after admitting that it could not find supporting information, but because the final implementation would use a hosted AI service, this model-level issue could not be fully mitigated. I handed the code to the developers, who integrated it into the broader system with relatively minor changes.

I explored how products named and surfaced their assistants, how they preserved conversation history, and how they structured the chat interface. I found that:

I explored a variety of options for both accessing the chat and presenting the interface. I then created prototypes and designed a simple internal test to gather feedback from people who used the tool every day. Together, the test results and stakeholder input helped me determine the right direction. We chose a top-right access button and a side panel because it allowed users to continue interacting with the page without making the chat feel too constrained.

The side panel was the most popular approach. However, each version included something users liked. I combined the strongest elements from each one in the final polished design.

The project was kept under wraps internally and revealed at the company Christmas party. Management wanted to end the year with one final surprise. What began as a 10-minute presentation became a 40-minute discussion as the audience asked question after question.

During the presentation, we demonstrated the AI's ability to find relevant articles quickly, answer questions accurately, provide links, and respond in any language.