AAI_2025_Capstone_Chronicles_Combined

‭ResolveAI‬

‭(RAG) system was built with LangChain to feed into our LLM for the chatbot. To keep low cost‬ ‭and quick inference we used ChatGPT version 3.5 turbo for the LLM to interact with the user.‬ ‭The full flow of the chatbot included collecting the query from the user, pulling the most relevant‬ ‭examples from our database using RAG, then prompting the LLM with both the user query and‬ ‭relevant examples. Finally the response from the LLM was provided back to the user to complete‬ ‭the chatbot interface.‬

‭Results‬

‭Model Effectiveness‬

‭During multi-class training (low, medium, high priority),‬‭the model’s training accuracy‬ ‭steadily increased while training loss sharply declined, suggesting that it was successfully‬ ‭learning the nuances of the data. However, the validation accuracy plateaued and remained‬ ‭noticeably lower, and the validation loss trended upward. These signs indicate the model‬ ‭struggled to generalize beyond the training set. Initially we attempted to improve regularization‬ ‭of the model by using both aggressive dropout and reducing model size to limit variance and‬ ‭overfitting. However the model did not improve significantly with these measures. The poor‬ ‭performance was likely due to data imbalance or insufficient representation of medium or high‬ ‭tickets, prompting a shift toward binary classification to improve performance. This could also‬ ‭be due to the fact that we’re working with the upper limit of the available data and perhaps the‬ ‭“priority” classes aren’t as linearly separable in this dataset as we had hoped. In a production‬ ‭environment, there would be a more refined or expanded dataset to address the potential‬ ‭separability issue and improve overall performance.‬

‭19‬

67