M.S. AAI Capstone Chronicles 2024

EDUCATION ASSISTANCE THROUGH A.I.

10

tagged with metadata that indicated such so that it would always be accessible. For our front end, we used Gradio which presented a visually appealing UI with very low effort on our part (Gradio, n.d.). Results As stated in our background information, our main goals are to assist students in a highly accessible manner as well as grading assignments. For this capstone, we were unable to complete the assignment grading model. However, we did complete multiple revisions of our A.I. on-demand teaching assistant. We gathered data using bench tests done internally and then we released a small-scale model to various students with different backgrounds for testing. A large problem with testing a model like this is how subjective some of the metrics can be. Similarity score, perplexity, and training loss all can be taken into account but this doesn’t represent how the model will overall perform on helping a student when they ask a question. We also tried various other techniques that did not come to fruition in time but could be enhancements to the project later. Our first bench tests involved testing the base model, the model with a RAG vector store, and the fine-tuned model with the modular RAG vector store. This was intended to be a markup of how it would be fully implemented in a full-scale session-based model for students.

Figure 1: A sample screenshot of the bench testing running on Google Colab

With the modular RAG system on the base model, Llama-2-7b-chat does not seem to behave as intended. The model responses on their own are generally adequate, but how it used the context that was fetched from the vector store left much to be desired. There

83