M.S. AAI Capstone Chronicles 2024

EDUCATION ASSISTANCE THROUGH A.I.

12

Figure 2: An illustration of the system concealing/revealing homework problem solutions based on the due date

It’s hard to perfectly imagine how an end user will treat the A.I. model until tested, so we created a scale testing model. The scale model was launched three times. Once with the base model, once with the fine-tuned model, and lastly once with the fine-tuned model and a RAG vector store. This was sent out to various discord channels for students at Cal Poly Pomona, and the slack channel for the USD-AAI program. We ended up with well over one thousand responses which exceeded the expected number of tests. In the most refined deployable iteration, we used a fine-tuned version of the model and a RAG vector store which both used the same custom dataset. This model showed great promise in answering based on the context that was retrieved. In order to ensure the model didn’t ramble, stopping criteria were implemented to reduce inference time and excess output. After that basic string enhancements were done to ensure that delimiters didn’t escape into user answers. This implementation of the model had some of the cleanest responses, but it didn’t go into as much detail as the bench-tested model.

Figure 3: A sample of the scale interface linked with a fine-tuned model connected to a RAG vector store

85