M.S. AAI Capstone Chronicles 2024

EDUCATION ASSISTANCE THROUGH A.I.

14

they asked for it. Some students, about 7%, were frustrated that it couldn’t go into greater detail. Often that was because the token limit would stop the model if the response was too long. This was done to minimize impact on the response time for all users in the scale implementation. However the supplemental data was only intended to sample a single course, so there were also limitations on what it knew. More advanced users, roughly 22%, that already knew data science and the abilities of other chatbots were frustrated by the lack of code generation. The bench version had some limited code generation ability but due to the data filtering and token capping on the deployed version, we were unable to provide code generation services. Overall this was a successful test. During this project we fine-tuned various models in an attempt to increase the ability of the model. For the majority of them, we used LoRA(Hu et al., 2021) and PEFT(Mangrulkar et al., 2022) with 4-bit loading which allowed us to train Llama-7B with limited resources. Some of our models attempted to use SQuAD(Rajpurkar et al., 2016) but we ended up getting better results after we were able to produce our own custom dataset which was based on the source material we were given. In future revisions, we plan on incorporating more open source materials using data mining techniques. In summary, we can say that as a proof of concept for an A.I. T.A. it looks to be a valid technology. With an approval rate from students at 63% without a highly refined implementation, we can see that not only is this a useful technology but also a desired technology. We have noticed an increase in the ability of our A.I. model through the usage of fine-tuning and RAG data stores, but it is critical that the data is formatted correctly for proper usage. Our RAG store really only gained traction once it was formatted to the same standard as the OpenAssistant conversations data(Köpf et al., 2023). Future Enhancements We have some possible strategies for making the model adhere to the context provided by the vector store even more closely. These include prompt engineering and adjusting inference parameters (e.g., top_p, temperature, repetition penalty). With

87