M.S. AAI Capstone Chronicles 2024

EDUCATION ASSISTANCE THROUGH A.I.

9

Many attempts to enhance the performance of our chosen model were made. The RAG system helped the model provide more helpful results by supplying the model with chunks of text relevant to user queries. Fine-tuning the model also helped substantially with the performance. This fine-tuning was done by a Quantized Low-Rank Adaptation (QLoRA) adapter (Dettmers et al., 2023a). This technique was used to conserve GPU memory. Depending on the setting, fine-tuning such a large model would require more GPU than an A100 card has (“A100 GPU’s offer power, performance, and efficient scalability,” n.d.). Using a batch size of 1 with a gradient accumulation of 4 and half-precision floats, we were able to reduce the required memory size to 14GB, which is enough to train on a T4 or RTX 4080. For chunking the data, we used LangChain’s Recursive Text Splitter (LangChain, n.d.). We experimented with using different embedding models, but we settled on all-MiniLM-L6-v2 (Wang et al., 2020) as it had a big enough corpus to embed all our data, whereas some other models we experimented with crashed when fed some uncommon characters. This model is used by default by ChromaDB. The system needed to take the date into account for concealing assignment solutions before the assignment due date. The team experimented with multiple ways of achieving this functionality and ended up using metadata tagging and filtering with ChromaDB. Assignment solutions were spread into separate files that had the week number in the file name. We extract the numbers to use to tag the data correctly when ingesting into the vector store. This is how the system achieved data poisoning. The first-week data poisoning file contained the questions for each assignment in the course. Each question was followed up with a message explaining that the answers cannot be revealed yet, thus concealing homework solutions for upcoming assignments. The second week contained answers for the first week’s assignment and data poisoning for the rest of the weeks. The third week contained the answers for weeks 1 and 2 and the concealment of the rest of the assignment solutions and so on. The content that was not time-sensitive, such as the textbook, was

82