M.S. AAI Capstone Chronicles 2024

EDUCATION ASSISTANCE THROUGH A.I.

8

where the team had to examine the pages in the documents for bad formatting. It was thought to be best to remove such content as to not feed noise to the LLM in a downstream task when fetching documents from our vector store. In several cases, important and badly formatted data were manually typed in text files and used in subsequent tasks. The second dataset we are attempting to work with is called the Stanford Question Answering Dataset v1.0 (SQuAD). It consists of questions posed by crowd workers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage. SQuAD contains 107,785 question-answer pairs on 536 articles, making it almost two orders of magnitude larger than previous manually labeled RC datasets such as MCTest according to Rajpurkar et al. (2016). Finally, our third dataset is in actuality a derivative of the first data source. Given the somewhat underwhelming performance of using only a RAG vector store with the first dataset, and fine-tuning with the second we could see that there were problems. As an experiment, we are taking the initial dataset and attempting to enhance its usability by developing a derivative dataset using OpenAI services. Leveraging a back-end API script, we comprehensively sorted through the original dataset, creating coherent questions and answers with chunked information from the source material. This process enabled us to extract and structure the information in a way that enhances the dataset’s functionality, making it more suitable for various machine-learning tasks. The resulting dataset will hopefully serve as a valuable resource for advancing research in data science-related reading comprehension and question-answering systems. Experimental Methods The team experimented with a few LLMs. The goal is to find the smallest model that would give satisfactory performance, which as mentioned, is to provide theoretical and technical help to students in the class material. The team experimented with several LLMs. It was decided unanimously that Llama 2 was the best offering in terms of both size and performance.

81