M.S. AAI Capstone Chronicles 2024

EDUCATION ASSISTANCE THROUGH A.I.

11

were a few cases where the model did use the fetched documents from ChromaDB to answer the queries precisely and accurately, though it was more common that the LLM did not seem to give much weight to the context. In some of the later testing, we found that the model would still answer questions, despite those questions being poisoned within the RAG. When testing the alternative, where the answer was fully revealed within the context, the model would sometimes answer independently. We theorized that a larger model might give more attention to the context and be able to adhere to the desired constraints more closely. During testing, there were moments where the model rambled until it reached the maximum output tokens. This was usually seen when the model was thrown off due to bad context data or a poor query. We were able to get much better results by doing the following: using Llama-2-13-b-chat, reducing the chunk (document) size for each document in the vector store, fetching only the most similar document to the query, further tuning the assistant prompt, rewording data poisoning files. The combination of these efforts resulted in a model that was effective in concealing homework answers when appropriate, and presenting the answers when passed to it.

84