AAI_2025_Capstone_Chronicles_Combined

5

students who are struggling with the material, and individualized student learning styles. The tutor is designed to engage students in a Socratic style, using open-ended questions to guide them to a genuine understanding of the material. Built-in safeguards ensure students stay on topic and help the tutor respond appropriately to student emotions like frustration or self-doubt. The current iteration of the application is particularly designed for mathematics students in the sixth through eighth grade range. The application is not expected to act as a replacement for human teachers, but rather as an additional tool for students to explore mathematics in or out of the classroom. In addition to students, course instructors are also considered primary end users for the application, with the course editing tools and insights dashboard offering additional ways to engage with students and gain insight into their learning processes. Live system data from conversation transcripts will eventually be used to create analytics for teachers. For this project, the goal is to enhance the teacher dashboard feature by providing insightful summaries derived from student-chatbot interactions, utilizing the Khan/tutoring-accuracy dataset as our primary data source (Miller & DiCerbo, 2024). Data Summary The dataset utilized in this project was initially curated as a benchmark to evaluate the accuracy of Large Language Models (LLMs) in discerning mathematical correctness from user responses. It comprises tutoring dialogues between human students and Khanmigo, Khan Academy’s GPT-4 powered tutor, with interactions truncated at the point where a student makes a mathematical claim (Miller & DiCerbo, 2024). Our objective is to leverage these dialogues to develop an unsupervised deep embedded clustering model. This model will identify class-wide trends and generate analytics from student conversations, providing teachers with actionable insights to support instructional practices. Features and Data Characteristics This human-curated dataset captures the nuanced, two-way feedback characteristic of tutoring sessions, moving beyond isolated math problems. Its design specifically targets the evaluation of AI model accuracy in tutoring scenarios, particularly in assessing student work and

229