M.S. AAI Capstone Chronicles 2024

EDUCATION ASSISTANCE THROUGH A.I.

5

There are A.I. tools now that can automatically assess code and even attempt to grade code-based assignments. A paper by Caiza details some of these technologies as well as their flaws (Caiza & Del Alamo, 2013). The main flaws are that they end up being rather limited in terms of languages accepted, they don’t adequately support teaching infrastructure, and they have a lack of normalized evaluation criteria. Continuing the topic of testing software-based assignments, recently a company known as Cognition has developed what they call the first A.I. Engineer (Wu, 2024). This is relevant for a feature that they report having called software unit testing. If this feature functions as intended, it could be used to grade student coding assignments. Simply have the A.I. run a unit test review and look at the code. This would allow for a basis of grading depending on how the feature works. Revisiting the application of ChatGPT, this A.I. shows promise in grading assignments in both code and essay formats. A paper by Jukiewicz shows there is a strong correlation between grades assigned by the teacher and grades given by ChatGPT (Jukiewicz, 2023). This, however, does have flaws as Jukiewicz outlined. Namely the model can sometimes be inconsistent, as well as experience hallucinations.

Figure 1: A comparison between a Human grader and ChatGPT grades (Jukiewicz, 2023)

Given these general limitations, we believe the best usage for the grading technology

78