M.S. AAI Capstone Chronicles 2024

A.S.LINGUIST

16

chatbot answers and make the entire project impossible to be carried out. Thus, considering our

project goal, the CNN model performance we obtained is absolutely desirable and does not

suggest the necessity to implement further model modifications.

Fine-Tuned Flan-T5-Base Chatbot Model

The learning task was properly learned by the chatbot model, as resulting from the

behavior of the training and validation loss, which both converge to zero after about 90 steps

(Figure 7).

Figure 7

Representation of training (blue) and validation (red) loss for the Flan-T5-Base chatbot model

The performance of the chatbot model was qualitatively evaluated through an initial list

of 20 questions on different topics, which should be further expanded in the future for a more

rigorous evaluation. The answers provided by the model to each questionnaire are reported in

Table 1. As one can see from the latter, model answers are relevant and consistent with

corresponding questions, although they are not always right and could be more precise. For

196