M.S. AAI Capstone Chronicles 2024

A.S.LINGUIST

19

instead, still shows a few inaccuracies. Midway through our project, we in fact realized that our

conversational dataset was too limited in scope, which restricted the variety of answers our

model could generate. A larger dataset would enable us not only to accommodate a broader

range of prompts but also to better evaluate the different responses produced by the model.

Another possibility for a better chatbot model could be employing a different pre-trained model.

In this work, we decided to opt for the Flan-T5- base model, by exploiting the “Parameter

Efficient Fine- Tuning” (PEFT) approach. In the future, we could compare the performance of

our current chatbot model with that obtained with other pre-trained fine-tuned models. In this

way, it could be possible to identify benefits and limitations of each solution in order to choose

the best one.

Another improvement involves making the application less hands-on and speeding up its

response time. Currently, the application requires the user to click buttons to capture a specific

frame, delete the last character in a string, reset the string, add a question mark if necessary, or

interact with the bot. Our future goal is to enable the application to recognize other signals that

can perform these tasks automatically, reducing the need for direct user interaction with the

screen. Our graphical user interface (GUI) currently experiences some lag. Due to the demanding

nature of capturing frames, modifying the string, and querying the bot, the Streamlit API

struggles to handle all these tasks smoothly. A possible solution could be to create a local GUI

using PyQt or Tkinter and deploy it on a separate server, which would help reduce latency

between actions and improve the overall user experience.

Finally, the creation of a virtual avatar to translate the textual answers from the chatbot

model into a live sign language feedback could be an additional improvement area. For our

application, we took the answer given by the chatbot, converted it into a list, and did some fancy

199