M.S. AAI Capstone Chronicles 2024

A.S.LINGUIST

18

frame, which is then processed by our CNN model to predict the associated ASL sign. Next, we

included buttons to manipulate the string generated by the CNN model's predictions. These

buttons allow users to reset the string, remove the last character, or add a question mark. Finally,

our application enables users to chat with the bot. Once the user is satisfied with the constructed

string, they can click the ‘Ask Bot’ button to input the string into our natural language model,

which generates a response. Alongside displaying the text generated by the model, the

application also offers a feature that visualizes the text using images (Figure 8).

Figure 8

Simulating a real-life avatar using images

Further Improvements

Given the chance to continue working on this project, we will probably work on the

chatbot model. As highlighted above, the CNN model already reached an optimal performance

and, at the moment, does not seem to require additional improvements. The chatbot model,

198