M.S. AAI Capstone Chronicles 2024

A.S.LINGUIST

8

and text outputs (Pardasani et al., 2018), and the type of conversations one can have with the

chatbot varies with the project scope, spanning from providing psychotherapy and psychological

counseling (Huang et al., 2021) to assisting users in retail environments (Berkley School of

Information). In terms of machine learning methods, different approaches can be adopted to

build well functioning sign language interpreters and chatbots, based on the characteristics of

inputs and outputs, the size of the datasets used to build the models, and other constraints like

time and machine capacity.

The project we present leverages supervised learning with neural networks, similar to

other notable projects, including a MIDS Capstone project from UC Berkeley (American Sign

Language Bot, n.d.) and a project developed by GitHub user Sophgdnfor (Sophgdn, n.d.). The

MIDS Capstone project from UC Berkeley focused on assisting ASL speakers in retail

environments using real-time video intelligence to interpret ASL. The team provided an

overview of data processing, model implementation and error analysis in their invention.

SignBot, an initial prototype developed by Sophgdn, uses LUIS, a Microsoft's machine learning

based service, to translate natural language speech into Australian Sign Language. The bot

operates by translating messages into videos that depict how they would be communicated in

sign language. The developers use LUIS to simplify complex sentences, making them

compatible with the Australian Sign Language vocabulary. According to their README file, the

application records a live feed of the user communicating via sign language, processes this feed

using LUIS, and presents interpretation options for the user to choose from.

Our project incorporates similar approaches to both mentioned projects but with some

notable differences. Like the UC Berkeley project, we used a live feed in our application, but

while their primary goal was to focus on ASL in retail, our vision was to create a more

188