AAI_2025_Capstone_Chronicles_Combined

5

Draw, Detect, Navigate ​

​

​

​

​

​

​

​

​

bounding box annotations. When the application is deployed, live input comes from a webcam

or USB connected camera.

The final product is an AR application that uses a YOLO model to detect doodles, place

3D representations on each symbol, and compute a navigation path using the A* algorithm.

Using a sheet of paper printed with OpenCV’s augmented reality university of cordova (ArUco)

marker, the user draws doodles of different objects, and the application will calculate the best

path for the starting doodle to reach the goal doodle (OpenCV: Detection of ArUco Markers,

n.d.). For example, a helicopter doodle routes to a hospital while avoiding obstacles, which are

the other remaining classes in our dataset.

Background Information

The original drawing data used in this project comes from user data from the website

game Quick, Draw! where users made small rapid drawings based on a given specific prompt

within 20 seconds for the purpose of training a classification model with labeled data. While

Google Creative Labs has not disclosed the model architecture of the classifier currently live on

the site, they do link to Tensorflow guidance on the construction of a recurrent neural network

for this purpose (Tensorflow, 2024). Niu et al. had robots use hand drawn maps and object

recognition to navigate unfamiliar environments (Niu et al., 2019). While these and other

projects have demonstrated high levels of accuracy in classification, bounding boxes are not

used, real-time continuous performance is not prioritized, the live surface of the drawing is not

incorporated, and in most implementations, the classification is the end goal itself, not a tool

that feeds into subsequent tasks.

While prior pictogram classification work and similar classification projects primarily

focus on image recognition, our project incorporates multiple layers of complexity by

combining CNN-based object classification with bounding box identification and dynamic path

optimization using A*. The A* algorithm is a graph traversal and path search algorithm that

estimates the most efficient path by using a heuristic function to guide the search toward the

32