AAI_2025_Capstone_Chronicles_Combined

2

Draw, Detect, Navigate ​

​

​

​

​

​

​

​

​

Abstract

This capstone project extends traditional sketch classification by incorporating real-time object

detection and augmented reality (AR) integration. While convolutional neural networks (CNNs)

have demonstrated high accuracy in identifying hand-drawn pictograms, most approaches omit

spatial localization and real-time interaction. Our system addresses these gaps by using

bounding box predictions and edge-optimized models capable of identifying multiple doodles

in real time within an AR environment. To overcome limitations in existing datasets, we

developed a synthetic data generation pipeline using Unity and Python, producing randomized,

annotated images that mirror real-world drawing variability. We trained and evaluated both

Faster R-CNN and YOLOv8 variants, ultimately selecting the YOLOv8 nano model for deployment

due to its speed, size, and high accuracy. The model achieved an F1 score of 0.96 and

processed images at 28 frames per second (FPS), enabling seamless AR integration. The final

application uses a webcam feed and ArUco marker tracking to detect hand-drawn symbols,

anchor 3D models to the drawings, and compute navigation routes using the A* algorithm. The

system combines symbolic vision with spatial reasoning to support interactive use cases such

as route planning, adversarial simulations, and strategic modeling. The model’s real-time

performance on new inputs, along with its successful deployment through Unity, supports its

use in live scenarios. These results show that sketch-based simulations can support quick

decision making in settings where fast, visual input is needed.

29