AAI_2025_Capstone_Chronicles_Combined

4

Draw, Detect, Navigate ​

​

​

​

​

​

​

​

​

Introduction

The recognition of hand-drawn symbolic representations plays a pivotal role in various

domains such as data capture, note taking, architecture and design planning, document

markup, wargaming, and strategy simulation. Translating abstract, low fidelity pictograms into

digital elements is typically done manually or through static classification models. These

approaches lack the spatial awareness or interactivity needed for dynamic computer vision

applications. The project addresses this gap by developing a system capable of detecting,

localizing, and classifying multiple hand-drawn symbols in real time.

This project builds on the success of classification models like Sketchnet (Zhang et al.,

2016) and transformer based architectures that use vectorized stroke data (Xu et al., 2022).

While these models perform well in identifying individual symbols, bounding boxes for the

symbol within the image are not typically used. They do not analyze spatial relationships

between symbols either. Our hypothesis is that a lightweight object detection model, supported

with synthetically generated data, can provide real-time sketch recognition with spatial

awareness on readily available devices.

The intended users of this system are individuals or teams who would benefit from

quick, visual input. These can include emergency responders mapping navigation paths.

Additionally, new opportunities for interactive applications in augmented reality (AR) can

include AI-enabled pathfinding, and adversarial simulations. The output of the model is meant

to support quick decision making by interpreting doodles as functional elements in a digital

environment.

This project used a subset of “doodles” from the publicly available Quick, Draw! dataset

(Jongejan et al.), which contains millions of user-submitted doodles labeled by class. From this

subset, we created synthetic data by combining multiple randomly placed doodles with

31