AAI_2025_Capstone_Chronicles_Combined

3

news aggregation services to provide continuous, automated message classification. The ultimate goal of this project is to develop a robust multi-label classification model that achieves a micro-averaged F1-score exceeding 0.75 and macro-averaged F1-score exceeding 0.60, with particular emphasis on achieving F1-scores above 0.80 for critical categories including medical help, water, and shelter. Test set evaluation demonstrates that while overall targets were not fully achieved (test micro F1=0.677, macro F1=0.518), the model successfully exceeded the 0.80 threshold for basic needs categories (food=0.84, water=0.81), confirming that modern NLP techniques combined with multi-label classification can accurately identify high-frequency critical needs. The central hypothesis driving this research is that modern NLP techniques, specifically Term Frequency-Inverse Document Frequency (TF-IDF) vectorization or BERT embeddings, combined with multi-label classification algorithms can accurately identify multiple simultaneous need categories within disaster messages, thereby reducing message triage time by 60-80% compared to manual processing. The final product would be a deployable system capable of processing incoming disaster messages in real-time, automatically assigning relevant category labels, prioritizing messages based on urgency (with medical emergencies flagged as highest priority), and routing communications to appropriate response teams, all within a matter of seconds rather than the hours or days required for manual processing. By successfully automating this critical component of disaster response, this system has the potential to significantly accelerate emergency relief operations, improve resource allocation efficiency, reduce responder workload

during peak crisis periods, and ultimately save lives by ensuring that urgent needs are identified and addressed with minimal delay. 2 Data Summary The dataset for this study is a corpus of labeled emergency communications collected during disaster events. It consists of approximately 26,200 messages, pre-split into a training set of about 21,000 messages, a validation set of 2,600, and a test set of 2,600 messages. Each observation includes the message content and its associated classification labels. The dataset contains 42 distinct variables, categorized into two functional groups: identification and source variables, and classification labels. Identification and source variables provide context for the communication, including a unique message ID, its split designation (train/validation/test), the primary message text (in English), the original text (if a translation was required), and the genre (e.g., "direct," "social," or "news"). The core of the dataset comprises 36 binary classification labels. Each label indicates the presence (1) or absence (0) of a specific disaster-related need or situation. These labels are grouped into several thematic categories to organize the project's multi-label classification goal: general categorization, critical resource needs, emergency services, infrastructure and services, social and demographic categories, and disaster type indicators. Exploratory data analysis (EDA) revealed several data quality issues that required specific preprocessing steps to prepare the data for modeling. These issues primarily involved

319