AAI_2025_Capstone_Chronicles_Combined

1 Automated Triage of Disaster Communications: Leveraging NLP for Real-Time Emergency Message Categorization

Gurleen Virk Master of Science in Applied Artificial Intelligence Shiley Marcos School of Engineering / University of San Diego​ gvirk@sandiego.edu

Victor Hsu Master of Science in Applied Artificial Intelligence Shiley Marcos School of Engineering / University of San Diego​ email@sandiego.edu

Abstract Rapid response is critical during natural disasters, yet emergency teams are often overwhelmed by the sheer volume of incoming communications, creating a bottleneck that delays life-saving interventions. This project developed an automated multi-label classification system to categorize emergency messages across 36 distinct need categories using the Figure Eight Disaster Response Messages dataset containing over 26,000 real-world communications from major disasters including the 2010 Haiti earthquake and Hurricane Sandy. We evaluated three machine learning architectures: XGBoost, Classifier Chains, and Deep Neural Networks, ultimately selecting XGBoost for systematic optimization. The final model achieved a micro-averaged F1-score of 0.677 on the test set, with macro F1=0.518, demonstrating strong performance on basic needs categories (food F1=0.84, water F1=0.81, shelter F1=0.74) and successfully exceeding the F1 > 0.80 target for food and water. Although extreme data scarcity limited performance on the rarest categories, per-label

threshold optimization delivered 9% performance improvements, successfully prioritizing recall for critical emergencies. The system is positioned as a viable filtering tool for hybrid human-AI disaster response teams, enabling automated triage of routine communications while flagging life-critical cases for mandatory human review. KEYWORDS Disaster Response, Multi-Label Classification, Natural Language Processing, XGBoost, Emergency Management 1 Introduction Natural disasters such as earthquakes, hurricanes, and wildfires generate urgent humanitarian crises that require rapid, coordinated response efforts. During these critical moments, disaster response organizations receive thousands of emergency messages per hour through various channels including social media, text messages, emergency hotlines, and news reports (Imran et al., 2015). Each message may contain requests for multiple types of assistance, such as medical help, water,

317