M.S. Applied Data Science - Capstone Chronicles 2025

32

identifying the distinctive characteristics of this class. Figure 16 Heatmap of All Metrics Across All Classes and Models During Validation

Note. Performance metrics for five machine learning models across three classes show random forest achieving highest overall scores while logistic regression demonstrates poorest performance, particularly for Class III.

The evaluation of model effectiveness for Class II instances revealed exceptional classification capabilities across all evaluated metrics for most algorithms. Tree-based models demonstrated particularly strong performance in this category, with random forest achieving near-perfect scores across precision, recall, and F1 metrics (all approximately 0.98). Figure 16 illustrates the pronounced disparity between logistic regression and other models, with logistic regression showing substantially lower recall (approximately 0.80) compared to its precision (approximately 0.95), indicating a tendency toward false negative classifications for Class II instances. This imbalance between precision and recall metrics for logistic regression suggests that while the model can reliably identify some Class II instances, it fails to recognize a significant portion of them. The neural network approach

MLP performed competitively with tree-based methods for this class, suggesting that Class II features may be effectively captured through both hierarchical decision boundaries and complex transformational approaches. The analysis of model performance for Class III revealed substantial variability in classification effectiveness, with notably lower metric values compared to other classes across all algorithms. This pattern suggests that Class III instances present the most challenging classification task among the three classes. Figure 16 demonstrates that random forest maintained its position as the top-performing model, though with diminished effectiveness (metrics ranging from approximately 0.65-0.75) compared to its performance on other classes. Interestingly, contrary to the pattern observed in Class I and II,

36