M.S. Applied Data Science - Capstone Chronicles 2025

38

6.2 Conclusion This study aimed to evaluate the performance of several classification models, including random forest, XGBoost, logistic regression, decision tree, and MLPClassifier, across three target classes using key performance metrics such as precision, recall, and F1-score. The comprehensive evaluation of model performance across three distinct classes revealed significant patterns in classification effectiveness and algorithm suitability. Random forest consistently emerged as the superior model across all three classes, demonstrating exceptional discriminatory power with metrics of approximately 0.97 for Class I, 0.98 for Class II, and 0.75 for the challenging Class III instances. This consistent performance advantage highlights random forest’s robust capacity to handle complex, multiclass classification problems through its ensemble approach of diverse decision trees. The model’s balanced precision-recall relationship across classes further emphasizes its reliability for deployment in production environments where consistent performance across all classes is essential. Class complexity notably influenced model performance, with all algorithms achieving their highest metrics for Class II, moderate performance for Class I, and significantly diminished effectiveness for Class III. This consistent pattern suggests inherent differences in the separability of features across classes, with Class III likely exhibiting greater overlap with other classes or higher within-class variability. The pronounced decline in performance metrics for Class III across all models indicates a fundamental classification challenge that persists regardless of algorithm selection, though tree-based approaches demonstrated greater

resilience to this challenge. random forest’s ability to maintain F1-scores of approximately 0.70 for Class III, compared to significantly lower values for other models, underscores its superior adaptability to challenging classification scenarios. While tree-based methods demonstrated superior performance overall, the analysis revealed interesting algorithm-specific patterns across classes. Decision tree maintained competitive performance with random forest for Classes I and II but showed a more substantial performance decline for Class III. MLP performed comparably to XGBoost across most metrics and classes, suggesting that both neural network approaches and gradient boosting techniques offer similar capabilities for this particular classification task. Logistic regression consistently underperformed across all classes, with particularly poor precision of approximately 0.20 for Class III compared to random forest’s precision of approximately 0.75. These findings provide valuable guidance for algorithm selection in similar classification contexts, emphasizing the advantages of ensemble methods, particularly random forest, for robust multiclass classification. The use of OOB evaluation in the random forest model provided an internal estimate of generalization performance without the need for additional cross-validation. This approach helped validate the model’s robustness and efficiency while reducing the risk of overfitting. The OOB score served as a reliable metric for comparing model performance during the tuning process. Additionally, the feature importance analysis revealed which variables most significantly influenced the final model’s predictions. Variables such as reason_word_count and

42