ADS Capstone Chronicles Revised

20

The F-1 score combines precision and recall into a single metric, allowing for a balanced view in terms of minimizing false positives and identifying positive instances. The F-1 score is crucial for providing a nuanced understanding of the model (Bruce & Bruce, 2017). Additionally, MCC was included as it considers all four confusion matrix categories: true positives, true negatives, false positives, and false negatives. The metric provides a balanced measure of the model’s performance by accounting for the ratio of correctly and incorrectly identified instances ( Matthews_Corrcoef , n.d.). These metrics provided insights into how well each model handled the complexities of the data, particularly in scenarios where false positives and false negatives have significant consequences. The key findings for each model are summarized below: Logistic Regression : As a baseline model, it achieved a validation accuracy of 0.76 and a test accuracy of 0.74. The confusion matrix revealed challenges in identifying fraud instances.

accuracies of 0.82 and 0.83, respectively. The ROC-AUC of 0.87 demonstrated its ability to distinguish classes well. Its F1-score of 0.61 reflected balanced precision and recall. XGBoost : Excelled with validation and test accuracies of 0.87. High ROC-AUC (0.95) and PR-AUC (0.88) scores underscored its effectiveness in handling class imbalance. Its F1-score of 0.78 indicated robust overall performance. Neural network : Achieved validation and test accuracies of 0.89. High ROC-AUC (0.97) and PR-AUC (0.91) scores, along with an F1-score of 0.81. The model demonstrated its capability to capture complex patterns. Quadratic Discriminant Analysis: Had validation and test accuracies of 0.80. The ROC-AUC of 0.89 and PR-AUC of 0.71 reflected moderate performance, with an MCC of 0.40 and F1-score of 0.41 highlighting its probabilistic approach. Bagging Classifier: Demonstrated the highest validation accuracy of 0.91 and test accuracy of 0.90. Its ROC-AUC of 0.97 and F1-score of 0.83 showed exceptional performance in distinguishing classes and balancing precision and recall.

Ridge classifier : Improved accuracy with a validation score of 0.82 and test accuracy of 0.83. The ROC-AUC score of 0.87 indicated strong class discrimination. Its F1-score of 0.61 showed a good balance between precision and recall. Lasso classifier : Also served as a baseline with validation and test

Stochastic Gradient Descent: Achieved validation and test

144