ADS Capstone Chronicles Revised

19

track and tune the hyperparameters in this large dataset because of the computational constraints. That is why the hyperparameter is set on a 20% validation set. Then, with the optimized hyperparameter, the models are run on a 20% test dataset. For hyperparameter tuning the model and making the model robust on the test dataset, k-fold cross-validation is used. Finding optimal collection of hyperparameters that translate well to new data is essential when undertaking hyperparameter tuning. The use of K-fold cross-validation guarantees that chosen hyperparameters will work effectively across several train-test splits, not just one. This raises the possibility that the selected hyperparameters will also function effectively on fresh data. Table 2 Evaluation of different Models on Test Dataset Random forest shows the model can successfully identify the underlying patterns in the data; accuracy, F1 score, precision, and recall are all in the 0.67 to 0.70 range. The hyperparameter tuned on the validation dataset was

'min_samples_leaf': 1, 'min_samples_split': 2. The results of the precision, recall, F1 score, and accuracy are the same. That is why it causes confusion as the model is imbalanced, overfitted or there might be some issues with cross validation processes. According to the confusion matrix shown in Figure 6, the model is not overfitted, and it is not imbalanced. It has high accuracy with some misclassification, but it performs well in most segmentations. They show better accuracy, recall, F1 score, and accuracy than other models. The XGBoost model demonstrates an accuracy of 66%, correctly classifying two-thirds of the instances. Precision and recall scores are balanced. It demonstrates the model can identify positive instances, but the F1 score is 0.64. So, there is room to improve the model with a higher F1 score and better feature engineering, class balancing, and hyperparameter tuning. With its balanced performance, the neural network model is a reliable candidate for improvement. It demonstrates 65% precision, 62% recall, 60% F1 score, and 62% accuracy. The model is reasonably balanced in identifying real positives and producing accurate positive

bootstrap=False,max_depth=None, 'min_samples_leaf=1, 'min_samples_split'=5, 'n_estimators' = 300. The precision and recall scores are highly relatable. It indicates the model can properly identify true positives and minimize false negatives. Decision tree shows the best result among other models, the hyperparameter it shows is: 'criterion': 'entropy', 'max_depth': None,

predictions. However, the precision and recall scores indicate that there is still a great deal of space for improvement. With an F1 score of 0.6011, performance is moderate but balanced, suggesting ongoing work is needed to improve the precision-recall trade-offs Figure 6

63