M.S. AAI Capstone Chronicles 2024

Figure 4 Training and validation loss curves for optimized CNN, Transformer and SCT models

The development of the Deterioration Index demonstrated success in improving model performance. The results of this were most evident in the 16% gain in precision performance observed in baseline models trained on a dataset that incorporated the DI compared to the same model trained on data without the DI. This helped reduce the rate of false positives, an important component of the project’s performance objective. Overall, the SCT model posted consistently strong performance metrics, even before the tuning and optimization routines outlined above were performed. Critically, the model struck a good balance between slightly optimizing for recall, while maintaining good precision performance. Once optimized through a tuning process, the SCT model posted the strongest performance metrics compared to all previous models and demonstrated the same strong generalizing characteristics observed during experimentation. This is most evident in the consistently high AUC-ROC scores of 0.93 on the validation dataset and 0.92 on test dataset. The final performance results of the tuned SCT model and top CNN and Transformer models are outlined in Table 1 below.

277