M.S. AAI Capstone Chronicles 2024

11

To correct the behavior seen, the pretrained model is fine-tuned with the UAV training set to

learn the relationships between the images and the labels. During this process the model is further

optimized by selecting hyperparameter values that achieve the best performance. These parameters

include batch size, epochs, learning rate, and an optimizer and are selected via a trial-and-error method.

Results/Conclusion

After the model development and optimization processes are complete, it is determined that the CNN

model with a filter size of 16, a 3x3 kernel, and 100 epochs achieves the best performance. The model

performance metrics are summarized in Table 1. From the metrics obtained, the model is not properly

learning to identify and classify objects in the images from the UAV. The low precision and high recall

indicate that the imbalanced dataset is drastically affecting the model results. The accuracy of the model

is illustrated in Figure 7 and shows that the accuracy is increasing as expected with the training data;

however, the validation accuracy is not improving as rapidly as the model learns. Figure 7 further

illustrates the poor model performance through the loss curve. The increase in validation loss indicates

that the model is overfitting to the training dataset and will not be able to generalize to new data.

Table 1

CNN model performance metrics

Evaluation Dataset

Accuracy

Precision

Recall 0.896 0.872

Validation

0.719 0.745

0.465 0.432

Testing

Figure 7

CNN model accuracy (left) and loss (right)

125