ADS Capstone Chronicles Revised

18

For the second model, each model was evaluated on 20% unseen test data. Key metrics such as precision, recall, F1 score and accuracy are used to evaluate different models. 5.1 Evaluation of Results 5.1.1 evaluation of results for recommendation . The results from the SVD model illustrate how increasing the number of components affects both prediction accuracy and recommendation quality. Initially, as the number of components increases from 5 to 100, the RMSE steadily decreases from approximately 177.91 to 127.52, indicating improved prediction accuracy. Concurrently, precision@5 and recall@5 both show significant improvements with increasing components, starting from 0.0627 and 0.0960 at 5 components, respectively, and rising to 0.3692 and 0.5626 at 100 components. This suggests that higher dimensionality enhances the model's ability to identify relevant items and capture a greater portion of relevant recommendations. At lower values of n_components, the model demonstrates relatively poor performance, with precision and recall values being quite low. However, as the number of components increases, the model’s performance improves substantially, achieving the best results at n_components=100 with a RMSE of 107.08, precision@5 of 0.2787, and recall@5 of 0.5878. This optimal configuration balances a low RMSE with high precision and recall, reflecting the model's effectiveness in both accurately predicting ratings and recommending relevant items to users. Overall, the SVD model benefits from higher dimensionality, which enhances both accuracy and recommendation quality. The results from the KNN model reveal a nuanced balance between prediction accuracy and recommendation quality as the number of neighbors k changes. At k=2, the model performs exceptionally well, achieving the lowest RMSE of approximately 70.56, which indicates that it

provides the most accurate predictions of user ratings compared to other k values tested. Precision@2 is relatively high at 0.60 and recall@2 is also robust at 0.62. This suggests that the model not only makes accurate predictions but also captures a substantial proportion of relevant items, reflecting a good balance between precision and recall for this k value. As k increases, a notable shift occurs in the model’s performance. The RMSE shows a gradual increase, peaking at 107.92 when k=100, indicating that prediction accuracy diminishes as more neighbors are considered. Precision decreases consistently with higher k, dropping to 0.03 at k=100, which implies that the model’s ability to correctly identify the most relevant items diminishes. Conversely, recall improves steadily with increasing k, reaching 0.81 at k=100. This indicates that while the model can identify a larger proportion of relevant items with more neighbors, it does so at the cost of precision, reflecting a trade-off between the breadth and accuracy of recommendations. Thus, selecting the optimal k involves balancing these competing metrics to best meet specific recommendation objectives. Overall, the evaluation of SVD and KNN models highlighted the importance of balancing multiple performance metrics to achieve optimal recommendation quality. The SVD model benefited from higher dimensionality and the KNN model demonstrated a trade-off between precision and recall as the number of neighbors increased. These insights are crucial for selecting and fine-tuning recommendation models to meet specific objectives in different application contexts. 5.1.2 evaluation of results for customer separation . The performance of each model depends on the fine-tuning of the model in the dataset. The dataset comprised up to 300k rows. It is hard to

62