M.S. AAI Capstone Chronicles 2024

ELECTRICITY DISTRIBUTION TOPOLOGY CLASSIFICATION

11

normalization is employed post-attention and post-FFN within each Transformer block to ensure that the outputs across different neurons have a consistent mean and variance, helping stabilize the learning process. The optimizer used, SGD with momentum and Nesterov acceleration, is tailored to adjust and refine the weights efficiently across these numerous neurons to minimize the loss function (sparse categorical cross-entropy), effectively enhancing the model's accuracy and generalization ability in multi-class classification tasks involving complex sequential inputs. Optimization of the transformer model focused on activation function, number of transformer block layers, optimizer types, and drop-off layer rate. After testing various activation functions, including linear, ReLu, and LeakeyReLu, the LeakeyRelu has the best result. Also, one or two transformer layers did not significantly affect the result. After the drop-off layer rate, try-out between 0.1 and 0.2; 0.1 is good enough to prevent overfitting and provide enough granularity. Also, the AdamW is the best optimizer compared to Adam and SGD. All the optimization and initial model training have been performed under 1000 epochs with training and test split at 85/15 ratio. CNN Model (2nd Experimental Model) The second deep learning approach for this project included the representation of the dataset in a varied format to fit the required format for the CNN model design. The same dataset was used, but instead, the hourly timestamps of each entry were used, and then the same min, max, average, and meter number was used to classify the transformer ID. The dataset was augmented to pad the grouping of meter number entries to achieve a square image of 22x22, corresponding to approximately three months and twenty-four entries per day for each meter number, with some data points filtered or padded as needed. The next step to prep the dataset for the model was to create images by using the voltage min, max, and average as R, G, and B

102