ADS Capstone Chronicles Revised
19
Feature transformation included converting categorical variables into one-hot encoded columns and transforming boolean columns into integer format for model compatibility. To addressskewness,transformationslikelog1pand inverse transformations were applied to continuous variables, ensuring a more normal-like distribution. Additionally, standardization was performed to scale all features, improving model performance, especially for algorithms sensitive to feature magnitude. 4.4.3 Feature Generation Polynomial features were generated tointroduce non-linear relationships between continuous variables, enriching the dataset with additional predictive power. This stepexpandedthefeature space by creating interaction terms and squared values,capturingcomplexdependenciesthatmay not be apparent in the original features. These new features had the potential to provide the selectedmodelwithmorenuancedinformationto improve predictions. 4.4.4 Encoding Target encoding was applied to categorical variablesbyreplacingtheirvalueswiththemean target value for each category. This technique added target-aware information to the dataset improving performance, particularly for models that benefit from capturing direct relationships between features and the target variable. It was especially useful for capturing the influence of specific categorical groups on the target. 4.4.5 Dimensionality Reduction To reduce the dimensionality of the dataset, incremental PCA was applied. This technique ensuredthedatasetwastransformedintoalower dimensional space while preserving as much of the variance as possible. The results presenteda datasetof10independentvariables,setapartbya defining threshold of 50% or more of explained variance.Byselectingcomponentsthatexplaina significant proportion of variance, risks such as
4.3.2 Consistency and Accuracy Consistency checks ensured geographic and temporal alignment of data. Coordinates from accident recordsweresynchronized,andmissing values were imputed wherenecessary.Similarly, weather data timestamps were matched with accident occurrences for accurate analysis. Categorical variables, including Weather_Condition and Wind_Direction were standardized to address inconsistencies, and missing categories were filled with a default value of “Unknown”. Accuracywasanotherkey consideration, though it is influenced by the reliability of the data sources. Accident data, derived from crowd-sourced and public agency reports,mayunder-representminororunreported incidents. Traffic data from SANDAG depends on sensor reliability, and weather data accuracy varies with station proximity to accident sites. 4.4 Feature Engineering Additional feature engineering techniques were carefully applied to enhance data quality and optimizemodelperformance,buildingoninsights uncovered during the EDA phase. To provide clarity and structure, these techniqueshavebeen grouped into distinct categories based on their purposeandmethodology.Thisapproachensures acomprehensiveyetaccessibleexplanationofthe work performed. 4.4.1 Feature Selection Feature selection involved dropping 50 columns deemedirrelevantorunsuitableformodeling.For these, identifiers, geographic coordinates, and redundant features such as the existence of a nearby railwayorfreewayexitaretargeted.This step streamlined the dataset by focusingonlyon variables with meaningful predictive power. By reducing the feature space, noise is minimized and computational efficiency is improved, ensuring modeling only considers the most relevant attributes. 4.4.2 Feature Transformation
259
Made with FlippingBook - Online Brochure Maker