Model performance relies heavily on its accuracy when predicting outcomes for unseen data. This accuracy is closely linked to the quality of the training data, as the system learns to generalize across various samples. If an unseen observation falls outside the distribution of the training dataset, it can significantly affect the reliability and trustworthiness of the deployed model. This is particularly important in critical infrastructure systems, such as nuclear power plants, where failures can have severe consequences. Traditional methods often assess model accuracy without adequately considering shifts in the distribution of training and real-world data. Such shifts can negatively impact model performance, especially in high-stakes environments. To address these challenges related to model reliability, we propose extending the Dynamic Model Agnostic Reliability Evaluation (DARE) methodology. Additionally, we aim to enhance it by incorporating advanced techniques such as Kernel Density Estimation (KDE) and variance decomposition. We present a novel approach to dynamically update the internal variance-covariance matrix of the Mahalanobis distance in response to changes in data distribution. Utilizing a Mode Detection Algorithm, we identify multimodal feature distributions to obtain a more accurate variance-covariance matrix, thereby improving reliability assessments. To validate our method, we introduce a Bootstrapping Validation Algorithm that evaluates feature-specific variances and ensures the robustness of our proposed approach for out-of-sample predictions. Our findings indicate that a KDE-driven data decomposition method significantly reduces false discovery rates and false negatives, ultimately improving the reliability of predictive models in critical infrastructure, such as nuclear power plants, amid changing data conditions.
