An Effective Uncorrectable Memory Error Prediction Framework by Exploiting UPH Indicators in Production Environments

UCEs (Uncorrectable memory errors) pose significant challenges to cloud computing systems, often resulting in catastrophic failures and crashes. Researchers have explored prediction approaches to address this issue. Previous studies have provided insights into memory error prediction, focusing on memory module part numbers and relationships between error code data. However, these efforts face challenges due to insufficient data features and suboptimal optimization, especially in production environments where hardware/software sparing techniques are widely deployed, the UCE ratio is low, and long lead time is required. To address these issues, our study first collect a large amount of memory data from different vendors in Huawei's production environment, which has deployed hardware/software sparing techniques, to provide more general data. Second, we exploit new indicators termed UPH (Unique, Pinx, and History) from this data, which play a crucial role in predicting UCEs. UPH offers a more profound understanding of the factors contributing to UCEs and demonstrates higher precision and recall. Then, we integrate existing indicators and UPH into our prediction framework and demonstrate the significance of UPH through indicator importance assessments. We also optimize the framework by determining an optimal sampling window. In production environments with long lead time and low UCE ratio, we improve the framework by implementing noise reduction, self-history learning, and a new scenario-based model selection approach. Experimental results demonstrate 19% - 27% increase in UCE prediction recall with 4%-11% increase in precision under different scenarios, outperforming state-of-the-art methods in production environments.