An Empirical Monte Carlo Simulation Framework for Risk Assessment and Uncertainty Quantification in Complex Stochastic Systems
Article Sidebar
Main Article Content
Abstract
Complex stochastic systems are shaped by uncertain input variables, nonlinear interactions, and low-probability failure events that cannot be adequately summarized by deterministic point estimates. This study develops and empirically evaluates a Monte Carlo simulation framework for risk assessment and uncertainty quantification in complex stochastic systems. The main empirical case uses the Artificial Intelligence for Industries (AI4I) 2020 Predictive Maintenance Dataset, which contains 10,000 operational observations and a binary machine-failure outcome. A calibrated random-forest risk model was trained on physical operating variables and then embedded within a Monte Carlo procedure using 100,000 stochastic scenarios generated by empirical resampling with controlled perturbation. The proposed framework estimates expected failure probability, uncertainty intervals, tail-risk indicators, risk-category shares, and sensitivity patterns. The main model achieved strong discrimination on the test set, with an area under the receiver operating characteristic curve (ROC-AUC) of 0.970, average precision of 0.744, and a Brier score of 0.015. The observed failure rate was 3.39%, while the simulated mean failure probability was 3.57%. Monte Carlo outputs showed that most scenarios were low risk, but 6.08% of scenarios exceeded a 10% failure-probability threshold, and the 95th-percentile risk index reached 19.21. A secondary credit-risk robustness case using 30,000 observations confirmed that the same simulation logic can be transferred to a financial default setting. The study contributes a transparent empirical framework that links stochastic inputs, calibrated risk prediction, tail-risk estimation, and decision-oriented interpretation of uncertainty.
Downloads
Article Details
Issue
Section
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
References
1. Aslett, L. J. M., Nagapetyan, T., & Vollmer, S. J. (2017). Multilevel Monte Carlo for reliability theory. Reliability Engineering & System Safety, 165, 188–196. https://doi.org/10.1016/j.ress.2017.03.003 DOI: https://doi.org/10.1016/j.ress.2017.03.003
2. Borgonovo, E., & Plischke, E. (2016). Sensitivity analysis: A review of recent advances. European Journal of Operational Research, 248(3), 869–887. https://doi.org/10.1016/j.ejor.2015.06.032 DOI: https://doi.org/10.1016/j.ejor.2015.06.032
3. Breiman, L. (2001). Random forests. Machine Learning, 45, 5–32. https://doi.org/10.1023/A:1010933404324 DOI: https://doi.org/10.1023/A:1010933404324
4. Carvalho, T. P., Soares, F. A. A. M. N., Vita, R., Francisco, R. P., Basto, J. P., & Alcalá, S. G. S. (2019). A systematic literature review of machine learning methods applied to predictive maintenance. Computers & Industrial Engineering, 137, Article 106024. https://doi.org/10.1016/j.cie.2019.106024 DOI: https://doi.org/10.1016/j.cie.2019.106024
5. Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. https://doi.org/10.1016/j.patrec.2005.10.010 DOI: https://doi.org/10.1016/j.patrec.2005.10.010
6. He, H., & Garcia, E. A. (2009). Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9), 1263–1284. https://doi.org/10.1109/TKDE.2008.239 DOI: https://doi.org/10.1109/TKDE.2008.239
7. Hong, L. J., & Liu, G. (2011). Monte Carlo estimation of value-at-risk, conditional value-at-risk and their sensitivities. In Proceedings of the 2011 Winter Simulation Conference (pp. 95–107). IEEE. https://doi.org/10.1109/WSC.2011.6147743 DOI: https://doi.org/10.1109/WSC.2011.6147743
8. Huang, C., El Hami, A., & Radi, B. (2017). Overview of structural reliability analysis methods—Part II: Sampling methods. Incertitudes et fiabilité des systèmes multiphysiques, 17(1), 1–10. https://doi.org/10.21494/ISTE.OP.2017.0116 DOI: https://doi.org/10.21494/ISTE.OP.2017.0116
9. Iooss, B., & Lemaître, P. (2015). A review on global sensitivity analysis methods. In Uncertainty management in simulation-optimization of complex systems (pp. 101–122). Springer. https://doi.org/10.1007/978-1-4899-7547-8_5 DOI: https://doi.org/10.1007/978-1-4899-7547-8_5
10. Joseph, D. (2021). Predicting credit default probabilities using Bayesian statistics and Monte Carlo simulations. arXiv. https://arxiv.org/abs/2108.03389
11. Kroese, D. P., Brereton, T., Taimre, T., & Botev, Z. I. (2014). Why the Monte Carlo method is so important today. WIREs Computational Statistics, 6(6), 386–392. https://doi.org/10.1002/wics.1314 DOI: https://doi.org/10.1002/wics.1314
12. Kucherenko, S., Tarantola, S., & Annoni, P. (2012). Estimation of global sensitivity indices for models with dependent variables. Computer Physics Communications, 183(4), 937–946. https://doi.org/10.1016/j.cpc.2011.12.020 DOI: https://doi.org/10.1016/j.cpc.2011.12.020
13. Lee, J., Bagheri, B., & Kao, H. A. (2015). A cyber-physical systems architecture for Industry 4.0-based manufacturing systems. Manufacturing Letters, 3, 18–23. https://doi.org/10.1016/j.mfglet.2014.12.001 DOI: https://doi.org/10.1016/j.mfglet.2014.12.001
14. Lee, J., Davari, H., Singh, J., & Pandhare, V. (2018). Industrial artificial intelligence for Industry 4.0-based manufacturing systems. Manufacturing Letters, 18, 20–23. https://doi.org/10.1016/j.mfglet.2018.09.002 DOI: https://doi.org/10.1016/j.mfglet.2018.09.002
15. Matzka, S. (2020a). Explainable artificial intelligence for predictive maintenance applications. In 2020 Third International Conference on Artificial Intelligence for Industries (AI4I) (pp. 69–74). IEEE. https://doi.org/10.1109/AI4I49448.2020.00023 DOI: https://doi.org/10.1109/AI4I49448.2020.00023
16. Matzka, S. (2020b). AI4I 2020 Predictive Maintenance Dataset. UCI Machine Learning Repository. https://doi.org/10.24432/C5HS5C
17. Montero Jimenez, J. J., Schwartz, S., Vingerhoeds, R., Grabot, B., & Salaün, M. (2020). Towards multi-model approaches to predictive maintenance: A systematic literature survey on diagnostics and prognostics. Journal of Manufacturing Systems, 56, 539–557. https://doi.org/10.1016/j.jmsy.2020.07.008 DOI: https://doi.org/10.1016/j.jmsy.2020.07.008
18. Niculescu-Mizil, A., & Caruana, R. (2005). Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning (pp. 625–632). Association for Computing Machinery. https://doi.org/10.1145/1102351.1102430 DOI: https://doi.org/10.1145/1102351.1102430
19. Plischke, E., Borgonovo, E., & Smith, C. L. (2013). Global sensitivity measures from given data. European Journal of Operational Research, 226(3), 536–550. https://doi.org/10.1016/j.ejor.2012.11.047 DOI: https://doi.org/10.1016/j.ejor.2012.11.047
20. Ran, Y., Zhou, X., Lin, P., Wen, Y., & Deng, R. (2019). A survey of predictive maintenance: Systems, purposes and approaches. arXiv. https://arxiv.org/abs/1912.07383
21. Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3), Article e0118432. https://doi.org/10.1371/journal.pone.0118432 DOI: https://doi.org/10.1371/journal.pone.0118432
22. Saltelli, A., Annoni, P., Azzini, I., Campolongo, F., Ratto, M., & Tarantola, S. (2010). Variance-based sensitivity analysis of model output: Design and estimator for the total sensitivity index. Computer Physics Communications, 181(2), 259–270. https://doi.org/10.1016/j.cpc.2009.09.018 DOI: https://doi.org/10.1016/j.cpc.2009.09.018
23. Saracco, P., & Pia, M. G. (2013). An exact framework for uncertainty quantification in Monte Carlo simulation. arXiv. https://arxiv.org/abs/1311.5221
24. Tao, F., Qi, Q., Liu, A., & Kusiak, A. (2018). Data-driven smart manufacturing. Journal of Manufacturing Systems, 48, 157–169. https://doi.org/10.1016/j.jmsy.2018.01.006 DOI: https://doi.org/10.1016/j.jmsy.2018.01.006
25. Wei, P., Lu, Z., & Song, J. (2015). Variable importance analysis: A comprehensive review. Reliability Engineering & System Safety, 142, 399–432. https://doi.org/10.1016/j.ress.2015.05.018 DOI: https://doi.org/10.1016/j.ress.2015.05.018
26. Yeh, I. (2009). Default of Credit Card Clients. UCI Machine Learning Repository. https://doi.org/10.24432/C55S3H
27. Yeh, I.-C., & Lien, C.-H. (2009). The comparison of data mining techniques for the predictive accuracy of the probability of default of credit card clients. Expert Systems with Applications, 36(2), 2473–2480. https://doi.org/10.1016/j.eswa.2007.12.020 DOI: https://doi.org/10.1016/j.eswa.2007.12.020
28. Zhu, H., Liu, T., & Zhou, E. (2020). Risk quantification in stochastic simulation under input uncertainty. ACM Transactions on Modeling and Computer Simulation, 30(1), 1–24. https://doi.org/10.1145/3329117 DOI: https://doi.org/10.1145/3329117
29. Zonta, T., da Costa, C. A., da Rosa Righi, R., de Lima, M. J., da Trindade, E. S., & Li, G. P. (2020). Predictive maintenance in Industry 4.0: A systematic literature review. Computers & Industrial Engineering, 150, Article 106889. https://doi.org/10.1016/j.cie.2020.106889 DOI: https://doi.org/10.1016/j.cie.2020.106889