PERFORMANCE PROFILES FOR BENCHMARKING OF GLOBAL SENSITIVITY ANALYSIS ALGORITHMS
Abstract
Nowadays, sensitivity analysis (SA) is a methodology commonly used to identify important parameters that determine the behavior of the model. The SA of a model allows to determine how uncertainties in the model responses (outputs) can be assigned to the values of the model parameters (input variables). The related literature indicates that there are several methods to perform SA. This work addresses the benchmarking of four widely used methods for Global SA (GSA): Sobol-Jansen, Sobol-Baudin, Sobol-Owen and Sobol 2007, based on the concept of performance profile introduced by Dolan and Moré (2002) and the extension by Mahajan et al. (2012). For the previous methods, a set of 21 models and their variations were considered, which correspond to various applications in chemical engineering (such as heap leaching, water distribution network, milling, flotation circuit, among others). These comparisons show that the Sobol-Jansen method has the best performance profile because it is the first to perform GSA in 83% of the models considered. The four SA methods analyzed proved to be quite stable since they performed the SA in 100% of the models tested.
References
Archer, G. E. B., Saltelli, A., and Sobol, I. M. (1997). Sensitivity measures, anova-like techniques and the use of bootstrap. J. Stat. Comput. Simul. 58, 99–120. DOI:10.1080/00949659708811825.
Austin, L. G. (1990) A mill power equation for SAG mills. Miner. Metall. Process. 7, 57–62.
Austin, L. G., Luckie, P. T., and Wightman, D. (1975). Steady-state simulation of a cement-milling circuit. Int. J. Miner. Process. 2, 127–150. DOI:10.1016/0301-7516(75)90017-4.
Baudin, M., Boumhaout, K., Delage, T., Iooss, B., and Martinez, J.-M. (2016). Numerical stability of Sobol indices estimation formula. Proc. 8th Int. Conf. Sensit. Anal. Model Output. University of Reunion Island, Le Tampon.
Beiranvand, V., Hare, W., and Lucet, Y. (2017). Best practices for comparing optimization algorithms. Optim. Eng. 18, 815–848. DOI:10.1007/s11081-017-9366-1.
Benson H. Y., Shanno, D. F., Vanderbei, R. J. (2000). Interior-point methods for nonconvex nonlinear programming: Jamming and comparative numerical testing. Operations Research and Financial Engineering. Princeton University, Princeton.
Billups, S., Dirkse, S., and Ferris, M.C., (1995). A Comparison of Algorithms for Large Scale Mixed Complementarity Problems. Technical Report. Computer Science Department, University of Wisconsin, Madison.
Bongartz, I., Conn, A. R., Gould, N. I. M., and Sauders, M. A. (1997). A numerical comparison between the LANCELOT and MINOS packages for large-scale nonlinear optimization. Technical Report. Namur University, Belgium.
Brevault, L., M., Balesdent, N., Bérend, and Le Riche, R. (2013). Comparison of different global sensitivity analysis methods for aerospace vehicle optimal design. Proc. 10th World Congress on Structural and Multidisciplinary Optimization, Orlando Florida, 1-12.
Cariboni, J., Gatelli, D., Liska, R., and Saltelli, A. (2007). The role of sensitivity analysis in ecological modelling. Ecol. Model. 203, 167–182. doi.org/10.1016/j.ecolmodel.2005.10.045
Confalonieri, R., Bellocchi, G., Bregaglio, S., Donatelli, M., and Acutis, M. (2010). Comparison of sensitivity analysis techniques: A case study with the rice model WARM. Ecol. Model. 221, 1897-1906. doi.org/10.1016/j.ecolmodel.2010.04.021
Cosenza, A., Mannina, G., Vanrolleghem, P. A., and Neumann, M. B. (2013). Global sensitivity analysis in wastewater applications: A comprehensive comparison of different methods. Environ. Model. Softw. 49, 40–52. DOI:10.1016/j.envsoft.2013.07.009.
Dolan, E. D., and Moré, J. J. (2002). Benchmarking optimization software with performance profiles. Math. Program. Ser. B 91, 201–213. DOI:10.1007/s101070100263.
Duong, P. L. T., Ali, W., Kwok, E., and Lee, M. (2016). Uncertainty quantification and global sensitivity analysis of complex chemical process using a generalized polynomial chaos approach. Comput. Chem. Eng. 90, 23–30. DOI:10.1016/j.compchemeng.2016.03.020.
Galvez, E. D., and Capuz-Rizo, S. F. (2016). Assessment of global sensitivity analysis methods for project scheduling. Comput. Ind. Eng. 93, 110–120. DOI:10.1016/j.cie.2015.12.010.
Gan, Y., Duan, Q., Gong, W., Tong, C., Sun, Y., Chu, W., Ye, A., Miao, C., and Di, Z. (2014). A comprehensive evaluation of various sensitivity analysis methods: A case study with a hydrological model. Environ. Model. Softw. 51, 269–285. DOI:10.1016/j.envsoft.2013.09.031.
Gonzalez-Cruz, M.B., Pire-Rivas, S.F., and Lopez-Jimenez, P.A. (2012). Sensitivity of conservative pollutants estimation using ISCST3 to input parameters for a case of study. Rev. Mex. Ing. Quim. 11, 287-298.
Heiselberg, P., Brohus, H., Hesselholt, A., Rasmussen, H., Seinre, E., and Thomas, S. (2009). Application of sensitivity analysis in design of sustainable buildings. Renew. Energy 34, 2030–2036. DOI:10.1016/j.renene.2009.02.016.
Herman, J. D., Kollat, J. B., Reed, P. M., and Wagener, T. (2013). Technical Note: Method of Morris effectively reduces the computational demands of global sensitivity analysis for distributed watershed models. Hydrol. Earth Syst. Sci. 17, 2893–2903. DOI:10.5194/hess-17-2893-2013.
Hernandez-Suarez, R., Zamora-Mata, J.M., Murrieta-Guevara, F., and Lugo-Leyte, R. (2008). Analysis and optimal retrofit of effluent lines entering a multicontaminant treatment unit. Rev. Mex. Ing. Quim. 7, 151-162.
Herrera, S., Cisternas, L. A., and Gálvez, E. D. (2015). Simultaneous Design of Desalination Plants and Distribution Water Network, Comp. Aided Chem. Eng. 37, 1193-1198. doi.org/10.1016/B978-0-444-63577-8.50044-9
Homma, T., and Saltelli, A. (1996). Importance measures in global sensitivity analysis of nonlinear models. Reliab. Eng. Syst. Saf. 52, 1–17. DOI:10.1016/0951-8320(96)00002-6.
Hopfe, C. J., and Hensen, J. L. M. (2011). Uncertainty analysis in building performance simulation for design support. Energy Build 43, 2798–2805. DOI:10.1016/j.enbuild.2011.06.034.
Ikonen, T. (2016). Comparison of global sensitivity analysis methods - Application to fuel behavior modeling. Nucl. Eng. Des. 297, 72–. DOI:10.1016/j.nucengdes.2015.11.025.
Iooss B., Lemaître P. (2015) A Review on Global Sensitivity Analysis Methods. In Dellino G., Meloni C. (Eds) Uncertainty Management in Simulation-Optimization of Complex Systems. Operations Research/Computer Science Interfaces Series 59. Springer, Boston, MA. DOI: 10.1007/978-1-4899-7547-8_5
Ishigami, T., Homma, T. (1990). An importance quantification technique in uncertainty analysis for computer models. Proc. First International Symposium on Uncertainty Modeling and Analysis, 398–403.
Jamett, N., Cisternas, L. A., and Vielma, J. P. (2015). Solution strategies to the stochastic design of mineral flotation plants. Chem. Eng. Sci. 134, 850–860. DOI:10.1016/j.ces.2015.06.010.
Jansen, M. J. W. (1999). Analysis of variance designs for model output. Comput. Phys. Commun, 117, 35–43. DOI:10.1016/S0010-4655(98)00154-4.
Kaghazchi, T., Mehri, M., Takht, M., and Kargari, A. (2010). A mathematical modeling of two industrial seawater desalination plants in the Persian Gulf region. Desalination 252, 135–142. DOI:10.1016/j.desal.2009.10.012.
Kucherenko, S., Feil, B., Shah, N., and Mauntz, W. (2011). The identification of model effective dimensions using global sensitivity analysis. Reliab. Eng. Syst. Saf. 96. DOI:10.1016/j.ress.2010.11.003.
Lamoureux, B., Mechbal, N., and Massé, J. R. (2014). A combined sensitivity analysis and kriging surrogate modeling for early validation of health indicators. Reliab. Eng. Syst. Saf. 130, 12–26. DOI:10.1016/j.ress.2014.03.007.
Lilburne, L., and Tarantola, S. (2009). Sensitivity analysis of spatial models. Int. J. Geogr. Inf. Sci. 23, 151–168. DOI:10.1080/13658810802094995.
Lorenz, E. N. (1963). Deterministic non-periodic flows. J. Atmos. Sci. 20, 130–141.
Lucay, F., Cisternas, L. A., and Gálvez, E. D. (2015a). A new group contribution method for mineral concentration processes. Comput. Chem. Eng. 74, 28–33. DOI:10.1016/j.compchemeng.2014.12.009.
Lucay, F., Cisternas, L. A., and Gálvez, E. D. (2015b). Global sensitivity analysis for identifying critical process design decisions. Chem. Eng. Res. Des. 103, 74–83. DOI:10.1016/j.cherd.2015.06.015.
Lucay, F., Cisternas, L. A., and Gálvez, E. D. (2015c). Retrofitting of Concentration Plants Using Global Sensitivity Analysis, Comp. Aided Chem. Eng. 37, 311-316. doi.org/10.1016/B978-0-444-63578-5.50047-5
Lucay, F.A., Gálvez, E.D., Sales-Cruz, M., Cisternas, L. A. (2017). Improving milling operation using uncertainty and global sensitivity analyses. Proc. 6th International Computational Modelling Symposium. MEI, Falmouth.
Mahajan, A., Leyffer, S., and Kirches, C. (2012). Solving mixed-integer nonlinear programs by QP- diving. Argonne National Laboratory, Argonne.
Mellado, M. E., Cisternas, L. A., and Gálvez, E. D. (2009). An analytical model approach to heap leaching. Hydrometallurgy 95, 33–38. DOI:10.1016/j.hydromet.2008.04.009.
Nash, S. G., and Nocedal, J. (1991). A numerical study of the limited memory BFGS method and the truncated-Newton method for large scale optimization. SIAM J. Optim. 1, 358–372. DOI:10.1137/0801023.
Nguyen, A., and Reiter, S. (2015). A performance comparison of sensitivity analysis methods for building energy models. Build. Simul. 8, 651–664. doi.org/10.1007/s12273-015-0245-4
Owen, A. B. (2013). Better estimation of small Sobol sensitivity indices. ACM Trans. Model. Comput. Simul. 23, 1–17. DOI:10.1145/2457459.2457460.
Pappenberger, F., Ratto, M., Vandenberghe V. (2010) Review of sensitivity analysis methods. In P.A. Vanrolleghem (Ed.), Modelling aspects of water framework directive implementation, IWA Publishing, 191–265.
Pianosi, F., Beven, K., Freer, J., Hall, J.W., Rougier, J., Stephenson, D.B., Wagener, T. (2016) Sensitivity analysis of environmental models: A systematic review with practical workflow. Environ. Model. Softw. 79, 214-232. DOI: 10.1016/j.envsoft.2016.02.008
Saltelli, A., Annoni, P., Azzini, I., Campolongo, F., Ratto, M., and Tarantola, S. (2010). Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index. Comput. Phys. Commun. 181, 259–270. DOI:10.1016/j.cpc.2009.09.018.
Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., and Tarantola, S. (2008). Global Sensitivity Analysis. The Primer, John Wiley & Sons, Ltd: Chichester, UK.
Saltelli, A., Tarantola, S., and Campolongo, F. (2000). Sensitivity analysis as an ingredient of modeling. Stat. Sci. 15, 377–395. DOI:10.2307/2676831.
Sepulveda, F. D., Cisternas, L. A., and Gálvez, E. D. (2013). Global sensitivity analysis of a mineral processing flowsheet. Comp. Aided Chem. Eng. 32, 913-918. https://doi.org/10.1016/B978-0-444-63234-0.50153-6
Sobol, I. M. (1993) Sensitivity analysis for non-linear mathematical models. Math. Comput. Simul. 1, 407–414.
Sobol, I. M. (2001). Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Math. Comput. Simul. 55, 271–280. DOI:10.1016/S0378-4754(00)00270-6.
Sobol, I. M. (2007). Global sensitivity analysis indices for the investigation of nonlinear mathematical models. Mat. Model. 19, 23–24. DOI:10.18287/0134-2452-2015-39-4-459-461.
Soto-Cruz, O., and Paez-Lerma, J. (2005) fermentation process balances: consistency and metabolic flux analysis. Rev. Mex. Ing. Quim. 4, 59-74.
Van Duc Long, N., and Lee, M. (2012). Dividing wall column structure design using response surface methodology. Comput. & Chem. Eng. 37, 119–124. doi.org/10.1016/j.compchemeng.2011.07.006
Vanderbei, R. J. (1999). LOQO user’s manual - version 3.10. Optim. Methods Softw. 11, 485–514. DOI:10.1080/10556789908805760.
Yianatos, J., Bucarey, R., Larenas, J., Henríquez, F., and Torres, L. (2005). Collection zone kinetic model for industrial flotation columns. Miner. Eng. 18, 1373–1377. DOI:10.1016/j.mineng.2005.01.014.
Yianatos, J. B., and Henríquez, F. D. (2006). Short-cut method for flotation rates modelling of industrial flotation banks. Miner. Eng. 19, 1336–1340. DOI:10.1016/j.mineng.2005.12.010.
Copyright (c) 2019 Revista Mexicana de Ingeniería Química
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
By publishing your paper in our journal you are also granting it the copyright of the information that it contains.
