Benchmarking ISO Risk Management Systems to Assess Efficacy and Help Identify Hidden Organizational Risk

Round 1

Reviewer 1 Report

It is a excellent topic to benchmarking ISO risk management systems in your paper. Below are some suggestions:

Firstly, the content statement in your paper is complex and wordy (such as: 2. Context for the Study; 5. Results), and the structure and language of the paper should be further refined.

Secondly, this study assesses the efficacy of ISO risk management systems. However, it lacks necessary feedback to find the defects of the model which should be improved.

Lastly, the language of the whole paper is good, but there are reference errors in many places in the paper(such as L507、518、551、579、584、589 ,and so on. ).

I hope the suggestions helpful to you.

Author Response

See authors’ comments and corrective actions (in red) after every comment from reviewer.

Paper ID: RA-00723-2020.R2

Paper Title: The Importance of Risk Management: What is missing in ISO standards?

Comments from Reviewer 1:

Firstly, the content statement in your paper is complex and wordy (such as: 2. Context for the Study; 5. Results), and the structure and language of the paper should be further refined.

Corrective action:

The manuscript has been reviewed and proofread with regard to English language and several corrections and improvements made.

The content statement was: “Section 2 describes the context for the study. Section 3 describes the research methodology, the research framework, and the hypotheses put forward in this article. Section 4 presents the results of this study. Section 5 discusses the results, the threefold research aim, and the hypothesis of this study. Section 6 contains conclusions and thoughts on future work.”

The content statement has been simplified and shortened: “In section 2 the context for the study is described; in section 3 the research methodology is illustrated; in section 4 the results are presented; in section 5 a discussion on the results is given; and in section 6 conclusions are drawn.”

Secondly, this study assesses the efficacy of ISO risk management systems. However, it lacks necessary feedback to find the defects of the model which should be improved.

Corrective action:

The following text was added in Conclusions: Although no defects of the model were observed during its use the model needs further refinement. It is adapted to ISO 31000, but the measurability of individual benchmarks needs further development in connection with use in diverse operations. For example, in this study, the measurement of risk criteria setting, and the treatment of residual risk consisted primarily in confirming that these factors were addressed. The way in which they were handled was examined but it was not possible to measure how effective the controls are. For this to be possible, the measurability needs to be investigated further and measurement techniques need to be developed.

Lastly, the language of the whole paper is good, but there are reference errors in many places in the paper (such as L507, 518, 551, 579, 584, 589, and so on).

Corrective action: The reference errors occurred when submitting the manuscript. The reason is unknown, but the authors apologize for this. These errors have all been fixed.

I hope the suggestions helpful to you.

Comment: The authors are grateful for the useful review. It has helped to improve the article.

Author Response File: Author Response.pdf

Reviewer 2 Report

Well-chosen research companies. Six different activities, thus the specifics of risk management in various activities were noticed.

Author Response

See authors’ comments and corrective actions (in red) after every comment from reviewer.

Paper ID: RA-00723-2020.R2

Paper Title: The Importance of Risk Management: What is missing in ISO standards?

Comments from Reviewer 2:

Well-chosen research companies. Six different activities, thus the specifics of risk management in various activities were noticed.

Comment: The authors are grateful for the review. The reference errors in the manuscript occurred when submitting it. The reason is unknown, but the authors apologize for this. These errors have all been fixed. The manuscript has been reviewed and proofread with regard to English language and several corrections and improvements made.

Author Response File: Author Response.pdf

Reviewer 3 Report

Dear authors:

Thank you for the honour of reviewing your publication. My comments are intended to improve your work as per clause 10 Continuous improvement of management systems standards.

In my opinion, the publication is more of a Review / Case Study than an Article.

Benchmarking in the publication belongs to the cross-sectoral type of benchmarking (comparator), I recommend to conclude.

---

"Error! Reference source not found"

in ... Line 278, 307, 507, 517, 534, 551, 579, 584, 588, 589, 592, 593, 603, 672, 676, 734, 738, 855, 857, 831, 973, 982, 1065, 1070, 1140 , 1194, 1224, 1225, 1283, 1286 It must be removed.

---

Always indicate the source from which the figures/tables were drawn or, if this is your own work, please provide this information.

Line 330 ÷ 332 I cannot agree with this statement. I recommend the authors to study the standard EN IEC 31010: 2019, Risk management - Risk assessment techniques specifically Bibliography [1, 2, 3, then selected publications up to 91], I would also recommend incorporating this standard into this publication.

Good job = Table 1

Line 572, I recommend the authors to look at ISO 29001: 2020 especially Annex C.

During the research, the authors found 6 organizations in which specific area they had their management systems certified according to ISO standards? I did not find this information. This information is usually stated on the certificate. This also significantly affects the view of the area of ​​risks that the authors examine. It would be interesting for readers to compile a table from this information.

Line 1272 ÷ 1274. The problem in organizations is also that they confuse the types of risks: managerial, safety, technical, environmental, etc.

Lots of success in your next job.

Sincerely

Your reviewer 

Author Response

See authors’ comments and corrective actions (in red) after every comment from reviewer.

Paper ID: RA-00723-2020.R2

Paper Title: The Importance of Risk Management: What is missing in ISO standards?

Comments from Reviewer 3:

Dear authors:

Thank you for the honour of reviewing your publication. My comments are intended to improve your work as per clause 10 Continuous improvement of management systems standards.

In my opinion, the publication is more of a Review / Case Study than an Article.

Comment: In the authors’ opinion, the manuscript is an article on a case study – broken down to six individual case studies and results summarized.

Benchmarking in the publication belongs to the cross-sectoral type of benchmarking (comparator), I recommend to conclude.

Corrective action: The authors agree that benchmarking in this study belongs to cross-sectoral type of benchmarking. This has been noted in the conclusions: “The benchmarking model in this study belongs to cross-sectoral type of benchmarking and it clearly helps identifying hidden risk, for example …”

"Error! Reference source not found"

in ... Line 278, 307, 507, 517, 534, 551, 579, 584, 588, 589, 592, 593, 603, 672, 676, 734, 738, 855, 857, 831, 973, 982, 1065, 1070, 1140, 1194, 1224, 1225, 1283, 1286 It must be removed.

Corrective action: The reference errors occurred when submitting the manuscript. The reason is unknown, but the authors apologize for this. These errors have all been fixed.

Always indicate the source from which the figures/tables were drawn or, if this is your own work, please provide this information.

Comment: Authors have gone through the manuscript and checked all figures and tables. Only Figure 1 and Table 1 have a source outside of the article, that is from ISO 31000. Reference to that source has been added in the figure text and the table text. A reference is also made in the text, for example, when explaining the figure: “ISO 31000 [12] is the main ISO guideline for risk management …”. All other figures and tables are created by the authors and based on research and case study results.

Line 330 ÷ 332 I cannot agree with this statement. I recommend the authors to study the standard EN IEC 31010: 2019, Risk management - Risk assessment techniques specifically Bibliography [1, 2, 3, then selected publications up to 91], I would also recommend incorporating this standard into this publication.

Corrective action:

Reviewer does not agree with the following statement: “It is in the risk management process where the identification and evaluation of risk takes place. The scientific basis of ISO risk management standards has been questioned in recent scientific literature [8], [31], [32], [33]. ISO standards do not reference scientific literature, only other ISO standards and sometimes risk assessment techniques and handbooks.”

This has been changed to: “According to ISO 31000, it is in the risk management process where the identification and evaluation of risk takes place, see Figure 1. The scientific basis of ISO risk management standards has been questioned in recent scientific literature [8], [31], [32], [33]. ISO standards do not reference scientific literature, only other ISO standards and sometimes risk assessment techniques and handbooks. The only bibliographic reference in ISO 31000 is IEC 31010. IEC was first published in 2009 and then updated in 2019. It is a dual logo IEC/ISO standard for supporting ISO 31000. It provides guidance on selection and application of systematic techniques for risk assessment. Some changes have been made regarding bibliographic references in the latest version of IEC 31010:2019. In version 2009 only 11 bibliographic references were made, all to other ISO/IEC standards. In the 2019 version, the bibliographic references are 91. Many of them are not standards but handbooks and they are categorized in the bibliography according to risk techniques with no direct reference to risk science.”

Good job = Table 1

Comment: Thanks!

Line 572, I recommend the authors to look at ISO 29001: 2020 especially Annex C.

Comment: The authors only found a heading in line 572: “4.2. Hypothesis”. Could the reviewer be meaning another line?

During the research, the authors found 6 organizations in which specific area they had their management systems certified according to ISO standards? I did not find this information. This information is usually stated on the certificate. This also significantly affects the view of the area of ​​risks that the authors examine. It would be interesting for readers to compile a table from this information.

Comment: It is correct, an accredited ISO certification entails a certificate with description of the management system. This description includes name of the organization, issue date and period of validity. The authors wanted to share this information in an appendix, but the organizations in this study do not want to be recognized and they do not want their certificates (with descriptions) to be revealed since it will make them recognizable. Table 3 contains information about the “Organization” in the second column and “Business operation” in the third column. The data in Table 3 was considered giving comparable information and all six organizations have read the manuscript and given their written consent regarding information revealed in this article. The process of review by representatives from the organizations (for example: CEOs, risk experts, lawyers, and project managers) and having a written consent from all organizations took many months.

Line 1272 ÷ 1274. The problem in organizations is also that they confuse the types of risks: managerial, safety, technical, environmental, etc.

Comment: Line 1274 – 1274: “During the time of the study (2014-2019) efforts to improve risk analysis were evident by the public supply system (organization B), the software company (organization E) and the pension fund (organization F). However unsubstantiated methods are used, such as two-dimensional risk matrices, by all organizations except the software company.”

The reviewer is right. The authors found that although the organizations have a certified risk management system, there is uncertainty in many things. Management usually wants things to be clear and understandable in a very short time. Nature of risk often does not allow that. Complex systems require thorough risk analysis and people often fail to admit the depth of complexity. Sometimes even the risk analysis method is considered “too complex” if it reveals true complexity and the depth of the understanding required.

Lots of success in your next job.

Comment: Many thanks! The authors are grateful for the useful review. It has helped to improve the article.

Author Response File: Author Response.pdf

Reviewer 4 Report

This study introduces a two-step benchmarking model to assess the efficacy of ISO risk management systems. It furthermore aims at verifying its usefulness in terms of finding hidden risk issues and improvement opportunities. The existence of all key elements of an ISO 31000 based risk management system are examined at the beginning of this study. Then the quality in terms of efficacy of important aspects of the risk management system is examined in more detail with special benchmarks. The application of the model to six ISO certified organizations follows and reinforces the novelty of this study which is to combine risk science knowledge with benchmarking theory in application of ISO risk management standards in organizations. The results show that the benchmarking model developed in this study provides rigor when assessing and evaluating the efficacy of an ISO risk management system. By applying the model, risk issues and risk factors can be found that had not previously been identified. The findings are of importance for risk management, the benchmarking science, and for the development of ISO risk management standards.

The topic is interesting but there are some points to ne addressed.

The aim of the analysis should be evidenced in the abstract and introduction sections.

The conclusions should be improved with the weaknesses of the analysis and the insights for future research.

Finally, the manuscript should be English proofread because some sentences are not clear.

Author Response

See authors’ comments and corrective actions (in red) after every comment from reviewer.

Paper ID: RA-00723-2020.R2

Paper Title: The Importance of Risk Management: What is missing in ISO standards?

Comments from Reviewer 4:

Comments and Suggestions for Authors

This study introduces a two-step benchmarking model to assess the efficacy of ISO risk management systems. It furthermore aims at verifying its usefulness in terms of finding hidden risk issues and improvement opportunities. The existence of all key elements of an ISO 31000 based risk management system are examined at the beginning of this study. Then the quality in terms of efficacy of important aspects of the risk management system is examined in more detail with special benchmarks. The application of the model to six ISO certified organizations follows and reinforces the novelty of this study which is to combine risk science knowledge with benchmarking theory in application of ISO risk management standards in organizations. The results show that the benchmarking model developed in this study provides rigor when assessing and evaluating the efficacy of an ISO risk management system. By applying the model, risk issues and risk factors can be found that had not previously been identified. The findings are of importance for risk management, the benchmarking science, and for the development of ISO risk management standards.

The topic is interesting but there are some points to be addressed.

The aim of the analysis should be evidenced in the abstract and introduction sections.

Corrective action: The abstract has been changed. The following text was added to the abstract as first sentence: “The overall aim of this article is to contribute to the further development of the area of bench-marking in risk management. The article introduces a two-step benchmarking model to assess the efficacy of ISO risk management systems. It furthermore …”

The conclusions should be improved with the weaknesses of the analysis and the insights for future research.

Corrective action:

Line x-y: “The limitation of this research lies in the data available, time required to analyze data, experts’ knowledge needed to evaluate the data, an understanding of specific and complex systems, and changes that occur in perpetual systems over time.”

This has been changed to: “The limitation of this research lies in the data available, time required to analyze da-ta, experts’ knowledge needed to evaluate the data, an understanding of specific and complex systems, and changes that occur in perpetual systems over time. The weakness of the risk analysis conducted in this study lies in the measurability of both risk and efficacy of risk management. This is difficult to standardize, and every organization must find an appropriate risk analysis technique where causal relationship of risk factors, risk criteria, risk acceptance and residual risk can be made understandable and measurable. It would be of great value if ISO standards would contain better guidance to help and support their users in this continuous process.”

Finally, the manuscript should be English proofread because some sentences are not clear.

Corrective action: The manuscript has been proofread.

Comment: Many thanks! The authors are grateful for the useful review. It has helped to improve the article.

Author Response File: Author Response.pdf

Round 2

Reviewer 3 Report

Dear Authors:

Line 572: Contradiction = However, ISO standards lack guidance on risk management as demonstrated by Björnsdóttir et al. in a previous study [8]. Consequently, it is expected that risk management, and particularly the analysis of risk, is executed in an unsatisfactory manner.

I recommend the authors to look at ISO 29001: 2020 especially Annex C.

Sincerely

Your reviewer