An Overview of Indicator Choice and Normalization in Raw Material Supply Risk Assessments

Round 1

Reviewer 1 Report

General

The manuscript provides an overview of indicators used in the supply risk axis of criticality assessment and their respective scaling. The coverage is extensive and the prose is generally well-written. What I miss is substantial discussion. With some exceptions (e.g. comments on the interpretation of the EPI or price volatility), the review does not go beyond being descriptive, leaving wildly differing choices of normalization simply standing next to each other and not discussing any of the aspects/reasoning/advantages/disadvantages of these choices. I think this is a lost opportunity but the contents fit the claim in the title well.

Specific comments:

line 10-12: increasing attention to criticality does not necessarily imply harmonization

all figures, y-axis label: I think "normalized supply risk score" is an incorrect term for this. Considering most studies in the scope are assessments (cf. Appendix A) and not collections of indicators, the normalized indicators

Fig 1: "Regulations" is a difficult grouping. The naming of the categories was chosen such that "more" of the category implies higher supply risk. This is not the case for "regulations", not only because of the differing interpretations in the case of environmental regulation, i.e. does "more regulation" equal "higher risk"?. I think not. What to do for categories like "

Fig 1: Unsubstitutability is a strange name. In my experience, even the word "substitutability" is difficult to pronounce correctly. Suggest using something else. "Lack of subsitution options" is certainly longer, but possibly easier to read and say.

Fig 2: There is enough room to write out the meaning of A01..J02. Please do. Also, consider using a colored background. instead of the horizontal lines -- these are hard to read next to the long timelines directly adjacent.

line 130, "the risk of concentration": strange formulation. Concentration is a category of indicators in the framework of the manuscript, not a risk to avert.

line 151 and also throughout the manuscript: the description of the normalization is done without considering the meaning of the normalization... are they useful beyond limiting the values to a given scale? Is there any basis or justification for choosing such a normalization?

around line 192 and elsewhere: it would be easier for the reader if the normalization description was briefly repeated instead of e.g., "the same as in A01" (l. 195-196)

Discarding earlier versions of the studies analyzed appears reasonable as long as the methodology stays the same. This is not the case at least for the EU criticality assessments, where there was a partial break in the methodology with the 2017 edition. A consequence of this is that some features are attributed to other studies that were actually already included in the 2010 and 2014 reports of the Ad-hoc working group tasked by the EU. This includes the simple rescaling of the WGI (equivalent to Eq. 5), the interpretation of strong environmental policies as "risk-reducing", and quantification efforts of substitutability across all applications for all candidate raw materials. 

lines 581-589: given that most studies use custom methods and the indicator values are evaluated for different regions at different times, data transparency allowing reproduction and adaptation appears more important that a harmonized normalization/scaling

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

This paper provides comprehensive review for the existing indicators to assess supply risk with focus on normalization to supply risk scores. This review will provide useful insights for future development and selection of criticality assessment methods. I think this paper fits this journal and is worth to be published. On the other hand, to improve the manuscript, I would like to ask the authors to take into account the following points and make corrections.

(L.53) I suggest that the authors state clearly the purpose and focus (i.e., supply risk) of this study, even though the focus is already implied in L.39.

(L.251-252) The authors conducted a simple linear transformation for the results of Ashby [27] in Figure 4. However, if the original literature (i.e., Ashby [27]) did not mention normalization, this normalization may be arbitrary and misleading.

(L.452) calculate?

(L.502) I suggest the authors explain the reason that Eggert and colleagues [19] use a decreasing normalization.

(L.509-510) The phrase “according to economic theory” appears twice.

(L.550-611) Most of the Results are reviews in terms of normalization of supply risk indicators. I think this perspective is very interesting and relevant. However, in the Discussion and Conclusion, the authors hardly discuss the normalization perspective (only discussion about the normalization based on the specific material scope in L.581-589). I suggest that the authors discuss and state the implications derived from this review in terms of normalization more, for example, variation of normalization methods and results, influence of difference in normalization on the criticality assessment results, and how should we select relevant normalization method, etc.

(L. 625) Some reference numbers in the manuscripts do not correspond to the reference lists. At least, I find the inconsistency for L.338, 594, and Table A. I suggest that the authors check all references.

(Supplementary Material spreadsheets, Sheet “S02_Indicators_summary”) Criterion code “X01-X53” follow the code “D01-D15”, not “J01-J04”. I suggest that the authors list “X01-X53” after “J01-J04”, if there is not specific reason for the order.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

The manuscript submitted by Christoph Helbig and colleagues is a well-supported review dealing with a relevant issue (normalization in raw material supply risk assessments). The literature review is extensive and results from different approaches are clearly (and correctly) reported, representing an excellent starting-point for future developments. I have no suggestions and in my modest opinion the manuscript should be accepted in its present form. 

Author Response

Thank you very much.