Application of AHP for the Weighting of Sustainable Development Indicators at the Subnational Level

Round 1

Reviewer 1 Report

The authors of this interesting article present a method for weighting indicators, more precisely for the construction of composite indices useful for assessing sustainable development at the subnational level, the case study discussed was Aburrá Valley region belonging to Antioquia state (Colombia). The topic of this manuscript falls within the objectives of the journal. The topic is well presented, and the English style and grammar are fine, as the text is easy to follow. However, further efforts are needed to make better use of the work. The authors are recommended as follows.
line 130: I would propose to report the three indicators that are not present in table 1.
line 144 and 145: better clarify how the indicators are compared.
line 243 to 251 : I would move the content of this part to an earlier section, even in the introduction.

Author Response

Dear Evay Yuan                                                                                             Economics
MDPI

We communicate with you through this letter in order to outline every change made in the article economies-1394433 “Application of AHP for the Weighting of Sustainable Development Indicators at the Subnational Level” based on the observations and suggestions provided by the reviewers (Reviewer 1, Reviewer 2). All changes and modifications made to the manuscript are highlighted in yellow in the revised version.

Comments from author to reviewers:
-Reviewer 1

The authors of this interesting article present a method for weighting indicators, more precisely for the construction of composite indices useful for assessing sustainable development at the subnational level, the case study discussed was Aburrá Valley region belonging to Antioquia state (Colombia). The topic of this manuscript falls within the objectives of the journal. The topic is well presented, and the English style and grammar are fine, as the text is easy to follow. However, further efforts are needed to make better use of the work. The authors are recommended as follows.

line 130: I would propose to report the three indicators that are not present in table 1.

R// A paragraph is added to explain why indicators related to SDG 9, SDG 13 and SDG 14 were not considered (lines 220-230):

“Note that indicators for SDG 9, SDG 13, and SDG 14 are not shown in Table 1 due to diverse causes. First, SDG 9 (Industry, innovation, and infrastructure) is not available in the MIO, and the availability of the indicator is essential for the construction of indices (Tanguay et al. 2010; Shaaban and Scheffran 2017). Second, indicators related to SDG 13 (Climate Action) are not available for all the municipalities of the Aburrá Valley, which is essential to compare them (Londoño 2018) and obtain a unified value for the region. Likewise, in the Aburrá Valley, no indicator covers the topic to be evaluated (Hák, Janousková, and Moldan 2016). Finally, SDG 14 (Life below water) does not apply to the Aburrá Valley because this region is located within the department of Antioquia, so it does not have beaches or access to the sea (Londoño and Cruz 2019).”

line 144 and 145: better clarify how the indicators are compared.

R// A column is added to Table 2 explaining how these indicators contribute directly to each SDG (line 246).

line 243 to 251 : I would move the content of this part to an earlier section, even in the introduction.

R// It was decided to remove this paragraph from the results section. In the introduction, several paragraphs were added to deepen on the methods for weighting criteria (lines 84-115).

“For indicator weighting, weighting methods are classified into equal weighting and rank-order weighting (Shao et al. 2020; Si and Marjanovic-Halburd 2018). In equal weighting, indicator weights are equally assigned, which does not require stakeholder preferences; however, it ignores the relative importance of the criteria (Shao et al. 2020; Wang et al. 2009). In rank-order weighting methods, indicators weights are distributed and influenced by stakeholder perspectives (Si and Marjanovic-Halburd 2018). The rank-order weighting methods include subjective weighting method, objective weighting method and combination weighting method (Wang et al. 2009). Subjective weighting methods depend only on the preference of decision-makers, while objective weighting methods emphasize the statistical evaluation of the given data, complex calculation process and intensive data requirement (Shao et al. 2020). Combination weighting method is used to balance merits and limitations of objective and subjective weighting methods, but the process is complex and not widely-used (Si and Marjanovic-Halburd 2018).”

“Subjective weighting methods include Analytic Hierarchy Process (AHP), Analytic Network Process (ANP), Best Worst Method (BWM), Delphi method, DEMATEL, Direct given method, Eigenvector method, Expert judgement, Full consistency method Step-Wise Weight Assessment Ratio Analysis (SWARA), Fuzzy preference programming, Least-square method, Quality function deployment (QFD), Simos method, Simple Multi-Attribute Rating Technique (SMART), Swing method. Objective weighting methods include Data envelopment analysis (DEA), Divergence measure method, Entropy method, Hesitant fuzzy entropy measure, Least mean square (LMS) method, Linear programming technique for multidimensional analysis of preference (LINMAP), Maximizing deviation method, Minmax deviation method, Multiple correlation coefficient, Ordered weight, Preference selection index, Principal component analysis (PCA), Relative preference relation, Time sequence weight, Technique for Order of Preference for Similarity Ideal Solution TOPSIS method, Variation coefficient, Vertical and horizontal method. Combination weighting methods include Additive synthesis, AHP and entropy method, Direct given and ordered weight, Entropy method and divergence measure method, Expert judgement and entropy method, Expert judgement and maximizing deviation method, Expert judgement and maximizing distance method, Expert judgement and statistical variance method, Multiplication synthesis, Optimal weighting based on minimum bias, Optimal weighting based on sum of squares (Wang et al. 2009; L. J. Zhang et al. 2020).”

Comments from author to reviewers:
-Reviewer 2

I am glad I have this opportunity to review your manuscript. Based on its title/abstract it looks very interesting. Anyway, my comments/notes are the follows:

Introduction. Stages of "assessing sustainable development using aggregated indices requires"... could be more described (specify what is the content of each stage by 1-2 sentences).

R// The following paragraph is added to describe each of the steps required for assessing sustainable development (lines 60-83)

“The conceptual framework development establishes the approach to analyze the sustainable development assessment, establishing the backgrounds, contexts, and relationships between the economic, social, environmental, and institutional dimensions. The indicator selection for sustainable development assessment should provide an estimate of the achievement of the SDGs, quantify both the determinants and the final impacts of sustainability, and influence the decision-making process with the information they provide. The indicator selection can be supported by several methods and should adopt the following principles: systematic, consistency, independency, measurability, comparability  (Wang et al. 2009; Rigo et al. 2020; Shao et al. 2020). Imputation of missing data recognizes the existence of variables that cannot be observed or measured, so an imperfect or approximate measurement can be achieved based on the valid values of other variables or based on a sample. It tends to be recurrent in sustainable development assessment at the subnational level due to the lack of structured and formal sources of information related to the SDGs. Data standardization normalizes the data obtained from the selected indicators so they can be used for quantitative calculations (Shao et al. 2020). Indicator weighting allows a trade-off between multiple indicators and a balance between different sustainable development perspectives, so it is necessary to select the weighting method to define or quantify the importance/relevance of the selected indicators (Rigo et al. 2020; Németh et al. 2019). The aggregation implies aggregating the normalized information of the selected indicators considering their weighting for calculating a total sustainable development score. Sensitivity analysis represents a validation method mainly performed by varying the indicator weights and indicators values to establish how stable the results are when changing the conditions considered in the sustainable development assessment (Shao et al. 2020; Rigo et al. 2020; Allen et al. 2020).”

Introduction. • three dimensions could be illustrated by any figure

R// Figure 1 is added to illustrate the three dimensions and four dimensions of sustainable development (line 46)

Figure 1. Three dimensions of sustainable development (a); four dimensions of sustainable development (b)

Introduction. • "For indicator weighting, several techniques..." - it is a poorly prepared overview of the methods used for criteria weighting (there are many other methods which could be also used). From my point of view, firstly should be these methods presented in general and after that, it could be more focused on one selected method (AHP),

R// We added two paragraphs dedicated to an overview of the methods used for criteria weighting (lines 84-115)

“For indicator weighting, weighting methods are classified into equal weighting and rank-order weighting (Shao et al. 2020; Si and Marjanovic-Halburd 2018). In equal weighting, indicator weights are equally assigned, which does not require stakeholder preferences; however, it ignores the relative importance of the criteria (Shao et al. 2020; Wang et al. 2009). In rank-order weighting methods, indicators weights are distributed and influenced by stakeholder perspectives (Si and Marjanovic-Halburd 2018). The rank-order weighting methods include subjective weighting method, objective weighting method and combination weighting method (Wang et al. 2009). Subjective weighting methods depend only on the preference of decision-makers, while objective weighting methods emphasize the statistical evaluation of the given data, complex calculation process and intensive data requirement (Shao et al. 2020). Combination weighting method is used to balance merits and limitations of objective and subjective weighting methods, but the process is complex and not widely-used (Si and Marjanovic-Halburd 2018).”

“Subjective weighting methods include Analytic Hierarchy Process (AHP), Analytic Network Process (ANP), Best Worst Method (BWM), Delphi method, DEMATEL, Direct given method, Eigenvector method, Expert judgement, Full consistency method - Step-Wise Weight Assessment Ratio Analysis (SWARA), Fuzzy preference programming, Least-square method, Quality function deployment (QFD), Simos method, Simple Multi-Attribute Rating Technique (SMART), Swing method. Objective weighting methods include Data envelopment analysis (DEA), Divergence measure method, Entropy method, Hesitant fuzzy entropy measure, Least mean square (LMS) method, Linear programming technique for multidimensional analysis of preference (LINMAP), Maximizing deviation method, Minmax deviation method, Multiple correlation coefficient, Ordered weight, Preference selection index, Principal component analysis (PCA), Relative preference relation, Time sequence weight, Technique for Order of Preference for Similarity Ideal Solution TOPSIS method, Variation coefficient, Vertical and horizontal method. Combination weighting methods include Additive synthesis, AHP and entropy method, Direct given and ordered weight, Entropy method and divergence measure method, Expert judgement and entropy method, Expert judgement and maximizing deviation method, Expert judgement and maximizing distance method, Expert judgement and statistical variance method, Multiplication synthesis, Optimal weighting based on minimum bias, Optimal weighting based on sum of squares (Wang et al., 2009; Zhang et al., 2020).”

Likewise, a paragraph is added to describe the functionality and practicality of the AHP for the weighting of indicators (lines 141-147):

“Moreover, the AHP is simple to understand and apply to complex issues using uses a 9-point scale to compare criteria relative importance. The AHP decomposes a large problem into smaller sub-problems at hierarchical levels, introduces the comparative importance of the criteria, showing a more reliable representation of the decision goal. The AHP is applicable for quantitative and qualitative criteria and checks the consistency of the decision, thus reducing the bias in the decision-making progression (Shao et al., 2020; Si & Marjanovic-Halburd, 2018).”

Indicator weighting for sustainable development. •         this is/could be a part of the methodology, not the theoretical section,

R // The name of the section “Indicator weighting for sustainable development” is changed to “Methodology” (line 180)

Indicator weighting for sustainable development. •         Figure 2 - correct "Consistency accepted" where "y" is on a new line,

R // The word “Consistency” is corrected to present it in a single line (line 202)

Indicator weighting for sustainable development. •         Table 1 - I expect more information about each criterion used (some base characteristics could be added, e.g. type of variable, source, some moment characteristics, ...),

R // A column with the description of each indicator is added to Table 1 (line 231)

Pairwise Comparisons •              I recommend changing the title of this section and do not divide next into short sub-sections (Normalized Matrix and Indicator Weighing Calculation, Consistency Test, ...). Is totally useless to have a section including one paragraph (Indicator Aggregation for SDG Weighing). They could be presented as steps of the calculation procedure, but for sure not as sub-sections.

R // The structure of the Methodology section is modified considering only two subsections: 2.1. Data and Indicator Description (line 204) and 2.2 AHP for indicator weighting (line 258)

Pairwise Comparisons •              you present your option (AHP, consistency test) as the only one possible option how to do it, there are many other approaches that are not considered or mentioned at least,

R// We added two paragraphs dedicated to an overview of the methods used for criteria weighting (lines 84-115)

“For indicator weighting, weighting methods are classified into equal weighting and rank-order weighting (Shao et al. 2020; Si and Marjanovic-Halburd 2018). In equal weighting, indicator weights are equally assigned, which does not require stakeholder preferences; however, it ignores the relative importance of the criteria (Shao et al. 2020; Wang et al. 2009). In rank-order weighting methods, indicators weights are distributed and influenced by stakeholder perspectives (Si and Marjanovic-Halburd 2018). The rank-order weighting methods include subjective weighting method, objective weighting method and combination weighting method (Wang et al. 2009). Subjective weighting methods depend only on the preference of decision-makers, while objective weighting methods emphasize the statistical evaluation of the given data, complex calculation process and intensive data requirement (Shao et al. 2020). Combination weighting method is used to balance merits and limitations of objective and subjective weighting methods, but the process is complex and not widely-used (Si and Marjanovic-Halburd 2018).”

“Subjective weighting methods include Analytic Hierarchy Process (AHP), Analytic Network Process (ANP), Best Worst Method (BWM), Delphi method, DEMATEL, Direct given method, Eigenvector method, Expert judgement, Full consistency method - Step-Wise Weight Assessment Ratio Analysis (SWARA), Fuzzy preference programming, Least-square method, Quality function deployment (QFD), Simos method, Simple Multi-Attribute Rating Technique (SMART), Swing method. Objective weighting methods include Data envelopment analysis (DEA), Divergence measure method, Entropy method, Hesitant fuzzy entropy measure, Least mean square (LMS) method, Linear programming technique for multidimensional analysis of preference (LINMAP), Maximizing deviation method, Minmax deviation method, Multiple correlation coefficient, Ordered weight, Preference selection index, Principal component analysis (PCA), Relative preference relation, Time sequence weight, Technique for Order of Preference for Similarity Ideal Solution TOPSIS method, Variation coefficient, Vertical and horizontal method. Combination weighting methods include Additive synthesis, AHP and entropy method, Direct given and ordered weight, Entropy method and divergence measure method, Expert judgement and entropy method, Expert judgement and maximizing deviation method, Expert judgement and maximizing distance method, Expert judgement and statistical variance method, Multiplication synthesis, Optimal weighting based on minimum bias, Optimal weighting based on sum of squares (Wang et al., 2009; Zhang et al., 2020).”

Likewise, a paragraph is added to describe the functionality and practicality of the AHP for the weighting of indicators (lines 141-147):

“Moreover, the AHP is simple to understand and apply to complex issues using uses a 9-point scale to compare criteria relative importance. The AHP decomposes a large problem into smaller sub-problems at hierarchical levels, introduces the comparative importance of the criteria, showing a more reliable representation of the decision goal. The AHP is applicable for quantitative and qualitative criteria and checks the consistency of the decision, thus reducing the bias in the decision-making progression (Shao et al., 2020; Si & Marjanovic-Halburd, 2018).”

Indicator weighting for sustainable development. •         this section should be marked as "Methodology" or "Material and Methods", clearly should be divided into sections focusing on the subject (region), data and criteria description, methods used.

R// The name of the section “Indicator weighting for sustainable development” is changed to “Methodology” (line 180). The structure of the Methodology section is modified considering only two subsections: 2.1. Data and Indicator Description (line 204) and 2.2 AHP for indicator weighting (line 258)

Results and Discussion •             Table 4 - without any description,

R // We improve and deepen the description of Table 4 (lines 314-331):

“As a result of the AHP method, Table 4 shows the matrix (A) of paired comparisons, which contains values from a 9-point scale to compare indicators' relative importance that corresponds to the original scale of preferences of Saatty.”

“These values reflect the importance of indicators defined by expert criteria by comparing elements of a row with elements of a column. For example, when comparing X₄ with X₂₀ (value of row 4 and column 20), a value of 9 was assigned, which means that the indicator X₄ (Maternal mortality rate) is of absolute importance concerning the indicator X₁₉ (Indebtedness index), then absolutely favoring the good health and well-being of pregnant women concerning the level of indebtedness of the municipalities, that is, prioritizing the social dimension over the economic and institutional dimension. Respectively, the value of the element in row 20 and column 4 corresponds to the inverse value of 9 (0,11).”

“Likewise, when comparing X₄ with X₂₃ (row 4 and column 23), a value of 1 was assigned, which means that the indicator X₄ (Maternal mortality rate) is equally important as the indicator X23 (Concentration of Particulate Material PM2.5) since air quality directly affects the health of citizens, specifically those who are at risk of respiratory diseases. Therefore, it is considered that both indicators contribute equally to the social dimension. Respectively, the element in row 23 and column 4 corresponds to the inverse of 1, which is also equal to 1.”

Results and Discussion •             Table 5 - totally useless (it is only an intermediate step that is ok, but it is not necessary to describe it by a table),

R// Table 5 is eliminated, and it is mentioned as a complement to the description of Table 4 how the normalized values are calculated (lines 334-336).

“Equation (1) is applied to the values of the matrix (A) of paired comparisons to obtain the Normalized Matrix, and the Indicator Weighing Calculation is obtained based on that matrix using Equation (2).”

Results and Discussion • you are not interpreting the results obtained at all, you just put some numbers there and advocate the use of the AHP method (which should be a part of the previous section),

R// Three paragraphs are added to interpret the results obtained (lines 355-378):

“The results of this work show that the sustainable development priorities for the municipalities of the Aburrá Valley should focus primarily on the health issue, defined by the CPS, since the indicators with the most considerable weight are related to SDG3, as they are the maternal mortality rate, mortality rate for HIV-AIDS, and the under-five mortality rate. The health issue also includes an indicator of SDG 11 related to concentrations of particulate matter PM2.5 since poor air quality constitutes a public health problem in the Aburrá Valley. This situation is relevant because this region is surrounded by mountains that do not allow adequate air circulation, concentrating particulate material inside the valley.”

“Moreover, this study prioritizes the issue of employment represented in the indicators that contribute to SDG₈. In this sense, the unemployment rate, underemployment rate, and dependency rate worsened due to the confinement generated by the Covid-19 pandemic; therefore, it will continue as a priority in the next quality of life surveys in this region. Similarly, the issue of education occupies the third place of preferences with the indicators related to SDG₄, such as the coverage of primary, secondary and higher education, and illiteracy rate indicators. The prioritization of this indicator indicates that governments of the municipalities of the Aburrá Valley must provide proper conditions for citizens to access education, especially higher education, to train citizens for the labor market that continually requires a more skilled workforce.”

“Accordingly, the indicators with the most considerable weight reflect the priorities of sustainable development in the Aburrá Valley corresponding to SDG₃ (Health and well-being), SDG₈ (Decent work and growth economic), and SDG₄ (Quality education). Therefore, development plans, programs, and the municipal budget of these municipalities should focus on improving health, education, and employment issues.”

Results and Discussion •             "This work was focused on indicator..." - the same problem, this should be a part of other section, real discussion is missing (you do not have any results interpretation to discuss),

R// The entire paragraph was moved and placed at the end of the Introduction section (lines 156-173)

Conclusions •   "However, the novelty of this study is providing an indicator weighting model that allows the assessment of sustainable development at the subnational level." - really? You just choose one method and applied it to one case study, this is novelty?

R// The novelty of this study is highlighted in the conclusions in a new paragraph added (lines 396-401):

“This study is novel because it allows prioritizing sustainable development dimensions through a methodology that articulates the technical approach derived from the 17 SDGs of the 2030 agenda (objective) with a participatory approach represented by the CPS (subjective) where citizens prioritize the most relevant life-quality issues. Moreover, a multi-criteria method based on the AHP supports the proposed methodology to lead political decision-making.”

Conclusions •   "Future research should focus..." - it should be a part of this manuscript, in the current version it looks like you just select a part of your research for publishing it separately.

R// The future research suggestion is modified in the conclusions, showing future lines of research that do not depend directly on the stages of our research (lines 404-409).

“Future works may implement the proposed methodology in regions like the Aburrá Valley to compare the sustainable development priorities and establish the factors generating variations in the assignment of preferences. Likewise, future studies should consider the uncertainty and vagueness in expert judgment through linguistic evaluations, gray-based methods, and fuzzy logic systems.”

Finally, we thank the reviewers for their suggestions and observations, which helped to improve the quality of the article, both in its form and content, ensuring that it is an article of high impact for the academic and scientific community.

Author Response File: Author Response.pdf

Reviewer 2 Report

Dear authors, 

I am glad I have this opportunity to review your manuscript. Based on its title/abstract it looks very interesting. Anyway, my comments/notes are the follows: 

• Introduction
  • Stages of "assessing sustainable development using aggregated indices requires"... could be more described (specify what is the content of each stage by 1-2 sentences),
  • three dimensions could be illustrated by any figure, 
  • "For indicator weighting, several techniques..." - it is a poorly prepared overview of the methods used for criteria weighting (there are many other methods which could be also used). From my point of view, firstly should be these methods presented in general and after that, it could be more focused on one selected method (AHP), 
• Indicator weighting for sustainable development
  • this is/could be a part of the methodology, not the theoretical section, 
  • Figure 2 - correct "Consistency accepted" where "y" is on a new line,
  • Table 1 - I expect more information about each criterion used (some base characteristics could be added, e.g. type of variable, source, some moment characteristics, ...), 
• Pairwise Comparisons
  • I recommend changing the title of this section and do not divide next into short sub-sections (Normalized Matrix and Indicator Weighing Calculation, Consistency Test, ...). Is totally useless to have a section including one paragraph (Indicator Aggregation for SDG Weighing). They could be presented as steps of the calculation procedure, but for sure not as sub-sections.
  • you present your option (AHP, consistency test) as the only one possible option how to do it, there are many other approaches that are not considered or mentioned at least,
  • this section should be marked as "Methodology" or "Material and Methods", clearly should be divided into sections focusing on the subject (region), data and criteria description, methods used.
• Results and Discussion
  • Table 4 - without any description,
  • Table 5 - totally useless (it is only an intermediate step that is ok, but it is not necessary to describe it by a table),
  • you are not interpreting the results obtained at all, you just put some numbers there and advocate the use of the AHP method (which should be a part of the previous section), 
  • "This work was focused on indicator..." - the same problem, this should be a part of other section, real discussion is missing (you do not have any results interpretation to discuss),
• Conclusions
  • "However, the novelty of this study is providing an indicator weighting model that allows the assessment of sustainable development at the subnational level." - really? You just choose one method and applied it to one case study, this is novelty? 
  • "Future research should focus..." - it should be a part of this manuscript, in the current version it looks like you just select a part of your research for publishing it separately.

There are many methodological shortcomings that should be solved. The results should be better prepared and much more interpreted. Check also the language level.

Author Response

Dear Evay Yuan                                                                                              Medellín, October 5^th, 2021

Economics
MDPI

We communicate with you through this letter in order to outline every change made in the article economies-1394433 “Application of AHP for the Weighting of Sustainable Development Indicators at the Subnational Level” based on the observations and suggestions provided by the reviewers (Reviewer 1, Reviewer 2). All changes and modifications made to the manuscript are highlighted in yellow in the revised version.

Comments from author to reviewers:
-Reviewer 1

The authors of this interesting article present a method for weighting indicators, more precisely for the construction of composite indices useful for assessing sustainable development at the subnational level, the case study discussed was Aburrá Valley region belonging to Antioquia state (Colombia). The topic of this manuscript falls within the objectives of the journal. The topic is well presented, and the English style and grammar are fine, as the text is easy to follow. However, further efforts are needed to make better use of the work. The authors are recommended as follows.

line 130: I would propose to report the three indicators that are not present in table 1.

R// A paragraph is added to explain why indicators related to SDG 9, SDG 13 and SDG 14 were not considered (lines 220-230):

“Note that indicators for SDG 9, SDG 13, and SDG 14 are not shown in Table 1 due to diverse causes. First, SDG 9 (Industry, innovation, and infrastructure) is not available in the MIO, and the availability of the indicator is essential for the construction of indices (Tanguay et al. 2010; Shaaban and Scheffran 2017). Second, indicators related to SDG 13 (Climate Action) are not available for all the municipalities of the Aburrá Valley, which is essential to compare them (Londoño 2018) and obtain a unified value for the region. Likewise, in the Aburrá Valley, no indicator covers the topic to be evaluated (Hák, Janousková, and Moldan 2016). Finally, SDG 14 (Life below water) does not apply to the Aburrá Valley because this region is located within the department of Antioquia, so it does not have beaches or access to the sea (Londoño and Cruz 2019).”

line 144 and 145: better clarify how the indicators are compared.

R// A column is added to Table 2 explaining how these indicators contribute directly to each SDG (line 246).

line 243 to 251 : I would move the content of this part to an earlier section, even in the introduction.

R// It was decided to remove this paragraph from the results section. In the introduction, several paragraphs were added to deepen on the methods for weighting criteria (lines 84-115).

“For indicator weighting, weighting methods are classified into equal weighting and rank-order weighting (Shao et al. 2020; Si and Marjanovic-Halburd 2018). In equal weighting, indicator weights are equally assigned, which does not require stakeholder preferences; however, it ignores the relative importance of the criteria (Shao et al. 2020; Wang et al. 2009). In rank-order weighting methods, indicators weights are distributed and influenced by stakeholder perspectives (Si and Marjanovic-Halburd 2018). The rank-order weighting methods include subjective weighting method, objective weighting method and combination weighting method (Wang et al. 2009). Subjective weighting methods depend only on the preference of decision-makers, while objective weighting methods emphasize the statistical evaluation of the given data, complex calculation process and intensive data requirement (Shao et al. 2020). Combination weighting method is used to balance merits and limitations of objective and subjective weighting methods, but the process is complex and not widely-used (Si and Marjanovic-Halburd 2018).”

“Subjective weighting methods include Analytic Hierarchy Process (AHP), Analytic Network Process (ANP), Best Worst Method (BWM), Delphi method, DEMATEL, Direct given method, Eigenvector method, Expert judgement, Full consistency method Step-Wise Weight Assessment Ratio Analysis (SWARA), Fuzzy preference programming, Least-square method, Quality function deployment (QFD), Simos method, Simple Multi-Attribute Rating Technique (SMART), Swing method. Objective weighting methods include Data envelopment analysis (DEA), Divergence measure method, Entropy method, Hesitant fuzzy entropy measure, Least mean square (LMS) method, Linear programming technique for multidimensional analysis of preference (LINMAP), Maximizing deviation method, Minmax deviation method, Multiple correlation coefficient, Ordered weight, Preference selection index, Principal component analysis (PCA), Relative preference relation, Time sequence weight, Technique for Order of Preference for Similarity Ideal Solution TOPSIS method, Variation coefficient, Vertical and horizontal method. Combination weighting methods include Additive synthesis, AHP and entropy method, Direct given and ordered weight, Entropy method and divergence measure method, Expert judgement and entropy method, Expert judgement and maximizing deviation method, Expert judgement and maximizing distance method, Expert judgement and statistical variance method, Multiplication synthesis, Optimal weighting based on minimum bias, Optimal weighting based on sum of squares (Wang et al. 2009; L. J. Zhang et al. 2020).”

Comments from author to reviewers:
-Reviewer 2

I am glad I have this opportunity to review your manuscript. Based on its title/abstract it looks very interesting. Anyway, my comments/notes are the follows:

Introduction. Stages of "assessing sustainable development using aggregated indices requires"... could be more described (specify what is the content of each stage by 1-2 sentences).

R// The following paragraph is added to describe each of the steps required for assessing sustainable development (lines 60-83)

“The conceptual framework development establishes the approach to analyze the sustainable development assessment, establishing the backgrounds, contexts, and relationships between the economic, social, environmental, and institutional dimensions. The indicator selection for sustainable development assessment should provide an estimate of the achievement of the SDGs, quantify both the determinants and the final impacts of sustainability, and influence the decision-making process with the information they provide. The indicator selection can be supported by several methods and should adopt the following principles: systematic, consistency, independency, measurability, comparability  (Wang et al. 2009; Rigo et al. 2020; Shao et al. 2020). Imputation of missing data recognizes the existence of variables that cannot be observed or measured, so an imperfect or approximate measurement can be achieved based on the valid values of other variables or based on a sample. It tends to be recurrent in sustainable development assessment at the subnational level due to the lack of structured and formal sources of information related to the SDGs. Data standardization normalizes the data obtained from the selected indicators so they can be used for quantitative calculations (Shao et al. 2020). Indicator weighting allows a trade-off between multiple indicators and a balance between different sustainable development perspectives, so it is necessary to select the weighting method to define or quantify the importance/relevance of the selected indicators (Rigo et al. 2020; Németh et al. 2019). The aggregation implies aggregating the normalized information of the selected indicators considering their weighting for calculating a total sustainable development score. Sensitivity analysis represents a validation method mainly performed by varying the indicator weights and indicators values to establish how stable the results are when changing the conditions considered in the sustainable development assessment (Shao et al. 2020; Rigo et al. 2020; Allen et al. 2020).”

Introduction. • three dimensions could be illustrated by any figure

R// Figure 1 is added to illustrate the three dimensions and four dimensions of sustainable development (line 46)

Figure 1. Three dimensions of sustainable development (a); four dimensions of sustainable development (b)

Introduction. • "For indicator weighting, several techniques..." - it is a poorly prepared overview of the methods used for criteria weighting (there are many other methods which could be also used). From my point of view, firstly should be these methods presented in general and after that, it could be more focused on one selected method (AHP),

R// We added two paragraphs dedicated to an overview of the methods used for criteria weighting (lines 84-115)

“For indicator weighting, weighting methods are classified into equal weighting and rank-order weighting (Shao et al. 2020; Si and Marjanovic-Halburd 2018). In equal weighting, indicator weights are equally assigned, which does not require stakeholder preferences; however, it ignores the relative importance of the criteria (Shao et al. 2020; Wang et al. 2009). In rank-order weighting methods, indicators weights are distributed and influenced by stakeholder perspectives (Si and Marjanovic-Halburd 2018). The rank-order weighting methods include subjective weighting method, objective weighting method and combination weighting method (Wang et al. 2009). Subjective weighting methods depend only on the preference of decision-makers, while objective weighting methods emphasize the statistical evaluation of the given data, complex calculation process and intensive data requirement (Shao et al. 2020). Combination weighting method is used to balance merits and limitations of objective and subjective weighting methods, but the process is complex and not widely-used (Si and Marjanovic-Halburd 2018).”

“Subjective weighting methods include Analytic Hierarchy Process (AHP), Analytic Network Process (ANP), Best Worst Method (BWM), Delphi method, DEMATEL, Direct given method, Eigenvector method, Expert judgement, Full consistency method - Step-Wise Weight Assessment Ratio Analysis (SWARA), Fuzzy preference programming, Least-square method, Quality function deployment (QFD), Simos method, Simple Multi-Attribute Rating Technique (SMART), Swing method. Objective weighting methods include Data envelopment analysis (DEA), Divergence measure method, Entropy method, Hesitant fuzzy entropy measure, Least mean square (LMS) method, Linear programming technique for multidimensional analysis of preference (LINMAP), Maximizing deviation method, Minmax deviation method, Multiple correlation coefficient, Ordered weight, Preference selection index, Principal component analysis (PCA), Relative preference relation, Time sequence weight, Technique for Order of Preference for Similarity Ideal Solution TOPSIS method, Variation coefficient, Vertical and horizontal method. Combination weighting methods include Additive synthesis, AHP and entropy method, Direct given and ordered weight, Entropy method and divergence measure method, Expert judgement and entropy method, Expert judgement and maximizing deviation method, Expert judgement and maximizing distance method, Expert judgement and statistical variance method, Multiplication synthesis, Optimal weighting based on minimum bias, Optimal weighting based on sum of squares (Wang et al., 2009; Zhang et al., 2020).”

Likewise, a paragraph is added to describe the functionality and practicality of the AHP for the weighting of indicators (lines 141-147):

“Moreover, the AHP is simple to understand and apply to complex issues using uses a 9-point scale to compare criteria relative importance. The AHP decomposes a large problem into smaller sub-problems at hierarchical levels, introduces the comparative importance of the criteria, showing a more reliable representation of the decision goal. The AHP is applicable for quantitative and qualitative criteria and checks the consistency of the decision, thus reducing the bias in the decision-making progression (Shao et al., 2020; Si & Marjanovic-Halburd, 2018).”

Indicator weighting for sustainable development. •         this is/could be a part of the methodology, not the theoretical section,

R // The name of the section “Indicator weighting for sustainable development” is changed to “Methodology” (line 180)

Indicator weighting for sustainable development. •         Figure 2 - correct "Consistency accepted" where "y" is on a new line,

R // The word “Consistency” is corrected to present it in a single line (line 202)

Indicator weighting for sustainable development. •         Table 1 - I expect more information about each criterion used (some base characteristics could be added, e.g. type of variable, source, some moment characteristics, ...),

R // A column with the description of each indicator is added to Table 1 (line 231)

Pairwise Comparisons •              I recommend changing the title of this section and do not divide next into short sub-sections (Normalized Matrix and Indicator Weighing Calculation, Consistency Test, ...). Is totally useless to have a section including one paragraph (Indicator Aggregation for SDG Weighing). They could be presented as steps of the calculation procedure, but for sure not as sub-sections.

R // The structure of the Methodology section is modified considering only two subsections: 2.1. Data and Indicator Description (line 204) and 2.2 AHP for indicator weighting (line 258)

Pairwise Comparisons •              you present your option (AHP, consistency test) as the only one possible option how to do it, there are many other approaches that are not considered or mentioned at least,

R// We added two paragraphs dedicated to an overview of the methods used for criteria weighting (lines 84-115)

“For indicator weighting, weighting methods are classified into equal weighting and rank-order weighting (Shao et al. 2020; Si and Marjanovic-Halburd 2018). In equal weighting, indicator weights are equally assigned, which does not require stakeholder preferences; however, it ignores the relative importance of the criteria (Shao et al. 2020; Wang et al. 2009). In rank-order weighting methods, indicators weights are distributed and influenced by stakeholder perspectives (Si and Marjanovic-Halburd 2018). The rank-order weighting methods include subjective weighting method, objective weighting method and combination weighting method (Wang et al. 2009). Subjective weighting methods depend only on the preference of decision-makers, while objective weighting methods emphasize the statistical evaluation of the given data, complex calculation process and intensive data requirement (Shao et al. 2020). Combination weighting method is used to balance merits and limitations of objective and subjective weighting methods, but the process is complex and not widely-used (Si and Marjanovic-Halburd 2018).”

“Subjective weighting methods include Analytic Hierarchy Process (AHP), Analytic Network Process (ANP), Best Worst Method (BWM), Delphi method, DEMATEL, Direct given method, Eigenvector method, Expert judgement, Full consistency method - Step-Wise Weight Assessment Ratio Analysis (SWARA), Fuzzy preference programming, Least-square method, Quality function deployment (QFD), Simos method, Simple Multi-Attribute Rating Technique (SMART), Swing method. Objective weighting methods include Data envelopment analysis (DEA), Divergence measure method, Entropy method, Hesitant fuzzy entropy measure, Least mean square (LMS) method, Linear programming technique for multidimensional analysis of preference (LINMAP), Maximizing deviation method, Minmax deviation method, Multiple correlation coefficient, Ordered weight, Preference selection index, Principal component analysis (PCA), Relative preference relation, Time sequence weight, Technique for Order of Preference for Similarity Ideal Solution TOPSIS method, Variation coefficient, Vertical and horizontal method. Combination weighting methods include Additive synthesis, AHP and entropy method, Direct given and ordered weight, Entropy method and divergence measure method, Expert judgement and entropy method, Expert judgement and maximizing deviation method, Expert judgement and maximizing distance method, Expert judgement and statistical variance method, Multiplication synthesis, Optimal weighting based on minimum bias, Optimal weighting based on sum of squares (Wang et al., 2009; Zhang et al., 2020).”

Likewise, a paragraph is added to describe the functionality and practicality of the AHP for the weighting of indicators (lines 141-147):

“Moreover, the AHP is simple to understand and apply to complex issues using uses a 9-point scale to compare criteria relative importance. The AHP decomposes a large problem into smaller sub-problems at hierarchical levels, introduces the comparative importance of the criteria, showing a more reliable representation of the decision goal. The AHP is applicable for quantitative and qualitative criteria and checks the consistency of the decision, thus reducing the bias in the decision-making progression (Shao et al., 2020; Si & Marjanovic-Halburd, 2018).”

Indicator weighting for sustainable development. •         this section should be marked as "Methodology" or "Material and Methods", clearly should be divided into sections focusing on the subject (region), data and criteria description, methods used.

R// The name of the section “Indicator weighting for sustainable development” is changed to “Methodology” (line 180). The structure of the Methodology section is modified considering only two subsections: 2.1. Data and Indicator Description (line 204) and 2.2 AHP for indicator weighting (line 258)

Results and Discussion •             Table 4 - without any description,

R // We improve and deepen the description of Table 4 (lines 314-331):

“As a result of the AHP method, Table 4 shows the matrix (A) of paired comparisons, which contains values from a 9-point scale to compare indicators' relative importance that corresponds to the original scale of preferences of Saatty.”

“These values reflect the importance of indicators defined by expert criteria by comparing elements of a row with elements of a column. For example, when comparing X₄ with X₂₀ (value of row 4 and column 20), a value of 9 was assigned, which means that the indicator X₄ (Maternal mortality rate) is of absolute importance concerning the indicator X₁₉ (Indebtedness index), then absolutely favoring the good health and well-being of pregnant women concerning the level of indebtedness of the municipalities, that is, prioritizing the social dimension over the economic and institutional dimension. Respectively, the value of the element in row 20 and column 4 corresponds to the inverse value of 9 (0,11).”

“Likewise, when comparing X₄ with X₂₃ (row 4 and column 23), a value of 1 was assigned, which means that the indicator X₄ (Maternal mortality rate) is equally important as the indicator X23 (Concentration of Particulate Material PM2.5) since air quality directly affects the health of citizens, specifically those who are at risk of respiratory diseases. Therefore, it is considered that both indicators contribute equally to the social dimension. Respectively, the element in row 23 and column 4 corresponds to the inverse of 1, which is also equal to 1.”

Results and Discussion •             Table 5 - totally useless (it is only an intermediate step that is ok, but it is not necessary to describe it by a table),

R// Table 5 is eliminated, and it is mentioned as a complement to the description of Table 4 how the normalized values are calculated (lines 334-336).

“Equation (1) is applied to the values of the matrix (A) of paired comparisons to obtain the Normalized Matrix, and the Indicator Weighing Calculation is obtained based on that matrix using Equation (2).”

Results and Discussion • you are not interpreting the results obtained at all, you just put some numbers there and advocate the use of the AHP method (which should be a part of the previous section),

R// Three paragraphs are added to interpret the results obtained (lines 355-378):

“The results of this work show that the sustainable development priorities for the municipalities of the Aburrá Valley should focus primarily on the health issue, defined by the CPS, since the indicators with the most considerable weight are related to SDG3, as they are the maternal mortality rate, mortality rate for HIV-AIDS, and the under-five mortality rate. The health issue also includes an indicator of SDG 11 related to concentrations of particulate matter PM2.5 since poor air quality constitutes a public health problem in the Aburrá Valley. This situation is relevant because this region is surrounded by mountains that do not allow adequate air circulation, concentrating particulate material inside the valley.”

“Moreover, this study prioritizes the issue of employment represented in the indicators that contribute to SDG₈. In this sense, the unemployment rate, underemployment rate, and dependency rate worsened due to the confinement generated by the Covid-19 pandemic; therefore, it will continue as a priority in the next quality of life surveys in this region. Similarly, the issue of education occupies the third place of preferences with the indicators related to SDG₄, such as the coverage of primary, secondary and higher education, and illiteracy rate indicators. The prioritization of this indicator indicates that governments of the municipalities of the Aburrá Valley must provide proper conditions for citizens to access education, especially higher education, to train citizens for the labor market that continually requires a more skilled workforce.”

“Accordingly, the indicators with the most considerable weight reflect the priorities of sustainable development in the Aburrá Valley corresponding to SDG₃ (Health and well-being), SDG₈ (Decent work and growth economic), and SDG₄ (Quality education). Therefore, development plans, programs, and the municipal budget of these municipalities should focus on improving health, education, and employment issues.”

Results and Discussion •             "This work was focused on indicator..." - the same problem, this should be a part of other section, real discussion is missing (you do not have any results interpretation to discuss),

R// The entire paragraph was moved and placed at the end of the Introduction section (lines 156-173)

Conclusions •   "However, the novelty of this study is providing an indicator weighting model that allows the assessment of sustainable development at the subnational level." - really? You just choose one method and applied it to one case study, this is novelty?

R// The novelty of this study is highlighted in the conclusions in a new paragraph added (lines 396-401):

“This study is novel because it allows prioritizing sustainable development dimensions through a methodology that articulates the technical approach derived from the 17 SDGs of the 2030 agenda (objective) with a participatory approach represented by the CPS (subjective) where citizens prioritize the most relevant life-quality issues. Moreover, a multi-criteria method based on the AHP supports the proposed methodology to lead political decision-making.”

Conclusions •   "Future research should focus..." - it should be a part of this manuscript, in the current version it looks like you just select a part of your research for publishing it separately.

R// The future research suggestion is modified in the conclusions, showing future lines of research that do not depend directly on the stages of our research (lines 404-409).

“Future works may implement the proposed methodology in regions like the Aburrá Valley to compare the sustainable development priorities and establish the factors generating variations in the assignment of preferences. Likewise, future studies should consider the uncertainty and vagueness in expert judgment through linguistic evaluations, gray-based methods, and fuzzy logic systems.”

Finally, we thank the reviewers for their suggestions and observations, which helped to improve the quality of the article, both in its form and content, ensuring that it is an article of high impact for the academic and scientific community.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Dear authors, 

thanks for the time you spent incorporating my comments, I really appreciate it. I consider each of them discussed and/or incorporated. Anyway, I recommend some other "light" changes":

• introduction - line 97 - we should recognize between methods of weight determination (SWARA, ...) and multi-criteria methods (TOPSIS, ...), please rewritten this section (see a comparison of some these methods to check the differences here:  https://www.worldscientific.com/doi/10.1142/S021962201950041X
• introduction - it is too long, so it would be useful to create a separate section "Theoretical background", the last paragraph of the introduction would be the same ("Therefore, this paper... see line 174),
• table 1/2 - the current version is too big, so I recommend two possibilities: a) replace it to the annex, b) create a shorter version in the text + add information that more comprehensive information could be found in the annex (it should be also replaced) 

Author Response

Dear Evay Yuan                                                                                             Economics
MDPI

We communicate with you through this letter in order to outline every change made in the article economies-1394433 “Application of AHP for the Weighting of Sustainable Development Indicators at the Subnational Level” based on the observations and suggestions provided by the Reviewer 2. All changes and modifications made to the manuscript are highlighted in blue in the revised version.

Comments from author to reviewers:
-Reviewer 2

Introduction - line 97 - we should recognize between methods of weight determination (SWARA, ...) and multi-criteria methods (TOPSIS, ...), please rewritten this section (see a comparison of some these methods to check the differences here: https://www.worldscientific.com/doi/10.1142/S021962201950041X.

R// The article suggested by the reviewer was analyzed and the paragraph was modified to state the methods of weight determination (lines 126-134)

“Subjective weighting methods include Analytic Hierarchy Process (AHP), Delphi method, Pairwise comparison, Ranking method, SIMOS method, Simple Multi-Attribute Rating Technique (SMART), Step-Wise Weight Assessment Ratio Analysis (SWARA), Swing Weighting (SW) (Vavrek 2019; Wang et al. 2009). Objective weighting methods include Criteria Importance Through Intercriteria Correlation (CRITIC), Entropy method, Mean Weight (MW), Standard Deviation (SD), Coefficient of Variance (CV), Statistical Variance Procedure (SVP), Integrated Determination of Objective Criteria Weights (IDOCRIW), Criterion Impact Loss (CILOS), Principal component analysis (PCA) (Vavrek 2019; Kumar et al. 2009). “

In this way, the works of (Vavrek 2019), (Wang et al. 2009) and (Kumar et al. 2009) were considered in the paper.

• Kumar, Rajesh, H R Murty, S K Gupta, and A K Dikshit. 2009. “An Overview of Sustainability Assessment Methodologies.” Ecological Indicators 9: 189–212. https://0-doi-org.brum.beds.ac.uk/10.1016/j.ecolind.2008.05.011.

• Vavrek, Roman. 2019. “Evaluation of the Impact of Selected Weighting Methods on the Results of the TOPSIS Technique.” International Journal of Information Technology & Decision Making 18 (6): 1821–43. https://0-doi-org.brum.beds.ac.uk/10.1142/S021962201950041X.

• Wang, Jiang Jiang, You Yin Jing, Chun Fa Zhang, and Jun Hong Zhao. 2009. “Review on Multi-Criteria Decision Analysis Aid in Sustainable Energy Decision-Making.” Renewable and Sustainable Energy Reviews 13 (9): 2263–78. https://0-doi-org.brum.beds.ac.uk/10.1016/j.rser.2009.06.021.

Introduction - it is too long, so it would be useful to create a separate section "Theoretical background", the last paragraph of the introduction would be the same ("Therefore, this paper... see line 174),

R// We create a separate section “2. Theoretical Background" in which the stages for sustainable development assessment (line 83) are addressed, then providing a shorter Introduction.

Table 1/2 - the current version is too big, so I recommend two possibilities: a) replace it to the annex, b) create a shorter version in the text + add information that more comprehensive information could be found in the annex (it should be also replaced) 

R // Appendix A (line 407) is created to show there in Table A1 and Table A2 the detailed information presented in Table 1 and Table 2.

Appendix A

Table A1. Description of indicators for assessing sustainable development.

Table A2. Relationship of indicators with SDGs

Finally, we thank the reviewers for their suggestions and observations, which helped to improve the quality of the article, both in its form and content, ensuring that it is an article of high impact for the academic and scientific community.

Author Response File: Author Response.pdf