A Review of Current Evaluation Urban Sustainability Indicator Frameworks and a Proposal for Improvement

Round 1

Reviewer 1 Report

The paper discusses how to measure sustainability and livability in cities and communities, with special focus on the Tools4Cities as the Next Generation Cities Institute Virtual Twin Community toolset. 

Rather the paper requires thorough revision and restructuring to avoid confusion to the readers. Please find below my comments in addition to others added to the soft copy attached.

Please add more information concerning the review approach, method, limits, tools, databases,.etc.

Suggestion to refer to the PRISMA chart for a systemic review process

Suggestion to pinpoint the interplay of environmental assessment methods; pointing out how different scales and scopes of analysis can work together for a better comprehensive tool

Point out of the aim focuses on environmental sustainability, or other sustainability aspects including social, community and economic…etc. This is somehow contradictory in different parts of the manuscript

Make sure to include the full term of all abbreviations once introduced, e.g. KPI…etc.

The structure and organization of the paper needs careful revision

Fig 1, 2, 3, 5 do not tell much information, please revise or delete, instead fig 2, 3, 5 can be replaced with graphs indicating the occurrence number of keywords which indicate their significance

Please revise the way intext citations are referred to without disrupting the language expression of the sentences, e.g. ‘Making cities sustainable has been a clear goal from [3] and the implementation of…etc.’

Please revise references, some are incomplete or old, missing important studies related to environmental assessment frameworks

List of abbreviations should be moved before the introduction section

Figure 4 showing the Clustering analysis is very interesting, please explain what tool/method did authors use to develop this analysis, and please better describe the information it tells

Some title seem awkward e.g. 3.3.5 is entitles ‘next steps’ please revise. Also, 4.1. Skin in the game[7]: against everything that does not work in current methodologies for evaluation.

Missing result section

Move 4.1.1., 4.1.2. to the method section

Section 4 Discussion, it is not clear what are authors discussing, what are the result to be discussed and how do they compare to previous studies

Method

I suggest authors can organize the research method in the following points: Pinpointing relevant frameworks and indicators. Selecting relevant indicators according to a number of factors (that have to be stated by the authors and discussed later in the discussion section), and their relative importance weight, and proposing tools and methods for measuring these indicators

Digital twin, data science have been mentioned but without pointing out the link to the current study, please revise

Many subsections for 4.1 and 4.2, please revise carefully as this is very confusing to follow

Fig. 10, 12, 13 are not clear, very small text size

Too many figures, please consider keeping only those relevant and othwise delete. Keep only those which support your scientific writing, please do not use them for presentation.

It is not usual to include figures in the conclusion section. Conclusions should relate to the study aims and objectives , show its novelty

Comments for author File: Comments.pdf

Moderate revision required. 

Author Response

Thanks for your inputs. They were precious and helped us improve our document. It has significantly changed.

Going individually into the mentioned points:

• Please add more information concerning the review approach, method, limits, tools, databases,.etc. DONE
• Suggestion to refer to the PRISMA chart for a systemic review process. Not concretely done, but explained much more the sources of the review process in the document
• Suggestion to pinpoint the interplay of environmental assessment methods; pointing out how different scales and scopes of analysis can work together for a better comprehensive tool. DONE, THANKS
• Point out of the aim focuses on environmental sustainability, or other sustainability aspects including social, community and economic…etc. This is somehow contradictory in different parts of the manuscript. DONE, THANKS
• Make sure to include the full term of all abbreviations once introduced, e.g. KPI…etc. DONE
• The structure and organization of the paper needs careful revision. It has only been slightly restructured, but explanations have been added to justify the different points
• Fig 1, 2, 3, 5 do not tell much information, please revise or delete, instead fig 2, 3, 5 can be replaced with graphs indicating the occurrence number of keywords which indicate their significance. DONE
• Please revise the way intext citations are referred to without disrupting the language expression of the sentences, e.g. ‘Making cities sustainable has been a clear goal from [3] and the implementation of…etc.’ DONE.
• Please revise references, some are incomplete or old, missing important studies related to environmental assessment frameworks. An important number of references have been added
• List of abbreviations should be moved before the introduction section. in the latex template, it was in the end. I will comment it with the editor.
• Figure 4 showing the Clustering analysis is very interesting, please explain what tool/method did authors use to develop this analysis, and please better describe the information it tells. DONE and deepened.
• Some title seem awkward e.g. 3.3.5 is entitles ‘next steps’ please revise. Also, 4.1. Skin in the game[7]: against everything that does not work in current methodologies for evaluation. DONE
• Missing result section. it is a discussion paper, no results foreseen..
• Move 4.1.1., 4.1.2. to the method section. DONE
• Section 4 Discussion, it is not clear what are authors discussing, what are the result to be discussed and how do they compare to previous studies. We tried to clarify it more
• I suggest authors can organize the research method in the following points: Pinpointing relevant frameworks and indicators. Selecting relevant indicators according to a number of factors (that have to be stated by the authors and discussed later in the discussion section), and their relative importance weight, and proposing tools and methods for measuring these indicators. addressed partially, although an effort to propose kpis and methodologies has been done and is clearer (we hope)
• Digital twin, data science have been mentioned, but without pointing out the link to the current study, please revise. ok, some writing done to clarify
• Many subsections for 4.1 and 4.2, please revise carefully as this is very confusing to follow. tried to link better the different topics. 
• Fig. 10, 12, 13 are not clear, very small text size. ok, the unnecessary figures have been deleted.

Author Response File: Author Response.pdf

Reviewer 2 Report

1. The main points and logical structure of the paper are clearly stated in the abstract. However, in the Discussion section, the corresponding relationship between problems and countermeasures is not well dealt with, which should be corrected.

2. The expression of the paper is not concise enough, which causes the logical structure of the whole paper to be unclear. Recommending further appropriate deletions and modifications。

Author Response

Thanks for the comments. A thorough analysis and cleaning of the document has been done to try to solve the points you made.

1. The main points and logical structure of the paper are clearly stated in the abstract. However, in the Discussion section, the corresponding relationship between problems and countermeasures is not well dealt with, which should be corrected. A deeper restructuring of the article has been done to try to solve the pinpointed problems
   
   2. The expression of the paper is not concise enough, which causes the logical structure of the whole paper to be unclear. Recommending further appropriate deletions and modifications。We tried to be clearer and more concise. Thanks

Author Response File: Author Response.pdf

Reviewer 3 Report

The paper discusses the concepts, methods and targets of current frameworks monitoring urban sustainability and highlight their common characteristics, limits and weaknesses as well as potential methodologies for improving the processes of urban sustainability evaluation and presents the development of a new tool towards this purpose. The subject of sustainability assessnment in urban communities is of high interest, the approach is original and the research is well structured and adequately referenced. The analysis is extensive and elaborate and the conclusions summarize the research results. The questions raised and the issues explored in the paper regarding urban sustainability frameworks’ improvement are critical but although there is a wide range of frameworks presented and analyzed, the actual sustainability indicators considered for the analysis are presented vaguely as general clustered groups. Maybe a clear list of indicators would be useful for the better comprehension of the proposed improvement. Also a more explicit description of the suggested tool in terms of its function and the expected results would probably add clarity and assist readers’ understanding.

Some more specific comments are noted below.

Line 77. Please define KPI

Lines 104-105. Please consider if the 24 word clouds could be presented in a graph or table.

Line 116. There is no Annex 1 in the paper. Please correct the reference appropriately.

Figure 2. It is not clear how the different clusters in the graph are distinguished, please add labels to indicate each group. 

Line 157. Please indicate which are the most critical indicators. Perhaps a list could be useful.

Figure 3. Please add explanations in the figure legend.

Lines 304-307. This statement is not very clear. Please rephrase or explain.

Line 308. Please clarify if the [15]’s graphs are those in figure 4.

Figure 4. The text in the graphs is not legible. Please increase font size. Also please add explanations in the figure legend.

Figure 7. Please increase figure size to allow for legible text

Figure 9. Please increase figure size to allow for legible text

Figure 10. Please increase figure size to allow for legible text

Lines 563-565. Even if no unique solution is consider optimal, an example would be useful.

Line 569. Please indicate the necessary criteria.

Figure 11. Please increase font size.

Line 617. In the abbreviations section there is a different explanation of NGCI compared to the one given in the Introduction (line 16). Please correct appropriately.

Author Response

Thank you very much for your comments. They have been helpful to improve the paper. Going into your comments

• Line 77. Please define KPI. DONE, THANKS
• Lines 104-105. Please consider if the 24 word clouds could be presented in a graph or table. Changed, thanks.
• Line 116. There is no Annex 1 in the paper. Please correct the reference appropriately. Changed, thanks.
• Figure 2. It is not clear how the different clusters in the graph are distinguished, please add labels to indicate each group. Labels added.
• Line 157. Please indicate which are the most critical indicators. Perhaps a list could be useful. Done.
• Figure 3. Please add explanations in the figure legend. Deleted.
• Lines 304-307. This statement is not very clear. Please rephrase or explain. Rephrased
• Line 308. Please clarify if the [15]’s graphs are those in figure 4. Rephrased
• Figure 4. The text in the graphs is not legible. Please increase font size. Also please add explanations in the figure legend. Deleted
• Figure 7. Please increase figure size to allow for legible text. Coming from other reference, not possible
• Figure 9. Please increase figure size to allow for legible text
• Figure 10. Please increase figure size to allow for legible text
• Lines 563-565. Even if no unique solution is consider optimal, an example would be useful. Example given.
• Line 569. Please indicate the necessary criteria. Indicated.
• Figure 11. Please increase font size. Deleted
• Line 617. In the abbreviations section there is a different explanation of NGCI compared to the one given in the Introduction (line 16). Please correct appropriately. Perfect, changed, thanks.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Some figures can be deleted e.g. fig 1, 5, 6, 7, 8

Figure 6 in particular is not authors’ elaboration, with very unclear text, so why add it

Figure 9, text is not clear, please revise

Some sections can be deleted e.g. section 3.1 as it is very well-known

Section 3.2 some paragraphs lack references, please revise

The manuscript requires further revision. There are some languages mistakes. Also in section 3.3.2., 3.3.3. the authors state ‘The result in ??’, also in section 4.1.3., (The proposed data schema from 5) please revise thoroughly before submitting

The method is not properly described

Section 3.3.2. , 3.3.3. can be moved to the result section

Authors mention (chapter), please revise, I believe it should be replaced with (section)

Please revise the naming of some titles/subtitles e.g. title 3.4

Many subtitles, please revise as this is very confusing. As mentioned before, the structure and organization of the paper needs careful revision. Please use the standard scientific writing (introduction, literature review, method, result, discussion, conclusion).

Please add an additional column for table 5 to state the references used for each indicator mentioned (selected)

Please revise the use of abbreviations and full terms

Please revise the writing style, it is very wordy so please stick to the scientific standard of writing and organization

Moderate revision required

Author Response

Thanks again for the excellent feedback. Answering each of the comments>

Some figures can be deleted e.g. fig 1, 5, 6, 7, 8

Most of them were deleted. Only kept Colouring Cities, we think it is important to understand the importance of visualization in platforms.

Figure 6 in particular is not authors’ elaboration, with very unclear text, so why add it

Ok, deleted.

Figure 9, text is not clear, please revise

Deleted.

Some sections can be deleted e.g. section 3.1 as it is very well-known.

Ok, reduced to the minimum essential text, to be linked to the rest of comments.

Section 3.2 some paragraphs lack references, please revise

References added.

The manuscript requires further revision. There are some languages mistakes. Also in section 3.3.2., 3.3.3. the authors state ‘The result in ??’, also in section 4.1.3., (The proposed data schema from 5) please revise thoroughly before submitting

Revised thoroughly. Thanks

The method is not properly described

Structure changed, and method described

Section 3.3.2. , 3.3.3. can be moved to the result section

Added to result section.

Authors mention (chapter), please revise, I believe it should be replaced with (section)

Changed naming.

Please revise the naming of some titles/subtitles e.g. title 3.4

Changed the full structure

Many subtitles, please revise as this is very confusing. As mentioned before, the structure and organization of the paper needs careful revision. Please use the standard scientific writing (introduction, literature review, method, result, discussion, conclusion).

Changed the full structure of the article, to follow the standard scientific writing and added numbering to gaps and proposals for solution of each of the gaps.

Please add an additional column for Table 5 to state the references used for each indicator mentioned (selected)

Column added

Please revise the use of abbreviations and full terms

Revised

Please revise the writing style, it is very wordy so please stick to the scientific standard of writing and organization

We tried to summarize the wording, although we feel the wording is necessary in the gaps and proposals area to explain our point of view.

Author Response File: Author Response.pdf

Round 3

Reviewer 1 Report

A figure can be added to explain the methodological steps.

The result section should be better structured. Currently, it is very long which makes it confusing to follow

 Moderate editing of English language required

Author Response

Thanks for the comments. I have tried to reply to each of your points.

A figure can be added to explain the methodological steps.

Thanks for the suggestion; we have implemented it. A new figure shows the methodological steps.

The result section should be better structured. Currently, it is very long which makes it confusing to follow

I understand the concern, but after changing the full structure of the doc, and adding the figure you suggested, I hope it can be clearer. Still, it is a bit long, but I hope it is easier to follow with the guide of the image

Moderate editing of English language required

Paragraphs in the first half were rewritten by one of the authors, from the UK, to make it clearer and with better English.

Author Response File: Author Response.pdf