Round 1

Reviewer 1 Report

Review for: Zitian Guo et al. "Accuracy Assessment of the FROM-GLC30 Land 2 Cover Dataset Based on Watershed Sampling Units: a 3 Continental Scale Study"

The paper makes use of a very comprehensive and detailed database of recent ground and satellite land assessment information to analyse the accuracy of a Chinese satellite data product for Global Land Cover at 30m resolution across the Pan 3^rd Pole area. The assessment and the ground survey data base are of great interest and the results will be useful for users of global land cover maps in general and users of the FROM-GLC30 Land Cover Dataset in particular. It is also of special interest for many applications in the Pan 3^rd Pole region.

The basic analysis is sound and very useful. However, it should only be published when it has made the methods (particularly for aggregation) it uses clear to readers and added some extra support for their decisions. These more critical modifications are listed below and then some detailed and minor points are suggested for improvement.

1. Data source

The data source being analysed is a GLC map – GLC meaning Global Land Cover. However, the various GLC maps described are not all from the same group and not all done in the same way. The authors should first discuss GLC maps in general (eg in 2.2.2) but make it clear that they are analysing just one example, the FROM-GLC30 data. It should be made clear that this GLC is a Chinese product generated by Qinghua University and made available at the web address given. I suggest the authors create a reference and provide information such as:

Host website: http://www.geodata.cn

DOI: https://0-doi-org.brum.beds.ac.uk/10.12041/geodata.25725976218589.ver1.db

Contact: [email protected]

The web site does not describe the methodology in detail. If it is not fully described in [17] and [18] the authors should seek an additional reference where it is. The map producers say it has been tested and has overall accuracy of 72.35%. This is very similar to the result of the present paper. Perhaps that can be mentioned in the paper? The big advantage of the current paper is (of course) in the more detailed overall accuracies by region as well as user and producer accuracies which are very important to know.

2. Methodology – important addition

It is very important for the authors to add some information not currently provided. It is not clear in the present paper how the aggregated accuracy data are generated. Section 2.3 is the key section to explain what is done. Equations (1), (2) and (3) arise from an error matrix (or Confusion Matrix). That is, a 7 by 7 matrix with areas (or pixels but better as area) of a primary sampling unit (PSU or SU) where the data class is “i” and the reference class is “j” in the (i,j) cells. This matrix result can be summarised by the overall accuracy (OA), the user accuracy (UA) and the producer accuracy (PA).

In standard methodology, to obtain aggregate statistics for a region or the whole map, the SU error matrices are summed to get aggregate error matrices for aggregate areas. These will also have overall OA, UA and PA statistics defined by Eq. (1), (2) and (3). But it does not seem this is done in the paper. My guess is that the OA, UA and PA statistics for regions and the whole map in the paper are not obtained by aggregating error matrices but by averaging the SU values of OA, UA and PA over regions or over all of the SUs to represent the whole map. But I cannot find this information clearly stated in the paper. I also cannot find what kind of average (e.g. mean, median, harmonic mean etc) is used.

The authors must state what is done clearly and indicate why they are not aggregating error matrices as the error matrices are the standard way to do this. Will the results be different from the current results if error matrices are aggregated? Will one way be better or are both similar? It at least needs some discussion. What the authors are doing is quite acceptable but they must tell the reader wat they are doing and why they are doing it.

I think what has been done should be stated clearly in section 2.2 and in some other places to help readers. The other places are:

1. In section 3.2, the first statement is that the OA of the GLC map is 73.49%. Presumably this is an average of OA values over all SUs. But this is out of place. I suggest this first sentence be moved to section 3.3;
2. Figure 6 shows a box-plot of the OAs for SUs by region. Some readers may not know what box-plots are so more explanation is needed. But assuming everyone understands, the upper and lower quantile values are not stated (what %?). Fig. 6 is useful but it may also be useful to indicate the average values by region as well as the median (unless the median is what is used);
3. Figure 7 shows the spatial distribution of OA by SU but it does not explain how the SU data are spread out to cover all the areas. Is this an interpolation? If so, how is it done? It is not clear.
4. In section 3.3 explain clearly how Table 2 is calculated. Presumably it is the average again. The overall OA (73.49%)) belongs here and not in section 3.2. It may be useful to compare it with claimed accuracy on the web site.
5. In Figure 8 (section 3.3), the 7 average UA and PA values are illustrated for the 8 regions. Please state in the text how they were averaged so that the reader understands. The Figure is currently very hard to understand. The letters on top of the bars are not explained. If they are explained it will also help.

Detailed and minor suggestions (In addition to the above) to improve expressions:

1. Page 1, Lines 19 and 43: “widely concerned” better is “of great concern”
2. Page 1 Line 43: “since it could” should be more positive “since it can”
3. Page 2, Lines 53, 54: “… which makes the obtain of reference dataset more conveniently and efficiently” should be “… which makes the use of a reference dataset more convenient and efficient”
4. Page 2, Line 54: What does “smallest expression unit” mean? Explain.
5. Page 2, Lines 59, 60: What does “data phase problem” mean? Please explain.
6. Page 2 Line 65: “widely concerned” better is “of great concern”.
7. Page 2, Line 70: “short release time” should be “short time since release”.
8. Page 2, Line 71: “It is an urgent need…” is better written “There is an urgent need…”.
9. Page 2, Line 78: “without reference data [21, 22], while the method based on small” should become “without reference data [21, 22]. The method based on small” (sentences too long).
10. Page 2, Line 79: “since the obtain of reference dataset is difficult.” Should be “since it is difficult to obtain a reference data set.”
11. Page 2, Line 80: “short release time” should be “short time since release”.
12. Page 2, Line2 82, 83: “method at the continental scale, which is one of the limitations in its application.” Should become “method at the continental scale. This is one of the limitations to its application.” (sentence too long).
13. Page 3, Line 108: Why use UTM? It seems there should be a more convenient and better projection for this project! Too many zone boundaries. Is there a specific reason for UTM? The reader may also ask this question.
14. Page 3, Line 111: “unequal probability” is better as “variable probability”. Variable Probability Sampling is VPS.
15. Page 3, Line 124: “In other 22%” better as “In the other 22%”.
16. Page 4, Line 131: “Generally speaking, the reference…” should just be “The reference…”. (No need for extra words like this).
17. Page 4, Line 133: Explain “Majority” in more detail.
18. Page 4, Line 147: “Existing research findings…” change to “Previous research findings…”.
19. Page 5, Line 163: “high-resolution satellite data of China” should be “high-resolution Chinese satellite data”
20. Page 5, Line164: “..data, and global night…” add Modis as “..data, MOD13Q1 (NDVI) and global night…”
21. Page 6, Lines 190-192: It is not clear where the areas come from. Are they from the SUs (sampling units)? If so you need to say this. Total area in each case is then total SU area.
22. Page 9, Lines 243, 244: “…it could be found that the of absolute value the difference” should be “…it can be found that the absolute value of the difference”
23. Page 10, Lines 261, 262: “in West Asia - Northeast Africa. was 98%, only 1.2% in Central and Eastern Europe.” Should be “in West Asia - Northeast Africa was 98%, and only 1.2% in Central and Eastern Europe.”
24. Page 10, Lines 263, 264: “China and West Asia - Northeast Africa., about 55%, while was lowest in Southeast Asia, about 40%.” Better is “China and West Asia - Northeast Africa., at about 55%, while it was lowest in Southeast Asia, at about 40%.”
25. Page 11, Lines 282,283: “In the related land cover data accuracy assessment researches” is better written “In previous research on land cover data accuracy assessment”
26. Page 11, Lines 282, 283: Can the authors provide references to these papers? If so, please add them.
27. Page 11, Lines 285, 286: This is a very interesting theory. Can the authors provide a reference to it? It will help the discussion.
28. Page 13, Line 348: “at the same time” is strange, do the authors mean “in the same place”?
29. Page 13, Lines 349-352. The problem of separating shrubland is common to all GLC maps! Grassland, Shrubland and Forest form a continuous gradient. But it may not matter when the classes are being used. Main thing to differentiate is fraction of cover. The Forest and Grassland growth forms should be able to be separated but shrubland is very hard.
30. Page 15. The authors should consider to additionally conclude that the results of the paper will be helpful for applications in different locations of the Pan 3^rd Pole as OA, UA and PA vary with location as well as land covers of interest. This is a very useful outcome of the paper.

Summary

The paper is basically sound and will be very interesting to people engaged in land cover based research over the large area covered. It is also a useful paper on accuracy assessment methodology. It should be published. In order to help readers understand what was done the authors must first add the information outlined at the start of this review. The more detailed suggestions by line have been made to help clarify the expressions.

Comments for author File: Comments.pdf

Author Response

Dear reviewers and editors,

Thank you very much for your comments to our manuscript, entitled ‘Accuracy Assessment of the FROM-GLC30 Land Cover Dataset Based on Watershed Sampling Units: a Continental Scale Study’, with the reference number of sustainability-942493. We have revised the manuscript according to the comments. Grammar and spelling have also been check by a native English speaker.

Following is the one by one reply to the reviewers’ comments. The Line numbers in this document refers to the revised line number with all the tracking displayed.

We highly appreciate your carefulness and conscientious suggestions, and your broad knowledge, which helped us improve the manuscript a lot.

Wish you all the best!

Sincerely yours,

Zitian Guo, Chunmei Wang*, Xin Liu, Guowei Pang, Mengyang Zhu, Lihua Yang

2020-09-29

Responses to reviewer #1’s comments

Comment 1: The paper makes use of a very comprehensive and detailed database of recent ground and satellite land assessment information to analyse the accuracy of a Chinese satellite data product for Global Land Cover at 30m resolution across the Pan 3rd Pole area. The assessment and the ground survey data base are of great interest and the results will be useful for users of global land cover maps in general and users of the FROM-GLC30 Land Cover Dataset in particular. It is also of special interest for many applications in the Pan 3rd Pole region.

The basic analysis is sound and very useful. However, it should only be published when it has made the methods (particularly for aggregation) it uses clear to readers and added some extra support for their decisions. These more critical modifications are listed below and then some detailed and minor points are suggested for improvement.

Response: Thank you for reviewer’s comments on the importance of this work. We have revised our manuscript according to your comments, one by one. And we believe it is now better for readers to understand and benefit.

Comment 2: Data source. The data source being analysed is a GLC map – GLC meaning Global Land Cover. However, the various GLC maps described are not all from the same group and not all done in the same way. The authors should first discuss GLC maps in general (eg in 2.2.2) but make it clear that they are analysing just one example, the FROM-GLC30 data. It should be made clear that this GLC is a Chinese product generated by Qinghua University and made available at the web address given. I suggest the authors create a reference and provide information such as:

Host website: http://www.geodata.cn

DOI: https://0-doi-org.brum.beds.ac.uk/10.12041/geodata.25725976218589.ver1.db

Contact: [email protected]

Response: Thank you for your suggestion. The description of GLC has been added in section 2.2.2. (Line 335-339) We have also added a reference ([32]) to the data site.

Comment 3: The web site does not describe the methodology in detail. If it is not fully described in [17] and [18] the authors should seek an additional reference where it is. The map producers say it has been tested and has overall accuracy of 72.35%. This is very similar to the result of the present paper. Perhaps that can be mentioned in the paper? The big advantage of the current paper is (of course) in the more detailed overall accuracies by region as well as user and producer accuracies which are very important to know.

Response: Thank you for your suggestion. The discussion of the OA values obtained in this manuscript and provided by the map producer has been added in Section 4.1 (Line 746-748).

Comment 4:  Methodology – important addition

In standard methodology, to obtain aggregate statistics for a region or the whole map, the SU error matrices are summed to get aggregate error matrices for aggregate areas. These will also have overall OA, UA and PA statistics defined by Eq. (1), (2) and (3). But it does not seem this is done in the paper. My guess is that the OA, UA and PA statistics for regions and the whole map in the paper are not obtained by aggregating error matrices but by averaging the SU values of OA, UA and PA over regions or over all of the SUs to represent the whole map. But I cannot find this information clearly stated in the paper. I also cannot find what kind of average (e.g. mean, median, harmonic mean etc) is used.

The authors must state what is done clearly and indicate why they are not aggregating error matrices as the error matrices are the standard way to do this. Will the results be different from the current results if error matrices are aggregated? Will one way be better or are both similar? It at least needs some discussion. What the authors are doing is quite acceptable but they must tell the reader wat they are doing and why they are doing it.

Response: This is quite interesting; we appreciate the method the reviewer mentioned. In the original submitted version of our manuscript, it was done by “obtain aggregate statistics for a region or the whole map, the SU error matrices are summed to get aggregate error matrices for aggregate areas” (METHOD1), which is exactly what reviewer said is normally used. But since reviewer mentioned another idea of method “averaging the SU values of OA, UA and PA over regions or over all of the SUs to represent the whole map” (METHOD2), we reconsidered and discussed, we believe METHOD2 for OA is exactly how we should do, and METHOD1 which is the way how we did is what we should do for PA and UA in a certain region.

The reason for this is that we were using not totally same sizes of SUs (ranges from 0.2-3km²), but we do hypnotized each SU would have same representing area. So METHOD2 will help us to make each sampling unit has equal importance in regional overall accuracy calculation as expected. While for PU and SU, since in some SUs there is only quite a small number or no pixels with a certain land cover type, which would make the results biased by using METHOD2.

We revised the values all over the manuscript. The values actually did not differ too much between METHOD1 and METHOD2 for OA after revise. That is because our SUs mostly have areas near 1km². See the following Table1.

Table 1 OA values by using METHOD1 and METHOD2

We believe by using METHOD2, the regional OA is more reasonable. We have made a clear state of this method in 2.3.2 (Line 416-437) as following:

“In this study, the overall accuracy was calculated at each sampling unit. Then the average values of sampling units within each region was calculated as the overall accuracy values of that region, which means each sampling unit has equal importance in the regional overall accuracy calculation. User's accuracy (UA) and producer's accuracy (PA) were calculated for each land cover type by summarizing all the SUs pixels in regions or the whole study area. By doing this, we could have equally importance for each pixel in calculating UA and PA within a certain domain, which also fits our expectations most, because in some SUs there is only quite a small number of pixels or no pixel with a certain land cover type”.

Comment 5: In section 3.2, the first statement is that the OA of the GLC map is 73.49%. Presumably this is an average of OA values over all SUs. But this is out of place. I suggest this first sentence be moved to section 3.3;

Response: Thank you for your suggestion. We discussed with author team, and now we prefer to stay it here. Section 3.2 mainly shows the calculation results of overall accuracy, involving the overall accuracy analysis of whole study area and the eight regions. This sentence together with Fig.6 is telling the story of OA in the whole area. Section 3.3 is mainly about the accuracy for different land cover types.

In order to make this part easier for understanding, we have revised the title of Section 3.2 from "Regional Differences in Overall Accuracy" to "Overall Accuracy in Different Regions". In addition, the overall accuracy is an average of overall accuracy values over all sampling units. (Line 478)

Let us know if you have further thinking on this.

Comment 6:  Figure 6 shows a box-plot of the OAs for SUs by region. Some readers may not know what box-plots are so more explanation is needed. But assuming everyone understands, the upper and lower quantile values are not stated (what %?). Fig. 6 is useful but it may also be useful to indicate the average values by region as well as the median (unless the median is what is used);

Response: Thank you for your suggestion. It has already been revised in the manuscript. The box-plot presents the standard five-number elements, and we have added the description of them. (Line 486-488)

In addition, we have added a table (Current Table 2) below Fig.7 to show the mean, median and standard deviation values of OA in each region and the whole area, which would help reader to obtain more useful information easily. (Line 523-534)

Comment 7: Figure 7 shows the spatial distribution of OA by SU but it does not explain how the SU data are spread out to cover all the areas. Is this an interpolation? If so, how is it done? It is not clear.

Response: Figure 7 (Current Figure 8) is not the interpolation result of the overall accuracy of each sampling unit, but is just a display of point data (the point is the geometric center of each SU). (Line 540-541)

Comment 8: In section 3.3 explain clearly how Table 2 is calculated. Presumably it is the average again. The overall OA (73.49%)) belongs here and not in section 3.2. It may be useful to compare it with claimed accuracy on the web site.

Response: Thank you for your suggestion. Table 2 (Current Table 3) was calculated by all the pixels of a certain land type. See response to Comment 4. The overall OA was still in 3.2 and has been compared with previous research; please see response to Comment 5.

Comment 9: In Figure 8 (section 3.3), the 7 average UA and PA values are illustrated for the 8 regions. Please state in the text how they were averaged so that the reader understands. The Figure is currently very hard to understand. The letters on top of the bars are not explained. If they are explained it will also help.

Response: According to your comment, we have revised Figure 8 into Table 4 which offers information of the user's accuracy and the producer's accuracy and F₁ for different land cover types in eight regions. And have added text to show how we calculated briefly in 2.3.2 as following: (Line 433-437)

“User's accuracy (UA) and producer's accuracy (PA) were calculated for each land cover type by summarizing all the SUs pixels in regions or the whole study area. By doing this, we could have equally importance for each pixel in calculating UA and PA within a certain domain, which also fits our expectations most, because in some SUs there is only quite a small number of pixels or no pixel with a certain land cover type”.

Comment 10: detailed and minor suggestions:

1. Page 1, Lines 19 and 43: “widely concerned” better is “of great concern”.

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 19 and 54)

2. Page 1 Line 43: “since it could” should be more positive “since it can”.

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 54)

3. Page 2, Lines 53, 54: “… which makes the obtain of reference dataset more conveniently and efficiently” should be “… which makes the use of a reference dataset more convenient and efficient”.

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 64)

4. Page 2, Line 54: What does “smallest expression unit” mean? Explain.

Response: It has already been modified in the manuscript. What we hope to express here is that a watershed is regarded as a whole geographically with the similar regional characteristics of climate, hydrology, soil, vegetation and so on with other watershed around it. (Line 64-66)

5. Page 2, Lines 59, 60: What does “data phase problem” mean? Please explain.

Response: It has already been revised from “data phase problem” to “problems related to the impacts of different seasons”. (Line 71)

6. Page 2 Line 65: “widely concerned” better is “of great concern”.

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 77)

7. Page 2, Line 70: “short release time” should be “short time since release”.

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 81)

8. Page 2, Line 71: “It is an urgent need…” is better written “There is an urgent need…”.

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 82-83)

9. Page 2, Line 78: “without reference data [21, 22], while the method based on small” should become “without reference data [21, 22]. The method based on small” (sentences too long).

Response: It has already been modified. The original long sentence has been split into two short sentences. Thank you for your suggestion. (Line 89)

10. Page 2, Line 79: “since the obtain of reference dataset is difficult.” Should be “since it is difficult to obtain a reference dataset.”

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 90)

11. Page 2, Line 80: “short release time” should be “short time since release”.

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 91)

12. Page 2, Line2 82, 83: “method at the continental scale, which is one of the limitations in its application.” Should become “method at the continental scale. This is one of the limitations to its application.” (sentence too long).

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 93)

13. Page 3, Line 108: Why use UTM? It seems there should be a more convenient and better projection for this project! Too many zone boundaries. Is there a specific reason for UTM? The reader may also ask this question.

Response: Thank you very much for your question. We have provided additional explanations in the manuscript. The FROM-GLC30 2017 dataset downloaded from the data publishing website was based on geographic coordinates and has no projected information. In the selection of projection, we referred to GlobeLand30, a 30-meter land cover data in the same series, and the projection of this data was UTM projection. Although the study area of the Pan Third-Pole had a large span and involves many different projection zones, we solved the complex problem of zoning projection conveniently by using Python programs. In addition, we found that SRTM, Lansat8 remote sensing images and some other global data were also released using UTM projection, showing the widely use of UTM projection in global scale datasets. (Line 182-185)

14. Page 3, Line 111: “unequal probability” is better as “variable probability”. Variable Probability Sampling is VPS.

Response: Thank you for your suggestion. Here we have made more detailed explanation of the sampling. (Line 187-192)

15. Page 3, Line 124: “In other 22%” better as “In the other 22%”.

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 216)

16. Page 4, Line 131: “Generally speaking, the reference…” should just be “The reference…”. (No need for extra words like this).

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 223)

17. Page 4, Line 133: Explain “Majority” in more detail.

Response: Thank you for your suggestion. Here the “Majority” means the new value of the pixel will be determined according to the most commonly used value in the filter window. We modified it as “The resampling method was Majority. That means the land cover type of each 30 m grid after resampling is consistent with that accounting for the largest proportion of the corresponding 900 grids with 1 m resolution”. (Line 225-227)

18. Page 4, Line 147: “Existing research findings…” change to “Previous research findings…”.

Response: Thank you for your suggestion. We modified this part to explain more focus on the process of reference data productive, and this sentence was not used in the revised version. Thank you anyway.

19. Page 5, Line 163: “high-resolution satellite data of China” should be “high-resolution Chinese satellite data”.

Response: Thank you for your suggestion. We modified according to your suggestion. (Line 346)

20. Page 5, Line164: “..data, and global night…” add Modis as “..data, MOD13Q1 (NDVI) and global night…”.

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 347)

21. Page 6, Lines 190-192: It is not clear where the areas come from. Are they from the SUs (sampling units)? If so you need to say this. Total area in each case is then total SU area.

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 449)

22. Page 9, Lines 243, 244: “…it could be found that the of absolute value the difference” should be “…it can be found that the absolute value of the difference”.

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 585)

23. Page 10, Lines 261, 262: “in West Asia - Northeast Africa. was 98%, only 1.2% in Central and Eastern Europe.” Should be “in West Asia - Northeast Africa was 98%, and only 1.2% in Central and Eastern Europe.”

Response: Thank you for your suggestion. This part has been largely revised, we added F1, and now most of analysis is based on F1. This sentence has been deleted.

24. Page 10, Lines 263, 264: “China and West Asia - Northeast Africa., about 55%, while was lowest in Southeast Asia, about 40%.” Better is “China and West Asia - Northeast Africa., at about 55%, while it was lowest in Southeast Asia, at about 40%.”

Response: Thank you for your suggestion. This part has been largely revised, we added F1, and now most of analysis is based on F1. This sentence has been deleted.

25. Page 11, Lines 282,283: “In the related land cover data accuracy assessment researches” is better written “In previous research on land cover data accuracy assessment”

Response: Thank you for your suggestion. It has already been modified as what you suggested. (Line 624)

26. Page 11, Lines 282, 283: Can the authors provide references to these papers? If so, please add them.

Response: Thank you for your suggestion. The references have already been cited. (Line 625)

27. Page 11, Lines 285, 286: This is a very interesting theory. Can the authors provide a reference to it? It will help the discussion.

Response: We must be sorry that we have not found a relevant reference here for the time being, but this work is still going on. (Line 407)

28. Page 13, Line 348: “at the same time” is strange, do the authors mean “in the same place”?

Response: Thank you for your suggestion. What you understand is exactly what we want to express. The expression here has already been modified as what you suggested. (Line 846)

29. Page 13, Lines 349-352. The problem of separating shrubland is common to all GLC maps! Grassland, Shrubland and Forest form a continuous gradient. But it may not matter when the classes are being used. Main thing to differentiate is fraction of cover. The Forest and Grassland growth forms should be able to be separated but shrubland is very hard.

Response: Thank you for your suggestion. Here we combine your comments to further explain that shrubland has low accuracy is a common conclusion. (Line 847-848)

30. Page 15. The authors should consider to additionally conclude that the results of the paper will be helpful for applications in different locations of the Pan 3rd Pole as OA, UA and PA vary with location as well as land covers of interest. This is a very useful outcome of the paper.

Response: Thank you for your suggestion. It is quite useful to us. We have already added it to the manuscript. Similar content has been added also in introduction, which actually help us answered another Reviewer’s questions quite well. We appreciate it.  (Line 959-961)

Reviewer 2 Report

The authors present a study for the accuracy assessment of the FROM-GLC30 Land cover dataset over an extended geographical area covering the Pan Third-Pole Area. The topic of the manuscript is interesting and it is relevant for the audience of the journal, while it has important implications for all the research community using such open global scale datasets.

However, the manuscript needs to be significantly improved since several points are not clear enough to follow and therefore cannot be evaluated in depth. Therefore, the authors should improve the clarity of the manuscript and provide more background details on their methods.

Further to this, the authors must include in their study a quantitative analysis of the uncertainty of the results from the accuracy evaluation they followed.

Detailed comments:

Line 28 “real field condition”. Not clear-revise

Line 85 “and its influencing factors”. In the discussion section, indeed the authors discuss this objective. However, the results do not include such an analysis

Lines 115-116. “Each sampling unit was a complete small 115 watershed of 0.2-3 km2, and a square area of 1 km by 1 km was selected in the flat terrain area”. This not clear. Was the reference unit the watershed or the 1 km square polygon??

Lines 138-139. Include the dates of the field survey in the text

Line 143 “systematically revised”. Not clear

Line 148 “acquisition accuracy”. Not clear

Line 165-168 “As the setting of the reference data classification system mainly considers the application 165 perspective of regional soil erosion, it was slightly different from the land cover classification system 166 of the FROM-GLC30 2017 dataset. Therefore, the classification system of the two datasets needed to 167 be partially consolidated and category mapped”. Again, the description for the nomenclature homogenization is not clear. Was the reference dataset from another project/research study or it has been developed within the specific research study presented within the manuscript?

Line 177 and elsewhere land use/land cover

Section 3.1. This part of the analysis is not described in the methodology

Figure 6. Does the boxplot presents the standard   five-number elements (i.e. median etc)? Please specify

Lines 243-244 ..absolute value of the difference

Lines 245. These figures (i.e. difference between user’s and producer’s) should be presented in a Table

Lines 248-258. It is hard to follow all these numbers by simply looking at Figure 8. Either revise Figure 8 or revise text

Lines 251 & 259 The “significant difference level” has not be mentioned in the methodology and it is not presented consistently within the results. Please explain this part

Lines 285-287 Not clear what the authors mean

Lines 288-312 This is “results”

Lines 334-336 “It could be concluded that the more complex the composition of land cover type is, the more broken it is in space, the lower the accuracy of land cover data is.” This is not supported by the methods and the results presented in the manuscript

Line 341 “centralized  distribution”. Not clear

Line 345 “The impervious surface is integrated in the FROM-GLC30 2017 dataset when it has a low  degree of accumulation, and its accuracy is low”. Again, this is not clear. How this was inferred?

Line 388 “relatively consistent”. This expression is vague

Author Response

Dear reviewers and editors,

Thank you very much for your comments to our manuscript, entitled ‘Accuracy Assessment of the FROM-GLC30 Land Cover Dataset Based on Watershed Sampling Units: a Continental Scale Study’, with the reference number of sustainability-942493. We have revised the manuscript according to the comments. Grammar and spelling have also been check by a native English speaker.

Following is the one by one reply to the reviewers’ comments. The Line numbers in this document refers to the revised line number with all the tracking displayed.

We highly appreciate your carefulness and conscientious suggestions, and your broad knowledge, which helped us improve the manuscript a lot.

Wish you all the best!

Sincerely yours,

Zitian Guo, Chunmei Wang*, Xin Liu, Guowei Pang, Mengyang Zhu, Lihua Yang

2020-09-29

Reponses to reviewer #2’s comments

Comment 1: The authors present a study for the accuracy assessment of the FROM-GLC30 Land cover dataset over an extended geographical area covering the Pan Third-Pole Area. The topic of the manuscript is interesting and it is relevant for the audience of the journal, while it has important implications for all the research community using such open global scale datasets. However the manuscript needs to be significantly improved since several points are not clear enough to follow and therefore cannot be evaluated in depth. Therefore, the authors should improve the clarity of the manuscript and provide more background details on their methods.

Response: We appreciate your comments on the importance of this work. In the following response, we have made deeply revision according to Reviewer’s detailed suggestion.

Comment 2: Further to this, the authors must include in their study a quantitative analysis of the uncertainty of the results from the accuracy evaluation they followed.

Response: We made two revision according to this suggestions: First, we added table 2 in section 3.2 to give information of mean, median and standard deviation of OA, analysed together with Fig.7, to show the accuracy difference is really a different among regions (Line 524-536); Secondly, we have added a quantitative and more clear explanation of our sampling method in section 2.2.1 (Line 188-209). By the more clear explanation, readers will understand our sampling method was stratified random sampling and the sample is large. This will make readers more confident about the accuracy we obtained since this sampling method is one the most commonly used method and has been proved to be efficient.

Comment 3: Line 28 “real field condition”. Not clear-revise

Response: Thank you for your suggestion. We replace “real field condition” with “the reference dataset”. (Line 28)

Comment 4: Line 85 “and its influencing factors”. In the discussion section, indeed the authors discuss this objective. However, the results do not include such an analysis

Response: Thank you for your suggestion. We quite agree with you. In this manuscript, there are discussions on the factors affecting accuracy, but they are not summarized in conclusions. The focus of this research is still on the description of land cover data accuracy. Therefore, we delete “and its influencing factors”. (Line 97)

Comment 5: Lines 115-116. “Each sampling unit was a complete small 115 watershed of 0.2-3 km2, and a square area of 1 km by 1 km was selected in the flat terrain area”. This not clear. Was the reference unit the watershed or the 1 km square polygon?

Response: The description of the sampling unit form has been modified. In this research, the sampling unit includes two forms, small watershed and square polygon. In general, small watershed is the main form of sampling unit. The area of the selected small watershed is 0.2-3 km². When the sampling area is a large area of farmland, sandy land and other flat terrain areas, it is difficult to define a complete small watershed. In this case, 1 km square polygon was selected as the sampling unit. (Line 192-209)

Comment 6: Lines 138-139. Include the dates of the field survey in the text

Response: Thank you for your suggestion. We have added the year of the field surveys. It has been modified to “In order to improve the quality of the reference data, four field surveys were organized in Thailand, Pakistan, Tibet, China, and Xinjiang, China in 2018 and 2019”. (Line 233-234)

Comment 7: Line 143 “systematically revised”. Not clear

Response: Thank you for your suggestion. We modified it as “According to the results of field surveys, some common errors in the interpretation were identified”. And more details of the processing of our reference dataset, including the process of revision have been described. (Line 236)

Comment 8: Line 148 “acquisition accuracy”. Not clear

Response: Thank you for your suggestion. It has already been re-write as Comment 7.

Comment 9: Line 165-168 “As the setting of the reference data classification system mainly considers the application perspective of regional soil erosion, it was slightly different from the land cover classification system of the FROM-GLC30 2017 dataset. Therefore, the classification system of the two datasets needed to be partially consolidated and category mapped”. Again, the description for the nomenclature homogenization is not clear. Was the reference dataset from another project/research study or it has been developed within the specific research study presented within the manuscript?

Response: Very good question! Yes, the reference land cover data in this research was originally the land cover interpretation results of project related to Pan-Third Pole soil erosion project, a very big project in China and our team is one part of that project, and our work in that project is the land cover filed survey and interpolation. So we have to be consistent with the project need in land cover classification, which is more reasonable for soil erosion assessment, and to some degree different as the global public datasets. We appreciate Reviewer’s comment on the description for the nomenclature homogenization. This part has been revised. We adjusted the two land cover data classification systems in combination, and gave a comparison table to describe the unified classification system, which we believe is clearer. (Line 350-356)

Comment 10: Line 177 and elsewhere land use/land cover

Response: Thank you for your suggestion. We have revised it into “land cover” all through the manuscript at similar situations.

Comment 11: Section 3.1. This part of the analysis is not described in the methodology.

Response: Thank you for your suggestion. We have moved Section 2.3 to Section 2.3.2, and added Section 2.3.1 to describe the method of the area proportion analysis. (Line 401-407)

Comment 12: Figure 6. Does the boxplot presents the standard   five-number elements (i.e. median etc)? Please specify

Response: Thank you for your suggestion. Yes, the boxplot presents the standard five-number elements. And the more detail description has already been added in the text. (Line 487-489)

Comment 13: Lines 243-244 .absolute value of the difference

Response: Thank you for your question. The difference between user's accuracy and producer's accuracy has been added to the revised Table 2. As shown in Table 3, this value could be both positive and negative values. That is why we have to use absolute values. (Line 566)

Comment 14: Lines 245. These figures (i.e. difference between user’s and producer’s) should be presented in a Table.

Response: Good suggestion! We have added a new column in Table 4 to show the difference between user's accuracy and producer's accuracy, and the calculation results of difference are obtained by subtracting the producer's accuracy from the user's accuracy. (Line 595)

We also added the F1 accuracy, which is a combined accuracy of UA and PA, because we found it easier for reader to understand the accuracy of different land cover types.  The calculation was in 2.3.2. (Line 439-441)

Comment 15: Lines 248-258. It is hard to follow all these numbers by simply looking at Figure 8. Either revise Figure 8 or revise text.

Response: Good suggestion! We have revised Figure 8 into table 4 (Line 595). Table 4 displays the user's accuracy and F1 of different land cover types in eight regions. We hope it is much clearer here now. And the text has also been revised to be more brief and clear and focusing on the combined accuracy. It is much better after revision. (Line 584-594)

Thank you.

Comment 16: Lines 251 & 259 The “significant difference level” has not be mentioned in the methodology and it is not presented consistently within the results. Please explain this part.

Response: Thank you for your suggestion. In the revised version, Section 3.3 has been revised as stated in Comment 15. We found the significant difference level is too complex for readers to distinguish after we made it into a table. Otherwise, we have modified our calculation of UA and PA method which is taking the whole region pixels to calculate UA and PA for a region, which made it difficult to provide a significance statement. Now Table 4 is clear and has information of interest to readers and the old figure together with the significant difference level has been deleted. (Line 595)

Comment 17: Lines 285-287 Not clear what the authors mean

Response: Thank you for your suggestion. This sentence has been modified into: “A small watershed is the basic unit with specified geographical characteristics, such as climate, soil, terrain, vegetation, and also land cover, which is similar as in the other nearby watersheds. That is why it is more likely to be able to represent the land cover characteristics and its accuracy in regions by using watersheds as sampling units”. (Line 603-607)

Comment 18: Lines 288-312 This is “results”

Response: Thank you for your suggestion. But we would prefer to leave it in Discussion. The reason is, in this paper, we aimed to give a large-scale assessment of land cover dataset, which is useful to large scale land cover applications. Here we talked about in this part, is just to compare our results to the results using different types of sampling units (we were using watersheds, while in most of previous research, were pixels because of the large amount of work in using watersheds). We would prefer to make it a “comparison between this research and other research methods”. That is why we chose to leave it here in Discussion.

Comment 19: Lines 334-336 “It could be concluded that the more complex the composition of land cover type is, the more broken it is in space, the lower the accuracy of land cover data is.” This is not supported by the methods and the results presented in the manuscript

Response: Thank you for your suggestion. Here the word “conclude” leads to misunderstanding. Yes, it is for now just a hypothesis according to what we found in the results, but we need more specific experiments to explore it in future study. We have modified here as following:

“Thus, we could assume the fragmentation of land cover in a region would influence its accuracy, more fragmented, lower accuracy, which would need more detailed exploration in the future study.”  (Line 793-795)

Comment 20: Line 341 “centralized distribution”. Not clear

Response: Thank you for your suggestion. We modified it as “The possible reason is that these four land cover types are usually have wider and more concentrated distributions.” (Line 816-817)

Comment 21: Line 345 “The impervious surface is integrated in the FROM-GLC30 2017 dataset when it has a low degree of accumulation, and its accuracy is low”. Again, this is not clear. How this was inferred?

Response: Thank you for your suggestion. It has been revised in the manuscript. What we wished to express here is that except for large-scale cities and towns or clustered villages, the impervious surface is usually scattered and has a small area, such as several scattered houses with an area of about hundreds square meters, or roads with a width of several meters. These can be interpreted in our reference data, and many of them are remained in the resampled reference data with the resolution of 30 m. However, in the FROM-GLC30 dataset, a great number of the scattered impervious surface cannot be accurately presented, which may be one of the reasons for the low accuracy of impervious surface. We have modified this sentence into:

“Scattered houses or buildings with the area of hundreds square meters or roads with a width of several meters failed be displayed in the FROM-GLC30 2017 dataset, which might be one of the reasons for the low accuracy of impervious”. (Line 820-822)

Comment 22: Line 388 “relatively consistent”. This expression is vague.

Response: Thank you for your suggestion. We replace the “relatively consistent” with “similar to”.  (Line 941)

Reviewer 3 Report

The manuscript (ms) assesses the accuracy of the FROM-GLC30 Land Cover Dataset at the continental scale. The FROM-GLC30 2017 dataset was evaluated by comparing with the reference data. The study area is the Pan-Third Pole Area. However, additional experiment results are required. More importantly, the ms must discuss the scientific significance of the study. I think the ms needs major revisions to be acceptable.

The ms provides a systematic comparison of reference data and FROM-GLC30 data product. How can this research potentially contribute to other works and the LULC community? What is the scientific significance inside? Why is the proposed assessment method better than other methods? 

The FROM-GLC30 2017 dataset and the reference dataset are two main data sources in the study. According to the description in section 2.2.1, the accuracy of reference data ranges from 80%-90%. In my opinion, this accuracy is too coarse for the ground truth. This error could significantly affect the evaluation result. How to avoid the influence of this error? 

The experiment of the current ms (section 3.1 and 3.2) mainly presents the quantitative result. It is a good way to assess the accuracy of the dataset. I suggest the authors add some qualitative comparison, such as some detailed mapping examples of FROM-GLC30 data along with reference data maps and Google Earth images, so readers can visually compare the difference as well.

Author Response

Dear reviewers and editors,

Thank you very much for your comments to our manuscript, entitled ‘Accuracy Assessment of the FROM-GLC30 Land Cover Dataset Based on Watershed Sampling Units: a Continental Scale Study’, with the reference number of sustainability-942493. We have revised the manuscript according to the comments. Grammar and spelling have also been check by a native English speaker.

Following is the one by one reply to the reviewers’ comments. The Line numbers in this document refers to the revised line number with all the tracking displayed.

We highly appreciate your carefulness and conscientious suggestions, and your broad knowledge, which helped us improve the manuscript a lot.

Wish you all the best!

Sincerely yours,

Zitian Guo, Chunmei Wang*, Xin Liu, Guowei Pang, Mengyang Zhu, Lihua Yang

2020-09-29

Responses to reviewer # 3's comments

Comment 1: The ms provides a systematic comparison of reference data and FROM-GLC30 data product. How can this research potentially contribute to other works and the LULC community? What is the scientific significance inside? Why is the proposed assessment method better than other methods?

Response: Thank you for reviewer's questions. This work is mainly about a continental scale accuracy assessment of a recent public land cover dataset. We suppose there would be mainly two types of readers who read our journal and search for land cover as follows.

(1) Readers who interest in the land cover application at a large scale, for example, crop yield prediction at a large scale. They would always hope to know, what is the accuracy of the land cover dataset being used, and then they would know how much their modeling would be away from the actual situation.

(2) Readers who are the producer of large scale land use dataset. They would hope to know what type of land cover, at what kind of geographical areas should they pay more attention to their processing. Then they would improve their product according to this knowledge.

For both the two types of readers, our research offered the accuracy assessment of a recent public land cover dataset, which is just what information they would be looking for. The study area covers 65 countries, a large area. This kind of information is rare in previous research. What is more important, we are confident about our reference dataset; it has been validated by field survey, instead of just high-resolution interpolation as other researches. It offers a detailed description of that process. Our sampling units (SUs) are not pixels, which is more easily for reference datasets obtain. Instead, we used watershed covers an area of 0.2-3km² as SUs; the reason for this, we have explained in our manuscript and compare the results with pixel-based SUs.

Of course, not all the readers would be using the dataset of FROM-GLC30; some of them might even not know about it. Our manuscript will still interest those readers who are more interested in another public land cover dataset, because the accuracy difference between regions or between land cover types could be similar even if they focus on other land cover datasets at similar resolutions. Most of the public dataset is based on a similar primary method that is remote sensing image processing nowadays. The method, results, and discussion in this manuscript will still make benefit from reading this manuscript.

We are sorry to take too long answering the questions. We hope this will make clear the whole story. We have revised the introduction especially the last paragraph about the importance of this work (Line 100-104) and the sentence about using watershed as SU (Line 65-68) to make it more clear to show the idea

Thank you for your questions which helped us make the introduction more clear to readers.

Comment 2: The FROM-GLC30 2017 dataset and the reference dataset are two main data sources in the study. According to the description in section 2.2.1, the accuracy of reference data ranges from 80%-90%. In my opinion, this accuracy is too coarse for the ground truth. This error could significantly affect the evaluation result. How to avoid the influence of this error?

Response: It's a good question and made our manuscript improved at this point. Ground truth is too hard to obtain. That is why in most of the research on this topic, no actual ground truth was given. And the readers believe the results from high resolution (sub-meter, for example) interpolation results.

The base image we used to get the reference dataset is in sub-meter resolution and used hand drawing of the land cover boundaries. This should be near 100% high quality and be ground truth as what other research believed. But actually, there are 2 reasons to make it hard to be totally the same as ground condition. That is because: (1) Different years. The survey time is 1-2 or even 3 years later than the interpolation year. Land cover ground truth might change. (2) Hard to identify all the land cover type right based on an image, even at sub-meter resolution. This makes the reference dataset hard to be 100% the same as ground truth, not only in our study but also in most of the research in this field.

The number of 80%-90% is calculated based on the reference dataset before revision according to field survey results. The good news is, after the field survey, we found that many of the interpretation errors were regionally common. For example, in Tibet Plateau, the most common mistake was to interpret grassland as bareland, which makes the reference dataset before revision only 80.1% in accuracy according to field surveyed ground truth. That give a chance to revise our reference dataset according to the common interpretation errors and improved the accuracy of the reference data not only at the survey sampling units but also units with similar conditions. This year in August, we went to Tibet Plateau again for validation of this revision, in different SUs as before, we found after the revison, in all the surveyed 16 SUs the reference dataset is 100% accuracy, that improved our reference a lot. But we could not say our reference dataset is now 100% in accuracy, because this validation after revision has only done in the Tibet Plateau. So we would be more comfortable to say, our reference data is more accurate than 80%.

Actually, the readers might only focus on the field survey and revised dataset processing. So we chose to delete the accuracy values but explain the revision process in more detail to make readers benefit and not be confused. (Line 233-332) stated as following:

"In order to improve the quality of the reference data, four field surveys were organized in Thailand, Pakistan, Tibet, China, and Xinjiang, China in 2018 and 2019. Based on the field survey of 53 small watershed sampling units (Figure 3). According to the results of field surveys, some common errors in the interpretation were summarized. For example, in Tibet Plateau, the most common mistake was to interpret grassland as bareland. In images, many objects look like bareland in color but actually low cover of grassland at high elevation. After the field survey, the reference data was revised according to the common errors not only at the surveyed SUs but also SUs at similar conditions and same regions. That helped improve the reference data."

Comment 3: The experiment of the current ms (section 3.1 and 3.2) mainly presents the quantitative result. It is a good way to assess the accuracy of the dataset. I suggest the authors add some qualitative comparison, such as some detailed mapping examples of FROM-GLC30 data along with reference data maps and Google Earth images, so readers can visually compare the difference as well.

Response: Thank you for your suggestion. Details of the two datasets together with Google Earth images have been given in Figure 4 in the revised manuscript. (Line 392-399)

Round 2

Reviewer 2 Report

The authors have successfully addressed all comments. Nice work

Reviewer 3 Report

The point-to-point response to my comments and suggestion was adequate. Thank you for the revisions.