Cartographic Vandalism in the Era of Location-Based Games—The Case of OpenStreetMap and Pokémon GO
Round 1
Reviewer 1 Report
The article "Cartographic Vandalism in the Era of Geo-Gaming -The Case of OpenStreetMap and Pokémon GO" addressed the problem of vandalism in VGI and revealed insightful phenomenon based on thorough analysis. The findings and the methodology should be interesting for VGI researchers.
The article used some innovative ways to address the information insufficiency of VGI research, including the selection of vandalizing & fixing changesets, the determination of user contributing areas, and the analysis schema to reveal what the vandalizing users edited. The statistical methods were generally well-chosen, where multiple robust statistical methods are combined to handle the long-tailed data.
There are several issues which can be improved:
Line 252: Fitting the distribution of fixing time using the power-law function can be inaccurate. Although power-law distributions are reported for VGI in previous literature (Juhász 2016), it may not apply to OpenStreetMap (https://0-www-mdpi-com.brum.beds.ac.uk/2220-9964/5/1/5) since VGI projects can differ in user contributing patterns (https://0-www-mdpi-com.brum.beds.ac.uk/2220-9964/9/1/10). Moreover, fitting long-tailed distributions is inherently hard and can be misleading even with high p-values (http://epubs.siam.org/doi/abs/10.1137/070710111). Figure 7a had several problems including 1) using linear regression on log scale, 2) fitting sample distribution instead of cumulative distribution on discrete values, 3) no p-value and confidential intervals 4) the line was not a good fit even visually. Because the fitting brought many problems but didn't add much value to the discussion, it is recommended that the authors simply use boxplots instead to describe the distribution.
Line 257: Linear regression on log-scale can introduce significant bias. Trend test methods specifically designed for non-linear data will be more appropriate.
Line 321: To present the sustaining nature of anti-vandalism actions even better, it is recommended that 1) "returning users" can be further divided into returning-once users and returning-twice-or-more users, and 2) all "fixing" changesets by fixers can be extracted to demonstrate how these contributors work against vandalization in a context wider than the Pokemon Go case.
Author Response
Please find our response in the attached PDF document.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Referring to abstract "As a result of OSM’s increased popularity, the worldwide audience ... is directly exposed to malicious edits which represent cartographic vandalism." and after line 35: "That is, VGI projects are vulnerable to cartographic vandalism". This statement hat to generalized since vandalism happenes to other (commercial) mapping projects - specifically to Google Maps - too through their editing and feedback channel.
In line 86 "1.3. Related Work on Cartographic Vandalism" I' dexpect that following talk is also being cited: "Can we validate every change on OSM?", Lukas Martinelli, State of the Map 2018, Milan:
https://2018.stateofthemap.org/2018/T079-Can_we_validate_every_change_on_OSM_/ .
Author Response
Please find our response in the attached PDF document.
Author Response File: Author Response.pdf
Reviewer 3 Report
The article offers an analysis of vandalism events linked to the use of VGI (OpenStreetMap) data in a popular geo-game (Pokémon Go). It describes the nature of these vandalism events and presents the response from the VGI community. The authors analyze the changes made to the data on both sides and characterize those who made these changes (vandals and correctors).
Overall Comments
Overall, the document is well constructed. The literature review is relevant. The methods and analysis seem scientifically sound. The text can be further improved to clearly state the objectives of the authors and make the reader aware of the analyzes and results to come.
Throughout a paper, the reader must be able to make the link between the objectives stated in the introduction, the analyzes carried out and the results obtained. These links are not (clearly) made in the text, which makes the reader lose interest in the paper. Make sure that the links between the research objectives, the methods used and the results obtained are clearly stated all over the text. The work you have done worth it.
Major Comments
|
Lines |
Comment |
|
215-216 |
Rephrase. Research objectives did not mention the interest of studying the temporal dynamics of vandalism. See overall comments |
|
217-221 |
See overall comments about links between objectives, analysis and results |
|
311-312 |
How do you know these users deleted their accounts? Although possible, it seems very unusual [1] |
|
313-317 |
If “age” refers to the editing experience of a contributor, you should rather use the time elapsed between their first and latest contribution. The registration date may overestimate contributors’ experience, especially if they registered before 2010 [3]. |
|
329-335 |
Interesting findings, but you should prepare the reader first. See overall comments about links between objectives, analysis and results. |
|
362-364 |
Interesting findings. A thought: According to your results and the scientific literature you have read so far, do you think this correction speed ratio (65/35) for vandalism is different from the correction speed ratio for casual editing errors? And therefore, if this correction speed ratio (65/35) is the same, could it be considered specific to the Linus’ law? |
|
408-410 |
Since you consider that question important, Bégin et al. [3] have shown that many types of events may make VGI visible to massive audiences and induces a gain in new contributors. Specifically regarding OSM and Pokémon GO, look at the first graph and the first comment of the “Contributor Lifespans” section of Jennings Anderson’s OSM diary [4] |
|
Section 5 |
Integrate with Discussion and Conclusion section, remove duplicated statements |
Minor Comments
|
Lines |
Comment |
|
63 |
Add parenthesis (Figure 1a) |
|
138 |
Add parentheses (e.g. with warning messages). |
|
146 |
“we first identify changesets that fixed PGO vandalism as a starting point.” Clever! |
|
177-179 |
It has already been said. |
|
180 |
What does the term discussion refer to here? Changeset comments? Clarify. |
|
182 |
Idem |
|
183 |
“ChangesetMD tool”: add a reference and briefly explain the objectives of its use |
|
198 |
“JOSM preset” Wouldn’t “JOSM pre-defined tag values” be any clearer |
|
201-202 |
Interesting, pragmatic point of view. |
|
206 |
“Edges” Is there a better word to express that relation? |
|
Table 1 |
“# of users deleted” Needs an explanation (see major comments). Furthermore, in line 311, you mention that 5 fixer users deleted their account. These users are not shown in Table 1. |
|
Figure 7 |
Power Law Distribution: it may be good, but it could be tricky [2] |
|
275-277 |
This was important to state, thank you |
|
278-279 |
Clarify/rephrase |
|
Figure 10 |
Nice Figure, very informative |
|
304 |
“edges,” same comment as before |
|
Figure 11 |
Nice Figure, very informative |
|
327 |
Unclear. Maybe having both types of users in the same graph would help? You decide. |
|
341-344 |
Something expected considering 94% of vandalism is made within a 5Km radius. Not sure that describing the extent of actions from about 1% of vandals is pertinent. |
|
386-387 |
Interesting |
|
407 |
“ …PGO vandals identified as identified in our study.” Rephrase |
|
444 |
“Manuscript” be replaced by paper? |
References
[1] https://wiki.openstreetmap.org/wiki/FAQ#How_can_I_close_my_account.3F
[2] Clauset, A., Shalizi, C. R., & Newman, M. E. (2009). Power-law distributions in empirical data. SIAM review, 51(4), 661-703.
[3] Bégin, D., Devillers, R., & Roche, S. (2017). Contributors’ enrollment in collaborative online communities: the case of OpenStreetMap. Geo-spatial information science, 20(3), 282-295
[4] (https://www.openstreetmap.org/user/Jennings%20Anderson/diary/390743
Author Response
Please find our response in the attached PDF document.
Author Response File: Author Response.pdf
Reviewer 4 Report
This is a well-developed and clearly written paper about a niche application of VGI that is intriguing in its revelations about human behavior.
Overall, the writing crisp and easy to understand. The paper does make some assumptions about the readers' knowledge of some background, including changesets, Linus' Law, tree maps, and basic concepts about PGO (e.g., how does a player win?). The content is sufficiently interesting to attract readers without background in either PGO or open source development.
A few minor notes about the writing: Is carto-vandalism a word or jargon? In line 54 - reminder should be remainder. In line 59 - is should be are. I do not think that Pokémons is the plural of Pokémon, but please check with a source that is more authoritative than me.
Line 109 writes about wasted time as a bad thing. Is that a judgement call? Or are we sure it is a positive thing?
Starting out, it was unclear to me whether the paper made a sufficient argument that it is addressing a problem sufficiently important to warrant a paper. How big a problem is it? Is bad VGI data really such a bad thing? I do think the paper is warranted, but it should establish its importance better.
The paper provided some background on VGI, but could have used a little more to show this as an area of theory and not just applications. Building on that, is it possible that the participatory nature of PGO is different than simple VGI? I think an extra paragraph or two on PPGIS and participation might help readers positions this (which I think is still quite new for them).
Overall, the use of graphics is strong. The maps in Figure 5 should identify their locations, otherwise some readers will be looking for Lake Geneva in the Eastern US. Figure 9 should be labelled to explain the type of infographic that is being used (even though it has been used before).
Since this paper is nearly ready for publication, I want to focus on one change that would improve it and broaden its audience. Currently, the final conclusions focus fairly tightly on the implications for PGO, OSM and VGI. While it might be a bit speculative, Section 5 could be broadened to reflect what this research suggests about mischief, motive and cartographic vandalism in the larger realm of VGI. Do popular applications like PGO threaten to bring a non-carto crowd into our previously closed world? Is self-interested mischief a threat to PPGIS applications in land use planning or could cartographic vandalism undermine VGI applications in mapping the spread of COVID-19?
This seemingly little paper is on the verge of some large questions, and I think it would be good for it to start to ask them, even if it doesn't have the answers.
Author Response
Please find our response in the attached PDF document.
Author Response File: Author Response.pdf
