VisWebDrone: A Web Application for UAV Photogrammetry Based on Open-Source Software

Round 1

Reviewer 1 Report

General comments:

The proposed manuscript describes an image-based modeling (photogrammetry) web service based on FLOSS solutions dedicated for aerial applications.

The content is well structured and quite well written (a simple yet clear and understandable English). It details the workflow and the technological choice to conceive a web service dedicated to aerial photogrammetric application with a correct but superficial approach.

From the current state of explanation, we can only presume that this web application is currently at advanced conception or prototyping stage but not yet an open and operative service (no screenshots, or link are provided).

The authors made the choice of relying on FLOSS packages, which is a praiseworthy intention but the advantages and inconveniences are not discussed above the consideration of ratio cost to performance.

The paper details all the modules required to develop a complete framework, from processing to visualisation, still without raising the doubt on the complexity of defining a user friendly and efficient system.

The first module operates a conventional photogrammetric reconstruction  (exclusively MicMac-based) aiming to compute dense point cloud and orthomosaic using already automated tools (cannot be considered as authors contribution).

From the result of the initial step, the architecture is logically divided in two branches, one handle orthophoto and DEM outputs while the other deals with 3D dense points.

On the first branch two modules are required to store and display the 2/2.5D map representations base on two packages. The second branch is a 3D viewer powered by Potree.

The supposed efficiency of the framework is illustrated on a single case study with two different area briefly analyzed and compared with a standalone commercial solution.

The results are discussed with minimal and basic metrics (processing time, RMSE) however the outcomes seems reasonably accurate, no details are given to validate to comparability of the results (processing stages, parameters).

The conclusion follows the content of the paper, highlighting only presumed strengths without exposing/discussing limits and weaknesses of the proposal.

Detailed comments section by section :

Title / general : The title is too generic and doesn't reflect the specificity of the content and application domain. I would really recommend to add UAV or aerial in title and to clarify within the article the fact that the web application seems to be dedicated to this specific kind of processing.

It doesn’t mean the platform will not be able to process other kind of dataset, however there is an important difference in technical and methodological requirements between aerial et terrestrial photogrammetric processing.

1. Introduction :

The authors give a quick overview of current photogrammetric solutions by comparing/opposing commercial and FLOSS, the technical choice of different packages implemented is briefly introduced as well.

However a specific paragraph introducing the scenario of web application (remote and/or computing) providing photogrammetric service is missing, neither compared with already available SaaS solutions (eg. Software as a Service) or historical review of research prototype (eg ARC 3D web service launched in 2006).

The authors might consider to add few words on the context of their development (academic and/or professional use), the target (open platform, paying service and potentiel users, private use) and potential users (in term of experience and quantity), currently unexposed.

2. System architecture :

The main diagram has to be improved to ease the readability with the textual content.

2.1. Module 1:

The authors detail the choice to rely on MicMac instead of others but without considering the hot-topic of interoperative processing (cooperation between different package), indeed MM is yet very powerful but it also comes with some important drawbacks. The main consideration above the versatility of MicMac to motivate the final choice is the density of final point cloud compared with ODM, it could be true but the authors give any information on the processing pipeline while high quality/density point cloud a very relative metric.

2.5 Software integration :

The architecture itself is briefly detailed but an important part is missing the interactivity between the system and the user.

As exemple, at no point the manuscript contains information about ; access to the portal, GUI or UX aspects, workspace with several projects, enumeration of processing mode, breakpoint and reprocessing, possible collaborative aspect. Just to name a few...

3.1. MicMac Implementation :

The authors detail some possible pipelines to handle aerial photogrammetry using MicMac. The pipeline itself is a sequential and linear operative chain articulating some automatized commands of the package.

No mention is made on coping errors or possible optimization required to reach robust and efficient processing for automated application. MicMac is a very versatile and expert solution, but in this case and without entering a high level technical explanation some basic processing details are unclear or not shared, all over the workflow (Tapioca mode and resolution to ZoomF in dense matching). As an experienced user of MicMac for almost 10 years there are misunderstanding points (see attached PDF), that have to be addressed or corrected.

It is also highly recommended to add or explain a paragraph describing how the data has to be formalized in input/output, moreover regarding the interactive step of GCP’s marking at image level (completely passed under silence in this manuscript).

In the current state of explanation, it is easy to prove that inexperienced or harsh user can make the process fails only by uploading images without metadata or dataset including rolling shutter.

Identically, how will behave the application if a user upload 890 pictures instead of 89 like presented in the case-study.

4. Case study :

The authors try to evaluate and prove the efficiency of their application based on a single dataset, which is the major and important flaw of the proposal.

Even if the comparison methodology seems correct, it is too weak to validate and assess objectively the presumed efficiency, robustness and accuracy of an automated web-service.

5. Results and discussion :

The results seems correct but are very discussable from a methodological point of view, moreover they cover just very narrow scope of what is expected for a web-service (compromising reliability, velocity, automation, computational cost.

There is a considerable flaw or bias in Table 2, as only the processing time of final tasks is mentioned, given the impression that the result are comparable.

Unfortunately the scientific community knows for a while that Pastis/Tapioca is a bottleneck and that Apero/Tapas is very demanding and very slow (because not parallelized).

In addition, a complement is required regarding the radiometric issue illustrated in Figure 7, also revealed by plenty of evaluation/comparison papers or reports including MicMac.

6. Conclusion

The conclusion confirms indeed the objective as being quite low or under evaluated. Actually what are the main interest and challenge of web application are not even addressed (complexity involved by automation level beyond a processing script running on a server, the deployment and scalability on a computational infrastructure).

It would have been more meaningful if the authors have detailed the technical implementation (hosting, processing, visualizing) and illustrated in different scenarios/case studies rather than focusing on a not so relevant comparison (a local computation using commercial solution VS a foreseen remote computation based on open-source) that could have been confirmed by a critical overview of the past few years literature.

Motivation of the decision :

Starting from the main comments that the proposal is not positioned within the state of the art (missing or too weak) and that the content illustrating an insufficient single case study to discuss and prove a presumed operatively of « completely automatic » web-service (not shown except at diagram level).

In addition of other major comments synthesized hereafter and several minors comment within the attached PDF.

This manuscript cannot be accepted until the condition of major revision is satisfied by addressing or responding most of the comments (main, majors and minors) detailed above and summarized below.

Main comment :

- Reinforce the position and technical soundness of the proposal.

Major comments :

- Clarify the scenario of application (dedicated to aerial mapping for academical/agricultural/urban purposes? Or multipurposes ?)

- Add or introduce related works in two folds 1/ to contextualize the proposition among other photogrammetric web-services 2/ to gain in technical soundness for comparison, qualitative evaluation.

- Provide more proof in term of web application (GUI, strategy of deployment, limitation of uses)

Minor comments :

- Provide complement more technical detail on data processing (for automation and comparison purposes)

- Correct some misunderstandings or errors (see PDF)

Comments for author File: Comments.pdf

Author Response

The authors would like to thank the reviewer by its valuable observations and suggestions.

Major changes to the manuscript text are highlighted.

Please, find the answers to your comments and suggestions in the attached PDF file.

Author Response File: Author Response.pdf

Reviewer 2 Report

The presented paper proposes a toolbox composed by the integration of several Free and Open Source Software (FOSS) for a photogrammetric web application, testing the performance of the workflow with UAV datasets.

BROAD COMMENTS

The presented work is interesting and there is overall merit in the article for developing a workflow integrating different FOSS in an innovative way. Nevertheless, there are some points to be clarified and areas for improvement in the presentation of results, as highlighted in the Specific Comments section of this review. The experiments carried out could benefit from further comments on critical aspects. English style and correctness should be revised and checked in some part of the paper.

SPECIFIC COMMENTS

Chapter 2.1 - The reason for choosing the software with which to carry out the photogrammetric processing seems quite weak and subjective. In particular, the point cloud density of 12.46M points against 10.85M (+15%) could be due to the parameters of the processing which is different in the two software. Those parameters are not mentioned in the paper. Moreover, a better quality is indicated in the MicMac results which can also be seen in a subjective or in any case qualitative way also from the images attached in Figure 2, but no numerical or scientific data is reported for this choice. Furthermore, Figure 2 could benefit from adding a scalebar.

Figure 4. The older pipeline (1) may be hidden in the workflow if it is not used in VisWebDrone processing.

Chapter 4.1 - It is mentioned that collected UAV images are georeferenced with a GNSS receiver. It should be better explained whether it was performed using an external geodetic-grade GNSS receiver for accurate positioning of the UAV during the capture of each image, or whether it is more likely the simpler GNSS receiver integrated within the drone providing an approximate position of those images. The total number of images for the second study site seem unlikely. Is there a mistake in flight height, number of images or overlapping values? Looking at Figure 6, Study site 1 (Ss1) seems about 1.3 ha, while Study site 2 (Ss2) should be around 4.2 ha. I tried to check the numbers with traditional flight planning formulas and for Ss1 I get a perfect confirmation for a single grid flight. But for Ss2 with 80% overlap and 70% sidelap, covering Ss2 with Phantom 4 camera from 80 m altitude would require just 80 images, or around 160 images with a double grid configuration. In the paper 324 images are mentioned for this area. Maybe some oblique images were also accounted in this number?

Line 292: “were performed” instead of “where performed”

Chapter 5.1 - Being GSD proportional to flight height (40 m for Ss1; a double height of 80 m for Ss2) and given the fact that the camera was the same (Phantom 4 camera, same resolution and focal length for both flights) we would expect the GSD for Ss2 to be double in respect to GSD in Ss1.

Chapter 5.2 - How many GCPs were surveyed and compared in both study sites? How many Control Points were considered for the RMSE analysis?

Conclusions - The process is described there as “completely automatic” when the photogrammetric module requires several user interventions to complete the processing. Furthermore, why do not provide a reference to the VisWebDrone application code available on GitHub?

Author Response

The authors would like to thank the reviewer by its valuable observations and suggestions.

Major changes to the manuscript text are highlighted.

Please, find the answers to your comments and suggestions in the attached PDF file.

Author Response File: Author Response.pdf

Reviewer 3 Report

The research presented in the paper is well structured and the tool is convincing. I suggest to authors to indicate the software or algorithm used to evaluate RMSE and mean error.

Author Response

The authors would like to thank the reviewer by its valuable observations and suggestions.

Major changes to the manuscript text are highlighted.

Please, find the answers to your comments and suggestions in the attached PDF file.

Author Response File: Author Response.pdf

Reviewer 4 Report

Please see the attachment

Comments for author File: Comments.pdf

Author Response

The authors would like to thank the reviewer by its valuable observations and suggestions.

Major changes to the manuscript text are highlighted.

Please, find the answers to your comments and suggestions in the attached PDF file.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

General comments:

The authors indeed took note and corrected most of the remarks among a quite consequent list.

To my sense the manuscript gains in soundness and clarity, improving reading experience and work’s credit.

Majors changes concerning technical/technological choice and methodological aspects has been successfully raised by the first revision.

Despite of some minors comment (at detail levels) that still have to be addressed, this submission shall be acceptable very soon.

Detailed comments section by section :

Title / general : The title is way more precise and meaningful.

1. Introduction :

I would have expected a better review of historical/competitive tools, citing the the ARC service (given just as example) is irrelevant.

Here a non exhaustive list of project/prototype that should have been brought by the authors  (MiaW, Culture 3D Cloud Platform, CMP SFM web Service, 3DNOW)

The manuscript newly mentionned the release as open-source, which is very valuable and made acceptable some flaws (pointed in the last review and/or still existing).

Indeed one cannot expect from FLOSS library same performance as payment for service platform.

2.1 System architecture :

Technical details given between line 129-139 are appreciable for expert readers. Therefore the density difference between result are expected because ODM is performed at MicMac level (≠BigMac).

I would homogenize the comparability (ODM expressed in pts/px  and/or MicMac expressed in px).

The integration of Figure 4 showing the platform is a great added value, proving that the platform is at operative prototyping stage.

3.1. MicMac Implementation :

I would add the Tapioca Mode (moreover because some are dedicated to UAV) and the size parameter (or rule to set the size).

I appreciate the simplified workflow (discarding outdated Malt version)

I still don’t understand how the pipeline can associate C3DC to Tawny without Pims2Mnt (incorrect).

Please consider this structure : 7 C3DC > 8 Pims2Mnt > 9 Tawny

I know that UAV-based photogrammetry benefit from controlled acquisition protocol (compared to terrestrial) but still no mention is made on coping errors or possible optimization required to reach robust and efficient processing for automated application.

4.2.2 Data processing using MicMac :

It still don’t get how Saisie QTtools are interfaced with the web service. I guess the management of GCP/Coord. System which could be quite complicated will be explained in the documentation (maybe just raise this issue this way is enough for the scope of this publication).

6. Conclusion

If possible, address in one sentence the technological choice for the deployment of the platform on the server (eg Docker, KVM,..).

Motivation of the decision :

I suggest a minor-revision to correct/complement all the point above-mentioned.

Author Response

The authors would like to thank the reviewer for his valuable comments and suggestions which helped to improve the manuscript. All changes performed to the manuscript are highlighted in the revised version.

Author Response File: Author Response.docx

Reviewer 4 Report

Congratulations

Author Response

Thank you very much.

Best Regards