Controller-Free Hand Tracking for Grab-and-Place Tasks in Immersive Virtual Reality: Design Elements and Their Empirical Study

Round 1

Reviewer 1 Report

This is a very well written manuscript that was enjoyable to read. The authors do a great job concisely articulating the ideas, study, and results which makes the manuscript very clear and reasonable to interpret. I do, however, have some small revision suggestions for the authors to incorporate for the final manuscript. First, by using a within-subjects approach, the power is notably less. As a result, the sample size is somewhat small for a within-subjects study that has three conditions. I believe this should be articulated somewhere in the manuscript, perhaps in the Limitations section (4.3). Second, there are a number of design changes between the prototype and traditional Leap API presented in section 1.4. While some of these changes can’t be illustrated in a manuscript (i.e., audio feedback or limiting pickup based on hand velocity), it would be nice to incorporate images highlighting some of these differences. For example, an image showing the smart object coloring and another one showing the transparent hand in comparison to the traditional Leap API hand would be great additions. Overall, this is a very well written manuscript that is ready for publication.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

The paper describes an experiment in which three different methods for hand-tracking in Virtual Reality are compared. To be much more precise, the paper compares two different devices (not methods), and for one of these devices (Leap Motion) it tests two different methods for performing a pick-and-place task.

The paper is well written and it is very easy to read. English is correctly used as far as I can tell. The design of the experiments is also correct.

The paper starts with one assumption (line 38: "It can be assumed that interacting with objects in VR as one would in real life increases the intuitiveness and naturalness of interaction - and with it the efficacy and outcome of VR-based applications (e.g., the user’s gain, safety and comfort)."). 

This assumption is sensible if we could make hyper-realistic devices, but that is not the case. Therefore, the assumption turns into: higher naturalness for hand tracking is beneficial and should increase usability and performance. This is assumed in the abstract.

However, the authors find (line 12) that this assumption is not true for the experiments they performed. In my opinion, this is not surprising. Some other papers have studied this issue and reached a similar conclusion.  For instance, in

McMahan RP, Lai C, Pal SK (2016) Interaction fidelity: the uncanny valley of virtual reality interactions. In: International Conference on Virtual, Augmented and Mixed Reality. Springer, pp 59–70

a similar conclusion is reached. The reason for this result may lie in the uncanny valley in natural interaction systems explained in McMahan et al. According to this idea, interaction systems that do not resemble real world interactions could provide better user performance and usability than more realistic interaction techniques. Tu put it in other words, non-natural and highly-natural interfaces are preferred over semi-natural interfaces.

Although the authors know about this uncanney valley, because they cite it at the end of the paper, I think the assumption they made in the beginning of the paper does not match the state of the art. I think they should not assume that higher naturalness is going to be automatically better for usability and performance. They know that this is not always the case.

In this regard, the paper should review the State of the Art of the question in much more detail before presenting the experiments and the results. I would add a Related Work section, because the Introduction section does not offer much information about this. In my opinion this is the only major missing point of the paper. The experiments are clear, the results are also clear, and the discussion and explanation of such effects are coherent and consistent, but a related work section is necessary in an empirical research paper like this.

In line 57, the authors write:  "Challenges when using a hand-tracking interface in VR (e.g., the Leap Motion) include that detection errors can occur when the user’s hands are occluded, which typically happens more often with camera-based than with controller-based position tracking". I think they should also comment that there might be an offset between the virtual hands and the user's real hands. This offset could be significant in some cases and interfere with usability.

In addition, the study is not quite gender-balanced (23-8). I would like to know why and if this could be a problem in terms of generalizing the results. I have a similar question about age. The experiment is performed with quite young people. These limitations should be emphasized in the paper. The question of video-game experience is commented, but gender is not...

I think the authors should also translate the text in Figure 2. Questionnaires' pictures are in german and are quite unreadable. In this regard, I would like to see the "single subjective questions" that the authors briefly comment on line 151. It is an empirical paper. Therefore, all the questions should be listed verbatim (or translated appropriately if they were in german).

I have also one question about line 161 ("Of the three interfaces you used, which did you like best?"). What is the criterion for "best"? They like best for WHAT?

I would also like to know why accidental drops were removed from the data. If there are accidental drops it means the hand-tracking system is not working well or the user is not capable of using it properly. Therefore these accidental drops are quite relevant. I would not remove them On the contrary, I would analyze and compare these drops between the 3 setups.

I am also curious about this statement: "we were not interested in the comparison between the B_Leap and Oculus" (line 178). Why not? You have all the data, why wouldn't you compare B_Leap with Oculus? I see no reason for excluding this comparison.

The asterisks (* and ***) in Figures 5 to 8 should be explained. They are only explained in Figure 9.

I also think the authors should perform some kind of statistical test for the data in section 3.2.5 (overall preference). Is 18 to 7 and 7 a significant result with 32 participants or could that happen by chance? For two-choice comparisons a binomial test is usually applied. In three-choice options there should be a similar test. I have the feeling that the result is significant, but I think it should be proven.

As I previously said, although the discussion found in the conclusion section is correct and they mentioned the uncanny valley, this concept should appear before (in a Related Work section) since this concept is not new at all. In fact, it also happens in robots, as explained in:

Mori M, MacDorman KF, Kageki N (2012) The uncanny valley [from the field]. IEEE Robotics & Automation Magazine 19:98–100

Finally, I don't understand the sentence in line 283: "For tasks that involve grabbing and placing, where high accuracy and quick grabbing are not the main focus, hand tracking can be a viable alternative". Why do you say that pick-and-place tasks do not require accuracy? I thinks it is just the other way around.

As a summary of my revision, I think this is a good paper, and I should commend the authors for it, but I also think it could be improved with the suggestions that I have made. For this reason my recommendation is "Accept after minor revision (corrections to minor methodological errors and text editing)" because I think the addition of a Related Work section won't change significantly the conclusions of the paper, won't reduce the contribution of the paper, and won't change, of course, the experiments and the results. Nevertheless, the changes that the paper needs are not really that minor, since I also recommend that the question of accidental drops, and other issues that I have listed, be addressed.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

Generally, this is a well-written but very condensed paper reporting about a decent study, which compares different interfaces to support object-grasping tasks in immersive VR.

There is some valuable contribution to the current state of knowledge, but not a really big one. Points limiting the value are the very specific question investigated (focusing on a special variant of the Leap Motion interface), the slightly skewed and a bit homogeneous participant sample, and the overall not exciting results.

Nevertheless, I have great sympathy with the general point this paper brings across. This stems from the fact that I have experienced in our own research several situations where a sophisticated interface, which was intended to be specially "natural", turned out as inferior to very basic and well-known interfaces. I believe it makes good sense to publish such results, to trigger a discussion whether it is just missing perfection or a general misconception which is behind this effect. So from my point of view, the results in themselves are worth to be published, and the statement of limitations given is quite adequate.

My main concern with the paper in its current state is that is looks a bit immature to me. Somehow it is a draft for a paper in some parts, whereas some other parts are worked out quite well. Here is a list of those parts which I found a bit unfinished:
- In section 1.4, the storyline gets a bit strange, by jumping harshly into very specific details. It is a bit confusing to explain the own modifications of the Leap Motion interface in detail before before having explained the general concept of interacting with the Leap Motion and its visual representation.
- In section 2.1, “the task” is mentioned without explaining what task it is. The task is finally described in 2.3. Alt least a forward reference would be needed.
- Pages 4, 5, and 6 contain a number of images, without much explanation - which makes the presentation not easy to consume. A similar effect appears on pages 11, 12, and 13.
- Section 2.4.1, for instance, and a few others, look to me just like a collection of bullet points waiting for further elaboration.
These issues can be resolved by a more smooth embedding into accompanying text.

The reporting of results is very dry, so for instance table 1 is not introduced very well. Still, the results are well understandable (looking at table 2).

The discussion in general sounds a bit defensive in favour of Leap-Motion-Style interfaces. I would have preferred a more general point of view, addressing the question whether we ever will achieve "naturalness" by improving interfaces in details. Somehow touch interfaces were for along time inferior to mouse pointing, but this has changed or is about to change - driven by ever more perfect touch interaction. A generalising tone in the discussion using analogies like this would make the paper more helpful for readers, I think.

I would like to state clearly that the English text, where it exists, is very well written and easy to understand, and that I did not find any typos worth mentioning.

Generally, my impression is that I clearly understand what the paper wants to tell, but just because I have a strong background: I have used Leap Motion for IVR myself, I have used the Oculus Touch controller, so I can imagine very well what went on in the study. Regarding the current state of this paper, I think that some more work has to be spent to make it helpful also for a larger audience.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 4 Report

A very well-written article examining usability of various effectors in a simple block-moving task in virtual reality accompanied by a wonderfully-documented github repository with code, data, and MSc thesis. The data presentation is also excellent. It is overall quite heavy on the number of statistical tests (particularly on all the various subscales), and I feel that the correction for multiple comparisons only within each pair of post hoc tests, without any correction across measures, is problematic. To their credit, the authors are pretty up front about the overall lack of difference between the two hand-tracking conditions in the abstract.

I have only a few minor comments

Line 184 – this link doesn’t appear to be correct

Line 204 – I believe it is more appropriate to use non-parametric statistics with the individual Likert Scales (for the omnibus scores, parametric should be fine)

Line 238 – I’ve not seen a report of leap-tracked hands inducing uncanny valley effects, nor seen any evidence that these might affect behaviour. Indeed, the graphics used here seem quite deliberately chosen to avoid that worry (quite simplified and cartoony). The two citations here don’t relate to CG animated hands.

Line 285 – I agree that the customization improved the number of drops, but the evidence of the other factors is quite weak statistically and graphically – I’d be more careful with the conclusion about the other factors and focus on the drops

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

The authors have addressed all the comments I raised in the previous revision round. I still think that the paper would be better with a State of the Art section, but the authors did not want to follow my advice. They just slightly modified the Introduction section.

Other than that, I have no more comments to make.

Reviewer 3 Report

The paper has been revised taken the reviewer's comments into account.

I still would have liked to see a deeper discussion of "naturalness" in interaction, but I can accept what is written now in the conclusion section.

Since only minor changes were requested, the paper is now in a state to be published in its current form.

Reviewer 4 Report

The authors have done a fine job of addressing my comments