Wearable Haptic Device for Stiffness Rendering of Virtual Objects in Augmented Reality

Abstract

We propose a novel wearable haptic device that can provide kinesthetic haptic feedback for stiffness rendering of virtual objects in augmented reality (AR). Rendering stiffness of objects using haptic feedback is crucial for realistic finger-based object manipulation, yet challenging particularly in AR due to the co-presence of a real hand, haptic device, and rendered AR objects in the scenes. By adopting passive actuation with a tendon-based transmission mechanism, the proposed haptic device can generate kinesthetic feedback strong enough for immersive manipulation and prevention of inter-penetration in a small-form-factor, while maximizing the wearability and minimizing the occlusion in AR usage. A selective locking module is adopted in the device to allow for the rendering of the elasticity of objects. We perform an experimental study of two-finger grasping to verify the efficacy of the proposed haptic device for finger-based manipulation in AR. We also quantitatively compare/articulate the effects of different types of feedbacks across haptic and visual sense (i.e., kinesthetic haptic feedback, vibrotactile haptic feedback, and visuo-haptic feedback) for stiffness rendering of virtual objects in AR for the first time.

1. Introduction

With the recent drastic advancement of sensing hardware along with estimation algorithms, the techniques for virtual reality (VR) and augmented reality (AR) have been tremendously researched and employed for many real-world engineering applications such as telerobotics [1,2], medical training [3], and brain–computer interface technology [4]. Since humans perceive stiffness of an object mainly relying on haptic sensation (contact force, compliance of the object, deformation of the finger-tip, tactile sensation), usage of wearable haptic devices has been tremendously researched for finger-based interaction [5,6] in VR, surely including studies for stiffness rendering [7,8,9,10].

Kinesthetic haptic devices would be a standard setup for rendering stiffness, which can directly reproduce kinematic/dynamic aspects of interaction with an object [11,12,13]. They are still normally bulky/heavy hardware with an expensive prices [14,15,16] which is not suitable for public usage. As alternative setups, cutaneous haptic devices (CHDs) are utilized, which can deliver stiffness via skin deformation of finger-tips [9,11,17], or vibrotactile haptic devices, which have the simplest structure for implementation, are also researched by modulating amplitude and frequency of vibration to deliver an illusion of stiffness [10,18]. In addition to employing a real haptic device, only utilizing visuo-haptic (i.e., pseudo-haptic) feedback [18,19] can provide a sense of stiffness to some extent without requiring additional devices.

However, compared to these VR scenarios, rendering an object’s stiffness in AR is much more challenging given the difference that the real objects in the field of view are inevitably displayed to the user. Particularly for finger-based haptic manipulation, the presence of two components in the scenes mainly would be a bottleneck, that is,

• Presence of Haptic Device: Since wearable haptic devices are typically designed for scenarios where the devices are invisible (e.g., VR applications or teleoperation of a remote robot), their visual appearances are not suitable for AR usage. For instance, the bulky exoskeleton structure of kinesthetic devices [15,16] can severely occlude or distort the visual appearance of the real hand. Even for cutaneous haptic devices with smaller form-factor [17,20,21], the fingertips, which are the most crucial parts for object manipulation, are severely occluded. These kinesthetic or cutaneous devices can lead to severe degradation of user experience and task performance in AR; thus, research changing the wearing location of the cutaneous device from the fingertip to the proximal phalanx [7,22] or utilizing vibrotactile haptic devices [23] are performed for AR finger-based manipulation.
• Presence of Real Hand: The presence of a real hand is another critical issue for AR finger-based manipulation since the fingertips of the real hand inevitably inter-penetrate a virtual object. In the studies about VR manipulation, the inter-penetration has been prevented by visually enforcing constraints in rendering [24] or by physically preventing penetration utilizing kinesthetic haptic devices [25]. These techniques increase user experience and preference, yet they cannot be directly applied for AR manipulation due to the presence of a real hand or bulky devices. Few studies propose solutions for AR (e.g., rendering of constrained avatar hands [26]), which are limited for general applications. Moreover, methods using visuo-haptic feedback for stiffness rendering in VR also cannot be directly adopted because the perception of an object’s physical features is fairly different in AR and VR [8] even displaying the same scenes.

This paper presents a novel wearable haptic device, which can render stiffness rendering of various virtual objects for AR finger-based interaction as shown in Figure 1. Our device can prevent inter-penetration problems by generating kinesthetic force, which is the most effective haptic feedback for VR/AR finger-based haptic manipulation [22,25]. Usage of passive actuation (i.e., locking mechanism), which allows for much smaller hardware than active kinesthetic devices [27,28], substantially reduces the visual distortion of the real hand in AR from the attachment of haptic devices. The tendon-driven system is also designed for transmission mechanism, which can minimize the finger-tip occlusion unlike CHDs with ensuring small-inertia, light-weight, and no back-lash [27]. Moreover, to deliver clearly different sensations of hardness/softness, we devise a selective locking algorithm (i.e., rigid mode and elastic mode) that can even be embedded in a small form-factor with only one actuator. The fine rendering of stiffness also can be achieved through the integration of vibrotactile modules. Total weight and cost of the proposed device per finger are only about 30 g and can be decreased to less than 10 $ (when replacing 3D-printed parts) respectably, which are much more comfortable and affordable than commercial kinesthetic devices or other haptic devices, simultaneously adequate for AR finger-based manipulation.

Moreover, this paper quantitatively compares the effects of various feedback conditions for stiffness rendering of AR objects in finger-based interaction. The way to select the feedback condition among numerous available options is crucial for the realistic rendering of physical attributes (i.e., weight, friction, stiffness) of objects in VR interaction. Thus, the studies comparing different feedback conditions are also researched [11,18,29]. As opposed to the many existing results in VR, a few studies research stiffness rendering in AR environments, yet they are limited on tool-based haptic interaction [12,30] or should utilize real tangible objects [7]. In addition, the studies about finger-based haptic interaction in AR [22,23,31] do not research about rendering physical attributes of AR objects.

In this study, we firstly demonstrate the effects of different feedback (rigid/soft kinesthetic, vibrotactile, and pseudo-haptic feedback—See Section 3) in an AR environment through an experiment with a two-finger grasping task, which has been widely used in many VR finger-based experiments [11,24,28,32]. Since the perceived stiffness is measured in comparison with real objects, our results would be more applicable for AR applications. Moreover, quantitative comparison for various feedback conditions is also demonstrated, which can be a design guideline for rendering stiffness of a specific virtual object in AR. In addition, by measuring the sense of realness in the experiment, we manifest the effect of each feedback in aspects of user experience at the same time. Through the experimental results, the efficacy of the proposed device is clearly verified for realistic rendering of stiffness corresponding to the rigidity/elasticity of AR objects.

The rest of the paper is organized as follows. Section 2 presents the design of the proposed wearable haptic device with a fabrication process. The experimental study verifies the efficacy of the kinesthetic haptic feedback from the proposed device, compared with vibrotactile and visuo-haptic feedback, for rendering stiffness in Section 3. The experimental results are discussed in Section 4, followed by concluding remarks in Section 5.

2. Wearable Haptic Device for Stiffness Rendering in AR

The main purpose of the proposed wearable haptic device is to generate enough kinesthetic force to prevent inter-penetration between finger-tips and a virtual object, at the same time, to minimize occlusion of the bare hand from wearing the device. For this, we employ a passive haptic mechanism [14], which can effectively halt the motions of fingers with a minimal-size actuator. We also adopt a tendon-driven system, which has a much smaller form-factor than other force transmission systems such as exo-skeleton [15,16]. More specifically, the structure of the developed haptic device comprises the following three components: (1) tendon locking part (Figure 2), which modifies the stiffness of the tendon using a selective locking mechanism (rigid mode or elastic mode), (2) tension transmission component (Figure 3A), which transfers the resistive force from the locking through the tendon, and (3) actuator fastening part (Figure 3B,C), which firmly fastens the above two components to the user’s hand when generating the force feedback, detailed descriptions of which are presented in the following sections. In addition, to make it easy to wear the devices, we attach the devices to a fabric glove. In this study, we intend to deliver the haptic feedback only to the two fingers (thumb/index), but extension to the other fingers would be trivial.

2.1. Tendon Locking Part

The main component of the proposed haptic device is the tendon locking part, the structure of which is shown in Figure 2A and the supplementary videos. The plate of each finger is connected with the locking part by the tendon. The tendon is fixed on a wall of the part, sequentially connected with a soft spring ($k_s\simeq 100$ N/m) which generates free tension to prevent slack in free hand motion, and a much stiffer spring ($k_e\simeq 2000$ N/m) which renders stiffness of elastic objects. Moreover, the tendon-spring is wound to two pulleys (an elastic mode pulley and a rigid mode pulley), where the braking force from the above servo motor is exerted. The bearings are located on the axes of the pulley to decrease friction in free motion. In the housing of the locking part, the placement of the tendon-spring is optimized to cover maximal displacement of the springs from finger motions (about 2 cm in a pilot test) while minimizing the size of the component.

The proposed device can render two types of stiffness corresponding to rigid/elastic objects by altering the effective stiffness of the tendon-spring structure via a selective locking of one of the pulleys. A servo motor is placed above the tendon-spring and pulleys, which has an arm to break the pulleys when actuated. To maximize the braking force, the upper side of the pulleys and the arm of the motor takes forms of gear and pawl, respectively.

Descriptive figures of the mechanism are shown in Figure 2B,C. In the free motion mode (Figure 2(B1,B2)), the configuration of the arm is horizontal, thus the pulleys can freely rotate, which leads to smooth free motion and slack prevention. In the elastic mode (Figure 2(C1,C2)), the servo motor is rotated in a counter-clockwise direction and the left pawl fixes the elastic mode pulley. Then, only the stiff spring is connected with the tendon so the user can feel the haptic sensation of elasticity when touching virtual objects. In the rigid mode (Figure 2(D1,D2)), the servo motor is rotated in a clockwise direction and the right pawl fixes the rigid mode pulley. Then, no spring is connected with the tendon so the user can sense the illusion of touching a rigid object while preventing the inter-penetration of the finger-tip simultaneously. In summary, the effective spring constant of the tendon–spring structure can be expressed in the following form:

$$k_{eff}=\left\{\begin{array}{cc}\frac{k_e{k}_s}{k_e+k_s}\simeq k_s\hfill & \mathrm{free}\mathrm{motion}\mathrm{mode}\hfill \\ k_e\hfill & \mathrm{elastic}\mathrm{mode}\hfill \\ k_{tendon}\gg k_e\hfill & \mathrm{rigid}\mathrm{mode}\hfill \end{array}\right.$$

(1)

2.2. Tension Transmission Part

When the locking module fixes the length of the tendon by breaking the gear, the resistive force is generated along with the stiffness of the selected mode as the user tries to bend their fingers more. The tension transmission part is designed to deliver this kinesthetic haptic feedback stably and efficiently, which is shown in Figure 3. In order to minimize the friction and fix the trajectory of the tendon, we guide the tendon by attaching teflon tubes on the glove as a tendon router. The tendon is connected to the contact plate, where the resistive force directly exerts on the finger. The contact plate consists of a 3D-printed upper part and a lower vibrotactile actuator, through which our device also can deliver vibrational haptic feedback. Plus, the length of the tendon is adjustable along with the user by adopting an adjustable knot structure before the contact plate.

Note that the contact plate is placed on the intermediate phalanx of the finger, not on the distal phalanx. This is because we want to alleviate the issues of the tendon-driven system (e.g., hysteresis, slack) by shortening the length of the tendon, assuming that the distal joints of the fingers have musculoskeletal dependencies with the parental joints (i.e., motion synergy [20,21,33]).

2.3. Fastening Part

The resistive force feedback preventing the bending motion of the fingers is delivered via the tendon mechanism. Since the tension exerts not only to the intended finger-tip parts but to the other end, the tendon locking part, this can lead to unexpected movement of the locking module. Therefore, we design the fastening part to fix the locking part on the hand to deliver the force feedback stably.

The structure of the part is shown in Figure 3B,C. We add the rigid hook with channel beam structure on the backside of the 3D printed locking part, and attach the velcro wrist band to the glove. In wearing the haptic device, the velcro band should be fastened with the hook structure tightly to fix the module on the dorsal/wrist of the hand. Moreover, we attach a polymer pad with high viscosity and friction, directly below the module inside the glove, to stick the module effectively by increasing the friction between the skin and the device/glove. Finally, our locking module can deliver the force feedback stably in the fixed positions.

2.4. Fabrication

In the aspect of fabrication, the proposed system is able to employ smaller size actuators than active actuation systems. Thereby, we adopt commercial servo motors (SG-990, Tower Pro, Hongkong, China), which have a low-cost and small form-factor. We also employ Super P.E., Dyneema^® as the tendon. The rigid parts of the device are 3D printed using the material, Verowhite Plus; thus, the total size is only about $50.2$ mm × $33.5$ mm × $29.3$ mm. In addition, we utilize a commercial low-cost glove, which is made of spandex fabric.

The total weight of the two haptic devices and glove is only about 60 g, which would be one of the lightest kinesthetic systems [14], thus ensuring the high wearability/usability and small occlusion of the real hand shape. Moreover, if the 3D-printed parts can be manufactured in way of mass-production in the future, the total cost of the device can be less than 10 $, which is affordable for general/commercial usage.

3. Experimental Study: Stiffness Rendering of Virtual Objects

The efficacy of kinesthetic haptic feedback from the proposed device for stiffness rendering is verified through the experiment. To clearly show the efficacy, we design a comparative experiment of the kinesthetic feedback with the other feedback. As a task for the experiment, we design a “two-point grasping” task of a virtual object, which is a rudimentary finger-based action for daily objects interaction, and, thus, frequently studied [11,24].

3.1. Experimental Setup

3.1.1. Hardware Setup

• Experimental Environment: An overall snapshot is presented in Figure 4A. A subject is instructed to sit in front of the table surrounded by the MOCAP cameras. The plane of the table is slightly sloped for the subject to look at the virtual object as vertical as possible, where the average vertical distance from the eyes to the plane is about 30 cm.
• Stereo Camera: To capture stereoscopic scenes of a real environment for AR scenes, we utilize Stereolabs^® ZED Mini [34], which is designed particularly for AR usage and can be equipped with commercial HMDs. The resolution/sampling rate of the captured scenes is 1280 × 720 and 60 Hz, respectively, for each stereo image.
• Head Mounted Display (HMD): To display an interactive AR environment, we employ Oculus^® Rift S [35] as the HMD. We compose AR scenes, where the real environment and virtual objects are co-presented, using the cross-platform game engine Unity^®. We also utilize the existing functionality of adopted hardware for rendering realistic AR scenes (e.g., mixing virtual/real scenes, depth-sensing).
• Motion Capture (MOCAP) System: For dexterous finger-based haptic interaction, one of the key prerequisites is to track the motions of finger-tips and the HMD in real time. In this study, we utilize an Optitrack^® MOCAP system, which consists of a set of IR reflective markers and surrounding IR cameras. The MOCAP system provides 3D positions of markers with an accuracy less than 1 mm at 200 Hz. We attach two MOCAP markers on the index/thumb finger-tips of the developed haptic glove and four MOCAP markers on the HMD. Note that, for real-world AR manipulation, the MOCAP system can be replaced with an affordable tracking algorithm (e.g., [20]), yet, the MOCAP system is employed here to minimize the effect of tracking error on the stiffness perception.

3.1.2. Feedback Conditions for Virtual Objects

In this experiment, we employ mainly four types of feedback conditions for stiffness rendering, which are already used for stiffness rendering. To be more specific (Figure 4B), the feedback conditions are (1) Rigid kinesthetic haptic feedback ($K{H}_r$), where the proposed device locks the tendon mechanism (except the rigid tendon) in contact with a virtual object, which can deliver stiff resistive force to the user preventing inter-penetration. (2) Elastic kinesthetic haptic feedback ($K{H}_e$), where our device partially locks the tendon-mechanism (except the rigid tendon and the stiff spring) in contact with a virtual object, which can deliver elastic resistive force to the user corresponding to the penetration distance. (3) Vibrotactile haptic feedback ($VH$), which shows an illusion of stiffness by delivering vibrational feedback to the finger-tip when the finger-tip is penetrated to the virtual object [10,18]. (4) Pseudo-haptic feedback ($PH$), which visually renders deformation of a virtual object along with the penetration distance of the finger-tip, which can modify the stiffness perception of virtual objects by modifying the visual appearance of the objects. During the following experiments, a total of five types of conditions (the above four feedbacks and no feedback) are randomly delivered to the subjects.

3.2. Procedures

3.2.1. Stiffness Referencing Phase

Over the experiment, the goal of the experiment is “how similar a feedback condition can deliver an illusion of stiffness with real objects in AR”. We think these would be more applicable results for general finger-based haptic manipulation, for instance, what type of feedback should be delivered for rendering the stiffness of a real object with specific material (e.g., steel, wood, rubber, sponge). For this, we want to directly compare the stiffness of virtual objects rendered by feedback, with the stiffness of real objects. Thereby, we design and perform a “stiffness referencing” of real objects phase before the main experiment.

For referencing the stiffness of real objects, we prepare a sensing box, which includes three sphere-shaped real objects $o_r$ (rigid), $o_e$ (elastic), and $o_s$ (soft) with different stiffness; a plastic ball, a rubber ball, and a sponge ball with the same shape as shown in Figure 5A. The stiffness of the objects is measured by $o_s\simeq 200$ N/m, $o_e\simeq 2000$ N/m, and $o_r>>o_e$ for the rigid object, which is selected with clearly the distinguished values to cover a wide range of stiffness of daily objects. The diameter of the objects is equally 4 cm, and the subjects are instructed to perform the two-finger grasping task for the objects. To block the visual feedback of the objects (e.g., color, texture, shape deformation) and enforce the subjects to sense the stiffness only through haptic feedback, the objects are occluded with a box during the task.

We instruct the subjects to grasp each object in turn as the object can be deformed enough. Since the users are asked to answer multiple-choice questions to represent their perceived stiffness and realness in the next section, we implement a GUI panel in the AR environment as shown in Figure 5B (left). Then, we mark the stiffness level of the grasping objects (i.e., levels 2, 4, and 6), and the subjects are given enough time to touch the real objects until the subjects can memorize the sense of the stiffness firmly. Moreover, we also give all types of experimental feedback conditions to the subjects ahead, to avoid the effect of learning or adaptation.

3.2.2. Stiffness Discrimination Phase

After the referencing phase, we perform a trial that consists of a grasp task and an answer task. The procedures of the grasp task of each trial are shown in Figure 5B (middle), where a virtual sphere is displayed on the surface of the table plane, and their position is randomly determined within the predefined range from the center of the plane (rectangular area of 20 cm width and 10 cm height). The size of the sphere is fixed to 4 cm, which is the same as the previous real objects in the referencing phase. The subjects can sense the stiffness of the sphere by two-finger for five seconds. The type of feedback for the subjects is also randomly selected among the mentioned conditions for each trial.

After the grasp task, the subjects are asked to answer two multiple-choice questions as shown in Figure 5B (right). The GUI panel, where the subjects can answer their responses by pointing gestures, is displayed in AR, which is same with the referencing phase. The first question asks the subjects to represent the perceived stiffness from the grasping task, compared with the real objects in the previous referencing phase. Thereby, there are seven levels of choices; over the object $o_r$, similar to the object $o_r$, between object $o_r$ and $o_e$, similar to the object $o_e$, between object $o_e$ and $o_s$, similar to the object $o_s$, below the object $o_s$. For example, if a subject senses the presented virtual object as hard as $o_e$, the subject rates the stiffness by level four as shown in Figure 5A.

The second question asks the subjects about the realness of the virtual sphere, which is one of the most important factors implementing virtual objects [36]. Thereby, we ask the users “Was touching the virtual object just as natural as touching the real object?”, which is a modified version of the question about realness in [36], and the answer is rated on a seven-point Likert type scale. Thus, the subjects are instructed to rate level seven when they feel a virtual object as a real object in the grasping task, or rate level one when they feel like touching a perfectly virtual object. The subjects answer the questions on the panel in AR immediately after each trial.

The number of trials is 50 times, which has a duration of about fifteen minutes for each subject. In addition, the subjects are instructed to perform the stiffness sensing phase in every 10 trials, to recall the sense of the stiffness of the real objects. A total of eight subjects participate in the experiments. The subjects are all right-handed males in the age range of 22 to 31 years, with no known perception disorders. The experiments are conducted in accordance with the requirements of the Helsinki Declaration. The overall procedures of the experiment are described in the supplementary videos.

3.3. Results

The experimental results for each feedback condition are presented in

Table 1. The experimental results of perceived stiffness and realness of each feedback condition; mean value, standard deviation, and standard deviation of mean values between subjects.

Stiffness	Mean	Standard	Std. Dev.
Rating	Value	Deviation	(Subjects)
$K{H}_r$	4.92	0.81	0.57
$K{H}_e$	3.56	0.97	1.34
$VH$	3.16	1.20	1.39
$PH$	1.45	0.60	0.59
No Feedback	1.06	0.29	0.36
Realness	Mean	Standard	Std. Dev.
Rating	Value	Deviation	(Subjects)
$K{H}_r$	4.82	0.90	0.77
$K{H}_e$	4.25	0.84	1.48
$VH$	3.37	0.91	1.34
$PH$	4.28	1.26	1.93
No Feedback	1.38	0.55	0.57

and plotted in Figure 6. Then, the mean values (and the standard deviations) of perceived stiffness and realness are obtained quantitatively, which clearly shows the effect of each feedback condition.

The condition of rigid kinesthetic haptic feedback $K{H}_r$ is obtained by $4.92$, which is the highest corresponding to the object between $o_r=6$ and $o_e=4$. This means that, although the value slightly lacks for rendering perfectly rigid objects, it is still larger than $o_e$ (a rubber ball) or any other feedback, which certifies that the kinesthetic haptic feedback is suitable than any other feedback for rendering fairly stiff objects in the AR environment. Moreover, as shown in the results of realness, the rigid kinesthetic feedback is also the highest value $4.82$, and this result verifies that the importance of the kinesthetic haptic feedback in finger-based interaction. The results of the standard deviations, $0.81$ and $0.57$ (between subjects) for the perceived stiffness and $0.90$ and $0.77$ (between subjects) for the perceived realness, are also rather small, which implies that the subjects perceive consistently for clear/strong kinesthetic haptic feedback regardless of users. From the results of standard deviation and realness, the efficacy of the proposed passive mechanism for the kinesthetic feedback is also verified.

For the condition of elastic kinesthetic haptic feedback $K{H}_e$, the perceived stiffness is measured by $3.56$, which is similar to the stiffness of the object $o_e$. This result means that the kinesthetic haptic feedback generated by a stiff spring can deliver the sense of stiffness of elastic real objects such as rubber, fabric, or skin of a human. Moreover, the realness of the feedback is measured by a fair high value, $4.25$, with a small standard deviation $0.84$; thus, the elastic haptic feedback can realistically generate the elasticity of virtual objects. These results verify the validity of the proposed selective locking module for elastic kinesthetic feedback, which can be used for rendering elastic/deformable materials (e.g., rubber, fabric, or skin of a human).

The stiffness rating of the vibrotactile feedback condition $VH$ shows a similar stiffness value of $3.16$ with that of $K{H}_e=3.56$ or $o_e=4$. This result implies that the vibrotactile feedback is also effective on a certain level for rendering elasticity similar to the elastic kinesthetic feedback, which again verifies the effectiveness of vibrotactile feedback for virtual object interaction as presented in previous studies [10,18]. However, the standard deviation of the feedback is a rather larger value ($1.20$ and $1.39$) than those of the elastic haptic feedback ($0.97$ and $1.34$), which means that the rendered stiffness from $VH$ is inconsistent for each trial or each user. The realness metric is also relatively smaller $3.37$ compared to the elastic kinesthetic feedback conditions ($4.82$ and $4.25$). This result indicates that the realness can be substantially increased from no feedback condition, which is also consistent with VR results (increased presence or task-performance by using the vibrotactile feedback [29]), yet a limitation of the vibrotactile feedback in AR that is solely utilizing the feedback lacks a vivid illusion of stiffness.

In case of the condition for the pseudo-haptic feedback $PH$, the stiffness value $1.45$ is fairly smaller than the other haptic feedbacks or even the soft real object $o_s=2$, which shows that the adopted visuo-haptic feedback is not suitable for rendering stiff virtual objects. However, the result of realness has a much higher value $4.28$ than no feedback condition $1.38$, implying the importance of proper visual haptic feedback in finger-based manipulation, which is already verified in VR [24]. Moreover, compared to the other haptic feedbacks, the perceived realness is even fairly higher ($VH$) or similar ($K{H}_e$), which shows the efficacy of visual feedback particularly for rendering elastic/deformable objects. Despite the large standard deviation $1.26$, which means that subjects perceive the visual deformation feedback diversely; this result presents some interesting facts that visually realistic rendering of virtual elastic objects would enhance the realness of interaction even without using additional haptic devices.

For the no feedback condition, the perceived stiffness is the lowest value $1.06$, which is even lower than the pseudo-haptic feedback rendering deformable objects. The perceived realness is also an extremely low value of $1.38$ close to level one, which would imply that, when rendering a rigid virtual object without appropriate feedback, the subjects perceive the object not as a rigid/real one but a virtual/intangible one. This fact certifies again that the design of appropriate feedback conditions is crucial for finger-based manipulation in AR environments, the same as with the VR cases. In addition, these results verify that the proposed tendon-spring structure, which employs a soft spring ($k_s$) to prevent slack motion, negligibly affects free finger motion.

4. Discussion

We propose a wearable haptic device for stiffness rendering, which can generate kinesthetic haptic feedback corresponding to elastic/rigid virtual objects in AR. The fundamental limitations of existing devices for AR finger-based manipulation are the co-presence of AR objects with the user’s hand, which raises problematic inter-penetration between the finger-tips and object, and the haptic device, which hampers the user experience and performance by occluding finger-tips. To minimize the size of the proposed device, we adopt a passive (i.e., locking) actuation mechanism with the tendon-driven transmission, which is a suitable hardware setup for AR (small-size form factor, lightweight). We also design a selective locking part, which can effectively generate two types of kinesthetic feedback for rigid and elastic objects.

From the experimental results, we quantitatively compare the various feedback conditions for stiffness rendering. The kinesthetic haptic feedback $K{H}_r$ is verified to deliver the highest stiffness, even stiffer than the real rubber ball consistently. The realness is also perceived as the highest value, which means that the kinesthetic haptic feedback would be the most adequate feedback for rendering a virtual rigid object, and other feedback (e.g., vibrotactile feedback) is not strong enough for rendering a rigid/realistic object in AR finger-based manipulation.

For rendering an elastic/deformable object, the elastic kinesthetic feedback $K{H}_e$ and vibrotactile feedback $VH$ are reported to render similar stiffness values; however, the kinesthetic feedback shows better consistent perception (small standard deviation) with a superior realness. Our device is able to render a wide range of elasticity by replacing the elastic spring $k_e$ along with the aiming applications, which would be applicable for many applications in AR/VR, (e.g., social interaction with an avatar, manipulation of a deformable object). However, given the simplicity of implementation of the vibrotactile feedback, the vibrotactile feedback would be a possible option for some applications.

Particularly, the subjects report the high realness of virtual objects for the pseudo-haptic feedback $VH$, which renders deformation along the inter-penetration. The realness is even larger than the haptic feedback conditions $VH$ or $K{H}_e$, which implies that the visual feedback can also substantially enhance the user experience of the AR finger-based manipulation. These complementary results of haptic feedbacks and visual (pseudo) feedback have an interesting implication for the integration of different types of feedback (i.e., across visual, vibrotactile, kinesthetic) for realistic rendering daily objects with a wide range of stiffness.

5. Conclusions

We propose a novel wearable haptic device generating multiple kinesthetic haptic feedback in a small form-factor, which is desirable for stiffness rendering of virtual objects for realistic AR finger-based manipulation. In the human subject study, the effects of various feedback conditions for stiffness rendering are quantitatively obtained and compared in a real AR finger-based interaction scenario. The efficacy of the proposed haptic mechanism with passive actuation and selective locking, which can render rigid and elastic/deformable virtual objects, is also verified via the high experimental results of the perceived stiffness and realness for the rigid/elastic kinesthetic feedbacks.

The future works of our research would be as follows: (1) In real AR finger-based applications, humans sense the stiffness and realness by combinations of multiple sensations simultaneously; thus, the effect of interplay across various haptic and visual feedback conditions should also be studied to provide the schemes for utilizing multiple feedback. (2) The two-point grasp task in our experiment is valid to measure the perception of stiffness quantitatively; however, it is a rather simple task among real finger-based interactions in VR/AR (e.g., peg-in-hole [20], throwing, or stacking of virtual objects [29]); thus, the efficacy of the stiffness using the proposed kinesthetic haptic feedback should be verified for many real-world applications in terms of task-performance or human perception. (3) The proposed haptic device can be integrated with existing wearable hand tracking systems (e.g., [20]), to implement the first wearable haptic interface applicable for general AR finger-based applications.