Progress in the Triboelectric Human–Machine Interfaces (HMIs)-Moving from Smart Gloves to AI/Haptic Enabled HMI in the 5G/IoT Era

Abstract

Entering the 5G and internet of things (IoT) era, human–machine interfaces (HMIs) capable of providing humans with more intuitive interaction with the digitalized world have experienced a flourishing development in the past few years. Although the advanced sensing techniques based on complementary metal-oxide-semiconductor (CMOS) or microelectromechanical system (MEMS) solutions, e.g., camera, microphone, inertial measurement unit (IMU), etc., and flexible solutions, e.g., stretchable conductor, optical fiber, etc., have been widely utilized as sensing components for wearable/non-wearable HMIs development, the relatively high-power consumption of these sensors remains a concern, especially for wearable/portable scenarios. Recent progress on triboelectric nanogenerator (TENG) self-powered sensors provides a new possibility for realizing low-power/self-sustainable HMIs by directly converting biomechanical energies into valuable sensory information. Leveraging the advantages of wide material choices and diversified structural design, TENGs have been successfully developed into various forms of HMIs, including glove, glasses, touchpad, exoskeleton, electronic skin, etc., for sundry applications, e.g., collaborative operation, personal healthcare, robot perception, smart home, etc. With the evolving artificial intelligence (AI) and haptic feedback technologies, more advanced HMIs could be realized towards intelligent and immersive human–machine interactions. Hence, in this review, we systematically introduce the current TENG HMIs in the aspects of different application scenarios, i.e., wearable, robot-related and smart home, and prospective future development enabled by the AI/haptic-feedback technology. Discussion on implementing self-sustainable/zero-power/passive HMIs in this 5G/IoT era and our perspectives are also provided.

1. Introduction

With the rapid development of the Internet of things (IoT) and 5G communication technology in recent years, human–machine interfaces (HMIs) have gradually evolved from traditional computer peripherals, e.g., keyboard, mouse, and joystick as illustrated in Figure 1, to a more intuitive interface that directly collects human’s original signals [1,2,3], such as voice and basic body motions, providing users with a more intuitive and easier interaction with computers and intelligent robots in the applications of healthcare, rehabilitation, industrial automation, smart home, virtual reality (VR) game control, etc. [3,4,5]. Current commercialized advanced HMIs include non-wearable solutions based on voice and vision recognition [6,7,8], and wearable solutions that use inertial measurement units (IMUs) [9,10], i.e., microelectromechanical system (MEMS) based accelerometers and gyroscopes, for the somatosensory information collection. However, these kinds of HMIs require complicated sensory information and high-performance signal collection/processing units, thus resulting in great power consumption in the system. Additionally, the MEMS-based sensing components are commonly bulky and rigid [11], making them relatively incompatible for wearable scenarios. For these issues, stretchable and flexible HMI solutions with minimalistic sensor design emerge recently, including sensors based on the sensing mechanisms of resistive [12,13,14,15,16,17,18,19,20], capacitive [21,22,23,24], and optical fiber [25,26,27], etc. Though the signal complexity and processing cost are greatly reduced to save energy and enhance the timeliness of the system, these sensors themselves still require a small amount of driving energy, and the power consumption may be large considering the massive number of sensor nodes in a sensor network. Moreover, repetitive charging is also annoying, especially for wearable or portable HMIs. So to address these issues, self-powered sensors using nanogenerator technologies of piezoelectric [28,29], triboelectric [30,31] have been developed frequently, to build low-power/self-sustainable human–machine interactive systems.

Triboelectric nanogenerator (TENG), first reported by Prof. Zhong Lin Wang in 2012 [48], has been widely developed as energy harvesters for mechanical energy scavenging, ranging from natural wind energy [49,50,51,52,53], blue energy [54,55,56,57,58,59] to the human body’s biomechanical energy [37,43,60,61,62,63,64], thanks to its exceptional merits of good output performance, wide material choices, good scalability, simple fabrication and low cost. Considering that the kinetic energies generated from the daily activities of humans, e.g., hand motion, joint rotation, foot motion, etc., contain valuable information of the corresponding motions, so such kinds of motion-induced energies could be collected by the nanogenerators towards a fully self-powered sensing strategy for human–machine interactions as well as health status monitoring purpose [65]. Compared with piezoelectric-based sensors that are commonly difficult for design customization due to the limitation of materials and complexity of the fabrication process [66,67,68,69], TENGs show the advantages of wide choices of stretchable and flexible materials, e.g., fabric, silicone rubber, plastic thin film, etc., and versatile operation modes, i.e., contact-separation mode, liner-sliding mode, single electrode mode and freestanding triboelectric-layer mode [70,71]. Therefore, TENGs have been successfully designed into various structures for different interactions (Figure 1), such as touchpad interface [35,36,37,38,41,72], auditory-based interface [39,73,74], 3D motion manipulator [33,40,42], etc., and can be further designed as self-powered wearable HMIs, e.g., electronic skin (e-skin) [43,75,76,77], data glove [32,44], wearable band [45,46], intelligent sock [78,79], breath-driven mask [80], etc., for advanced robotic manipulation, IoT control, VR game control/rehabilitation, personal identification and advanced sport analysis, showing the wide application prospects of triboelectric in HMIs area.

The new era of artificial intelligence (AI) provides a new possibility to enhance the functionalities of HMIs via machine learning (ML) enabled data analytics [81,82], where the subtle features hidden behind the real-time signal spectrum could be extracted automatically towards more advanced human–machine interactions, e.g., gesture recognition [83], voice recognition [84], pose estimation [85], personal identification [86], object classification [87,88], etc. Thus, due to these benefits, combining TENGs with ML-enabled analytics reveals a promising research direction for the development of HMIs with enhanced intelligence and low power consumption, which has attracted great attention in the past few years [73,89,90,91,92,93]. Besides, integrating TENGs with other sensing mechanisms, such as piezoresistive [76,94], pyroelectric [95,96,97,98,99], etc., to implement a multimodal sensing system capable of perceiving different sensory information simultaneously, i.e., tactile, strain, temperature, etc., is also a good strategy to broaden the applications of TENG-based HMIs. In addition, to build a complete human–machine interaction system, the haptic feedback function enabled by mechanical actuators [100,101], i.e., wire actuator, pneumatic actuator, vibration motor, etc., is indispensable to provide users with a more immersive experience for specific applications, e.g., robotic collaborative operation [102,103], VR game manipulation [104,105,106,107,108], etc., and deserves to be further integrated into current TENG-based HMI solutions to boost up the capability of information interpretation. Furthermore, by integrating self-powered TENG sensors with energy harvesters or passive wireless techniques, e.g., near-field communication (NFC) [109], surface acoustic wave (SAW) [110], etc., battery-free systems can also be achieved towards fully self-sustainable/zero-power HMI terminals under the future IoT framework.

Herein, in this review, we systematically introduce the recent progress in the TENG-based HMIs from the following sections: (1) glove-based HMIs for advanced manipulation, gesture recognition and tactile sensing; (2) wearable HMIs for other biomechanical signal collection, e.g., eye motion, facial expressions, voice, posture, etc.; self-powered HMIs for (3) robotic perception and (4) smart home applications; (5) ML-enabled intelligent HMIs, and the possible future research direction enabled by the (6) haptic-feedback technology and towards (7) self-sustainable/zero-power/passive HMI terminals. In the end, current issues and the potential development trends for TENG-based HMIs are also provided for future research in this 5G/IoT era.

2. Glove-Based HMIs

As we all know, nearly all the conventional HMIs are inseparable from the fine operation of our hands: the keyboard needs ten-finger tapping, the mouse needs to be moved and clicked by the hand, and the gamepad needs the finger to press the button and operate the joystick, etc. Compared with the proportion of nerve areas in other parts of the body, the sensory and motor nerve areas hidden in hands and fingers are huge [111,112], making each finger can bend and stretch independently to a certain extent, and can also expand and shrink the lateral intervals. This degree of flexibility enables them to complete quite complex gestures to achieve diversified operations. A glove, as a common wearable item, is quite suitable to be further designed into glove-based HMI by integrating sensors for highly sensitive finger motion tracking. One of the most mature sensing techniques for data gloves is to use IMUs [113,114], consisting of accelerometers and gyroscopes. However, the rigid sensing elements, complex data format and relatively high energy consumption remain concerns. Other flexible solutions based on resistive [13,14,15,115,116,117], optical fiber [25,27], etc., also reveal their own drawbacks, e.g., temperature effect, limited sensing range, etc. Thus, the emerging sensing technology based on triboelectric, with the advantages of diversified material selection, extremely simplified design and self-powered characteristics, provides a new research direction for designing the next generation of low-power data gloves [118,119,120,121,122,123,124].

He et al. proposed a glove-based HMI using TENG textile as the sensing unit with the minimalist design as shown in Figure 2a [125]. The conducting polymer made of poly (3,4-ethylene dioxythiophene): poly (styrene sulfonate) (PEDOT: PSS), due to its advantages of good physical and chemical stability, was chosen as the coating material to fabricate the highly stretchable electrodes as well as the positive triboelectric parts in this fabric-based contact-separation-mode TENG sensor. Additionally, a layer of silicone rubber thin film was coated on the textile glove serving as the negative triboelectric contacting layer. There are two kinds of sensor configurations in this design, where the arch-shaped sensor is utilized to measure the finger bending motions, while the sensors mounted at the sides of the fingers are to detect the contacts between the index finger and its adjacent fingers. Based on these two configurations, the movement of the index finger in all directions can be recognized to realize the intuitive control of aircraft minigame in cyberspace, and the car/drone movement in the real 3D space. Moreover, the function of a mouse can also be mimicked by this glove-based HMI for web surfing and alphabet writing, providing a simpler, power-compatible interactive method in daily life. However, for the actual applications of such a kind of textile glove, the humidity in the environment or the sweat generated from the human body may negatively affect the triboelectric output [126,127]. So to solve this issue, a more advanced design with superhydrophobicity of the triboelectric textile for performance improvement enabled by a facile carbon nanotubes/thermoplastic elastomer (CNTs/TPE) coating approach was reported by Wen et al. as shown in Figure 2b [128]. The anti-sweat capability enables the TENG sensor to remain at 80% voltage output even after a 1 h exercise. By leveraging machine learning technology, complex gestures’ recognition function could be realized with the minimal sweat effect, which was successfully demonstrated to be further utilized for real-time VR/AR control applications, i.e., shooting games, baseball pitching and floral arrangement.

The abovementioned two works are all based on the arch-shaped TENG strain sensors [129,130], meaning that a large air space needs to be reserved between two contact layers and a relatively limited sensing range: obvious sensor response only occurs at the moment of contact and separation of the friction layers. For this problem, Zhou et al. proposed a TENG strain sensor based on a unique yarn structure as illustrated in Figure 2c [131]. The core of the sensing unit is composed of a conductive yarn coiled around a rubber microfiber, with the entire body sheathed by a polydimethylsiloxane (PDMS) sleeve. Varying degrees of deformation will result in a constant and continuous change in the contact area between the PDMS sleeve and the coiled conductive yarn, enabling the sensor with good linearity and sensitivity within a large strain range (20–90%). After integrating a wireless printed circuit board (PCB) for signal collection, processing and transmission, a wearable sign-to-speech translation system could be achieved with the multi-class support vector machine (SVM) machine-learning algorithm, whose overall accuracy could be maintained higher than 98.63% with fast response time (<1 s), showing a cost-effective approach for assisted communication between signers and non-signers, as well as the prospect of TENG-based HMI in the field of healthcare.

To further enrich the function of the glove-based HMI, Zhu et al. integrated TENG-based finger bending sensors, palm sliding sensors, and also piezoelectric mechanical stimulators onto one 3D-printed glove which realizes the multidirectional bending sensing, sliding event detecting, as well as haptic stimulation simultaneously for augmented AR/VR experiences (Figure 2d) [132]. By attaching multiple elastomer-based TENG tactile sensors onto different joints of each finger, the perception of motions of each phalanx with multiple degrees of freedoms (DOFs) could be achieved, where these sensors provide more useful subtle features regarding the finger bending compared with other common solutions that installing one sensor node per finger [136,137,138,139]. In the meanwhile, the sensory information of the normal and shear force could also be collected by the TENG palm sliding sensor to realize diversified sensing, especially for grasping-related tasks, revealing a new possibility of multifunctional HMI solution based on TENGs for VR entertainment and training applications.

Though these works based on the contact-separation mode TENG strain sensors have shown the great potential of TENG enabled glove-based HMI as effective gesture interfaces, limitations are still there and need to be further improved. One of the main issues is that most of these studies use the generated output peaks’ amplitudes to judge the bending degree [120,140], which is unstable and could be easily influenced by the varying environmental factors, e.g., humidity, etc. Due to the fast decline of the stimuli-induced electrical states caused by electrostatic equilibrium, the generated pulse-like signals can only reflect the momentary motion of finger bending. Other valuable information, e.g., bending speed, the degree at a certain moment during the entire bending movement, etc., will be lost in such kind of signal format. One possible solution is to use grating-sliding mode TENG for better quantifying the bending degree/speed by counting the generated output peak number. A joint motion TENG quantization sensor for the robotic collaborative operation application was developed by Pu et al. as illustrated in Figure 2e [133]. When the slider is driven by the finger to slid forward/backward across the well-designed interdigitated electrodes, the alternating output signal in a series of periodic narrow pulses will be generated, where the bending degree and speed could be easily distinguished based on the pulse number and width respectively. The minimum resolution of the TENG joint sensor can reach 3.8° and could be further improved with finer grating segments. With such kind of measurement method, the real-time continuous robot bending control can be realized with high precision, demonstrating a more stable and reliable solution of TENG-based strain/displacement sensor with strong tolerance to environmental interference.

Besides the strain sensors for the finger bending monitoring, the tactile sensory information also plays a key role in glove-based HMI for mimicking the biological perception system of human skin to provide a more complete anthropomorphic feedback in contact force detecting, roughness recognition, as well as temperature sensing applications [141,142,143]. Due to the high sensitivity and fast response to tiny stimuli, TENGs have been frequently investigated to simulate the fast adapting (FA) sensory cells of the skin that respond to dynamic touch and vibration [144,145] and can be further integrated with other traditional sensors based on resistance or capacitance that mimic the slow adapting (SA) cells due to the signal maintenance capability, to form a multimodal tactile sensory system [76,146]. As depicted in Figure 2f, an electronic skin that simultaneously perceives the lateral and vertical movements of the fingertip during grasping tasks was reported by Chen et al. [94]. In this design, the carbon nanotube-poly-dimethylsiloxane (CNT-PDMS) electrode layer works as a freestanding TENG sensor for roughness differentiation according to the generated output peaks when the device slides across the object surface, where rougher objects commonly contribute more peaks during the sliding motion. The porous CNT-PDMS layer serves as the static pressure sensor based on the mechanism of piezoresistive, making the tactile HMI also capable of real-time status monitoring, e.g., holding or releasing, etc., during the grasping/tapping operation process. A similar but more advanced multifunctional tactile sensor was proposed by Wang et al. as shown in Figure 2g [134]. The main difference between this device and previous tactile designs lies in the functionality of the TENG layer. The electrification layer made of hydrophobic polytetrafluoroethylene (PTFE) film here is utilized for material identification based on the varying electron affinity ability of different materials [147], and 10 different flat materials have been successfully demonstrated to be inferred with a simple lookup table algorithm. Moreover, apart from the pressure sensing ability brought by the piezoresistive property, the sponge-like graphene/polydimethylsiloxane (PDMS) composite also shows thermoelectric effects, which can be used for detecting the temperature of contacted objects with a high resolution of 1 kelvin, reveling the possibility to further enhance the diversity of the functions for TENG-integrated tactile HMIs.

Limited by the pulse-like output signals as mentioned above, the TENG-based tactile sensing units developed in most works were used for dynamic touch/vibration sensing and cannot be used to detect the continuous force variation due to the fast shift of electrical states. In this case, combining with resistive/capacitive sensors becomes necessary, and a fully self-powered sensing system could not be achieved. Actually, the problem of the electrical state shifts could be suppressed by using a high-impedance readout circuit to maintain the triboelectric output [148,149]. However, a complicated sensing system with an amplifying circuit for small current collection is needed. To address this issue, Dong et al. proposed a strategy by using the robust nanophotonic aluminum nitride (AlN) modulators to read the TENG output, as shown in Figure 2h [135]. The TENG sensor can work in the open-circuit condition with negligible charge flows due to the electrically capacitive nature of AlN modulators, and the stimuli-induced triboelectric voltage can be transformed to AlN modulators’ optical output based on the electro-optic Pockels effect. Owing to the negligible charge flow and the high-speed optical information carrier, continuous force sensing with good linearity and stability was successfully achieved based on the TENG tactile sensor with nanophotonic readout circuits, demonstrating the potential to replace resistive/capacitive sensors as SA sensing units in anthropomorphic skin, and to realize a fully TENG-based self-powered multifunctional tactile HMI.

3. Other Wearable HMIs

In addition to the glove-based HMIs, due to the advantages of high output, lightweight, high flexibility/stretchability and applicability of various structural designs [150,151], TENGs have also been developed into sundry biomechanical sensors for other biosignal collection, e.g., eye motion, facial expressions, voice, posture, etc., as self-powered wearable HMIs to bring great convenience to people [152,153,154], especially those with disabilities, in the era of information and IoT.

Among these mechanical motions, eye blinking has been proven as a new, simple and effective triggering method for handicapped people to realize convenient electrical appliance control for smart healthcare/home purposes [155,156,157]. A TENG-based micromotion sensor for eye blink motion monitoring with high sensitivity was reported by Pu et al. as shown in Figure 3a [158]. This sensor works in single-electrode mode and has a multilayered structure, where a fluorinated ethylene propylene (FEP) thin film and a natural latex thin film serve as the tribo-layers in this device, with an acrylic thin annulus as the spacer to reserve the necessary space for the contact and separation process. This tiny sensor could be easily mounted on the arms of glasses by fixators to capture the mechanical micromotion of the skin around the eyes, and function as an intuitive HMI to control electrical appliances, e.g., table lamp, electric fan and doorbell, via the simple trigger signal generated while eye blinking. By using a wireless module for data transmission, a hands-free typing system based on the input method of adjusting the number of blinks in a period was also successfully demonstrated, which may bring great convenience to our daily life, especially for the disabled, or people whose hands are fully occupied while working. Similarly, a non-attached electrode-dielectric TENG sensor for eye blinking sensing was reported by Anaya et al. as illustrated in Figure 3b [159]. This sensor is fully made of soft materials, i.e., PEDOT:PSS coated conductive textile and silicone rubber, and shows good comfortability when placed on the lateral skin tissue of the eye to detect the orbicularis oculi muscle motion. Due to the near-field transmission of signals based on the non-contact electrostatic induction, the output electrode could be mounted on the temple of the eyeglasses without a wire connected to the main sensor unit, which significantly improves the simplicity of the whole system. By analyzing the trigger signal induced by eye blinking, diversified hands-free human–machine interaction applications, including cursor control, car/drone control, etc., could be realized to assist people with mobility impairment. Additionally, the developed HMI could also be used for eye fatigue monitoring to evaluate the tiredness of a driver and give proper warnings to avoid danger.

Apart from the eye movements, fluctuations induced by the movements of other facial muscles, e.g., masseter muscle, etc., are also good choices to be translated into control command as communication aid HMI for the disabled [160]. Inspired by the frogs’ croaking behavior, Zhou et al. reported a bionic TENG-based sensor for masseter muscle motion monitoring as illustrated in Figure 3c [161]. By imitating the oral structure and acoustic capsule, the flexible PDMS elastomer was made into a sensing membrane and a deformable vibrating membrane, with an air layer as the spacer, to amplify the small fluctuations of the masseter muscle into a significant movement of the vibrating membrane, due to the varying deformations of films with different radiuses under the same volume change. TENG technology was then integrated to convert the vibration of the film into an electrical signal, which could be further utilized in a Morse code communication system, a hands-free typing tool, as well as an intelligent authentication system, to achieve the efficient collaboration between the disabled and the digitalized world.

Another biosignal that can be effectively utilized for the human–machine interaction purpose is the human voice, which has attracted great attention recently due to the rapid development of voice recognition based on artificial intelligence technology [162,163,164]. TENG-based acoustic devices have also been investigated a lot towards self-powered microphones [39,73,74,165,166,167,168]. Although these flexible acoustic devices show good performance and functionality, they are still rigid, which hinders their applications in the wearable device platform that can be comfortably integrated with the human skin. For this issue, Kang et al. developed a skin-attachable microphone with high transparency and adhesion by using hybrid freestanding nanomembranes (NMs) combined with a micropatterned PDMS and a holey PDMS film in a sandwich structure as illustrated in Figure 3d [169]. The holey PDMS film effectively enhances the vibration of the freestanding NM membrane compared with a planar PDMS film, result in a larger output voltage and excellent sensitivity. The time-dependent waveform of a speech recorded by the wearable TENG microphone was also demonstrated, which was very much in line with the original sound waveform, proving the good acoustic sensing capability of the developed microphone especially when attached to a person’s neck. Then a personal voice security system was successfully established to recognize the users’ identity with high accuracy by perceiving the collected voiceprint, showing its potential as HMI for biometric purposes.

In addition, a tactile sensing patch is also a common form of wearable HMI by collecting finger touching/sliding movements to realize simple manipulation commands [174,175,176,177,178,179] Currently, TENG tactile HMIs commonly consist of a large number of sensing pixels, where each pixel is connected to an independent output, resulting in complex readout circuits and output signals [180,181,182,183,184,185,186,187]. In order to simplify the outputs to effectively reduce the difficulty and cost of data collection and processing, some novel solutions with minimalistic electrode design have been demonstrated recently [188,189,190,191,192,193]. A two-dimensional TENG patch with a 5 × 5 pixel matrix for finger trajectory sensing was developed by Chen et al. as shown in Figure 3e [170]. With only four edge electrodes, the accurate position of the finger sliding or touching along the x and y axis of the patch surface could be achieved with a minimum resolution of ~1.6 mm based on the output ratio of two pairs of opposite electrodes, which greatly reduce the total number of output channels for multi-pixel sensing. By utilizing another one-dimensional TENG patch to detect the position along the z-axis, the real-time three-dimensional manipulation of a robotic arm was demonstrated and could be applied in various complicated operations, e.g., welding, handling, spraying, etc. A more advanced design was proposed by Shi et al., as depicted in Figure 3f [171], where a bio-inspired spider-net-coding TENG interface was developed for multi-directional sliding sensing with only one output electrode. By adjusting the electrode widths and positions along with different directions, the output signal patterns could be differentiated according to the relative amplitude and positions of the generated peaks in the time domain and could be further transferred into binary codes for more straightforward signal processing. Such a coding method enables the minimalistic TENG tactile interface with good reliability and robustness, and makes it applicable for diversified human–machine interaction applications, e.g., drone control, security code, etc.

Though some of the abovementioned HMIs have shown the possibility to realize the robotic control by defining specific commands according to the sensor signals, the number of commands is limited by the sensor design and data format, and as a result, diversified operations for multi-tasks may not be achieved. Additionally, some customized interaction commands also require a certain learning cost for the users. To realize a more intuitive HMI for parallel manipulation with enhanced degrees of freedom in the applications of advanced industrial automation or virtual reality interactions, the sensory information of the human pose is of great significance [194,195,196,197]. Zhu et al. proposed a customized exoskeleton enabled by triboelectric bidirectional sensors for upper limbs’ joint motion monitoring as depicted in Figure 3g [172]. With the well-designed grating patterns and the bistable switch integrated into the sliding-mode TENG rotational sensor, the rotating degree, direction, and speed can be achieved simultaneously to accurately reflect the real-time status of shoulder rotations, wrist twisting, as well as finger bending with a minimum resolution of 4°, and can be further utilized to collaboratively control the robotic arms and virtual character in both real and cyber space. Moreover, a ping-pong/boxing game was demonstrated to verify the capability of the control system for complex and coherent movements, revealing its potential for virtual sports training and rehabilitation applications. Apart from the movements of upper limbs, spinal bending is also essential towards a more complete pose monitoring. A badge-reel-like stretch sensor based on a similar grating-structured TENG was reported by Li et al. as shown in Figure 3h [173]. By analyzing the generated peaks’ number, high sensitivity of 8 V mm⁻¹ and a minimum resolution of 0.6 mm with good robustness and low hysteresis can be achieved for this wearable badge reel, which can be used for patients’ spinal shape change monitoring when serving as a rehabilitation brace. In addition, such a kind of spinal information is valuable when combined with other limb motions towards a whole-body movement detection, which can be widely used for human motion capture and reconstruction in the 3D animation/game industry.

4. Robotic-Related HMIs

With the gradual rollout of AI technology around the world, intelligent robots will play a more important role in our society and will gradually replace humans in labor-intensive or dangerous tasks, from the industries of service, manufacturing and medical, to daily life assistant, as well as future scientific-related space exploration [198]. As the medium to perceive the external world and enhance the interactions with humans, sensors based on TENGs have also been investigated a lot to mimic the bionic sensory system for robots’ tactile sensing [199,200,201,202,203], gesture/motion monitoring [204,205], gait analysis [206], etc., thanks to their good compatibility brought by TENGs’ wide material choices.

As shown in Figure 4a, a typical TENG tactile sensor for robot perception was reported by Yao et al. [207]. With the novel interlocking microstructures enabled by the surface morphology of natural plants, and the polytetrafluoroethylene (PTFE) tinny burrs fabricated on the tribo-layers, the pressure sensing sensitivity was effectively enhanced by 14-fold compared with sensors using flat tribo-surfaces. Owing to the high flexibility, the developed sensor can be easily attached to a robotic hand, to measure the pressure distribution, as well as finger bending, during handshaking with humans. Additionally, the capability of the sensor for surface roughness and object hardness recognition was also demonstrated, showing its potential for more advanced robotic dexterous operation and human–machine interactions.

In addition to the perception of external tactile stimulation, auditory information is also a straightforward communication strategy for robots to receive feedback information from the outside world and interact with humans [208,209]. A TENG-based auditory system for social robotics was reported by Guo et al. as illustrated in Figure 4b [210]. This developed tiny auditory sensor shows ultrahigh sensitivity of 110 mV/dB based on the triboelectricity generated by contact-separation motions of the Kapton and FEP layers during the air flow-induced vibrations. With the special design of the annular or sectional inner boundary architecture, a broadband response from 100 to 5000 Hz could be achieved, which almost covers the human voice’s frequency range. After being integrated onto a smart robot, the auditory sensor was successfully utilized to capture the music sound waveform with high quality, which can be further used for high-accuracy voice recognition in human–robot interaction applications.

During the parallel control process of robots, the sensing of the robot’s pose and motions is also critical, which can be used as feedback information to achieve more precise control and status monitoring [211,212]. As depicted in Figure 4c, Wang et al. reported a self-powered angle sensor that can be mounted on robotic arms for high-resolution angular monitoring [213]. Two rotary sliding TENGs are integrated with the difference in the overlaps of electrodes, to form a detectable phase difference between the two output electrodes for rotating direction detecting, i.e., clockwise or anti-clockwise. By counting the generated pulse number and calculating the corresponding consumption time, the overall rotation angles and the angular velocity can be obtained respectively, with a high resolution (2.03 nano-radian), high sensitivity (5.16 V/0.01°), and good signal-to-noise ratio (98.68 dB). A robotic arm equipped with this angle sensor was successfully controlled to reproduce the traditional Chinese calligraphy, proving the effectiveness of the collected signals for accurately reflecting the movement trajectory of the robot.

Thanks to the merits of good flexibility and multi-degree deformation [214,215,216], soft robots [217,218,219] made of soft materials, e.g., silicone rubber, thermoplastic polyurethane (TPU), etc., can perform conformal contact with external objects and environments, which make them applicable to various scenarios instead of making specific designs for different product lines like the widely used rigid robotic manipulators, thereby greatly reducing the costs. Considering that many commonly used TENG and soft robot materials have similar Young’s modulus, flexible TENG-based sensors have been frequently developed recently to realize the tactile/deformation perception for soft robots with good compatibility [220,221,222,223].

A TENG robotic skin for soft robot tactile perception was proposed by Lai et al. as shown in Figure 4d [224]. The sensing skin is made of silicone rubber with triangular micro prisms patterned on the surface to enhance the pressure sensitivity (9.54 V kPa⁻¹). With the high stretchability of 100%, the sensor could be easily integrated into a soft gripper to sense the different motions during grasping tasks, including approaching, grabbing, lifting, lowering and dropping, according to the variation of the open-circuit voltage in different stages. The self-muscle motion perception of a robotic crawler could also be achieved by integrating multiple tribo-skins, showing its scalability for large-area soft robot tactile sensing. Besides the tactile sensing skin, Jin et al. also integrated a gear-structural TENG bending sensor into a soft gripper to form a multifunctional sensory system to detect the tactile and deformation related information simultaneously as indicated in Figure 4e [225]. With the distributed electrodes along with the tactile sensor patch, the accurate position of the external stimuli on the sensor surface could be extracted based on the output ratio of different channels. Moreover, the continuous bending monitoring of the pneumatic finger could be realized by counting the generated peak numbers of the TENG bending sensor, with a minimum resolution of about 12°. By fusing the data from these two types of sensors, more valuable information, including grasping position, contact area, object shape and size, could be achieved for grasping tasks, providing the possibility for the soft manipulator to implement high-accuracy object recognition with the help of machine learning analysis.

However, as mentioned in the tactile sensor part, the pulse-like output signals make the TENGs more sensitive to the external dynamic stimuli, and the real-time static status monitoring, i.e., deformation or pressure, of the soft robots can be achieved by fusing with other sensing mechanisms, e.g., capacitive, resistive, photonic, etc., towards a multifunctional sensory system [229,230]. As shown in Figure 4f, Bu et al. proposed a triboelectric-photonic smart skin for robots’ tactile and gesture sensing [226]. By doping the aggregation-induced emission (AIE) powder into the flexible silicone rubber substrate with the grating-structured metal film for exposure area adjustment, the photoluminescence and photocurrent are tunable for continuous tensile measurement under the lateral stretching range of 0–160%. Due to the high electron affinity ability of the silicone rubber, the triboelectric output can also be generated when external stimuli occur, which can be used for vertical static pressure detection with a maximum sensitivity of 34 mV Pa⁻¹. After integrating onto a robotic hand, the precise joint bending and touch pressure monitoring can be achieved simultaneously, demonstrating the applicability of the triboelectric-photonic fused sensory system for soft robot-related HMI applications. Another hybridizing strategy based on the triboelectric and potentiometric for soft robot perception was reported by Wu et al. as illustrated in Figure 4g [227]. Inspired by the slow adapting (SA) and fast adapting (FA) capabilities of human skin, the potentiometric sensing mechanism that is sensitive to the static or slowly varying stimuli is utilized to detect the compressive strain induced by the external force, while the triboelectric mechanism suitable for dynamic stimuli sensing can provide the instantaneous signal information at the beginning or ending moments of the stimulation. With this multifunctional sensing capability brought by the complementary effects of these two mechanisms, more valuable information, including the pressure and duration during the process of approaching, touching, holding and releasing, can be obtained towards a more detailed object manipulation monitoring for soft robotic grippers.

Expert for these humanoid robots, animal-like exploration robots have also become a hot topic recently. By sending these robots to natural or harsh environments where people cannot go, valuable sensory data or physical tasks could be achieved for environmental monitoring or space exploration purposes [231,232,233,234]. To endow the robots with the capability to respond to complex environmental situations, e.g., avoiding obstacles, etc., varying advanced sensing technologies based on the mechanisms of piezoresistive, piezoelectric, optical and magnetic have been investigated [235,236,237,238]. However, thanks to the advantages of lightweight and low power consumption, TENGs have become a new strategy for developing self-sustainable exploration robots [230]. As shown in Figure 4h, inspired by the hair-based sensory system that animals used to explore the environment, An et al. developed a TENG-enabled self-powered biomimetic whisker mechanoreceptor for robotic tactile sensing [228]. The sensor consists of a fluorinated ethylene propylene (FEP) layer serving as the animal whisker, covered by a biomimetic hair follicle made of two metal electrodes. When the whisker swings between the two electrodes, the potential distribution will change due to the triboelectricity and electrostatic induction, which can be used to reflect the deflection direction and amplitude based on the signs and magnitude of the transferred charges, respectively. When integrated into a quadruped robot, the biomimetic whisker can help robots detect the movements of surrounding objects by analyzing the amplitude and frequency of the vibrational signal. The real-time pressure applied on its feet can also be achieved to reflect the gait and ground environment information, which can effectively help it pass complex roads in harsh environments for exploration applications.

5. HMIs for Smart Home Applications

The rapid development of numerous smart electronics enables a large variety of applications in the ambient environment to realize the smart home, with the intelligent monitoring and response systems for healthcare monitoring, elderly/children care, fall detection, body motion monitoring, automation, and security [239,240]. The current mainstream technologies for smart home applications include camera-based image recognition and commercial humidity/temperature sensors [241]. Although these technologies are continuously developing, drawbacks still exist such as video-based privacy concerns, large power consumption, bulky volume, and rigidity. Considering the fast development of the energy harvesting/self-powered sensing technology (triboelectric, etc.), lowpower HMIs can be realized with high convenience, low cost, and self-powered sensing ability to reduce the overall power consumption and extend the lifetime of the system [5,242,243]. Additionally, because of the energy harvesting capabilities, these HMIs can also scavenge the ambient energy and convert it into electricity for potential wireless communication.

As a good substitute for mechanical switches, flexible touchpads with multiple arrays/units have been investigated frequently by using TENG technology to achieve self-powered HMIs to interact with household appliances [244,245,246,247]. For these multi-array touchpads, there are two common ways for wire connection. One is connecting each unit to an output channel, which will lead to a messy wiring layout with complex processing circuits. Another method is to build cross nodes of X-axial channels and Y-axial channels, by which the total output channels will be reduced greatly, but the issues of crosstalk between the intersected channels could not be avoided. To solve these problems, Pu et al. developed a subdivision-structural 3D touchpad as illustrated in Figure 5a [248]. With the novel subdivision structure, the overlapping area between the intersected electrodes is significantly reduced, which can effectively suppress the crosstalk problems and improve the position sensing resolution. Additionally, the pressure sensing function was also realized by integrating another three-layer TENG unit for the touch or press differentiation purpose. This sensing array was successfully further designed into an anti-peek built-in code lock, where the information of the position and pressure can be integrated to form a more complex access password compared with the methods that are only based on the number location, effectively improving the safety factor of the authorization system in the smart home.

Another smart method to simplify the output channels of touchpads is using binary codes to define more functions with as few electrodes as possible. As shown in Figure 5b, Qiu et al. reported a sliding-mode TENG-enabled control disk interface based on the binary coding mechanism [249]. Two sensing electrodes with specific patterns are attached to the panel, where one serves as the binary bit “1”, and the other serves as the binary bit “0”. Due to the variation in the width and location of these two electrodes along with different sliding directions, eight sensing transitions can be achieved based on the 3-bit binary code, e.g., “001”,”010”, etc., for smart appliances control. Additionally, the sliding speed can also be utilized to adjust the light brightness or fan speed. After integrating a soler cell to form a hybrid energy harvester, the self-sustainable capability could be achieved by scavenging both the light energy and the mechanical energy from hand tapping. Moreover, this control disk can also be designed into a password input interface with more than 262,000 potential combinations for smart home authentication applications.

Apart from the password input panel, the barcode is also a widely-used strategy as the information carrier for personal identification/authentication applicable in smart home applications [255,256], and can be developed with the TENG technology towards a self-powered identification system. As depicted in Figure 5c, a triboelectric-based transparent secret code was proposed by Yuan et al. [250]. The information is hidden in the patterned indium tin oxide (ITO) electrodes, with a fluorinated ethylene propylene (FEP) film covered on top serving as the tribo-layer. The elongated design of the ITO stripes can effectively enlarge the contacting area within a short sliding time, and the different lengths will contribute to the variation of the output amplitudes, which can be further translated into the binary information of “1” or “0” with a reasonable threshold value by a sliding-check or roll-to-roll reader for security defense purposes. However, the peak height enabled sensing method is sliding velocity-dependent, and the non-uniform sliding speed may result in the error in the peak amplitudes, as well as the corresponding identified information. To solve this issue, Chen et al. proposed to use a reference barcode component to improve the reliability of the identification system [251]. As illustrated in Figure 5d, the information card is double-sided, where one side is patterned with the standard barcode electrodes with equal intervals as the reference component, and the other side is patterned with the information barcode that is aligned, swiped and measured simultaneously with the reference one. By comparing the number and positions of the generated positive and negative peaks from two sides, the coded information could be easily converted into the digital information of “1” and “0” with high accuracy even under non-uniform sliding speed by human hands, demonstrating a more reliable coding method for self-powered access system in practical usage.

In this fast-paced era, people are under increasing pressure from work and life, leading to more and more people being annoyed by the sleeping disorder, which may further increase the risk of other health problems, e.g., obesity, heart disease, and diabetes [257]. To realize the long-term sleep behavior monitoring for sleep quality assessment, many works have been done based on real-time pressure sensing [258,259]. However, most of these technologies are still restricted by the issues of low sensitivity, high fabrication cost, and non-washability, limiting their practical applications for large-area sleeping monitoring in daily usage. Recently, the fast development of TENG-based textile technology shows the great potential of TENGs to be further designed into smart clothes for wearable or household scenarios with high sensitivity and low cost [260,261,262]. By utilizing this technology, a large-scale and washable smart textile based on TENG arrays was developed by Lin et al. as shown in Figure 5e [252]. The proposed bedsheet has three layers, where a layer of wave-shaped PET film is sandwiched by two layers of Ag-coating conductive fabrics. When external stimuli occur, the applied pressure will enlarge the contact area between the PET layer and the two conductive layers, thus resulting in the electrical potential change and generating the outputs. With this optimal design, the smart textile shows a good sensitivity of 0.77 V Pa⁻¹ with a fast response time (<80 ms), as well as high durability and stability even after being washed in tap water. By connecting multiple TENG units to form a sensing array, the information of the body’s posture, position and pressure distribution over an entire night could be collected, providing a reliable dataset for analyzing the sleep quality. Additionally, the smart bedsheet can also serve as a warning system to prevent the elderly from falling off the bed, demonstrating a new possibility to realize the real-time remote healthcare service in the smart home.

To avoid the privacy concerns introduced by cameras, floor-embedded sensors can be implemented to extract the abundant sensory information associated with human activities in the smart home, such as the indoor position and gait-based individual identity [263,264,265]. Thanks to the advantages of low cost, easy fabrication and wide choices for triboelectric materials, TENG-based sensors commonly show good scalability and are compatible with large-scale manufacturing, making them suitable to be further designed into the floor-based HMIs [266,267,268]. A self-powered identity recognition carpet system enabled by TENG-based e-textile for safeguarding entrance was reported by Dong et al. as illustrated in Figure 5f [253]. The carpet consists of 128 black and white squares of the same size, where the black blocks are attached with the TENG fabrics serving as the sensing region, and the white blocks are used to reduce the interference between the adjacent sensing regions. According to the generated output peaks in these black sensing blocks, the walking trajectories of a visitor can be mapped with high accuracy and good stability, which can be further compared with the correct password path to validate the authentication. However, in this design, each sensing block is connected to an independent output channel, which means a complex wire connection and high signal processing cost. To simplify the output channels of the multi-array sensing system, a more advanced design of the smart mat was reported by Shi et al. as shown in Figure 5g [254]. Six distinct electrode patterns with varying coverage rates, i.e., 0% to 100% with 20% intervals, were designed and fabricated by screen printing, acting as sensing arrays for this floor mat. Due to that different electrode areas will contribute to different amounts of induced charge, the arrays are self-distinguishable and can be connected in an interval parallel manner to form a 3 × 4 mat array with minimal two-electrode outputs. So that the indoor positioning and activity monitoring can be achieved with minimal output terminals and minimized system complexity, which also benefits the backend signal processing and data analysis. Furthermore, with deep learning enabled data analytics, the identity information associated with gait patterns can be extracted from the output signals, and high recognition accuracy of 96% could be achieved for 10 persons based on their specific walking gaits, enabling diversified applications in the smart home such as position sensing, activity/healthcare monitoring, and security.

6. ML-Enabled Advanced HMIs

In terms of complex data analysis, the new trend of AI-enabled machine learning has shown a new direction for enhancing the functionalities of sensors [1,269,270]. Due to the powerful feature extraction capabilities of machine learning, more comprehensive/detailed sensory information can be utilized to realize diversified applications, e.g., gesture/pose estimation, voice recognition, object recognition, etc. [81,87,164], for advanced human–machine interactions as illustrated in Figure 6.

Recently, several AI-enabled TENG-based HMIs have been successfully developed [89,92,276,277,278] based on the algorithms of support vector machine (SVM), neural network (NN), etc., where the subtle features hidden in the triboelectric waveform, including contact sequence, impact vibration, etc., have been proven to effectively enhance the recognition capability of the intelligent sensory system. Compared with state-of-art works [279,280,281] that using a large number of resistive/capacitive sensor nodes for high-accuracy ML analysis, the minimalistic approach of TENGs shows comparable performance with significantly lower power consumption. Here, in this section, some typical examples of TENG-based intelligent HMIs for diversified applications are reviewed.

For cybersecurity applications, keystroke dynamics enabled authentication system has been proven as an effective approach to enhance the security level based on people’s typing attributes with non-invasive monitoring characteristics [282,283,284]. A TENG-based smart keyboard for keystroke dynamics monitoring was developed by Wu et al. as illustrated in Figure 6a [271]. There are a total of 16 silicone-based keys with high flexibility and stretchability forming a multichannel keypad array, where each key consists of a contact-separation-mode TENG to convert the typing behavior into electrical signals, and a shield electrode to minimize the environmental interference. By using an analog-to-digital converter (ADC) to collect the open-circuit-voltage signals, keystroke-related features, i.e., typing latencies, hold time and signal magnitudes, can be acquired simultaneously with specific signal processing, e.g., denoising, baseline elimination, etc. Following the principal component analysis (PCA) for feature dimensionality reduction, a multi-class SVM classier was utilized to recognize the identities of 5 users based on the established dataset (150 sets of data for each user), and high accuracy of 98.7% could be achieved, showing the feasibility of the keystroke dynamics enabled authentication system for practical usage.

In addition to the keystroke dynamics, identification based on gait analysis is also a promising technology for biometric authentication applications [46,285]. With the help of artificial intelligence, complex personal information regarding the identity, health status as well as real-time activity of the users could be delivered at the same time by analyzing the gait patterns acquired from the floor- or sock-based sensory system. Zhang et al. proposed deep learning-enabled TENG socks for gait analysis as depicted in Figure 6b [272]. A textile-based TENG pressure sensor, consisting of a silicone rubber film with patterned frustum structures as the negative tribo-layer, a nitrile thin film as the positive tribo-layer and two conductive textiles as output electrodes, was fabricated and integrated onto a smart sock for gait monitoring with high sensitivity (0.4 V kPa⁻¹) and large sensing range (>200 kPa). With an optimized four-layer one-dimensional (1D) convolutional neural network (CNN) model for automatic feature extraction from the original walking spectrums, a high recognition accuracy of 96% could be achieved for 5 participants with varying weights, and could still be maintained higher than 93.5% when the number of people increased to 13, making it applicable for most indoor scenarios, e.g., home or office. By combining this intelligent sock with an IoT module for wireless communication, an artificial intelligence of thing (AIoT)-enabled two-stage recognition platform was established at the cloud server to realize the functions of family member identification and real-time indoor activities, i.e., run, walk and jump, monitoring simultaneously, revealing its potential for future smart home applications.

With AI for comprehensive sensory information extraction and autonomous learning, sophisticated hand gestures could be discriminated for glove-based HMIs towards advanced control or sign language interpretation applications [286,287]. Though several works have demonstrated the feasibility of developing TENG-based sign language perception platforms with minimalistic design and high recognition accuracy [131,139,288], most of them are limited to the identification of only several discrete and simple words or letters and are not suitable for real-time sentence recognition. To deal with these issues, Wen et al. developed a more advanced TENG-enabled sign language recognition system with improved glove design and training strategies as shown in Figure 6c [273]. Apart from the strain sensors that are mounted on each finger for finger bending monitoring, some sensors are placed on the wrists, fingertips and palms of the signers to capture subtle features towards more comprehensive sign language gestures. By utilizing a 5-layer CNN for feature extraction, high accuracy of 91.3% and 93.5% could be achieved for 50 words and 20 complete sentences respectively based on a non-segmentation method. To further extend its potential for new/never-seen sentences recognition, a segmentation strategy was used, where the signal spectrum of a whole sentence was split into word units first and then based on the correlation between the word units and the sentence, the original sentence’s information could be reconstructed by the AI framework with an accuracy of 85.58%. With this novel learning approach, new/never-seen sentences could also be identified with an average accuracy of 86.67%, showing a new methodology to effectively expand the dataset and improve the practicality of the sign language recognition system.

As mentioned in the section on robotics-related HMIs, TENG-based sensors have been frequently investigated for developing low-power-consumption robot perception systems due to the advantages of high flexibility and self-powered property. However, most of the abovementioned works mainly focus on the simple manual analysis of the collected sensory information [221,223], e.g., deformation, tactile, etc. For realizing more advanced interactive functions, e.g., pose estimation, surface roughness perception and object recognition, AI-enabled analytics are needed, to capture more valuable features and endow the self-discrimination ability [225]. A TENG-enhanced smart soft robotic manipulator for AIoT enabled virtual shop applications was reported by Sun et al. as shown in Figure 6d [274]. A TENG bending sensor consisting of a rotating gear was utilized to monitor the real-time deformation of the pneumatic finger according to the generated peak numbers during the stretching process, and a TENG tactile sensor with distributed electrodes design was used for contact position and area detection. With the aid of 1D CNN ML algorithm for data processing, an intelligent robotic gripper that fuses these two kinds of sensory information could be easily achieved to realize a high recognition accuracy of 97.143% for 28 grasped objects with different shapes and sizes. Moreover, temperature distribution information could also be implemented by a poly(vinylidene fluoride) (PVDF) pyroelectric temperature sensor, towards a comprehensive and fully self-powered perception system. This AI-enhanced smart gripper with multifunctional sensing capability was further applied in a digital-twin-based virtual shop to provide users with real-time feedback information of the goods, as well as a more immersive experience of online shopping, showing great potential to realize advanced human–machine interactions in unmanned working space.

Due to the existence of environmental interferences, the performance of sensors could be affected in different environmental conditions, e.g., humidity effect for triboelectric sensors, etc. [127], which may result in poor stability of the established recognition system. For those TENG HMIs based on the sensing mechanism of grating sling or output ratio as mentioned above, though the output amplitudes will be influenced under varying environmental conditions, the generated peak numbers or the output ratio of distributed electrodes will not change, resulting in strong resistance to environmental interference. So, the ML result achieved from such kinds of output data will show better robustness without the effect of environmental elements. While for the ML results obtained from the real-time output spectrum, the variation in voltage amplitude under different environmental conditions will inevitably affect the results of machine learning. However, if we collect the data under different environmental conditions and combine them into a more generalized data set, i.e., each category contains the data that captured under different environmental conditions, the influence of environmental elements on accuracy will be avoided to a certain extent due to the more generalized trained model. Similarly, the influence from the different human behavior habits could also be avoided by enhancing the generalization ability of the dataset, where the data from different users are collected, thus taking into account the individual differences.

Another possible solution for this issue is to fuse sensors based on different mechanisms to build a more robust sensory system by utilizing the complementary effects of different sensors [289,290,291]. In Figure 6e, Wang et al. proposed a bioinspired deep-learning-based data fusion architecture that integrates the vision data and somatosensory data for high-accuracy gesture recognition in a harsh/dark environment [275], where the somatosensory information could contain complementary features for maintaining recognition accuracy especially when images are noisy and under- or over-exposed. To ensure high compatibility with visual information, a highly transparent stretchable strain sensor using single-walled carbon nanotubes (SWCNTs) was developed, which can collect high-quality somatosensory data without affecting the visual information. With the visual data captured by commercial cameras, a bioinspired somatosensory-visual (BSV) associated architecture was then reported for further multimodal data fusion, which can mimic the biological visual and somatosensory interactions in the area of the multisensory neuron of the human brain. By using this approach for human gesture recognition, high accuracy of 100% can be maintained even with low-quality images, proving the complementary effect of the somatosensory information on vision-based gesture recognition. Additionally, this architecture has also been demonstrated for more accurate robot manipulation, where 98.3% accuracy could be achieved with enough illumination and 96.7% accuracy could still be maintained in the dark environment, illustrating its applications for human–machine interactions in harsh environmental scenarios. Such a data fusion strategy provides a good development direction for future TENG-based HMIs with better stability and applicability.

7. Haptic-Feedback Enabled HMIs

The completeness of an HMI system not only rely on the dexterous sensing units which can monitor various human physiological signals and motions, but also requires a specialized feedback module that can provide necessary stimulations to assist the cognition of the manipulation status, so that a control-feedback loop is able to be established [292]. Noticeably, the consistency between the artificially stimulated sensation and the real sensation to humans is the key to improve the user experience and the value of the feedback information. Therefore, a majority of the feedback research is also focusing on the biomimetic stimulators, include kinesthetic feedback which can reflect the spatial movement of different body parts, and cutaneous feedback which is in charge of performing the tactile and thermal stimulations to various skin’s receptors [293,294,295,296].

With the concern of conformability to the human body, wire and pneumatic actuators, as commonly used actuation techniques, are also adopted in feedback systems for achieving kinesthetic actuation [297,298]. Kang et al. reported a polymer-based soft wearable exo-glove using tendon-driven feedback, as shown in Figure 7a [299]. The main functional part consists of a body with stretchable designed finger modules, and the thimbles which are connected to the corresponding finger modules. The actuation wires are then covered by the sheaths embedded in the main body and linked to the motor box. This device can help disabled people to recover the capability of hand grasping. Motor-based vibrators are frequently used in mobile phones and joysticks as feedback units. Yu et al. developed a skin-integrated wireless haptic interface for VR and remote interactions by making the electromagnetic vibrator array (Figure 7b) [300]. This elastomer packaged device can be directly attached to the human skin, and it is powered wirelessly via an NFC coil. The design of serpentine Cu connectors ensures the stable performance under strain.

Dielectric elastomer actuator (DEA) is becoming another popular research direction in recent years. The electric field induced deformation of DEA can be reshaped into vibration, stretching, bending and other modes, by applying the different designs [312]. In Figure 7c, Wang et al. presented a hydraulically amplified self-healing electrostatic actuator that can mimic the muscle contraction motions under the applied electric field [301]. The actuator is made of rectangular polymer shells filled with a liquid dielectric, and the electrodes are coated at both ends. Hence, the applied voltage can induce a Maxwell stress cause the electrode to zip together and the shell is then contracted due to the hydraulic pressure. This actuator achieved a maximum linear contraction of about 24% at a loading of 0.2 N. On the other hand, as shown in Figure 7d, a feel-through low-voltage DEA made by multi-layer PDMS based DEA sandwiched by single-walled carbon nanotube (SWCNT) electrodes for fingertip haptic feedback is demonstrated by Ji et al. [302]. The combination of three active layers is only 18 µm thick, which can ensure the mechanoreceptors of the skin remain sensitive to external stimuli. By applying the voltage, the surface area of the elastomer increases or decreases to stretch or compress the skin for feedback. Similarly, Pyo et al. presented a pyramid microstructured DEA layer. The applied AC voltage can also deliver the vibrational stimulus (Figure 7e) [303]. Interestingly, Hwang et al. developed a light-driven and low-power vibrotactile actuator using thin poly(3,4-ethylenedioxythiophene) doped with p-toluenesulfonate (PEDOT-Tos) and PET film, as illustrated in Figure 7f [304]. The irradiation of near-infrared (NIR) light from LED can initiate a thermoelastic bending deformation due to the mismatch of thermal expansion coefficient. Therefore, with the assistance of LED array, a programmable vibrotactile feedback system is demonstrated.

There are also researches focus on electrostatic force-based feedback. The applied voltage can cause the attraction of two electrodes under the difference of electrical potential. In Figure 7g, Hinchet et al. designed a high force density electrostatic clutch for making the VR feedback glove [305]. The clutch is simply made by the conductive textile with poly vinylidene fluoride, trifluoroethylene, 1,1-chlorotrifluoro- ethylene (P(VDF-TrFE-CTFE) based high friction insulation layer. The proposed device can generate frictional shear stress up to 21 N cm⁻² at 300 V, which can be used for blocking the finger motion during VR events. In the meantime, Leroy et al. presented a multimode electrostatic actuator with hydraulically amplified haptic feedback, as illustrated in Figure 7h [306]. The liquid dielectric is encapsulated within a chamber formed by the flexible membrane and central stretchable membrane. The electrode pairs are then located at the surroundings of the center. The voltage-induced electrostatic force will force the electrode to attract and squeeze the outer region of the chamber, and hence, the central membrane will be expanded under hydraulic pressure. By controlling the electrode designs and activation sequences, the device can realize the multimode haptic feedback.

To enhance the feedback functionality of the pneumatic actuator with minimal air inlet, various techniques are also integrated to tune the activation region of the device. Besse et al. reported a large reconfigurable pneumatic haptic array with the aid of flexible shape memory polymer (SMP) membrane, as depicted in Figure 7i [307]. The heater can change Young’s modulus of the SMP membrane and allow it the deform under positive or negative air pressure. As a result, the array design of the chambers and membranes enables the dense reconfigurable tactile system. On the other hand, Qu et al. developed a refreshable braille display system based on pneumatic actuation and TENG-based DEA (Figure 7j) [308]. A similar air chamber with a membrane design is adopted. As mentioned earlier, the DEA can be stretched under applied TENG voltage. Together with the air pressure, the DEA membrane is then raised up to form a braille dot.

Moving forward, TENG is also directly used as a feedback unit via the generated voltage. As shown in Figure 7k, Shi et al. designed a TENG array with ball electrodes to make a feedback system for VR [309]. When the slider is sliding across the TENG array, the electrical output can be generated via the triboelectrification effect. With the direct contact of the ball electrodes to human skin, the TENG electrostatic discharge can be delivered as electrical virtual tactile stimulation. Hence, the sliding trajectory can be sensed on the skin as a feedback function.

Thermal sensation, as another important function of human skin, can bring a new dimension of feedback to enrich the reconstruction of the environment in cyber or remote space. The collection of temperature information is also critical to prevent potential damage during remote control. Oh et al. developed a multimodal sensing and feedback glove with a resistive strain sensor, vibrator, and thermal feedback units (Figure 7l) [310]. By applying the direct ink writing technique, eGaIn liquid metal is printed into meandered shape to make the units with both strain sensing and thermal feedback functions via the power supply. Moving forward, the capabilities of both heating and cooling is the next challenge for those wearable and flexible feedback systems. As illustrated in Figure 7m, a skin-like thermo-haptic device with thermoelectric units is developed by Lee et al. [311]. This device consists of Cu serpentine electrode, thermal conductive elastomer, and n-type/p-type thermoelectric-based pellets. Based on Peltier effect, this flexible thermo haptic skin can alternate the heating and cooling modes with the temperature difference of 15 °C.

In general, haptic feedback technologies can drastically boost up the capability of information interpretation, instead of receiving digitized data. Equipment with wearable feedback systems not only paves the way for immersive interaction, but also improves the efficiency of manipulation via rapid cognition of the target environment [313,314]. The research of more feedback parameters, such as smell and airflow, will eventually establish a fully biomimetic feedback system.

8. Towards Self-Sustainable/Zero-Power/Passive HMI Terminals in the 5G/IoT Era

In this information age, the expansion of the Internet of things not only comes from mobile phones, tablets, and computers, but also thanks to other millions of smart devices [315] that are connected to the Internet through wireless communication technologies, e.g., WI-FI, Bluetooth, etc. Though the sensors based on self-powered mechanisms [154,316], i.e., triboelectric and piezoelectric, in the IoT devices may do not need a power supply, such signal transmission modules still mean high power consumption, especially when there are massive sensor nodes under the IoT framework. In addition, as the traditional power supply for portable devices, batteries experience drawbacks of high contamination and limited lifespan, as well as the annoying replacement or recharging process. To effectively extend the lifespan of portable or remote devices, energy harvesters, which can convert biomechanical energies or wasted energy in the ambient into electricity based on the mechanisms of electromagnetic, piezoelectric, and triboelectric, have been frequently investigated recently to replace batteries as the power supply units towards self-sustainable IoT systems/HMIs [317,318,319].

As shown in Figure 8a, an intelligent walking stick, consisting of a top press TENG for contact point sensing, a rotational TENG for gait abnormality detection and a rotational electromagnetic generator (EMG) for energy harvesting, was reported by Guo et al. [320]. With the deep-learning-enabled data analytic, complicated sensing functions including disability evaluation, identity recognition and activity status identification could be implemented through the TENG sensor outputs. Additionally, the linear-to-rotary structure enables the integrated EMG to harvest the ultra-low-frequency biomechanical energy from human motions and provide an average output power of 27.5 mW under 1 Hz stimuli. By stacking two EMGs for higher power output and using customized circuits for power management, a self-sustainable system capable of locomotion tracing and temperature/humidity monitoring was successfully achieved, where a temperature/humidity sensing module could be driven continuously with 1 Hz working, and a GPS and wireless module could work 6 s after every 90 s charging. Moreover, by transmitting the TENG sensor signal to an AI cloud/server via the 5G network for further analysis, real-time monitoring of the user’s location and well-being status could be achieved in outdoor environments, showing the feasibility of sustainable HMIs enabled by energy harvesters for future IoT applications.

However, current sustainable HMIs enabled by the integrated energy harvesting units are not suitable for continuous wireless monitoring or fast information exchange due to the inevitable charging phase [61,324,325,326]. The additional energy harvesters or power management circuits also increase the size and cost of the whole device, reducing the applicability for wearable scenarios. To these issues, some works chose to utilize the high-voltage transient of the TENG to realize wireless coupling, and achieve complete zero power consumption on the sensor terminals with the minimalistic design [327,328]. As shown in Figure 8b [321], Mallineni et al. reported a wireless TENG tactile patch consisting of graphene polylactic acid (gPLA) nanocomposite and Teflon serving as the tribo-layers, which can generate a high electric field in the surrounding air enabling ~3 m wireless sensing, thanks to the extremely high output voltage (>2 kV) through simple mechanical stimuli, e.g., hand tapping. By connecting a customized signal processing circuit to the receiver, the hand tapping signal could be real-time collected wirelessly without a power supply or even signal transmitter at the sensor side, realizing a truly zero-power sensing terminal. Such a self-powered HMI could function as a controller for diversified smart home applications, e.g., activating lights, displays, photo frames, or even security systems via Morse code based passwords, showing great advantages in terms of size and cost when compared with conventional HMIs that need bulky power supply components or wireless circuit modules. However, the wireless sensing solution using signal amplitude as the sensing parameter is easily affected by the ambient environment, so a more advanced self-powered wireless network that detects the frequency of the TENG output via the mechanical switch enabled frequency boosting up strategy was proposed by Wen et al. as shown in Figure 8c [322]. With the enhanced frequency brought by the switch-induced instantaneous discharging, the TENG output signal could be easily transmitted wirelessly through a couple of coils, serving as a reliable reference for force calibration with high sensitivity (434.7 Hz N⁻¹) and a large sensing range thanks to the stable resonant frequency. By changing the wire connection of TENG sensor layers, i.e., in series or parallel, or adjusting the capacitors connected to different pixels in a sensor array, the tunable frequency could be achieved for complicated manipulation, e.g., the multiple-freedom-degree 2D car control or 3D VR drone control, revealing the feasibility of zero-power HMI based on the TENG-enabled frequency shift wireless sensing for IoT applications.

Another strategy to realize battery-free sensor nodes with continuous sensing capability under the IoT framework is to use passive wireless technologies, including near-field communication (NFC) [329,330], radio frequency identification (RFID) [331,332], and surface acoustic wave (SAW) sensors [333,334]. Compared with the abovementioned wireless transmission methods based on the TENG output enabled electric/magnetic coupling, these passive wireless approaches commonly show better stability in varying environmental circumstances, as well as smaller wireless sensing and receiving units. A passive wireless TENG sensor using a surface acoustic wave resonator (SAWR) was reported by Tan et al. as illustrated in Figure 8d [323]. By connecting a TENG force sensor to a SAWR, the frequency and amplitude of the SAWR’s response signal can be modulated via the TENG output, making the radio frequency (RF) signal contain the tactile related sensory information which can be further extracted by the RF reader through the demodulating process. All the energy in this wireless transmission system is provided by the remote RF reader, making the sensor terminal a completely passive sensing node capable of continuous monitoring with a transmission distance larger than 2 m and high sensitivity of 23.75 kHz V⁻¹. This system could also act as a passive controller for trigger signal detection and Morse code based wireless manipulation, or as a wireless matrix keyboard with resonators working under different center frequencies, showing the promising prospects of distributed sensing or human–machine interaction in this 5G/IoT era.

9. Conclusions and Prospects

With the advantages of the wide material choices and diversified structural designs, TENG-based sensing technology has been frequently investigated towards low-power HMIs in this 5G/IoT era. In this review, we systematically summarize the key technologies and progress of TENG-based HMIs in terms of different application scenarios, including wearable HMIs, robotic-related HMIs and HMIs for smart home applications. By using the ML analytic for automatic feature extraction, advanced interactions, such as biometric authentication, gait analysis, gesture recognition and object recognition, can be implemented to enhance the functionalities of TENG-based HMIs towards intelligent interactive systems in the age of AI. The state-of-art haptic feedback technologies have also been reviewed and discussed, which yields a promising research direction for developing self-driven immersive HMIs when bringing in the TENG technology. Furthermore, the integration of self-powered TENG sensors and energy harvesters or passive wireless transmission technology also provides the possibility to realize battery-free/self-sustainable HMI terminals.

Despite the viable progress in developing TENG-based HMIs for diversified applications, there are still challenges that remain to be solved for the current solutions. First, the peak-like output signals make the TENG-based sensors only suitable for dynamic stimuli sensing and cannot be used to detect the continuous variation due to the fast shift of electrical states. Though methods based on grating-sliding mode TENG have been frequently reported [133,172], the resolution is limited by the size and spacing of the grating, and is difficult to reach a very high level without advanced fabrication processes [335], e.g., MEMS process, screen printing, micro-machine, etc., which means high fabrication cost. Other methods based on high-impedance readout circuits [149] or nanophotonic modulators [135,336] need complicated measurement systems, which are unlikely to be made into portable sensing devices for daily usage. Therefore, a reliable and convenient way to achieve continuous sensing capability for TENG-based HMIs that is compatible with IoT mobile platforms is needed. Second, though the energy harvesting or passive wireless transmission technology shows the feasibility to realize battery-free/self-sustainable HMI terminals with TENG sensors, limitations still exist, e.g., intermittent operation for energy harvester integrated system due to the charging phase, short transmission distance for electric/magnetic coupling or passive wireless transmission, etc. Further investigation on battery-free/self-sustainable systems with continuous operation capability and long wireless transmission distance is still desired. Third, the durability of TENG-based HMIs for long-term usage, and the robustness of the sensor performance in varying environmental conditions are also major concerns, considering the inevitable friction layer loss and the triboelectric charge loss under the high humidity condition, which may cause instability of the output and bring great challenges to the sensor collaboration. Moreover, TENG sensors can be fused with other sensing mechanisms towards a multimodal sensor fusion with AI analysis [337] for multifunctional purposes or more robust performance under varying environmental conditions. In this regard, the seamless implementation of the low-power/self-sustainable TENG-based HMIs will definitely shed light on the harmonic coexistence of humans and machines in the future of the IoT era, along with the immersive and efficient interactions in numerous scenarios.