Wearable Device for Residential Elbow Joint Rehabilitation with Voice Prompts and Tracking Feedback APP

Abstract

In this paper, we propose a wearable device for residential elbow joint rehabilitation with voice prompts and a tracking feedback app (WDRTFAPP). We have developed the app as well as the Arduino embedded system, which we have integrated together. In this research, the patients were simulated by our team not real patients. By using this wearable device, the elbow joint rehabilitation could be executed at home for the simulated patients with mild and moderately mild elbow joint symptoms. During the rehabilitation, data captured by the wearable device were sent to the tracking feedback APP, using automatic real time via Bluetooth transmission. After TFAPP received the rehabilitation data from the wearable device, the rehabilitation data was sent to the cloud database by Wi-Fi or 5G communication automatically in real time. When the performance of the elbow joint rehabilitation was incorrect the patients received a voice prompt by TFAPP. The simulated patients could query their rehabilitation data using different search strategies, namely by date or TFAPP, at any time or location. In the experimental results, it showed that the correct detecting rate of elbow joint rehabilitation could be up to 90% by WDRTFAPP. The medical staff also could track the rehabilitation status of each simulated patient by the tracking feedback APP (TFAPP) with remote accessing, such as the Internet. Moreover, the rehabilitation appointments could be set up by the clinical staff with TFAPP, using the Internet. Furthermore, the medical staff could track the rehabilitation status of each simulated patient and give feedback at any time and location. The costs of the rehabilitation could be reduced (in terms of time and money spent by the simulated patients) and the manpower required by the hospital.

1. Introduction

Recently, combining wireless communication with other fields’ research is an important issue since the technological development changes suddenly. Intelligent medical is one of the interdisciplinary fields whose research is combining medical with information communication techniques. In the existing medical treatments, the rehabilitation treatment often needed more medical resources since the physician, nurse, and physical therapist were all required in the process of rehabilitation treatment. The numbers of people with rehabilitation symptoms were in proportion to the age. The age of people with rehabilitation symptoms was decreased due to advances in medical technology. Hence, the numbers of people with rehabilitation symptoms were increased recently [1,2,3].

The treatment of rehabilitation was different based on the different parts of body and symptoms. Some rehabilitation could be treated at home without in-hospital rehabilitation [4,5,6,7,8,9,10,11]. Among the existing rehabilitation symptoms, elbow joint rehabilitation occurred most often, such as tennis elbow [12,13,14]. After discussion with orthopedists, the elbow joint rehabilitation most often occurred and could be executed at home without in-hospital rehabilitation as is the case with other rehabilitation symptoms [15,16,17,18,19,20,21,22,23,24,25,26,27,28,29].

From a paper survey, to our best knowledge, the existing rehabilitation systems were divided into a visual recognition and non-visual recognition system. In the rehabilitation systems with visual recognition, it was often combined with virtual reality (VR) and motion-sensing [17,18,19,20,21,22,23]. In [17], it integrated the exoskeleton systems with virtual reality (VR) to improve the traditional rehabilitation. The exoskeleton systems were composed of a load cell and a servo motor. The load cell detected the force of rehabilitation. The servo motor supplied the assisted force of rehabilitation and obtained the open angle of rehabilitation. These factors were used as the inputs of VR to perform the rehabilitation exercises with the simulated weight and real weight.

In [18], the authors proposed a telerehabilitation combined with VR and Kinect II motion capture sensor for rehabilitation at home. The therapists could monitor and adjust the rehabilitation program of each patient by telerehabilitation, such as a website. The patient did the exercises of rehabilitation by VR and Kinect. The data is logged by the system. Hence, the therapists could examine the status of rehabilitation and adjust the program of rehabilitation for the patients.

In [19,20], the rehabilitation systems were composed of VR and motion-sensing, such as Kinect. The patients did the exercises of rehabilitation based on VR. The motion of rehabilitation was captured by Kinect. In [21,22], the rehabilitation systems were also composed of VR and motion-sensing, the same as [19,20]. However, the motion-sensing technology used was by Wii. In [23], the rehabilitation system was only built by Kinect. The patients could know that the action was correct or not and then could adjust the motion by the captured image from Kinect.

Although the rehabilitation by VR or motion-sensing could be interactive and interesting, it was still suitable for some patients only. Firstly, it needed the additional hardware cost. Secondly, the user interface was not often friendly since it often needed the additional installation and settings, especially for elder patients. Finally, it required the additional sufficient space for hardware implementation and capturing by motion-sensing. The rehabilitation was thus constricted to be executed in the fixed location without movement to other places. These issues showed that the rehabilitation by VR or motion-sensing could not be applied to all patients.

For the non-visual recognition system, it was divided into rehabilitation assistance and rehabilitation detection. In [24,25], the authors designed a robot arm to be the simulated human arm as the exoskeleton to rotate the motion for some critical patients who cannot do rehabilitation by themselves. Although the rehabilitation assistance could help the patients to do rehabilitation automatically without anyone’s assistance, they were still constrained by the external hardware space and cost, because these rehabilitation assistance systems were the tethered systems. Hence, both of the visual recognition rehabilitation system and rehabilitation assistance system could not be applied for the residential rehabilitation due to the constraint of fixed space and hardware cost.

To address the above issues, the wearable technology for rehabilitation was thus proposed. In [26,27,28], the authors proposed the rehabilitation detection combined with wearable technology. In [26], the authors designed the rehabilitation system with E-AR sensor to detect the electrocardiography, ECG, and photoplethysmography, PPG, for the knee rehabilitation with a longitudinal feasibility study. In [27,28], a review of wearable sensors for rehabilitation was given. Each wearable sensor was used based on the request of rehabilitation. For example, an electromyography, EMG, was needed to be detected due to rehabilitation of triceps. In this condition, a rehabilitation system with EMG sensor was required. Hence, the rehabilitation system with wearable sensors could be applied for the residential rehabilitation. Based on different symptoms of rehabilitation, the rehabilitation system with wearable technology needed to be developed individually. Therefore, a wearable device for residential elbow joint rehabilitation detection with voice prompts and the tracking feedback APP, WDRTFAPP, was proposed in this paper, since no elbow joint rehabilitation detection system was designed.

In [29], the authors proposed the applications of wearable inertial sensors in estimation of upper limb movements. However, it only focused on using the kinematic modeling to estimate orientation and joint position in both the simulation and real condition among the shoulder, elbow, and wrist without addressing the angle of elbow rehabilitation. Although it proved that the motion rehabilitation by using kinematic modeling was almost the same as the orientation and joint position in real condition, no APPs, databases, user interfaces, nor systems were implemented in this paper. Hence, the patients could not query the status of rehabilitation. The medical staff also could not track the status of rehabilitation for each patient. Therefore, the applications of wearable inertial sensors in estimation of upper limb movements could not be used to overcome the issues in our paper.

In COPDTrainer [30], an APP was designed for Rehabilitation Training. However, it focused on motion rehabilitation, such as high performance classes, but addressed nothing for the angle of elbow joint rehabilitation. The patients could query the status of rehabilitation, but the medical staff could not track the status of each patient since no database was implemented. Although smartphones exist in life, they may not be used as the detector since the smartphones were often heavier than the general detector. Moreover, the smartphones were often larger than the general detector. Hence, the smartphones were not prone to be attached on the lower/upper limbs. Once the patients or medical staff wanted to feedback the message to each other, it could not be used by COPDTrainer. Due to the above reasons, the applications of wearable inertial sensors in estimation of upper limb movements and COPDTrainer could not solve the issues in our paper.

Therefore, a wearable device and APP were needed to be designed to address the above issue. A wearable device for residential elbow joint rehabilitation with voice prompts and tracking feedback APP was thus proposed in this paper. WDRTFAPP included a wearable device with voice prompts (WDVP) and a tracking feedback APP (TFAPP) for elbow joint rehabilitation detection. A WDVP aimed to detect the elbow joint rehabilitation and then transmit the rehabilitation data, such as the times, date, and correct rate of rehabilitation, to a cloud database via TFAPP, by Bluetooth, automatically in real time. The patients were simulated by our team not real patients. While the action of rehabilitation was incorrect, the voice prompts were sent to the simulated patients by TFAPP immediately. The simulated patients then could correct the failed action of rehabilitation immediately. The elbow joint rehabilitation thus could be rehabilitated at home for the simulated patients.

To track the status of elbow joint rehabilitation by the simulated patients or medical staff, TFAPP was designed in this paper. In TFAPP, the data of rehabilitation from WDVP was received by Bluetooth in real time automatically. In addition, the simulated patients and medical staff could query the data of elbow joint rehabilitation by mobile devices via the internet. For the simulated patients, they could query the data of elbow joint rehabilitation based on different dates. For the medical staff, they could track the elbow joint rehabilitation status of each simulated patient. They could set the times and date of rehabilitation for each simulated patient remotely via the internet by TFAPP. The medical staff could send a notification to the simulated patient based on his status of rehabilitation by TFAPP. The simulated patient could also give feedback responses to the medical staff by TFAPP.

In Taiwan, no wearable device for residential elbow joint rehabilitation combined with voice prompts and tracking feedback APP existed until now. Most elbow joint rehabilitation system focused on the exoskeleton systems with virtual reality and the motion-sensing. However, they may not be suitable for each simulated patient. Moreover, they may require more hardware cost and space. Hence, they could not be used in any place and time. To address these issues, the wearable device for residential elbow joint rehabilitation combined with voice prompts and tracking feedback APP was proposed in this paper.

The main contributions of this paper are as follows: (1) a voice prompt and tracking feedback app and interaction system for a wearable device for residential elbow joint rehabilitation, and, (2) a wearable device for residential elbow joint rehabilitation that uses lightweight wearable. The novel contribution of our work is our app and interaction system.

2. Methods

To detect the elbow joint rehabilitation at home in real time and to track the status of elbow joint rehabilitation by both of the simulated patients and medical staff with mobile devices remotely, a wearable device for elbow joint rehabilitation detection with voice prompts and tracking feedback APP (WDRTFAPP) was proposed in this paper. In our experiments, the patients were simulated by our team users and not real patients. WDRTFAPP could be divided into two main parts to be developed. One was the cloud wearable device with voice prompts, WDVP. Another was the tracking feedback APP, TFAPP. The architecture of WDRTFAPP was shown in Figure 1. While the rehabilitation data were captured by WDAP, the data were transmitted to the smart phone by Bluetooth immediately. At the same time, the data were transmitted to the cloud database by TFAPP via 4G/5G or Wi-Fi in real time.

2.1. Wearable Device Development

In WDVP, it aimed to calculate the angle of elbow joint rehabilitation and then transmitted the angle data to TFAPP via Bluetooth in real time. The hardware of WDVP was composed of embedded board (Arduino Uno), Bluetooth module, and tri-axial accelerometer module, respectively, as shown in Figure 2, where the name of the embedded board is Arduino UNO R3, the code of the tri-axial accelerometer module is GY-61 ADXL335, and the code of the Bluetooth module is HC-05. The connections between the various modules were shown in Figure 3. Finally, these modules were integrated into a box, as shown in Figure 3. The software of WDVP was developed by Arduino C++.

2.2. APP Design

The angle of elbow joint rehabilitation by WDVP was captured and transmitted to the cloud database by TFAPP in real time. Voice prompts occurred by TFAPP to help the simulated patients to correct their action of elbow joint rehabilitation, when the action of elbow joint rehabilitation is incorrect. TFAPP was designed by App Inventor 2. The cloud database was built by firebase. The user interface (UI) was designed based on the different roles of users, such as simulated patient and medical staff. In the simulated patient UI, the user only could query his/her data of elbow joint rehabilitation, but could not query other users’ data to ensure the privacy of simulated patients.

While the role of user is the medical staff, he/she could track the status of each simulated patient by simulated patient ID. Moreover, the medical staff could judge the setting of elbow joint rehabilitation, such as the required times and date of elbow joint rehabilitation, based on the status of simulated patient’s rehabilitation remotely. The simulated patient thus could increase or decrease strength of elbow joint rehabilitation based on the setting by the medical staff to improve the rehabilitation therapy next time.

Moreover, while the user has any question which was needed to be replied by medical staff, the user could transmit the message to the medical staff by TFAPP. After the medical staff receives the messages from the simulated patients, they could reply the messages to the simulated patients by TFAPP. Hence, the tracking feedback could be completed between the simulated patients and medical staff by TFAPP.

2.3. Experimental Design

This paper aimed to assess the feasibility of WDRTFAPP for judging the correct action of elbow joint rehabilitation. The assessment of medical effectiveness was not addressed in this paper. The experimental results were permitted by the orthopedist with the standard rehabilitation procedures. The rate of correct elbow joint rehabilitation detection (R) was calculated as (1). The main object of this paper was to ensure R up to 90%. The correct elbow joint rehabilitation was defined that the angle of habilitation between the start position and end position was more than 90 degree within 3 s, as shown in Figure 4. The reasons for the 3-s time limit are discussed in the discussion section. The angle was measured by the acceleration value of Y axis, as listed in

Table 1. Angle measurement.

Angle	Acceleration Value of Y Axis
0°	270–272
10°	273–276
20°	277–280
30°	281–290
40°	291–292
50°	293–309
60°	310–318
70°	319–323
80°	324–337
90°	338–350
100°	351–359
110°	360–375
120°	376–381
130°	382–389
140°	390–392
150°	393–407

.

Since WDRTFAPP not only judges the correct action of elbow joint rehabilitation, but it also provides the APP functions, such as data querying, remote setting, feedback messages, and voice prompts. Hence, it aimed to ensure the functions, such as data querying, remote setting, feedback messages, and voice prompts, were working in the experiment normally.

3. Results

The purpose was to implement a wearable device and design an APP for elbow joint rehabilitation detection at home. Hence, the experimental results not only included the performance evaluation but also the APP function verification. The rate of correct elbow joint rehabilitation detection was evaluated under the correct and incorrect actions of elbow joint rehabilitation in real situation. The experimental results were the pilot studies. Ten participants took part in this pilot study. Since the experimental results were the pilot studies not the clinical trials, the ethics committee could not be performed. This research is academic in nature, no personal information is saved, and the system is still in the development stage. After the system is perfected, it will apply to the Institutional Review Board (IRB) for experiments. After discussion with the Orthopedist, such as the third author, there is no ethical permission in this research, since our research was the pilot study and the patients were simulated by our team users not real patients. However, we will send the IRB our research in the future.

3.1. Correct Detection of Elbow Joint Rehabilitation

In the experimental results, the total times of elbow joint rehabilitation were 3000. Each round contained 1000 times. In the first round, the ratio of correct elbow joint rehabilitation to all elbow joint rehabilitation (r) was 100% in real situation. In the second round, the ratio of correct elbow joint rehabilitation to all elbow joint rehabilitation was 90%. In the final round, the ratio of correct elbow joint rehabilitation to all elbow joint rehabilitation was 70%. The rate of correct elbow joint rehabilitation detection, R, was evaluated under these three different conditions. Figure 5 showed that R was up to 90% in each round. It showed that CWDER we developed could be applied for the residential elbow joint rehabilitation.

The frequency of rehabilitation may be different from the age and symptom of simulated patients. Since CWDER examined the correct rehabilitation within 3 s, R was evaluated under different frequency of rehabilitation, such as 1 time/second, 2 times/second, and 5 times/second. The frequency of rehabilitation may be different from the age and symptom of simulated patients. Since CWDER examined the correct rehabilitation within 3 s, R was evaluated under different frequency of rehabilitation, such as 1 time/second, 2 times/second, and 5 times/second. Figure 6 showed that R was below 60% while the frequency of rehabilitation was high and low, such as 1 time/second and 5 times/second. Addressing this issue will be the future work in this paper.

3.2. APP Function Verfication

APP function verification was another important experimental result in this paper. Here, it showed the data querying, remote setting, feedback messages, and voice prompts based on the role of login user. Firstly, everyone needed to register an account to login, as shown in Figure 7. The role of the registered user will be determined based on the registered information. While the role of user is the patient, the login function was shown in Figure 8. While the simulated patient clicked the ‘‘Start rehabilitation” function, the action of rehabilitation began. After the number of rehabilitation (N_r) was larger than the number of recommended rehabilitation (N_rr), the rehabilitation was completed. The data of rehabilitation was then transmitted to the cloud database in real time, as shown in Figure 9. The rehabilitation record could be queried based on the setting date, as shown in Figure 10. Finally, the simulated patient could get the feedback the recommendation in “Doctor’s recommendation”, as shown in Figure 11. The designed wearable device actually worn by the user and the developed application on the smartphone during the real user rehabilitation activities was shown in Figure 12.

While the role of user is the medical staff, the login function was shown in Figure 13. In this paper, the setting of elbow joint rehabilitation was not judged by the simulated patients. The setting of elbow joint rehabilitation was judged by medical staff based on the status of the simulated patient’s rehabilitation remotely. Since the simulated patients had no medical professional ability, the setting of elbow joint rehabilitation judged by medical staff could have better efficiency of rehabilitation. Hence, the setting of elbow joint rehabilitation was judged by medical staff was shown in Figure 14. The medical staff could track the rehabilitation status of each simulated patient based on the name and date. Moreover, the medical staff could also send the recommendation to the simulated patient while tracking this simulated patient’s rehabilitation status, as shown in Figure 15. Finally, the simulated patient could feedback the reminder to the simulated patient in any time, as shown in Figure 16.

4. Discussion

The main object of this paper was to detect the elbow joint rehabilitation at home without the assistance by medical staff. How to examine the action of elbow joint rehabilitation which was correct or incorrect was important. Hence, the rate of correct elbow joint rehabilitation detection (R) was the main performance evaluation in our experimental results. In a general situation, R could be up to 90%.

However, the correct rehabilitation may be misjudged as the incorrect rehabilitation and the incorrect rehabilitation may be misjudged as the correct rehabilitation while the frequency of rehabilitation was too high or too low. To solve this problem, the medical staff could judge the capturing time of WDVP for each simulated patient based on his/her status of rehabilitation. This issue will also be a future work in this paper.

For the APP function verification, they were all well working. However, it may be difficult to be used by elderly people since the elderly people were usually not familiar with the operation of a smart phone. Hence, how to improve the UI of APP for elderly people was needed due to the aging society.

It was difficult to find the suitable exercise time in the experimental results initial since the experimental results were not only affected by the exercise time but also the different rehabilitation speed on different simulated patients, where the simulated patients were simulated by our team users not the real patients. After discussion with orthopedists, the symptom of the patients was assumed to be the elbow joint symptom. The age of the patients was assumed to be more than sixty years old, since the elder people may easily have the elbow joint symptom. However, young people may also have the elbow joint symptom. Since the experiments in this paper were the pilot studies not the clinical trials, the users were simulated as the patients with medical conditions to do rehabilitation for more than 3 s while the exercise time was set to 3 s in our experiment. In this situation, the rate of correct elbow joint rehabilitation detection was decreased. However, the rate of correct elbow joint rehabilitation detection could increase while the exercise time was longer. To address this issue, the exercise time was set to 3 s finally because this paper focused on the mild and moderately mild elbow joint symptoms after discussion with orthopedists. The number of simulated patients was ten.

The number of users trying the APP was ten. In this paper, the APP focused on data querying, remote setting, feedback messages, and voice prompts. Moreover, the experiments in this paper were the pilot studies not the clinical trials. Hence, to ensure the functions were working normally was the main objective of the APP. Since our researches were the pilot studies, the users needed to download the APP by our research team. The time to perform specific functions was more than three days for each simulated patients. Some mistakes occurred; however, these mistakes were mostly caused by the wireless connectivity not the APP. In the future, the APP will be uploaded to the APP store, such as Google Play, and the performance and feedback of the APP will also be evaluated after the IRB is passed.

5. Conclusions and Future Work

By the progress of 5G and wireless communication techniques, the interdisciplinary research integrated with wireless domain has become an important research issue recently. By the advancement of medical technology, the number of people with rehabilitation was increased. Hence, how to integrate the wireless communication with medical techniques was the main object of this paper. Among all kinds of rehabilitation, the elbow joint rehabilitation occurred commonly and most of elbow joint rehabilitation could by executed at home after discussion with an Orthopedist. To reduce the time and money for patients with elbow joint rehabilitation, and to reduce the resource and manpower for a hospital, a wearable device for residential elbow joint rehabilitation detection with voice prompts and a tracking feedback APP, WDREJRAPP, was proposed in this paper.

WDREJRAPP included the wearable device for residential elbow joint rehabilitation detection with voice prompts (WDREJHVP) and a tracking feedback APP (TFAPP). The WDREJHVP aimed to capture the action of rehabilitation and examine whether the action was correct or incorrect. Then, WDREJHVP automatically transmitted the detected data of rehabilitation to TFAPP via Bluetooth. While TFAPP received the data of rehabilitation from WDREJHVP, TFAPP automatically uploaded the data to the cloud database in real time.

Once WDREJHVP examined the incorrect action of rehabilitation, the simulated patients could receive the voice prompts from TFAPP to calibrate the action of rehabilitation to achieve the required number of rehabilitation. In the experimental results, the rate of correct elbow joint rehabilitation detection (R) could be up to 90% in a general situation. Another contribution of this paper is to provide the data querying, remote setting, feedback messages, and voice prompts in TFAPP that we designed for the simulated patients and medical staff. Different from other APPs, the main contribution of TFAPP was that the program of elbow joint rehabilitation was set and judged by medical staff, since the simulated patients had no medical professional ability. The simulated patients cannot set or judge the program of elbow joint rehabilitation. In this condition, it could have the better efficiency of rehabilitation than other APPs. In the experimental results, it showed that each function was working normally and could be used for the simulated patients and medical staff in any time and any place via the internet.

In conclusion, WDREJRAPP provided a wearable device and APP for the residential elbow joint rehabilitation detection in real time. In addition, WDREJRAPP reduced the time, money, resources, and manpower for the simulated patients and hospital. However, some issues were found in the experimental results. The first issue was that the rate of correct elbow joint rehabilitation detection may be decreased and below 90% while the frequency of elbow joint rehabilitation was too high or too slow. For this issue, WDVP will be improved to judge the capturing time of WDVP by the medical staff based on the simulated patient’s status of rehabilitation. Another issue was that TFAPP may not be suitable for iOS. Hence, we will design TFAPP through Node.js replaced by App Inventor 2 to be used in both of Android and iOS mobile operation systems in the future.

Since our research was the pilot study and the patients were simulated by our team users not real patients, we will send the IRB our research in the future.