A Comparative Study of Markerless Systems Based on Color-Depth Cameras, Polymer Optical Fiber Curvature Sensors, and Inertial Measurement Units: Towards Increasing the Accuracy in Joint Angle Estimation

Abstract

This paper presents a comparison between a multiple red green blue-depth (RGB-D) vision system, an intensity variation-based polymer optical fiber (POF) sensor, and inertial measurement units (IMUs) for human joint angle estimation and movement analysis. This systematic comparison aims to study the trade-off between the non-invasive feature of a vision system and its accuracy with wearable technologies for joint angle measurements. The multiple RGB-D vision system is composed of two camera-based sensors, in which a sensor fusion algorithm is employed to mitigate occlusion and out-range issues commonly reported in such systems. Two wearable sensors were employed for the comparison of angle estimation: (i) a POF curvature sensor to measure 1-DOF angle; and (ii) a commercially available IMUs MTw Awinda from Xsens. A protocol to evaluate elbow joints of 11 healthy volunteers was implemented and the comparison of the three systems was presented using the correlation coefficient and the root mean squared error (RMSE). Moreover, a novel approach for angle correction of markerless camera-based systems is proposed here to minimize the errors on the sagittal plane. Results show a correlation coefficient up to 0.99 between the sensors with a RMSE of 4.90${}^{\circ }$, which represents a two-fold reduction when compared with the uncompensated results (10.42${}^{\circ }$). Thus, the RGB-D system with the proposed technique is an attractive non-invasive and low-cost option for joint angle assessment. The authors envisage the proposed vision system as a valuable tool for the development of game-based interactive environments and for assistance of healthcare professionals on the generation of functional parameters during motion analysis in physical training and therapy.

1. Introduction

There is a clear and growing interest in developing technological-based tools that systematically analyze human movement. Notably, there are many advantages to implement automated systems to detect human motion for applications associated with children in a healthcare context or to assess mobility impairment of ill and elderly people [1]. Automated quantification of body motion to support specialists in the decision-making process such as stability, duration, coordination, and posture control is the desired result for those technological-based approaches [2,3]. Despite recent advances in this area, automated quantification of the human movement for children with sensory processing and cognitive impairments, and adults with mobility disabilities presents multiple challenges due to factors such as accessibility barriers, attached to the body requirements, and the high cost of the system.

Automated analysis of body movements typically involves obtaining 3D joint data such as position and orientation, which are estimated in two different ways using an intrusive or a non-intrusive approach, also known as wearable and non-wearable technologies [4]. Wearable systems are portable and can be used by people with movement impairments in unstructured scenarios [5]. However, advances in non-wearable sensing technologies and processing techniques have appeared to measure with high accuracy human biomechanics in highly structured environments [6]. For instance, camera-based markerless systems can be used in scenarios when the user does not admit a wearable device to capture data [4].

Commonly, analysis of joint angles is conducted through portable wearable sensors, which include electrogoniometers and potentiometers mounted in a single axis; however, they are bulky and limit natural patterns of movement [7]. Flexible goniometers adapt better to body parts and are not sensitive to misalignments due to movements of polycentric joints. In addition, they employ strain gauges, which must be carefully attached to the skin due to their high sensitivity [8]. Advancements in micro-electro-mechanical systems (MEMs) have enabled the growth of new wearable sensors.

Despite the wide range of applications, inertial measurement units (IMUs) present high sensitivity to the magnetic field and need frequent calibration [9]. New sensor technologies, such as optical sensors (with advantages such as compactness, lightweight, flexibility, and insensitivity to electromagnetic interference [10]), are employed for joint angle measurements using different approaches. Those approaches include intensity variation [11], fiber Bragg gratings [12] and interferometers [13]. Considering their advantages such as low cost, simplicity on the signal processing, and portability [14], intensity variation-based sensors emerge as a feasible alternative for joint angle assessment in movement analysis and robotic applications [15]. However, to date, POF-based curvature sensors share the same limitations of the aforementioned goniometers, since angle assessment is limited to a single plane.

Non-wearable human motion analysis systems use either multiple image sensors or multiple motion sensors [16]. The latter can be classified as either marker-based or markerless capture [17]. Although commercial marker-based systems can track human motion with high accuracy, markerless systems have many advantages [18]. Markerless systems can eliminate the difficulty of applying markers to users with physical or cognitive limitations [19].

However, most of the markerless systems require a surface model. Therefore, 3D surface reconstruction is a prerequisite to markerless capture [17,20]. The advent of off-the-shelf depth sensors, such as Microsoft Kinect [21], makes it easy to acquire depth data. Such depth data are beneficial for gesture recognition [22]. However, motion capture with depth sensors is a challenging issue due to the limitations of accuracy and range [23].

To build an efficient system, it is necessary to identify its functionalities and requirements to effectively contribute to the user’s necessities [24]. For example, people with disabilities that harm the use of wearable sensors impose difficulties on the acquisition of mobility parameters by clinical staff. Consequently, it is necessary to propose valid and straightforward methods for acquiring such parameters. The main assumption of the proposed technique for accuracy enhancement is the correlation between the measurement errors with the anthropometric parameters of the user. Thus, a compensation equation was obtained, which correlates the errors with the user’s anthropometric parameters. Although the proposed approach requires a longer calibration step prior to the sensor system application, it results in lower errors when compared to conventional approaches for angle analysis using markerless systems. The novel approach proposed here is capable of enhancing the accuracy of non-wearable systems. In addition, it is important to emphasize that such an approach can be applied in different systems (even the more accurate ones) for the enhancement of their accuracy. Furthermore, this technique can, with a few adjustments on the calibration step, be applied on the assessment of 3D dynamics.

This paper presents the analysis and comparison among a multi-red green blue-depth (RGB-D) vision system, an intensity variation-based POF sensor, and IMUs, which are intended for human joint elbow angles assessment. These wearable sensors were chosen due to their compactness, which would not cause occlusions for the markerless camera system. In addition, a marker-based camera system was not used in this comparison, since the markers can aid or harm the joint detection by the markerless system, which would not represent the condition for a practical application. This systematic comparison aims to study the trade-off between the markerless feature of the vision system and its accuracy by comparing it with wearable technologies for joint angle measurements. Thus, a compensation technique based on anthropomorphic measurements of the participants was proposed and validated using the POF curvature sensor measures.

2. Materials and Methods

2.1. RGB-D Fusion System

This system was designed as a distributed and modular architecture using the open source project Robot Operating System (ROS). The architecture developed here was built using a node graph approach. This system consists of several nodes to local video processing, distributed around several different hosts and connected at runtime in a peer-to-peer topology. The inter-node connection is implemented as a hand-shaking and occurs in XML-RPC protocol (Remote Procedure Call protocol which uses XML to encode its calls). The node structure is flexible, scalable, and can be dynamically modified, i.e., each node can be started and left running along an experimental session or resumed and connected to each other at runtime.

A computer vision framework composed of an unstructured and scalable network of RGB-D cameras is used to automatically estimate joint position, this visual sensor network counteracts typical problem as occlusion and narrow field of view. Consequently, the system uses a distributed architecture for processing the videos of each sensor independently. In this structure, a human body analysis algorithm is executed for each camera. Subsequently, the data from each sensor are transformed and represented respect to a shared reference to fusion them and generate the global joint position.

The vision system is composed of two RGB-D cameras, as shown in Figure 1, which each camera is connected to a workstation equipped with a processor Intel Core i5 and a GeForce GTX GPU board (GTX960 board, and GTX580 board) to execute local data processing. All workstations are connected through a local area network synchronized using Network Time Protocol (NTP) and managed through ROS. The data fusion estimation is sent for a third-party software interaction, which are executed on the highest processing capacity workstation.

Each workstation has two primary processes: user detection and position/orientation estimation of 15 joints. The detection process targets the candidate points to be people in the scenario, using the software NiTE, which does the client task, sending the movement estimation to the server over the network. The extrinsic (transformation from 3-D world coordinate system to the 3-D camera’s coordinate system) and intrinsic (transformation from the 3-D camera’s coordinates into the 2-D image coordinates) calibration of each RGB-D camera is performed using both the OpenCV package and the multi-camera network calibration tool provided by OpenPTrack (see Figure 2).

The workstation with the highest processing capacity is also used as the system server, which is responsible for the vision fusion process, using a Kalman filter. When the server receives a message with the position data of a client, it checks the time interval between the last received message and the current message. If this interval is greater than 33 ms, the system discards the received measurement and resumes counting the intervals from the next measure to receive. This is made to do not use the merging data with a very discrepant time. If the data is within the time interval, the system transforms the client coordinate system into the global coordinate system defined in the extrinsic calibration process. Then, the data is inserted into the Kalman filter. The saved data is processed through a low-pass Butterworth filter used to eliminate noise and to achieve a smoother estimate (see Figure 3).

The RGB-D fusion process is executed to obtain kinematic parameters and corporal patterns to be shown in an assessment interface that detects and quantifies the user’s movements. The system can produce parameters such as range of motion and positions of the tracked body articulations in three dimensions. In the same way, the system can be configured to show specific articulation angles. We use the law of cosine to calculate the elbow angle, as suggested by different authors [25,26,27,28]. Equation (1) shows the relation between the forearm $d_1$ and upper arm $d_2$ lengths as shown in Figure 4. The blue dots shown in Figure 4 are identified using the software NiTE, The blue dots shown in Figure 4 are identified using the software NiTE, such as aforementioned, where three points are identified (on the shoulder, elbow and hand), with (X, Y, Z) coordinates represented at each point.

$$\theta ={cos}^{-1}(\frac{-d_3+d_1+d_2}{2\ast d_1\ast d_2})$$

(1)

2.2. POF Sensor

The POF curvature sensor used in this work is based on the intensity variation principle in which, considering the fiber input connected to a light source and the output to a photodetector, there is a power variation on the POF output when it is subjected to curvature. Such power variation is proportional to the curvature angle and occurs due to some effects on the fiber such as radiation losses, light scattering, and stress-optic effects, as thoroughly discussed in Leal-Junior et al. [29]. To increase the sensor linearity and sensitivity as well as the hysteresis reduction, a lateral section is made on the fiber, removing the fiber cladding and part of its core (considering the core-cladding structure of conventional solid-core optical fibers [30]). This material removal on the fiber creates a so-called sensitive zone, where the fiber is more sensitive to curvature variations.

The POF employed in this work, multimode HFBR-EUS100Z POF (Broadcom Limited, Singapore), has its core made of Polymethyl Methacrylate (PMMA) with 980 $\mathsf{\mu }$m, whereas the cladding thickness is 10 $\mathsf{\mu }$m. In addition, the fiber also has a polyethylene coating for mechanical protection, which results in a total diameter of 2.2 mm for this fiber. The lateral section is made on the fiber through abrasive removal of material, where the depth and length of the sensitive zone are about 14 mm and 0.6 mm, respectively. The sensitive zone length and depth were chosen as the ones that result in high sensitivity, linearity with low hysteresis, as experimentally demonstrated in Leal-Junior et al. [31]. The POF with the lateral section has one end connected to a light emitting diode (LED) IF-E97 (Industrial Fiber Optics, Tempe, AZ, USA) with central wavelength at 660 nm, whereas the other end (output) is connected to the photodiode IF-D91 (Industrial Fiber Optics, Tempe, AZ, USA). The POF curvature sensor needs a characterization step prior to its application in movement analysis, where this characterization is performed by positioned the POF sensor on the experimental setup shown in Figure 5. In this setup, a DC motor with controlled position and angular velocity performs flexion and extension movements on the POF sensitive zone on an angular range of 0–90${}^{\circ }$. Then, the power attenuation is compared with the angle measured by the potentiometer (also shown in Figure 5) and a linear regression is made.

To get user movement parameters, the sensor is positioned on their elbow joints. Since the mounting conditions of the POF curvature sensor has direct influence on its response (as discussed in Leal-Junior et al. [31]), each user is asked to perform a 90${}^{\circ }$ angle in the sagittal plane (in both cases, i.e., sensor at elbow and knee joints). Thus, the 90${}^{\circ }$ movement on the sagittal plane is used to calibrate the sensor as a function of its positioning on each subject.

2.3. IMU

Two wireless MTw IMU sensors (Xsens, Enschede, The Netherlands) are used, which are placed on the users’ body segments, (weight: 10 g, size: 47 × 30 × 13 mm). Each MTw sensor comprises a 3D gyroscope, a 3D accelerometer and a 3D magnetometer. IMUs sends its data to a station (Awinda, Xsens) via a proprietary wireless communication protocol at 100 Hz of sampling frequency. The orientation data, in quaternion format, was estimated with a customized Kalman filter developed by the manufacturer (XKF3-hm) [32].

We implemented a method using two IMU units to estimate the elbow flexion-extension angle. The IMU on the proximal body limb (i.e., upper arm) was used as a reference, hereafter referred to as $S_1$. The second IMU, which was located on the distal body limb, will be referred as $S_2$. The algorithm uses five coordinate systems for its operation. $S_1$ and $S_2$ are the local coordinate system of the IMU sensors, these are embedded in each sensor. The technical-anatomical coordinate systems ($B_1$ and $B_2$) correspond to the body limbs’ orientation [33]. Finally, the global coordinate system, G, is a frame formed by gravity acceleration vector and Earth’s magnetic north. The reference IMU sensor $(S_1)$ was carefully aligned with the body limb, in such a way, during calibration moment, with a neutral standing posture and after applying a gravity alignment, $B_1$ ($B_2$) coordinate system is estimated from $S_1$ ($S_2$).

The algorithm aims the estimation of the angle between the orientation of the proximal and distal limb. Therefore, the limbs’ orientations should be projected to a common coordinate system (i.e., global coordinate system). This is done through the orientation quaternions, ${}^G{\mathbf{q}}_{B_1}(t)$ and ${}^G{\mathbf{q}}_{B_2}(t)$. The estimation process presupposes that x-axis of both technical-anatomical coordinate system are aligned with gravity vector at the calibration moment in a straight neutral posture, as proposed in previous works [33,34,35]. Besides the elbow joint is simplified as a 1-DOF hinge joint. The flexion-extension angle ($\alpha$) is calculated following the steps below:

• To calculate the rotation quaternion, ${\mathbf{q}}_c$, that aligns the x-axis of the $S_1$ coordinate system, ${\mathbf{x}}_{S_1}$, with the gravity vector is expressed in the global coordinate system (${\mathbf{z}}_G$) at the calibration moment, as proposed in [33].
• To define the technical-anatomical coordinate systems ${}^G{\mathbf{q}}_{B_1}(0)$ and ${}^G{\mathbf{q}}_{B_2}(0)$, at the calibration moment, applying Equations (2) and (3).

  $${}^G{\mathbf{q}}_{B_1}(0)={\mathbf{q}}_c{\otimes }^G{\overline{\mathbf{q}}}_{S_1}(0)$$

  (2)

  $${}^G{\mathbf{q}}_{B_1}(0){=}^G{\mathbf{q}}_{B_2}(0)$$

  (3)

  where ⊗ denotes the Hamilton product. Please note that due to the assumptions mentioned above, at the calibration moment, ${}^G{\mathbf{q}}_{B_1}(0)$ and ${}^G{\mathbf{q}}_{B_2}(0)$ are equals. Which means that the initial angle in a straight neutral posture is zero.
• To calculate sensor orientation $S_1$ ($S_2$) with respect to its associated body segment $B_1$ ($B_2$) using Equation (4). Please note that ${}^{B_1}{\mathbf{q}}_{S_1}$ and ${}^{B_2}{\mathbf{q}}_{S_2}$ are constant at all time instants.

  $${}^{B_1}{\mathbf{q}}_{S_1}={\left({}^G{\mathbf{q}}_{B_1}(0)\right)}^{-1}{\otimes }^G{\overline{\mathbf{q}}}_{S_1}(0)$$

  (4)
• To estimate the body segments’ orientation ${}^G{\mathbf{q}}_{B_1}(t)$ and ${}^G{\mathbf{q}}_{B_2}(t)$ using Equation (5). Please note that Equations (4) and (5) are expressed in terms of $B_1$ and $S_1$, but they are also applicable to $B_2$ and $S_2$.

  $${}^G{\mathbf{q}}_{B_1}(t){=}^G{\mathbf{q}}_{S_1}(t)\otimes {\left({}^{B_1}{\mathbf{q}}_{S_1}\right)}^{-1}$$

  (5)
• To calculate the relative orientation between ${}^G{\mathbf{q}}_{B_1}(t)$ and ${}^G{\mathbf{q}}_{B_2}(t)$. In that way, ${}^{B_1}{\mathbf{q}}_{B_2}(t)$ is then decomposed into ’ZXY’ Euler angles. The rotation about z-axis represents the elbow flexion-extension angle, $\alpha$, consistent with the ISB recommendations [16,36].

  $${}^{B_1}{\mathbf{q}}_{B_2}(t)={\left({}^G{\mathbf{q}}_{B_1}(t)\right)}^{-1}{\otimes }^G{\mathbf{q}}_{B_2}(t)$$

  (6)

2.4. Sensors Characterization

To validate the measurements of the camera-based and IMU-based systems a goniometer as a standard reference is used. This reference has two adjustable lever locks, which are positioned to limit the flexion-extension motion between 20${}^{\circ }$ (lower bound) and 90${}^{\circ }$ (upper bound). The goniometer was placed and aligned with the elbow joint of the subject M1, then, this was asked to perform flexion-extension movements on the sagittal plane in such a way to reach both locks. Lastly, the data was acquired by the camera-based system and the IMUs. The POF curvature sensor is characterized using the procedure mentioned in Section 2.2.

2.5. Experimental Protocol

Eleven participants without motor impairments were enrolled in this study. Six females, referred as F1, F2, F3, F4, F5, and F6, age: 27.3 ± 4.9 years old, corporal weight 56.8 ± 16.3 kg, and five males, referred as M1, M2, M3, M4, and M5, age: 27.4 ± 3.3 years old, corporal weight 70.2 ± 3.8 kg, as shown in

Table 1. Characteristics of the participants of this research.

Subject	Age (Years)	Height (cm)	Weight (Kg)
M1	30	163	67
M2	22	176	66
M3	30	173	73
M4	27	183	75
M5	28	170	70
F1	30	156	57
F2	32	158	46
F3	22	160	48
F4	23	163	53
F5	24	158	48
F6	33	176	89

. This research was approved by the Ethical Committee of UFES (Research Project CAAE: 64797816.7.0000.5542). As shown in Table 1, there is a variability not only in gender, but also on subjects’ weight and height, which provides evidence for a further generalization of the proposed study.

Two IMU sensors, one POF curvature sensor and two RGB-D cameras were used to estimate elbow joint angles. The IMU reference sensor was placed on the superior third of the right upper arm, and the second IMU was attached dorso-distally on the right forearm, as shown in Figure 6e. In a standing neutral posture, both sensors were positioned with x-axis pointing cranially, z-axis laterally and y-axis an orthogonal axis to x and z axes. These positions have been suggested by different authors [9,35,37]. Moreover, the POF curvature sensor was carefully aligned with the elbow joint in such a way that the sensitive zone of the optical fiber is located on the axis of rotation (flexion-extension axis).

In this test, the 11 participants (see Table 1) perform, a comfortable self-velocity, flexion-extension movements on three planes: sagittal, transverse, and frontal. Each participant was standing at the center of the room, observing the middle point between the two RGB-D cameras, see Figure 6d. All trials start with a synchronization movement, which consists of keeping the elbow in maximum extension on standing posture, then performing an elbow flexion of 90${}^{\circ }$ and returning to the extended elbow position, where each transition last 5 s. Then, the subject was asked to perform three repetitions of flexion-extension on a specific plane. In the sagittal plane, the shoulder is in a neutral position and the participant performs elbow flexion-extension to get the maximum angle as possible (see in Figure 6a). In the transverse plane, the shoulder is in abduction (at max 90${}^{\circ }$) and kept in that position for 5 s before the elbow flexion-extension movements (Figure 6b). In the frontal plane, the shoulder is in abduction (at max 90${}^{\circ }$) and external rotation, so the palm of the hand is facing forward as shown in Figure 6c. These described steps can be summarized in Figure 7.

3. Results and Discussion

The comparison variables of the three systems were: (i) the correlation coefficient and (ii) the root mean squared error (RMSE) between the RGB-D cameras, IMUs, and POF. After the comparison among the sensors, a novel approach for angle correction on the markerless camera-based system was proposed and validated for the correction of angular errors on the sagittal plane. The proposed technique is based on anthropometric measurements of each subject, where the errors of the camera-based system compared with the POF curvature sensor and IMU system are corrected through a correlation between the measured errors and the length of the arm of each subject ($d_1$, $d_2$ and $d_3$ of Equation (1)). In this way, an equation for error correction considering the length $d_3$ of each subject was obtained.

3.1. Sensors Characterization

3.1.1. IMUs and RGB-D Cameras

Figure 8 presents the results for RGB-D camera system and IMUs for the characterization phase using a goniometer as a reference for upper and lower bounds. The performed movement was divided into three cycles, and each cycle presents correlation between cameras and IMUs higher than 0.98.

Table 2. Maximum and minimum angles of camera-based system and IMU of each cycle for the first test.

	Cycle 1		Cycle 2		Cycle 3
	Max [°]	Min [°]	Max [°]	Min [°]	Max [°]	Min [°]
Camera	87.4	21.0	82.6	12.8	99.0	22.2
IMU	94.6	22.9	94.4	22.7	94.7	23.0
Goniometer	90.0	20.0	90.0	20.0	90.0	20.0

shows the maximum and minimum angles for cameras system and IMUs, compared to upper and lower bounds, respectively. The average error for the cameras system was ${4.9}^{\circ }$, with a maximum error of $9^{\circ }$, when compared with the goniometer for two values (${90}^{\circ }$ e ${20}^{\circ }$), which is lower than the mean error presented in Tannous et al. [38] ($14,6^{\circ }$). However, in Tannous et al. [38] only one camera was used, carrying a higher self-occlusion leading to errors on the angle assessment. Since our system consists of two cameras, the self-occlusion decreases, consequently, reducing the errors. Compared to the goniometer, the IMUs average error was ${3.7}^{\circ }$, which is lower than the camera-based system.

3.1.2. POF

Figure 9 presents the normalized POF power attenuation as a function of the angle on the experimental setup shown in Figure 6, where a high correlation with the measured angle was obtained, since the determination coefficient (R${}^2$) between the POF power variation and the angle was 0.996.

Thereafter, the POF curvature sensor is positioned on the subject’s elbow joint as previously described (see Section 2.5). The data of the elbow movement at each plane (sagittal, frontal and transverse) were recorded, which are presented in Figure 10 for subject M3. The initial movement of 90${}^{\circ }$ in the sagittal plane was used to adjust the sensor response with respect to the differences aroused from the positioning on the subject elbow.

The sensor response has the pattern of the flexion/extension movements performed by the subject. Furthermore, the angles estimated by the POF curvature sensor are within the range of movement described by the literature for elbow (0–145${}^{\circ }$) [39]. Thus, the results presented in Figure 10 in conjunction with the high correlation of 0.996 (obtained on the sensor characterization as a function of the angle measured by a potentiometer) shows the feasibility of the POF sensor on the angle assessment, which makes it a suitable option for the comparison with angles estimated by the camera-based system.

3.2. Comparison among the Sensors

After the first evaluation of the sensors and the POF curvature sensor characterization, the sensors used in this work were compared using the experimental protocol depicted in Section 2.5. In this case, the camera-based system was compared with both IMU and POF. The comparison was made with respect to the correlation coefficient and RMSE (as also performed in the previous sections). Figure 11 shows the results obtained for all sensors in different planes, i.e., sagittal, transverse, and frontal planes, for subject M1.

The results presented in Figure 11 show a good correlation between the errors of the POF curvature sensor and IMU, especially on sagittal and frontal planes. Although we used the same number of cycles to compare the sensors, the period of each movement is different, due to the fact that each subject was allowed to perform the movements at a comfortable self-velocity.

Furthermore, the range of movement at each plane is different, i.e., the movement at the sagittal plane occurs in a range of about 0–145${}^{\circ }$, whereas the one at the transverse plane reaches angles lower than 130${}^{\circ }$. Similarly, the angles at the frontal plane can be as high as 145${}^{\circ }$ (as in the sagittal plane). From the tests, the mean deviation between POF curvature sensor and IMUs was about 6.5% on the tests in the sagittal plane. However, such deviation increased to about 10% on the transverse and frontal planes. The reason for this increase can be related to the POF positioning on the tests, since it is a critical factor on the angle assessment using such technology. In addition, it can be also related to the increase of the errors of the IMUs when the test was performed in planes different from the sagittal one, as reported in Vargas-Valencia et al. [33]. Regarding the camera-based system, the results at the sagittal plane show an overestimation of the angle, when compared to IMU and POF curvature sensor. In this case, the angles estimated by the markerless camera system had a maximum value of about 160${}^{\circ }$, which is higher than the elbow range of motion [39]. In contrast, the camera-based system underestimates the angles at the frontal plane when compared with the other two systems for angle assessment.

Such as aforementioned, the errors on the markerless camera system for angle assessment are related to issues, such as frame errors, exploitation of multiple image streams and, especially, due to self-occlusions. To further evaluate the errors obtained by the camera-based system,

Table 3. RMSE and correlation coefficient for Cameras and IMU.

	Sagittal		Frontal		Transverse
	R2	RMSE	R2	RMSE	R2	RMSE
M1	0.994 ± 0.0041	13.53${}^{\circ }$ ± 2.87${}^{\circ }$	0.955 ± 0.0164	14.31${}^{\circ }$ ± 6.37${}^{\circ }$	0.988 ± 0.0043	18.62${}^{\circ }$ ± 1.32${}^{\circ }$
M2	0.994 ± 0.0023	10.21${}^{\circ }$ ± 1.86${}^{\circ }$	0.983 ± 0.0213	11.96${}^{\circ }$ ± 4.67${}^{\circ }$	0.991 ± 0.0052	14.48${}^{\circ }$ ± 4.53${}^{\circ }$
M3	0.986 ± 0.0052	7.42${}^{\circ }$ ± 2.51${}^{\circ }$	0.993 ± 0.0063	15.03${}^{\circ }$ ± 1.55${}^{\circ }$	0.984 ± 0.0060	18.84${}^{\circ }$ ± 7.23${}^{\circ }$
M4	0.984 ± 0.0058	13.02${}^{\circ }$ ± 3.12${}^{\circ }$	0.982 ± 0.0026	9.99${}^{\circ }$ ± 3.94${}^{\circ }$	0.992 ± 0.0031	15.89${}^{\circ }$ ± 6.59${}^{\circ }$
M5	0.991 ± 0.0025	9.60${}^{\circ }$ ± 3.08${}^{\circ }$	0.989 ± 0.0095	14.34${}^{\circ }$ ± 3.85${}^{\circ }$	0.978 ± 0.0230	11.21${}^{\circ }$ ± 2.90${}^{\circ }$
F1	0.993 ± 0.0011	11.83${}^{\circ }$ ± 2.93${}^{\circ }$	0.976 ± 0.0098	16.42${}^{\circ }$ ± 2.41${}^{\circ }$	0.991 ± 0.0077	17.61${}^{\circ }$ ± 3.67${}^{\circ }$
F2	0.990 ± 0.0043	11.26${}^{\circ }$ ± 2.89${}^{\circ }$	0.986 ± 0.0045	8.49${}^{\circ }$ ± 5.61${}^{\circ }$	0.921 ± 0.0833	15.28${}^{\circ }$ ± 4.31${}^{\circ }$
F3	0.974 ± 0.0136	12.04${}^{\circ }$ ± 5.39${}^{\circ }$	0.990 ± 0.0046	14.89${}^{\circ }$ ± 1.90${}^{\circ }$	0.956 ± 0.0543	16.73${}^{\circ }$ ± 2.89${}^{\circ }$
F4	0.996 ± 0.0019	11.90${}^{\circ }$ ± 3.53${}^{\circ }$	0.989 ± 0.0057	15.98${}^{\circ }$ ± 6.78${}^{\circ }$	0.990 ± 0.0052	16.29${}^{\circ }$ ± 5.98${}^{\circ }$
F5	0.993 ± 0.0079	9.89${}^{\circ }$ ± 0.77${}^{\circ }$	0.994 ± 0.0020	12.56${}^{\circ }$ ± 3.74${}^{\circ }$	0.987 ± 0.0037	19.42${}^{\circ }$ ± 5.03${}^{\circ }$
F6	0.995 ± 0.0042	11.81${}^{\circ }$ ± 0.52${}^{\circ }$	0.989 ± 0.0034	17.31${}^{\circ }$ ± 4.04${}^{\circ }$	0.992 ± 0.0006	19.98${}^{\circ }$ ± 6.52${}^{\circ }$

presents the correlation coefficient and RMSE between the markerless system and the IMUs for each of the 11 participants in all three planes tested, whereas

Table 4. RMSE and correlation coefficient for Cameras and POF.

	Sagittal		Frontal		Transverse
	R2	RMSE	R2	RMSE	R2	RMSE
M1	0.988 ± 0.0066	10.01${}^{\circ }$ ± 2.12${}^{\circ }$	0.972 ± 0.0175	15.32${}^{\circ }$ ± 6.92${}^{\circ }$	0.994 ± 0.0033	19.32${}^{\circ }$ ± 3.30${}^{\circ }$
M2	0.977 ± 0.0055	10.90${}^{\circ }$ ± 1.67${}^{\circ }$	0.985 ± 0.0065	11.28${}^{\circ }$ ± 0.97${}^{\circ }$	0.985 ± 0.0080	14.98${}^{\circ }$ ± 4.83${}^{\circ }$
M3	0.978 ± 0.0110	6.90${}^{\circ }$ ± 2.25${}^{\circ }$	0.967 ± 0.0025	16.14${}^{\circ }$ ± 2.74${}^{\circ }$	0.981 ± 0.0123	19.83${}^{\circ }$ ± 3.80${}^{\circ }$
M4	0.955 ± 0.0208	12.00${}^{\circ }$ ± 2.83${}^{\circ }$	0.916 ± 0.0106	13.35${}^{\circ }$ ± 4.23${}^{\circ }$	0.965 ± 0.0165	13.53${}^{\circ }$ ± 5.28${}^{\circ }$
M5	0.994 ± 0.0017	10.87${}^{\circ }$ ± 0.15${}^{\circ }$	0.973 ± 0.0069	13.26${}^{\circ }$ ± 3.39${}^{\circ }$	0.954 ± 0.0289	11.60${}^{\circ }$ ± 1.17${}^{\circ }$
F1	0.975 ± 0.0178	11.52${}^{\circ }$ ± 3.24${}^{\circ }$	0.978 ± 0.0196	14.84${}^{\circ }$ ± 1.09${}^{\circ }$	0.970 ± 0.0205	17.84${}^{\circ }$ ± 3.44${}^{\circ }$
F2	0.983 ± 0.0120	10.35${}^{\circ }$ ± 0.60${}^{\circ }$	0.987 ± 0.0051	9.42${}^{\circ }$ ± 4.27${}^{\circ }$	0.945 ± 0.0375	17.44${}^{\circ }$ ± 4.77${}^{\circ }$
F3	0.982 ± 0.0085	7.72${}^{\circ }$ ± 1.59${}^{\circ }$	0.978 ± 0.0070	17.17${}^{\circ }$ ± 1.80${}^{\circ }$	0.982 ± 0.0083	17.86${}^{\circ }$ ± 5.20${}^{\circ }$
F4	0.978 ± 0.0098	12.76${}^{\circ }$ ± 1.63${}^{\circ }$	0.975 ± 0.0076	16.95${}^{\circ }$ ± 4.46${}^{\circ }$	0.987 ± 0.0001	17.71${}^{\circ }$ ± 6.52${}^{\circ }$
F5	0.989 ± 0.0097	8.11${}^{\circ }$ ± 0.81${}^{\circ }$	0.977 ± 0.0070	13.56${}^{\circ }$ ± 4.30${}^{\circ }$	0.979 ± 0.0131	19.28${}^{\circ }$ ± 5.49${}^{\circ }$
F6	0.964 ± 0.0076	13.52${}^{\circ }$ ± 0.95${}^{\circ }$	0.892 ± 0.0165	19.13${}^{\circ }$ ± 4.53${}^{\circ }$	0.979 ± 0.0047	18.92${}^{\circ }$ ± 5.36${}^{\circ }$

presents the correlation and RMSE between the markerless system and the POF curvature sensor.

Table 3 and Table 4 show a correlation coefficient higher than 0.9 in all analyzed cases, which indicates a high correlation between the responses of the sensors. In addition, the standard deviation of the correlation coefficient was below 0.01 in all the analyzed cases. Thus, it is possible to verify not only a high correlation between the data of the camera-based system compared to the wearable ones, but also that the results present a promising evidence of repeatability of such systems. The mean of correlation coefficients between the camera-based system and the IMUs were 0.990, 0.984 and 0.979 on the sagittal, transverse, and frontal planes, respectively. It is noteworthy that higher correlations were obtained between the camera-based system and the IMUs than the ones comparing the markerless system with the POF curvature sensor. The mean of the correlation coefficients considering the later comparison was 0.978, 0.964 and 0.975 for sagittal, transverse, and frontal planes, respectively.

Even though the proposed camera-based system presented high correlation with the wearable sensors in all scenarios, the errors of such system are generally high. Such as can be observed in Figure 11, there are deviations on the angle estimation of the camera-based system when compared with the wearable sensors, where, considering all the performed tests, these errors can be as high as 15${}^{\circ }$ on the worst case. In addition, the mean errors are about 10${}^{\circ }$ when compared with the wearable sensors. It is noteworthy that these errors are lower than the ones reported on the literature [40], which is mainly due to the use of two cameras to reduce the errors related to occlusions. However, errors of about 10${}^{\circ }$ are still not sufficient when a reliable system for movement analysis is concerned. Nevertheless, the high correlations obtained in all tests for the comparison with the wearable sensors (see Table 3 and Table 4) indicate that the proposed markerless camera-based system can be a feasible solution for angle estimation if a post-processing technique for the correction of the angular errors is applied, as also discussed in Schmitz et al. [40].

3.3. Technique for Angle Correction in Markerless Camera-Based Systems

The primary assumption for the proposed compensation technique for angle errors in markerless camera-based systems is that the errors mainly occur due to occlusions, or errors on computer vision algorithm for the tracking of the anatomical points used to calculate the parameters $d_1$ and $d_2$ in Figure 4. If these parameters are incorrectly estimated, errors on the angle assessment will occur. Thus, it is possible to assume that these angular errors have a correlation with the anthropometric measurements of each participant. To verify this assumption and develop the compensation technique for the markerless system, each participant performed 3 flexion/extension cycles only on the sagittal plane (see Section 2.5), and the angles estimated with the markerless camera-based system were compared with the ones measured by the POF curvature sensor. We used the POF curvature sensor for the development of the compensation technique, since it was already evaluated with respect to the potentiometer, presenting low errors in this characterization. However, we must emphasize that other sensor systems can be used as reference for the proposed compensation technique, including IMUs, marker-based camera systems and goniometers. The technique proposed here is based on the basic premise that the errors are mainly related to the detection of the parameters $d_1$, $d_2$ and $d_3$ (due to self-occlusions, numerical errors on the computer vision algorithm, among other reasons). Therefore, the errors can be correlated (and then compensated) by considering the actual value of the anthropometric measurements ($d_1$ and $d_2$) used on the angle estimation, which can be measured on each subject or estimated using the subject’s height [41].

For the first characterization of the technique, the flexion/extension cycles of five subjects (M1, M3, M5, F2 and F5) are analyzed, and a polynomial regression between the angles estimated by the camera-based system and the POF curvature sensor is performed for each of the five subjects, where each equation has the type shown in Equation (7):

$$an{g}_{ref}=a·{an{g}_{cam}}^3+b·{an{g}_{cam}}^2+c·an{g}_{cam}+d,$$

(7)

where $an{g}_{cam}$ is the angle estimated by the camera, $an{g}_{ref}$ is the angle measured by the POF curvature sensor. In addition, a, b, c and d are polynomial regression coefficients experimentally obtained through the regression between the angular responses of both sensor systems, i.e., markerless camera-based and POF sensor, using the least squares method. The coefficient d is the offset on the sensor response (in ${}^{\circ }$). Therefore, if the sensors responses are normalized in the beginning of the test, the offset will be null. For this reason, the coefficient d is not employed on the analysis of correlation between the coefficients of the angular error correction and the anthropometric measurements of the subjects.

Figure 12 shows the regression between the angle measured by the camera-based system and the POF curvature sensor for the third flexion cycle (as an example) of subject F5. The results show a high correlation (0.998) between the responses using a third-order polynomial regression. Actually, such high correlation occurs for all the cycles of the five subjects analyzed, where the correlation coefficient was higher than 0.9 in all cases. Hence, the assumption of correlation between the errors of both sensor systems holds true (based on the analyses performed). Then, the next step is to correlate the polynomial coefficients (a, b and c) with the anthropometric measurements of each participant.

As discussed in Section 2.1, the parameters used on the angle estimation by the camera-based system are the anthropometric distances ($d_1$ and $d_2$), which are detected through computer vision algorithms. Thus, errors on the detection of such points will lead to errors on angle estimation, where such errors can be related to those anthropometric distances. However, these parameters ($d_1$ and $d_2$) are intrinsic of each subject and can be easily measured. In addition, it is possible to use the height of each subject (as showed in Table 1) in conjunction with anthropometric data for males and females to correlate the arm length with the subject’s height. Figure 13 shows the correlation of the polynomial regression coefficients (a, b and c) with the subjects’ arm lengths (D).

The results as well as the equations presented in Figure 13 indicate the feasibility of using the anthropometric measurements of each subject on equations for angular errors corrections in camera-based systems. The correlation coefficient is higher than 0.9 for all analyzed coefficients, indicating the possibility of using the proposed compensation technique for angle correction. Then, by substituting the equations shown in Figure 13 in Equation (7), it is possible to obtain a corrected angle as depicted in Figure 14 for three flexion/extension cycles for subject F1. In addition, the uncompensated response, i.e., the response of the camera-based system without applying the equations for angle correction, is also presented for comparison purposes. The RMSE for the compensated response is also presented in order to verify the accuracy enhancement provided by the proposed technique. Compared to the uncompensated responses, where the RMSE was 15.04${}^{\circ }$, 9.25${}^{\circ }$ and 10.23${}^{\circ }$ for cycles 1, 2, and 3, respectively, the proposed angular error compensation was able to reduce the errors substantially in all three cycles. To further verify the performance of the proposed technique, the aforementioned compensation equations were applied for the responses in the sagittal plane for all subjects. The comparison between the RMSEs for the cases with and without the compensation technique is presented in

Table 5. Comparison between RMSEs of the compensated and uncompensated responses for each subject.

Subject	RMSE-Compensated	RMSE-Uncompensated
M1	3.52${}^{\circ }$ ± 0.84${}^{\circ }$	10.01${}^{\circ }$ ± 2.12${}^{\circ }$
M2	3.99${}^{\circ }$ ± 1.15${}^{\circ }$	10.90${}^{\circ }$ ± 1.67${}^{\circ }$
M3	4.79${}^{\circ }$ ± 0.74${}^{\circ }$	6.90${}^{\circ }$ ± 2.25${}^{\circ }$
M4	6.24${}^{\circ }$ ± 1.66${}^{\circ }$	12.00${}^{\circ }$ ± 2.83${}^{\circ }$
M5	4.64${}^{\circ }$ ± 1.23${}^{\circ }$	10.87${}^{\circ }$ ± 0.15${}^{\circ }$
F1	3.52${}^{\circ }$ ± 1.25${}^{\circ }$	11.52${}^{\circ }$ ± 3.24${}^{\circ }$
F2	4.22${}^{\circ }$ ± 2.25${}^{\circ }$	10.35${}^{\circ }$ ± 0.60${}^{\circ }$
F3	4.11${}^{\circ }$ ± 1.31${}^{\circ }$	7.72${}^{\circ }$ ± 1.59${}^{\circ }$
F4	5.36${}^{\circ }$ ± 2.05${}^{\circ }$	12.76${}^{\circ }$ ± 1.63${}^{\circ }$
F5	5.74${}^{\circ }$ ± 0.77${}^{\circ }$	8.11${}^{\circ }$ ± 0.81${}^{\circ }$
F6	4.79${}^{\circ }$ ± 1.06${}^{\circ }$	13.52${}^{\circ }$ ± 0.95${}^{\circ }$

for each subject in all three flexion/extension cycles analyzed, where the mean and standard deviation of the three cycles are presented for each participant.

The results presented in Table 5 show the feasibility of the proposed technique, where the RMSE was reduced for all 11 subjects analyzed. The highest reduction occurred in Subject F1, in which the RMSE reduced from 11.52${}^{\circ }$ to 3.52${}^{\circ }$ after applying the correction equations. The mean of the RMSEs for the compensated responses is about 4.90${}^{\circ }$, whereas the uncompensated one is 10.42${}^{\circ }$, which means a two-fold reduction of the RMSE when the proposed compensation is applied. It is also worth to mention that the lowest RMSE reduction for the compensated case occurred in subject M3, where the RMSE reduced 2.11${}^{\circ }$. However, one should note that the RMSE of the uncompensated response of this subject was already low (6.90${}^{\circ }$) when compared to the ones of the other subjects and even when compared to the errors presented in the literature for similar systems [40].

The proposed technique for angular errors correction in markerless camera-based system is a feasible and straightforward option to enhance the angular accuracy in such systems. There is a calibration step in which the response of the camera-based system has to be compared with the one of a reference sensor system, e.g., wearable or marker-based camera systems. Then, the errors obtained on the markerless camera-based system are compared with the subject’s anthropomorphic parameters (arm length in this case) in order to obtain an equation that relates the angle correction with the parameters of each subject. Therefore, an important caveat should be mentioned: the calibration routine must be performed with respect to a reliable reference, and the movements should be performed at one plane, i.e., sagittal, frontal, or transverse planes movements. In addition, the calibration has to be performed on the same range at which the angle analysis will be performed, i.e., if an angular interval of 0 to 160${}^{\circ }$ will be analyzed, the calibration has to be made at this same angular range (0 to 160${}^{\circ }$). By following these steps, it is possible to obtain accurate single plane angle measurements with a markerless camera-based system. Therefore, the main limitation of this approach is the necessity of a calibration stage prior to the application of the proposed sensor system in the same range and planes of movement envisaged on the proposed application. However, it is worth noting that the proposed approach can be extended for movement analysis of different degrees of freedom by adjusting the calibration stage accordingly and correlating the errors with the anthropometric parameters of each subject.

4. Conclusions

This paper presented the analysis and comparison of a markerless camera-based system for elbow angles assessment. The proposed markerless system uses two RGB-D cameras to reduce errors and inaccuracies related to self-occlusion issues. The non-wearable system performance was compared with two wearable solutions, namely POF curvature sensor and IMUs, in flexion/extension movements performed in different planes (sagittal, transverse, and frontal planes). Even though the proposed markerless camera-based system showed lower errors than some similar systems proposed in the literature, the errors are still high for movement analysis applications. To tackle this limitation, a comprehensive analysis of the system showed that despite the high errors, the markerless camera-based system response has a high correlation with the response of the wearable sensors. This also indicated the possibility of applying post-processing techniques aimed at error reduction in those systems. Thus, a compensation technique based on anthropometric measurements of the subjects was proposed and validated using the POF curvature sensor measures, resulting in a significant decrease of the RMSE.

The proposed RGB-D vision system and the novel compensation technique proposed here indicate the suitability of the system on the movement analysis, since the mean error obtained is about 4${}^{\circ }$ for the angles tested, i.e., up to 120${}^{\circ }$ in the sagittal plane, which is in agreement with the errors obtained in some sensing approaches for movement analysis [42]. Thus, this work can pave the way for movement analysis applications with markerless camera-based system. Future works include the further investigation of this technique in the other motion planes (frontal and transverse plane) and also in 3D movement scenarios. In addition, further evaluation and new sensor fusion techniques of markerless camera-based systems and optical fiber sensors will also be developed. Currently, to decrease the error, this work can be understood as the foundation of such developments.