Detection of Movement Intention for Operating Methods of Serious Games

Abstract

In many post-stroke cases, patients show dysfunctions in movement, cognition, sense, and language, depending on the damaged area of the brain. Active and repetitive physical rehabilitation centered on the stroke-affected side is essential for effective and rapid neurological recovery of upper extremity dysfunction due to hemiplegia. A symmetric upper extremity trainer is utilized to assist the patient body, depending upon the degree of hemiplegia. In this study, we developed a novel balance handle as a symmetric upper extremity trainer capable of extension, flexion, pronation, and supination of the upper extremity. We collected the surface electromyogram (sEMG) signal data while the subjects were playing a serious game and recorded the electroencephalogram (EEG) signal data while the subjects were performing basic movements with the balance handle, to analyze the effectiveness of the device as an assistive tool for rehabilitation. The triceps brachii were activated during the extension movements, whereas the biceps brachii and deltoid muscles were activated during the flexion movements. With the balance handle, the peak event-related desynchronization (ERD) values were relatively lower while showing higher peak event-related synchronization (ERS) values compared to other types of operating methods, such as hand gripping and gamepad operation. Movement intention of tilting the balance handle for the α and β waves was clearly distinguished from the other tasks. These data demonstrated the potential of various applications using the developed proof-of-concept upper extremity trainer to bring out an excellent rehabilitative effect not only through muscle growth but also via identification of large movement intentions inducing brain activation exercise.

1. Introduction

1.1. Background

A commendable outcome, in recent years, of the enormous advancement in medical science is the continuous decrease in mortality rate of stroke [1]. However, in many post-stroke cases, patients show dysfunctions in movement, cognition, sense, and language, depending on the damaged area of the brain [2]. In particular, hemiplegia that occurs on the opposite side of the damaged brain reduces locomotor function of the affected extremity and causes an unstable balance sense, which in turn reduces the activities of daily living (ADL), such as walking, eating, and dressing [3,4]. Symptoms of movement disorders due to hemiplegia are normally more notable in the upper extremity than the lower one [5]. Upper extremity functions are closely related to the activities of exploring and manipulating the surrounding environment; therefore, the upper extremity is particularly important for the ADL and workability. Thus, active and repetitive physical rehabilitation centered on the stroke-affected side is essential for effective and rapid neurological recovery of upper extremity dysfunction due to hemiplegia [6].

The methods to improve the stroke-affected movement functions of the upper extremity include constraint-induced movement therapy, rehabilitation robotics, mirror therapy, and bilateral movement training [7,8,9]. Specifically, the bilateral movement training utilizes substitution, in which the neural network of the healthy side compensates for the functions of the neural network of the stroke-affected side by simultaneously activating the neural networks of both the healthy and pathologic sides [10]. In general, a symmetric upper extremity trainer is utilized to assist the patient body, depending upon the degree of hemiplegia. Recent studies have attempted to associate the rehabilitation equipment used to improve or treat physical abilities with serious games, applying game elements such as fun and challenge. Such programs are considered to increase the patients’ intention to rehabilitate and effectively recovers the physical abilities by reducing the rejection or boredom of treatment that patients may have [11]. Moreover, it can help users actively perform steady repetitive exercises, which are the most important in rehabilitation, by offering them a sense of reality beyond the limited user interface of the serious games, consisting of a keyboard, mouse, and specific buttons and sticks [12,13].

In this study, a novel balance handle was manufactured as a symmetric upper extremity trainer to enable bilateral rehabilitation movements, such as extension, flexion, pronation, and supination of the upper extremity. The balance handle was connected to a serious game, and a surface electromyogram (sEMG) was measured for the triceps brachii, biceps brachii, and deltoid to assess the muscular activation while the users played the game. The sEMG signal has the advantage of quantitatively evaluating physical function and tracking the results of the rehabilitation treatment by measuring and analyzing the motor signals expressed in the central nervous system (CNS) in muscle nerves. Additionally, an electroencephalogram (EEG) was used to evaluate brain activation while the subjects performed the rehabilitation movement process with the symmetric upper extremity trainer. Reorganization of the damaged CNS was confirmed by evaluating the activation of the motor cortex from the EEG signals [14]. In order to analyze the activation of the motor cortex quantitatively, the signals measured by the symmetrical upper extremity trainer were compared with other types of operating methods for serious games, such as hand gripping and gamepad.

The EEG signals associated with the movement function helps analyze all the time points of body movement pertaining to an idle rhythm, a state in which a set of neurons in the motor cortex simultaneously displays a periodic signal. Movement intention can then be predicted by analyzing the signal that appears before actual movement occurs [15]. When movement intention of the body is found in the idle rhythm state, an event-related desynchronization (ERD) phenomenon, in which the power decreases at a specific frequency band by the excitatory postsynaptic potential (EPSP), occurs. Following the motion, an event-related synchronization (ERS) phenomenon, in which the power increases again at a specific frequency band by the inhibitory postsynaptic potential (IPSP), occurs. The movement intention can be analyzed using ERD and ERS and employed as a good indicator to quantitatively measure the willingness and movement of patients participating in training, as it demonstrates a statistical difference between rest and movement as well as between different types of movement [16]. The movement intention was calculated through the ERD/ERS analysis that can show the CNS activities based on the spontaneous shifting of the measured EEG signals. The difference between the groups according to the operating methods was compared by verifying the normality and performing the two-way analysis of variance (ANOVA).

1.2. Related Work

Recent studies compared the effects of upper extremity trainers as well as attempted to quantitatively measure movements or increase the effectiveness of rehabilitation. A variety of customized trainers were developed to rehabilitate specific areas of the upper extremities. Movements of an end-effector robot capable of horizontal plane movement of the unilateral forearm were quantitatively evaluated by moving a hand on a desk [17]. Assist-as-needed training was conducted using an end-effector upper limb rehabilitation robot that helps patients keep their arm close to a specific trajectory [18]. Information technology (IT), including virtual reality (VR) and mobile applications, was integrated for accessible and structured rehabilitation. VR has great strength in interaction and was employed as a therapeutic treatment tool for rehabilitation of the upper extremities of stroke patients [19]. A mobile application was also developed as a useful tool for subject-specific rehabilitation of the upper extremity following stroke [20].

The balance handle developed in this study can assist movement with respect to the various axes, including extension, flexion, pronation, and supination of the upper extremity, by allowing movements on the coronal and sagittal planes unlike traditional trainers applying along a single axis. Improved effectiveness of rehabilitation is expected by symmetrically performing bilateral movements of both upper extremities at the same time compared to performing unilateral movements. Furthermore, this balance handle can be utilized to quantitively measure movement data with respect to various tilting angles and collect real time feedback data via Bluetooth connection.

2. Materials and Methods

2.1. Manufacture of the Balance Handle

We developed a novel balance handle as a symmetrical upper extremity trainer for extension, flexion, pronation, and supination of the upper extremity. The balance handle comprised a balance ball, arm holders, and handles (Figure 1).

The spherical balance ball positioned at the center between the arm holders was designed to calculate its slope through an inertial measurement unit (IMU) sensor and send the data through a Bluetooth module that did not interfere with the user movement. A user with weak upper extremity muscle can comfortably place both upper extremity forearms on the holders, hold the handle, and move it freely towards the longitudinal and lateral axes. Furthermore, the user can push the buttons on the edges of both handles to add more input data if needed. We used an AM530 sensor (Laxtha, Daejeon, Korea) to measure the sEMG signals and a cap-type EMOTIV EPOC Flex (Emotiv, San Francisco, CA, USA) to measure the EEG signals through up to 32 channels to examine the muscular (sEMG data) and brain (EEG data) activation of the subjects while using the balance handle.

2.2. Measurement of the Muscular Activation Using the Balance Handle

Users activated the major upper extremity muscles by moving the balance handle forward, backward, leftward, or rightward. Figure 2 displays the serious game developed to induce these movements. Figure 2a shows the jet ski moving up and down as the user tilted the balance handle forward and backward through extension and flexion of the upper extremity. Figure 2b demonstrates the jet ski moving leftward and rightward as the user tilts the balance handle to the left and to the right through pronation and supination of both upper extremity forearms. The user gained score points, the competitive element of the serious game, by acquiring the displayed fuel and increase the content execution time. The score was deducted if the user failed to avoid an obstacle. The highest score among the participants was displayed on the screen to encourage competition.

2.3. Movement Intention Test

Ten adults (eight males and two females) in their twenties (average age 25.0 ± 1.7 years old) with no experience of a damaged upper extremity participated in the experimental tests to perform three operations: hand gripping, gamepad operation, and operation of the symmetric upper extremity trainer similar to the Oddball Paradigm, to compare the movement intention according to the operation methods for the serious game content (Figure 3).

All the tasks were divided into preparation, stimulation, and rest stages, to reduce their mutual influences before and after each task. A performance of 5 s was classified as a single trial, and each operation was repeated 20 times to minimize any error during the test process. Figure 3a demonstrates the rest stage after the stimulation stage between the repeated tasks as a blank page in the preparation stage. The rest stage gives a measure of the reference in the idle rhythm state before presenting a visual stimulus. The purpose of the data measured at the preparation stage for 5 s was to measure the changes in the data related to movement, and the relative amplitude was then calculated from these data. The EEG signals of the reaction to the visual stimulus on the screen were recorded during the stimulation stage. The rest stage was the idle moment before performing the next task; therefore, no data during the rest stage were used in the analysis.

The grip task was to perform extension and flexion of the finger used in an intuitive operation, such as using a keyboard or touching the screen, while the upper extremity remained fixed. The contents were composed to indicate one of the hands to grip; this for the user to perform the same task shown on the screen without bending the wrist (Figure 3b). A product from Joytron, consisting of nine general buttons, two analog sticks, one cross button, and two trigger buttons, was utilized to conduct the gamepad task. The screen displayed red arrows to instruct the user to move to the right or to the left with the analog stick or to press the specified general button (Figure 3c). Lastly, for the task involving the trainer interfaced with the serious game, the screen showed red arrows to instruct the users to tilt the symmetrical upper extremity trainer (i.e., the balance handle) forward, backward, leftward, or rightward, or press the button on the handle (Figure 3d). The system was configured so that the task instruction was displayed in a random order to prevent the test subjects from predicting and acting on the stimulus in each task, and the number of repetitions was set to be the same in all the directions for straightforward comparison.

2.4. Measurement of the EMG and EEG Signals

With the balance handle, the subjects performed extension and flexion of their upper extremity and pronation and supination movements of their joints, including shoulder, elbows, and forearms, according to the device tilting movement. In order to collect and analyze the sEMG signal data, an AM530 active surface EMG sensor (Laxtha, Daejeon, Korea) was attached to the deltoid, biceps, and triceps while the reference electrodes were attached to the elbow joint where there was no change in muscle activity (Figure 4). The sEMG signal data were collected at a 1-kHz sampling frequency via serial communication.

The EEG signal data were collected for analysis of brain activation at a sampling frequency of 1024 Hz and filtered through a band pass filter (BPF) ranging from 0.16 to 43 Hz and a notch filter at 50 and 60 Hz, such that the EEG signal data in the frequency band were selectively collected and the noises induced by the power were removed. Following the 10–20 international system of EEG electrode placement, 32 signal electrodes (hollow white) and 2 ground electrodes (solid black) were configured with two channels (C3 and C4) used to analyze the movement intention, displayed in red in Figure 5.

The C3 and C4 channels were positioned on the sensorimotor cortex encompassing the motor cortex and somatosensory cortex in the central part of the cerebrum. Activation of the sensorimotor cortex is closely associated with human body movement, which can be directly correlated with the analysis of movement intention [21]. Moreover, the ERD and ERS related to movement occur prominently in the opposite hemisphere of the moving body part, and therefore the EEG data collected in the opposite hemisphere of the moving hand were utilized for analyses of the hand gripping and gamepad operation. For the task of using the balance handle with both hands, arithmetic averages of the collected data from each hemisphere were analyzed.

2.5. Signal Processing and Data Analysis

The root mean square (RMS) values of the sEMG signal data measured from the upper extremities were calculated to quantify muscle energy. A 50-ms window was used to capture the rapid dynamic contraction of the joint movement of the upper extremity. In order to compare the muscle activation trends between the main upper extremity muscles according to basic movement, down-sampling was performed at 10-ms intervals, followed by smoothing through a moving average filter at 300 ms (Figure 6).

On the other hand, as the time series data of the EEG signals were vulnerable to noises from the surrounding environment, a signal-processing protocol, containing a suitable filter and quantification processes, was designed to calculate the ERD/ERS data (Figure 7).

The EEG data collected at a frequency of 1024 Hz were down-sampled to 128 Hz, and since the length of the input signal was sufficiently long, the FIR (finite impulse response) filters for the α (8–12 Hz) and β (12–30 Hz) waves were employed to obtain the time series data for a specific frequency band and remove the high-frequency noises at the same time. The relative amplitude (RA) values for the idle rhythm state and the upper extremity movement due to the stimulus were calculated for quantification using Equations (1)–(3). Equation (1) signifies the rectification process that converts a negative EEG signal into a positive amplitude to assure that the sum of the amplitudes would not be zero. Equations (2) and (3) refer to extraction of the RA. In Equation (2), the pre-stimulus baseline is calculated to examine the relative impact of the stimulus. The RA is then calculated by subtracting the reference value from the time series data at each point in Equation (3). Here, $N$ refers to the total number of executions, y to the BPF process data of the jth sample of the ith trial, Act(j) to the average of the squared jth sample, and R to the reference, indicating the average of the section of 1 to 2 s prior to the stimulus.

$$Ac{t}_{\left(j\right)}=\frac{1}{N}{\displaystyle \sum }_{i=1}^N{y}_{ij}$$

(1)

$$R=\frac{1}{k+1}{\displaystyle \sum }_{j=r_0}^{r_0+k}Ac{t}_{\left(j\right)}$$

(2)

$$R{A}_{\left(j\right)}\left(\%\right)=\left(\frac{Ac{t}_{\left(j\right)}-R}{R}\right)\times 100$$

(3)

Due to the difficulty in identifying the RA pattern only with a single trial, the average of the multi-trial data was employed to eliminate the extreme outliers in the data and minimize noise. Moreover, the double moving average was used to more clearly represent the data patterns. As the ERD peak was detected before a stimulus was given, the minimum value was selected among the data collected 1 to 2 s before a stimulus occurred. The ERS peak was selected as the maximum value at 1 s after occurrence on the stimulus. The minimum ERD and the maximum ERS values are the key states of the CNS that express participation in the activity of responses to movement or stimulus, indicating an induced response of changes in EEG oscillation [22]. Averaging the induced response allowed us to minimize the noise data and increase the signal-to-noise ratio (SNR). As the change in the stimulus became more notable in the idle state, the amplitudes were compared with each other to determine the cognitive characteristics.

The normality of each data group was verified to compare the movement intention according to the operating method, and the two-way ANOVA was performed with a statistical significance level of 0.05. This method allowed to solve the increase in type-I error due to the multi-test problem when repeatedly using t-tests in multiple group comparisons, and also to validate the difference between groups with two or more independent variables. Therefore, we distinguished the tasks as a group of independent variables and set the α and β waves as independent variables and the ERD/ERS peak values as a dependent variable. The resulting F statistic was the difference between groups for intra-group differences, and the larger the F-value, the more pronounced the difference between the groups.

3. Results

3.1. sEMG during Operation of the Balance Handle

The sEMG signal data were recorded while the subjects operated the balance handle. In Figure 8a, “A” shows the inclination angles of the balance handle while the subjects performed the serious game by tilting the balance handle forward and backward. The sEMG signals measured while doing the extension and flexion movements from the triceps, biceps, and deltoid muscles are displayed in “B”, “C”, and “D”, respectively. During extension movement by tilting the balance handle forward, the greatest muscle activation was found in the triceps muscle (“B”). During flexion movement by tilting the balance handle backward, the largest muscle activation was observed in the biceps (“C”). In Figure 8b, “A” demonstrates the inclination angles of the balance handle while the subjects performed the serious game by tilting the balance handle leftward and rightward. The muscle activation in the right deltoid muscle during tilting the balance handle to the left was displayed in “B”, and the muscle activation in the left deltoid during tilting the balance handle to the right was displayed in “C”. As the muscle activation while using the balance handle demonstrated a similar tendency as other conventional upper extremity trainers, it was successfully validated that our balance handle was appropriate for interfacing with a serious game to conduct rehabilitation exercise programs [23].

3.2. EEG for the Movement Intention Test

Figure 9 shows representative RA patterns of the three types of operational methods (hand gripping, gamepad operation, and balance handle operation). It was clearly observed that the ERD phenomenon of the α and β waves temporarily decreased immediately after the “start”, which represented the occurrence time of the stimulus in the “stimuli” section. ERD generally refers to a temporary change in negative potential due to an instantaneous increase in the excitatory signals of the cortical neurons during the preparation for body movement; therefore, an analysis of ERD allows to determine the movement intention [24]. On the other hand, ERS occurring after an action is performed is accompanied with an increase in the α and β waves due to the positive potential changes following deactivation of the nervous system due to inhibitory signals. This refers to conversion into an idle state and, in the EEG tests, it was confirmed that ERS occurred after ERD in the “stimuli” section. Although the times of ERD/ERS in the RA patterns of the three operational methods were found to be similar, the magnitude of the amplitude with the balance handle was the largest, and the smallest magnitude of the amplitude was found with the gamepad.

Figure 10 demonstrates the box and whisker plots of the α and β waves of each task to quantitatively compare the magnitude of the amplitudes. As tilting the balance handle using the upper extremity was a physically different movement from pressing the buttons, tilting and button pressing were classified into Handle_I and Handle_B, respectively. The box represents the range of the first and third quartiles, the horizontal line in the box defines the second quartile, the x mark refers to the median, and the whisker indicates the range of the minimum and maximum values. The ERD/ERS peak distributions of the α and β waves were similar. The ERD peak indicated the lowest average value (α: −105.7%; and β: −81.1%) in the Handle_I task, whereas the ERS peak indicated the higher peak value of the Handle_I task (α: −188.9%; and β: 141.9%) and the Handle_B task (α: 171.3%; and β: 146.1%). A comparison of the average ERD values showed that tilting the balance handle resulted in the highest movement intention. Although pressing the button on the balance handle was physically not much different from hand gripping or pressing the button on the gamepad, a low ERD peak and a high ERS peak values were observed, expressing relatively high movement intention. However, several outliers were found in the Handle_B task due to the relatively low reproducibility of the ERS, resulting in large deviation. These outlier data are consistent with previous studies that reported that ERS might not be necessarily associated with signal generation or muscle activation in the motor cortex and rather reflected the short-term state to inhibit the motor cortex network [25]. Therefore, we intended to analyze the variance to verify the statistical significance of the ERD and ERS peak values for each task.

Table 1. Normality tests for the ERD and ERS peak values (* p < 0.05).

Peak	Rhythm	Task	Shapiro–Wilk Significance
ERD	α 8–12 Hz	Grip	0.111
		Gamepad	0.328
		Handle_I	0.278
		Handle_B	0.746
	β 12–30 Hz	Grip	0.198
		Gamepad	0.605
		Handle_I	0.052
		Handle_B	0.802
ERS	α 8–12 Hz	Grip	0.006 *
		Gamepad	0.023 *
		Handle_I	0.271
		Handle_B	0.589
	β 12–30 Hz	Grip	0.080
		Gamepad	0.090
		Handle_I	0.956
		Handle_B	0.005 *

shows the outcomes of the normality test. The Shapiro–Wilk method, suitable for less than 2000 samples, was used in the normality test. A normal distribution was considered if the p-value was larger than 0.05. The p-values for both the α and β waves of ERD were larger than 0.05, confirming that these data had a normal distribution. However, for the α wave ERS in the hand gripping and gamepad operation tasks, and for the β wave ERS in the balance handle button task, the normality was not verified, presumably attributing to the outliers found in the process of extracting the ERS peak values.

Table 2. Two-way ANOVA and post-hoc analysis (data shown as the mean ± SD, * p < 0.05).

	Rhythm	Task	Relative Amplitude (%)	F-Value	p-Value	Post-hoc
ERD	α 8–12 Hz	Grip (a)	−35.2 ± 26.4	6.914	<0.00 *	a, b, d > c (Bonferroni)
		Gamepad (b)	−23.1 ± 23.5
		Handle_I (c)	−105.7 ± 72.0
		Handle_B (d)	−46.8 ± 36.3
	β 12–30 Hz	Grip (a)	−33.0 ± 16.4	3.567	<0.02 *	b > c (Bonferroni)
		Gamepad (b)	−21.4 ± 44.1
		Handle_I (c)	−81.1 ± 60.9
		Handle_B (d)	−36.3 ± 42.2
ERS	α 8–12 Hz	Grip (a)	114.0 ± 48.2	0.782	0.512
		Gamepad (b)	141.7 ± 175.4
		Handle_I (c)	188.9 ± 122.7
		Handle_B (d)	171.3 ± 86.5
	β 12–30 Hz	Grip (a)	120.4 ± 67.1	0.247	0.863
		Gamepad (b)	107.1 ± 117.9
		Handle_I (c)	141.9 ± 40.5
		Handle_B (d)	146.1 ± 186.1

demonstrates the comparative data from the two-way ANOVA at a significance level 0.05 to identify the difference between the tasks with the operational methods and the post-hoc analysis using the Bonferroni method.

The α wave ERD revealed that the Handle_I task was more significant than the other three tasks (F(3,6) = 6.914, p < 0.05), and the F-value indicating the difference between the groups was clearly larger than the other groups. The β wave ERD showed that the Handle_I task significantly decreased for the gamepad operation task (F(3,6) = 3.567, p < 0.05). On the other hand, the ERS of the α and β waves demonstrated a large difference within the group but a small difference between the groups, and outliers were found in some data, indicating that these data were not suitable for comparison. Therefore, movement intention of tilting the balance handle for the α and β waves was clearly distinguished from the other tasks through ERD comparison; however, as the normality test of the ERS was not passed, there was no process characteristics found while recovering to the idle rhythm state after movement.

4. Discussion

Most of the post-stroke patients with hemiplegia have dysfunction of the upper extremity. The usage of an assistive trainer capable of bilateral exercise to help these patients exercise is an effective rehabilitation strategy [26]. Moreover, studies using serious games to reduce the rejection and boredom of patients toward treatment and induce voluntary participation have been actively ongoing in recent years [27]. In the present study, we developed a novel balance handle as a symmetric upper extremity trainer capable of extension, flexion, pronation, and supination of the upper extremity. We collected the sEMG signal data while the subjects were playing a serious game and recorded the EEG signal data while the subjects were performing basic movements with the balance handle, to analyze the effectiveness of the device as an assistive tool for rehabilitation. In particular, the effectiveness test using the EEG signals was conducted to compare the movement intention with the balance handle to the movement intention in other types of operational methods (hand gripping and gamepad operation) via analysis of the ERD and ERS values. Normality tests were also conducted, and two-way ANOVA and post-hoc analyses were performed for comparative studies.

The triceps brachii were activated during the extension movements, whereas the biceps brachii and deltoid muscles were activated during the flexion movements. The deltoid muscles on the opposite side were activated during the tilting of the balance handle to the left or to the right. This ensured that the extensor and flexor muscle activities, essential for increasing the hand and foot functions while moving the balance handle, were properly executed, demonstrating similar sEMG characteristics to other upper extremity trainers [28]. With the balance handle, the ERD peak values were relatively lower while showing higher ERS peak values compared to other types of operating methods. It was noteworthy that the task of pressing the balance handle button revealed a decrease in the ERD peak compared to the task of hand gripping and gamepad operation, which did not show much difference in exercise volume. The lower the ERD peak was, the higher the movement intention was; therefore, it can be inferred that a higher level of movement intention was expressed in the task of utilizing the balance handle [29]. However, for the ERS peak, the higher values reflected more short-term states of inhibiting movement, which corresponds to a previous study that the ERS phenomenon was not necessarily associated with muscle activation in motor cortical networks [30]. As there were some outliers found and data normality was not achieved, detailed classification of the frequency bands needs to be further conducted to reduce individual differences related to motor functions.

Most of the previous studies have demonstrated the prediction of movement intention for only a single type of upper extremity movement or either left or right side of the body movement using ERD analysis [31,32]. By contrast, the present study derived movement intention while users operated the balance handle, and showed a considerable difference compared to the other types of operational methods. These data demonstrated the potential for various applications using the upper extremity trainer to bring out an excellent rehabilitative effect, not only through muscle growth but also via identification of large movement intentions inducing brain activation. To develop specific applications to robot rehabilitation or nerve control using brain–computer interface (BCI) technologies, further studies are required to achieve consistent outcomes in securing and comparing the movement intention patterns; for example, by improving the signal processing methods.