1. Introduction
Brain computer interfaces (BCIs) interpret brain signals, allowing communication with computers by bypassing the usual neuronal muscular pathways [
1]. Brain signals can be obtained through various techniques, including electroencephalography (EEG), electrocorticography (ECoG), functional near-infrared spectroscopy (fNIRS), or functional magnetic resonance imaging (fMRI), with EEG being the most commonly used. When a patient loses the ability to move all voluntary muscles except the eyes, they are referred to as being in a locked-in state (LIS) [
2,
3]. This state is commonly associated with motor neuron degenerative diseases, such as ALS, as well as stroke and traumatic brain injury. Several studies report that patients in LIS can use BCI and eye-tracking assistive technologies to communicate with people in their environment [
4,
5]. However, when patients’ voluntary eye movements fade, and a patient enters a completely locked-in state (CLIS), they lose the ability to control eye-tracking devices, making the use of BCIs the only available means of communication. Unfortunately, BCIs still remain very ineffective for the CLIS population (see
Table 1 for main results related to the current state of the art). Even binary YES/NO communication is very difficult to obtain accurately and systematically. CLIS patients cannot control eye and eye lid movements, leading to eye dryness and the potential development of corneal ulcers without proper care. Visual-only BCIs might not provide the desired outcome of a possible method of communication, although Steady State Visual Evoked Potential (SSVEP) interfaces have proved to be effective in [
6]. In this study, authors used SSVEPs evoked by LEDs and instructed patients to either attend or ignore the presented stimuli, detecting user intention though SSVEP power spectral density (PSD). For one of the patients transitioning from LIS to CLIS and experiencing loss of eye movement control, the LED was positioned approximately 15° from the center of the optic axis to enable attention shifts without requiring the patient to direct their gaze. Auditory BCIs usually have a lower classification result in healthy populations than visual BCIs but remain one possible via for patients in CLIS, as the auditory system is believed to remain intact until late stages of the disease. Pires et al. [
7] used an auditory and a hybrid audiovisual oddball paradigm with spoken words to try to communicate with a CLIS patient, showing a better neurophysiological signal with the auditory interface than with the hybrid interface, although no effective communication was achieved. Guger et al. [
8] used vibrotactile stimulation with two or three stimulators (one in each wrist, which were associated with yes and no answers, and a third one in the shoulder that acted as a distractor) and used the oddball paradigm to provide a binary communication system. Han et al. [
9] used a left motor imagery and mental subtraction task to implement a binary classifier, achieving up to 87.5% online performance with a CLIS patient. Naito et al. [
10] tested an fNIRS-based BCI with 17 CLIS patients performing mental tasks associated with yes/no answers. Ardali et al. [
11] also used an fNIRS-based BCI to obtain yes/no answers from CLIS patients by associating semantic content with the questions. Chaudhary et al. [
12] tested, very recently and for the first time, an intracortical BCI with a CLIS patient. The patient was implanted with two 64-microelectrode arrays in the supplementary and primary motor cortex. The BCI paradigm was based on an auditory neurofeedback strategy to communicate. The brain signals allowed differentiation between low-pitched and high-pitched tones, with an accuracy above 80% on most days. This strategy was then employed in a speller interface that allowed the patient to communicate freely. The patient could correctly spell 131 characters per day, which is a promising result, although it has the disadvantage of being highly invasive. For a recent review on BCIs used with patients in LIS and CLIS, please refer to [
13].
The state of the art presented above shows that clinical experiments with CLIS patients are quite insipient, very experimental, and currently limited to yes/no detection. Further studies are needed, and these studies should be longitudinal in nature to gain a better understanding of patients’ brain states, including attention spans, fatigue, and variability [
14], and to adjust interfaces to the specificities of each patient. The trial-and-error approach in the design process takes time, especially considering the possibility that patients may not fully comprehend the mental tasks proposed [
15,
16]. Achieving a long-term, at-home BCI system remains a distant goal that requires substantial progress.
In this study, our primary objective was to establish communication with a CLIS patient utilizing a P300-based BCI. The option of using a P300 oddball paradigm relied on the fact that this approach has been the most common and reliable for long-term use of BCI with patients in late ALS stages including LIS, offering the benefit of possibly being seamlessly used when patients transition from LIS to CLIS [
17]. Over the course of ten months, we followed the patient while developing and testing various P300-BCI versions that we expected to address communication challenges and overcome the patient’s difficulties. Therefore, the interfaces were designed with a patient-centered approach, but the iterative process design turned into a blind trial-and-error endeavor due to the patient’s lack of communication. We developed and tested nine interfaces encompassing different visual, auditory, and hybrid audiovisual stimuli. Some of these interfaces were also tested in a systematic way by a control group of five participants for validation and comparison.
2. Methods
2.1. Participants
Each developed interface underwent pilot testing with healthy volunteers before being evaluated with the patient to ensure proper functionality of the interfaces and the intended effects. Systematic tests involving a control group and a subset of the developed BCIs were then performed for reference and to compare neuronal patterns and performance with those obtained with the patient.
2.1.1. CLIS Patient
The patient, a 54-year-old female with limb onset ALS diagnosed 46 months prior to the initial BCI session, in February 2023, had an ALS functional rating scale-revised (ALSFRS-R) score of 1. At the beginning of the experiments, she had no eyelid and vertical eye movements and could no longer use the eye-tracking device she used previously. She had slight horizontal movement of the eyes, interpreted as a ‘Yes’ response, while the absence of movement indicated a ‘No’. Only the patient’s husband could interpret the slight eye movement.
Although she had lost control of the eye-tracking device, it continued to serve as a means for her family to discern whether her eye was open and to detect attempts at eye movement, which were indicative of communication efforts. The patient subsequently entered CLIS some months later. Despite the loss of direct communication, the ERP waveform analysis suggested that the patient retained visual, hearing, and cognitive functions, which motivated us to continue the experiments.
Table 2 summarizes the main clinical characteristics of the patient 8 months after the initial BCI session.
This study was approved by the Ethical Committee for Health of the University Hospital Center of S. João, Porto (CHUSJ), complying with the code of Ethics of the Declaration of Helsinki. Informed consent was obtained from the patient’s husband. The decision to participate in this study was made by the patient’s family after meetings with the research team and the medical staff of the CHUSJ palliative care service where the patient was being followed.
2.1.2. Control Group
The control group consisted of 5 (5 female) healthy volunteers (22.0 ± 2.0 years) with normal or corrected vision and hearing. An informed consent was given by each participant.
2.2. Data Acquisition
A 16-channel g.USBamp acquisition device (g.tec medical engineering GmbH, Schiedlberg, Austria) was used to acquire EEG signals at a sampling rate of 256 Hz. EEG was acquired with 16 g.Ladybird electrodes (Fz, Cz, C3, C4, CPz, Pz, P3, P4, P07, P08, POz, Oz, FPz, FCz, FC1, and FC2) placed in a g.GAMMAcap2 (g.tec medical engineering GmbH, Austria) according to the extended international 10–20 System. The right earlobe was used as reference and the AFz channel for ground. All electrodes were connected to a g.GAMMAbox (g.tec medical engineering GmbH, Austria). Signals were filtered by a band-pass filter between 0.1 and 30 Hz and a notch filter at 50 Hz to eliminate the powerline interference. Data were acquired, processed, and classified in real-time in a Highspeed online processing Simulink framework.
2.3. Experimental Protocol
The experiments with the patient were conducted at her home. The patient was in a wheelchair, in a reclined position. The visits to the patient were done in the late morning when the patient showed longer attention spans, according to information provided by the husband. Each visit lasted no longer than two hours. We performed 16 visits to the patient for data acquisition and testing. At each visit, we performed one or more sessions, where we tested the same or different interfaces. A total of 31 sessions were conducted during the 16 visits.
The developed interfaces comprised visual (text, symbols, and images), auditory (spoken words and white noise), or hybrid stimuli (combining visual and auditory stimuli). The visual stimuli were presented via a screen located at approximately 1 m from the patient’s line of sight, and the auditory stimuli were presented through earphones. The stimuli differed on the different interfaces, as explained in
Section 2.4.1, but the number of symbols remained the same during our experiments. The BCIs evaluated in this study consisted of a P300 oddball paradigm (1 Target and 6 standards (non-Targets)), consisting of seven stimuli that provide a small Portuguese lexicon, namely ‘SIM’ (Yes), ‘NÃO’ (No), ‘TOSSE’ (Cough), ‘TV’, ‘AJUDA’ (Help), ‘SONO’ (Sleep), and ‘STOP’, which, according to the patient’s family members, were the main needs of the patient. The stimuli had a duration of 550 ms and an interstimulus interval (ISI) of 100 ms.
Each online session was preceded by a calibration to obtain training data for fitting the classification model. The calibration process (
Figure 1a) consisted of the patient focusing on one word at a time and mentally counting the number of times each word appeared on the screen and/or was spoken. Each target word was repeated 9 times interleaved with the non-target words, according to the oddball paradigm, to induce the target-related potentials. In total, there were 12 words in the training sequence, corresponding to 108 (12 × 9) target trials and 648 (12 × 9 × 6) non-target trials.
During the online session (
Figure 1b), the patient was asked to perform a copy-paste task, wherein she was instructed with one word and required to mentally count the number of times the specified word occurred. The number of repetitions for each selection was adjusted to 9, although different numbers of repetitions were tested in some sessions for comparison. Each copy task consisted of 10 selections.
After the fourth visit to the patient, due to the very low BCI performance achieved, two modifications were introduced to the experimental procedure: (1) only the words ‘SIM’ and ‘NÃO’ were used in the training sequence, while the P300 oddball paradigm with the seven symbols remained the same; and (2) in the copy task, we started applying a filter in the classification algorithm, limiting the selections to ‘SIM’ and ‘NÃO’ choices, i.e., transforming the 7-class problem into a binary classification. The oddball paradigm with the seven symbols remained the same as in calibration.
The control group performed the experiments in a laboratory environment, always using the 7-class BCI.
2.4. BCI Paradigm Design: Visual and Auditory Stimuli
The visual and auditory stimuli were implemented recurring to the Psychophysics toolbox [
18,
19] in Matlab R2021b, running on a laptop with the Windows 11 operating system. Visual stimulation was displayed to the participants on the laptop’s built-in 15.6″ screen to avoid multi-screen desynchronization of the stimuli, and the auditory stimulation was delivered through Mi In-Ear Headphones Basic (Xiaomi Communications Co., Ltd., Beijing, China).
2.4.1. Stimuli
The BCI was implemented using the g.tec’s Highspeed Simulink framework. The g.USBamp driver set the sampling rate for EEG acquisition and triggered both visual and auditory stimuli. The psychophysics routines for stimuli implementation were embedded in a 2-level Simulink s-function. The stimuli consisted of text words, spoken words, white noise audio, standard face images, and family face images.
The audio files were recorded in a Huawei P20 lite phone with Lexis Audio Editor version 1.2.158, converted from mono to stereo, and saved in the wav audio format. These files were then edited to increase the volume and filtered to reduce background noise in Audacity (Audacity Team, version 3.3.2). The white noise audio file was generated in Matlab 2021b.
The standard face images were obtained from the Radboud Faces Database [
20]. Patient’s family member’s photos were obtained from the patient’s caregiver. The background was removed from the images and changed to black, leaving only the family members’ faces visible.
2.4.2. BCI Versions and Iterations
The different interfaces tested were developed according to the patient’s needs for basic communication, with the goal of providing a small lexicon. The interfaces were designed iteratively, aiming to overcome the challenges posed by the lack of eye movement and aiming to simplify the interface and increase the patient’s attention and engagement.
Table 3 lists the iterations of the developed BCIs. In the following section, we describe the several iterations and their rationale (videos for each interface are available at
https://home.isr.uc.pt/~gpires/videos/BCI4ALL/videos.html (accessed on 11 March 2024), except for the familiar faces’ interface due to privacy reasons).
As a starting point, we used the visual interface tested in [
7], adjusted to seven Portuguese text words (‘Sim’, ‘Não’, ‘Tosse’, ‘Ajuda’, ‘TV’, ‘Sono’, and ‘Stop’) associated to arrow symbols. The stimuli were highlighted by switching color from gray to green. The symbols were positioned close together to be within the patient’s foveal region. Given the poor ERP results obtained with this interface obtained in the first sessions, we decided to replace the text/arrow symbols with larger images covering the entire range of the screen. At the flash time, the text word was replaced by an image of an emotionally neutral person. The rationale of using faces as stimuli was to trigger ERPs related to face detection and processing and to increase task engagement [
21,
22,
23]. Given the improvements in ERPs and in performance, we replaced the standard face with 7 images of the family, expecting to further improve task engagement and enhance ERPs.
Given the vision impairments in CLIS patients related to the absence of gaze movements and eye dryness, auditory stimuli were added to the previous interfaces. The auditory stimuli were the 7 spoken words. Purely auditory interfaces and hybrid visuo-auditory interfaces were created.
Expecting to increase selective attention to ‘SIM’ and ‘NÃO’ options, the previous versions were modified, replacing the spoken words ‘Sono’, ‘Ajuda’, and ‘Stop’ with white audio noise.
To further improve the visual component of the ERP, a new grid layout was developed, consisting of two rows and four columns with both the words and the familiar and standard faces displayed in a larger size. The distance between the patient and the screen was adjusted to ensure full view of the grid layout in the foveal region.
Given the significant gaze restriction, the last modification to the visual component of the interface involved relocating the ‘SIM’ and ‘NÃO’ words from opposite columns to the same column within the layout. This adjustment was made to ensure that the target stimuli were in a smaller visual local field.
All visual interfaces mentioned above had an audiovisual version with all spoken words and an audiovisual version with three words as white noise and the remaining as spoken words.
Figure 2 shows the final visual layout with the ‘AJUDA’ word in the ON state.
2.5. Classification Pipeline
The classification pipeline is similar to that used in [
7,
24]. A 1 s EEG segment (epoch) is extracted for each event in the oddball paradigm. The segments are averaged according to the number of repetitions used and then normalized to have zero mean and standard deviation of 1. Then, the EEG of the 16 channels is projected with a statistical spatial filter called Fisher Criterion Beamformer (FCB) [
25] into two projections, aiming to simultaneously increase the discrimination between target and non-target epochs and to reduce space dimensionality. The best features of the projections are then selected using the r-squared method and used for classification with a Naïve Bayes classifier. The BCIs tested with the patient include a postprocessing filter to exclusively select ‘YES’ or ‘NO’ target options. The BCI pipeline from stimulation to classification is depicted in
Figure 3. A demonstration code for this classification pipeline is available in Matlab and Python at
https://github.com/gpiresML/FCB-spatial-filter (accessed on 11 March 2024).
4. Discussion
The main goal of this study was to establish an effective communication channel with a CLIS patient. With this purpose, we tested multiple stimulation modalities with various features and combined different layouts and stimulation strategies. Our aim was to identify an effective combination that allows the exploration of perceptual and cognitive functions believed to be common among individuals in CLIS. The rationale behind the modifications implemented in the nine BCI versions was two-fold. Firstly, we aimed to adjust the interfaces to accommodate the perceptual limitations of the patient, which involved testing different visual layouts and stimuli sizes. Secondly, we sought to enhance selective attention and task engagement by introducing stimuli based on images with emotional content and using auditory white noise stimuli associated to non-target events to draw attention to target events. The study shows that the different types of BCI stimulation caused different responses but also that the responses greatly varied within the same type of stimulation. Both the AV and AVN conditions showed larger discrimination windows when compared to visual and auditory-only conditions. This positively correlates with the online performance, where these conditions also attained better results more times. The visual-only conditions showed lower results, likely influenced by possible eye vision deterioration and the patient’s tendency to close their eyelids, impacting BCI performance. Nevertheless, the visual component remained important, especially when combined with auditory stimuli, as the combined performance surpassed that of visual-only and auditory-only conditions.
The control group achieved very high classification results for all interfaces requiring a low number of repetitions to control the BCI system. This outcome was anticipated due to the simplicity of the BCIs affirming their functionality and the effectiveness of the classification pipeline. The best performing interface for this group was the AVNsf condition, with 98.0 ± 4.5% accuracy for 2.6 ± 1.5 repetitions for the seven-class BCI, which englobed audiovisual stimulation with the white noise audio component. However, no conclusions about the influence of the white noise component can be made for the CLIS patient since the corresponding interface for the patient had both the highest (90%) and lowest (30%) accuracies for the two-class BCI employed. This leads us to conclude that performance results heavily rely on the mental state of the participant during the sessions. The performance fluctuation across sessions may be related to variation in attention and vigilance, as well as in motivation and mood, due to antidepressant medication. The significant gap between calibration and online results obtained with the patient underscores the necessity of exploring alternative classification approaches that are more adaptable in contexts of high EEG variability. While the current processing methods applied are well-established and have shown effectiveness in healthy populations and individuals in advanced stages of ALS, exhibiting relatively stable data, they proved ineffective when applied to the highly variable data of the CLIS patient.
The neurophysiological results obtained with the control group clearly show the expected ERPs in oddball paradigms, namely the N200 and P300 components. However, the neural responses observed in the CLIS patient are inconsistent and do not exhibit evident N200 and P300 ERPs. Moreover, there is no clear waveform discrimination between target and non-target responses, as shown in
Figure 5. A notable exception comes from the data collected during one session under AVNsf condition, as depicted in
Figure 4, where the P300 is evident and indicates discriminative power. The N200 is also present but appears weak and lacks discriminative power.
Figure 5 highlights the low discrimination between target and non-target stimuli, with much smaller time windows observed for the CLIS patient compared to the control group.
There are several possible causes contributing to the ERP variability, resulting in atypical average signals and the consequent low SNR and reduced discriminative power. These factors may include fluctuations in attention and vigilance levels, the cognitive workload demanded by the task, and the patient’s limited working memory for sustaining attention on target symbols. These processes, which can be related to reduced executive functions, can lead to high variability in ERP latency and amplitude. Additionally, external factors such as repetitive stimulation, inducing SSVEPs, and distractors from adjacent stimuli may have also contributed to the variability and decrease in target discrimination.
Nevertheless, the ERPs were more consistent than those obtained in our exploratory study [
7] involving a CLIS patient. Previous research has shown that, while a unique common pattern in the EEG signal of CLIS patients could not be found, it was common for the EEG signal to show slowing and attenuation of alpha activity, along with significantly altered auditory-evoked potential responses compared to those of healthy individuals [
29]. Additionally, an ECoG study involving a late-stage ALS patient [
30] revealed that the P300 component of the signal, detectable when the patient was still able to communicate through voluntary muscle control, became undetectable three months after the last communication was made.
Given the highly atypical nature of the observed patient’s ERP waveforms, further methods and studies are required to understand what potential (confounding) factors affects target ERPs and to identify other neural correlates that can help predict BCI performance. To further elucidate potential signaling effects, future studies should incorporate BCI paradigms that systematically assess neural responses at different levels, including reflexive, perceptual, and cognitive processes and isolating the effects of different stimulus characteristics. This should occur early on at the LIS stage, when patients can provide more reliable feedback.
Our study provides valuable insights into potential strategies to enhance the feasibility and efficacy of BCIs for CLIS patients. However, both neurophysiological and classification results cannot be generalizable to other CLIS patients, given the significant variability in individual characteristics and conditions within this population.
EEG-based BCIs are still not a reliable communication system for at-home, day-to-day use for CLIS patients [
17]. Moreover, there are only a handful of successful studies utilizing a functional BCI-based system for CLIS patients. Although we achieved three sessions with an online classification equal to or above 80%—surpassing the 56.9% achieved in [
11] and comparable to accuracies in [
6,
8], all achieved with non-invasive techniques—our results were not consistent and cannot be considered successful as there was no repetition of those results. However, this underscores the potential of the proposed approaches to be effective for these patients.
The use of BCIs for communication in patients in CLIS would improve the quality of life of these patients by allowing them to regain a voice in the decision-making process of their caretaking needs, as well as allowing a level of social interaction with their surroundings, family members, and caretakers. In other words, this would represent a qualitative improvement within the goals of personalized medicine.
Longitudinal studies are essential for achieving more effective BCIs and could provide us with additional insights into the neuronal changes during the transition from LIS to CLIS [
31,
32]. For this to happen, BCI studies should start once a patient enters LIS, a state in which patients can still communicate with eye blinks, eye movements, or alternative communication tools, such as eye-tracking spellers. The implementation of BCI technology for patients in LIS allows experimenters to directly question a patient’s comprehension on how to use the system, and possibly facilitates them becoming proficient users before entering CLIS. BCI studies starting when the patient enters LIS would also allow a longitudinal assessment of the hypothesis of the extinction of goal-directed thinking [
15,
31]. This hypothesis states that there is a reduction in arousal, vigilance, and working memory in patients in CLIS due to the lack of social-cognitive interaction. The continuous use of BCIs would allow uninterrupted communication, even after losing eye movements, with family members and caretakers, preventing a decline in executive function. In [
6], a patient who progressed from LIS to CLIS retained the ability to use the BCI acquired in LIS. Additionally, introducing the interface before entering in CLIS would enable patients to offer valuable insights, providing feedback on interface design, and express any concerns or doubts about how the system operates, which were not possible to ascertain in our study. This would enable improved personalization. The introduction of BCI for assistive contexts before an ALS patient even reaches LIS has been explored in [
33,
34] showing that the mental strategies used can become helpful in translating these mental strategies when the patient reaches more advanced states. Thus, early BCI introduction in earlier stages of ALS, even in LIS, could help prolong patients’ communication capabilities.
Future work is expected to include larger-scale longitudinal studies, with the inclusion of both LIS and CLIS patients, to address the aforementioned aspects and achieve a more optimized combination of stimulation strategies and classification algorithms tailored to patients’ specific needs. Furthermore, BCI sessions with patients should occur in periods when the patients are known to be awake/vigilant. It is essential to find biomarkers of patients’ vigilance levels as well as performance predictors. With these goals, an automatic vigilance detector is underway towards implementation. Additionally, shorter calibration sessions are required to reduce patient fatigue. Within our group, efforts have been made to reduce calibration times, albeit so far only having been tested with healthy participants [
28].