Next Article in Journal
Uncertainty-Aware Deep Learning-Based Cardiac Arrhythmias Classification Model of Electrocardiogram Signals
Previous Article in Journal
On the Effect of Standing and Seated Viewing of 360° Videos on Subjective Quality Assessment: A Pilot Study
Previous Article in Special Issue
Machine-Learning-Based Emotion Recognition System Using EEG Signals
Article

Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems

1
School of Engineering, RMIT University, Bundoora, VIC 3038, Australia
2
THALES Australia—Airspace Mobility Solutions, WTC North Wharf, Melbourne, VIC 3000, Australia
3
Northrop Grumman Corporation, 1550 W. Nursery Rd, Linthicum Heights, MD 21090, USA
*
Author to whom correspondence should be addressed.
Distribution Statement A: Approved for Public Release; Distribution is Unlimited; #21-0655; Dated 26 April 2021.
Academic Editor: Larbi Boubchir
Received: 19 April 2021 / Revised: 9 June 2021 / Accepted: 11 June 2021 / Published: 16 June 2021
(This article belongs to the Special Issue Machine Learning for EEG Signal Processing)
With increasingly higher levels of automation in aerospace decision support systems, it is imperative that the human operator maintains the required level of situational awareness in different operational conditions and a central role in the decision-making process. While current aerospace systems and interfaces are limited in their adaptability, a Cognitive Human Machine System (CHMS) aims to perform dynamic, real-time system adaptation by estimating the cognitive states of the human operator. Nevertheless, to reliably drive system adaptation of current and emerging aerospace systems, there is a need to accurately and repeatably estimate cognitive states, particularly for Mental Workload (MWL), in real-time. As part of this study, two sessions were performed during a Multi-Attribute Task Battery (MATB) scenario, including a session for offline calibration and validation and a session for online validation of eleven multimodal inference models of MWL. The multimodal inference model implemented included an Adaptive Neuro Fuzzy Inference System (ANFIS), which was used in different configurations to fuse data from an Electroencephalogram (EEG) model’s output, four eye activity features and a control input feature. The online validation of the ANFIS models produced good results, while the best performing model (containing all four eye activity features and the control input feature) showed an average Mean Absolute Error (MAE) = 0.67 ± 0.18 and Correlation Coefficient (CC) = 0.71 ± 0.15. The remaining six ANFIS models included data from the EEG model’s output, which had an offset discrepancy. This resulted in an equivalent offset for the online multimodal fusion. Nonetheless, the efficacy of these ANFIS models could be confirmed by the pairwise correlation with the task level, where one model demonstrated a CC = 0.77 ± 0.06, which was the highest among all of the ANFIS models tested. Hence, this study demonstrates the suitability for online multimodal fusion of features extracted from EEG signals, eye activity and control inputs to produce an accurate and repeatable inference of MWL. View Full-Text
Keywords: mental workload; EEG; eye tracking; control inputs; closed loop system adaptation; adaptive automation; multimodal data fusion; machine learning; ANFIS mental workload; EEG; eye tracking; control inputs; closed loop system adaptation; adaptive automation; multimodal data fusion; machine learning; ANFIS
Show Figures

Figure 1

MDPI and ACS Style

Planke, L.J.; Gardi, A.; Sabatini, R.; Kistan, T.; Ezer, N. Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems. Computers 2021, 10, 81. https://0-doi-org.brum.beds.ac.uk/10.3390/computers10060081

AMA Style

Planke LJ, Gardi A, Sabatini R, Kistan T, Ezer N. Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems. Computers. 2021; 10(6):81. https://0-doi-org.brum.beds.ac.uk/10.3390/computers10060081

Chicago/Turabian Style

Planke, Lars J., Alessandro Gardi, Roberto Sabatini, Trevor Kistan, and Neta Ezer. 2021. "Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems" Computers 10, no. 6: 81. https://0-doi-org.brum.beds.ac.uk/10.3390/computers10060081

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop