Next Article in Journal
A New Readout Method for a High Sensitivity Capacitance Sensor Based on Weakly Coupled Resonators
Previous Article in Journal
Automatic Detection of Arrhythmias Using a YOLO-Based Network with Long-Duration ECG Signals
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Proceeding Paper

A Workflow for Affective Computing and Stress Recognition from Biosignals †

Section Medical Psychology, University of Ulm, Frauensteige 6, 89075 Ulm, Germany
*
Author to whom correspondence should be addressed.
Presented at the 7th Electronic Conference on Sensors and Applications, 15–30 November 2020; Available online: https://ecsa-7.sciforum.net/.
Published: 14 November 2020
(This article belongs to the Proceedings of 7th International Electronic Conference on Sensors and Applications)

Abstract

:
Affective computing and stress recognition from biosignals have a high potential in various medical applications such as early intervention, stress management and risk prevention, as well as monitoring individuals’ mental health. This paper presents an automated processing workflow for the psychophysiological recognition of emotion and stress states. Our proposed workflow allows the processing of biosignals in their raw state as obtained from wearable sensors. It consists of five stages: (1) Biosignal Preprocessing—raw data conversion and physiological data triggering, relevant information selection, artifact and noise filtering; (2) Feature Extraction—using different mathematical groups including amplitude, frequency, linearity, stationarity, entropy and variability, as well as cardiovascular-specific characteristics; (3) Feature Selection—dimension reduction and computation optimization using Forward Selection, Backward Elimination and Brute Force methods; (4) Affect Classification—machine learning using Support Vector Machine, Random Forest and k-Nearest Neighbor algorithms; (5) Model Validation—performance matrix computation using k-Cross, Leave-One-Subject-Out and Split Validations. All workflow stages are integrated into embedded functions and operators, allowing an automated execution of the recognition process. The next steps include further development of the algorithms and the integration of the developed tools into an easy-to-use system, thereby satisfying the needs of medical and psychological staff. Our automated workflow was evaluated using our uulmMAC database, previously developed for affective computing and machine learning applications in human–computer interaction.

1. Introduction

Affective Computing is a multidisciplinary field with high potential in many human–computer interaction applications [1]. Since Picard proposed the concept of affective computing, researchers from various disciplines have been investigating diverse perspectives, ranging from theories to applications and from design to evaluation. Among them is affective computing for healthcare technologies and medical applications [2]. One emerging application associated with the medical field is the emotion and stress recognition [3]. It is a promising topic seeing its wide prospect in daily applications such as early intervention, stress management, risk prevention as well as monitoring individuals’ mental health. In this context, various modalities ranging from facial, speech, text and biosignal analysis have been adopted for the purpose of emotion and stress recognition [4]. Among these modalities, psychophysiological signals have the valuable advantage as “honest signals”: they cannot be easily triggered by any conscious or intentional control and are continuously available [5]. Affective computing from biosignals acquired through wearable sensors, adds the convenience of mobile implementation in real-life in-the-wild applications [6,7].
For the implementation of affective computing and stress recognition in daily life applications, several prerequisites have to be achieved. On one side, the reflection of affective states in physiological patterns demands a particularly robust and reliable biosignal analysis [8,9]. On the other side, the integration of the recognition process into daily applications of non-experts in biosignal processing and machine learning requires an automated processing of the workflow stages [10,11]. This paper presents an automated processing workflow for the psychophysiological recognition of emotion and stress states. Our proposed workflow allows processing biosignals in their raw state as obtained from wearable sensors. It consists of five stages, allowing biosignal preprocessing, feature extraction, feature selection, affect classification and model validation. These aspects will be discussed in the following sections.

2. Materials and Methods

The proposed automated biosignal processing workflow for affective computing and stress recognition consists of the stages illustrated in Figure 1.
These workflow steps are described in the following subsections.

2.1. Psychophysiological Biosignals

Our workflow allows processing biosignals in their raw state as obtained from the bioamplifiers. They are usually acquired via electrodes and sensors connected to the subjects. The biosignals currently implemented in the workflow include the following physiological channels:
  • Electrocardiography (ECG): It measures the electrical cardiac activity and is related to the activity of the sympathetic nervous system. The magnitude of the electrical potential is obtained from the difference between positive and negative electrodes placed on the skin surface.
  • Electromyography (EMG): It measures the electrical muscle activity and is also related to the activity of the sympathetic nervous system. Relevant EMG channels include the Zygomaticus, Corrugator and Trapezius muscles known to be active during emotions.
  • Electrodermal activity (EDA) in terms of Skin Conductance Level (SCL): It indicates the activity of the sweat glands in the skin and is directly regulated by the sympathetic nervous system and therefore sensitive to external stimuli.
  • Respiration (RSP): It measures breathing patterns such as breathing frequency or relative depth of breathing. Respiration data can be acquired using piezoelectric sensors that react to pressure variations caused by the thoracic and abdominal fluctuations during respiration. The sensors are usually embedded into an elastic belt system worn around the thorax.
  • Temperature (TEMP): It detects the changes in hotness and coolness of the skin. Body temperature is measured using a temperature sensor placed on the skin (for example on the fingers) and that converts the temperature changes into an electrical signal.
The biosignal data used for developing the present workflow were acquired with the bioamplifier system g.MOBIlab+ from g.tec. The data are part of our previously acquired uulmMAC dataset—A Multimodal Affective Corpus for Affective Computing in Human–Computer Interaction—that is freely available for academic use and research applications [12].

2.2. Biosignal Preprocessing

The first analysis step after the biosignal acquisition and prior the feature extraction is the preprocessing of the biosignal raw data. This includes extracting the raw data from the bioamplifiers and converting them into a readable format, triggering the physiological data with the help of subject-specific logger files, cutting irrelevant data and selecting relevant information from the data, and performing various filtering steps to the different biosignal channels.
The main goal of filtering is to remove various artifacts and reduce the amount of noise within each specific biosignal channel. For instance, ECG signals are affected by different kinds of noise including: baseline wander mainly caused by patient respiration, power line interference due to differences in the electrode impedances and to stray currents through patient and cables (considered as narrow-band noise and centered around 50 Hz and 60 Hz), as well as high frequency noise and other artifacts generated from equipment and the environment. Therefore, various signal-specific algorithms and filtering strategies are used for removing different kinds of noise and artifacts including bandpass filtering, signal detrending and artifact correction. In our workflow, we implemented butterworth bandpass filters and low-pass filters. Butterworth bandpass filters can be used to eliminate noise and artifacts from ECG signals, to isolate the bursts of EMG signals that contain relevant information about muscle activity, and to optimize the RSP signals. Low-pass filters are implemented to smooth the EDA and TEMP signals. The cut-off frequencies and filter orders can be easily adapted to individual channels and different datasets.
Finally, the resulting filtered data are saved and exported into a readable Matlab format for further processing.

2.3. Feature Extraction

The next step after data preprocessing is to extract appropriate features from the biosignals for an accurate further analysis. Therefore, each selected and preprocessed signal is first decomposed into small sliding windows. Reasonable window sizes for biosignals are between 5 s and 10 s with sliding steps in the range of 1 s or 2 s. Standard feature extraction methods based on pure morphological characteristics are not always sufficient for a reliable representation of specific affective states. Therefore, extended features shown to be efficient in the pain recognition field are adopted and implemented in our workflow for affective computing and stress recognition [13]. The features are extracted from six mathematical groups including amplitude, frequency, linearity, stationarity, entropy and variability. This set of features from the different mathematical groups is calculated for each of the biosignal channels. While these features contain meaningful information for the muscle, skin, respiration and temperature signals, the analysis of cardiovascular signals requires the extraction of specific ECG wave characteristics of the heart beats, which are essential for reliable information interpretation. Therefore, additional ECG specific features such as QRS complex detection, various interval durations and heart rate variability are also computed based on our ecgFEAT toolbox for cardiovascular feature extraction and analysis [14].
The extracted features are then normalized by performing a feature z-transformation for each participant and channel. The standardization converts the feature values into z-scores by subtracting the mean and dividing by the standard deviation.

2.4. Feature Selection

Among the large number of extracted features, only non-redundant and relevant ones should be selected for further processing. This is necessary in order to enhance the speed of the algorithms and to increase the efficiency of the recognition. A pre-selection is first performed using feature reduction to exclude similar redundant features based on statistical analysis with the Pearson correlation coefficient r. Thereby, the dimension of the features is reduced by examining each pair of features under the condition r > 0.95. In case of correlated features, the feature requiring the higher computation time is removed.
For further enhancing the efficiency of the classification, only relevant decisive features should be selected for further processing. Therefore, various feature selection methods are available based on different strategies. In our workflow, the following three feature selection methods are implemented including: Forward Selection (FS), Backward Elimination (BE) and Brute Force (BS) methods. The Forward Selection method begins with an empty selection of features and subsequently adds one feature (from the features set obtained after removing correlated features), which leads to the highest increase in performance in each iteration. The process is repeated until no increase in performance is observed anymore. On the other hand, the Backward Elimination method starts with the whole set of extracted features (obtained after removing correlated features) and subsequently removes one feature which results in the least decrease in performance in each iteration. Thus, the iteration of the Backward Elimination runs as long as there is no decrease in performance. Further, the Brute Force method tries all the possible combinations of features and selects the ones leading to the highest performance. Since the computation time associated with this method is quite expensive, the Brute Force is usually the last option to be used when only a small set of features is left after feature reduction.

2.5. Affect Classification

For the classification task, several machine learners are commonly used based on different decision algorithms. In our workflow, the following three classification algorithms are implemented including: Support Vector Machine (SVM), Random Forest (RF) and k-Nearest Neighbor (kNN). Among these classifiers, k-Nearest Neighbor is the simplest algorithm where the category of an unseen sample is determined by a majority vote of its k nearest neighbors. The performance and computation time of kNN depend on the size of the training sample. The larger the training sample is, the longer is the computation time and the better is the performance. The parameter k is set as an odd number (e.g., 3, 5, 7, 9). A too small value of k results in too much noise, while a large value of k leads to expensive computations. Random Forests are an ensemble learning, consisting of a great deal of decision trees. The following parameters are important for RF classifiers: the number of trees (e.g., 11) and the depth of each tree. The higher the number of trees is, the better is to learn the data but the longer is the training process. Further, the deeper the tree is, the more information about the data can be captured. Support Vector Machines are based on finding an optimal hyperplane which has the largest distance to the nearest training data points of any category. With the help of kernels, SVM algorithms are only suitable in linear problems, but also perform efficiently in non-linear classification tasks. For SVM with a RBF (radial basis function) kernel, the parameter C controls the generalization ability of the model, while the parameter gamma represents the degree of nonlinearity of the model. The higher the value of C is, the easier the over-fitting. While the higher the value of gamma is, the more nonlinear is the model.

2.6. Model Validation

In order to evaluate the performance of the feature selection and classification process, various validation methods are available for computing a performance matrix. In our present workflow, three validation methods are implemented including k-Cross Validation (CV), Leave-One-Subject-Out (LOSO) cross validation and Split Validation (SV). k-Cross Validation divides the whole dataset into k subsamples and utilizes one of these subsamples for testing and the remaining subsamples for training within one iteration. This is repeated k times and the performance is computed as the average of the k different classification rates. Leave-One-Subject-Out cross validation is similar to the k-Cross Validation, but divides the whole dataset according to the participants. That is, each subsample in the case of Leave-One-Subject-Out cross validation comprises the complete data of one subject. Finally, Split Validation is not based on the cross principle. Instead, it just splits the whole sample into training and test subsamples according to a percentage predefined by the user.

3. Results and Discussion

The described stages of the proposed workflow for affective computing and stress recognition, illustrated in Figure 1, were developed and integrated into automated biosignal processing and machine learning tools. The first steps of the workflow, including biosignal preprocessing and feature extraction, are handled in MATLAB, while the second part of the workflow, including feature selection, affect classification and model validation, is processed in RapidMiner.
For the first part, various Matlab-based functions were developed and implemented, allowing performance of the different steps consisting of raw data conversion and physiological triggering, data cutting and information selection, signal filtering and feature extraction. These steps are integrated into embedded functions, allowing an automated processing of this workflow part. The results are structured into separate RawData, TriggerData, CutData, FilteredData and ExtractedFeatures output files. As for the second part, various operators were designed in RapidMiner making it possible to automatically run the recognition process based on a set of extracted features obtained from the first part. The first layer of the process contains the main operators allowing data import, feature reduction, class selection and model training and testing. The last operator for model training and testing allows switching to the next layers consisting of various operators for feature selection, affect classification and model validation. Particularly, the second layer consists of operators defining the classification algorithms, while the third layer is placed within the classifiers and includes both feature selection and model validation operators. Moreover, various operators for optimizing the parameters of the different classification algorithms are implemented, including optimization of the C and gamma parameters in the SVM classifiers, the number of trees in the RF classifiers and the k value in the kNN classifiers.
The workflow was evaluated using biosignal data from our uulmMAC database for affective computing and machine learning applications [12]. The psychophysiological data were acquired in a human–computer interaction through an experimental setting using the g.MOBIlab+ wireless bluetooth bioamplifier system equipped with several physiological sensors. All biosignal channels were synchronously recorded at a fixed sampling rate of 256 Hz. We mainly used the classes overload and underload to evaluate the functional capability and general operability of the whole workflow. The next steps include further development of the algorithms and the integration of the developed tools into an easy-to-use system with a graphical interface [10], satisfying the needs and requirements of medical and psychological staff. In addition, a systematic evaluation of the different feature selection, classification algorithms and validation methods will be performed for various classes and stress-related recognition tasks.

4. Conclusions

This paper presents our workflow for affective computing and stress recognition from biosignal data obtained from physiological sensors. After a description of the biosignal channels included in the workflow, each step involved in the processing of the psychophysiological data is described, including signal preprocessing, feature extraction, feature selection, affect classification and model validation. Finally, the implementation and integration of the described steps are presented in the results. The present workflow is a valuable step towards automated affective computing and stress recognition for real-life applications in the medical and psychological fields.

Author Contributions

Conceptualization: D.H.-R.; methodology: D.H.-R. and L.Z.; validation: D.H.-R. and L.Z.; formal analysis: D.H.-R. and L.Z.; investigation: D.H.-R.; resources: D.H.-R. and H.C.T.; data curation: D.H.-R.; writing—original draft preparation: D.H.-R. and L.Z.; writing—review and editing: D.H.-R.; visualization: D.H.-R.; supervision: D.H.-R. and H.C.T.; project administration: D.H.-R.; funding acquisition: D.H.-R. and H.C.T. All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported by a Margarete von Wrangell (MvW) habilitation scholarship funded by the Ministry of Science, Research and Arts (MWK) of the state of Baden-Württemberg for Dilana Hazer-Rau.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Traue, H.C.; Ohl, F.; Limbrecht, K.; Scherer, S.; Kessler, H.; Schwenker, F.; Kozyba, M.; Brechmann, A.; Hoffmann, H.; Scheck, A.; et al. Framework for Emotions and Dispositions in Man-Companion Interaction. In Coverbal Synchrony in Human-Machine Interaction; Campbell, N., Rojc, M., Eds.; CRC Press: Boca Raton, FL, USA, 2013; pp. 99–140. [Google Scholar]
  2. Luneski, A.; Kostantinidis, E.; Bamidis, P. Affective Medicine: A review of Affective Computing efforts in Medical Informatics. Methods Inf. Med. 2010, 49, 207–218. [Google Scholar] [CrossRef] [PubMed]
  3. Greene, S.; Thapliyal, H.; Caban-Holt, A. A Survey of Affective Computing for Stress Detection: Evaluating. technologies in stress detection for better health. IEEE Consum. Electron. Mag. 2016, 5, 44–56. [Google Scholar] [CrossRef]
  4. Calvo, R.A.; D’Mello, S. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Trans. Affective Comput. 2010, 1, 18–37. [Google Scholar] [CrossRef]
  5. Pentland, A. Honest Signals: How They Shape Our World; MIT Press: London, UK, 2008. [Google Scholar]
  6. Zhang, L.; Traue, H.C.; Hazer-Rau, D. Individual Emotion Recognition and Subgroup Analysis from Psychophysiological Signals. Signal Image Process. Int. J. (SIPIJ) 2018, 9, 1–16. [Google Scholar] [CrossRef]
  7. Larradet, F.; Niewiadomsk, R.; Barresi, G.; Caldwell, D.G.; Mattos, L.S. Toward Emotion Recognition from Physiological Signals in the Wild: Approaching the Methodological Issues in Real-Life Data Collection. Front. Psychol. 2020, 11, 1111. [Google Scholar] [CrossRef] [PubMed]
  8. Hazer-Rau, D.; Zhang, L.; Traue, H.C. Performance Evaluation of Various Emotion Classification Approaches from Physiological Signals. Int. J. Artif. Intell. Appl. (IJAIA) 2018, 9, 31–41. [Google Scholar] [CrossRef]
  9. Shu, L.; Xie, J.; Yang, M.; Li, Z.; Li, Z.; Liao, D.; Xu, X.; Yang, X. A review of emotion recognition using physiological signals. Sensors 2018, 18, 2074. [Google Scholar] [CrossRef] [PubMed]
  10. Hazer-Rau, D. Towards an automated stress recognition system for cardiovascular risk assessment from biomedical signals. In Proceedings of the International Society for Research on Emotions Conference, Amsterdam, The Netherlands, 10–13 July 2019. [Google Scholar]
  11. Daucher, A.; Gruss, S.; Jerg-Bretzke, L.; Walter, S.; Hazer-Rau, D. Preliminary classification of cognitive load states in a human machine interaction scenario. In Proceedings of the International Conference on Companion Technology (ICCT’17), Ulm, Germany, 11–13 September 2017; pp. 1–5. [Google Scholar]
  12. Hazer-Rau, D.; Meudt, S.; Daucher, A.; Spohrs, J.; Hoffmann, H.; Schwenker, F.; Traue, H.C. The uulmMAC Database—A Multimodal Affective Corpus for Affective Computing in Human-Computer Interaction. Sensors 2020, 20, 2308. [Google Scholar] [CrossRef] [PubMed]
  13. Gruss, S.; Treister, R.; Werner, P.; Traue, H.C.; Crawcour, S.; Andrade, A.; Walter, S. Pain Intensity Recognition Rates via Biopotential Feature Patterns with Support Vector Machines. PLoS ONE 2015, 10, e0140330. [Google Scholar] [CrossRef] [PubMed]
  14. Schach, S.; Traue, H.C.; Hazer-Rau, D. The ecgFEAT Toolbox for Automated Cardiovascular Feature Extraction and Analysis. In Proceedings of the International Conference on Frontiers of Signal Processing (ICFSP’16), Warsaw, Poland, 15–17 October 2016; pp. 67–72. [Google Scholar]
Figure 1. The biosignal processing workflow for affective computing and stress recognition. It is based on psychophysiological biosignals as input. The workflow consists of the main steps: Biosignal Preprocessing, Feature Extraction, Feature Selection, Affect Classification and Model Validation. The output consists of recognition rates of different emotion and stress states.
Figure 1. The biosignal processing workflow for affective computing and stress recognition. It is based on psychophysiological biosignals as input. The workflow consists of the main steps: Biosignal Preprocessing, Feature Extraction, Feature Selection, Affect Classification and Model Validation. The output consists of recognition rates of different emotion and stress states.
Engproc 02 00085 g001
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Hazer-Rau, D.; Zhang, L.; Traue, H.C. A Workflow for Affective Computing and Stress Recognition from Biosignals. Eng. Proc. 2020, 2, 85. https://0-doi-org.brum.beds.ac.uk/10.3390/ecsa-7-08227

AMA Style

Hazer-Rau D, Zhang L, Traue HC. A Workflow for Affective Computing and Stress Recognition from Biosignals. Engineering Proceedings. 2020; 2(1):85. https://0-doi-org.brum.beds.ac.uk/10.3390/ecsa-7-08227

Chicago/Turabian Style

Hazer-Rau, Dilana, Lin Zhang, and Harald C. Traue. 2020. "A Workflow for Affective Computing and Stress Recognition from Biosignals" Engineering Proceedings 2, no. 1: 85. https://0-doi-org.brum.beds.ac.uk/10.3390/ecsa-7-08227

Article Metrics

Back to TopTop