Next Article in Journal
Consistent Time-to-Failure Tests and Analyses of Adhesive Anchor Systems
Next Article in Special Issue
EEG-Based Emotion Recognition Using Logistic Regression with Gaussian Kernel and Laplacian Prior and Investigation of Critical Frequency Bands
Previous Article in Journal
Special Issue “Plasma Technology for Biomedical Applications”
Previous Article in Special Issue
Application of Texture Descriptors to Facial Emotion Recognition in Infants
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Deep Learning for EEG-Based Preference Classification in Neuromarketing

1
Information System Department, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia
2
Information Technology Department, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia
3
Computer Science Department, College of Computer and Information Sciences, Imam Muhammad bin Saud University, Riyadh 11432, Saudi Arabia
*
Author to whom correspondence should be addressed.
Current address: Riyadh 11432, Saudi Arabia.
Submission received: 14 January 2020 / Revised: 15 February 2020 / Accepted: 19 February 2020 / Published: 24 February 2020
(This article belongs to the Special Issue Ubiquitous Technologies for Emotion Recognition)

Abstract

:

Featured Application

This article presents an application of deep learning in preference detection performed using EEG-based BCI.

Abstract

The traditional marketing methodologies (e.g., television commercials and newspaper advertisements) may be unsuccessful at selling products because they do not robustly stimulate the consumers to purchase a particular product. Such conventional marketing methods attempt to determine the attitude of the consumers toward a product, which may not represent the real behavior at the point of purchase. It is likely that the marketers misunderstand the consumer behavior because the predicted attitude does not always reflect the real purchasing behaviors of the consumers. This research study was aimed at bridging the gap between traditional market research, which relies on explicit consumer responses, and neuromarketing research, which reflects the implicit consumer responses. The EEG-based preference recognition in neuromarketing was extensively reviewed. Another gap in neuromarketing research is the lack of extensive data-mining approaches for the prediction and classification of the consumer preferences. Therefore, in this work, a deep-learning approach is adopted to detect the consumer preferences by using EEG signals from the DEAP dataset by considering the power spectral density and valence features. The results demonstrated that, although the proposed deep-learning exhibits a higher accuracy, recall, and precision compared with the k-nearest neighbor and support vector machine algorithms, random forest reaches similar results to deep learning on the same dataset.

1. Introduction

Neuromarketing is an emerging field that links the cognitive and affective sides of the consumer behavior by using neuroscience. Since its origin in 2002, this field has rapidly achieved credibility among the advertising and marketing specialists, and many such specialists are adopting neuromarketing strategies [1].
Neuromarketing can assist marketers in understanding how a consumer’s brain evaluates the diverse brands and recognizing the factors that affect the consumers’ choices when purchasing products. Neuromarketing research has demonstrated that people do not always recognize what happens in their unconscious brains. Furthermore, it has been demonstrated that people are not always explicit in their preferences or intentions [2].
The use of traditional marketing tools, such as interviews and questionnaires, to assess consumer preferences, needs, and buying intentions can lead to the generation of biased or incorrect conclusions [3,4]. Similarly, an oral expression of preferences can produce conscious or unconscious biases. It is difficult to extract the consumer preferences directly through choices, owing to the high product costs, ethical caution considerations, or the product not having been invented at the time of evaluation [3]. These elements highlight a contradiction in the users’ opinions during the usability assessments and their actual opinions, feelings, and senses regarding the use of a product [4].
Therefore, neuromarketing requires more effective methodological alternatives to evaluate the consumer behavior. Novel neuroimaging procedures provide an effective approach to study consumer behavior. Such methods ultimately help marketers examine the consumers’ brains to obtain valuable insights into the subconscious procedures underlying successful or failed marketing messages. This information is obtained by eliminating the primary problem in traditional advertising research, that is, trusting people; in particular, people should be trusted, whether they are consumers or workers who report on how the consumers are influenced by a specific part of an advertisement [1].
Brain–computer interfaces (BCIs) are promising neuroimaging tools in neuromarketing. This technology allows the users to communicate effectively with computer systems. A BCI does not require the use of any external devices or muscle interference to produce commands [5]. Furthermore, a BCI employs voluntarily generated user brain activity to control a system through signals, which provides the ability to communicate or interact with the nearby environment. Electroencephalography (EEG) is one of the main instruments used to examine brain activity. The EEG technique is the only practical, versatile, affordable, portable, non-invasive BCI to perform repetitive, real-time analysis of brain interactions in high temporal resolution [5,6,7].
Therefore, in the present research study, EEG was adopted as the input brain signal for a BCI system. Using classification algorithms, BCIs can be used as neural measures to distinguish the preference patterns from brain signals and translate them into actions to promote a product. In addition, the performance of a deep neural network (DNN) was implemented and examined to model a benchmark dataset for the preference classification.
The main objective of this research was to deeply investigate EEG-based preference recognition in neuromarketing to enhance the accuracy of classification prediction by comparing the performance of deep-learning with other conventional classification algorithms, such as support vector machine (SVM), random forest (RF), and k-nearest neighbors (KNN).

2. Background

This section provides a review of the main concepts used in this research: neuromarketing, BCI, and EEG.

2.1. Neuromarketing

Traditional marketing approaches include surveys, interviews, questionnaires, and focus groups, in which consumers openly and consciously report their experiences and opinions. However, such traditional approaches cannot evaluate the unconscious side of the consumer behavior. Neuroscience has the potential to discern the unconscious motivations that influence the act of making choices. It has been reported that approximately 90% of data are processed subconsciously in the human mind [8]. In the field of neuromarketing and consumer neuroscience, the evaluation of the subconscious activities exposes the true preferences of consumers more accurately than traditional marketing research does. Furthermore, neuromarketing can reveal information regarding the consumer preferences/ratings that cannot be accurately determined through the traditional methods. This is because subconscious opinions play a key role in consumer decision-making. Traditional market research approaches fail to assess the subconscious activities in the consumer brain, which leads to an inequality between the results of the traditional market research and the real behavior of the consumers at the points of purchase [8].
The term “neuromarketing” is derived by combining the prefix “neuro” and the term ’marketing”, indicating the integration of two study areas: neuroscience and marketing [1]. Neuroscience is a field that examines the facets of the brain at the biological level and from a psychological perspective [2]. In addition, neuroscience has significantly enlightened the field of marketing, and the interaction between these fields assists in intuiting the consumer behavior [8].
The term “neuromarketing” began to emerge organically around 2002. At that time, a few corporations, such as Brighthouse and SalesBrain, began offering neuromarketing studies and consultations, motivating the application of technology and knowledge from cognitive neuroscience to the field of marketing. Neuromarketing values the study of the consumer behavior from a psychological perspective [1]. Recently, several high-profile companies have begun exploiting neuromarketing approaches to assess the advertisements before introducing products to consumers [9]. This neuromarketing approach has gradually gained favor with brand executives in major corporations, such as Coca-Cola and Campbell’s [10].
Neuromarketing researchers aim to use neuroscientific procedures to exploit consumer behavior (i.e., requests, needs, and preferences) when shoppers purchase goods. This factor represents the researchers’ primary motivation for examining the consumers’ sensorimotor, mental, and effective feedback for products and advertisements through various modalities [9]. There are several neuromarketing modalities besides BCI, such as eye-tracking, galvanic skin response, skin conductance, facial coding, and facial electromyography. Each modality records different neural measure [11]. Eye tracking is used to determine eye locations and eye movement to grasp the consumer’s attention and natural responses to marketing stimuli. Galvanic skin response measures the moisture activity, which is related to the emotional state. Electromyography is used to evaluate the physiological features of facial muscles. Facial coding measures emotional states through facial expressions.

2.2. BCIs

BCIs are some of the most promising neuroimaging technologies in the neuromarketing domain. This technology helps facilitate effective communication between the users and computer systems. BCIs do not require any nerves, muscles, or movement interferences to issue a command [5] and employ the voluntarily-generated user brain activity to control a system through signals to communicate or interact with the nearby environment. Such environments can include wheelchairs, artificial arms/hands, and entertainment applications that involve skillful visualization, digital painting, and game playing [6].
BCI systems have contributed to numerous fields, including manufacturing, education, marketing, smart transportation, biomedical engineering, clinical neurology, and neuroscience [5,12]. A BCI system includes an input (i.e., the user’s mental activity), output (i.e., states or commands), a decoder component between the input and output, and a protocol that regulates the beginning, offset, and timing of the action [6]. The BCI research is expected to lead to an approach in which the brain signals are operated to aid people in interaction actions [6].
In a BCI, the brain signals require processing in non-clinical situations, which corresponds to a new challenge in computational neuroscience research. Currently, most of the application-oriented BCI research is focused on endowing users—not only disabled people—the ability to control systems or sensors with various environments [6].
Different neuroimaging techniques can be recorded with non-invasive BCI, such as EEG, fNIRS, fMRI, PET, MEG, SST, and TMS. EEG has better temporal resolution than fNIRS, which is a relatively new neuroimaging technique in neuromarketing research. However, recent fNIRS research is still in the substantial validation phase [13,14]. EEG is most commonly used in neuromarketing research due to its advantages that are detailed in the next subsection.

2.3. EEG

The EEG is a widely used tool that examines the brain activity. The electrical activity is recorded on the scalp by evaluating the voltage variations from neurons firing in the brain. These electrical activities are logged over a period of time using several electrodes positioned on the scalp directly above the cortex. The electrodes are connected in a hat-like device [5,7]. The EEG has the following key benefits: it is non-invasive, portable, cost effective, and relatively simple to use, and it has an exceptional temporal resolution (up to milliseconds). However, the signal-to-noise ratio and spatial resolution are restricted compared with those of other techniques. Nevertheless, EEG is considered to be the only practical, non-invasive BCI input to realize a repetitive, real-time brain interactive analysis [5,6,7]. Therefore, EEG was selected as the input brain signal for the BCI in this research.
The international 10-20 system is a method used to name electrodes based on their location on the scalp. The approach relates information pertaining to the inter-electrode space, specifically, 10–20% of the front-to-back or right-to-left of the scalp boundaries. In other words, the distance between the nearby electrodes is either 10% or 20% of the scalp diameter, as depicted in Figure 1. The 10-20 standard has been frequently used across diverse EEG systems to increase the dependability of the signals and decrease the signal-to-noise ratio [5,7].

EEG Signals

The brain produces abundant neural activity, which can be captured as EEG signals for the BCI. These neural activities consist of two types: (1) rhythms; and (2) transient activities. The EEG activity can be further categorized on the basis of these types of activities [6,7].
  • Rhythms:
    Rhythms, neural oscillations, or brainwaves are repetitive forms of neural activity. The rhythms are measures of collective synaptic, neuronal, and axonal activities of the neuronal sets. The EEG activity is characterized by separating the frequencies into bands, denoted as delta, theta, alpha, beta, gamma, and mu rhythms. Table 1 presents the details of the EEG rhythms, ranges of frequency, amplitude, and shape, as well as the brain regions in which these activities are the most common along with the events usually associated with the type of band [6,7].
    These frequency bands have been linked to affective reactions. The theta band in the front-center of the brain reflects the emotional processing when a consumer looks at a product. The alpha band on the prefrontal cortex differentiates between the positive and negative emotional valences. The beta band is correlated with the alterations during affective arousal. Finally, the gamma band is largely associated with the arousal effects [15].
  • Transient activities
    The transient activities or field potentials replicate the action potentials of certain neurons in a manner similar to spikes. These spikes can be recognized by their position, frequency, amplitude, shape, recurrence, and operational properties. The event-related potentials (ERPs) and event-related spectral perturbations (ERSPs) are common types of transient activities [3,16].
    ERP is the most common spike and arises as a reaction to a specific event or stimulus. These spikes have extremely small amplitudes. Consequently, the EEG samples must be averaged over many iterations to uncover the ERPs and eliminate noise fluctuations [16]. Table 2 presents the common ERPs used in the neuromarketing research. ERSPs compute the reaction to a stimulus over a period of time and are similar to the ERPs. However, the ERSPs split the EEG signals into the diverse frequency bands to test whether a variation exists in the power of a specified frequency band over time [3].

3. Literature Review

This section details EEG-based preference recognition, specifically, the neural correlation of the preference, predictive features of the preference, and preference classification algorithms.
Preference can be defined as a human attitude toward a collection of entities that can be mirrored in an explicit decision-making procedure. This aspect can also be an evaluative judgment in the sense of liking or disliking an object [19]. The possibility of measuring the conscious and unconscious brain activity in assessing advertisements, through processing the consumer’s processing of the advertisement message, cognitive workload, and emotional state, cannot be disregarded. The idea of a ’buy button’ in the brain may be overexaggerated; however, the research efforts to utilize the neural measures in monitoring the consumer thought processes are not trivial [20]. Understanding the neural process behind the preference, feelings, and decision-making can enhance the prediction of the user preferences and choices, and neuromarketing provides a precise objective determination of the implicit preferences of the consumers [21].
Several studies [10,22,23,24] have shown that the EEG can be used to determine the consumer preferences. To better utilize the EEG in consumer neuroscience research, the psychological processes underlying the consumer preferences must be understood.
In the following subsection, we describe the neural correlations of the EEG-based preference. Next, we classify the relevant studies into: (1) predictive features; and (2) classification algorithms of the preference recognition. Finally, we explain how the preferences can be detected using BCI.

3.1. Neural Correlations of the Preference

This subsection explains the neural elements correlated with the preferences. Certain areas of the brain are responsible for various cognitive and mental functions. To determine the positions of EEG electrodes, the underlying brain regions that are responsible for preference processing must be understood. Studies have demonstrated that the preference is linked to the frontal brain regions, specifically, the medial prefrontal cortex, nucleus accumbens [19,25], and medial orbitofrontal cortex [19,26].
Knutson et al. [25] linked the choice prediction of the products to the nucleus accumbens. When a consumer views the product, a higher activation of this region indicates a higher probability of the consumer purchasing that product. Furthermore, Kirk et al. [26] proved the relationship between the contextual preference and the medial orbitofrontal cortex; a higher activation in this region is related to higher level of preferences.
Recording the neural activity correlated with a certain function requires placing the electrodes directly above the corresponding brain area. Figure 2 shows the main electrode positions and the associated neural activity according to the 10-20 [27].
Although many researchers [24,25,26] have proved that the medial-frontal cortex is responsible for the preference function, no consensus exists on which electrodes should be used within the same brain area. Table 3 summarizes the electrode positions used in the preference recognition research.
The authors of [24] proved that the medial-frontal cortex is linked to the individual preference in the beta range (16–18 Hz) on the mid-frontal areas on electrodes AFz, F2, FC1, and FCz. Moreover, the authors proved that the population preference is linked to the frontocentral areas in the theta range (60–100 Hz) on electrodes F1, F2, F4, FC3, FC1, FCz, FC2, FC4, C5, C3, C1, C4, and CP5.
Agarwal et al. [27] linked preference attributes such as attention, emotions, and liking to electrodes F3, C3, P3, Pz, Fz, Cz, and C4.
Vecchiato et al. [28] found that asymmetrical increments of the theta and alpha bands are linked to watching pleasant (unpleasant) advertisements, as noted in the left (right) brain areas at electrodes Fp1 (Fp2), AF7 (AF8), F7 (F8), and F1 (F2). The spectral power for alpha bands increases noticeably for liked advertisements at electrode F1 and for the disliked advertisements at electrodes AF8 and AF4. In the theta band, increased activity occurs at electrodes F2, AF8, and F3 for the disliked advertisements and at Fp1 for the liked advertisements.
Touchette et al. [29] found that the frontal asymmetry in the alpha band is linked to the consumers’ unconscious reactions to the product attractiveness at electrodes F3 and F4. Vecchiato et al. [28] found that the asymmetrical frontal activity is statistically significantly positive in the alpha and theta bands between F1 and F2. In addition, this activity is significantly negative in the theta band between Fp2 and Fp1, AF8 and AF7, and F8 and F7.

EEG Indices

Based on our literature review, we identified four autonomic EEG indices that have been utilized for evaluating the reactions of the people in marketing stimuli: (1) the approach–withdrawal (AW) motivation index; (2) effort index; (3) choice index; and (4) valence. Such indices assist marketers in understanding customer responses to products [30,31].
  • AW Index
    The AW index is also known as the frontal alpha asymmetry, which indicates motivation, desire, or approach avoidance. The frontal asymmetry theory, which was initiated in 1985, states that the frontal regions of the left and right hemispheres are responsible for positive feelings (approach motivation) and negative feelings (withdrawal motivation) [29], respectively. This index can be defined as the difference between the two hemispheres in the prefrontal alpha band, that is, the relative engagement of the frontal left hemisphere compared with the right one. Positive AW values correspond to positive motivation (approach behaviors), expressed in terms of the higher activation of the left frontal cortex. In contrast, negative AW values correspond to negative motivation (avoidance behaviors), expressed in terms of the higher activation of the left frontal cortex [29,30,31,32].
    Numerous researchers have demonstrated the reliability and dependability of the frontal alpha asymmetry as an effective marker in the emotion and neuromarketing research [29,30,31,32,33,34]. Touchette [29] calculated the frontal alpha asymmetry scores by considering the difference between the right and left power spectral densities divided by their sum, as obtained using electrodes F4 and F3.
    A W = a l p h a ( F 4 ) a l p h a ( F 3 ) a l p h a ( F 4 ) + a l p h a ( F 3 )
  • Effort Index
    The effort index is defined as the frontal theta activity in the prefrontal cortex. A higher theta power in the frontal region has been linked to higher levels of task difficulty and complexity. This index acts as a sign of cognitive processing that results from mental fatigue [33], and it has been investigated extensively in neuromarketing research [3,24,28,33,35]. This factor demonstrates the importance of positive and negative emotional processing for the creation of the steady memory traces during advertising [30].
  • Choice Index
    The choice index is defined in terms of the frontal asymmetric gamma and beta oscillations, which are mostly linked to the real decision-making stage. It is also the most related element to willingness-to-pay responses, especially in the gamma band, for evaluating consumer preference and choice. Higher values of gamma and beta bands indicate a stronger activation of the left prefrontal region, and lower values are linked to relatively stronger activation of the right region [32]. Ramsoy et al. calculated the choice index for each band individually (gamma and beta) using electrodes AF3 and AF4 according to Equation (2):
    Choice index = l o g ( A F 3 ) l o g ( A F 4 ) l o g ( A F 3 ) + l o g ( A F 4 )
  • Valence
    Frontal asymmetry has been linked to preferences expressed as valence (i.e., the direction of a customer’s emotional state). Left and right frontal activation is related to positive and negative valence, respectively. Numerous studies have supported the hypothesis that the frontal EEG asymmetry is an indicator of valence [34].

3.2. Predictive Features for the Preferences

This section reports on the studies that focused on the predictive features of neuroscience methods that can aid marketers in forecasting consumer preferences, as described in Table 4. Most of these studies employed distinctions of the standard regression analyses toward their prediction models. We classified these predictive features based on the EEG signal types: (1) rhythms; and (2) transient activities.

3.2.1. Rhythms as Features

The beta and gamma oscillations from consumers who watched movie trailers were utilized to predict the box office sales and recall [24]. These factors were also used as an indicator of the willingness-to-pay to evaluate consumer preference and choice [32].
The alpha oscillations were used to compute the neural likeness and forecast recall and ticket sales. High-frequency EEG components were connected to both the individual preference (beta wave) and population preference (gamma wave) [24].
In addition, alpha frontal asymmetry was linked to the consumers’ unconscious reactions to the product attractiveness [29]. Similarly, Modica et al. [33] linked the higher alpha frequencies to comfort food as well as foreign food products. Moreover, awarded campaigns (i.e., the campaigns that received prizes) in anti-smoking public service announcements were linked to higher alpha frequencies [30].
Lower theta frequencies were associated with the negative results toward choosing products [3]. Moreover, these frequencies have been linked to effective anti-smoking public service announcements [30] and foreign products compared with local products [33].

3.2.2. Transient Activities as Features

In the cognitive processes associated with preferences, several research studies considered the ERP components N400, N200, and P300, each of which can be described as follows.
  • N400
    Some researchers [10] found that the N400 component can reflect familiarity in forecasting hits in brand extension. A powerful association with well-known brand names was replicated in the case of larger N400 amplitudes, foreseeing greater consumer preference. Another brand extension study [38] reported that the N400 component is associated with the unconscious conceptual categorization of products and brands, albeit not with conscious assessments.
  • N200
    Some other researchers [36] observed that the N200 amplitude exposed a relationship between the emotional state and brand extension categories. This relationship appeared only with negative emotions and moderate brand extensions. A second study [43] suggested that N200 could indicate the product preferences, as determined by spontaneous procedures, whereas the LPP and PSW could indicate the product preferences, as determined by the conscious cognitive procedures. In a third study, the Cerebro system [27] combined the N200 mean, N200 minima, and ERSP to rank products according to customer preferences. Similarly, in a fourth study [3], the researchers used two methodologies, ERSP and ERP, to forecast the product preferences by examining theta brainwaves, N200, and FNR.
  • P300
    The consumer preferences for the expanded brand labels were clarified using greater P300 amplitudes [17]. The authors of [40] used P300 as a measure of the consumer preferences for certain product features.
    Other researchers [18,39] have considered factors that influence the purchasing decisions. The authors of [18] investigated the roles of mathematical ability, gender, pricing, and discount promotions in the process of consumer purchasing using the active BCI. The authors correlated the ’buy’ decisions with ERP components, such as P200 and P300. To understand the product preferences, the authors also evaluated the relative importance (mutual information) of the diverse product (i.e., cracker) characteristics involved in the decision-making process by evaluating the cognitive processing by using the EEG alpha, beta, and theta brainwaves [39]. The researchers used eye tracking for choosing the preferred product. Michael et al. [45] used the same approach (EEG with eye tracking) to investigate the emotional reactions of tourism preferences by using different stimuli (words, images, and video). The authors observed that the images had higher affective responses than those of words in travel decision-making driven by the unconscious preference.
    The authors of [22] built a predictive model for consumer product choice from the EEG data. The researchers studied the roles of gender and age in the process of consumer preferences in terms of liking/disliking by using a passive BCI. Another research study [20] involved the use of an inductive research method to evaluate three successful and three unsuccessful advertisements by using a dense array EEG data. The results suggest that statistically significant ERP differences existed between the successful and unsuccessful advertisements.

3.3. Preference Classification Algorithms

Although considerable progress has been made in connecting the brain activities to the user choice, indications that neural assessment could genuinely be beneficial for forecasting the success of marketing activities remain limited [24]. The neural assessments can significantly increase the predictive power above and beyond that of the traditional assessments. Because the neural assessments are better predictors than self-reported assessments, the capability of neuroscience methods to forecast the preferences in real-world situations has incredible consequences for marketers. The first study to address this was published in 2007, and it was concluded that the pre-decisional activation in the related brain areas could be used to forecast the consequent choices [46]. Since then, many neuromarketing studies have published similar conclusions.
In recent years, it has become common practice to use multivariate methods, such as pattern classification, to predict choices. For example, a classification approach can be used to predict the out-of-sample choices from “non-choice’ neural responses to different products. The resulting models, which were founded on basic neuroscience methods, are more reliable for predicting the new states and settings compared to traditional market methods, such as focus groups and questionnaires. Moreover, these methods are more likely to be scalable, providing marketers with a deeper understanding of consumers and crucial economic outcomes [46].
Preference modeling using data-mining approaches can be classified into three general signal fields: time, frequency, and a combination of time and frequency. Time-based preference modeling exploits the discovery of the ERPs, as discussed in Section 3.2.2. Frequency-based modeling is accomplished by understanding the features gained by performing power spectrum analyses by generating delta, theta, alpha, beta, and gamma frequency bands, as explained in Section 3.2.1. In addition, different frequency-based feature extraction methods can be used; for instance, common spatial patterns (CSPs) and spectral filters were used in the preference classification for music with an SVM, and an accuracy of 74.77% and 68.22%, respectively, was obtained [47]. Fast Fourier transform (FFT) as the feature extraction method was used, and the SVM obtained an accuracy of 82.14% [48]. In another study, the researchers used the FFT with the radial SVMs for the preference classification and obtained an accuracy of 75.44% [49]. The last preference model combines time and frequency by analyzing the power spectrum at the time intervals that cover the entire duration of the post stimuli interval to assess the brain signals. Several traditional data-mining algorithms have been applied to classify preferences, and the utilization of different time–frequency analysis (TF) approaches has been considered [15,23,50] to detect the user preferences for music. The use of KNN led to an accuracy of 86.5% and 83.34% with different TF approaches, namely the Hilbert–Huang Spectrum (HHS) and spectrogram, respectively [15]. In their extended study, Hadjidimitriou and Hadjileontiadis [51], using familiar music data, managed to obtain a considerably higher accuracy of 91.0%. Another work involved the performance of the music preference classification by using the TF approaches, namely, the discrete Fourier transform with a KNN, and an accuracy rate of 97.99% was achieved. The researchers could achieve a similar accuracy result when using the quadratic discriminant analysis (QDA) at 97.39% [52].
Most researchers applied variations of standard regression analysis to their prediction models. However, numerous techniques and methods have been developed to process EEG to determine the preference state of consumers by using classification algorithms. A review of some experimental neuromarketing articles and comparisons of computational approaches is presented in Table 5.
Some preference studies involved the use of more than two classification algorithms to discover well-matched classifiers for a definite feature set [12]. Chew et al. [54] measured the user preferences for the aesthetics presented as virtual 3D shapes by using EEG. The researchers used the frequency bands as features to classify EEG into two classes—liked and disliked—by using the KNN and SVM and achieved high classification accuracies of 80% and 75%, respectively. However, these results cannot be considered reliable because the authors used an extremely small dataset of five subjects. In their extended study [55,56], the authors increased the number of subjects to 16 but better results were not obtained. Hakim et al. [44] achieved an accuracy of 68.5% by using the SVM to predict the most and least favored products by combining EEG measures with questionnaire measures.
Classifier combinations such as boosting, voting, or stacking can be used to join numerous classifiers, by merging their outputs and/or training them to complement each other and improve their performance [57]. The selection of the classification algorithms in a BCI system is mostly based on both the form of the acquired mental signals and the context in which the application is expected to be used. However, LDA and SVM are the most commonly applied classification algorithms and have been used in more than half of the EEG-based BCI articles.
Another categorization of the classifications was based on the survey research [57], which considered the BCI and machine learning literature from 2007 to 2017. The findings of the recently designed classification algorithms were divided into four main categories: adaptive classifiers, matrix and tensor classifiers, transfer learning, and deep learning. The adaptive classifiers are classifiers whose parameters, such as the feature weights, are gradually re-assessed and revised over time as new EEG data are presented. The matrix and tensor classifiers (multi-way array) avoid the use of the filters and feature selections and map the data directly onto a certain space with appropriate measures. The transfer learning approach aims to improve the performance of a learned classifier trained on a domain based on the information acquired, while learning another domain or task.
In recent times, deep learning has been employed in EEG-based preference recognition. DNNs are gatherings of the artificial neurons organized in layers to estimate the nonlinear resolution border. The most popular type of DNN used for BCIs is the multi-layer perceptron (MLP), which normally consists of only one or two hidden layers. Other DNN types have been explored less frequently, such as the Gaussian classifier neural networks or learning vector quantization neural networks [57]. Furthermore, Teo et al. [55,56] proposed deep learning approaches for preference recognition by using 3D rotating objects. The results prove that the use of the deep network could obtain a higher accuracy compared to that of the other machine learning classifiers, such as the SVM, RF, and KNN algorithms. In their extended research, Teo et al. [23] improved the result accuracies by using a deep network plus dropout architecture, with rectified linear units and tanh for activation at 79.76%.
Table 6 presents some neuromarketing studies that used different classification algorithms to obtain the most accurate results in predicting the consumer preferences. The review highlights the need to use more features and hybrid classifiers to improve the accuracy results of the predictions [22,44].

3.4. Preference Detection Using a BCI

This section explains the design process of the neuromarketing experiment to predict the consumer preferences and choices. First, a BCI device must be placed on the head of a consumer. Next, the consumer is asked to look at the products. During the recording phase, the EEG data are recorded concurrently while the consumer views a product. After viewing each product, the user is asked for his or her preference toward the product in terms of a five- or nine-point scale of subjective rank. When all products are displayed, the subjective ranks must be manually labeled as liked or disliked classes. Next, the EEG signals undergo a signal preprocessing and feature extraction. The classification module is developed based on the ground truth completed by the consumer’s selection (subjective ranks).
Figure 3 presents a proposed BCI system for the preference detection composed of three main modules: signal preprocessing, feature extraction and selection, and classification.

4. Proposed System for the EEG-based Preference Detection

The performance of EEG recognition systems is based on the selection of a feature extraction technique and a classification algorithm. In our study, we investigated the possibility of detecting two preference states, namely pleasant and unpleasant, by using EEG and classification algorithms. To this end, we performed rigorous offline analysis to investigate the computational intelligence for the preference detection and classification. We used deep learning classification from the DEAP dataset to explore how to employ intelligent computational methods in the form of classification algorithms. This could effectively mirror the preference states of the subjects. Furthermore, we compared our classification performance with those of the KNN and RF classifiers. We built our model in the open source programming language Python and used the Scikit-Learn toolbox for machine learning, along with SciPy for EEG filtering and preprocessing, MNE for EEG-specific signal processing, and the Keras library for deep learning.
In this section, we discuss our methodology along with some implementation details of the proposed system for EEG-based preference detection. We begin with describing the benchmark dataset and ground truth of the preference labeling. Next, we explain the feature extraction. Finally, we illustrate the DNN classification model.

4.1. Dataset Description

DEAP [58] is a benchmark EEG database developed for affective analysis. The DEAP database was built at the Queen Mary University in London, and it has been used in several research studies for preference detection [59,60]. Table 7 summarizes some characteristics of the DEAP dataset.

4.2. Preference Modeling and Ground Truth

To set the true preferences (ground truth table), we used the DEAP self-assessment reports to identify the preference states using a nine-point Likert scale for valence dimension. In this study, we considered the valence dimension as a preference indicator to align with the target preference state: pleasant and unpleasant. Moreover, we considered EEG trials that had at least two different valence levels—low and high. The valence levels are classified as follows: (1) low valence if the valence rating is between 1 and 5; and (2) high valence if the valence rating is between 6 and 9. The presence of a low or high valence is an indicator of an unpleasant or pleasant preference state, respectively.

4.3. Data Pre-Processing

We used the preprocessed EEG dataset from the DEAP database, where the sampling rate of the original recorded data of 512 Hz was down-sampled to a sampling rate of 128 Hz, with a bandpass frequency filter that ranged from 4.0 to 45.0 Hz, and the EOG artifacts were eliminated from the signals using a blind source separation method, namely independent component analysis ICA. The data were averaged and segmented to 60-s trials. Then, we applied a channel selection step with a dimensionality reduction technique. The aim of this step was to reduce the number of features and/or channels used by selecting a subset that excludes very high-dimensional and noisy data. Ideally, the features that are meaningful or useful in the classification stage are identified and selected, while others, including outliers and artifacts, are omitted. Moreover, it reduces the computational cost of the subsequent steps. Therefore, we keep only the channels in which we are interested (Fz, AF4, AF3, F4, and F3).

4.4. Feature Extraction

Feature extraction plays a crucial role in building EEG-based BCI applications. Thus, we extracted the EEG frequency bands by using a power spectral density (PSD) method called the Welch method. Subsequently, we used the resulting frequency bands to calculate the valence as a preference indicator. Figure 4 presents the block diagram of feature extraction.

4.4.1. PSD

The PSD is one of the most popular feature extraction methods based on the frequency domain analysis in the neuromarketing research. Research studies [11,37,39] have demonstrated that the PSD obtained from the EEG signals works well for determining consumer preferences. The PSD method converts the data in the time domain to the frequency domain and vice versa. This conversation is based on the FFT, which calculates the discrete Fourier transform and its inverse.
We used the PSD technique in this study to divide each EEG signal into four different frequency bands: theta (4–8 Hz), alpha (8–13 Hz), beta (13–30 Hz), and gamma (30–40 Hz). The Python signal processing toolbox (MNE) was used for PSD calculation, and the average power over the frequency bands was computed to build a feature using the avgpower function in the MNE toolbox.

4.4.2. Valence

The valence was selected as the measure of preference in this study. Strong valence is reflected in the activation of frontal EEG asymmetry [34]. In DEAP dataset [58], there was high correlation between valence and EEG frequency bands, as shown in Figure 5. The increment in valence led to power increment in alpha, which is consistent with the results in a similar study [34]. We did not use liking rating in the DEAP dataset because the data owners [58] reported conflicting findings between the activation in left alpha power and liking.
We applied the different valence equations and investigated the relationship with the DEAP self-assessment valence measurement. For the valence calculation, we used the extracted alpha and beta band powers from the DEAP data and considered only the following electrodes: Fz, AF3, F3, AF4, and F4. Finally, we computed the values of the valence by using four different equation (Equations (3)–(6)), which have been well-explained in a previous paper [34] authored by an author of this paper.
Valence = b e t a ( A F 3 , F 3 ) a l p h a ( A F 3 , F 3 ) b e t a ( A F 4 , F 4 ) a l p h a ( A F 4 , F 4 )
Valence = l n [ a l p h a ( F z , A F 3 , F 3 ) ] l n [ a l p h a ( F z , A F 4 , F 4 ) ]
Valence = a l p h a ( F 4 ) b e t a ( F 3 )
Valence = a l p h a ( F 4 ) b e t a ( F 4 ) a l p h a ( F 3 ) beta ( F 3 )

4.5. DNN Classification

Deep learning has been proved as an effective tool to help make the EEG signals meaningful because of its ability to learn the feature representations from the raw data. DNNs are models consisting of the combined layers of “neurons” in which each layer applies a linear transformation to the input data. Then, the transformation result of each layer undergoes processing on the basis of a nonlinear cost function. The parameters of such transformations are deduced by minimizing a cost function [61]. The DNN operates in one forward direction, from the input neurons through the hidden ones (if available) to the output neurons in the forward directions. Assuming that the length window of the samples is s, the input of the DNN for the EEG signals consists of a multidimensional array X i R e × s that contains s samples associated with a window for all e electrodes. The fully connected layer, which is the most common type of layer used in building a DNN, consists of fully connected neurons. The input of every neuron is the activation of each neuron from the previous layer [61].
Our study aimed to detect two preference states in the EEG data. Therefore, we employed intelligent classification algorithms that could effectively mirror the preferences of the subjects. We proposed a DNN classifier and compared its performance with those of KNN and RF classifiers.
The block diagram of the proposed DNN classifier is shown in Figure 6. First, the extracted features are normalized using minimum–maximum normalization (Equation (7)) and then fed into the DNN classifier.
x scaled = ( x min ) / max min
In this study, the considered DNN architecture is a fully connected feed-forward neural network with three hidden layers, which contain units involving rectified linear activation functions (ReLu). The output is obtained as a soft-max layer with a binary cross-entropy cost function. The input layer consists of 2367 units, and each hidden layer consists of 75% units from its predecessor (previous) layer. In particular, the first, second, and third hidden layers involve 1800, 1300, and 800 units, respectively. The output layer dimensions pertain to the number of target preferences state (2) units. To train the DNN classifier, we used Adam gradient descent with three objective loss functions: binary cross-entropy, categorical cross-entropy, and hinge cross function. For transfer learning, we considered the reasonable defaults and followed the established best practices: the start learning rate was 0.001. Then, we linearly reduced the rate with each epoch such that the learning rate for the last epoch was 0.0001. We set the dropout for the input and hidden layers as 0.1 and 0.05, respectively. The stopping criterion of the network training was determined according to the model performance on a testing set. If the network started to over-fit, the network training was stopped. This stopping criterion is helpful for reducing the possibility of over-fitting of the validation data. The network was tested on a test set, which contained approximately 20% of the data samples in the dataset.

5. Results and Discussion

We predicted the preference states (pleasant or unpleasant) using different classification algorithms: DNN, RF, KNN, and SVM. We used different evaluation measurements: accuracy, recall, and precision. The accuracy was calculated as the average of the binary measurements in which the score of every class was weighted by its availability in the real data. Precision is the proportion of pleasant preference predictions that were actually correct. Recall is the proportion of actual pleasant preferences that were successfully predicted. To evaluate the performance of classification algorithms, we used different cross validation methods: holdout (train/test splitting), k-folds cross validation, and leave-one-out cross validation (LOOCV). Table 8 presents the accuracy results of DNN, RF, KNN, and SVM for each cross validation method. In LOOCV, RF reached the best accuracy results at 90% while DNN reached similar results to RF at 93% in the holdout validation method. In k-fold validation method, KNN achieved the best accuracy results at 90% and 91% when k was set to 10 and 20, respectively. Because the best accuracy results were achieved using the holdout validation, this method was chosen as the base validation for comparison and the tuning the DNN hyper parameter (loss function).
The proposed DNN model was compared with three conventional classification algorithms for EEG signals: SVM, RF, and KNN. Table 9 presents the accuracy, recall, and precision results of RF, KNN, and DNN using three different loss functions in the DNN: the categorical cross-entropy function, binary cross-entropy function, and hinge function. The KNN classifier led to a better accuracy of 88% when K was set to 1. Although the RF achieved a high accuracy of 92%, the DNN reached the highest accuracy result of 94% with hinge cross-entropy function compared to the other conventional classification algorithms. To ensure that the DNN does not have over-fitting problem, we presented the loss per epoch for each cross-entropy function. The average loss per epoch DNN with the categorical, binary, and hinge function reached a value of 0.28, 0.24, and 0.23, respectively, as shown in Figure 7.

6. Conclusions

This study proposed a DNN model to detect the preferences from the EEG signals by using the pre-processed DEAP dataset. Two types of features were extracted from the EEG: the PSD and valence. This aspect resulted in a group of 2367 unique features illustrating the EEG activity in each trial. We used different evaluations measures (accuracy, recall, and precision) and various validation methods (holdout, LOOCV, and k-fold cross validation) to test classifiers’ performance. We built four different classifiers, namely the DNN, RF, SVM, and KNN classifiers, which achieved an accuracy of 94%, 92%, 62%, and 88%, respectively. The results demonstrate that, although the proposed DNN exhibits a higher accuracy, recall, and precision compared with the KNN and SVM, RF reaches similar results to DNN on the same dataset. Future research directions will involve exploring the DNNs in the context of transfer learning for preference detection.

Author Contributions

M.A. conceived, designed, and performed the experiment; analyzed and interpreted the data; and drafted the manuscript. A.A.-N. co-supervised the analysis, reviewed the manuscript, and contributed to the discussion. M.Y. supervised this study. All authors have read and approved the submitted version of the manuscript.

Acknowledgments

The authors would like to thank the deanship of scientific research for funding and supporting this research through the initiative of DSR Graduate Students Research Support (GSR) at King Saud University.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:
BCIBrain Computer Interface
EOGElectrooculography
EEGElectroencephalography
ERSPEvent Related Spectral Perturbations
PSDPower Spectral Density
ICAIndependent Component Analysis
ERPEvent Related Potential
DNNDeep Neural Network
SVMSupport Vector Machine
AWApproach-Withdrawal
KNNk-Nearest Neighbor
RFRandom Forest
LOOCVLeave-One-Out Cross Validation
PCAPrincipal Component Analysis
ReLuRectified Linear Unit
LPPLate Positive Component
FRNFeedback Related Negativity
MLPMulti-Layer Perceptron
GSRGalvanic Skin Response
HMMHidden Markov Model
LDALinear Discriminant Analysis
EMGFacial electromyography
TFtime–frequency analysis
CSPCommon Spatial Pattern
FFTFast Fourier Transform
HHSHilbert–Huang Spectrum

References

  1. Morin, C. Neuromarketing: The New Science of Consumer Behavior. Society 2011, 48, 131–135. [Google Scholar] [CrossRef] [Green Version]
  2. Ait Hammou, K.; Galib, M.H.; Melloul, J. The Contributions of Neuromarketing in Marketing Research. J. Manag. Res. 2013, 5, 20. [Google Scholar] [CrossRef] [Green Version]
  3. Telpaz, A.; Webb, R.; Levy, D.J. Using EEG to Predict Consumers’ Future Choices. J. Mark. Res. 2015, 52, 511–529. [Google Scholar] [CrossRef] [Green Version]
  4. Barros, R.Q.; Tavares, A.S.; Albuquerque, W.; da Silva, J.C.; de Lemos, I.A.; de Albuquerque Cardoso, R.L.S.; Soares, M.M.; Cairrao, M.R. Analysis of Product Use by Means of Eye Tracking and EEG: A Study of Neuroergonomics; Lecture Notes in Computer Science; Springer International Publishing: Cham, Switzerland, 2016; Volume 9747, pp. 539–548. [Google Scholar] [CrossRef]
  5. Abdulkader, S.N.; Atia, A.; Mostafa, M.S.M. Brain computer interfacing: Applications and challenges. Egypt. Inform. J. 2015, 16, 213–230. [Google Scholar] [CrossRef] [Green Version]
  6. Ramadan, R.A.; Refat, S.; Elshahed, M.A.; Ali, R.A. Brain-Computer Interfaces. In Intelligent Systems Reference Library; Springer International Publishing: Cham, Switzerland, 2015; Volume 74, pp. 31–50. [Google Scholar] [CrossRef] [Green Version]
  7. Ramadan, R.A.; Vasilakos, A.V. Brain Computer Interface: Control Signals Review. Neurocomputing 2016, 223, 1–19. [Google Scholar] [CrossRef]
  8. Agarwal, S.; Dutta, T. Neuromarketing and consumer neuroscience: current understanding and the way forward. Decision 2015, 42, 457–462. [Google Scholar] [CrossRef]
  9. Murugappan, M.; Murugappan, S.; Balaganapathy; Gerard, C. Wireless EEG signals based Neuromarketing system using Fast Fourier Transform (FFT). In Proceedings of the 2014 IEEE 10th International Colloquium on Signal Processing and its Applications, Kuala Lumpur, Malaysia, 7–9 March 2014; pp. 25–30. [Google Scholar] [CrossRef]
  10. Lin, M.H.J.; Cross, S.N.N.; Jones, W.J.; Childers, T.L. Applying EEG in consumer neuroscience. Eur. J. Mark. 2018, 52, 66–91. [Google Scholar] [CrossRef]
  11. Ohme, R.; Reykowska, D.; Wiener, D.; Choromanska, A. Analysis of Neurophysiological Reactions to Advertising Stimuli by Means of EEG and Galvanic Skin Response Measures. J. Neurosci. Psychol. Econ. 2009, 2, 21–31. [Google Scholar] [CrossRef] [Green Version]
  12. Hwang, H.J.; Kim, S.; Choi, S.; Im, C.H. EEG-Based Brain-Computer Interfaces: A Thorough Literature Survey. Int. J. Hum.-Comput. Interact. 2013, 29, 814–826. [Google Scholar] [CrossRef]
  13. Krampe, C.; Gier, N.R.; Kenning, P. The Application of Mobile fNIRS in Marketing Research—Detecting the “First-Choice-Brand” Effect. Front. Hum. Neurosci. 2018, 12. [Google Scholar] [CrossRef]
  14. Meyerding, S.G.; Mehlhose, C.M. Can neuromarketing add value to the traditional marketing research? An exemplary experiment with functional near-infrared spectroscopy (fNIRS). J. Bus. Res. 2020, 107, 172–185. [Google Scholar] [CrossRef]
  15. Hadjidimitriou, S.K.; Hadjileontiadis, L.J. Toward an EEG-based recognition of music liking using time-frequency analysis. IEEE Trans. Biomed. Eng. 2012, 59, 3498–3510. [Google Scholar] [CrossRef] [PubMed]
  16. Nezamfar, H. FlashLife™, a Context-Aware Code-VEP Based Brain Computer Interface for Daily Life Using EEG Signals. Ph.D. Thesis, Northeastern University, Boston, MA, USA, 2016. [Google Scholar]
  17. Ma, Q.; Wang, X.; Shu, L.; Dai, S. P300 and categorization in brand extension. Neurosci. Lett. 2008, 431, 57–61. [Google Scholar] [CrossRef] [PubMed]
  18. Jones, W.J.; Childers, T.L.; Jiang, Y. The shopping brain: Math anxiety modulates brain responses to buying decisions. Biol. Psychol. 2012, 89, 201–213. [Google Scholar] [CrossRef] [PubMed]
  19. Ramsøy, T.Z.; Friis-Olivarius, M.; Jacobsen, C.; Jensen, S.B.; Skov, M. Effects of perceptual uncertainty on arousal and preference across different visual domains. J. Neurosci. Psychol. Econ. 2012, 5, 212–226. [Google Scholar] [CrossRef] [Green Version]
  20. Daugherty, T.; Hoffman, E.; Kennedy, K. Research in reverse: Ad testing using an inductive consumer neuroscience approach. J. Bus. Res. 2016, 69, 3168–3176. [Google Scholar] [CrossRef]
  21. Alvino, L.; Constantinides, E.; Franco, M. Towards a Better Understanding of Consumer Behavior: Marginal Utility as a Parameter in Neuromarketing Research. Int. J. Mark. Stud. 2018, 10, 90. [Google Scholar] [CrossRef] [Green Version]
  22. Yadava, M.; Kumar, P.; Saini, R.; Roy, P.P.; Dogra, D.P. Analysis of EEG signals and its application to neuromarketing. Multimed. Tools Appl. 2017, 76, 19087–19111. [Google Scholar] [CrossRef]
  23. Teo, J.; Chew, L.H.; Chia, J.T.; Mountstephens, J. Classification of Affective States via EEG and Deep Learning. Int. J. Adv.. Comput. Sci. Appl. 2018, 9, 132–142. [Google Scholar] [CrossRef] [Green Version]
  24. Boksem, M.A.S.; Smidts, A. Brain Responses to Movie Trailers Predict Individual Preferences for Movies and Their Population-Wide Commercial Success. J. Mark. Res. 2015, 52, 482–492. [Google Scholar] [CrossRef] [Green Version]
  25. Knutson, B.; Rick, S.; Wimmer, G.E.; Prelec, D.; Loewenstein, G. Neural Predictors of Purchases. Neuron 2007, 53, 147–156. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  26. Kirk, U.; Skov, M.; Hulme, O.; Christensen, M.S.; Zeki, S. Modulation of aesthetic value by semantic context: An fMRI study. NeuroImage 2009, 44, 1125–1132. [Google Scholar] [CrossRef] [PubMed]
  27. Agarwal, M.; Sivakumar, R. Cerebro: A Wearable Solution to Detect and Track User Preferences using Brainwaves. In Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, Seoul, Korea, 12 June 2019; pp. 47–52. [Google Scholar] [CrossRef]
  28. Vecchiato, G.; Toppi, J.; Astolfi, L.; Fallani, F.D.V.; Cincotti, F.; Mattia, D.; Bez, F.; Babiloni, F. Spectral EEG frontal asymmetries correlate with the experienced pleasantness of TV commercial advertisements. Med. Biol. Eng. Comput. 2011, 49, 579–583. [Google Scholar] [CrossRef] [PubMed]
  29. Touchette, B.; Lee, S.E. Measuring Neural Responses to Apparel Product Attractiveness: An Application of Frontal Asymmetry Theory. Cloth. Text. Res. J. 2017, 35, 3–15. [Google Scholar] [CrossRef]
  30. Cartocci, G.; Caratu, M.; Modica, E.; Maglione, A.G.; Rossi, D.; Cherubino, P.; Babiloni, F. Electroencephalographic, Heart Rate, and Galvanic Skin Response Assessment for an Advertising Perception Study: Application to Antismoking Public Service Announcements. Jove-J. Vis. Exp. 2017. [Google Scholar] [CrossRef]
  31. Cherubino, P. Application of Neuro- Marketing techniques to the wine tasting experience. In Proceedings of the 11th Annual Conference of the EuroMed Academy of Business, Valletta, Malta, 12–14 September 2018; pp. 290–298. [Google Scholar]
  32. Ramsøy, T.Z.; Skov, M.; Christensen, M.K.; Stahlhut, C. Frontal brain asymmetry and willingness to pay. Front. Neurosci. 2018, 12. [Google Scholar] [CrossRef]
  33. Modica, E.; Cartocci, G.; Rossi, D.; Martinez Levy, A.C.; Cherubino, P.; Maglione, A.G.; Di Flumeri, G.; Mancini, M.; Montanari, M.; Perrotta, D.; et al. Neurophysiological responses to different product experiences. Comput. Intell. Neurosci. 2018, 2018. [Google Scholar] [CrossRef]
  34. Al-Nafjan, A.; Hosny, M.; Al-Wabil, A.; Al-Ohali, Y. Classification of Human Emotions from Electroencephalogram (EEG) Signal using Deep Neural Network. Int. J. Adv. Comput. Sci. Appl. 2017, 8, 419–425. [Google Scholar] [CrossRef]
  35. Vecchiato, G.; Astolfi, L.; Fallani, F.D.V.; Cincotti, F.; Mattia, D.; Salinari, S.; Soranzo, R.; Babiloni, F. Changes in brain activity during the observation of TV commercials by using EEG, GSR and HR measurements. Brain Topogr. 2010, 23, 165–179. [Google Scholar] [CrossRef]
  36. Ma, Q.; Wang, K.; Wang, X.; Wang, C.; Wang, L. The influence of negative emotion on brand extension as reflected by the change of N2: A preliminary study. Neurosci. Lett. 2010, 485, 237–240. [Google Scholar] [CrossRef]
  37. Ohme, R.; Reykowska, D.; Wiener, D.; Choromanska, A. Application of frontal EEG asymmetry to advertising research. J. Econ. Psychol. 2010, 31, 785–793. [Google Scholar] [CrossRef]
  38. Wang, X.; Ma, Q.; Wang, C. N400 as an index of uncontrolled categorization processing in brand extension. Neurosci. Lett. 2012, 525, 76–81. [Google Scholar] [CrossRef] [PubMed]
  39. Khushaba, R.N.; Wise, C.; Kodagoda, S.; Louviere, J.; Kahn, B.E.; Townsend, C. Consumer neuroscience: Assessing the brain response to marketing stimuli using electroencephalogram (EEG) and eye tracking. Expert Syst. Appl. 2013, 40, 3803–3812. [Google Scholar] [CrossRef]
  40. Wang, J.; Han, W. The impact of perceived quality on online buying decisions: An event-related potentials perspective. Neuroreport 2014, 25, 1091–1098. [Google Scholar] [CrossRef]
  41. Vecchiato, G.; Maglione, A.G.; Cherubino, P.; Wasikowska, B.; Wawrzyniak, A.; Latuszynska, A.; Latuszynska, M.; Nermend, K.; Graziani, I.; Leucci, M.R.; et al. Neurophysiological Tools to Investigate Consumer’s Gender Differences during the Observation of TV Commercials. Comput. Math. Methods Med. 2014. [Google Scholar] [CrossRef]
  42. Bosshard, S.S.; Bourke, J.D.; Kunaharan, S.; Koller, M.; Walla, P. Established liked versus disliked brands: Brain activity, implicit associations and explicit responses. Cogent Psychol. 2016, 3. [Google Scholar] [CrossRef]
  43. Goto, N.; Mushtaq, F.; Shee, D.; Lim, X.L.; Mortazavi, M.; Watabe, M.; Schaefer, A. Neural signals of selective attention are modulated by subjective preferences and buying decisions in a virtual shopping task. Biol. Psychol. 2017, 128, 11–20. [Google Scholar] [CrossRef] [Green Version]
  44. Hakim, A.; Klorfeld, S.; Sela, T.; Friedman, D.; Shabat-Simon, M.; Levy, D.J. Pathways to Consumers Minds: Using Machine Learning and Multiple EEG Metrics to Increase Preference Prediction Above and Beyond Traditional Measurements. bioRxiv 2018. [Google Scholar] [CrossRef] [Green Version]
  45. Michael, I.; Ramsoy, T.; Stephens, M.; Kotsi, F. A study of unconscious emotional and cognitive responses to tourism images using a neuroscience method. J. Islamic Mark. 2019, 10, 543–564. [Google Scholar] [CrossRef]
  46. Plassmann, H.; Venkatraman, V.; Huettel, S.; Yoon, C. Consumer Neuroscience: Applications, Challenges, and Possible Solutions. J. Mark. Res. 2015, 52, 427–435. [Google Scholar] [CrossRef]
  47. Pan, Y.; Guan, C.; Yu, J.; Ang, K.K.; Chan, T.E. Common frequency pattern for music preference identification using frontal EEG. Int. IEEE/EMBS Conf. Neural Eng. NER 2013, 505–508. [Google Scholar] [CrossRef]
  48. Tseng, K.C.; Lin, B.S.; Han, C.M.; Wang, P.S. Emotion recognition of EEG underlying favourite music by support vector machine. In Proceedings of the ICOT 2013—1st International Conference on Orange Technologies, Tainan, Taiwan, 12–16 March 2013; pp. 155–158. [Google Scholar] [CrossRef]
  49. Kim, Y.; Kang, K.; Lee, H.; Bae, C. Preference Measurement Using User Response Electroencephalogram; Springer: Berlin/Heidelberg, Germany, 2015; pp. 1315–1324. [Google Scholar] [CrossRef]
  50. Cohrdes, C.; Wrzus, C.; Frisch, S.; Riediger, M. Tune yourself in: Valence and arousal preferences in music-listening choices from adolescence to old age. Dev. Psychol. 2017, 53, 1777–1794. [Google Scholar] [CrossRef] [PubMed]
  51. Hadjidimitriou, S.K.; Hadjileontiadis, L.J. EEG-Based classification of music appraisal responses using time-frequency analysis and familiarity ratings. IEEE Trans. Affect. Comput. 2013, 4, 161–172. [Google Scholar] [CrossRef]
  52. Moon, J.; Kim, Y.; Lee, H.; Bae, C.; Yoon, W.C. Extraction of user preference for video stimuli using eeg-based user responses. ETRI J. 2013, 35, 1105–1114. [Google Scholar] [CrossRef]
  53. Bercik, J.; Horska, E.; Wang, R.W.Y.; Chen, Y.C. The impact of parameters of store illumination on food shopper response. Appetite 2016, 106, 101–109. [Google Scholar] [CrossRef] [PubMed]
  54. Chew, L.H.; Teo, J.; Mountstephens, J. Aesthetic preference recognition of 3D shapes using EEG. Cognit. Neurodyn. 2016, 10, 165–173. [Google Scholar] [CrossRef] [Green Version]
  55. Teo, J.; Hou, C.L.; Mountstephens, J. Deep learning for EEG-Based preference classification. AIP Conf. Proc. 2017, 1891. [Google Scholar] [CrossRef] [Green Version]
  56. Teo, J.; Hou, C.L.; Mountstephens, J. Preference classification using Electroencephalography (EEG) and deep learning. J. Telecommun. Electron. Comput. Eng. 2018, 10, 87–91. [Google Scholar]
  57. Lotte, F.; Bougrain, L.; Cichocki, A.; Clerc, M.; Congedo, M.; Rakotomamonjy, A.; Yger, F. A review of classification algorithms for EEG-based brain-computer interfaces: A 10 year update. J. Neural Eng. 2018, 15, aab2f2. [Google Scholar] [CrossRef] [Green Version]
  58. Mühl, C.; Lee, J.s. DEAP: A Database for Emotion Analysis Using Physiological Signals. IEEE Trans. Affect. Comput. 2011, 2011. 3, 18–31. [Google Scholar] [CrossRef] [Green Version]
  59. Shin, S.; Jang, S.J.; Lee, D.; Park, U.; Kim, J.H. Brainwave-based mood classification using regularized common spatial pattern filter. KSII Trans. Internet Inf. Syst. 2016, 10, 807–824. [Google Scholar] [CrossRef]
  60. Lan, Z.; Liu, Y.; Sourina, O.; Wang, L. Real-time EEG-based user’s valence monitoring. In Proceedings of the 2015 10th International Conference on Information, Communications and Signal Processing (ICICS), Singapore, 2–4 December 2015; pp. 1–5. [Google Scholar] [CrossRef]
  61. Roy, Y.; Banville, H.; Albuquerque, I.; Gramfort, A.; Falk, T.H.; Faubert, J. Deep learning-based electroencephalography analysis: A systematic review. J. Neural Eng. 2019, 16, 051001. [Google Scholar] [CrossRef] [PubMed]
Figure 1. The international 10-20 system.
Figure 1. The international 10-20 system.
Applsci 10 01525 g001
Figure 2. EEG electrodes and related neural functions.
Figure 2. EEG electrodes and related neural functions.
Applsci 10 01525 g002
Figure 3. EEG-based consumer preference prediction system.
Figure 3. EEG-based consumer preference prediction system.
Applsci 10 01525 g003
Figure 4. Feature extraction block diagram.
Figure 4. Feature extraction block diagram.
Applsci 10 01525 g004
Figure 5. The average correlations for all subjects of the valence ratings with the power of different frequency bands. The highlighted electrodes correlate significantly (p < 0.05) with the valence ratings. © [2011] IEEE. Reprinted, with permission, from: IEEE Trans. Affect. Comput., DEAP: A Database for Emotion Analysis Using Physiological Signals, Mühl, C.; Lee, J.s. [58].
Figure 5. The average correlations for all subjects of the valence ratings with the power of different frequency bands. The highlighted electrodes correlate significantly (p < 0.05) with the valence ratings. © [2011] IEEE. Reprinted, with permission, from: IEEE Trans. Affect. Comput., DEAP: A Database for Emotion Analysis Using Physiological Signals, Mühl, C.; Lee, J.s. [58].
Applsci 10 01525 g005
Figure 6. Block diagram of the DNN classifier.
Figure 6. Block diagram of the DNN classifier.
Applsci 10 01525 g006
Figure 7. Loss per epoch on the training and validation sets in DNN using different cross-entropy functions: (a) categorical cross-entropy (average loss rate = 0.28); (b) Binary cross-entropy (average loss rate = 0.24); and (c) hinge cross-entropy (average loss rate = 0.23).
Figure 7. Loss per epoch on the training and validation sets in DNN using different cross-entropy functions: (a) categorical cross-entropy (average loss rate = 0.28); (b) Binary cross-entropy (average loss rate = 0.24); and (c) hinge cross-entropy (average loss rate = 0.23).
Applsci 10 01525 g007
Table 1. Categorization of the EEG rhythms based on their frequency.
Table 1. Categorization of the EEG rhythms based on their frequency.
RhythmsFrequency (Hz)Amplitude ( μ V)Brain RegionState of Mind
δ (Delta)< 450 – 100Central regionDeep sleep
θ (Theta)4 – 8<100Central regionDrowsiness, first
step of sleep
μ (Mu)Approximately 10< 50Somatosensory cortexMovement
α (Alpha)8 – 13< 10Posterior brainOpening the
and focusing attention
β (Beta)13 – 22< 20Central regionHigh state of
wakefulness, alert
and focused
γ (Gamma)22 – 30< 2Highly localizedHigher mental activity,
including perception
and consciousness
Table 2. Common ERP components used in neuromarketing studies.
Table 2. Common ERP components used in neuromarketing studies.
ERPDescription
P300P300 is a positive potential that arises approximately 300 ms after the onset of a stimulus as a result of internal decision-making [16]. It shows the activity of attention in working memory as well as the adjustment to responses [17].
N200N200 is a negative potential related to unfamiliarity, and reaches its peak between 200 and 350 ms after the onset of a stimulus [16].
LPCThe late positive component is a positive potential that arises approximately 400–800 ms after the onset of a stimulus associated with explicit recognition memory [18].
LPPThe late positive potential indicates enabled attention to emotional stimuli of either a positive or negative valence. This can be translated as neutral, pleasant, or unpleasant contextual stimuli.
N400N400 is a negative potential related to oddness experiments such as brand names with products [10]
FN400FN400 is located in the front-center of the brain related to familiarity and arises approximately 300–500 ms after the onset of a stimulus. FN400 has a more negative potential for new words than for similar and familiar words [18]
FRNFeedback related negativity is a front-central negative potential related to a subject’s choices. It arises in response to passively observed products 200–300 ms after the presentation of unfavorable versus favorable products [3].
PSWPositive slow waves are correlated with sustained attention to visual emotional stimuli and can be detected long after the appearance of an emotional stimulus [10].
Table 3. Common rhythms/ERP and electrode positions used for the preference detection in neuromarketing.
Table 3. Common rhythms/ERP and electrode positions used for the preference detection in neuromarketing.
ReferenceRhythmsElectrode Channel
[24]Beta range (16 – 18 Hz)AFz, F2, FC1, and FCz.
[24]Theta range (60 – 100 Hz)F1, F2, F4, FC3, FC1, FCz, FC2, FC4, C5, C3, C1, C4, and CP5.
[28]ThetaFp1/Fp2, AF7/AF8, F7/F8, and F1/F2.
[28]AlphaFp1/Fp2, AF7/AF8, F7/F8, and F1/F2.
[29]AlphaFp3, and F4.
[27]N200F3, C3, P3, Pz, Fz, Cz, and C4.
Table 4. EEG-based neuromarketing studies.
Table 4. EEG-based neuromarketing studies.
YearReferenceStimuli and Marketing ElementSignal TypeEEG Waves or ERP ComponentRecording Modality
2008[17]Viewing brands and productsERPP300BCI
2009[11]Watching TV adsEEGAlpha (for emotional state)EMG, BCI and GSR
2010[36]Viewing brands and imagesERPN200 and P300BCI
2010[37]Watching TV adsEEGAlpha band frontal asymmetryBCI
2010[35]Watching TV adsEEGTheta and gammaHeart rate, BCI and GSR
2011[28]Watching TV adsEEGAsymmetrical increase in theta and alpha in PSDBCI
2012[38]Viewing brand namesERPN400BCI and EOG for eye movement
2012[18]Viewing products and pricesERPFN400, LPC, and P200BCI
2013[39]Viewing productsEEGAlpha, beta, theta, gamma, and deltaBCI, Eye tracking
2014[40]Viewing productsERPP300BCI
2014[41]Watching TV adsEEGTheta and alphaHeart rate, BCI and GSR
2015[3]Viewing productsERSP and ERPTheta, N200, and FRNBCI
2015[24]Watching ads (movie trailers)EEG (64 electrodes)Beta and gamma oscillationsBCI and EOG for eye movement (2 electrode)
2016[20]Watching TV adsdense-array EEGThree epochs: 200-350, 350-500, and 500-800BCI
2016[42]Viewing brand namesERPLPPBCI
2017[43]Viewing product imagesERPN200, LPP, and PSWBCI
2017[30]Watching ad videosEEGTheta and alphaHeart rate, BCI and GSR
2017[22]Viewing product imagesEEGDelta, theta, alpha, beta, and gammaBCI
2017[29]Viewing product imagesEEGAlphaBCI
2018[44]Viewing ads of food productsEEGDelta, theta, alpha, beta, and gammaBCI
2018[32]Viewing products and pricesEEGThetaBCI
2018[31]Tasting drinksEEGAlphaBCI
2018[33]Viewing and touching productsEEGAlpha and thetaBCI
2019[27]Viewing product imagesERSP, ERPTheta, beta and N200BCI
2019[45]Viewing tourism images, videos and wordsEEGDelta, theta, alpha, beta and gammaBCI and GSR
Table 5. Computational approaches for assessing the customer preferences.
Table 5. Computational approaches for assessing the customer preferences.
MethodNumber
of Studies
References
Low-pass filter3[22,24,38]
PreprocessingBand-pass filters10[3,11,15,20,28,39,40,41,44,53]
Down sampling3[11,15,44]
Average7[3,15,27,37,38,40,54]
Statistical features: median,
standard deviation, t-test,
z-score transformation
9[3,11,24,28,35,37,38,40,41]
Time–frequency analysis2[15,54]
Feature Extraction SelectionWavelet domain: entropy,
energy >> using wavelet transform
2[22,39]
Frequency domain: power
spectral density, band power,
spectrum power >>
using Fourier transform
8[11,27,37,39,44,53,54,55]
Correlation analysis: mutual
information, correlation coefficient
3[18,27,39]
Univariate linear regression analysis2[3,24]
SVM4[15,22,54,55]
KNN3[15,54,55]
Regression/ClassificationDNN3[22,23,55]
RF1[22,55]
HMM1[22]
Linear regression (Lasso, Ridge)1[27]
Table 6. Classification algorithms applied for recognizing the consumer preferences.
Table 6. Classification algorithms applied for recognizing the consumer preferences.
ReferenceClassification AlgorithmFeatureClassBest Accuracy
ChannelRhythms
[54]SVMF3 and F4(Alpha and theta),
and (alpha, beta and theta)
1.Liked 2.Disliked75%
KNNFz, F3, F4Alpha, theta and delta 80%
[49]SVMAF3, F7, F3, FC5, T7,
P7, O1, O2, P8, T8,
FC6, F4, F8, and AF4
Alpha, beta, theta, and gamma1.Preferred image
2. Unnoticed image
75.44%
F8 and T8Alpha 83.64%
[15,51]KNN + TFAF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4Delta, theta, alpha, beta and gamma1.Liked 2.Disliked83.34%
Beta and gamma 86.5%
91.02%
[47]SVM + Spectral filterFP1 and FP2Delta, theta, alpha, beta and gamma1.Liked 2.Disliked68.22%
SVM + CSP 74.77%
[52]QDA + Fourier transformAF3, AF4, F7, F8, F3, F4, FC5, FC6, T7, T8, P7, P8, O1, and O2Theta, alpha, beta and gamma1.Most preferred
2.Preferred
3.Less preferred
4.Least preferred
97.39%
KNN +Fourier transform Theta, alpha, beta, gamma
and their corresponding
symmetric differences
97.99%
DNN63.99%
[23,55,56] SVMPOz, Fz, Cz, C3, C4, F3, F4, P3, and P4 Delta, theta, alpha, beta and gamma 1.Liked
2.Disliked
60.19%
DNN with dropout and ReLu 79.76%
DNN with dropout and tanh 74.38%
Logistic Regression67.32%
[44] SVMF7, Fp1, Fpz, Fp2, F8, Fz, Cz, and Pz Delta, theta, alpha, beta and gamma 1.Most favored
2.Least favored
68.50%
KNN 59.98%
Decision trees with Adaboost M1 63.34%
DNN (2 layers + Sigmoid)60.10%
[22] SVMAF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4 Delta, theta, alpha, beta and gamma 1.Liked
2.Disliked
62.85%
RF 68.41%
HMM 70.33%
Table 7. DEAP dataset description.
Table 7. DEAP dataset description.
Affective ModelDimensional emotion model (Valence-Arousal-Dominance)
StimuliVisual- and audio-based stimuli (1-min music video)
Participants32 participants aged 19 to 37 years
Trials1280 trials (40 trials for each subject)
EEG device32 EEG channels of the Biosemi Active Two system. The EEG data stream was collected using 32 Ag/AgCl electrodes, which were arranged in accordance with the 10-20 international system.
Experimental ProtocolEach participant watched and rated his or her emotional responses to 40 music videos on scales of arousal, valence, and dominance using self-assessment manikins (SAM). Participants also reported their liking of and familiarity with the videos.
DEAP DatabaseDifferent datasets that are publicly available in the DEAP database include recorded signal data, frontal face videos for a subset of participants, stimuli-volunteers’ self-reported data, and subjects’ self-assessments.
Table 8. Accuracy results of preference detection with DNN, RF, KNN, and SVM using different cross validation methods.
Table 8. Accuracy results of preference detection with DNN, RF, KNN, and SVM using different cross validation methods.
Cross Validation MethodLOOCVHoldoutk-fold
(K = 5)(K = 10)(K = 20)
DNN (Binary Cross-entropy)73%93%60%68%71%
SVM47%62%49%48%47%
RF90%93%86%87%89%
KNN92%88%88%90%91%
Table 9. Results of preference recognition using holdout validation and different classifiers: DNN, SVM, RF, and KNN.
Table 9. Results of preference recognition using holdout validation and different classifiers: DNN, SVM, RF, and KNN.
ClassifierDNNSVMRFKNN
Metrics Hinge CrossCategorical CrossBinary crossK = 5K = 3K = 1
Accuracy94%91%93%62%92%73%79%88%
Recall94%91%93%62%92%73%79%88%
Precision94%92%94%64%93%75%81%90%

Share and Cite

MDPI and ACS Style

Aldayel, M.; Ykhlef, M.; Al-Nafjan, A. Deep Learning for EEG-Based Preference Classification in Neuromarketing. Appl. Sci. 2020, 10, 1525. https://0-doi-org.brum.beds.ac.uk/10.3390/app10041525

AMA Style

Aldayel M, Ykhlef M, Al-Nafjan A. Deep Learning for EEG-Based Preference Classification in Neuromarketing. Applied Sciences. 2020; 10(4):1525. https://0-doi-org.brum.beds.ac.uk/10.3390/app10041525

Chicago/Turabian Style

Aldayel, Mashael, Mourad Ykhlef, and Abeer Al-Nafjan. 2020. "Deep Learning for EEG-Based Preference Classification in Neuromarketing" Applied Sciences 10, no. 4: 1525. https://0-doi-org.brum.beds.ac.uk/10.3390/app10041525

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop