A New Sentiment-Enhanced Word Embedding Method for Sentiment Analysis

Abstract

Since some sentiment words have similar syntactic and semantic features in the corpus, existing pre-trained word embeddings always perform poorly in sentiment analysis tasks. This paper proposes a new sentiment-enhanced word embedding (S-EWE) method to improve the effectiveness of sentence-level sentiment classification. This sentiment enhancement method takes full advantage of the mapping relationship between word embeddings and their corresponding sentiment orientations. This method first converts words to word embeddings and assigns sentiment mapping vectors to all word embeddings. Then, word embeddings and their corresponding sentiment mapping vectors are fused to S-EWEs. After reducing the dimensions of S-EWEs through a fully connected layer, the predicted sentiment orientations are obtained. The S-EWE method adopts the cross-entropy function to calculate the loss between predicted and true sentiment orientations, and backpropagates the loss to train the sentiment mapping vectors. Experiments show that the accuracy and macro-F1 values of six sentiment classification models using Word2Vec and GloVe with the S-EWEs are on average 1.07% and 1.58% higher than those without the S-EWEs on the SemEval-2013 dataset, and on average 1.23% and 1.26% higher than those without the S-EWEs on the SST-2 dataset. In all baseline models with S-EWEs, the convergence time of the attention-based bidirectional CNN-RNN deep model (ABCDM) with S-EWEs was significantly decreased by 51.21% of ABCDM on the SemEval-2013 dataset. The convergence time of CNN-LSTM with S-EWEs was vastly reduced by 41.34% of CNN-LSTM on the SST-2 dataset. In addition, the S-EWE method is not valid for contextualized word embedding models. The main reasons are that the S-EWE method only enhances the embedding layer of the models and has no effect on the models themselves.

1. Introduction

Word embedding maps a word into a vector space, playing an important role in natural language processing (NLP). Word vectors contain rich information, such as local contextual information [1], global co-occurrence information [2], and global contextual information [3]. Moreover, word embedding can apply to many NLP tasks, such as sentiment analysis [4,5,6,7,8], part-of-speech tagging [9], and named entity recognition [10]. It can also apply to many interdisciplinary tasks, such as influence maximization [11] and emotion role identification [12].

Word embedding learned by deep learning carries contextual semantic and syntactic information, which is beneficial for NLP tasks. Sitaula et al. [13] evaluated many machine learning and deep learning methods on sentiment analysis tasks. However, for feature-based word embedding, such as Word2Vec [1], if two words with opposite sentiment polarity have similar context in a corpus, the performance of word embedding may not be so good [14,15]. This is because feature-based word embeddings only assign a unique word embedding to each word and cannot generate word embeddings based on the context in downstream tasks. For example, the words “good” and “terrible” from two different sentences, i.e., “The weather is good today” and “The weather is terrible today,” have similar contexts in a corpus and have similar word embeddings. In this situation, sentiment classification models cannot determine which is a positive or negative sentence. Yu et al. [15] concluded that about 30% of the top 10 semantically similar words have opposite sentiment orientations. In most tasks, sentiments could divide into two (positive, negative) or three dimensions (positive, neutral, and negative). Ekman [16] even divided sentiments into six categories (anger, disgust, fear, happiness, sadness, and surprise). Based on the six categories, Xu [17] further divided sentiments into seven categories (anger, disgust, fear, sadness, happiness, good, and surprise). The latest research of Google [18] found that sentiments could divide into 28 categories. However, the sentiment dimensions are also much fewer than the dimensions of word embedding. The dimensions of Word2Vec [1] and global vectors for word representation (GloVe) [2] are usually 300; the dimensions of the bidirectional encoder representations from the transformers base (BERT-base) [3] number 768; and the dimensions of BERT-large [3] number 1024. Thus, the dimensions of sentiment and word embedding are different.

Sentiment enhancement is a way of injecting sentiments into word embedding, making word embedding contain sentiment information. However, sentiment enhancement concerning sentiment and word embedding faces two challenges (see Figure 1).

1.

Due to the different dimensions of sentiment word embeddings and sentiment orientations, they are hard to fuse into one vector.

2.

Since sentiment and word embeddings belong to two different vector spaces, sentiment classification models could not directly operate on them.

To solve the above problems, we expect to find a method to build the mapping relationship between words and their sentiment orientations in sentiment lexicons and fuse sentiment information into these words. Fortunately, we are inspired by translations in the embedding space (TransE) [19] and contrastive learning [20,21]. TransE is a knowledge graph embedding method. It shortens the distance between two related entities and increases the distance between two unrelated entities in vector space. Meanwhile, contrastive learning narrows the distance between positive examples and increases the distance between negative examples in vector space. The bootstrap your own latent (BYOL) [20] minimizes the distance between two similar images; the supervised contrastive pre-training (SCAPT) [21] used supervised contrastive learning to cluster explicit and implicit positive sentiments, cluster explicit and implicit negative sentiments, and separated these two clusters. By borrowing these ideas, we propose a novel sentiment-enhanced word embedding (S-EWE) method to improve the performances of sentiment classification models. Specifically, we convert words in the sentiment lexicon to word embeddings and arrange each word embedding for a sentiment mapping vector. Then, we add original word embeddings and their sentiment mapping vectors to obtain the sentiment-enhanced word embeddings with the information of word and sentiment. Thirdly, we adopt one fully connected layer to reduce the dimensions of sentiment-enhanced word embedding to get the predicted sentiment orientations. Finally, we calculate the loss of the predicted and true sentiment orientations by the cross-entropy function. We further train the sentiment mapping vectors through backpropagating the loss so that the trained sentiment mapping vectors can find the mapping relationship between words in sentiment lexicons and their sentiment orientations. Under different sentiment classification models, we confirmed the effectiveness of the S-EWE method. The main contributions are as follows:

1.

A new sentiment-enhanced word embedding method is proposed. This method establishes the mapping relationship between words and their sentiment orientations by vector addition.

2.

Abundant experiments were performed on TextCNN [22], TextRNN [23], TextRCNN [24], TextBiRCNN [24], an attention-based bidirectional CNN-RNN deep model (ABCDM) [25], and CNN-LSTM [26] using Word2Vector [1]; and GloVe [2] with or without sentiment-enhanced word embeddings on the semantic evaluation 2013 (SemEval-2013) dataset [27] and the Stanford sentiment treebank (SST-2) dataset [28]. Experimental results show that sentiment-enhanced word embeddings can improve the ability of these models’ sentiment classification.

The subsequent parts of this paper are organized as follows. Section 2 introduces the related work. The sentiment-enhanced word embedding method is shown in Section 3. Datasets, empirical results, and model analysis are presented in Section 4. Finally, we summarize our paper and put forward a future outlook in Section 5.

2. Related Work

2.1. Word Embedding

Word embedding is a knowledge enhancement technique because it can capture syntactic and semantic information in textual corpora. Word2Vec [1] has two models: a continuous skip-gram model (Skip-gram) and a continuous bag-of-words model (CBOW). They all use sliding windows to capture contextual information between words. Word2Vec achieved the best results on the Semantic-Syntactic Word Relationship dataset and the Microsoft Sentence Completion Challenge dataset. However, Word2Vec could not get global information of words. GloVe [2] used a co-occurrence probability matrix to obtain the global information of words. It also used slide windows to capture contextual information between words. Experimental results showed that GloVe achieved state-of-the-art performance on word analogies, similarity, and named entity recognition tasks. Since word embeddings trained by Word2Vec and GloVe could only assign a vector to each word, they could not solve the ambiguity problem of words. In addition, Word2Vec and GloVe could only capture the contextual information of words in a window due to the limited computing power. With the increased computing power, the embedding from language models (ELMo) [29] used two reversed recurrent neural network (RNN) layers to encode one input sequence. For a word i, the forward RNN captured the above information of the word i, and the backward RNN captured the following information of the word i. By concatenating the forward hidden state and backward hidden state of the word i, the contextual information of the word i was obtained. The ELMo model achieved remarkable performance on CoNLL 2003 NER and CoNLL 2000 chunking tasks.

Due to the improvements in computing power, BERT [3] used bidirectional transformers [30] to encode information of a whole sentence. Since the self-attention mechanism [30] can capture the information of an entire sequence, the contextual information between words is no longer limited to the sliding window. BERT broke all records for the general language understanding evaluation (GLUE), with an average improvement of 6.7% over the other models. Moreover, BERT achieved significant performance on SQuAD, CoNLL-2003, and SWAG datasets. Liu et al. [31] averaged Chinese and English word embeddings in multilingual BERT (m-BERT) [3], respectively. They obtained the Chinese–English difference vector by subtracting the average English vector from the average Chinese vector. The experimental results showed that a Chinese word embedding minus the difference vector could get English word embedding and vice versa.

Even though word embedding has shown its power in many natural language processing tasks, it cannot distinguish the sentiment orientation of semantically similar words well. Singh et al. [32] extracted hashtags, mentions, and keywords from tweets. They performed centrality-aware random walks on them to get their word/node embeddings. Then, they put the embeddings into a deep learning model for sentiment classification. The performance of their method is better than those of the original baseline models. Sent2Vec [33] used the smoothed inverse frequency (SIF) [34] or the unsupervised smoothed inverse frequency (uSIF) [35] to encode a sentence into a vector. Then, this vector is input into a multi-layer perceptron (MLP) to learn sentimental embedding. These methods could achieve good results based on the syntactic and semantic knowledge inside a dataset in tasks of different sentiment classifications. However, these methods could not utilize external knowledge about the dataset.

2.2. Knowledge Enhancement

Knowledge enhancement is a way of injecting external knowledge into models, improving the performances of the models on downstream tasks. Liang and Yi [36] proposed the two-stage three-way enhanced technique for inclusive policy text classification. They first used the ensemble convolution neural network to extract text representation in the first stage. Then, they used a new determination method to reduce the decision risk. If the classification accuracy at the first state was not within the confidence interval, they used a traditional machine learning technique to reclassify texts. They conducted some experiments on Chengdu and Xiamen datasets. Their experimental results showed that their method outperforms baseline models. Bai et al. [37] proposed the multi-view document clustering with enhanced semantic embedding (MDCE) to solve the problems, including the sparseness, the high dimensionality, and the inconsistency of document clustering. This model used two deep neural networks to obtain document representation and neighbors representation. Then, the model adopted one fusion layer to fuse different representations and used a clustering layer to output the cluster results. The proposed MDCE model achieved the best performance on Aminer, Aminer (700), 3-sources, Reuters, and Multi-source news datasets. ERNIE [38] incorporated knowledge graphs (KGs) into BERT [3]. It used two encoders [30] to encode tokens and KGs obtained by TransE [19], respectively. It also adopted a fusion layer to fuse these two representations into the same vector space. This method achieved excellent performance on FIGER, Open Entity, FewRel, and TACRED datasets. It also obtained comparable results with BERT in GLUE. However, it took much time in the process of entity alignment. To shorten the running time of entity alignment, the knowledge embedding and pre-trained language representation (KEPLER) [39] used the robustly optimized BERT approach (RoBERTa) [40] to represent definitions of considerable knowledge in knowledge graphs. This method combined knowledge graph and text representations, achieving better performance than ERNIE on TACRED, FewRel, and OpenEntity datasets. Saxena et al. [41] proposed a temporal question answering system. They first extracted knowledge from temporal knowledge graphs. Then, they used the proposed temporal KGE model to obtain temporal knowledge representation. They further used BERT [3] to get text representation. Finally, they fused temporal knowledge representation and text representation to answer questions. Their model achieved dramatic performance on the CRONQUESTIONS dataset. Even though injecting external knowledge into models could improve the performances of different models [38,42,43], all the above models have not adopted sentiment information.

2.3. Sentiment Enhancement

In recent times, some scholars have used the sentiments as external knowledge. Shi et al. [44] proposed a sentiment-enhanced neural graph recommender. This model applied an attention mechanism [30] to extract a representation of a user’s review and an item review representation. They further extracted another user representation and item representation based on a graph convolutional network (GCN) [45]. The model obtained the final user and item representations by concatenating the two user representations and item representations. Then, the model adopted a fully connected network to extract an auxiliary sentiment representation from the output of the self-attention network. Finally, they trained their sentiment-enhanced neural graph recommender with a loss between sentiment auxiliary representation and user-item representation. Their experiments achieved the best performance on Toys, Kindle, Yelp2017, and Yelp2018 datasets. However, this model only adopted sentiment as an additional feature to implement a recommendation system. It did not enhance word embedding with sentiments. Gavilanes et al. [46,47] proposed an unsupervised system with sentiment propagation across dependencies (USSPAD) model [48]. Their method found the relationship between each emoji’s description and sentiment orientation. The classification results of their model are close to those of manual annotations. Yu et al. [15] used k-nearest neighbors (KNN) to zoom in on positive samples and pushed out negative samples in a sentiment lexicon. Their refinement method outperformed original baseline models on SemEval-2013 and SST-2 datasets. Wei et al. [49] used sentiment lexicon as external sentiment knowledge and injected this knowledge into the bidirectional long short-term memory (BiLSTM) to analyze implicit sentiment. They achieved the best results on SMP2019-ECISA, COAE (2015), and SemEval (2013–2017) datasets. Wang et al. [6] injected sentiment lexicon and labeled sentiment corpora as the pre-training corpora for two bidirectional gate recurrent unit (BiGRU) layers. They pre-trained their model with three objectives: predicting target words, predicting word sentiment, and predicting sentence sentiment. After pre-training, they concatenated word embeddings and the hidden states of two BiGRUs, and obtained the sentiment-enhanced word embedding. The experimental results show that their model achieved comparable results to contextualized word embedding models (such as BERT) on three datasets with only 1–7 M parameters. However, these models considered sentiment an additional feature or combined other features to analyze sentiment, but fewer scholars injected sentiment information into word embeddings.

In the field of injecting sentiment into word embedding, Sent2Vec [33] used SIF [34] or uSIF [35] to encode a sentence into a vector. The vector is further input into a fully connected network to learn sentimental embedding. This method achieved better results than other word embeddings on Dbpedia and Yahoo datasets. Naderalvojoud and Sezer [14] proposed Approaches 1 and 2 for enhancing sentiment. Specifically, Approach 1 adopted an MLP to find the mapping relationships between words and their sentiment intensity; and Approach 2 trained Word2Vec [1] and GloVe [2] with an additional sentiment vector if a target word is a sentiment word. The two methods achieved better results than Word2Vec and GloVe on SemEval-2013 and SST datasets. Therefore, few studies are focused on injecting sentiment information into word vectors.

3. The Proposed Method and Its Applications on Downstream Tasks

This section illustrates a novel sentiment-enhanced word embedding method and its applications on downstream tasks. Figure 2 shows their workflows. For example, in the pre-training stage, given the word “happy” and its sentiment orientation “1” in a sentiment lexicon, we assign “happy” a randomly initialized sentiment mapping vector ${\mathbf{\nabla }}_{\mathrm{happy}}$. Then, we add the word embedding of “happy” to the sentiment mapping vector and obtain a sentiment-enhanced word embedding. We feed it to a fully connected layer, predicting the probability of the “happy” sentiment label. By making a loss between the predicted sentiment label and the gold standard (i.e., 1), we can train ${\mathbf{\nabla }}_{\mathrm{happy}}$ by backpropagation. The ${\mathbf{\nabla }}_{\mathrm{happy}}$ will be the unique sentiment mapping vector for “happy.” In the application stage, we add the well-trained ${\mathbf{\nabla }}_{\mathrm{happy}}$ and the original word embedding of “happy”; we obtain the sentiment-enhanced embedding of “happy” and apply the sentiment-enhanced embedding into the embedding layer of the downstream tasks.

3.1. The Sentiment-Enhanced Word Embedding Method

The S-EWE method aims to establish the mapping relationship between words in sentiment lexicons and their sentiment orientations to obtain trained sentiment-enhanced word embeddings for downstream tasks. When we convert one word to a word embedding, the dimensions of the word embedding are much larger than those of the sentiment orientation, and they belong to two different vector spaces. Therefore, we assign a sentiment mapping vector to the word embedding. This sentiment mapping vector can help word embedding find the mapping relationship between it and its sentiment orientation.

Let the $\mathcal{V}$ be a set of all words in a sentiment lexicon, and $\mathcal{Y}$ be a set of sentiment orientations of all words in the sentiment lexicon. Given a word $w_i\in \mathcal{V}$, its word embedding ${\mathit{w}}_i$, and its sentiment orientation $y_i\in \mathcal{Y}$, we assign a sentiment mapping vector ${\mathbf{\nabla }}_i$ to the word embedding ${\mathit{w}}_i$. Then, the goal of the S-EWE method is to solve the mapping relationship function among $w_i,{\mathit{w}}_i$ and ${\mathbf{\nabla }}_i$, denoted by $f\left(y_i\right|w_i,{\mathbf{\nabla }}_i)$. Figure 3 shows the running process of the S-EWE method. The method implements $f\left(y_i\right|w_i,{\mathbf{\nabla }}_i)$ by four modules, including the embedding layer, addition layer, MLP layer, and loss function as follows.

3.1.1. Embedding Layer

The embedding layer converts words to word embeddings, which is the first step in finding the relationship between $w_i$ and $y_i$. Let sentiment words $w=\{w_1,w_2,\dots ,w_n\}$ where $w_i\in \mathcal{V}$, and n is the total number of sentiment words, and their sentiment orientations $Y=\{y_1,y_2,\dots ,y_n\}\in \mathcal{Y}$. Then, we first turn words into word embeddings as follows.

$$\mathit{w}=\mathrm{embed}\left(w\right)=\{{\mathit{w}}_1,{\mathit{w}}_2,\cdots ,{\mathit{w}}_n\},$$

(1)

where ${\mathit{w}}_i\in {\mathbb{R}}^{ed}$ is the word embedding of the word $w_i$ ($ed$ is the dimension of the embedding size), and $\mathrm{embed}(·)$ is the function that turns words into their word embeddings.

3.1.2. Addition Layer

The addition layer implements simple addition between vectors. We define a sentiment mapping matrix $\mathcal{R}=\{{\mathbf{\nabla }}_1,{\mathbf{\nabla }}_2,\dots ,{\mathbf{\nabla }}_n\}$, where ${\mathbf{\nabla }}_i\in {\mathbb{R}}^{ed}$ is the mapping relationship vector between ${\mathit{w}}_i$ and $y_i$. Let $\mathit{E}=\{{\mathit{e}}_1,{\mathit{e}}_2,\dots ,{\mathit{e}}_n\}$ be the sentiment-enhanced word embeddings of sentiment words $w=\{w_1,w_2,\dots ,w_n\}$, where ${\mathit{e}}_i\in {\mathbb{R}}^{ed}$. Then, by adding word embedding ${\mathit{w}}_i$ to its sentiment mapping vector ${\mathbf{\nabla }}_i$, we can obtain the sentiment-enhanced word embedding ${\mathit{e}}_i$ of the word $w_i$ as follows.

$${\mathit{e}}_i={\mathit{w}}_i+{\mathbf{\nabla }}_i.$$

(2)

Then, the sentiment-enhanced word embedding of $\mathit{w}$ is as follows.

$$\mathit{E}=\mathit{w}+\mathcal{R}.$$

(3)

3.1.3. MLP Layer

The MLP layer contains one fully connected network. It will reduce the dimensions of sentiment-enhanced word embeddings and map them and their sentiment orientation labels into the same vector space. Considering that the addition of vectors does not change the vectors’ dimensions, the dimensions of ${\mathit{e}}_i$ are still much more than those of $y_i$. We use one MLP to reduce the dimensionality of the sentiment-enhanced word vector. The MLP will map the sentiment-enhanced word embedding and label to the same vector space [38]. The predicted sentiment orientation label of the word $w_i$, denoted by ${\widehat{y}}_i$, is as follows.

$${\widehat{y}}_i=\mathrm{softmax}(\mathit{W}{\mathit{e}}_i+\mathit{b}),$$

(4)

where $\mathit{W}$ and $\mathit{b}$ are the weight matrix and bias of the output layer, respectively.

3.1.4. Loss Function

The loss function measures the difference between true sentiment orientation labels in a sentiment lexicon and predicted sentiment orientation labels of words at the final module of the S-EWE method. We use the cross-entropy function

$$\mathcal{L}=-\frac{1}{N}\sum _{i=1}^N\left[y_i\log \left({\widehat{y}}_i\right)+(1-y_i)\log (1-{\widehat{y}}_i)\right],$$

(5)

to calculate the loss, where N is the total number of samples.

In order to better learn the mapping relationship between original word embeddings and their true sentiment orientation labels in the S-EWE method, we freeze the word embedding layer. All word embeddings and true sentiment orientation labels are fixed in this case. The parameters W, b, and sentiment mapping matrix are trainable. By backpropagation, the method can better learn the sentiment mapping matrix and find the mapping relationship between original word embeddings and their corresponding sentiment orientations. After training, we can obtain the well-trained sentiment mapping matrix, denoted as ${\mathcal{R}}^t=\{{\mathbf{\nabla }}_1^t,{\mathbf{\nabla }}_2^t,\cdots ,{\mathbf{\nabla }}_n^t\}$. Finally, the well-trained sentiment-enhanced word embeddings, denoted by ${\mathit{E}}^t=\{{\mathit{e}}_1^t,{\mathit{e}}_2^t,\cdots ,{\mathit{e}}_n^t\}$, are obtained as follows.

$${\mathit{E}}^t=\mathit{w}+{\mathcal{R}}^{\mathit{t}}.$$

(6)

The detailed running procedure of the S-EWE method is described in Algorithm 1.

Algorithm 1 The S-EWE algorithm.

Input: Set of words in a sentiment lexicon $w=\{w_1,w_2,\cdots ,w_n\}$, set of sentiment orientations in the sentiment lexicon Y, number of epochs denoted by $epochs$, batches denoted by $batch$

Output: Well-trained sentiment-enhanced word embeddings ${\mathit{E}}^t$

1: Initialization $\mathcal{R}\leftarrow$ xavier Gaussian initialization

2: $\mathit{w}\leftarrow \mathrm{embed}\left(w\right)$ // Equation (1)

3: $i\leftarrow 0$

4: while $i<epochs$ do

5:      for $(\mathit{w},\mathit{y})\in batch$ do

6:           ${\mathit{e}}_i={\mathit{w}}_i+{\mathbf{\nabla }}_i$ // Equation (2)

7:           $\widehat{\mathit{y}}\leftarrow \mathrm{Softmax}(\mathit{W}{\mathit{e}}_i+\mathit{b})$ // Equation (4)

8:           $\mathcal{L}\leftarrow -\frac{1}{\mathrm{len}\left(batch\right)}{\sum }_{i=1}^{\mathrm{len}\left(batch\right)}\left[y_i\log \left({\widehat{y}}_i\right)+(1-y_i)\log (1-{\widehat{y}}_i)\right]$ // Equation (5)

9:          Backpropagation

10:         Optimize parameters

11:     end for

12:     $i\leftarrow i+1$

13:     Obtain well-trained sentiment-enhanced word embeddings ${\mathcal{R}}^{\mathbf{t}}$

14: end while

15: return well-trained sentiment mapping matrix ${\mathit{E}}^t=\mathit{w}+{\mathcal{R}}^{\mathit{t}}$ // Equation (6)

3.2. Applications of the Sentiment-Enhanced Word Embedding Method on Downstream Tasks

Since the S-EWE method needs to work with a sentiment lexicon, we need a robust sentiment lexicon. We selected the extended version of Affective Norms of English Words (E-ANEW) [50] and the Subjectivity Clue Lexicon [51] as our base sentiment lexicons, for they are well-known English sentiment lexicons. For the E-ANEW [50], we define sentiment words with sentiment intensity between $[1,5)$ as negative, sentiment words with sentiment intensity of 5 as neutral, and sentiment words with sentiment intensity between $(5,9]$ as positive. For the Subjectivity Clue Lexicon [51], we directly extract the sentiment orientation (positive, neutral, negative).

To improve the robustness of these two lexicons, we adopted the following four rules to integrate them: (1) if a word appears in only one lexicon, then the word is directly added into the fused lexicon; (2) if a word has the same sentiment orientation in the two lexicons, then the word is directly added into the fused lexicon; (3) if the sentiment orientation of a word is neutral in one lexicon and the sentiment orientation is not neutral in another lexicon, then the word is assigned with non-neutral sentiment orientation and added into the fused lexicon; and (4) if a word has opposite sentiment orientations (positive and negative) in the two lexicons, then the word is discarded and does not belong to the fused lexicon. After finishing the above four rules, we can obtain the fused lexicon.

Table 1. The details of the sentiment lexicons.

	E-ANEW	Subjectivity Clue Lexicon	Fused Lexicon
Total number of words	13,915	6869	17,293
The number of positive words	7761	2296	8672
The number of negative words	5945	4149	8216
The number of neutral words	209	424	405

shows the details of the E-ANEW, Subjectivity Clue Lexicon, and fused lexicon, respectively.

The words in the sentiment lexicon may not all appear in the corpus in most cases. Meanwhile, the S-EWE method will waste some storage space to train the words in the lexicon and may not obtain good sentiment-enhanced word embeddings. Inspired by sentiment-aware word embeddings (SAWE) [14], we extract all words from the corpus and build a final lexicon by the following two rules. (1) If a word appears in both the corpus and the fused lexicon, we directly add it and its corresponding sentiment orientation in the fused lexicon to the final lexicon. (2) If a word appears in the corpus but not in the fused lexicon, we add it to the final lexicon with the neutral sentiment.

Based on the fused and final lexicons, the sentiment-enhanced word embeddings are applied to downstream tasks for sentiment analysis, as shown in Figure 4. When the sentiment-enhanced word embeddings are trained by the S-EWE method on the fused lexicon (respectively, the final lexicon), we denote the well-trained embeddings by $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$ (respectively, $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$). Figure 4 shows the applications of the sentiment-enhanced word embeddings trained by the S-EWE method based on the final and fused lexicons for downstream tasks. Figure 4a shows how we apply $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$ to downstream tasks, and Figure 4b represents how we apply $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ to downstream tasks. When the fused lexicon does not contain some words in the corpus, we adopt the following two methods to get the embedding layer of the downstream sentiment analysis models (Figure 4a). (1) If a word is in the fused lexicon, we input the well-trained sentiment-enhanced word embedding by the S-EWE method in the fused lexicon as the word embedding with sentiment information into the embedding layer of the downstream tasks. (2) If a word is not in the fused lexicon, we use the original word embedding as the word embedding with sentiment information into the embedding layer of the downstream tasks. When the final lexicon contains all words in the corpus (Figure 4b), we can directly input the well-trained sentiment-enhanced word embeddings by the S-EWE method in the final lexicon as the word embedding with sentiment information into the embedding layer of the downstream tasks.

4. Experiment

4.1. Datasets and Evaluation Metrics

The SemEval-2013 [27] and SST-2 (the preprocessed SST-2 dataset was selected as an experimental dataset; the preprocessed data are at https://github.com/clairett/pytorch-sentiment-classification, accessed on 26 November 2021) [28] datasets were selected as experimental datasets. The SemEval-2013 dataset is a Tweet sentiment analysis dataset containing five categories of labels (positive, negative, neutral, objective, and objective-or-neutral). Our experiment only considered the positive and negative data by SemEval-2013 (binary). The SST-2 dataset contains movie reviews containing two labels: (1) positive and negative, and (2) very negative, negative, neutral, positive, and very positive. From this dataset, our experiment only used the positive and negative data.

Table 2. The details of SemEval-2013 and SST-2 datasets.

	SemEval-2013 (Binary) [27]			SST-2 [28]
	Positive	Negative	Total	Positive	Negative	Total
Train	2974	1159	4133	3610	3310	6920
Develop	483	280	763	444	428	872
Test	1281	472	1753	912	909	1821

and

Table 3. The numbers of sentiment words in SemEval-2013 and SST-2.

Datasets	Total Words	Positive Words	Negative Words	Neutral Words
SemEval-2013	15,133	2755	1412	10,966
SST-2	16,008	4001	2994	9013

show the details of all the datasets.

The accuracy and macro-F1 score [14] were adopted to evaluate the performance of the proposed S-EWE method. Let $TP$ (resp. $TN$) be the number of samples whose true labels and model prediction labels are both positive (resp. negative), and let $FN$ (resp. $FP$) be the number of samples whose true labels are positive (resp. negative) and model prediction labels are negative (resp. positive). Accuracy and macro-F1 score are defined as

$$\mathrm{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN},$$

(7)

and

$$\mathrm{Macro}\text{-}\mathrm{F}1=\frac{1}{k}\sum _{i=1}^{k}F{1}_i,$$

(8)

where $P_i=\frac{T{P}_i}{T{P}_i+F{P}_i},R_i=\frac{T{P}_i}{T{P}_i+F{N}_i}$ and $\mathrm{F}{1}_i=\frac{2P_i{R}_i}{P_i+R_i}$ stand for the precision, recall, and F1 scores of the category i; and k is the total number of categories.

4.2. Comparison Experimental Models and Their Training Details

TextCNN [22], TextRNN [23], TextRCNN (our main reference codes come from: https://github.com/649453932/Chinese-Text-Classification-Pytorch, accessed on 5 December 2021) [24], ABCDM [25], and CNN-LSTM [26] were selected as sentiment classification models to verify the effectiveness of trained sentiment-enhanced word embeddings by the S-EWE method in fused and final lexicons.

TextCNN. TextCNN [22] convolves word vectors in the convolution kernel. It obtains the feature maps of these convolution results through max pooling.

TextRNN. TextRNN [23] is a bidirectional LSTM. It inputs word embeddings and outputs the hidden state of each word.

TextRCNN. TextRCNN [24] uses LSTM (resp. BiLSTM) to obtain the forward (resp. bidirectional) hidden state. It applies max pooling to get sentence embeddings.

ABCDM. ABCDM [25] adopts BiLSTM and BiGRU to encode one sentence to obtain long-term and short-term hidden states. Then, it uses two parallel attention layers to obtain the attention score of each hidden state. Finally, ABCDM applies 1D-CNN, 1D-max pooling, and 1D-average pooling to get the final sentence embedding.

CNN-LSTM. CNN-LSTM [26] uses TextCNN [22] to obtain local sentence embedding, and adopts LSTM to get the long-term information.

In these sentiment classification models, Adam [52] optimizer is regarded as the optimizer. The cross-entropy function is selected as the loss function. The classical feature-based word embedding methods, including Word2Vec [1] and GloVe [2], are used to pre-train word embeddings. Specifically, GoogleNews-vectors-negative300 (https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?resourcekey=0-wjGZdNAUop6WykTtMip30g, accessed on 5 November 2021 and glove.840B.300d (https://nlp.stanford.edu/data/glove.840B.300d.zip, accessed on 10 December 2021) were adopted. The embedding dimensions of Word2Vec and GloVe were 300. All parameters of the S-EWE method and downstream task models are shown in

Table 4. Parameter details of different models.

	S-EWE	TextCNN	TextRNN	TextRCNN	TextBiRCNN	ABCDM	CNN-LSTM
epochs	200	40	40	40	40	100	100
batch size	64	64	64	128	128	64	64
dropout	-	0.5	0.5	1.0	1.0	0.5	0.5
learning rate	1 × 10${}^{-3}$	1 × 10${}^{-4}$	1 × 10${}^{-4}$	1 × 10${}^{-4}$	1 × 10${}^{-4}$	1 × 10${}^{-4}$	1 × 10${}^{-4}$
early stop	512	256	256	512	512	512	512
filter sizes	-	(3, 4, 5)	-	-	-	(4, 6)	3
number of filters	-	100	-	-	-	32	600
hidden sizes	-	-	128	256	256	128	600
hidden layers	-	-	1	1	1	1	1

.

4.3. Experimental Results

Let the sentiment classification (SC) models enhanced by $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ and $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$ be denoted by SC${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$ and SC${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$, respectively. When Word2Vec was used to pre-train word embeddings for sentiment classification tasks on SemEval-2013 [27] and SST-2 [28] datasets, the accuracies and macro-F1 scores under the sentiment classification models TextCNN, TextRNN, TextRCNN, TextBiRCNN, ABCDM, and CNN-LSTM with and without $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ and $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$ are shown in

Table 5. The accuracies and macro-F1 scores of different models using Word2Vec with and without $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ and $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$ on SemEval-2013 and SST-2 datasets (%). For each model, we ran it five times to achieve five accuracy values and macro-F1 values, respectively. Then, we took their average accuracies and macro-F1 values as the final results. The value in parentheses is the variance of the five values for each model. The bold text represents the best result achieved by each model, and the underlined text stands for the lowest variance achieved by each model.

Models	SemEval-2013		SST-2
	Accuracy	Macro-F1	Accuracy	Macro-F1
TextCNN [22]	80.02 (0.34)	73.42 (1.22)	79.81 (0.83)	79.81 (0.83)
TextCNN${}_{\mathrm{S}\text{-}\mathrm{E}\text{-}{\mathrm{WE}}_{\mathrm{f}}}$	80.55 (0.17)	74.13 (0.52)	80.89 (0.15)	80.88 (0.15)
TextCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$	80.52 (0.23)	74.55 (0.71)	80.69 (0.12)	80.68 (0.12)
TextRNN [23]	73.07 (0)	42.22 (0)	49.92 (0)	33.30 (0)
TextRNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$	73.07 (0)	42.22 (0)	49.92 (0)	33.30 (0)
TextRNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$	73.07 (0)	42.22 (0)	49.92 (0)	33.30 (0)
TextRCNN [24]	73.82 (0.09)	60.31 (44.45)	73.64 (1.13)	73.47 (1.34)
TextRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$	74.87 (1.73)	65.38 (2.85)	75.65 (1.05)	75.61 (1.1)
TextRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$	75.42 (0.19)	64.81 (22.84)	75.72 (0.15)	75.66 (0.13)
TextBiRCNN [24]	74.07 (1.24)	68.57 (1.14)	74.15 (0.74)	74.08 (0.75)
TextBiRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$	76.67 (1.05)	69.95 (1.03)	75.97 (0.33)	75.95 (0.34)
TextBiRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$	76.13 (0.55)	71.22 (1.89)	76.17 (0.24)	76.16 (0.24)
ABCDM [25]	74.85 (2.98)	69.36 (0.61)	74.26 (0.54)	74.22 (0.59)
ABCDM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$	76.08 (0.34)	68.84 (0.52)	75.15 (0.13)	75.10 (0.12)
ABCDM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$	75.09 (0.32)	69.58 (1.21)	75.51 (0.06)	75.47 (0.06)
CNN-LSTM [26]	76.39 (2.54)	68.43 (7.85)	75.75 (0.35)	75.73 (0.33)
CNN-LSTM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$	75.48 (3.17)	68.47 (0.25)	76.06 (0.12)	76.01 (0.11)
CNN-LSTM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$	76.19 (1.35)	69.62 (0.52)	76.40 (0.36)	76.38 (0.36)

. We conclude that: (1) the sentiment classification models with Word2Vec enhanced by $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$ and $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ had higher accuracies and macro-F1 than the original Word2Vec except for TextRNN and CNN-LSTM; (2) in most cases, the sentiment classification models with Word2Vec enhanced by $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ achieved the best accuracies and macro-F1 scores on two datasets; (3) TextRNN with Word2Vec enhanced by $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$ and $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ achieved the same accuracies and macro-F1 scores of the TextRNN with original Word2Vec—that is, the S-EWE method did not improve the performance of TextRNN on the two datasets; and (4) on the two datasets, the variance of the results obtained by the model using the proposed method was minor, indicating that using S-EWE method can make the model more stable.

When GloVe is used to pre-train word embeddings for sentiment classification tasks on SemEval-2013 [27] and SST-2 [28] datasets, the accuracy and maro-F1 scores under sentiment classification models, including TextCNN, TextRNN, TextRCNN, TextBiRCNN, ABCDM, and CNN-LSTM, enhanced by $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ and $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$ are shown in

Table 6. The accuracies and macro-F1 scores of different models using GloVe with/without $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ and $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$ on SemEval-2013 and SST-2 datasets (%). The accuracies and macro-F1 values were obtained in the same way as those in Table 5. Moreover, the values in parentheses, bold text, and underlined text represent the common meanings in Table 5.

Models	SemEval-2013		SST-2
	Accuracy	Macro-F1	Accuracy	Macro-F1
TextCNN [22]	80.07 (0.02)	73.17 (0.02)	79.88 (0.31)	79.88 (0.31)
TextCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$	80.11 (0.02)	73.27 (0.03)	79.99 (0.01)	79.99 (0.01)
TextCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$	80.62 (0.05)	73.68 (0.19)	80.14 (0.16)	80.14 (0.16)
TextRNN [23]	73.07 (0)	42.22 (0)	49.92 (0)	33.30 (0)
TextRNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$	73.07 (0)	42.22 (0)	49.92 (0)	33.30 (0)
TextRNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$	73.07 (0)	42.22 (0)	49.92 (0)	33.30 (0)
TextRCNN [24]	72.62 (1.00)	64.63 (10.90)	72.56 (3.51)	72.45 (4.00)
TextRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$	74.03 (0.78)	66.40 (10.34)	74.83 (0.30)	74.78 (0.35)
TextRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$	73.25 (2.26)	67.40 (1.38)	75.65 (0.63)	75.63 (0.63)
TextBiRCNN [24]	73.66 (2.57)	66.37 (1.26)	73.32 (0.83)	73.29 (0.83)
TextBiRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$	74.92 (1.13)	66.81 (1.15)	75.12 (0.85)	75.10 (0.9)
TextBiRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$	75.70 (0.26)	67.50 (2.58)	75.29 (0.19)	75.25 (0.19)
ABCDM [25]	73.77 (5.42)	64.29 (11.64)	73.48 (0.71)	73.43 (0.66)
ABCDM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$	74.76 (0.68)	67.56 (1.50)	74.97 (1.45)	74.94 (1.46)
ABCDM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$	73.94 (1.72)	67.10 (0.27)	74.31 (0.17)	74.28 (0.15)
CNN-LSTM [26]	74.82 (3.77)	67.84 (2.02)	71.65 (3.74)	71.56 (4.30)
CNN-LSTM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$	76.85 (0.39)	68.91 (1.87)	72.38 (0.27)	72.35 (0.26)
CNN-LSTM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$	76.95 (0.49)	68.72 (1.72)	72.56 (0.06)	72.51 (0.07)

. Compared with these results, we can conclude that: (1) the sentiment classification models with GloVe enhanced by $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$ and $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ had higher accuracies and macro-F1 scores than the original GloVe except for TextRNN; (2) in most cases, the sentiment classification models with GloVe enhanced by $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ achieved the best accuracies and macro-F1 scores on two datasets; (3) TextRNN is not trainable on these two datasets with the proposed method; and (4) models using our method achieved more robust results on both datasets.

As shown in Table 5 and Table 6, TextCNN, TextRCNN, TextBiRCNN, ABCDM, and CNN-LSTM with Word2Vec or GloVe, enhanced by $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ and $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$, achieved better classification performances than these models without the sentiment-enhanced word embeddings on the two datasets. In particular, these models with $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ obtained better classification performances than those with $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$. TextRNN with Word2Vec or GloVe enhanced by $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ and $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$ kept the same classification performance as the model without the sentiment-enhanced word embedding on the two datasets. Meanwhile, models using $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ and $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$ can achieve more robust results on the two datasets.

4.4. Analysis of Convergence Time for Downstream Tasks

Under TextCNN, TextRNN, TextRCNN, TextBiRCNN, ABCDM, and CNN-LSTM using Word2Vec and GloVe with or without $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ and $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$ on SemEval-2013 and SST-2 datasets, their convergence times were analyzed on one NVIDIA 2080Ti (12GB), as shown in

Table 7. Comparisons with the convergence times of TextCNN, TextRNN, TextRCNN, TextBiRCNN, ABCDM, and CNN-LSTM using Word2Vec and GloVe with or without $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}$ and $\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}$ on two datasets. The contents in parentheses indicate the increased or decreased convergence speeds compared to the original models.

Input Embedding		Dataset	SemEval-2013	SST-2
	Model
	TextCNN [22]		19.36	18.54
	TextCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$		21.08 (8.88% ↑)	28.76 (55.12% ↑)
	TextCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$		21.50 (11.05% ↑)	25.36 (36.79% ↑)
	TextRNN [23]		67.50	54.49
	TextRNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$		68.86 (2.01% ↑)	54.49 (0%)
	TextRNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$		64.71 (−4.13% ↓)	54.30 (−0.35% ↓)
	TextRCNN [24]		392.75	459.07
	TextRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$		526.43 (34.04% ↑)	346.01 (−24.63% ↓)
	TextRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$		536.14 (36.51% ↑)	499.65 (8.84% ↑)
Word2Vec	TextBiRCNN [24]		770.44	647.97
	TextBiRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$		825.24 (7.11% ↑)	647.19 (−0.12% ↓)
	TextBiRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$		925.44 (20.12% ↑)	715.88 (10.48% ↑)
	ABCDM [25]		97.04	46.77
	ABCDM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$		47.35 (−51.21% ↓)	60.61 (29.59% ↑)
	ABCDM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$		53.70 (−44.66% ↓)	61.65% (31.82% ↑)
	CNN-LSTM [26]		190.94	71.09
	CNN-LSTM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$		174.97 (−8.36% ↓)	177.24 (149.32% ↑)
	CNN-LSTM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$		105.59 (−44.70% ↓)	384.11 (440.32% ↑)
	TextCNN [22]		18.12	34.44
	TextCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$		21.93 (21.03% ↑)	29.07 (−15.59% ↓)
	TextCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$		21.74 (19.98% ↑)	31.55 (−8.39% ↓)
	TextRNN [23]		66.32	55.65
	TextRNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$		66.78 (0.69% ↑)	70.94 (27.48% ↑)
	TextRNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$		66.63 (0.47% ↑)	69.94 (25.68% ↑)
	TextRCNN [24]		339.23	576.37
	TextRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$		390.31 (15.06% ↑)	461.90 (−19.86% ↓)
GloVe	TextRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$		437.42 (28.94% ↑)	424.92 (−26.28% ↓)
	TextBiRCNN [24]		822.67	719.04
	TextBiRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$		822.35 (−0.04% ↓)	790.64 (9.96% ↑)
	TextBiRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$		824.52 (0.22% ↑)	794.55 (10.50% ↑)
	ABCDM [25]		47.59	74.07
	ABCDM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$		64.50 (35.53% ↑)	78.82 (6.41% ↑)
	ABCDM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$		119.70 (151.52% ↑)	95.43% (28.84% ↑)
	CNN-LSTM [26]		164.84	189.07
	CNN-LSTM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$		159.77 (−3.08% ↓)	222.25 (17.55% ↑)
	CNN-LSTM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$		121.59 (−26.24% ↓)	110.91 (−41.34% ↓)

. For all baseline models using Word2Vec enhanced by S-EWEs on the SemEval-2013 dataset, the convergence times of TextCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$, TextRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$, and TextBiRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$ were on average increased by 22.56% over their original models; however, the convergence times of TextRNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$, ABCDM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$, and CNN-LSTM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$ on average decreased by 31.16%. Meanwhile, for all baseline models using Word2Vec enhanced by S-EWEs on the SST-2 dataset, the convergence time of CNN-LSTM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$ was the most significant increase, by 440.32% over CNN-LSTM; the convergence time of TextRNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$ decreased the convergence time of TextRNN by 0.35%; and the convergence times of TextCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$, TextRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$, TextBiRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$, and ABCDM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$ were on average increased by 21.98% of their original models.

For all baseline models using GloVe enhanced by S-EWEs on the SemEval-2013 dataset, the convergence times of TextCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$, TextRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$, TextBiRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$, and ABCDM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$ were on average increased by 50.17% compared to their original models. The convergence time of CNN-LSTM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$ was decreased by 26.24% compared to CNN-LSTM. Meanwhile, using GloVe enhanced by S-EWEs on the SST-2 dataset, the convergence times of TextBiRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$ and ABCDM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$ were on average increased by 19.67% compared to their original models; the convergence times of TextCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$, TextRCNN${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$, and CNN-LSTM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$ were on average decreased by 25.34% compared to their original models.

In all baseline models with the S-EWE method, the convergence time of ABCDM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$ was significantly decreased by 51.21% compared to ABCDM on the SemEval-2013 dataset, and the convergence time of CNN-LSTM${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{C}}}$ vastly decreased by 41.34% compared to CNN-LSTM on the SST-2 dataset.

Based on the performances of various sentiment classification models and models’ convergence times, we conclude that the proposed S-EWE method can enhance the ability of sentiment classification on SemEval-2013 and SST-2 datasets.

4.5. Comparisons with the Contextualized Word Embedding Models and the BERT with/without the S-EWEs

The contextualized word embedding models, including BERT [3] and CoSE-T [6], were selected to compare with the sentiment-enhanced word embedding method.

BERT [3]. It is a classic transformer-based [30] model. Its backbone is the encoder in a transformer. It adopts the masked language model (MLM) and next sentence prediction (NSP) as the pre-training objectives. When it is doing the downstream sentiment analysis task, the hidden state of the special token (CLS) it outputs will go through a pooler layer and output the classification result. We adopt the BERT base as our experimental model.

CoSE-T [6]. It has two BiGRU layers. In the pre-training stage, the target words, word sentiment, and sentence sentiment are predicted by training on the labeled sentiment corpus with a sentiment lexicon. In the fine-tuning stage, the pre-trained word embedding and the hidden states of the two BiGRUs are concatenated as the sentiment representation. Finally, the representation will be fed into one fully connected layer with a softmax function to predict sentence sentiment.

BERT with S-EWEs. We extract the weights in the embedding layer of BERT and use S-EWE${}_f$ and S-EWE${}_c$ to enhance these weights. After enhancing the sentiment of the BERT embeddings, we replace these weights with the S-EWEs. Finally, we feed them into the BERT model.

Table 8. Theaccuracies and macro-F1 scores of BERT (with and without S-EWE) and CoSE-T on SemEval-2013 and SST-2 datasets (%). ${}^{†}$ represents that the results are from [6].

Model	SemEval-2013		SST-2
	Accuracy	Macro-F1	Accuracy	Macro-F1
CoSE-T ${}^{†}$	90.80	-	89.80	-
BERT	90.80	88.88	90.38	90.38
BERT${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$	81.66	77.54	79.41	79.40
BERT${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$	75.83	68.67	72.66	72.65

shows the accuracies and macro-F1 values of the BERT and CoSE-T models on SemEval-2013 and SST-2 datasets. From these results, we can observe that: (1) Contextualized word embedding can achieve better results than feature-based word embedding (shown in Table 5 and Table 6) in sentiment analysis tasks. Specifically, CoSE-T achieved 90.80% accuracy on the SemEval-2013 dataset and 89.80% accuracy on the SST-2 dataset; BERT achieved 90.80% accuracy and 88.88% macro-F1 on the SemEval-2013 dataset, and 90.38% accuracy and 90.38% macro-F1 on the SST-2 dataset. (2) BERT with S-EWE embeddings performs worse than BERT and CoSE-T without S-EWE embeddings. Specifically, on the SemEval-2013 dataset, BERT${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$ was 9.14% less accurate than BERT and CoSE-T, and 11.34% lower than BERT in macro-F1; BERT${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$ was 14.97% less accurate than BERT and CoSE-T and 20.21% lower than BERT in macro-F1. On the SST-2 dataset, BERT${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{f}}}$ was 10.39% lower than CoSE-T and 10.97% lower than BERT in accuracy, and 11.34% lower than BERT in macro-F1; BERT${}_{\mathrm{S}\text{-}{\mathrm{EWE}}_{\mathrm{c}}}$ was 17.14% lower than CoSE-T and 17.72% lower than BERT in accuracy, and 17.73% lower than BERT in macro-F1. From these observations, we can conclude that the S-EWE method is unsuitable for enhancing contextualized word embedding. The main reasons are that the S-EWE method only enhances the embedding layer of the model and has no effect on the model itself.

5. Conclusions

This paper proposes a sentiment enhancement method, i.e., the sentiment-enhanced word embedding method. This model finds the relationship between the words in the sentiment lexicon and their corresponding sentiment orientations. After training the sentiment mapping matrix, this matrix and word embeddings are fused as sentiment-enhanced word embeddings. Then, the sentiment-enhanced word embeddings are fed into sentiment classification models to classify the sentiment orientations of sentences on SemEval-2013 and SST-2 datasets. Experimental results show that these models using Word2Vec and GloVe enhanced by the sentiment-enhanced word embeddings perform better than those with original Word2Vec and GloVe embeddings. Moreover, the convergence times of the models using Word2Vec and GloVe enhanced by the sentiment-enhanced word embeddings were acceptable. Since the S-EWE method only enhances the embedding layer of the model and has no effect on the model itself, it does not effectively enhance contextualized word embedding.

In the future, there will be some sustainable research directions. (1) The proposed sentiment mapping embeddings could be considered external knowledge in other sentiment classification models. (2) Since sentiment orientations contain less information than sentiment intensity, sentiment intensity could be considered in the S-EWE method. (3) Since Chinese words contain one or more tokens, various Chinese sentiment-enhanced word embedding methods need further study by expanding the S-EWE method. (4) The methods of injecting sentiment into contextualized word embeddings, such as BERT, need to be used in sentiment analysis.