Compressing BERT for Binary Text Classification via Adaptive Truncation before Fine-Tuning

Featured Application

The proposed approach can be used to reduce the memory footprint, training and inference time of BERT based binary text classifiers which could serve as spam or anomaly detector, sentiment discriminator, and so on.

Abstract

Large-scale pre-trained language models such as BERT have brought much better performance to text classification. However, their large sizes can lead to sometimes prohibitively slow fine-tuning and inference. To alleviate this, various compression methods have been proposed; however, most of these methods solely consider reducing inference time, often ignoring significant increases in training time, and thus are even more resource consuming. In this article, we focus on lottery ticket extraction for the BERT architecture. Inspired by observations that representations at lower layers are often more useful for text classification, we propose that we can identify the winning ticket of BERT for binary text classification through adaptive truncation, i.e., a process that drops the top-k layers of the pre-trained model based on simple, fast computations. In this way, the cost for compressing and fine-tuning, as well as inference, can be vastly reduced. We present experiments on eight mainstream binary text classification datasets covering different input styles (i.e., single-text and text-pair), as well as different typical tasks (e.g., sentiment analysis, acceptability judgement, textual entailment, semantic similarity analysis and natural language inference). Compared with some strong baselines, our method saved 78.1% time and 31.7% memory on average, and up to 86.7 and 48% in extreme cases, respectively. We also saw good performance, often outperforming the original language model.

1. Introduction

Binary text classification, abbreviated to BTC throughout this article, is an important and classical research problem that can support a long list of practical applications, such as sentiment analysis [1], paraphrase identification [2] and spam detection [3]. Recently, the state of the art in BTC has significantly advanced via the use of large-scale, pre-trained language models such as BERT (bidirectional encoder representations from transformers) [4]. However, such models commonly have millions, if not billions, of parameters, and therefore require enormous resources for training, fine-tuning and inference. Even for relatively modest-sized languages models such as BERT_base [4], fine-tuning sometimes takes several hours per epoch [5]. This prohibits them from being applied in resource-limited, on-device or streaming data scenarios [6,7].

Model compression is one approach to mitigating this issue. Various methods for compressing large-scale language models have been proposed in the last two years [8,9,10,11,12,13]. From the perspective of downstream tasks, current model compression methods could be classified as task-agnostic and task-specific compression. The former aims at compressing pre-trained language models for any downstream NLP task, whereas the latter shrinks models that are fine-tuned for a particular task. As discussed in [8,14,15], task-specific compression has two important advantages over task-agnostic compression: First, it is often faster and more sample-efficient. In contrast, task-agnostic compression requires (re-)training on very large (unlabeled) datasets, and hence is resource-hungry. Second, models obtained by task-specific compression can adopt specialized attention patterns that are useful for downstream tasks, but not necessarily useful for general language modeling.

Contemporary task-specific compression methods fall into four categories: quantization, pruning, distillation and matrix decomposition [8,16,17,18]. Among them, distillation and pruning are perhaps the most used. The general procedure of these two kinds of methods is illustrated in Figure 1 (upper box). (Note that pruning methods commonly require fine-tuning again after model compression, which has been omitted in Figure 1 due to this process being very simple and rarely mentioned in the literature.) As shown, knowledge distillation implements compression via transferring knowledge from a large teacher to a much smaller student, whereas the pruning methods finish compression based on one model. There exist two common weaknesses of these methods: First, they reduce the calculations needed in the inference stage at the cost of a much longer training time (fine-tuning and compressing). In fact, the resources consumed by fine-tuning are often prohibitive of the application of these methods [6,7]. Second, few of them consider the complexity of downstream tasks, as well as how the language models and target tasks correlate, when determining the architecture of the student model or the units in the language model to be pruned.

To address these weaknesses, we present a new task-specific model compression method and apply it to the BERT language model [4], which is also illustrated in Figure 1 (lower box) for ease of comparison. As shown, the proposed method relies on adaptively truncating BERT for a given BTC task before fine-tuning. Specifically, by dropping the top-k Transformer layers in BERT, we observe that the full BERT is not always the best choice when fine-tuning on downstream BTC tasks. The classification accuracy achieved by some truncated ones can get very close to or even higher than that of the full BERT. This indicates that compressing BERT via appropriate truncation could cause no significant loss and even bring notable gain to the prediction accuracy of downstream BTC tasks. To properly and efficiently determine the truncation position, we conduct some theoretic analysis of the fine-tuning process, together with a series of tentative experiments, which motivate us to choose a class separability index as a proxy. The chosen measure can be rapidly calculated before fine-tuning and is found to be highly correlated with the final classification accuracy. As a result, only truncated BERT, plus a task-specific head, needs to be fine-tuned, and hence resources consumed in both fine-tuning and inference stages could be saved with negligible additional cost. For validation, we conduct experiments on eight commonly used BTC tasks. The results show that our method, albeit simple, is very effective, outperforming both knowledge distillation and weight pruning methods.

In summary, the main contributions of this article include:

• Proposition of a simple, but effective model compression method that adaptively truncates BERT by layer-dropping prior to fine-tuning, substantially reducing the cost of fine-tuning and inference. We observe that the full BERT model is not always the best point of departure for fine-tuning models for particular downstream tasks. Sometimes shrinking BERT by truncating layers leads to better performance, as well as faster fine-tuning and inference.
• A criterion to estimate truncating position, according to pre-trained representations of instances. As a result, our method finds winning tickets [19,20,21] more before fine-tuning. In addition, the provided criterion makes the balance between the loss in classification accuracy and reduction of the model size controllable, and thus is adaptive for downstream tasks and supports compression on-demand.
• Experiments that show that, compared with some strong baselines, the proposed method reduces fine-tuning time by 78% and memory requirements by 31.7%, on average, with up to 86.7 and 48% reductions in some extreme cases, respectively. On average, 39.6% of storage can be saved compared with original BERT-based models in both fine-tuning and inference. In addition, the accuracy achieved surpasses all our baselines on six out of eight tasks, and is higher than the full BERT model on five tasks.

The remainder of this article is structured as follows: Section 2 briefly reviews related work; Section 3 discusses how to truncate BERT and how the resulting models perform on BTC tasks; Section 4 presents our compression method of truncating BERT before fine-tuning; Section 5 describes the experimental settings, the results of our approach and competitive methods. Finally, we draw a conclusion and evaluate future work in Section 6.

2. Related Work

Three broad classes of related work pertinent to that presented in this article are briefly surveyed in the following section.

2.1. Pre-Trained Language Models

Since about 2017, complex language models began to gradually appear. The widely used BERT, proposed by Devlin et al. [4], achieved pretty good results on GLUE [22], SQuAD [23] and SWAG [24] Before long, XLNet [25], RoBERTa [26] and some others [27,28,29] emerged and pushed the evaluation scores of NLP tasks much higher. With the development of pre-trained language models, a wide range of NLP tasks obtained new state-of-the-art results again and again, including question answering [13], language inference [30,31], text classification [32] and text summarization [33]. Consequently, language models have become an indispensable part in solving NLP assignments. Although powerful, these models often contain huge quantities of parameters, and hence are resource-hungry. To deploy them in a resource-limited scenario, compression, i.e., reducing model size without significant accuracy loss, becomes necessary.

2.2. Neural Model Compression

As previously mentioned, contemporary neural model compression methods fall into four main categories: quantization, matrix decomposition, knowledge distillation and model pruning [17]. All have been successfully applied to compressing BERT. Among them, the last category is most closely related to our proposed method. Thus, we only recap the key ideas of the first three, and devote more attention to existing pruning methods below.

Quantization uses a smaller number of unique values to represent model weights, and hence reduces the resources required to store and compute them. Matrix decomposition refers to decomposing large parameter matrices into small ones, and thus reduces resource consumption. Knowledge distillation is transferring knowledge from one or more highly regularized large models (called the teachers) into a much smaller, distilled model (called the student). Pruning accomplishes compression by identifying and dropping unimportant or redundant model weights or units. The lottery ticket hypothesis (LTH), first proposed by Frankle and Carbin [19], substantiates the existence of sub-networks that can reach test accuracy that is comparable with the original network, and has inspired many later works on pruning. Existing methods can be grouped into two classes, viz., structured and unstructured pruning. The former reduces models by dropping several entire layers or some coherent groups of blocks [10,34,35]. In contrast, unstructured pruning is a form of dropout that compresses model by reducing weights independently [19,20,21,36].

Generally, these existing compression methods, distillation and pruning in particular, require a lot of extra training to transfer knowledge from teachers to student or identify units or weights to be removed. They sometimes even need to conduct fine-tuning followed by iterative compressing. In comparison, our approach, though it looks like a kind of structured pruning as it drops contiguous model units (viz., Transformer layers), differs significantly from existing compression methods in two main aspects: (i) it only needs to fine-tune the (truncated) model once and requires no extra training to determine the truncation position, and (ii) it decides the number of Transformer layers to be retained for each task using a metric evaluated on the corresponding labeled dataset, and hence is adaptive to downstream tasks. Thus, it not only saves resources for inference, but also those for training. It is noteworthy that the method proposed in this article is complementary to quantization, matrix decomposition, distillation, and even some pruning methods, as it can provide a better start point than the full BERT, in terms of its smaller model size and often higher classification accuracy, for these existing compression methods.

2.3. Class Separability

Class separability refers to a collective geometric feature of a set of labeled samples that gauges how the samples with different class labels separate from each other in the representation space. Various separability indices have been proposed, e.g., those in [37,38,39]. The key idea shared by most of them is to measure and compare the (maximal) intra-class distance to the (minimal) inter-class distance. As it is impossible to know the performance of machine learning models before they are applied, separability indices make it possible to get a prior glimpse into the model outcome [40,41,42]. Moreover, it is used in data-driven analysis for data peculiarities [43,44], providing a better understanding of the data to obtain better prediction results. In the current era of deep learning, it is also used to provide different insights into the deep neural network and the spatial distribution of the resulting representations [39,45,46]. In this article, we select a simple separability measure that is found to closely correlate with the classification accuracy of a classifier after being fine-tuned. We use this separability measure as a proxy to determine, before fine-tuning, the number of top-most layers in BERT to be dropped for each downstream BTC task, under the constraint that no significant accuracy decline would be incurred.

3. Full BERT Does Not Always Perform the Best

This section first provides an overview of BERT, which is the target model to be compressed, and then describes how to truncate it and how the resulting models perform on various BTC tasks.

3.1. A Brief Introduction to BERT

A BERT model [4] consists of multiple stacked Transformers [47]. Its size is mainly determined by three hyper-parameters: the number of Transformer layers, the dimensionality of hidden state vectors and the number of self-attention heads in each Transformer. Typical settings include (12, 768, 12) and (24, 1024, 16). The corresponding models are called BERT_base and BERT_large, which contain 110M and 360M parameters, respectively, and hence need huge computation resources and enormous corpora for training. To mitigate this, BERT models are pre-trained in a self-supervised manner using large-scale unlabeled texts, such as those collected from Wikipedia, on two tasks, i.e., masked language modeling (MLM) and next sentence prediction (NSP). In addition, the model parameters learned via pre-training have been made publicly available. Thus, a BERT-based model for a downstream task such as BTC (see Figure 2 for example) can be easily obtained by (i) appending a task layer to the top of BERT, (ii) loading the pre-trained parameter values as initialization for BERT, (iii) randomly initializing the parameters in the task layer and (iv) training those model parameters, i.e., those both in BERT and the task layer, using a set of labeled samples of the downstream task. As the dataset involved in step (iv) commonly contains no more than tens of thousands of samples, which is quite small compared with the number of model parameters, the training process can often slightly change the parameter values, particularly those in BERT, and hence is called fine-tuning. In short, the training process of a BERT-based model consists of two stages: pre-training and fine-tuning. Subsequently, the trained model could be applied to unseen test samples to predict their labels. The latter process is called inference.

As pre-trained BERTs have been shared online, the resource issues that need to be considered are mainly regarding those consumed during fine-tuning and inference. However, as previously mentioned, existing studies chiefly focus on the problem involved in inference, while ignoring those involved in fine-tuning. Even worse, they usually reduce the resources needed for inference at the cost of much larger investments required for the training process.

3.2. Truncating BERT for Binary Text Classification

It is widely accepted that BERT is over-parameterized [20,21], which motivates us to conduct a series of experiments to empirically study how this would influence BTC tasks that are the focus of this study. As BERT_base and BERT_large have identical structures but different sizes, we focus only on the former and refer to it as BERT from now on for notation simplicity, unless otherwise specified. Further study on BERT_large should be pursued in future work.

As illustrated in Figure 3, a model for BTC can be built via truncating BERT in two steps: The first step is to drop the top-k Transformer layers from the pre-trained BERT and preserve the bottom part as an encoder. The second step is to add a task head to the top of the resulting encoder, composed of a fully connected (FC) layer and a sigmoid layer to construct a binary classifier. In this way, when $k$ ranges from 11 to 0, we obtain 12 different classifiers, which are denoted by $M_1$,…, $M_{12}$, respectively. To train such a classifier $M_m\left(m=1,\dots ,12\right)$, the parameters in different parts are initialized differently: those in the encoder, i.e., the bottom Transformer layers retained, are initialized using their weights learned in pre-training, whereas those in the task head are randomly initialized. After that, $M_m$ is fine-tuned using labeled datasets of downstream BTC tasks.

To investigate the performance of these truncated models, we fine-tune and test them on the datasets of three different BTC tasks:

• ChnSentiCorpHou [48] is a Chinese sentiment classification corpus consisting of hotel comments collected by Tan and Zhang [48]. It consists of 3000 positive and 3000 negative reviews of different hotels. Given a comment, the task is to predict the corresponding sentiment polarity.
• CoLA [49] consists of 10,657 English sentences from published linguistics literature. Given a sentence, the task is to judge the grammatical acceptability of it. Note that, as the test set is unlabeled, we only utilize publicly available training and dev sets in our experiments.
• RTE [22] refers to the Recognizing Textual Entailment dataset, consisting of 2.7 k English sentence pairs. Each pair is labeled with a binary label indicating whether one sentence entails the other or not.

For fairness, different truncated models adopt the same splitting of training (or rather, fine-tuning), validation and test set. The test accuracy each model achieved on these three datasets is given in

Table 1. Test accuracy (%) of $M_1$,…, $M_{12}$ on three BTC datasets. Each row corresponds to a classifier with a truncated BERT with a different number of Transformers as its encoder. Column 2–4 are the classification accuracies these classifiers achieve on the three datasets. Bold denotes the highest accuracy.

Classifiers	ChnSentiCorpHou	CoLA	RTE
$M_1$	88.08	69.07	52.00
$M_2$	91.08	66.22	52.29
$M_3$	91.52	69.64	56.38
$M_4$	92.07	72.87	63.06
$M_5$	92.65	75.52	64.73
$M_6$	92.45	77.80	63.57
$M_7$	93.05	77.99	64.29
$M_8$	92.07	80.83	66.03
$M_9$	93.28	84.25	63.68
$M_{10}$	92.02	85.96	63.71
$M_{11}$	93.02	83.49	64.94
$M_{12}$	93.30	83.87	65.81

, which is evaluated by

$$accuracy=\frac{\#TP+\#TN}{\#TP+\#TN+\#FP+\#FN}$$

(1)

where $\#TP$, $\#TN$, $\#FP$ and $\#FN$ denote the number of true positives, true negatives, false positives and false negatives, respectively.

Note that, as each Transformer has more than 9M parameters, fine-tuning and inference times would dramatically increase as the number of Transformer layers grows. However, based on the results in Table 1, it is evident that layers seem to come with diminishing returns in terms of classification accuracy. As more layers are added, the accuracy fluctuates and then plateaus. More concretely, for CoLA, truncated models with 9 or 10 Transformer layers outperform the full BERT model, i.e., $M_{12}$, and models with 8 and 11 Transformer layers exhibit competitive performance; for RTE, a model containing 8 Transformers has the best test accuracy; and for ChnSentiCorpHou, no truncated model outperforms the full BERT model, but those with 2 to 11 Transformers are competitive. In addition, the optimal numbers of Transformer layers are different for different datasets, as reflected in Table 1 by the fact that the highest accuracy (bold) in different datasets are located in different rows.

These observations indicate that BERT can be compressed for BTC tasks through truncation with very minor loss or even notable gain in terms of classification accuracy, and the number of Transformer layers to be dropped should be determined according to the downstream BTC task, i.e., adaptively.

4. Truncating BERT before Fine-Tuning

Below, we first conduct some theoretical analysis of the feasibility of truncating BERT before fine-tuning, then describe how to use class separability as a criterion to determine truncation position adaptively for downstream BTC tasks, and finally present the proposed compression method.

4.1. Feasibility Analysis

Recall that, in Section 3, we constructed 12 binary classifiers, denoted by $M_1$,… ,$M_{12}$, each of which contains a truncated BERT with $m$ ($=1,\dots ,12$) Transformer layers, followed by an FC layer and a sigmoid layer. More formally, for a given sample (e.g., a text or text-pair) vectorized as $x_n$, the prediction function of $M_m$ is

$$M_m\left(x_n\right)=\sigma \left(g_m\left(h_m\left(x_n\right)\right)\right),$$

(2)

$$h_m\left(x_n\right)=f_m\left(f_{m-1}\left(\dots f_1\left(x_n\right)\dots \right)\right),$$

(3)

where $\sigma$ is the sigmoid function, $g_m\left(\cdot \right)$ denotes the transform function of a fully-connected layer, $h_m\left(\cdot \right)$ is the function of the encoder, and $f_j\left(\cdot \right)$ is the function of the $j$-th ($j=1,\dots ,m$) Transformer layer.

For binary classification tasks, $\sigma \left(g_m\left(\cdot \right)\right)$ in Equation (2) is a generalized linear transform targeted to find a hyperplane best separating examples without transforming the input representations, namely the embedding vectors of the input examples. Such embeddings are produced by the encoder $h_m\left(\cdot \right)$, which plays a more critical role as it determines the space that the separating hyperplane is located in and the feature vectors of the input examples, and hence the degree of how the examples are (linearly) separable. More precisely, the prediction accuracy of $M_m\left(·\right)$, denoted by $ac{c}_m$, on a dataset $\mathbb{S}$ containing $N$ examples vectorized as $x_1,\dots ,x_N$, relies mainly on the linear separability of the embedding set $\left\{h_m\left(x_1\right),\dots ,h_m\left(x_N\right)\right\}$ generated by the encoder $h_m\left(\cdot \right)$. The higher the separability of the embeddings set, the higher the test accuracy $ac{c}_m$ attained by the model $M_m\left(·\right)$.

In addition, for BTC tasks, we observed relatively small changes were brought by the fine-tuning process to the parameters of the encoder $h_m\left(\cdot \right)$ in our experiments, particularly for the deeper layers. We owe this to three main factors: (i) the labeled dataset used for fine-tuning is of a small size, commonly ranging from hundreds to tens of thousands, (ii) the vanishing effect caused by computing the gradients of model parameters by the chain rule and (iii) the often very slow learning rate, e.g., 5 × 10⁻⁵, applied to the pre-trained encoder. Similar phenomena have been previously observed by Merchant et al. [50] on the MNLI dataset [31]. Even the very recent studies conducted by Zhou and Srikumar [51,52] found that fine-tuning largely preserved the spatial structure of the data points, i.e., the vectorized examples.

Inspired by the above analyses and observations, we conjecture that the ordering of the separability of the embedding sets generated by different pre-trained encoders remains consistent during fine-tuning. Thus, the separability of the embedding set produced by $h_m\left(\cdot \right)$ before fine-tuning could be used as a precursor for the ranking of the accuracy that the corresponding classifier $M_m\left(\cdot \right)$ could achieve after being fine-tuned.

4.2. Using Class Separability as a Proxy

We conducted additional experiments to validate the conjecture described above. Before that, three widely used separability indices, viz., class scatter matrices (CSM) [53], Thornton’s separability index (SI) [54] and hypothesis margin (HM) [55], are briefly reviewed in the following section. To simplify notation, the embedding vector generated by $h_m\left(\cdot \right)$ from an input example $x$ is denoted by $z$ from now on, viz., $z=h_m\left(x\right), m=1,\dots ,12$, in which the model subscript is omitted.

CSM is the fraction of the trace of between-cluster scatter matrix ($S_B$) and within-cluster scatter matrix ($S_W$). It could be formulated as follows for two-class datasets:

$$CSM=tr\left(S_B\right)/tr\left(S_W\right),$$

(4)

$$S_B={\sum }_{c=1}^2{N}_c(\overline{{\mathit{z}}_c}-\overline{\mathit{z}}){(\overline{{\mathit{z}}_c}-\overline{\mathit{z}})}^{\mathrm{T}},$$

(5)

$$S_W={\sum }_{c=1}^2{\sum }_{j=1}^{N_c}\left(z_{cj}-\overline{z_c}\right){\left(z_{cj}-\overline{z_c}\right)}^{\mathrm{T}},$$

(6)

where $tr\left(\cdot \right)$ denotes the trace operation, calculating the sum of the diagonal elements of a given matrix; $S_B$ is the between-cluster scatter evaluating the distance between the mean vector of each category and the mean vector of the whole dataset; $S_W$ is the within-cluster scatter measuring the distance between an instance and the mean vector of the class it belongs to; $N_c\left(c=1,2\right)$ is the number of examples in the $c$-th class; $z_{cj}$ is the $j$-th instance in class $c$; and $\overline{z_c}$ and $\overline{z}$ denote the mean vectors of class $c$ and the whole dataset, respectively.

SI measures the inter-class overlap by calculating the ratio of instances that share the same class label with their nearest neighbors. Formally, it is defined to be

$$SI=\frac{{\sum }_{i=1}^N\left(\varphi \left(z_i\right)+\varphi \left(z_i^{\prime }\right)+1\right) \mathrm{mod} 2}{N},$$

(7)

in which $N$ is the total number of instances in the dataset, $z_i$ is the $i$-th instance, and $\varphi \left(z_i\right)\in \left\{0,1\right\}$ and $z_i^{\prime }$ refer to the class label and the nearest neighbor of $z_i$, respectively.

HM evaluates the sum of the distance between the hypothesis and the closest hypothesis that assigns the alternative label to the given instance.

$$HM=\frac{1}{2}{\sum }_{i=1}^N\left(\parallel z_i-\mathrm{nearmiss}\left(z_i\right)\parallel -\parallel z_i-\mathrm{nearhit}\left(z_i\right)\parallel \right),$$

(8)

where $N$ denotes the number of instances in the dataset, $\mathrm{nearmiss}\left(z_i\right)$ and $\mathrm{nearhit}\left(z_i\right)$ are the nearest instances to $z_i$ with the same and different class labels, respectively, and $\parallel \cdot \parallel$ stands for the Euclidean norm.

To evaluate the appropriateness of these three separability indices for use as proxies for truncating BERT before fine-tuning, we computed them on the embedding sets generated by the pre-trained (but not fine-tuned) encoders, $h_1\left(\cdot \right),\dots ,h_{12}\left(\cdot \right)$, from the three BTC datasets described in Section 3.2. Figure 4 shows the results. For the convenience of comparison, we draw the results of one separability index together with the classification accuracy of the corresponding classifiers in each subplot, i.e., $M_1$,…,$M_{12}$, on the same dataset. Intuitively, the trends of CSM, compared with SI and HM, are more similar to that of test accuracy across encoders and datasets. As shown in the first column in Figure 4, CSM has similar test accuracy to that achieved by the corresponding classifiers after fine-tuning. This somewhat validates our conjecture presented at the end of Section 4.1. We further compute the Pearson’s correlation coefficient between classification accuracy and each separability index. The results are given in

Table 2. The Pearson’s correlation coefficients between test accuracy and each of the three separability indices.

Corpus	CSM	SI	HM
ChnSentiCorpHou	0.7942	0.8204	0.4208
CoLA	0.9655	0.5444	0.8269
RTE	0.8720	0.1090	−0.2889
Average	0.8772	0.4913	0.3196

, which quantitatively confirm that CSM most closely correlates with test accuracy, and hence is the best choice among the three indices to be used as a proxy.

4.3. The Compression Method

Now, we describe how to use CSM as a proxy to truncate BERT before fine-tuning, and hence reduce resources to be consumed in both fine-tuning and inference without a significant loss in classification accuracy.

Let ${\mathbb{S}}_{tr}$ be the training set of a BTC task. The maximum and minimum of the CSMs of the embedding sets generated from ${\mathbb{S}}_{tr}$ by encoders with different numbers of Transformer layers are denoted by $CS{M}^{*}$ and $CS{M}_{*}$, respectively. That is,

$$CS{M}^{*}=\underset{m}{\max }\left\{CSM\left(h_m\left({\mathbb{S}}_{tr}\right)\right)\right\},$$

(9)

$$CS{M}_{*}=\underset{m}{\min }\left\{CSM\left(h_m\left({\mathbb{S}}_{tr}\right)\right)\right\},$$

(10)

where $h_m\left({\mathbb{S}}_{tr}\right)$ is the embedding set generated by $h_m$, a pre-trained encoder consisting of $m$ Transformer layers. The encoder indices/subscripts corresponding to $CS{M}^{*}$ and $CS{M}_{*}$ are denoted by $m^{*}$ and $m_{*}$, respectively. As the goal of compression is to shrink the model under consideration as much as possible without significantly diminished accuracy, we define $\rho$($\in \left[0, 1\right]$) to be the maximally tolerable ratio of (relative) accuracy decline, and determine the number of Transformers to be preserved during truncation, denoted by $m^{△}$, as follows

$${\Delta }_{CSM}=CS{M}^{*}-CS{M}_{*},$$

(11)

$$m^{△}=\underset{CSM\left(h_m\left({\mathbb{S}}_{tr}\right)\right)\ge CS{M}^{*}-\rho {\Delta }_{CSM}}{\min }\left\{1,\dots ,m^{*}\right\}.$$

(12)

By definition, $\rho$ should be a small constant. Otherwise, significant decreases in accuracy may occur. In our experiments (see Section 5.3.1), we find $0.20\le \rho \le$ 0.25 to be a good choice that ensures very small accuracy loss while achieving a considerable compression ratio. As an additional advantage, $\rho$ makes the balance between accuracy loss and model size reduction somewhat controllable, and hence provides the possibility to implement compression on-demand.

To summarize, truncating BERT before adaptively fine-tuning for a specified BTC task can be implemented in three steps: (i) Vectorize each example in the training set ${\mathbb{S}}_{tr}$ of the target task as $x_n\left(n=1,\dots ,N\right)$. (ii) Feed each $x_n$ into a pre-trained BERT, and collect all the hidden states, each of which is denoted as $z_n^{\left(m\right)}\left(m=1,\dots ,12\right)$, generated by each Transformer layer into an embedding set $Z^{\left(m\right)}$. Namely, $Z^{\left(m\right)}=h_m\left({\mathbb{S}}_{tr}\right)=\left\{z_1^{\left(m\right)},\dots ,z_N^{\left(m\right)}\right\}$. (iii) Calculate the CSM of each embedding set $Z^{\left(m\right)}$ for $m=1,\dots ,12$, and find the truncation position according to Equation (12), described above.

5. Experiments

To validate the effectiveness of the proposed method, we conduct extensive experiments on eight BTC datasets, including CoLA, ChnSentiCorpHou and RTE, described in Section 3, and five additional ones. In addition, we empirically compare our method to some strong baselines to analyze their relative merits and complementarity.

5.1. Datasets

In the following, we provide a brief introduction to the five additional datasets used in our experiments:

• WaiMai (https://github.com/SophonPlus/ChineseNlpCorpus/tree/master/datasets/waimai_10k (accessed on 8 October 2022)) is a Chinese sentiment corpus consisting of comments of delivery service and takeaway food. It contains about 4000 positive and 8000 negative examples.
• SST-2 refers to the Stanford Sentiment Treebank [56]. It consists of about 70 k English sentences excerpted from movie reviews, each of which is assigned to a positive or negative label.
• Nikon-JD is a Chinese aspect-level sentiment classification dataset constructed by ourselves [57]. It consists of 2.6 k product reviews from an E-commerce platform, namely JingDong (https://www.jd.com (accessed on 8 October 2022)). Given a review and a target aspect category, three human annotators are asked to judge whether the review expresses a positive sentiment toward the specified aspect. Then, a label is assigned according to the judgement of the majority.
• AFQMC stands for the Ant Financial Question Matching Corpus [58], released by the Ant Technology Exploration Conference Developer competition. There are 42.5 k instances in this corpus, each of which is a sentence pair. The task is to judge whether a given pair of sentences are semantically similar.
• MRPC is the Microsoft Research Paraphrase Corpus [59], consisting of English sentence pairs automatically extracted from online news platforms. Each pair is annotated with a label indicating whether the sentences in a pair are semantically equivalent. There are approximately 5.8 k pairs in this corpus.

Table 3. Statistics of the eight datasets used in this article. Pos. Rate indicates the ratio of positive instances in the corresponding dataset.

Corpus	Samples	Language	Input	Task	Pos. Rate
ChnSentiCorpHou	6 k	Chinese	Single text	Sentiment classification	0.50
WaiMai	12 k	Chinese	Single text	Sentiment classification	0.33
SST-2	67.8 k	English	Single text	Sentiment classification	0.50
CoLA	9 k	English	Single text	Grammatic acceptability judgement	0.70
Nikon-JD	2.6 k	Chinese	Text pair	Semantic similarity	0.78
AFQMC	38.6 k	Chinese	Text pair	Semantic similarity	0.31
MRPC	5.8 k	English	Text pair	Semantic similarity	0.67
RTE	2.7 k	English	Text pair	Semantic similarity	0.50

gives the statistics of these datasets, together with those described in Section 3. For better diversity, these datasets involve English and Chinese texts, single text and text-pair inputs, and various typical BTC tasks such as sentiment polarity identification and grammatical acceptability judgement. Note that the ordering of these datasets has been rearranged to place together tasks dealing with identical languages and input formats.

Among these datasets, all samples in ChnSentiCorpHou, WaiMai, SST-2, Nikon-JD and MRPC are available, and hence used in the experiments. In contrast, only the training and dev sets of CoLA, AFQMC and RTE are used, as their test sets have not been opened. For the two larger ones, i.e., CoLA and AFQMC, we take the original dev sets as our new test sets and randomly sample 10% of instances from the original training sets to form the new dev sets while the remaining examples form the new training sets. Then, we fine-tune each classifier on the training and dev sets and test their accuracy on the test sets for WaiMai, SST-2, CoLA, AFQMC and MRPC. We conduct five-fold cross-validations on the remaining three smaller datasets, namely ChnSentiCorpHou, Nikon-JD and RTE, which involves randomly partitioning the examples into five (nearly equal) groups, and then repeating the following process five times: choosing a different fold as the test set with accuracy calculated, and leaving the remaining four folds to form the training and dev set. We use the average test accuracy of five rounds as the result.

5.2. Implementation Details and Comparative Methods

We use 12-layer Chinese BERT-wwm-ext (https://huggingface.co/hfl/chinese-bert-wwm-ext (accessed on 8 October 2022)) and BERT-base-uncased (https://huggingface.co/bert-base-uncased (accessed on 8 October 2022)) as our basic language models for Chinese and English tasks separately. All classification models are fine-tuned for five epochs. The batch size and other implementation details are provided in

Table 4. Parameter settings in our implementations.

Corpus	Batch Size	Learning Rate	Max Seq. Length	Device
ChnSentiCorpHou	32	1 × 10−5	512	RTX3090 × 2
WaiMai	32	1 × 10−5	256	RTX3090 × 2
SST-2	32	2 × 10−5	128	V100 × 1
CoLA	32	2 × 10−5	128	RTX3090 × 2
Nikon-JD	32	1 × 10−5	256	RTX3090 × 2
AFQMC	32	5 × 10−6	256	RTX3090 × 2
MRPC	32	1 × 10−5	128	RTX3090 × 2
RTE	32	1 × 10−5	256	V100 × 1

. Following each epoch of training, we give a test on dev set and record the better model. Finally, the one that performs best on dev set will be saved and applied.

For comparison, we set up the following baselines:

• Full BERT: We construct binary classification models based on BERT_base [4] as the standard model.
• Distil-Pretrain: We build up classification models based on the distilled language models (DistilBERT [12] and ALBERT-Chinese-tiny (https://huggingface.co/clue/albert_chinese_tiny (accessed on 10 October 2022))).
• Distil-task-biLSTM: We fine-tune a BERT-based classification model as the teacher, and construct a student using the biLSTM as the encoder with the assistance of the attention mechanism. Then, we let the student imitate predictions of the teacher.
• Distil-task-kimCNN: The teacher is the same as in the last one, but the student is constructed based on the kimCNN [60].
• Pruning-AGP: We employ the automated gradual pruner (AGP) [61] to implement task-specific magnitude pruning on our tasks. Here, we refer to the implementation provided in the Distiller library [62].

The BERT embeddings, the batch size and max sequence length adopted in all these baselines are the same as those of our models (see Table 4). As an exception, we set the learning rate to be 1 × 10⁻³ during task-specific distillation for both Distil-task-biLSTM and Distil-task-kimCNN. In the following sections, we will compare these methods with ours in terms of predicting accuracy, training time and number of parameters. Note that, to ensure fairness, the above baselines and proposed truncation methods adopt the same splitting of training (or rather, fine-tuning), validation and test set on each dataset.

5.3. Results and Analysis

5.3.1. The Balance between Accuracy Decline and Model Size Reduction

We investigate how the classification accuracy and model size vary along with different $\rho$ (maximally tolerable ratio of relative accuracy decline) to find an appropriate trade-off between them. To check whether there exists some general regularity, we first treat the aforementioned eight BTC tasks as a whole by computing the average accuracy loss and compression ratio on them for different $\rho$. Figure 5 depicts how the average accuracy loss (measured using the mean accuracy loss ratio relative to the full BERT; see the left vertical axis) and average compression ratio achieved (measured using the average number of Transformers dropped during compression; see the right vertical axis) vary when $\rho$ changes from 0 to 1 with a stride of 0.05. It can be observed that both accuracy loss and compression ratio increase when $\rho$ grows larger. This is consistent with our expectations. Notably, when $\rho >0.40$, the mean accuracy loss ratio (see the red curve in Figure 5) starts accelerating going up, whereas the increase in compression ratio (see the blue curve in Figure 5) slows down. Both curves are nearly horizontal in the interval of $\rho \in \left[0.35, 0.40\right]$; for a better trade-off, we would limit $\rho \le 0.35$. If we prefer near to no accuracy loss, we may limit the mean loss ratio to be less than 0.02 (i.e., within 2%). Correspondingly, $\rho$ should be no more than 0.25. Based on the figure, it is obvious that $\rho$ should be limited to $\left[0.20, 0.25\right]$ if we want to obtain a compression ratio as large as possible at the same time. In addition, as the red curve is steeper than the blue one when $\rho \in \left[0.20, 0.25\right]$, we think $\rho =0.20$ is an even better choice, which restricts accuracy loss more rigidly while achieving a close compression ratio.

Figure 6 further depicts how the test accuracy achieved (the left vertical axis in each subplot) and number of Transformer layers retained (the right vertical axis in each subplot) covary with different $\rho$ on each dataset. We highlight the test accuracy and size of compressed model corresponding to $\rho =0.20$ using a vertical dashed line in each subplot. The results show that this $\rho$ setting helps to find a compressed model with the highest accuracy or accuracy very close to the highest on all datasets but AFQMC. This verifies $\rho =0.20$ to be a generally good choice. Moreover, the resulting model is of different complexity, as it contains a different number of Transformers for different BTC datasets. In other words, the number of Transformers to be preserved in the encoder is adaptively determined for the downstream BTC task.

5.3.2. Performance of the Compressed Models

To validate the effectiveness of our proposed method, we comprehensively evaluate the performance of the compressed model it produces when adopting an optimal balance between accuracy loss and compression ratio, namely with a setting of $\rho =0.2$. The performance is measured in different angles including the test accuracy achieved, the number of parameters remained and the training time spent. The results are shown in

Table 5. Test accuracy (%) on the 8 tasks in this article. Bolded values denote the highest among all models (i.e., including full BERT), and underlined values denote the best among compressed models.

Model	ChnSentiCorpHou	WaiMai	SST-2	CoLA	Nikon-JD	AFQMC	MRPC	RTE
Full BERT	93.30	90.12	92.26	83.87	97.01	74.12	83.59	65.81
Distil-Pretrain	91.17	89.82	91.71	81.40	90.72	51.88	82.09	59.56
Distil-task-biLSTM	87.50	89.57	87.10	68.31	89.32	64.18	64.17	50.88
Distil-task-kimCNN	86.48	88.45	85.23	70.59	91.29	63.51	67.59	52.08
Pruning-AGP	92.65	87.11	91.32	82.73	95.23	69.95	77.33	57.50
Truncated-CSM	92.65	90.78	93.03	84.25	97.31	68.95	76.52	66.03

,

Table 6. Number of parameters in the model needs to be stored and updated during training/fine-tuning for the 8 BTC tasks in this study. All values have been scaled by 1M. Bold denotes the smallest values. The quantity of Distil-Pretrain is not available, and thus also not shown in the table. The quantity of a distillation based model counts the parameters in both the teacher and student.

Model	ChnSentiCorpHou	WaiMai	SST-2	CoLA	Nikon-JD	AFQMC	MRPC	RTE
Full BERT	102	102	109	109	102	102	109	109
Distil-Pretrain	-	-	-	-	-	-	-	-
Distil-task-biLSTM	102 + 9	102 + 9	109 + 9	109 + 9	102 + 9	102 + 9	109 + 9	109 + 9
Distil-task-kimCNN	102 + 7	102 + 7	109 + 7	109 + 7	102 + 7	102 + 7	109 + 7	109 + 7
Pruning-AGP	102	102	109	109	102	102	109	109
Truncated-CSM	53	53	95	88	95	53	60	81

and

Table 7. Training time (s) for the 8 BTC tasks in this study. Bold denotes the shortest times. For distillation-based method, it includes fine-tuning time of teacher model and distillation time of student model. For pruning methods, it includes fine-tuning time and weight pruning time, whereas for truncation-based method, it includes calculation time of separability measure and fine-tuning time. Note that the total training time of Distil-Pretrain is not available, and hence not given here.

Model	ChnSentiCorpHou	WaiMai	SST-2	CoLA	Nikon-JD	AFQMC	MRPC	RTE
Full BERT	347	439	3918	334	104	1617	159	177
Distil-Pretrain	-	-	-	-	-	-	-	-
Distil-task-biLSTM	347 + 841	439 + 613	3918 + 5493	334 + 761	104 + 248	1617 + 3933	159 + 371	177 + 284
Distil-task-kimCNN	347 + 738	439 + 570	3918 + 5052	334 + 1312	104 + 455	1617 + 3836	159 + 1113	177 + 261
Pruning-AGP	10,516	5175	21,389	2326	1970	16,643	2467	2004
Truncated-CSM	158	202	3286	268	96	771	83	122

, respectively, in which our compressed model is called Truncated-CSM. For the convenience of comparison, we also provide the results of the baseline models on identical datasets in these tables, using the same devices as those listed in Table 4.

As shown in Table 5, compared with other compressed models, ours achieves the highest accuracy on six out of eight datasets. For the other two datasets, it wins the second best on AFQMC and the third best on MRPC, and performs very closely to Pruning-AGP in that the accuracy difference between them is no more than one point. It is also noteworthy that though our model is often much smaller than the full BERT (see the corresponding columns in Table 6), it outperforms the full BERT on five datasets in terms of test accuracy. This shows that BERT is over-parameterized, as previously found, and justifies our method of compressing it via truncation.

We further compare the quantities of parameters in different compressed models, together with that of the full BERT, which need to be stored and updated during training. Note that, for inference, distillation-based methods only load and use the student, which is often much lighter and runs very fast. However, as shown in Table 6, such model reduction and speed-up are gained at the cost of far more resources invested into training. When looking back at training, the results in Table 6 demonstrate that our compressed model is much smaller than all competing models, including the full BERT, which leads to significantly less training time. This is indicated by the results in Table 7, showing that our model consumes the least training time on all datasets. It is worth highlighting that the training time of our model is only 46~92% of that needed to fine-tune a full BERT. In contrast, the other compression methods, including both distillation and pruning, require far longer training times than the full BERT as they need extra training to learn the pruning mask or transfer knowledge from the teacher to the student, sometimes iteratively.

5.4. Discussion

Overall, the above results show that our method can achieve an averaged compression ratio of about 40% relative to the full BERT while keeping the classification accuracy on a very high level. Specifically, our method achieved the highest accuracy among all models, including the full BERT, on five out of eight datasets, and beat other compressed models on six out of eight datasets while being the runner-up and third winner on the remaining two datasets.

Moreover, our method is simple to implement, requires very little additional cost for compression, only needs to fine-tune a truncated BERT with fewer Transformers, and thus enjoys much shorter training times compared with the full BERT. This is impossible for other conventional compression methods, including distillation and pruning, because they need to transfer knowledge from teacher to student or learn the pruning weights/masks in addition to fine-tuning. This generally leads to significant increases in the time and space consumption of the training process.

Currently, we only concentrate on BTC tasks. Although these tasks can support a long list of practical applications, for greater real-world applicability, they need to be formulated into more complicated tasks, such as multi-class classification and sequence labeling. As the separability indices also work for multi-class classification, we believe the work presented in this study could provide a basis for future studies on compression methods for more complex (downstream) tasks. In addition, we also note in the experiments that distillation methods often use only the learned students, e.g., a biLSTM-based binary classifier, for inference, and hence have shorter inference times compared with other compression methods, including ours. This merit could be exploited in the future through a combined paradigm, in which our method is applied first to find a lighter and more accurate teacher for distillation.

6. Conclusions and Future Work

In this study, we conduct a series of experiments and some theoretic analysis to explore whether there exist winning tickets when transferring language models to downstream BTC tasks and how to conveniently and effectively find them before fine-tuning. Different from conventional model compression methods, we truncate BERT by dropping out several consecutive Transformer layers once. In addition, a simple way to locate an appropriate position for truncation is provided, which uses class separability as a proxy and manages to appropriately balance between compression ratio and accuracy decline. To validate the proposed method, we conduct extensive experiments on eight BTC datasets covering two main languages (Chinese and English), different input formats (single text and text pair) and various tasks (sentiment analysis, acceptability judgement, textual entailment, semantic similarity analysis and natural language inference). The experimental results reveal that, using our compression method, the resources consumed in inference, as well as fine-tuning, could be significantly reduced without losing too much accuracy or adding extra computational burdens or memory footprint. It outperforms some strong baselines, including the full BERT, distillation methods and structured pruning, in terms of classification accuracy, parameter quantity and training time. In addition, the classification accuracy achieved by our method is even higher than that of full BERT on most of the datasets. In contrast, as well-known conventional compression methods cannot reach the accuracy level of full language models. We think this not only verifies the effectiveness and superiority of our method, but also proves that BERT is over-parameterized for certain downstream tasks and should be adaptively shrunken.

In future studies, we aim to extend these experiments to other more complex NLP tasks, e.g., multi-class classification and sequence labeling, and other language models, e.g., BERT_large, and explore more effective ways to find winning tickets in language models. In addition, we intend to find out indicators correlating more closely with classification accuracy than the CSM index used in this article, and hence to further improve the performance of compressed models on some difficult BTC datasets, such as AFQMC and MRPC.