Unknown Health States Recognition with Collective-Decision-Based Deep Learning Networks in Predictive Maintenance Applications

Abstract

At present, decision-making solutions developed based on deep learning (DL) models have received extensive attention in predictive maintenance (PM) applications along with the rapid improvement of computing power. Relying on the superior properties of shared weights and spatial pooling, convolutional neural networks (CNNs) can learn effective representations of health states from industrial data. Many developed CNN-based schemes, such as advanced CNNs that introduce residual learning and multi-scale learning, have shown good performance in health states recognition tasks under the assumption that all the classes are known. However, these schemes have no ability to deal with new abnormal samples that belong to state classes not part of the training set. In this paper, a collective decision framework for different CNNs is proposed. It is based on a one-vs.-rest network (OVRN) to simultaneously achieve classification of known and unknown health states. OVRNs learn class-specific discriminative features and enhance the ability to reject new abnormal samples incorporated to different CNNs. According to the validation results on the public dataset of the Tennessee Eastman process (TEP), the proposed CNN-based decision schemes incorporating an OVRN have outstanding recognition ability for samples of unknown heath states while maintaining satisfactory accuracy on known states. The results show that the new DL framework outperforms state-of-the-art CNNs, and the one based on residual and multi-scale learning has the best overall performance.

1. Introduction

Modern industry has achieved a high degree of automation, and it is very important to ensure the long-term operation of industrial processes and the stability of product quality. However, many plant accidents have been reported despite being equipped with advanced process control systems, resulting in serious consequences, including casualties, economic losses, and environmental damage [1]. This is because the early detection of abnormal operating conditions and the execution of reliable decisions mainly rely on experienced operators, who are not always effective in preventing the occurrence and further deterioration of accidents [2].

In industrial systems, predictive maintenance is critical in ensuring the safe and reliable operation of production processes [3]. Effective predictive maintenance systems provide early warning of the operating status of the system and equipment, which can avoid unexpected shutdown or even casualties due to various health states. In recent years, methods based on machine learning and statistical analysis have been developed and improved, enabling valuable information extraction and health state monitoring performance enhancement from massive process data. Principal component analysis (PCA) [4,5], partial least squares (PLS) [6,7], canonical correlation analysis (CCA) [8], and Fisher discriminant analysis (FDA) [9,10] have become widely employed feature extractors for obtaining linear feature representations [11]. As nonlinear feature extractors, kernel PCA [12], kernel FDA [13], kernel PLS [14], support vector machines (SVMs) [15], and artificial neural networks (ANNs) [16] overcome the limitation of application in nonlinear systems. In addition, dynamic PCA [17], dynamic PLS [18], independent component analysis (ICA) [19], and dynamic ICA [20] have been used to describe other complex data characteristics, such as dynamic characteristics and non-Gaussian distribution assumptions. The above traditional data-driven methods can be considered as shallow learning models, and have achieved good performance in many health state monitoring tasks. However, these shallow learning models may not be able to fully learn the complex mapping relationship between massive data and health states, resulting in an insufficient generalization performance for big data [21].

As one of the important tasks of predictive maintenance, heath states recognition aims to achieve fast and efficient discrimination between the system’s faulty states. In the field of industrial system states recognition, deep learning (DL) models have received attention as an important branch of machine learning. Compared with shallow learning models, DL models perform well in learning more abstract and generalized latent representations under the influence of multi-layer nonlinear mappings. Also, DL-model-based recognition methods tend toward end-to-end learning; that is, integrating feature learning and recognition tasks as one training subject, which is more helpful for extracting state-related discriminative features. Convolutional neural networks (CNNs), stacked auto-encoders (SAEs), deep belief networks (DBNs), and recurrent neural networks (RNNs) are the main families that have been widely employed and discussed in published research [22,23,24,25,26,27,28,29,30]. Relying on the superior properties of shared weights and spatial pooling, CNNs can learn effective representations of health states from industrial data, and have been developed and improved in many publications. In industrial processes, CNNs have been validated to have excellent recognition performance compared to other DL models [31,32]. Considering the importance of limited labeled samples and sufficient unlabeled samples, Li et al. proposed a semi-supervised convolutional ladder network, which achieves the goal of extracting latent manifold features and enhancing incipient fault representations [33]. In some scenarios, the mismatch of training and test sets also brings difficulties to the application of CNNs, which has been overcome and discussed in some studies. Li et al. integrated a CNN and domain adaptation techniques, including maximum mean discrepancy (MMD) and domain adversarial learning, reducing the mismatch between computer simulation and physical processes [34]. Wu and Zhao proposed a recognition model for multimode processes, which is based on joint MMD by aligning the joint distribution of multiple domain-specific layers [35]. Aiming to update the recognition model to include both existing heath states and new abnormal samples, Yu and Zhao proposed a broad CNN with incremental learning capability to avoid retraining processes [36].

Most of the health states recognition methods based on DL models are based on the closed set assumption, in which all states are known a priori. Under this assumption, the features extracted by DL models from the training and test sets share the same class characterization [37]. The health states recognition frameworks based on deep learning can achieve high accuracy with the support of complete datasets with guaranteed quantity and heath state diversity, but the acquisition cost is high. In practice, infrequent health states and expensive labeling costs make it difficult to identify and label all the states. In this scenario, new samples from unknown states will be misclassified into existing states. Therefore, it is necessary to consider samples of unknown states, which can be regarded as an important element for intelligent self-learning systems [38]. In industrial process health states recognition, several publications have suggested schemes to meet the needs of classifying known states while detecting unknown samples. Atoui et al. developed a single conditional Gaussian network with a modified distance criterion that allows for the simultaneous processing of known and unknown health states [39]. Considering the dynamic characteristics of industrial processes, Lou et al. further proposed an enhanced fault recognition method, which learns by extracting time-independent components instead of original data [40]. To overcome the limitation that common discriminative methods cannot identify unknown states, a modified discriminant rule is proposed to provide new statistical spaces for quadratic discriminant analysis (QDA), Fisher discriminant analysis (FDA), exponential discriminant analysis (EDA), orthonormal discriminant vector (ODV), and kernel Fisher discriminant (KFD) to ensure that each state is statistically isolated from other states [41].

Most of the research on unknown health states recognition has focused on shallow learning methods. At present, almost all DL-based methods take the recognition of known states as the classification target. These DL methods set generalized boundaries to divide the closed decision space into subspaces for known states. They are effective under the assumption that all the new samples in the testing phase are part of the training set. But, they may fail in practice. Indeed, these closed set approaches may assign new samples from an unknown state to one of the states part of the training set.

To overcome the above limitations, this paper proposes a new collective-decision-based framework for DL networks that introduces a one-vs.-rest network (OVRN) as a classifier for CNN-based feature extractors. CNNs are responsible for extracting deep representations from input measurements, while the OVRN further learns class-specific discriminative features and enhances the ability to reject samples of unknown health states. Furthermore, the OVRN with a collective decision rule provides final recognition results through decision boundaries built for known health states.

Experiments were conducted on the public dataset of the Tennessee Eastman process (TEP), and the results show that the proposed CNN-based collective decision schemes have satisfactory accuracy in both known and unknown health states recognition. The main contributions of this work can be summarized as:

• A new framework combining CNN-based feature extractors and an OVRN classifier is constructed, allowing deep learning methods like CNNs and their variants to process samples from known and unknown health states simultaneously.
• The features extracted by the OVRN and the boundaries established by the collective decision rule are beneficial to the rejection of unknown faults and the preservation of the recognition performance of known states.
• The public dataset from the Tennessee Eastman process (TEP) is used to verify the accuracy and effectiveness of the proposed methods.

The remainder of this paper is organized as follows. In Section 2, relevant preliminaries on CNN-based models are briefly introduced. In Section 3, the details of proposed deep learning models are presented, including the construction of proposed network structures and the implementation of the recognition procedures. Section 4 demonstrates proposed methods on the TEP dataset and presents the application results. Finally, the conclusions are given in Section 5.

2. Convolutional Neural Network

Aiming at realizing the known health states recognition of systems, CNN-based DL methods have been widely developed. In the existing publications, residual learning and multi-scale learning have become the main strategies for improving the feature extraction ability of CNNs.

2.1. Standard Learning

CNNs have shown outstanding performance in feature extraction and health states recognition tasks, relying on the unique characteristics of local receptive fields, spatial sub-sampling, and weight sharing. Generally, the basic structure of a typical CNN includes an input layer, several alternately stacked convolutional and pooling layers, fully connected layers, and an output layer [42,43,44]. The task of the CNN as a feature extractor is accomplished by the forward propagation process.

In forward propagation, the entire network model extracts spatial information from the preprocessed input. As an important part of this process, convolutional layers utilize a series of convolutional kernels as filters to generate feature maps after sliding on the whole input neurons at the previous layer [45]. The pooling layer performs scaled mapping immediately after each convolutional layer to reduce data dimensions and network parameters. Max-pooling, average-pooling, and stochastic-pooling are common methods for generating feature maps with low resolution [46]. After some stacked convolutional layers and pooling layers, the fully connected layer integrates the discriminative features from the previous layer into different classes.

2.2. Residual Learning

In general, CNNs with deeper network structures are capable of extracting complex features, which can improve the performance of health states recognition. However, related studies have shown that when the number of network layers increases, the phenomenon of accuracy saturation and model performance degradation occurs, leading to higher training errors [47]. Residual learning is proposed as a solution to the problem that deeper networks cannot be easily trained [48]. The basic concept of residual learning is that it is easier for a network to learn its residuals than it is to use several network layers to fit a hidden nonlinear map [49].

Residual learning is achieved by constructing residual blocks, which typically consist of several convolutional layers, batch normalizations (BNs), ReLU activation functions, and one identity shortcut [50]. CNNs incorporating residual learning are characterized by easy optimization, and the classification accuracy can be improved by increasing the depth of the network structure.

2.3. Multi-Scale Learning

Due to the fixed kernel scale in each convolutional layer, the multi-scale feature extraction capability of CNNs is limited. It is known that the small-size convolution kernel focuses on the details of the signal while the large-size convolution kernel is responsible for capturing the characteristics that reflect the overall trend of the distribution. The information contained in the complementary features from different scales can help to better express health state information and improve the discrimination ability. The multi-scale learning strategy was developed to capture local features at different scales from the input measurement signals [51].

The multi-scale convolutional architectures can be realized by feature extraction in parallel through multi-branch CNNs with different convolution kernel sizes. CNNs with multi-scale learning can provide feature representations that contain complementary and rich state information compared with the single-scale model.

2.4. CNN’s General Framework

The softmax layer is usually designed in the output layer as a classifier after CNN-based feature extractors to calculate the probability distribution of samples belonging to different classes. Consider a training data matrix $z_{tr}^{\left(i\right)}\in R^{w\times m}$, and corresponding training label $y_{tr}^{\left(i\right)}$, where w and m, respectively, represent the number of samples and variables in a training data matrix. The value of $y_{tr}^{\left(i\right)}$ is between 1 and K, where K is the number of known health states. The softmax layer outputs the conditional probability for class k:

$$\begin{array}{c}\hfill P({\widehat{y}}^{\left(k\right)}|l^{\left(1\right)},\dots ,l^{\left(K\right)})=\frac{\exp \left(l^{\left(k\right)}\right)}{{\sum }_{k=1}^K\exp \left(l^{\left(k\right)}\right)},\end{array}$$

(1)

where the sum of the probabilities of all classes for sample $z_{tr}^{\left(i\right)}$ is 1. A CNN-based network with a softmax classifier is trained to increase the value $l^{\left(k\right)}$ of the target class k relative to the other classes. The samples will be considered to belong to the predicted class with a relatively large probability.

For unknown health states recognition tasks, the mechanism of the general CNN-based framework with softmax is shown in the upper part of Figure 1. Since the softmax function is designed to measure the relative likelihood of a known class compared to other known classes, unknown samples are easily assigned to the most similar known class with the maximum confidence score. Therefore, the unknown sample in Figure 1 is likely to be classified into the pentagram class that is closer than other classes, which is unfavorable for unknown states recognition.

3. The Proposed DL-Based Framework to Deal with Unknown States

Based on the advantages of the OVRN with the learning ability of state specific discrimination features, a new DL-based framework incorporating an OVRN for CNNs is proposed to be applied to industrial process health states recognition.

3.1. The New OVRN-Based Framework

The recognition framework that introduces the one-vs.-rest network (see the lower part of Figure 1) gives a new solution to assign samples of unknown health states. For each known state, the OVRN classifier is independently trained to learn specific features and provide a binary response, which indicates whether the sample belongs to this known state. Due to multiple low posterior probabilities of sigmoid layer outputs, the unknown sample may be indicated as not belonging to any known state.

The proposed OVRN-based framework consists of a CNN-based feature extractor and an OVRN classifier, as shown in Figure 2. In this work, a conventional CNN, residual CNN (RCNN), and multi-scale residual CNN (MRCNN) are employed as feature extractors.

Let $\mathcal{F}$ be the feature extractor, which can be one of the traditional CNN, RCNN, and MRCNN, and let ${\mathcal{G}}_1,{\mathcal{G}}_2,\dots ,{\theta }_{{\mathcal{G}}_K}$ be K parallel branch networks of the OVRN. The OVRN can better learn the discrimination information for target classes, and the output probability of the k-th sigmoid layer ${\widehat{y}}^{\left(k\right)}$ for class k can be expressed as:

$$\begin{array}{c}\hfill P({\widehat{y}}^{\left(k\right)}|z_{tr}^{\left(i\right)})={\mathcal{G}}_k\left(\mathcal{F}\left(z_{tr}^{\left(i\right)}\right)\right).\end{array}$$

(2)

Since the sigmoid function is applied to node $l^{\left(k\right)}$ in the last output layer, the relationship between the probability $P({\widehat{y}}^{\left(k\right)}|z_{tr}^{\left(i\right)})$ in (2) and the output of node $l^{\left(k\right)}$ can be expressed as:

$$\begin{array}{c}\hfill P({\widehat{y}}^{\left(k\right)}|l^{\left(k\right)})=\frac{1}{1+\exp (-l^{\left(k\right)})}.\end{array}$$

(3)

Obviously, each output layer with a sigmoid activation function provides an independent decision to distinguish whether samples belong to this class or not. Therefore, the OVRN using sigmoid layers can provide more supporting information for unknown health states recognition compared with the traditional softmax layer.

In the recognition framework using the OVRN as the classifier, the loss function for model training is expressed as the sum of K binary cross-entropy losses as follows [52]:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{tr}=& -\frac{1}{N}\sum _{i=1}^N\sum _{k=1}^K(\mathbb{I}(y_{tr}^{\left(i\right)}={\widehat{y}}^{\left(k\right)})P({\widehat{y}}^{\left(k\right)}|z_{tr}^{\left(i\right)})\hfill \\ \hfill & +\mathbb{I}(y_{tr}^{\left(i\right)}\ne {\widehat{y}}^{\left(k\right)})(1-P({\widehat{y}}^{\left(k\right)}|z_{tr}^{\left(i\right)})),\hfill \end{array}$$

(4)

where N is the batch size and $\mathbb{I}$ represents the indicator function. During training, this loss function allows each branch network of the OVRN to learn relevant feature representations that contribute to its binary recognition task. The parameter set $\theta =\{{\theta }_{\mathcal{F}},{\theta }_{{\mathcal{G}}_1},{\theta }_{{\mathcal{G}}_2},\dots ,{\theta }_{{\mathcal{G}}_K}\}$ can be updated through the gradient descent algorithm to minimize the loss as the optimization object. ${\theta }_{\mathcal{F}},{\theta }_{{\mathcal{G}}_1},{\theta }_{{\mathcal{G}}_2},\dots ,{\theta }_{{\mathcal{G}}_K}$ are the required parameters of the feature extractor and the OVRN branch networks.

3.2. Recognition Rule Based on Collective Decisions

The collective decision score for state k is computed, given the sample matrix $z_{tr}^{\left(i\right)}$, based on the logit of OVRN outputs $P({\widehat{y}}^{\left(1\right)}|z_{tr}^{\left(i\right)}),P({\widehat{y}}^{\left(2\right)}|z_{tr}^{\left(i\right)}),\dots ,P({\widehat{y}}^{\left(K\right)}|z_{tr}^{\left(i\right)})$, and is expressed as [52]:

$$\begin{array}{cc}\hfill S_{cd}^{i,k}=& \log \left(P({\widehat{y}}^{\left(k\right)}|z_{tr}^{\left(i\right)})\right)\hfill \\ \hfill & -\frac{1}{K-1}\sum _{j\ne k}\log \left(P({\widehat{y}}^{\left(j\right)}|z_{tr}^{\left(i\right)})\right).\hfill \end{array}$$

(5)

Therefore, samples belonging to one of the known states receive higher scores for the target state and lower scores for other states.

Since K collective decision scores are obtained for the sample matrix $z_{tr}^{\left(i\right)}$, a collective decision rule is proposed. Considering samples from unknown health states, the rule is expressed as:

$$\begin{array}{c}{z}_{tr}^{\left(i\right)}\in \mathrm{State}{k}^{*},\mathrm{if}{S}_{cd}^{i,k^{*}}\ge {\epsilon }_{k^{*}},\\ \mathrm{subject}\mathrm{to}{k}^{*}=\underset{k=1,\dots ,K}{argmax}{S}_{cd}^{i,k},\end{array}$$

(6)

where ${\epsilon }_k$ is the collective decision score threshold for health states k, and it can be obtained by ensuring that 95% of the training samples from class k can be diagnosed as the target class based on their collective decision scores. Under this rule, samples are recognized as belonging to the known state with the largest collective decision score value above the corresponding threshold. Otherwise, the samples are assigned to the unknown health states.

3.3. Outline of the Proposed Recognition Strategy

Conventional CNN-based models applied to predictive maintenance can efficiently perform recognition tasks under the assumption that labeled samples are sufficient. However, these methods fail to assign samples from unknown health states that did not participate in the training process. This work allows CNN-based models to overcome their limitations in applications where the classes of test samples are unknown. The proposed recognition framework has been applied to three CNN-based models, including a conventional CNN, RCNN, and MRCNN. These three improved CNN-based models for unknown health states recognition are defined as the CNN-OVRN, RCNN-OVRN, and MRCNN-OVRN. Compared, respectively, to the CNN, RCNN, and MRCNN, the proposed methods can simultaneously achieve classification of known and unknown health states. This good performance benefits from the enhanced ability of the OVRN to reject unknown samples and the complex decision boundaries established by the collective decision rule. In addition, the classification performance of the proposed framework to recognize known health states can also be guaranteed, as shown, for instance, below by the CNN-OVRN, RCNN-OVRN, and MRCNN-OVRN.

The training and recognition processes of the proposed framework are shown in Algorithm 1. The whole procedure can be divided into two stages, offline stage and online stage, which can be summarized as follows.

• Offline stage:

  -

  Data from the training set are pre-processed. The samples in the adjacent w sampling windows are formed into a sample matrix with the size of $w\times m$ and then normalized to zero mean and one deviation.

  -

  The model is trained to obtain the optimal parameters. The training process stops when the loss ${\mathcal{L}}_{tr}$ of the training set tends to converge.

  -

  The collective decision rule is trained. The collective decision scores of training sample matrices are first calculated. The threshold ${\epsilon }_k$ is then set by ensuring that 95% of the fault k sample matrices are assigned to the target health state.

• Online stage:

  -

  The test data are preprocessed using the same sampling window and normalization parameters as the training set.

  -

  The output values of the model are calculated, and the decision scores corresponding to all health states are further obtained.

  -

  The health state corresponding to the test data is predicted according to the collective decision rule.

Algorithm 1 Unknown Health States Recognition with Collective-Decision-Based Deep Learning Networks

Input: training data set $X_{tr}=\{x_{tr}^{\left(1\right)},x_{tr}^{\left(2\right)},\dots ,x_{tr}^{n_{tr}}\}$, $X_{tr}\in R^{n_{tr}\times m}$, and training label set,   $Y_{tr}\in R^{n_{tr}}$; validation data set $X_{val}=\{x_{val}^{\left(1\right)},x_{val}^{\left(2\right)},\dots ,x_{val}^{n_{val}}\}$, $X_{val}\in R^{n_{val}\times m}$, and validation   label set $Y_{val}\in R^{n_{val}}$; testing data set $X_{te}=\{x_{te}^{\left(1\right)},x_{te}^{\left(2\right)},\dots ,x_{te}^{n_{te}}\}$, $X_{te}\in R^{n_{te}\times m}$; Output: predicted healthy state  $Z_{tr}\in R^{(n_{tr}-w+1)\times m},Z_{val}\in R^{(n_{val}-w+1)\times m},Z_{te}\in R^{(n_{te}-w+1)\times m}$← data pre-processing  ▹ Training a CNN-based model  $\theta =\{{\theta }_{\mathcal{F}},{\theta }_{{\mathcal{G}}_1},{\theta }_{{\mathcal{G}}_2},\dots ,{\theta }_{{\mathcal{G}}_K}\}$← initialize parameters  repeat      ${\mathcal{L}}_{tr}=-\frac{1}{N}{\sum }_{i=1}^N{\sum }_{k=1}^K(\mathbb{I}(y_{tr}^{\left(i\right)}={\widehat{y}}^{\left(k\right)})P({\widehat{y}}^{\left(k\right)}|z_{tr}^{\left(i\right)})+\mathbb{I}(y_{tr}^{\left(i\right)}\ne {\widehat{y}}^{\left(k\right)})(1-P({\widehat{y}}^{\left(k\right)}|z_{tr}^{\left(i\right)}))$      $\theta =\{{\theta }_{\mathcal{F}},{\theta }_{{\mathcal{G}}_1},{\theta }_{{\mathcal{G}}_2},\dots ,{\theta }_{{\mathcal{G}}_K}\}$← updata parameters using gradients of $\mathcal{L}$      ${\mathcal{L}}_{val}$← to avoid overfitting  until the convergence of parameters $\theta$  ▹ Training the collective decision rule  for $\mathit{i}\mathbf{=}\mathbf{1}$ to ${\mathit{n}}_{\mathit{tr}}$ do       $S_{cd}^{i,k}=\log \left(P({\widehat{y}}^{\left(k\right)}|z_{tr}^{\left(i\right)})\right)-\frac{1}{K-1}{\sum }_{j\ne k}\log \left(P({\widehat{y}}^{\left(j\right)}|z_{tr}^{\left(i\right)})\right)$← decision score  end for  ${\epsilon }_1,{\epsilon }_2,\dots ,{\epsilon }_K$← collective decision score thresholds for Fault 1, Fault 2, ⋯, Fault K  ▹ Online recognition procedure  for $\mathit{i}\mathbf{=}\mathbf{1}$ to ${\mathit{n}}_{\mathit{te}}$ do       $P({\widehat{y}}^{\left(k\right)}|z_{te}^{\left(i\right)}),k=1,\dots ,k$← the outputs of OVRN       $S_{cd}^{i,k},k=1,\dots ,k$← collective decision scores of all health states       $k^{*}=\underset{k=1,\dots ,K}{argmax}{S}_{cd}^{i,k}$← Fault $k^{*}$ with the largest collective decision scores       if $S_{cd}^{i,k^{*}}\ge {\epsilon }_{k^{*}}$ then           $z_{te}^{\left(i\right)}$ is classified as state $k^{*}$       else           $z_{te}^{\left(i\right)}$ is recognized as unknown health states       end if   end for

4. Application on Tennessee Eastman Process

4.1. Experimental Settings

The Tennessee Eastman process (TEP) is a simulation program developed from an actual chemical plant that has been widely employed in the verification of various health states recognition methods [53]. The TEP consists of five major units: reactor, condenser, compressor, separator, and stripper, as shown in Figure 3 [54]. The whole process can be described as: four gaseous reactants A, C, D, E and a small amount of inert B are sent into the reactor to form liquid products F, G, and H.

The additional TEP simulation datasets for anomaly evaluation were published online by Rieth et al. [55] and are publicly available and can be downloaded at website https://0-doi-org.brum.beds.ac.uk/10.7910/DVN/6C3JR1 (accessed on 1 April 2022). Both the training and test sets provide data of 500 random simulation runs for 20 health state types. For the training and test sets, each simulation run lasts 25 h and 48 h with a sampling time of 3 min, generating 500 and 960 samples. A total of 52 variables, including 41 measured variables and 11 manipulated variables, are employed for health states recognition. From the training dataset, 15 simulation runs are used for training. One simulation run from the test set is used as test data, which include 15 known and 5 unknown health states. After the samples for training, validation, and testing are determined, the adjacent 20 samples (1 h) are formed into a sample matrix with a size of $20\times 52$. To realize the validation of the proposed method in known and unknown health states recognition tasks, the required data are set as shown in

Table 1. TEP datasets.

	Health State	Number of Samples	Number of Sample Matrices
Training	IDV 1-18 (except for IDV 3, 9, 15)	500 × 15 × 15	481 × 15 × 15
Testing (known)	IDV 1-18 (except for IDV 3, 9, 15)	960 × 15 × 1	941 × 15 × 1
Testing (Unknown)	IDV 3, 9, 15, 19, 20	960 × 5 × 1	941 × 5 × 1

.

A new deep learning framework based on CNN-based feature extractors and an OVRN classifier is proposed for known and unknown health states recognition. This work allows CNN-based models such as conventional CNNs, RCNNs, and MRCNNs to overcome limitations when faced with new observations that are not part of the training set. The network structures adopted for the CNN-OVRN, RCNN-OVRN, and MRCNN-OVRN are shown in Figure 2. All deep learning models in experiments were trained on an NVIDIA GeForce RTX 2080 Ti GPU and an Adam optimizer with a learning rate of 0.0005 and 50 epochs. These models are implemented based on an open-source deep learning framework named Apache MXNet.

4.2. Methods Comparison

When the proposed CNN-OVRN, RCNN-OVRN, and MRCNN-OVRN perform online recognition, the samples are labeled as one of the known health states or the unknown health state (UHS). Therefore, for a comprehensive comparison with other related models, this work evaluates the recognition performance of the proposed frameworks on both known and unknown states. The quantitative metrics used for evaluation mainly include: recognition accuracy of known states, recognition accuracy of unknown states, and overall recognition accuracy.

To verify the recognition effectiveness of the proposed methods, statistical-analysis-based models and CNN-based baseline networks from the related literature are compared according to the above evaluation metrics.

• Statistical-analysis-based models: discriminant analysis methods show inherent superiority in health states recognition tasks based on the assumption that the training dataset is available and complete. The existing literature has proposed improved discriminant rules for common discriminant analysis methods such as quadratic discriminant analysis (QDA), Fisher discriminant analysis (FDA), and kernel Fisher discriminant (KFD) analysis [41]. The improved discriminant analysis methods are able to handle observations belonging to new health states, named UQDA, UFDA, and UKFD.
• CNN-based baseline networks: the proposed CNN-OVRN, RCNN-OVRN, and MRCNN-OVRN are first completed for fault recognition on the test dataset. The MCNN-OVRN is constructed to verify the mitigation of residual learning for performance degradation, where the multi-scale RCNN is replaced by a multi-scale CNN with the same network structure. To illustrate the contribution of the collective decision rule in the proposed methods, deep learning models of MRCNN-Softmax and the MRCNN-OVRN without collective decision rules are used for comparison. MRCNN-Softmax employs the softmax layer instead of the OVRN classifier for decision making. Both models assign samples to known states with the largest output value greater than a pre-set threshold; otherwise, they are considered to be unknown faults. Here, the threshold is set to 0.5, which is a common value in unknown states recognition [52].

4.3. Health States Recognition Result and Performance Comparison

Considering the test data in Table 1, the health states recognition accuracies of proposed methods and related comparative models for fifteen known and unknown states are shown in

Table 2. The comparison of health states recognition results.

Item	(a)	(b)	(c)	(d)	(e)	(f)	(g)	(h)	(i)	(j)	(k)
IDV 1	0.8	0.8094	0.8	0.8087	0.8232	0.8200	0.8232	0.8165	0.8393	0.8413	0.8253
IDV 2	0.8125	0.8167	0.8115	0.8082	0.8272	0.8196	0.8183	0.8231	0.8463	0.8400	0.8244
IDV 4	0.8	0.8135	0.8063	0.7991	0.7985	0.8032	0.8094	0.7835	0.8346	0.8321	0.8083
IDV 5	0.7167	0.8135	0.7740	0.8009	0.7902	0.8043	0.7815	0.7932	0.8380	0.8394	0.8043
IDV 6	0.6208	0.4781	0.3854	0.8234	0.8298	0.8395	0.8402	0.8257	0.8474	0.8478	0.8356
IDV 7	0.8208	0.8146	0.7865	0.7829	0.7756	0.7926	0.7836	0.7865	0.8476	0.8493	0.7909
IDV 8	0.7083	0.7208	0.5896	0.7122	0.7256	0.7046	0.7046	0.7301	0.7868	0.8046	0.7387
IDV 10	0.65	0.8354	0.7865	0.7538	0.7547	0.7411	0.7352	0.7595	0.7879	0.7931	0.7498
IDV 11	0.4740	0.8208	0.675	0.7825	0.7953	0.7872	0.7870	0.7826	0.8295	0.8336	0.7988
IDV 12	0.6792	0.7906	0.6167	0.8045	0.8136	0.8081	0.8102	0.8099	0.8399	0.8360	0.8104
IDV 13	0.6552	0.7583	0.4479	0.7758	0.8049	0.8102	0.8091	0.7805	0.7613	0.8064	0.8130
IDV 14	0.6865	0.8198	0.8052	0.8062	0.8149	0.8172	0.8074	0.8081	0.8457	0.8459	0.8157
IDV 16	0.6552	0.7521	0.7313	0.8266	0.8544	0.8461	0.8308	0.8471	0.9245	0.9078	0.8168
IDV 17	0.5042	0.6448	0.5010	0.7763	0.7751	0.7743	0.7749	0.7761	0.7751	0.7747	0.7730
IDV 18	0.2698	0.2052	0.1552	0.7937	0.7962	0.7953	0.7974	0.7963	0.7951	0.7974	0.7960
UFC	0.0902	0.1643	0.1075	0.6163	0.6556	0.6997	0.6786	0.6241	0.2965	0.3180	0.7194

(a): UFDA; (b): UQDA; (c): UKFD; (d): CNN-OVRN with kernel size of $5\times 5$; (e): RCNN-OVRN with kernel size of $3\times 3$; (f): RCNN-OVRN with kernel size of $5\times 5$; (g): RCNN-OVRN with kernel size of $7\times 7$; (h): MCNN-OVRN; (i): MRCNN-Softmax; (j): MRCNN-OVRN without the collective decision rule; (k): MRCNN-OVRN.

. For each experimental result, the accuracies from ten independent runs of the deep-learning-based models were averaged to avoid random interference. It can be observed that UFDA, UQDA, and UKFD cannot achieve good recognition results for some health states such as states 6, 16, 17, and 18. Also, the recognition accuracy of these three methods for samples of unknown states is quite low compared to deep-learning-based models. The reasons why discriminant analysis methods perform poorly on both known and unknown recognition tasks can be explained from two perspectives. One is that the limited learning ability of these shallow learning-based methods makes it impossible to guarantee that the extracted features can best represent the discriminative information of health states. Second, the complex mapping relationship between input data and health states may be difficult to be fully learned due to the independent execution of feature extraction and health states recognition.

The results in Table 2 show that the proposed methods have superior recognition performance for both known and new health states. On the one hand, the proposed methods achieve satisfactory single-recognition accuracy relying on strong nonlinear feature extraction capabilities and an end-to-end recognition framework. On the other hand, observations belonging to new health states have been well identified by the proposed methods, as demonstrated by the fact that the recognition accuracies of the CNN-OVRN, RCNN-OVRN, and MRCNN-OVRN for unknown states are all maintained above 0.6. Due to the advantages of residual learning and multi-scale learning, the MRCNN-OVRN perform best, with a recognition accuracy of 0.7194. As a comparison method, the recognition accuracy of the MCNN-OVRN for unknown states is 0.6241 lower than that of the MRCNN-OVRN with the same network structure except that there is no residual learning. It can be explained that the degradation problem caused by the deep network structure may limit the feature learning ability so that the distinguishability between known and unknown states is slightly poor. The results of MRCNN-Softmax and the MRCNN-OVRN without the collective decision rule are to demonstrate that the decision rule of the proposed method is capable of providing robust recognition results. These two methods slightly improve the recognition accuracy for known states due to the loose decision boundaries constructed; however, the recognition accuracy for unknown states is only around 0.3.

Figure 4 provides the recognition accuracy of all methods on the same test set, including the average recognition accuracy of known states, the recognition accuracy of unknown states, and the overall accuracy. The recognition results of unknown states prove that the online recognition performance of the proposed CNN-OVRN, RCNN-OVRN, and MRCNN-OVRN is superior. Although there is a small reduction in the accuracy of known states compared to models such as the MCNN-OVRN and MRCNN-Softmax that do not employ collective decision rules, the recognition performance of the proposed method is within an acceptable range. The overall recognition accuracies of the CNN-OVRN, RCNN-OVRN, and MRCNN-OVRN are all higher than 0.74, and the MRCNN-OVRN benefits from multi-scale residual learning to improve the accuracy to 0.7801.

Furthermore, the effectiveness of the proposed methods is demonstrated by comparing the distribution histograms of decision scores for known and unknown health states, as shown in Figure 5. Greater separation between the histograms of known states (red part) and unknown states (blue part) indicates a higher recognition ability for unknown states. The proposed CNN-OVRN, RCNN-OVRN, and MRCNN-OVRN can obtain clear separation results between samples of known and unknown states with small overlap, which benefits from the the collaboration of CNN-based feature extractors and the OVRN classifer. The MRCNN-OVRN provides the best separation histogram, which is reflected in the fact that the decision scores of samples from an unknown state are concentrated around small values. Compared with the MRCNN-OVRN, the MCNN-OVRN without residual learning has a larger overlapping area of the distribution of decision scores due to its limited ability to learn deep features. For MRCNN-Softmax and the MRCNN-OVRN without the collective decision rule, the maximum posterior probability output by the softmax layer and sigmoid layer is taken as the decision score, respectively. The softmax layer, which provides high confidence scores for unknown samples, results in a poor ability to distinguish between known and unknown states by identifying the state with the highest relative likelihood among all known states. The MRCNN-OVRN without the collective decision rule provides slightly lower decision scores than MRCNN-Softmax since each node of the output layer provides an independent decision to distinguish whether a sample belongs to this state or not.

For one of ten experimental runs, Figure 6 shows the confusion matrix for test samples of 15 known health states and the unknown state, where the rows and columns represent true and predicted states. In this confusion matrix, one can clearly see how all the test data are assigned by the MRCNN-OVRN. It can be observed that the MRCNN-OVRN can achieve the recognition of most unknown samples while maintaining satisfactory recognition performance for known states. This is mainly due to the multi-scale feature learning ability of the MRCNN and the complex decision boundary established by the OVRN classifier. It can also be found that a small number of samples of known states are misassigned to states 8, 10, 13, and 16. In addition, there are also samples that belong to known health states that are considered unknown. To enhance the discriminative performance for unknown states, the proposed method establishes a strict and compact decision boundary to reject new samples as unknown states. Therefore, under the collective decision rule, some samples of known states are inevitably misidentified as unknown states. For the 14,115 known class sample matrices and 4705 unknown class sample matrices shown in Table 1, the computational cost (CPU time) of the MRCNN-OVRN is approximately 75.81 s. The low computational cost demonstrates the ability of the proposed method to achieve real-time fault classification.

5. Conclusions

In this article, a novel collective decision framework combining a CNN-based feature extractor and an OVRN classifier is proposed to address the important challenge of deep learning models facing new test samples. The work extends the CNN-based health recognition schemes, for instance, conventional CNNs, residual CNNs (RCNNs), and multi-scale residual CNNs (MRCNNs). It allows these methods to simultaneously achieve recognition of known and unknown health states. CNN-based models learn abstract and generalized latent representations, while the features extracted by the OVRN and decision boundaries established by the collective decision rule are beneficial for rejecting unknown health states and preserving satisfactory recognition performance for known states. The public dataset from the Tennessee Eastman process (TEP) was carried out to validate the proposed methods. Specifically, the effectiveness of the proposed CNN-OVRN, RCNN-OVRN, and MRCNN-OVRN was validated by a comparison with several statistical analysis-based models and CNN-based baseline networks. The proposed methods are demonstrated to have satisfactory accuracy in both known and unknown states recognition by comparing the recognition accuracy and analyzing the distribution histogram of decision scores obtained by the test methods. The significance of this work was to provide a generalizable deep-learning-based framework for the states recognition of industrial processes under the assumption that not all health states are known.

Further research of this work can be implemented from the following aspects. First, how to merge the identified samples from unknown health states into the original training dataset and how to retrain the existing model is a meaningful challenge. Second, the selection of hyperparameters relies on manual setting and repeated experiments, which can consume much time to obtain satisfactory recognition performance. It might be a possible development direction that allows the optimal usage of machine learning with deep learning models to realize feature learning in a fast and automatic way.