Diversified Kernel Latent Variable Space and Multi-Objective Optimization for Selective Ensemble Learning-Based Soft Sensor

Abstract

The improvement of data-driven soft sensor modeling methods and techniques for the industrial process has strongly promoted the development of the intelligent process industry. Among them, ensemble learning is an excellent modeling framework. Accuracy and diversity are two key factors that run through the entire stage of building an ensemble learning-based soft sensor. Existing base model generating methods or ensemble pruning methods always consider the two factors separately, which has limited the development of high-performance but low-complexity soft sensors. To work out this issue, a selective ensemble learning-based soft sensor modeling method based on multi-kernel latent variable space and evolutionary multi-objective optimization is proposed, referred to as MOSE-MLV-VSPLS. This method designs a multiple diversity enhancement mechanism in the base model generation stage. Diversified input variable subspaces are first constructed using the maximum information coefficient on the bootstrapping random resampling subset. Then a set of base models that combine accuracy and diversity are generated on supervised latent variable subspaces under multiple kernel function perturbations. Further, two quantifiable parameters are designed for accuracy and diversity, and the multi-objective gray wolf optimization algorithm is used to select the base models that maximize these two important parameters to achieve effective ensemble pruning at the model ensemble stage. The MOSE-MLV-VSPLS method is applied to two typical industry processes, and the experimental results show that the method is effective and superior in selective ensemble-based soft sensor modeling.

1. Introduction

The accurate prediction of key process parameters of industrial processes is one of the important aspects in the development of smart manufacturing in process industries [1,2,3]. Data-driven soft sensors that are constructed based on state variables and detection variables in industrial processes are widely used in chemical, pharmaceutical, and metallurgical industries because of their ability to describe actual process states more realistically and accurately, which strongly supports the development of intelligent levels of industrial processes [4,5,6,7,8,9]. However, it is worth noting that industrial big data often have more complex data characteristics. On one hand, the operational variables of the production process obtained according to a large number of instruments available at the production site show strong correlation and redundancy characteristics. On the other hand, the internal mechanism of the industrial production process is complex, and the relationship between process variables and detection indicators is strongly nonlinear [10,11]. The complex data relationships bring challenges to the performance requirements of accuracy and reliability of industrial process soft sensors.

According to the differences in the modeling framework, data-driven soft sensors are classified into global models and local models. The framework of global models uses some learning algorithms such as the artificial neural network [12,13,14,15], the Gaussian process regression algorithm [16], etc. to generate a single soft sensor model that can describe the whole process of the data. However, it is difficult for either of the superior learning algorithms to comprehensively and adequately learn all the information embedded under the complex data relationships of the data. The result is that those soft sensors tend to exhibit unstable generalization performance in applications. On the contrary, the local modeling framework adopts the idea of “divide and conquer”, by generating a series of “good but different” base learners for the same problem, and takes advantage of the “group wisdom” to reach a more reasonable conclusion [17,18]. Among them, the ensemble learning modeling approach, which has developed rapidly in the last decade or so, has achieved good results in practical tasks such as classification and parameter prediction [19,20,21].

The improvement of the generalization performance of the ensemble learning-based soft sensor consists of two important steps. The first one is to generate a set of base models with high accuracy and diversity levels [17]. Research shows that subspaces of input variables that are strongly correlated but have weak redundancy with output variables are a prerequisite for accurate and robust base model construction [22,23,24]. The maximum information coefficient algorithm (MIC) is a filter method that has the ability to measure the correlation of variables taking into account the linear and nonlinear characteristics of industrial process data and solves the problem of redundant input variables using the max-dependency metric [25,26]. The latent variable models (LVMs) are constructed based on the kernel method, by using the nonlinear mapping method, and can deeply explore the potential data relationships between variables and extract more useful data information only through a relatively simple and straightforward training process, which is suitable for the rapid construction of soft sensor base models for industrial data [27,28,29].

In addition, maintaining a high level of difference between base models is the key to the success of the ensemble learning soft sensor models [17]. Therefore, the design of the variable subspace diversity mechanism at the base model generation stage is particularly important. Research scholars add perturbation mechanisms to enhance the randomness of each stage at the level of input samples, variable feature extraction, etc. to achieve the goal of enhancing the level of diversity. Examples include multiple random resampling [30,31,32], random feature subspace method, etc. However, these methods make it difficult to obtain a subspace of modeling variables with large variability when the sample size is not sufficient or when there is little variability in the relationship of variables between local sample sets, and the diversity among base models is difficult to ensure. It is noteworthy that different kernel function types and kernel parameters have different characteristics in the LVMs [33,34,35]; thus, the same input variable structure will be mapped to different latent variable modeling subspaces (LVS). This character of LVMs can be added to the base model diversity enhancement mechanism [29,36].

In ensemble learning, many could be better than all [37,38]. Therefore, selectively choosing ensemble parts of base models that are based on certain standards and show better comprehensive performance and lower similarity from base models is another key step for high-performance ensemble learning soft sensor model construction. The selection process of base models is also called ensemble pruning [39,40]. Whether the ensemble pruning effect is satisfactory depends on reasonable quantitative indexes of base model accuracy and diversity criteria on one hand and quick and easy selection methods on the other hand. For soft sensor models, the accuracy is often measured by quantitative indicators such as root-mean-square error (RMSE), mean square error (MSE), etc. [7,18]. However, there is no unified standard for the quantitative parameters of diversity. Mutual information of variables or covariance and other correlation or complementarity indicators are usually chosen [41].

The based models in subensemble sets obtained by ensemble pruning are expected to be precise and different, but this work often is NP-complete hard and even intractable to approximate, so the selection method of base models has always been the focus of researchers. Based on the quantitative indicators of accuracy and diversity, a comprehensive performance evaluation function of the subensemble set was constructed by using weighting algorithms and other methods. Then the greedy search algorithm or optimization algorithm was used to solve the evaluation function for ensemble pruning. However, a multi-step analysis and a relatively large amount of overall computation are always required in these methods. In terms of detailed parameter design, not only does the weight value between the two indicators need to be constantly adjusted according to the changeable model condition and data characteristics, but the number of base models in the subensemble set also needs to be artificially set in advance [40,42,43].

Motivated by the above problems of ensemble soft sensor modeling, the mechanism of base model diversity generation and the method of base model integrated pruning are the focus of this study research. The main contributions of this study are as follows:

(1) A diversified base model construction method based on adaptive multi-kernel supervised LVSs is proposed. The method takes advantage of the diversification and variation of different kernel function types and kernel function parameters, uses multiple kernel functions and an adaptive kernel parameter choosing algorithm to map each variable space obtained by variable selection to different and reasonable modeling LVSs, adds random perturbations to the construction of variable subspace, and generates base model parameters.

(2) According to the predictions of each base model on the verification set, a diversity quantification parameter called the comprehensive Euclidean distance (CED) is designed, which is used to establish the diversity level of the subensemble model set during the ensemble pruning.

(3) A subensemble set generation method based on formalizing ensemble pruning with simultaneous diversity and accuracy into a multi-objective optimization problem is proposed. The selection of each base model is defined as the decision objective, and the comprehensive quantitative parameter of accuracy and diversity is used as the objective function. The multi-objective gray wolf optimization algorithm is applied to solve the problem, which simplifies the integrated pruning process.

(4) A detailed comparative experiment is designed to verify the effectiveness of this research method.

2. Methods

In this section, the preliminaries about the maximal information coefficient (MIC), kernel partial least squares (KPLS), and multi-objective gray wolf optimization algorithm (MOGWO) are briefly introduced.

2.1. Maximal Information Coefficient (MIC)

The maximal information coefficient has been proposed by Reshef et al. in 2011, which was called “A correlation for the 21st century” [25]. Relying on its universal, fair, and symmetrical properties, it can capture and quantify a wide range of complex relations between variables, provide similar scores to equally noisy relationships of different types, and be robust to outliers. Therefore, MIC is a good indicator applicable to measure the strengths of correlations between process variables and target variables of industrial non-linear process data, helps to remove redundant variables, and constructs a highly robust model.

MIC is the maximum value of normalized mutual information obtained from an optimal discrete partition. The range of MIC is between 0 and 1. A higher MIC value indicates the two variables have a closer relationship, whereas MIC(X,Y) = 0 describes the independent relationship between two variables of X and Y. MIC(X,Y) = 1 shows the two variables have deterministic relationship.

The specific steps for the correlation analysis between m-dimensional process variable X and target variable y using MIC are as follows:

First, combine variable X and variable y into a finite ordered dataset $D=\left\{\left(x_1,y_1\right),\left(x_2,y_1\right),\cdots \left(x_m,y_1\right)\right\}$. Define a grid G = (r,j), gridding a scatterplot consisting of the dataset D using the grid G. When the number of rows r and columns c are fixed, different grids can be obtained by different partitioning methods.

Then, calculate mutual information between variables X and Y according to the frequency of scattered points in each subgrid of grid G. It can be formed as

$$I\left(X;Y\right)=H\left(X\left|G\right.\right)+H\left(Y\left|G\right.\right)-H\left(X,Y\left|G\right.\right)=I\left(D\left|G_{r_i{c}_j}\right.\right)$$

(1)

$H\left(X\left|G\right.\right)=-{\displaystyle \sum _{i=1}^x\frac{n_i}{n}}·{\log }_2\frac{n_i}{n}$, $H\left(Y\left|G\right.\right)=-{\displaystyle \sum _{j=1}^y\frac{n_j}{n}}·{\log }_2\frac{n_j}{n}$, $H\left(X,Y\left|G\right.\right)=-{\displaystyle \sum _{i=1}^x{\displaystyle \sum _{j=1}^y\frac{n_{ij}}{n}·{\log }_2}\frac{n_{ij}}{n}}$, where $n$ is the total number of scattered points, $n_i$ is the scattered points in the ith row of grid G in the scatter chart,$n_j$ is the scattered points in the jth column of grid G in the scatter chart, and $n_{ij}$ is the scatter number in the subgrid formed by the ith row of the jth column of grid G in the scatter graph.

Changing the grid division produces different mutual information values, with the maximum value being

$$I_{\max }\left(D,r,c\right)=\max I\left(D\left|G_{rc}\left(k\right)\right.\right),k=1,2,\dots$$

(2)

At last, for any number of rows $r$ and columns $c$, the MIC standardizes the maximum mutual information to [0, 1] by Equation (3):

$$MIC(D)=\underset{rc<B}{\max }\left\{\frac{I_{\max }\left(D,r,c\right)}{{\log }_2\min \left(r,c\right)}\right\}$$

(3)

where B(n) = n^0.6 is the mesh division fineness, which is a function of sample size.

2.2. Kernel Partial Least Squares Algorithm (KPLS)

KPLS algorithm is an extension of PLS by using the kernel function, the original data matrix is replaced by the kernel matrix, and then PLS is used to model the kernel matrix linearly [44].

Set the low-dimensional sample input variable space as X, $X=\left[x_1,x_2,\cdots ,x_d\right]\in R^{n\times d}$, where $R^{n\times d}$ is a d-dimensional real space, and n is the number of samples. Assume that the nonlinear mapping relationship is $\Phi :X\to L$; X will be projected into the high-dimensional latent space variable L through this mapping. All input variables can be expressed as $\phi (X)=\left(\phi (x_1),\phi (x_2),\cdots \phi (x_n)\right)$.

In fact, when constructing the latent variable space, the known form of the kernel function is always introduced to replace the nonlinear mapping function, and then the corresponding kernel matrix of the kernel function is calculated and the eigenvector of the kernel matrix is solved to project the dataset in the high-dimensional space.

The kernel function is expressed as

$$K\left(x_i,x_j\right)={\phi }_i^T{\phi }_j$$

(4)

where $x_i$ and $x_j$ is the input variable of low-dimensional space, and $i,j=1,2,\cdots ,n$; K(x) is a kernel function.

Therefore, the process of KPLS is briefly described as follows:

First, using the kernel function K, the inner product of high-dimensional latent variable space data is replaced by the $n\times n$ Gram kernel matrix:

$$\mathsf{\Phi }(X)=K(X,X)=\phi {(X)}^{\prime }\phi (X)={\displaystyle \sum _{i=1}^n{\phi }_i^T}{\phi }_i$$

(5)

Next, like the PLS algorithm, the KPLS model is decomposed into

$$\left\{\begin{array}{c}K=T{P}^T+E\\ Y=U{Q}^T+F\end{array}\right\}$$

(6)

where T is the score matrix of kernel matrix K,$T=\left[t_1,t_2,\cdots ,t_l\right]\in R^{n\times l}$, and U is the score matrix of target variable Y, $U=\left[u_1,u_2,\cdots ,u_l\right]\in R^{n\times l}$. P and Q are the loading matrices of K and Y, respectively, $P=\left[p_1,p_2,\cdots ,p_l\right]\in R^{d\times l}$, $Q=\left[q_1,q_2,\cdots ,q_l\right]\in R^{d\times l}$, E and F are the residual matrices of K and Y, respectively, and l is the number of latent variables.

At last, nonlinear iterative partial least squares (NIPALS) can be used to solve the model, and the specific steps can be referred to in the literature [19].

According to the above steps, LVS is closely related to a kernel function; there are many kinds of kernel functions, including local kernel functions, Gaussian kernel functions, Cauchy kernel functions, Laplacian kernel functions, global kernel functions, and polynomial kernel functions. The forms of common kernel functions include the following:

$$\mathrm{Polynomial} \mathrm{kernel} \mathrm{function}: K_{\mathrm{Poly}}(x,y)={\left(x^{T}y+c\right)}^d$$

(7)

where d is the order of the polynomial kernel function. At x = 0.2, the kernel function curves of different d are shown in Figure 1. The characteristic curve of the kernel function shows that the distance from the test point has little effect on the result of the polynomial kernel function under the same kernel parameters.

Three typical kernel functions with local characteristics are as follows:

$$\mathrm{Gaussian} \mathrm{kernel} \mathrm{function}: K_{\mathrm{Gau}}(x,y)=\exp \left(-\frac{{‖x-y‖}^2}{2{\sigma }^2}\right)$$

(8)

$$\mathrm{Cauchy} \mathrm{kernel} \mathrm{function}: K_{\mathrm{Gauchy}}(x,y)={\left(-\frac{{‖x-y‖}^2}{{\delta }^2}+1\right)}^{-1}$$

(9)

$$\mathrm{Laplacian} \mathrm{kernel} \mathrm{function}: K_{\mathrm{Laplace}}(x,y)=\exp \left(-\frac{‖x-y‖}{\gamma }\right)$$

(10)

where $\sigma$, $\delta$, and $\gamma$ are the width parameters of the three kernel functions, respectively.

At x = 0.2, Figure 2 shows the characteristic curves of three kernel functions with local characteristics under the same width parameter.

All three kernel functions have strong local characteristics, as shown by the fact that the values of the kernel functions are relatively large near the test point, while the values of the kernel functions become progressively smaller for data far away from the test point. However, the sensitivity of the three kernel functions to the width parameter has some differences.

Based on the above analysis, it can be seen that for the same original low-dimensional nonlinear input variable space, using different types of kernel function types and kernel function parameters can characterize the inherent properties of the original variable space from different perspectives and map it to a diverse LVS.

2.3. Multi-Objective Gray Wolf Optimization Algorithm (MOGWO)

Using a swarm intelligence optimization algorithm to solve the Pareto optimal solution set is a good solution to multi-objective optimization problems [45]. This study considers the convergence speed of the optimization algorithm and the global search mechanism; the multi-objective gray wolf optimization algorithm proposed by Mirjalili and other researchers in 2015 is adopted [46,47].

$$D=\left|C·X_p(t)-X(t)\right|$$

(11)

$$X(t+1)=\left|X_p(t)-A·D\right|,$$

(12)

where t indicates the current iteration, A and C are coefficient vectors, X_p is the position vector of the prey, and X indicates the position vector of a gray wolf.

The vectors $\mathit{A}$ and $\mathit{C}$ are calculated as follows:

$$\mathit{A}=2\mathit{a}\cdot {\mathit{r}}_1-\mathit{a}$$

(13)

$$\mathit{C}=2\cdot {\mathit{r}}_2$$

(14)

where elements of a linearly decrease from 2 to 0 over the course of iterations, and r₁, r₂ are random vectors in [0, 1].

Gray wolf groups $\alpha$, $\beta$, and $\delta$ update their positions under the leadership. Suppose the positions of wolves after t + 1 iterations as X_t+1.

$$\left\{\begin{array}{c}{\mathit{D}}_{\alpha }=\left|{\mathit{C}}_1\cdot {\mathit{X}}_{\alpha }-\mathit{X}\right|\\ {\mathit{D}}_{\beta }=\left|{\mathit{C}}_2\cdot {\mathit{X}}_{\beta }-\mathit{X}\right|\\ {\mathit{D}}_{\delta }=\left|{\mathit{C}}_3\cdot {\mathit{X}}_{\delta }-\mathit{X}\right|\end{array}\right.$$

(15)

$$\left\{\begin{array}{c}{\mathit{X}}_1=\left|{\mathit{X}}_{\alpha }-{\mathit{A}}_1{\mathit{D}}_{\alpha }\right|\\ {\mathit{X}}_2=\left|{\mathit{X}}_{\beta }-{\mathit{A}}_2{\mathit{D}}_{\beta }\right|\\ {\mathit{X}}_3=\left|{\mathit{X}}_{\delta }-{\mathit{A}}_3{\mathit{D}}_{\delta }\right|\end{array}\right.$$

(16)

$$\mathit{X}(t+1)=\frac{{\mathit{X}}_1+{\mathit{X}}_2+{\mathit{X}}_3}{3}$$

(17)

$\left|A\right|$ is in the range of [−2, 2]; when $\left|A\right|\le 1$, the wolves capture prey and obtain the global optimal solution.

Compared with the GWO algorithm, the MOGWO algorithm still maintains the social characteristics of gray wolf hunting behavior. The fittest solution is considered as the alpha (α) wolf, and the second and third best solutions are named beta (β) and delta (δ) wolves, respectively. The rest of the candidate solutions are assumed to be omega (ω) wolves. The hunting is guided by α, β, and δ. The ω wolves follow these three wolves in order to search for the optimum. However, in a multi-objective search space, the solutions cannot easily be compared due to the Pareto optimality concepts [47].

Two important components are added to solve the Pareto optimal solution. The first one is an archive, which can store the non-dominated Pareto optimal solution so far. This component ensures the diversity of the population and at the same time prunes the population to avoid flooding a large number of similar solutions, reduce the complexity of the algorithm, and speed up the optimization. The second one is a leader selection method, which is performed by a roulette wheel method, and to solve the alpha, beta, and delta solutions of the hunting process from the archive, it enhances the exploration ability of the algorithm.

The specific steps of solving the Pareto optimal solution set based on the MOGOW algorithm are referred to in reference [47].

3. Ensemble Soft Sensor Model Construction Based on Diversity Enhancement and Ensemble Pruning

In order to construct an ensemble soft measurement model with a high diversity level of the base model and a strong generalization performance, this study proposes a selective ensemble modeling algorithm framework called MOSE, MLV-VSPLS for short; the modeling framework is illustrated in Figure 3. The core contents include base model generation based on the diversity enhancement mechanism on training samples and ensemble pruning based on the multi-objective optimization algorithm on validation samples.

3.1. Diversified Base Model Construction Method Based on Multi-Kernel Latent Variable Space

The effective generation mechanism of diversified base models is an important factor to ensure the generalization performance of the ensemble soft sensor model. In this study, a base model generation method based on the most relevant variable selection and adaptive multi-kernel supervised latent variable subspace is proposed for homogeneous integration. The basic idea is as follows.

First, a set of subsample sets ${\left\{X_i,Y_i\right\}}_{i=1}^M$ is obtained by bootstrapping random resampling, where M is the times of resampling, representing the number of sample subsets. Each subsample set will form a modeling subspace; for example, the kth subspace is $S_k=\left\{X_k,Y_k\right\}$, $k\in (1,M)$, and $X=\left[x_1,x_2,\dots ,x_m\right]\in R^{n\times m}$ is the set of process variables, where m is the number of input variables. $Y=\left[y_1,y_2,\dots ,y_m\right]\in R^{n\times l}$ is the target variable, where l is the number of output variables. This is the most common diversity mechanism used in the sample set generation phase.

Next, delete the process variables whose MIC results are less than the threshold $\sigma$ on each subsample set, and keep only the process variables with the strongest correlation with the target variables to build the initial modeling variable subspace S_in = { X_in, Y}, where in each initial variable subspace, $X_{in}\in R^{n\times d}$ means selected process variables, and d is the number of variables most relevant to the output variable Y. The relationships between variables in each subsample set are not the same, which means the initial constructed variable subspaces will be different. This is the diversity perturbation mechanism incorporated in the variable stage.

Then, multi-kernel supervised LVS are constructed using the KPLS algorithm based on different types of kernel functions with an adaptive kernel parameter. The number of initial subspaces S_in equal to the number of subsample sets is M, and the types of kernel functions are k_n. After that, the number of LVS MS_LVM is M × k_n. For each LVS, the SOGWO algorithm is used to optimize the parameters of each kernel function with the fitting error as the accuracy evaluation index. This is the diversity perturbation mechanism incorporated in the modeling variable space and model parameters.

3.2. Ensemble Pruning Based on Multi-Objective GWO Optimization

The selection of base models before the ensemble is another important factor that affects the final performance of the ensemble soft sensor model. This study takes into account the diversity and accuracy of the two indicators; in the ensemble pruning of the base models, the problem of base model selection is formalized as a multi-objective optimization problem, and an evolutionary algorithm is used to solve the problem in order to better weigh these two criteria, to achieve the goal of “both good and different”. The basic idea is as follows:

First, define a small-scale subensemble model set for the final ensemble, which is called SEM_sel. The RMSE is proposed as the accuracy index to measure the comprehensive prediction performance of SEM_sel, which is formed as RMSE_val. Take the distance value between the predicted outputs of the models as the model similarity index to measure the diversity of models in SEM_sel, which is formed as CED_val. Then, transform the model pruning problem into a multi-objective optimization problem that is formulated as follows:

$$\mathrm{minimize} F(x)={\left[f_1(x),f_2(x)\right]}^{\mathrm{T}}$$

(18)

$$\mathrm{subject} \mathrm{to} x\in [a,b]$$

(19)

where x is the decision variable vector, and F(x) is the object vector.

The specific steps to build the SEM_sel base on MOGWO that takes into account both accuracy and diversity parameters are as follows:

(1) Define the decision variables.

The construction of the SEM_sel essentially corresponds to the question of whether all the base models are selected. Number all base models from 1 to MS_LVM, and use a 0–1 code to describe the selection status of those base models. Code 1 is assigned to those selected base models. Code 0 is assigned to the unselected base models. Then a binary number of MS_LVM bits represents the selecting status of the base models. Therefore, the decision variable of the multi-objective optimization problem in this study is defined as a binary number of MS_LVM bits.

(2) Define the constraints.

The purpose of ensemble pruning is to select appropriate base models to form a subensemble model set. Therefore, the size of the subensemble models set m_sem is taken as the constraint condition.

(3) Define the objective function.

Definition 1.

Prediction accuracy objective function.

On the verification set $D_{val}=\left\{Q_{val},T_{val}\right\}$, RMSE is used as the prediction accuracy index. First, the prediction accuracy of each base model is estimated as ${\mathrm{RMSE}}_{val}^i$; then the average value of RMSE of all base models is used as the overall prediction accuracy. The ensemble pruning accuracy objective function is defined as follows:

$$f_1(M)={\mathrm{RMSE}}_{val}=\frac{{\displaystyle {\sum }_{i=1}^{m_{sem}}{\mathrm{RMSE}}_{val}^i}}{m_{sem}}$$

(20)

where m_sem represents the number of actually selected model sets SEM_sel, and ${\mathrm{RMSE}}_{val}^i$ represents the RMSE of the ith base model on the validation set.

Definition 2.

Diversity objective function.

In ensemble learning modeling, there is no unified definition of quantitative parameters of diversity. This study defines a comprehensive diversity quantification parameter called CED based on the Euclidean distance of the representation of the base model on the verification set.

On the verification set $D_{val}=\left\{Q_{val},T_{val}\right\}$, the output predictive value of the base model is named as yh_val. The smaller the similarity between the output predictive values of any two base models, the greater the difference between the characterization base models. The similarity measurement s_yh between any two models is defined as

$$s_{yh}(y{h}_{val}^i,y{h}_{val}^j)=\exp (-di{s}_{i,j}), i,j=1,2,\cdots ,M$$

(21)

$$di{s}_{i,j}=\sqrt{{{\displaystyle {\sum }_{k=1}^{n_{val}}\left(y{h}_{val}^i(k)-y{h}_{val}^j(k)\right)}}^2}$$

(22)

n_val is the number of samples in the validation set, and dis denotes the Euclidean distance between any two base models.

All the similarity values obtained from the above equation are averaged and then inversed as the comprehensive diversity quantification parameter as follows:

$$f_2(M)={\mathrm{CED}}_{mval}=-\frac{{\displaystyle \sum _{j=i+1}^{m_{sem}}{\displaystyle \sum _{i=1}^{m_{sem}-1}{s}_{yh}(y{h}_{val}^i,y{h}_{val}^j)}}}{\left(m_{sem}^2-m_{sem}\right)/2}$$

(23)

According to the above analysis, the ensemble pruning problem is converted into a minimization multi-objective optimization solution problem:

$$\min \left[{\mathrm{RMSE}}_{val},{\mathrm{CED}}_{val}\right]$$

(24)

The above multi-objective optimization problem is solved using the MOGWO algorithm. Each resolution in the Pareto optimal solution set corresponds to a set of selected results of the base model after ensemble pruning.

3.3. Model Ensemble

Each base model belongs to unprivileged equality construction when it is initially constructed. Therefore, in this study, average the output values of test samples on the $m_{sem}$ base models selected by ensemble pruning as the final ensemble soft sensor model output.

4. Case Study

In this section, two typical industrial process case studies were used to verify the effectiveness of the proposed MOSE-MLV-VSPLS algorithm. One was to predict the penicillin content in the penicillin fermentation process, and the other was the butane content in the debutanizer process.

The methods used for comparison were as follows:

(1) EPLS (ensemble partial least squares regression): an ensemble soft sensor modeling method that ensemble all base models generated by the PLS algorithm on the diversified subsample space obtained based on bootstrap resampling.

(2) VSEPLS (variable selection-based EPLS): an ensemble soft sensor model that uses the MIC algorithm for input variable selection on each subsample set of the constructed EPLS.

(3) LV-VSEPLS (latent variable space-based VSEPLS): an ensemble soft sensor modeling method that diversified base models are constructed on the latent variable space, which built based on an adaptive single kernel function on each variable space of VSEPLS.

(4) MLV-VSEPLS (multiple latent variable space-based VSEPLS): an ensemble soft sensor modeling method that diversified base models are constructed on the latent variable space which built base on adaptive multiple kernel functions on each variable space of VSEPLS.

(5) MOSE- MLV-VSPLS (selective ensemble MLV-VSPLS based on the muti-objective optimization algorithm): a selective ensemble soft sensor modeling method that ensemble those base models selected by the ensemble pruning method proposed in this paper that content diversity and accuracy simultaneously.

Take the root-mean-square error (RMSE) and coefficient of determination (R²) as the prediction performance indicators of the model established by the above method.

$${\mathrm{RMSE}}_{test}=\sqrt{\frac{1}{n_t}{\displaystyle \sum _{i=1}^{n_t}{\left({\widehat{y}}_{ti}-y_{ti}\right)}^2}}$$

(25)

$$R^2=1-\frac{{\displaystyle \sum _{i=1}^{n_t}{\left({\widehat{y}}_{ti}-y_{ti}\right)}^2}}{{\displaystyle \sum _{i=1}^{n_t}{\left({\widehat{y}}_{ti}-{\overline{y}}_t\right)}^2}}$$

(26)

where n_t means the total number of sample points in the test set, ${\widehat{y}}_{ti}$ and $y_{ti}$ are the predicted output value and actual output value of the ith test sample, and ${\overline{y}}_t$ represents the mean value of the sample points.

4.1. Fermentation Process for Penicillin Production

4.1.1. Process Description

The penicillin fermentation process is a typical biochemical batch benchmark for soft sensor and fault diagnosis algorithms. The penicillin fermentation industrial process simulation platform, which can comprehensively reflect the actual process of penicillin fermentation, etc., and provides standardized biochemical reaction data, was developed by a process modeling and monitoring research group led by Professor Cinar at the Illinois Institute of Technology (IIT) from 1998 to 2002. This simulation platform considers the effects of control variables such as the pH value, temperature, aeration rate, agitator power, and substrate acceleration rate on bacterial and penicillin production and comprehensively includes factors such as bacterial growth, carbon dioxide concentration, penicillin production, and generation of heat at the same time. The technological process of the penicillin fermentation process is shown in Figure 4, and the detailed process description and dynamic model are referred to [45].

Different reaction results can be obtained for different application requirements by adjusting the initial values of some process variables in the fermentation tank. In the construction of the soft sensor model in this study, penicillin concentration is used as the output variable, and 14 process variables are used as the input variables. See

Table 1. Input variables in the penicillin fermentation process.

Input Variables	Variable Description	Units	Input Variables	Variable Description	Units
x1	Aeration rate	L/h	x8	Substrate feed temperature	K
x2	Agitator power	W	x9	Dissolved oxygen concentrate	g/L
x3	Substrate feed rate	L/H	x10	Culture volume	L
x4	Acid flow rate	L/H	x11	Carbon dioxide concentrate	g/L
x5	Base flow rate	L/H	x12	Fermenter temperature	K
x6	Cooling water flow rate	L/H	x13	pH	-
x7	Hot water flow rate	L/H	x14	Generated heat	kcal

for the meaning of specific variables.

Based on the simulation platform, the fermentation cycle was set to 400 h, the sampling interval was 1 h, and the initial value of the substrate concentration for each process was increased or decreased by 0.1–0.2 from the default value. Eight batches of fermentation process data were counted, and five batches of process data were randomly selected as training samples (62.5% of the total data volume) for model training. Two batches of process data were used as the validation set (25% of the total data volume) for model parameter optimization and integration pruning. Two batches of process data were used as the validation samples (25% of the total data volume) for model parameter optimization and integration pruning, and one batch of process data was used as the test set samples (12.5% of the total data volume) for soft measurement model performance evaluation.

The model parameters affect the prediction performance of the model. In this study, the parameters in the base model were optimized adaptively using the GWO algorithm with the objective of minimizing the RMSE criteria, and the specific content and search range of the parameter optimization included the following:

(1) The number of supervised latent variables $n_l$ in EPLS and VSEPLS, where $n_l=[1,2,\cdots m]$, and m represents the characteristic dimension of the input variable.

(2) Three kernel function parameters in LV-VSEPLS and MLV-VSEPLS, where $\sigma \in [0.5,0.7,0.9,1,2]$ for the Gaussian kernel function, $d\in [2,3,4,5,6]$ for polynomial kernel functions, and $\delta =[0.1,0.3,0.5,0.7,0.9,1,2,3]$ for the Cauchy kernel function.

(3) In the MOGWO algorithm, the number of populations is set to 200, the number of iterations is 100, and the one with the best performance among the Pareto optimal solutions is selected as the final screening combination of the base model.

4.1.2. Performance Test

In this study, to demonstrate the effectiveness of the diversity perturbation mechanism in the base model generation stage, we use an ensemble of all base models generated on different diversity mechanisms at first. At this stage of the experiment, the number of base models under each method is the same, which is set to 12, 30, 45, and 60. In the EPLS, VSEPLS, and LV-VSEPLS methods, the number of base models equals the resampling times. In MLV-VSEPLS, three kinds of kernel functions are used for each resampled sample space, and three different base models are constructed; the number of base models is three times the resampling times.

The performance of the penicillin fermentation process under different diversity perturbation mechanisms in the framework of the full ensemble soft sensor is given in

Table 2. Performance of fully ensemble soft sensor models in penicillin fermentation under different diversity perturbation mechanisms.

Method	Kernel Function	Resampling Times	Number of Subspaces	RMSE	R2
EPLS	-	12	12	0.04977	0.9975
		30	30	0.04891	0.9976
		45	45	0.04890	0.9976
		60	60	0.04889	0.9976
VSEPLS	-	12	12	0.02794	0.9982
		30	30	0.02793	0.9982
		45	45	0.02791	0.9982
		60	60	0.02790	0.9982
LV-VSEPLS	Gaussian kernel function	12	12	0.01416	0.9979
		30	30	0.01406	0.9986
		45	45	0.01378	0.9986
		60	60	0.01375	0.99980
	Poly kernel function	12	12	0.01410	0.9987
		30	30	0.01466	0.9990
		45	45	0.01395	0.9994
		60	60	0.01451	0.9989
	Cauchy kernel function	12	12	0.01466	0.9977
		30	30	0.01464	0.9987
		45	45	0.01470	0.9987
		60	60	0.01441	0.9990
MLV-VSEPLS	Multiple kernel function	4	12	0.01267	0.9998
		10	30	0.01223	0.9998
		15	45	0.01220	0.0999
		20	60	0.011934	0.9998

.

According to the RMSE criteria, the optimal set of data in each method is bold. From this, the EPLS method incorporates only input data perturbations in the base model generation and shows the worst prediction accuracy on the test set. The VSEPLS method adds an input variable selection mechanism, which improves the prediction performance of the ensemble model to some extent compared with EPLS, but the accuracy of the two methods is not high. LV-VSEPLS combines the perturbation input features with the perturbation kernel function parameters on the basis of perturbed input data and improves the accuracy of the prediction to a certain extent. By comparison, in the MLV-VSEPLS method, the latent data relationships of the samples are fully explored from different perspectives, and a diversity of latent variable modeling subspaces are constructed. In the full ensemble framework, the number of resampling is reduced, and the optimal performance is achieved with the same number of base models. It is also clear from the data that the rate of improvement in the model accuracy is higher when the number of modeling sample spaces is increased from 15 to 30. However, the subsample set of the base model for constructing the ensemble learning algorithms in this study was obtained through bootstrap resampling, and the base learners were homogeneous. Due to the limited data samples, the base learners were not independent of each other, so when the number of base models was increased from 45 to 60, the improvement rate of prediction accuracy of ensemble learning soft sensor models showed a decreasing and stabilizing trend.

Next, comparative experiments are conducted on the full ensemble method and the selective ensemble method based on different selection criteria and ensemble pruning methods. In the SE-MLV-VSPLS algorithm, the quantitative parameters of accuracy or diversity are used as the criteria for ensemble pruning, and the GWO algorithm is applied to optimize the selection of the base model, which can maximize the parameters. In the MOSE-MLV-VSPLS method proposed in this study, after multi-objective optimization, one of the Pareto optimal solutions is selected and decoded to the ensemble for the soft sensor.

Table 3. Performance of selective ensemble soft sensor models under ensemble pruning with different indicators.

Ensemble Type	Method	Criteria	Number of Subspaces	Size of Subensemble Model	RMSE	R2
ALL	MLV-VSEPLS	-	30	30	0.01223	0.09994
SEL	SE-MLV-VSPLS	Accuracy	30	9	0.01220	0.09992
SEL	SE-MLV-VSPLS	Diversity	30	8	0.01271	0.09990
SEL	MOSE-MLV-VSPLS	Accuracy and diversity	30	10	0.01063	0.09998

shows the performance of each model under the experiment.

The data show that the same effect of the full ensemble is obtained with a smaller number of base models after ensemble pruning is completed based on a single indicator of accuracy or diversity. In contrast, after pruning with dual objectives of accuracy and diversity, the redundant base models are effectively eliminated, the complexity of the ensemble modeling is reduced, and the optimal prediction performance is obtained with a smaller number of base models. Figure 5 shows the performance comparison of each soft sensor built based on four different methods; the horizontal axis in the figure represents the sample points, while the vertical axis represents the values of penicillin concentration. Method 1 is MOSE-MLV-VSPLS, method 2 is MLV-VSEPLS, method 3 is LV-VSEPLS, and method 4 is VSEPLS.

From the prediction trend curves, there is a significant difference between the real values and the predicted output values obtained using the VSEPLS and LV-VSEPLS methods. In contrast, almost no deviation can be found between the predicted output values and the real output values under the MOSE-MLV-VSPLS method, and these two curves fit well with each other.

4.2. Debutanizer Column

4.2.1. Process Description

In naphtha, propane and butane need to be removed from the process, in which the debutanizer column is an important part of the desulfurization and naphtha separation unit in the petroleum refinery production process. Referring to reference [48], a brief schematic diagram of the process is shown in Figure 6.

The goal of the debutanizer column is to minimize the concentration of butane at the bottom of the column while maximizing the yield of purified gasoline. Therefore, accurate estimation of the butane content is important to improve the effectiveness of the debutanizer column. This industrial process includes 7 process variables, and the specific meanings of the variables are shown in

Table 4. Input variables in the debutanizer column.

Input Variables	Variable Description
u1	Top temperature
u2	Top pressure
u3	Reflux flow
u4	Flow to next process
u5	Sixth tray temperature
u6	Bottom temperature
u7	Bottom pressure

.

In this study, 2285 data points were collected in the process: 1142 random sample points in the dataset were divided into training set samples, accounting for 50% of the total data volume, for model training; 343 random sample points were used as validation set samples, accounting for 10% of the total data volume, for the optimization and integration pruning of model parameters; and 800 sample points were used as test set samples, accounting for 40% of the total data volume.

Similar to the penicillin fermentation process in Section 4.1, the GWO algorithm was used to optimize the kernel function parameters and the number of hidden variables adaptively in the base model construction process with the goal of minimizing the prediction error. According to the data characteristics of the debutanizer column, the selected kernel functions include the Gaussian kernel function, the Cauchy kernel function, and the Laplace kernel function, and the search range of the corresponding kernel functions is set as $\sigma \in [0.03,0.05,0.07,0.09,0.1,0.2]$ for the Gaussian kernel function, $\delta \in [0.01,0.02,0.03,0.05,0.07,0.09,0.1,0.2]$ for the Cauchy kernel function, and $\gamma \in [0.07,0.1,0.2,0.3,0.4,0.5,0.7]$ for the Laplace kernel function.

4.2.2. Performance Test

Similarly, in the EPLS, VSEPLS, and LV-VSPLS methods, the number of base models is equal to the number of sample spaces generated under resampling, which is 15, 30, and 45, respectively. In the MLV-VSEPLS method, the number of resampling times is 5, 10, and 15, and the number of base models is 15, 30, and 45, respectively.

Table 5. Performance of fully ensemble soft sensor models in test sets under different diversity perturbation mechanisms.

Method	Kernel Function	Resampling Times	Number of Subspaces	RMSE	R2
EPLS	-	12	12	0.1555	0.1345
		30	30	0.1553	0.1300
		45	45	0.1554	0.1321
VSEPLS	-	15	15	0.1713	0.1250
		30	30	0.1621	0.1293
		45	45	0.1613	0.1295
LV-VSEPLS	Gaussian kernel function	15	15	0.0646	0.8576
		30	30	0.0564	0.8976
		45	45	0.0541	0.9001
	Laplace kernel function	15	15	0.0527	0.9053
		30	30	0.0519	0.9081
		45	45	0.0494	0.9196
	Cauchy kernel function	15	15	0.0552	0.8960
		30	30	0.0511	0.9110
		45	45	0.0497	0.9159
MLV-VSEPLS	Multiple kernel functions	5	15	0.0516	0.9142
		10	30	0.0513	0.9138
		15	45	0.0503	0.9195

shows the performance of the full ensemble-based soft sensor in the debutanizer column test set under different diversity perturbation mechanisms.

From the experimental results in Table 5, for the debutanizer column, the prediction accuracy of the EPLS method using only the input data perturbation is very low because of the strong nonlinearity of the seven process variables, and the prediction accuracy is improved to some extent by adding input variable perturbation, but it is far from meeting the requirements. Based on the kernel trick, under the framework of the full ensemble model, the accuracy of the soft sensor ensemble model has been improved to a certain extent. However, a single kernel function is used to construct the latent variable subspace, so even with the increase in resampling times in the LV-VSEPLS method, the precision improvement space of the soft sensor model is limited. By comparison, in the MLV-VSEPLS method, multiple diversity enhancement mechanisms are used in the process of subsample set generation, initial variable space, and latent variable subspace construction for diversified base models. Under the fully ensemble framework, the MLV-VSEPLS method has achieved better results.

However, due to the existence of poor performance and redundant base models, the full ensemble method used above cannot achieve the optimal performance. Therefore, in order to improve the performance of the ensemble soft measurement models, the ensemble pruning work is essential.

Similarly, a comparative study of ensemble pruning methods is designed to display the superiority of the multi-objective optimization selection method based on GWO. A multi-objective optimization method is used to decode one of the Pareto optimal solutions under 45 base models, and the target variables are predicted on the test set; the experimental results are shown in

Table 6. Performance of fully ensemble and selective ensemble under diversity enhancement mechanism.

Ensemble Type	Method	Criterion	Number of Subspaces	Size of Subensemble Model	RMSE	R2
ALL	MLV-VSEPLS	-	45	45	0.0503	0.9195
SEL	SE-MLV-VSPLS	Accuracy	45	12	0.0518	0.9051
SEL	SE-MLV-VSPLS	Diversity	45	13	0.0550	0.8942
SEL	MOSE-MLV-VSPLS	Accuracy and diversity	45	16	0.0453	0.9302

.

In order to more clearly show the performance, the box plots in Figure 7 are used to show comparatively the RMSE obtained with the selective ensemble and the full ensemble under the statistical results of 25 experiments.

The results in the Figure 7 are the same as those in Table 6. The ensemble pruning method based on the multi-objective optimization algorithm achieves optimal performance at a smaller base model size. In many experiments, the maximum RMSE generated is still smaller than the minimum RMSE of the full ensemble soft sensor model.

In detail, the prediction results of MLV-VSEPLS and MOSE-MLV-VSPLS in the test set are given separately in Figure 8.

The curves in Figure 8 show that for the MOSE-MLV-VSPLS method proposed in this study, by using the ensemble pruning method, the predicted output values match well with the actual values in data locations with large changes especially and achieve the most satisfactory prediction performance.

5. Conclusions

This study proposes a soft sensor construction method named MOSE-MLV-VSPLS in an ensemble learning framework to accurately measure industrial processes’ key parameters under complex data relationships. As an ensemble learning-based approach, at the base model generation stage, the proposed base model generation method first uses the MIC algorithm on each subsample space obtained by bootstrap random resampling to optimize the variable space. Then it uses the adaptive multi-kernel functions to obtain the latent variable space describing the data relationships from different perspectives, followed by the construction of the diversified base model. Compared with the EPLS algorithm, which does not use variable selection, and the LV-VSEPLS algorithm, which is based on a single kernel function, the proposed algorithm has the advantages of effectively removing redundant variables and significantly increasing the level of diversity of the underlying model’s strong nonlinear data relationships.

Since both the accuracy and diversity affect the performance of an ensemble learning-based soft sensor model, in the base model ensemble phase, a comprehensive diversity quantification indicator is designed, and the base model ensemble pruning, which considers the accuracy and diversity quantification metrics simultaneously, is transformed into a multi-objective optimization problem. The solution is performed using a multi-objective gray wolf optimization algorithm, which is an average ensemble of a selected set of high-performing and complementary base models. Compared with the full ensemble approach, the complexity of the ensemble modeling is significantly reduced.

The comparative experimental results for two application examples show that the ensemble soft sensor model with the MOSE-MLV-VSPLS method achieves satisfactory performance by an effective diversity enhancement mechanism and the selective mean ensemble of smaller base models. The MOSE-MLV-VSPLS method has strong generality in homogeneous integration modeling. The ensemble strategy can be updated, and other base models can be used according to different application requirements.

It is worth noting that in this study, for the generation of the multi-kernel LVS in the base model construction, the kind of kernel function for each LVS is single, and in the future, hybrid kernel functions can be used for a better description of the data space. On the other hand, after converting the ensemble pruning problem into a multi-objective optimization, the success of the proposed method depends to a large extent on the design of a suitable diversity of definitions of the objective functions, which is open to question. As new process data continue to be added to the data space, the model update mechanism of this algorithm still needs further research and strengthening.