Rate of Penetration Prediction Method for Ultra-Deep Wells Based on LSTM–FNN

Abstract

The drilling process is complex, especially for ultra-deep wells, which face the problems of high temperature, high pressure and poor drilling resistance in their formation. In order to establish an ROP (the Rate of Penetration) prediction model for ultra-deep wells, the characteristics of ultra-deep well drilling operations, such as formation temperature and formation pressure, are fully considered in the process of parameter optimization. Combined with the drilling mechanism and mutual information correlation coefficient, the final input parameters are determined. The powerful nonlinear fitting ability of the artificial intelligence method is very suitable for predicting the ROP. Considering the time sequence of multi-source data, this paper combines the powerful timing information-based mining ability of the LSTM (Long Short-Term Memory Neural Network) with the nonlinear fitting ability of FNN (Fully Connected Neural Network), and establishes an intelligent prediction model of the ROP based on a LSTM–FNN. The results show that the average relative error and $R^2$ of the LSTM–FNN model on the data of well 1 and well 2 are better than the FNN and LSTM models. In addition, the accuracy of the LSTM–FNN model on the data of adjacent wells is reduced by only 5%, which further verifies the good mobility of the model.

1. Introduction

Deep and ultra-deep well drilling has become the key focus of the future of oil and gas exploitation [1]. With the increase of well depth, the geological conditions become more complex, well formation becomes more difficult and drillability worsens, resulting in a low drilling efficiency, high safety risk, long drilling cycles and other problems [2]. However, the prediction of the rate of penetration (ROP) can lower drilling risks and optimize drilling parameters in advance, which is of great significance to the improvement of the efficiency of deep drilling [3].

With the progress of information technology, intelligent drilling and completion technology has begun to enter the field. Information technology, microelectronics technology, robotics technology, communication network technology and drilling and completion engineering have been closely combined and integrated, which has greatly promoted the innovation of oil drilling and production technology [4]. The intelligent prediction of the ROP is one of the most important aspects to intelligent drilling and completion. Although the ROP prediction is very important to improve drilling efficiency, it is difficult to match these factors with the ROP and establish an accurate ROP prediction model [5].

In recent years, with the deepening of the oilfield digitization process, the massive drilling data and efficient computing power of computers provide good conditions for data-driven ROP prediction. There are several Machine Learning (ML) methods [2,6], which have been applied in ROP prediction, such as Random Forest (RF) [7], Gradient Boosting Machines (GBM), K-nearest Neighbors (KNN) [3] and Support Vector Machine (SVM) [8,9]. However, when these machine learning models are trained using a large amount of drilling data, they facing the challenges of long training times and overfitting. When the volume of drilling data is very large, deep learning possesses a clear advantage, as computational efficiency and single-well prediction accuracy are significantly improved. A large number of researchers have begun to use fully connected neural network (FNN) to establish the ROP prediction [4,5,10,11,12], and compare it with machine learning models and ROP equations [13]. The results show that the established model has high precision and a fast training speed, which is better than the conventional machine learning and ROP equation model [14,15,16]. Abiodun Ismail Laval [4] et al. optimized the key parameters of the ROP and selected the density of rock, porosity and point load index as input parameters, and ROP as an output parameter, and a drilling data set with a total sample of 71 was established. Using these drilling data, an ROP prediction model is established based on the FNN algorithm, and the relative error on the test set is 14%. Furthermore, considering the timing characteristics between the drilling parameters and ROP, some scholars established an ROP prediction model by using recurrent neural network (RNN) [17,18,19,20], long short-term memory neural network (LSTM) and gated recurrent neural network (GRU). The hyperparameters of the neural network algorithm, such as network structure, activation function and weight parameters, have significant influence on the prediction effect of the model [21,22]. Some scholars have attempted to use a genetic algorithm (GA) [23], particle swarm optimization (PSO) and other algorithms to optimize the model’s hyperparameters [24,25] and to obtain the optimal hyperparameters and achieve the best prediction effect. Some scholars have also improved the effect of the model by combining the structures of various neural networks, as seen with Shuo Zhu [26].

However, for the problem of deep well drilling, the existing intelligent prediction models for the ROP tend to ignore the characteristics of deep well drilling, such as high temperature and high pressure. In addition, multiple drilling parameters do have time series correlation with ROP. However, the direct prediction of the ROP only after RNN or LSTM calculation is likely to lose other correlation characteristics. Therefore, the current intelligent prediction model of ROP still needs to be optimized.

Based on the characteristics of deep well drilling and data correlation, this paper analyzes and optimizes the input parameters of a deep well ROP intelligent prediction model. At the same time, considering the time series correlation and other complex correlations between multiple parameters and the ROP, the existing LSTM structure is optimized, and an intelligent prediction model of the ROP based on an LSTM–FNN algorithm is established. The accuracy and mobility of the model are analyzed through the field actual drilling data, which verifies the reliability and superiority of the model established in this paper.

2. Artificial Intelligence Algorithm

2.1. Fully Connected Neural Network

A fully connected neural network (FNN) is a computational model that can be used to approximate functional relationships between variables, identify features or perform other complex tasks. A typical fully connected neural network is shown in Figure 1, which consists of multiple layers, each of which contains many nodes.

The neural network starts from the input layer, and the output of the previous layer acts as the input of the next layer until the final result is output. As neurons receive input data from other neurons, each with some weight, the total input value received by the neurons will be compared to the neuron threshold and then processed through the activation function to generate the neuron output. As shown in Equation (1), the input matrix of the neural network is recorded as $X_{in}$, the output matrix is recorded as $Y_{out}$, the output matrix of n layer is recorded as $Y_n$, the weight matrix is $W_n$ and the bias matrix is recorded as $B_n$, where $n\in \left[1,4\right]$. The structure of the fully connected neural network is expressed as follows:

$$\{\begin{array}{c}{Y}_1=X_{in}\hfill \\ Y_2={\phi }_{active}\left(W_2{Y}_1+B_2\right)\hfill \\ Y_3={\phi }_{active}\left(W_3{Y}_2+B_3\right)\hfill \\ Y_{out}=Y_4=W_4{Y}_3+B_4 \hfill \end{array}$$

(1)

${\phi }_{active}$ is the activation function. Tanh and ReLU activation functions are widely used.

Tanh function: As shown in Equation (2), the range of the tanh function is (−1,1), and the average value of the activation function is close to 0, which is more conducive to improving the accuracy of the next layer of learning, but easily causes the problem of gradient disappearance. x is the input parameter of the activation function.

$$tanh\left(x\right)=\frac{e^x-e^{-x}}{e^x+e^{-x}}$$

(2)

ReLU function: As shown in Equation (3), only the linear function exists in the ReLU function, and the calculation speed is faster. There is no gradient saturation problem when the input parameter is positive.

$$ReLU\left(x\right)=max\left(0,x\right)$$

(3)

2.2. Long Short-Term Memory Neural Network

A long short-term memory neural network (LSTM) is a neural network with the ability to remember short- and long-term information, which was proposed by Hochreiter and Schmidhuber [27] in 1997. Due to the prosperity of deep learning, LSTM has been modified by Felix Gers [28] and others many times. Eventually, a relatively systematic and complete LSTM algorithm was formed, and it has been widely used in many realms.

LSTM is a special recurrent neural network algorithm. Recurrent neural networks (RNNs) are sensitive to short-term information and seldom use long-term information. It is therefore difficult to deal with the dependencies between long term and short term information. Traditional RNNs always entail gradient vanishing, but LSTM can effectively avoid this problem. The specific structure of LSTM is shown in Figure 2.

LSTM has two transmission states; cell state ($C_t$) and hidden state ($h_t$). Throughout the entire process, $C_t$ changes quite slowly. Usually, the output $C_t$ in each different node is $C_{t-1}$ of the previous cell plus several signals, while $h_t$ varies greatly. Additionally, in the LSTM algorithm, individual neurons are replaced by a unit structure. Each unit contains three gates: the forgetting gate, the input gate and the output gate. The whole process of LSTM can be briefly described by the Equation (4), below. In this equation, W is the weight in the whole process whereas b is the bias.

$$\begin{array}{c}\left[\begin{array}{c}{\tilde{c}}_t\\ o_t\\ i_t\\ f_t\end{array}\right]=\left[\begin{array}{c}tanh\\ \sigma \\ \sigma \\ \sigma \end{array}\right]\left(W\left[\begin{array}{c}{x}_t\\ h_{t-1}\end{array}\right]+b\right)\\ c_t=f_t·c_{t-1}+i_t·{\tilde{c}}_t\\ h_t=o_t·\mathit{tan}h\left(c_t\right)\end{array}$$

(4)

Forgetting gate: The forgetting gate is a key component in the LSTM unit. It can control which information of the input is retained and which is forgotten, and avoid the problems of gradient vanishing and gradient explosion. The structure of the forgetting gate is shown in Figure 3a.

The forgetting gate determines what information the neural network discards from the previous cell state. The gate reads $h_{t-1}$ and $x_t$, then maps them to a signal $f_t$ between 0 and 1 by a sigmoid function. Finally, this signal $f_t$ is multiplied $c^{t-1}$ of the previous cell state to determine what information is discarded. The signal of 1 indicates that the information is completely retained, while the signal of 0 indicates the information is completely discarded. The calculation expression is shown in Equation (5):

$$f_t=\sigma \left(W_f\cdot \left[h_{t-1},x_t\right]+b_f\right)$$

(5)

Input gate: The input gate is used to control how much of the input information is saved to the memory cell. The input gate consists of two parts. First, the input gate generates candidate state ${\tilde{c}}_t$ of the current unit through a tanh layer as seen in Equation (6):

$$\begin{array}{c}{\tilde{C}}_t=\mathit{tan}h\left(W_C\cdot \left[h_{t-1},x_t\right]+b_C\right)\end{array}$$

(6)

Second, the sigmoid function in the input gate produces a signal $i_t$ between 0 and 1. This is used to control how much information the candidate state ${\tilde{c}}_t$ needs to save. The structure of the signal acquisition in the input gate is shown in Figure 3b. The signal $i_t$ will be generated by the following equation (Equation (7):

$$\begin{array}{c}{i}_t=\sigma \left(W_i\cdot \left[h_{t-1},x_t\right]+b_i\right)\end{array}$$

(7)

The structure of obtaining $C_t$ in the input gate is shown in Figure 3c. Through the input gate, the old cell state will be updated to the new cell state. With the signal $f_t*C_{t-1}$ generated by the forgetting gate, the signal $i_t$ generated by the sigmoid function and the candidate state ${\tilde{c}}_t$ generated by the tanh layer, the new cell state $C_t$ will be obtained by Equation (8):

$$C_t=f_t\ast C_{t-1}+i_t\ast {\tilde{C}}_t$$

(8)

Output gate: Output signals are based on cell states, and there is also a filtering process. The structure of the output gate is shown in Figure 3d below. There are also two operation parts: first, the sigmoid function in the output gate will generate the control signal $o_t$ between 0 and 1. The signal $o_t$ will be generated by Equation (9):

$$\begin{array}{c}{o}_t=\sigma \left(W_o\left[h_{t-1},x_t\right]+b_o\right)\end{array}$$

(9)

Second, the final output signal $h_t$ is obtained by the output signal $tanh(C_t)$ multiplied with the control signal $o_t$. The signal $h_t$ will be generated by Equation (10):

$$\begin{array}{c}{h}_t=o_t\ast \mathit{tan}h\left(C_t\right)\end{array}$$

(10)

The output gate controls the influence of the current cell unit, and also determines which part of the cell unit will be output at the time of step t. The structure of the output gate is shown in Figure 3d.

3. Field Data Processing

3.1. Data Introduction

Compared with deep well drilling operations, ultra-deep well drilling operations face more difficulties and challenges. Under the harsh drilling environment of high temperature and high pressure in deep formation, the properties of drilling fluid will change, so it is difficult to both carry rock cuttings and cool the bit in time, resulting in a reduced ROP and accelerated bit wear. In addition, with the increase of well depth, the formation abrasiveness increases, the drillability is poor, the formation pressure system is complex and the safe density window is narrow, which increases the difficulty of drilling parameter optimization. Therefore, in the process of establishing the prediction model of ROP in ultra-deep wells, it is necessary to focus on the complex correlation between parameters such as bottom hole temperature (T), formation pressure, rock physical properties, bit type and ROP.

The drilling and logging data used in this paper come from two vertical and ultra-deep wells in a block of the Tarim oil fields, which are marked as well 1 and well 2, respectively. The straight-line distance between the two wells is 5098 m, and the well depth is more than 6000 m. The deep formation is mainly interbedded with mudstone and siltstone, and the difference in rock physical properties is small. The specific input parameters are shown in

Table 1. Drilling parameters.

Items	Parameter
Geological	Gamma Ray (GR), Spontaneous Potential (SP), Acoustic (AC)
Engineering	Well Depth, Rotational Speed (RPM), Weight on Bit (WOB), Torque, Hook Load (HL), Riser Pressure (RP), Inlet Flow (IF), Outlet Flow (OF), Inlet Temperature (IT), Outlet Temperature (OT), Inlet Density of Drilling Fluid (ID), Outlet Density of Drilling Fluid (OD), Inlet Conductivity (IC), Outlet Conductivity (OC), Pump Impulse Speed (PIS)
Trajectory	Deviation Angle, Borehole Curvature
Drilling Fluid	Density, Viscosity, Plastic Viscosity, Yield
Drilling Bit	Bit Size, Bit Type, Bit Footage (BF)

.

3.2. Feature Correlation Analysis

Through feature selection, on the one hand, the correlation between various parameters and the ROP can be clarified; on the other hand, irrelevant parameters can be removed to reduce the complexity and improve the calculation efficiency of the model. The depth neural network model has a strong nonlinear fitting ability, and the Spearman and Pearson test methods are mostly used to test the linear relationship between multiple parameters and the ROP. For the nonlinear correlation test, this paper introduces the mutual information feature selection method.

Mutual information I (X: Y) of continuous random variables X and Y is as follows:

$$I\left(X;Y\right)=H\left(X\right)+H\left(Y\right)-H\left(X,Y\right)={{\displaystyle \int }}^{}{{\displaystyle \int }}^{}p\left(x,y\right)\log \frac{p\left(x,y\right)}{p\left(x\right)p\left(y\right)}dxdy$$

(11)

In Equation (11), $H\left(X\right)$ and $H\left(Y\right)$ are the unconditional entropy of variables X and Y, respectively, $H\left(X,Y\right)$ is the joint entropy of variables X and y, and $p\left(x,y\right)$ is the joint probability distribution of X and Y; $p\left(x\right)$ and $p\left(y\right)$ are marginal probability distributions, respectively.

Mutual information is easily affected by the amount of data. In order to eliminate the influence of the amount of data on the value of mutual information, it is necessary to normalize the mutual information, as shown in the following formula:

$$\mathrm{NMI}\left(X;Y\right)=2\frac{I\left(X,Y\right)}{H\left(X\right)+H\left(Y\right)}$$

(12)

The value range of NMI is [0, 1]. If the NMI value of X and Y is 0, it means that the two variables are independent variables or mutually independent variables; that is, they have no information contained in each other. On the contrary, the closer the mutual information value is to 1, the higher the degree of correlation between the two variables, and the more the same amount of information is contained.

This paper focuses on the correlation analysis of engineering parameters. As shown in Figure 4, the NMI of the rotary table speed, riser pressure, outlet flow and outlet conductivity are about 0.08, which is relatively small, indicating that the correlation between the above parameters and mechanical ROP is low and should be removed.

At the same time, the NMI value of the main pump impulse and inlet flow remains large (0.81) and the NMI value of the inlet flow and ROP is greater than that of the main pump impulse and ROP. In order to reduce the characteristic dimension of the ROP prediction model, the total pump impulse speed is removed.

3.3. Time Series Data Processing

As shown in the Figure 5, before the time series transform processing for drilling data, the drilling data shall be sampled every 1.0 m. The series data containing different characteristic parameters and ROP values, and the data matrix shape is [n, m], where n is the number of samples and m is the characteristic number of each sample. The time series data needs to combine the data within a time interval to form a sequential data set. The depth was taken as the timing scale to divide the time series. Since the drilling rate before the t time step also belonged to known data, the drilling rate before the t time step was taken as the new input feature when constructing the time series data, and the input of the time series data included as all sampling points 10 m before the t time [(x^t−10, y^t−10), (x^t−9, y^t−9), (x^t−8, y^t−8),…, (x^t−3, y^t−3), (x^t−2, y^t−2), (x^t−1, y^t−1), (x^t, y^t)].

The output time t + 1 is the drilling rate of one meter below the sampling point (y^t+1). The time steps of the input parameters have an important influence on the models. In order to ensure the precision of the ROP prediction models, the time steps of the drilling data should be greater than 1 m. For the input parameters, the longer the time steps are, the more information is usually contained. However, with the increase of time steps length, the complexity of the model increases and the calculation speed decreases. In order to ensure the timeliness of the model, the time steps of input parameters should not be too large.

4. Model Construction

4.1. Model Workflow

The frame of the intelligent ROP prediction model established in this paper is shown in Figure 6. According to the drilling characteristics of ultra-deep wells, the key influencing factors are analyzed and the model input parameters are preliminarily determined. The characteristic parameters are analyzed by cross relation number, the input parameters are optimized, and the characteristic dimension is reduced. The original data were transformed into time series data, and the drilling rate prediction model was built by LSTM and FNN in a series. By optimizing the super-parameters of the model, the optimal prediction model was obtained.

4.2. Parameter Optimization

In this paper, well depth is the basis of the time series data division. From the perspective of the model, different time series data lengths contains different information, which leads to a difference in model training accuracy. From the analysis of the actual drilling process, the depth range reflects how much information of engineering geological parameters before the time step is considered when the model predicts the drilling rate of the next time step, so the step length of the time series data is a model hyperparameter that needs to be prioritized. The hyperparameters to be optimized in the process of model establishment include: the timing data input step, that is, the well depth range of each time series data; the number of LSTM layers; the number of LSTM hidden layer neurons; the number of FNN layers; the number of FNN neurons; the FNN activation function; and the model learning rate. The specific hyperparameter search range is shown in the

Table 2. LSTM–FNN model parameter combination.

LSTM			FNN			Model Parameters
Step Length	Layers	Neural	Layer	Neural	Active Function	Learning Rate
5	1	10	1	32	ReLU	1 × 10−3
10	2	20	2	64
15	3	30	3	128	Tanh	1 × 10−4
20	4	40	4	256

.

80% of the drilling data of well 1 were randomly selected as the training set and 20% as the test set to optimize the hyperparameter of the model. The mean relative error (MRE) and regression determination coefficient ($R^2$) were selected as the standards to test the prediction accuracy of the model, As shown in Equations (13) and (14).

$$\mathrm{MRE}=\frac{1}{n}{\displaystyle \sum }_{i=1}^n\left[\frac{\left|y_i^p-y_i\right|}{y_i}\right]\times 100$$

(13)

$$R^2=1-\frac{{{\displaystyle \sum }}_{i=1}^n{\left(y_i-y^p\right)}^2}{{{\displaystyle \sum }}_{i=1}^n{\left(y_i-\overline{y}\right)}^2}$$

(14)

where, n is the number of samples, $y_i^p$ is the predicted value, $y_i$ is the true value and $\overline{y}$ is the average of the actual values.

According to the calculation results, when the parameters in

Table 3. LSTM–FNN model parameter selection.

LSTM			FNN			Model Parameters
Step Length	Layers	Neural	Layer	Neural	Active Function	Learning Rate
10	2	30	3	128	ReLU	1 × 10−3

are selected for the model, the model achieves the optimal prediction accuracy. At this time, the prediction accuracy of the model on the test set, where the MRE is 12%,$R^2$, is 0.90.

5. File Application

In order to verify the migration effect of the intelligent ROP prediction model based on LSTM–FNN, we will take 80% of the data of well 1 as the training set and 20% as the test set, and use the optimal ROP prediction model to predict the ROP of well 2. In addition, prediction models of the ROP are established based on LSTM and FNN algorithms. The prediction accuracy of the model is shown in

Table 4. Precision comparison of ROP prediction models.

Model	Train Set Well 1 (80%)		Test Set 1 Well 1 (20%)		Test Set 2 Well 2 (100%)
	MRE	${\mathit{R}}^2$	MRE	${\mathit{R}}^2$	MRE	${\mathit{R}}^2$
FNN	18.12	0.72	20.02	0.69	26.30	0.58
LSTM	17.23	0.83	20.03	0.71	25.56	0.65
LSTM + FNN	13.05	0.90	15.11	0.87	18.23	0.83

. The prediction results of the model are shown in Figure 7 and Figure 8. Figure 7a–c are the partial prediction results of the FNN, LSTM and LSTM–FNN on test set 1, respectively. Figure 8a–c are the partial prediction results of the FNN, LSTM and LSTM–FNN on test set 2, respectively.

As shown in Table 4, the relative error difference between the FNN and LSTM on the training set, test set 1 and test set 2 is small, about 1%. Although the LSTM ROP prediction model increases the time series information processing process compared with the FNN model, the accuracy of the model has not been greatly improved.

As shown in Figure 9a, on the training set, the relative errors of the LSTM–FNN model are 4.17% and 5.07% lower than those of the LSTM and FNN models, respectively; on test set 1, the relative errors of the LSTM–FNN model are 4.92% and 4.91% lower than those of the LSTM and FNN models, respectively; on test set 2, the relative errors of the LSTM–FNN model are 7.33% and 8.07% lower than those of the LSTM and FNN models, respectively. After adding an FNN network to the output layer of the LSTM, the accuracy of the model is greatly improved.

It can be seen from Figure 9a that the MRE of the FNN model in the training set, test set 1 and test set 2 are 18.12%, 20.02% and 26.30%, respectively. The maximum MRE difference of the FNN model in the three data sets is 8.18%; the MRE of the LSTM model in the training set, test set 1 and test set 2 are 17.23%, 20.03% and 25.56%, respectively. The maximum MRE difference of the model in the three data sets is 8.33%; the MRE of the LSTM–FNN model in the training set, test set 1 and test set 2 are 13.05%, 15.11% and 18.23%, respectively. The maximum MRE difference of the model in the three data sets is 5.18%; the results show that LSTM–FNN not only has higher accuracy than the other two models on similar data sets and migrated data sets, but also has the lowest reduction in model accuracy.

$R^2$ is the fitting degree of the established model to the data, and the straight range is [0, 1]. The closer $R^2$ is to 1, the better the model is. According to Figure 9b on the three data sets, the $R^2$ of the LSTM–FNN model is 0.9, 0.87 and 0.83, respectively; these results are higher than those of the other two models. This further demonstrates that the LSTM–FNN model fully excavates data information to establish an ROP prediction model.

6. Conclusions

Ultra-deep well drilling is faced with challenges such as high temperature and high pressure at the bottom of the well, strong formation abrasiveness and staggered soft and hard formations. Accurate ROP prediction is of great significance for bit tool optimization and engineering parameter optimization.

The influence of temperature, pressure and other parameters on the ROP should be fully considered in the process of the ROP prediction of ultra-deep wells. In this paper, the drilling mechanism and data correlation analyses are combined to optimize the input parameters of the model.

In the process of establishing the ROP prediction model, the LSTM–FNN model is established to fully mine the time series information and nonlinear information between multiple parameters and the ROP. Compared with the single FNN and LSTM ROP prediction models, the MRE of the model is reduced by 8.07% and the $R^2$ is increased by 0.27 at most. In addition, on the new data set, the accuracy and $R^2$ reduction of the LSTM–FNN model are the smallest, indicating that the model has strong migration ability. This paper verifies that the optimal combination of network structures can improve the fitting ability of the algorithm, and this plays an important role in improving the accuracy and mobility of the model.