Adaptive Proportional Integral Robust Control of an Uncertain Robotic Manipulator Based on Deep Deterministic Policy Gradient

Abstract

An adaptive proportional integral robust (PIR) control method based on deep deterministic policy gradient (DDPGPIR) is proposed for n-link robotic manipulator systems with model uncertainty and time-varying external disturbances. In this paper, the uncertainty of the nonlinear dynamic model, time-varying external disturbance, and friction resistance of the n-link robotic manipulator are integrated into the uncertainty of the system, and the adaptive robust term is used to compensate for the uncertainty of the system. In addition, dynamic information of the n-link robotic manipulator is used as the input of the DDPG agent to search for the optimal parameters of the proportional integral robust controller in continuous action space. To ensure the DDPG agent’s stable and efficient learning, a reward function combining a Gaussian function and the Euclidean distance is designed. Finally, taking a two-link robot as an example, the simulation experiments of DDPGPIR and other control methods are compared. The results show that DDPGPIR has better adaptive ability, robustness, and higher trajectory tracking accuracy.

1. Introduction

A robotic manipulator is similar to the human arm and can replace or assist humans to complete the tasks of picking, placing, painting, welding, and assembling. The manipulator plays an important role in industrial production, underwater exploration, medical application, aerospace, and other fields [1,2,3,4]. To achieve a better control effect and meet the control requirements of different fields, the manipulator must have the ability to track a trajectory with high precision. Due to the highly nonlinear, dynamic characteristics of a robotic manipulator, and the influence of joint friction and time-varying external interference in practical applications, it is difficult to obtain accurate information about model parameters. Therefore, when designing a control strategy, good adaptability and high-precision trajectory tracking abilities are necessary for the uncertainty of the n-link robotic manipulator system.

In order to better control the robot manipulator, the robustness of the control strategy has attracted extensive attention. Robustness here refers to the ability to produce good dynamic behavior in the face of modelling errors and unmodelled dynamics of the robot manipulator [5,6]. Loucif and Kechida [7] and Elkhateeb et al. [8] used a whale optimization algorithm and an artificial bee colony algorithm, respectively, to optimize the parameters of the proportion integral differential (PID) controller, improve the trajectory tracking accuracy of the robot manipulator under unmodeled dynamics, and make the controller have a certain robustness. In order to model the control process of the robot manipulator more accurately, Ardeshiri et al. [9,10] proposed a fractional order fuzzy PID controller. A fractional order controller summarizes the design of an integer order PID controller and extends it from point to plane. This extension increases the flexibility of control system design and can realize the control process more accurately. With the help of a fractional order PID controller, the controller can be designed to ensure that the closed-loop system has a stronger robustness to gain variation and aging effect [11]. Therefore, PID and other model-based control strategies have proved to be effective, but these methods need to obtain dynamic model information of the controlled object. In the actual control process, it is difficult to obtain accurate information due to the complexity of the manipulator mechanism and the uncertainty of external interference [12,13]. To solve this problem, it is necessary to compensate or approximate the uncertainty and time-varying external disturbance of the nonlinear dynamic model to meet the demand of actual control. Wang [14] used the robust controller to compensate for the uncertainty, unmodeled dynamics, and external interference of the dynamic model parameters of the robot manipulator, and so realized accurate tracking for it. Yang and Jiang [15] and Rouhani [16] used a fuzzy logic system to approximate the nonlinear dynamic model of the robot manipulator. However, the design of the fuzzy logic system depends on expert knowledge. A neural network is good at approximating the uncertain mathematical model, and it is one of the effective solutions to nonlinear system control problems. Yang et al. [17] proposed an adaptive neural network control method based on a nonlinear observer. The joint speed of the manipulator is estimated by a nonlinear observer, and, based on the estimated value of the speed, an adaptive radial basis function neural network is used to compensate for the uncertainty of the robotic manipulator system, to improve the tracking accuracy of its end force and its joint position. Guo et al. [18] proposed an adaptive neural network control method, which uses the weighted performance function to control the joint angle and the trajectory tracking error within an expected range, approximates the dynamic model of the manipulator through the radial basis function neural network, and uses the adaptive law to adjust the weights of the neural network, to improve the robustness of the controller. Although the neural network has a good compensation or approximation effect for the uncertainty and time-varying disturbance of the nonlinear dynamic model, training the network is likely to converge to the local optimal problem. Therefore, a robust term based on deep reinforcement learning is proposed to compensate for the modeling error of the nonlinear dynamic model of an n-link robot manipulator. Under conditions of structural parameter perturbation, time-varying external interference, and friction resistance, the influence of the uncertainty of the dynamic model on the controller can be reduced so as to maintain the stability of the control system and improve the trajectory tracking performance of an n-link robotic manipulator.

As an important branch of artificial intelligence technology, reinforcement learning mainly selects actions through interactive learning between agents and the environment. The environment responds to the actions of agents and transforms them into a new state. At the same time, it generates a reward. The agent’s goal is to maximize the accumulated discount reward value [19,20]. Compared with the classical control method, reinforcement learning does not need to obtain an accurate dynamic model, which is very advantageous in solving the decision sequence problem under highly nonlinear and uncertain conditions [21]. Kukker and Sharma [22] and Runa et al. [23] used the fuzzy Q-learning algorithm to realize trajectory tracking control of the manipulator. Kim et al. [24] used the State-Action-Reward-State-Action (SARSA) algorithm to locate fixed and random target points of the end effector of a three-link Planar Arm. Although Q-learning and SARSA can effectively solve some typical reinforcement learning tasks, the algorithms need to be built in discrete space. Therefore, in the control problem, it is often necessary to discretize the continuous process. However, sparse discretization can easily reduce the control accuracy, and dense discretization can easily fall into the curse of the dimension problem [25,26]. One method for solving this problem is to effectively combine deep learning with reinforcement learning. A deep neural network is used, in traditional reinforcement learning, to model solutions to continuous reinforcement learning tasks [27,28]. Based on this method, Lillicrap et al. [29] proposed a depth deterministic strategy gradient algorithm based on the actor critic framework. Shi et al. [30] used the DDPG algorithm to deal with controlling the zinc electro winning (Zep) process, which effectively solved the problems of inaccurate modeling and time delay, while also being more energy-saving than the traditional control method. Sun et al. [31] used this algorithm to solve the heavy vehicle adaptive cruise decision-making problem, which has good adaptability in a strange and complex environment. Zhao et al. [32] solved the cooperative control problem of wind farms through the DDPG algorithm and reduced the learning cost in the learning process. Therefore, the DDPG algorithm seems to be effective in solving multiple continuous-state space reinforcement learning tasks.

The purpose of this paper is to establish an n-link robotic manipulator control system with a model for uncertainty and time-varying external disturbances. An adaptive PIR control method based on deep reinforcement learning is proposed. The modeling error of the nonlinear dynamic model of an n-link manipulator is compensated by robust control, and the parameters of the controller are adjusted by a DDPG algorithm to improve the adaptability of the controller to the uncertain, nonlinear, dynamic model. The main contributions of this paper are as follows:

• Considering the uncertainty and time-varying disturbance of the dynamic model of the n-link robot manipulator system and the influence of friction resistance, the adaptive robust term is used to compensate for the uncertainty of the system. An adaptive PIR control method based on the DDPG is proposed, which has good adaptability and high-precision trajectory tracking ability for the uncertainty of the n-link robot manipulator system.
• A reward function combining a Gaussian function and the Euclidean distance is proposed, which can ensure the reinforcement learning agent learns efficiently and stably and can effectively avoid a convergence of the deep neural network to the local optimal problem.
• Taking a two-link robotic manipulator as an example, the simulation results show that the proposed method is effective compared with an adaptive control based on radial basis function neural network (RBFNN) approximation and PIR control with fixed parameters.

2. Dynamic Model of the n-Link Robot Manipulator

The dynamic model of the n-link robotic manipulator system expresses the relationship between the joint torque and the position, velocity, and acceleration of the connecting rod:

$$M\left(q\right)\ddot{q}+C\left(q,\dot{q}\right)\dot{q}+G\left(q\right)+F_f\left(\dot{q}\right)+{\tau }_d=\tau$$

(1)

where $q\in {ℝ}^n$ is the joint position vector of the manipulator, $\dot{q}\in {ℝ}^n$ is the velocity vector of the manipulator, $\ddot{q}\in {ℝ}^n$ is the acceleration vector of the manipulator, $M\left(q\right)\in {ℝ}^{n\times n}$ is the mass inertia matrix, $C\left(q,\dot{q}\right)\in {ℝ}^{n\times n}$ is the Coriolis force and the centrifugal force vector, $G\left(q\right)\in {ℝ}^{n\times n}$ is the gravity vector, $F_f\left(\dot{q}\right)\in {ℝ}^n$ is the friction vector, ${\tau }_d\in {ℝ}^n$ is the time-varying external disturbance, and $\tau \in {ℝ}^n$ is the torque vector acting on the joint.

The precise values of the $M\left(q\right)$, $C\left(q,\dot{q}\right)$, and $G\left(q\right)$ parameters in the dynamic model are difficult to obtain due to a series of influential factors, such as the complexity of the manipulator mechanism, environmental variations, and measurement errors in the actual operation of the manipulator. Therefore, the actual values for $M\left(q\right)$, $C\left(q,\dot{q}\right)$, and $G\left(q\right)$ are divided into the model part and the error part as follows:

$$M\left(q\right)=M_0\left(q\right)+\Delta E_M\left(q\right)$$

(2)

$$C\left(q,\dot{q}\right)=C_0\left(q,\dot{q}\right)+\Delta E_C\left(q,\dot{q}\right)$$

(3)

$$G\left(q\right)=G_0\left(q\right)+\Delta E_G\left(q\right)$$

(4)

The dynamic model Formula (1) of the n-link robot manipulator can also be expressed as follows:

$$M_0\left(q\right)\ddot{q}+C_0\left(q,\dot{q}\right)\dot{q}+G_0\left(q\right)+E\left(q,\dot{q}\right)+F_f\left(\dot{q}\right)+{\tau }_d=\tau$$

(5)

$$E\left(q,\dot{q}\right)=\Delta E_M\left(q\right)\ddot{q}+\Delta E_C\left(q,\dot{q}\right)\dot{q}+\Delta E_G\left(q\right)$$

(6)

Property 1

([33]). The mass inertia matrix $M_0\left(q\right)$ is symmetric, positive definite and bounded and can be expressed as follows:

$${\mu }_m<\Vert M_0\left(q\right)\Vert <{\mu }_n$$

(7)

where $\Vert M_0\left(q\right)\Vert$ is the norm of the mass inertia matrix $M_0\left(q\right)$; ${\mu }_n$ and ${\mu }_m$ are the upper and lower boundaries, respectively, and both are positive numbers.

Property 2

([34]). Coriolis force and centrifugal force matrix is $C_0\left(q,\dot{q}\right).$ The following equation is satisfied:

$${\xi }^T\left(\dot{M_0}\left(q\right)-2C_0\left(q,\dot{q}\right)\right)\xi =0$$

(8)

Among them,$\left(\dot{M_0}\left(q\right)-2C_0\left(q,\dot{q}\right)\right)$. It is a skew symmetric matrix,$\xi \in {ℝ}^n$.

Property 3

([35]). The gravity vector $G\left(q\right)$ satisfies $\Vert G\left(q\right)\Vert <\rho$, $\rho \in \left(0,\infty \right)$.

3. DDPGPIR Control Design

In this paper, a control strategy for DDPGPIR for the n-link robotic manipulator system with a model for uncertainty and time-varying external disturbances is proposed. The control strategy includes PIR control design, reinforcement learning and policy gradient method, DDPG adaptive PIR control, DDPGPIR network design, the DDPGPIR learning process, and the reward function.

3.1. PIR Control Design

In the n-link robotic manipulator system, the position error $e\left(t\right)$ is the difference between the expected joint angle $q_d\left(t\right)$ and the actual joint angle $q\left(t\right)$. The position error and the error function are defined as follows:

$$e\left(t\right)=q_d\left(t\right)-q\left(t\right)$$

(9)

$$s=\dot{e}+\mathsf{\Lambda }e$$

(10)

where $\mathsf{\Lambda }={\mathsf{\Lambda }}^T=\left(\begin{array}{ccc}{K}_{r1}& \cdots & 0\\ ⋮& \ddots & ⋮\\ 0& \cdots & K_{rn}\end{array}\right)$. Take ${\dot{q}}_s=s\left(t\right)+\dot{q}\left(t\right)$. Then:

$${\dot{q}}_s={\dot{q}}_d+\mathsf{\Lambda }e$$

(11)

$${\ddot{q}}_s={\ddot{q}}_d+\mathsf{\Lambda }e$$

(12)

Therefore, the dynamic model Equation $\left(3\right)$ of the n-link robotic manipulator can be expressed as follows:

$$\begin{array}{cc}\hfill \tau & =M\left(q\right)\ddot{q}+C\left(q,\dot{q}\right)\dot{q}+G\left(q\right)+F_f\left(\dot{q}\right)+{\tau }_d\hfill \\ & =M\left(q\right)\left(\dot{s}+\ddot{q}\right)+C\left(q,\dot{q}\right)\left(s+\dot{q}\right)+G\left(q\right)-M\left(q\right)\dot{s}-C\left(q,\dot{q}\right)s+F_f\left(\dot{q}\right)+{\tau }_d\hfill \\ & =M\left(q\right){\ddot{q}}_s+C\left(q,\dot{q}\right){\dot{q}}_s+G\left(q\right)-M\left(q\right)\dot{s}-C\left(q,\dot{q}\right)s+F_f\left(\dot{q}\right)+{\tau }_d\hfill \\ & =E_m-M\left(q\right)\dot{s}-C\left(q,\dot{q}\right)s+E_s\hfill \end{array}$$

(13)

where

$$E_m=M_0\left(q\right){\ddot{q}}_s+C_0\left(q,\dot{q}\right){\dot{q}}_s+G_0\left(q\right)$$

(14)

$$E_s=\Delta M\left(q\right){\ddot{q}}_s+\Delta C\left(q,\dot{q}\right){\dot{q}}_s+\Delta G\left(q\right)+F_f\left(\dot{q}\right)+{\tau }_d$$

(15)

In PIR control [36], the control law is designed as follows:

$$\tau ={\tau }_m+{\tau }_{pi}+v$$

(16)

$${\tau }_m=M_0\left(q\right){\ddot{q}}_s+C_0\left(q,\dot{q}\right){\dot{q}}_s+G_0\left(q\right)$$

(17)

$${\tau }_{pi}=K_{p}s+K_i{\displaystyle \int }sdt$$

(18)

$$v=K_{s}sgn\left(s\right)$$

(19)

where $\tau$ is the torque applied to each joint of the n-link robotic manipulator, ${\tau }_m$ is the torque control term of the model, $K_p$ and $K_i$ are the gain of the proportional term and the gain of the integral term, respectively, and ${\tau }_s$ is the robust term used to compensate the nonlinear dynamic model error and external disturbance. From Equation (13) and Equation (16), it can be concluded that:

$$M\left(q\right)\dot{s}+C\left(q,\dot{q}\right)s+K_i{{\displaystyle \int }}_0^{t}sdt=-K_{p}s-K_{s}sgn\left(s\right)+E_s$$

(20)

Select the Lyapunov function as follows:

$$V=\frac{1}{2}{s}^{T}Ms+\frac{1}{2}{\left({{\displaystyle \int }}_0^{t}sd\tau \right)}^T{K}_i\left({{\displaystyle \int }}_0^{t}sd\tau \right)$$

(21)

The derivation on both sides of the equation leads to:

$$\begin{array}{cc}\hfill \dot{V}& =s^T\left[M\dot{s}+\frac{1}{2}\dot{M}s+K_i{{\displaystyle \int }}_0^{t}sd\tau \right]\hfill \\ & =s^T\left[M\dot{s}+Cs+K_i{{\displaystyle \int }}_0^{t}sd\tau \right]\hfill \\ & =s^T\left[-K_{p}s-K_{s}sgn\left(s\right)+E_s\right]\hfill \\ & =-s^T{K}_{p}s-{\displaystyle \sum _{i=1}^n}{K}_{sii}{\left|s\right|}_i+s^{T}E\hfill \end{array}$$

(22)

Because of $K_{sii}\ge \left|E_i\right|$, $\dot{V}\le -s^T{K}_{p}s\le 0$. Therefore, the control system is asymptotically stable.

3.2. Reinforcement Learning and Policy Gradient Method

Reinforcement learning is an important branch of machine learning, which is mainly composed of environment, agent, reward, state, and action. When the agent performs action $a_t$ on the environment in state $s_t$, the environment will give the agent a reward $r_{t+1}$, the state changes to the next state $s_{t+1}$, and the future reward value passes through the discount coefficient $\gamma \left(0\le \gamma \le 1\right)$ After weighting, the cumulative reward $r_t$ can be expressed as:

$$r_t={\displaystyle \sum }_{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}=r_{t+1}+\gamma r_{t+2}+{\gamma }^2{r}_{t+3}+\cdots$$

(23)

The policy of reinforcement learning is the functional relationship $\pi$ between state space and action space. The objective of a policy-based reinforcement learning method is to try to find the optimal strategy ${\pi }^{*}$ to maximize the cumulative reward. In the strategy gradient method, the optimal strategy is updated along the gradient direction of the expected cumulative reward as follows:

$$J\left(\theta \right)=\mathbb{E}\left({\displaystyle \sum }_{l=0}^{N}r\left(s_l,a_l\right)|{\pi }_{\theta }\right)={\displaystyle \sum }_{\sigma }P\left(\sigma |\theta \right)r\left(\sigma \right)$$

(24)

$${\theta }_{h+1}={\theta }_h+\vartheta {\nabla }_{\theta }J\left(\pi \left({\theta }_h\right)\right)$$

(25)

where $\theta$ is the parameter vector of the policy, $J\left(\theta \right)$ is the objective function of reinforcement learning, $\sigma =\left(s_0,a_0,s_1,a_1,\cdots s_l,a_l\right)$ is a group of state action sequences, and $P(\sigma |\theta )$ is the action sequence $\sigma$ Probability of occurrence, $\vartheta$ is the learning rate and $h$ is the number of the current update.

3.3. DDPG Adaptive PIR Control

The schematic diagram of the DDPGPIR control system of the n-link robot manipulator is shown in Figure 1. The input of the controller is the error vector $e=\left(e_1,e_2,\cdots e_n\right)$ of the n-link robot manipulator. The output is the torque vector acting on the joint $\tau =\left({\tau }_1,{\tau }_2,\cdots {\tau }_n\right)$. The control performance of the DDPGPIR mainly depends on the parameter vector $g=\left(K_{p1},K_{i2},K_{s2},K_{r2},\cdots K_{pn},K_{in},K_{sn},K_{rn}\right)$. The control problem of the n-link robot manipulator can be expressed as:

$$\underset{g}{\min }{\displaystyle \sum }_{j=1}^n{e}_j\left(q_d{}_j,q_j\left(g_j,p_j\right)\right)$$

(26)

where the vector $q_d$ is the expected joint angle, $q$ is the actual joint angle, $p$ is the physical parameter of n-link robot manipulator, and $j$ is the $j$-th link.

To improve the adaptability and trajectory tracking accuracy of the n-link robot manipulator, the parameter vector of DDPGPIR needs to be adjusted and optimized in real time. However, the process of setting the parameter is time-consuming, and the optimization process is continuous; it is not advisable to adjust the parameters manually. Therefore, it is necessary to find the optimal strategy function ${\mu }^{*}\left(x\right)$, which is one of the effective methods for solving this problem. The state vector $x_t=\left({\tau }_1,e_1,{\displaystyle \int }{e}_{1}dt,\cdots ,{\tau }_n,e_n,{\displaystyle \int }{e}_{n}dt\right)$ is input into the optimal strategy function to obtain the optimal parameter vector $g_t$. The goal of reinforcement learning is to find the optimal strategy for maximizing cumulative rewards. The objective function can be expressed as:

$$J_{\beta }\left(\mu \right)=\underset{\mu }{\max }\mathbb{E}\left[{\displaystyle \sum }_{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}|x_t=x,g_t=\mu \left(x_t\right)\right]$$

(27)

where $\beta$ is the behavior strategy and $\gamma \in \left(0,1\right)$ is the discount factor.

To find the optimal strategy for maximizing the objective function, the strategy gradient method is usually used to select and execute actions from the distribution function of strategy probability in each time step. However, this method needs to sample continuous actions in each time step, which is a huge calculation process. To solve this problem, the deterministic strategy gradient method is used to simplify the calculation process. Therefore, the gradient of the objective function is calculated as follows:

$$\begin{array}{cc}\hfill {\nabla }_{{\theta }^{\mu }}J& \approx {\mathbb{E}}_{x_t\sim {\rho }^{\beta }}\left[{\nabla }_{{\theta }^{\mu }}{Q}_{\mu }\left(x_t,\mu \left(x_t\right)\right)\right]\hfill \\ & \hfill \\ & ={\mathbb{E}}_{x_t\sim {\rho }^{\beta }}\left[{\nabla }_{{\theta }^{\mu }}Q\left(x,g;{\theta }^Q\right){|}_{x=x_t,g=\mu \left(x_t{|\theta }^{\mu }\right)}\right]\hfill \\ & ={\mathbb{E}}_{x_t\sim {\rho }^{\beta }}\left[{\nabla }_{g}Q(x,g|{\theta }^Q){|}_{x=x_t,g=\mu \left(x_t\right)}{\nabla }_{{\theta }^{\mu }}\mu (x_t|{\theta }^{\mu }){|}_{x=x_t}\right]\hfill \end{array}$$

(28)

3.4. Network Design of DDPGPIR

The network structure of DDPGPIR includes an actor network, a critic network, and a corresponding target network. The structure of the actor network is shown in Figure 2. The input is the state vector $x_t$ of the n-link robot manipulator, the two middle hidden layers are the full connection layer and the activation layer, and the output layer is the parameter vector $g_t$. The structure of the critic network is shown in Figure 3. The input includes state vector $x_t$ and parameter vector $g_t$. The four middle hidden layers are the full connection layer, activation layer, superposition layer and activation layer. The output layer is the $Q$ value of action.

To make the training data relatively independent, to accelerate the convergence speed, and to improve the stability of the network update process, the data used for the current network update are not the previous state data obtained by decision-making, but $M$ small batch sample data randomly selected from the experience replay memory. The critic network includes the current critic network $Q(g,x|{\theta }^Q)$ and the target critic network $Q^{\prime }\left({\theta }^{Q^{\prime }}\right)$.The current critic network is updated using a gradient descent method by minimizing the loss function as follows:

$$Q_{target}=r_i+\gamma Q^{\prime }\left(x_{i+1},{\mu }^{\prime }\left(x_{i+1}|{\theta }^{{\mu }^{\prime }}\right)|{\theta }^{Q^{\prime }}\right)$$

(29)

$$L=\frac{1}{M}{\displaystyle \sum }_{i=1}^M{\left(Q_{target}-Q\left(x_i,g_i|{\theta }^Q\right)\right)}^2$$

(30)

$$\nabla L\left({\theta }^Q\right)=\frac{1}{M}\left[Q_{target}-Q\left(x,g|{\theta }^Q\right){\nabla }_{{\theta }^Q}Q\left(x,g|{\theta }^{\mu }\right)\right]$$

(31)

where $Q_{target}$ is the value of the target critic network, $Q(x,g|{\theta }^{\mu })$ is the value of the critic network, $i$ is the $i$th sample data, and $\gamma \left(0\le \gamma \le 1\right)$ is the discount rate. The actor network includes the current actor network $\mu (x|{\theta }^{\mu })$ and the target actor network ${\mu }^{\prime }\left({\theta }^{{\mu }^{\prime }}\right)$. The current actor network is updated with the deterministic strategy gradient as follows:

$${\nabla }_{{\theta }^{\mu }}{J}_{\beta }\left(\mu \right)\approx \frac{1}{M}{\displaystyle \sum }_i\left({\nabla }_{g}Q\left(x,g|{\theta }^Q\right){|}_{x=x_i,g=\mu \left(x_i\right)}{\nabla }_{{\theta }^{\mu }}\mu \left(x;{\theta }^{\mu }\right){|}_{x=x_i}\right)$$

(32)

where ${\nabla }_{{\theta }^{\mu }}{J}_{\beta }\left(\mu \right)$ represents the gradient direction of the $Q$ value caused by the action strategy $\mu$, ${\nabla }_{g}Q\left(x,g|{\theta }^Q\right){|}_{x=x_i,g=\mu \left(x_i\right)}$ represents the change in the $Q$ value caused by action $\mu \left(x_i\right)$ in the current state, and ${\nabla }_{{\theta }^{\mu }}\mu \left(x;{\theta }^{\mu }\right){|}_{x=x_i}$ is the gradient direction of the current strategy.

The target critic network and the target actor network update the network with a soft update with an update rate of $\rho$ as follows:

$$\{\begin{array}{c}{\theta }_{i+1}{}^{Q^{\prime }}\leftarrow \rho {\theta }^Q+\left(1-\rho \right){\theta }_i{}^{Q^{\prime }}\\ {\theta }_{i+1}{}^{{\mu }^{\prime }}\leftarrow \rho {\theta }^{\mu }+\left(1-\rho \right){\theta }_i{}^{{\mu }^{\prime }}\end{array}$$

(33)

3.5. Learning Process of DDPGPIR

The DDPGPIR learning process applied to the manipulator is shown in Figure 4. $\mu \left(x|{\theta }^{\mu }\right)$ and ${\mu }^{\prime }\left(x|{\theta }^{{\mu }^{\prime }}\right)$ are the current actor network and the target actor network, respectively, and $Q\left(x,g|{\theta }^Q\right)$ and $Q^{\prime }\left(x,g|{\theta }^{Q^{\prime }}\right)$ are the current critic network and the target critic network, respectively. The learning process is described as Algorithm 1. First, parameters $\left(Q,\mu ,Q^{\prime },{\mu }^{\prime }\right)$, memory playback space $RM$, and noise $G$ of the online network and the target network are initialized. After the dynamic information $x_t$ of the manipulator is input into the DDPGPIR agent, according to strategy $\mu$ and noise $G$ to determine the optimal parameter $g_t$ of the PIR controller, the output torque of the controller acts on the manipulator. In addition, the system monitors the joint angle $q_d\left(t\right)$ in real time. If $q_d\left(t\right)$ is within a reasonable range, the corresponding reward value will be obtained after this action is executed, and the next state $x_{t+1}$ will be input. Otherwise, the action is immediately stopped, a negative reward is given, and the agent re-selects the new action and executes it. The data $\left(x_t,g_t,r_t,x_{t+1}\right)$ tuple formed in this process will be stored in the experience replay memory $RM$. Small-batch tuple data are randomly extracted from $RM$, the minimal loss function method is used to update the critic network, the deterministic strategy gradient method is used to update the actor network, and the target network is updated by the soft update method.

Algorithm 1. DDPGPIR Algorithm.

Initialize the critic network $Q(g,x|{\theta }^Q)$ and the actor network $\mu (x|{\theta }^{\mu })$

Initialize the target network $Q^{\prime }\left({\theta }^{Q^{\prime }}\right)$ and ${\mu }^{\prime }\left({\theta }^{{\mu }^{\prime }}\right)$ with the same weights

Initialize replay memory $RM$

Initialize Gaussian noise $G$

for episode = $1\cdots M$ do

Receive initial observation state $x_1$

for $t=1\cdots T$ do

select action $g_t=\left(K_{p1},K_{i1},K_{s1},K_{r1},\cdots K_{pn},K_{in},K_{sn},K_{rn}\right)=\mu \left(x_t|{\theta }^{\mu }\right)+G$

select execution action $g_t$

if  $q\left(t\right)\notin \left[-\epsilon ,\epsilon \right]$

reject $g_t$ and add a negative number to $r$

else:

execute $g_t$ and get observed reward $r_t$ and observe new state $x_{t+1}$

store transition $\left(x_t,g_t,r_t,x_{t+1}\right)$ in $RM$

sample mini-batch of $M$ transitions $(x_i,g_i,r_i,x_{t+1})$ from $RM$

set $Q_{target}=r_i+\gamma Q^{\prime }\left(x_{i+1},{\mu }^{\prime }\left(x_{i+1}|{\theta }^{{\mu }^{\prime }}\right)|{\theta }^{Q^{\prime }}\right)$

update critic according to Equations $\left(29\right)$ and $\left(31\right)$

update actor according to Equation $\left(32\right)$

update the target networks according to Equation $\left(33\right)$

end for

end for

3.6. Reward Function

As stated, for most reinforcement learning tasks, there is always a reward function, which can reward each behavior of the agent accordingly, so that the agent can make a corresponding behavior when facing different states and obtain a higher cumulative reward value. To adapt to different reinforcement learning tasks, the reward function must be universal and provide abundant information for the reinforcement learning agents. In the problems discussed in this paper, the trajectory tracking error $e\left(t\right)$ and the joint angle $q\left(t\right)$ of the manipulator are the variables of most concern. When the tracking error $e\left(t\right)$ increases, or the joint angle $q\left(t\right)$ exceeds the reasonable range, a negative reward value should be given; otherwise, a positive reward value should be given. Therefore, the reward function combining the Gaussian function and the Euclidean distance is as follows:

$$r=\alpha r_1+\beta r_2+\delta r_3$$

(34)

$$r_1={\displaystyle \sum }_{j=1}^n-\frac{{\left(q_{d_j}\left(t\right)-q_j\left(t\right)\right)}^2}{2c^2}$$

(35)

$$r_2=\sqrt{{\displaystyle \sum }_{j=1}^n{\left(q_{d_j}\left(t\right)-q_j\left(t\right)\right)}^2}$$

(36)

$$r_3=\{\begin{array}{cc}0,\hfill & \left|q_j\left(t\right)\right|<\epsilon \\ -1,\hfill & other\end{array}$$

(37)

where $\alpha$, $\beta$ and $\delta$ are the coefficients of the reward items, $q_{d_j}\left(t\right)$ and $q_j\left(t\right)$ are the expected joint angle and the actual joint angle of the $j$-th joint, respectively, and $\epsilon$ is a reasonable critical value of the joint angle.

4. Experiment and Results

To verify the control performance of DDPGPIR, taking a two-link robotic manipulator as an example, the DDPGPIR, PIR and RBFNN are simulated and compared in MATLAB/Simulink. The dynamic model of two joint manipulators can be deduced by the Lagrange method [37,38]. The widely studied kinetic models and parameters can be expressed as follows [39]:

$$M\left(q\right)=\left[\begin{array}{cc}{p}_1+p_2+2p_3\cos q_2& p_2+p_3\cos q_2\\ p_2+p_3\cos q_2& p_2\end{array}\right]$$

(38)

$$C\left(q,\dot{q}\right)=\left[\begin{array}{cc}-p_3{\dot{q}}_2\sin q_2& -p_3\left({\dot{q}}_1+{\dot{q}}_2\right)\sin q_2\\ p_3{\dot{q}}_1\sin q_2& 0\end{array}\right]$$

(39)

$$G\left(q\right)=\left[\begin{array}{c}{p}_{4}g\cos q_1+p_{5}g\cos \left(q_1+q_2\right)\\ p_{5}g\cos \left(q_1+q_2\right)\end{array}\right]$$

(40)

$$p={\left[\begin{array}{ccccc}{p}_1& p_2& p_3& p_4& p_5\end{array}\right]}^T={\left[\begin{array}{ccccc}2.9& 0.76& 0.87& 3.04& 0.87\end{array}\right]}^T$$

(41)

In order to achieve better control effect and facilitate comparisons with other control methods, the simulation sampling step size is set at 0.1 s and the simulation cycle is set at 20 s. The initial state of the system is $q_1\left(0\right)=-0.5 \mathrm{rad}$, $q_2\left(0\right)=-0.5 \mathrm{rad}$, The expected trajectory path is $q_{d1}=\sin 0.5\pi t$, $q_{d2}=\sin 0.5\pi t$. The friction force is $F_f=5sgn\left(\dot{q}\right)$. The external interference is ${\tau }_d=10\sin \left(\dot{q}\right)$. After many attempts, a set of appropriate PIR controller parameters are selected as $K_{p1}=60, K_{i1}=45, K_{s1}=35, K_{r1}=3, K_{p2}=60, K_{i2}=45, K_{s2}=35, K_{r2}=3$.

RBFNN has good function approximation and generalization ability and is widely used in nonlinear function modeling [40,41]. The adaptive control of manipulators based on RBFNN approximation is as follows [42]:

$$\mathsf{\tau }=W^{*T}\phi \left(x\right)+K_{v}s-v$$

(42)

where $W^{*}$ is the network weight vector and $x$ is the input signal of the network, $\phi \left(x\right)$ is the column vector of the basis function, $K_v$ is the coefficient of error function term, and $v$ is the robust term used to overcome the approximation error of neural network.

4.1. Learning Results for DDPGPIR

In Figure 5, the reward value obtained by the DDPGPIR agent in the initial learning process is low, because the process is in the exploratory stage. However, as the learning times increase, the reward value gradually increases and tends to be stable and close to the expected cumulative reward value, which verifies that the reward function proposed in this paper can effectively avoid the convergence of a deep neural network to the local optimum. At the same time, the correctness and stability of the DDPGPIR model are proved. Figure 6 shows the changing process of the controller parameters. Because the desired trajectory is constantly changing, the controller parameters are also adjusted in real time, to improve the tracking accuracy of the trajectory.

4.2. Control Effect Comparison of the Controller

Figure 7, Figure 8 and Figure 9 show the trajectory tracking performance of the RBFNN, PIR, and DDPGPIR controllers. The figures show that the DDPGPIR controller has a shorter response time and higher trajectory tracking accuracy than the PIR and RBFNN controllers in the case of friction and time-varying external interference. Figure 10 and Figure 11 show the trajectory tracking errors of the DDPGPIR, PIR, and RBFNN controllers, respectively. It can be seen that compared with DDPGPIR, the PIR and RBFNN controllers have larger overshoot and trajectory tracking errors.

4.3. Control Performance Index Comparison

To further highlight the effectiveness of the DDPGPIR controller, the integral absolute error (IAE) and the integral time absolute error (ITAE) were used to evaluate the performance of the controller. The definitions of IAE and ITAE are as follows:

$$\mathrm{IAE}={\displaystyle \int }\left|e\right|dt$$

(43)

$$\mathrm{ITAE}={\displaystyle \int }t\left|e\right|dt$$

(44)

Table 1. Performance index calculation results.

Controller	Indicator	Joint 1	Joint 2
RBFNN	IAE	1.5978	0.5440
	ITAE	13.4532	3.9454
PIR	IAE	0.4217	0.3476
	ITAE	1.6596	1.5451
DDPGPIR	IAE	0.0866	0.0410
	ITAE	0.0285	0.0848

shows the IAE and ITAE values of RBFNN, PIR, and DDPGPIR. The table shows that DDPGPIR has smaller IAE and ITAE values than PIR and RBFNN. Therefore, DDPGPIR has better adaptability and robustness in the case of friction and external disturbance.

5. Conclusions

An adaptive PIR control method based on deep reinforcement learning is proposed for the n-link robot manipulator system with model uncertainty and time-varying external disturbances. In this method, the parameters of the PIR controller are adjusted and optimized in real time by using the DDPG algorithm. Among them, the adaptive robust term is used to compensate for the uncertainty of the robot manipulator system. In addition, the model-free reinforcement learning method does not need to rely on expert knowledge and human intervention. The agent of the deep neural network can effectively avoid reduction of control accuracy caused by sparse discretization and the curse of dimension caused by dense discretization. In addition, a reward function combining the Gaussian function and the Euclidean distance is designed to ensure efficient and stable learning of the reinforcement learning agent.

The proposed method was applied to control the two-link robot manipulator with a model for uncertainty and external disturbance. The experimental results show that the reward value obtained by the DDPGPIR agent increases gradually with the increase of learning times, and finally tends to be stable and close to the expected reward value, which proves the correctness and stability of the DDPGPIR model. In addition, compared with PIR and RBFNN, DDPGPIR has better adaptability and robustness, and a higher precision trajectory tracking ability, for the uncertainty of the n-link robot manipulator system. At the same time, it is better than PIR and RBFNN in the performance evaluation of IAE and ITAE.

In future work, since the proposed control method can control the n-link robot arm system, this method may be applied to more complex control tasks, such as unmanned aerial vehicles. However, the ability of the control system to handle emergencies remains a thorny issue. Therefore, our follow-up work will continue to carry out in-depth research for this problem.