Adaptive Nonlinear Model Predictive Horizon Using Deep Reinforcement Learning for Optimal Trajectory Planning

Abstract

This paper presents an adaptive trajectory planning approach for nonlinear dynamical systems based on deep reinforcement learning (DRL). This methodology is applied to the authors’ recently published optimization-based trajectory planning approach named nonlinear model predictive horizon (NMPH). The resulting design, which we call ‘adaptive NMPH’, generates optimal trajectories for an autonomous vehicle based on the system’s states and its environment. This is done by tuning the NMPH’s parameters online using two different actor-critic DRL-based algorithms, deep deterministic policy gradient (DDPG) and soft actor-critic (SAC). Both adaptive NMPH variants are trained and evaluated on an aerial drone inside a high-fidelity simulation environment. The results demonstrate the learning curves, sample complexity, and stability of the DRL-based adaptation scheme and show the superior performance of adaptive NMPH relative to our earlier designs.

1. Introduction

Path planning and trajectory tracking control are compelling domains for researchers working with autonomous robotic systems. Some formulations require accurate system dynamics models to design the control and navigation algorithms [1]. However, obtaining accurate models is challenging in practice, especially if the system dynamics are vary by time or task. Changes in system dynamics require updating the system model and/or the associated control and navigation algorithms. For instance, adaptive control designs adjust the controller’s parameters in response to changes in the system dynamics and the environment [2]. Adaptive control methods can be traced back to the 1950s and early 1960s [3]. Richard Bellman showed how dynamic programming is related to the different aspects of adaptation [4], and various adaptive flight control systems from this era are reported in [5]. One of the simplest instances of adaptive control is dynamically adjusting the gains of a PID control law; some techniques proposed by researchers for online PID tuning include [6,7,8,9].

The world is witnessing rapid progress in the use of artificial intelligence (AI) techniques for self-adaptive systems [10]. In particular, some AI-based techniques have generated great interest for adaptive control designs for mobile robots [11,12,13]. One of the most productive paradigms in AI is reinforcement learning (RL), which is a learning method for an agent interacting with its environment [14]. In the literature, RL has been used by researchers as an adaptive control strategy, for instance, a Q-learning-based cruise control method was developed by [15] to control a vehicle’s speed on curved lanes. Q-learning [16] is an RL algorithm that learns the value of an action for a given state of the system. For online tuning purposes, [17] used the Q-learning method to auto-tune fuzzy PI and PD controllers for both single- and multi-input/output systems, while [18] used an actor-critic RL technique to tune the weights of an LQR controller to adjust to different payloads being carried by a robot arm manipulator.

Recent developments in RL have made it possible to use neural networks as approximators of the RL value and policy functions [14]. In general, RL methods that use neural networks in their structure are called deep reinforcement learning (DRL). One class of DRL methods that support continuous-time system models belongs to the actor-critic family [19], including the deep deterministic policy gradient (DDPG) [20], twin delayed deep deterministic (TD3) [21], soft actor-critic (SAC) [22], and asynchronous advantage actor–critic (A3C) [23]) algorithms. Actor-critic methods simultaneously learn policy and value functions that are maintained independently using separate memory structures [14]. The actor is a policy function that selects the best action for the current observations, and the critic is a value function that criticizes the actions made by the actor. The algorithms listed above have recently begun being used to implement adaptive control. For example, the DDPG algorithm was used by [1] for self-tuning gains of PID controllers onboard mobile robots, while [24] utilized the A3C algorithm to tune the gains of a PID controller used for position control of a two-phase hybrid stepping motor. DRL-based algorithms can also be used to autonomously tune the parameters of algorithms other than control, for instance path planning. This will be the focus of the present paper.

Recently, the authors introduced a path planning methodology called nonlinear model predictive horizon (NMPH) [25], which produces optimal, consistent, collision-free, and computationally efficient trajectories that respect the internal and external constraints of a mobile robot (in our case, an aerial drone). By design, the NMPH algorithm compensates for the system’s nonlinearities to reduce or even remove the non-convexity of its underlying optimization problem. This is done by combining the nonlinear plant model with various nonlinear feedback control design methodologies, such as feedback linearization (FBL) [25] and backstepping control (BSC) [26]. The optimization problem embedded within NMPH contains various parameters that affect its cost function, as further explained in Section 2.1. In our previous works, these parameters were selected empirically; however, in the present paper, a new framework is proposed that dynamically adjusts these parameters to optimize the path planning performance in real time. Our approach uses DRL algorithms (DDPG or SAC) to automatically tune the NMPH parameters based on system states and observations from the environment. This framework is called ‘adaptive NMPH’.

The research contributions of this paper are as follows:

• Introducing an adaptive NMPH framework that uses a DRL-based method to tune the parameters of the underlying optimization problem of generating the best possible reference trajectories for the vehicle.
• Designing the RL components (the agent, the environment, and the reward scheme) of the proposed system.
• Implementing two different actor-critic DRL algorithms—the deterministic DDPG approach and the probabilistic SAC algorithm—within the adaptive NMPH framework, comparing them in terms of learning speed and stability.
• Evaluating the performance of the overall system with each of the above DRL algorithms in a lifelike simulation environment.

The remainder of this paper is organized as follows: Section 2 describes the various methodologies used in this work. Section 3 presents the adaptive NMPH framework for trajectory planning. Section 4 evaluates the proposed designs in simulation, and Section 5 concludes the paper and proposes future work directions.

2. Methodologies

This section provides a background on the different methodologies used within the adaptive NMPH framework.

2.1. Nonlinear Model Predictive Horizon Based on Backstepping Control

Nonlinear model predictive horizon (NMPH) was originally proposed by the authors in [25]. NMPH is an optimization-based method used to generate reference trajectories for a closed-loop system. Within its optimization problem, NMPH uses a model of the nonlinear plant, a nonlinear control law (here, backstepping control), and a set of constraints representing input limits plus static and dynamic obstacles in the environment. Connecting the nonlinear plant with the control law aims to reduce the nonlinearity of the overall closed-loop system and consequently the non-convexity of the associated optimization problem. This greatly improves the efficiency of the optimization calculations, which enables real-time trajectory generation to run onboard the drone vehicle.

Consider a nonlinear system with state, input, and output vectors $x\in X\subseteq {\mathbb{R}}^{n_x}$, $u\in U\subseteq {\mathbb{R}}^{n_u}$, and $\xi \in$$\subseteq {\mathbb{R}}^{n_{\xi }}$, respectively. The output vector is assumed to be a subset of the system state, $\subseteq X$. In addition, let $f\left(x\left(n\right),u\left(n\right)\right):X\times U\to X$ be the smooth map that represents the plant dynamics, and $g\left(x\left(n\right),{\xi }_{{}_{ref}}\left(n\right)\right):X\times$$\to U$ the smooth nonlinear control law map.

NMPH is designed to generate estimated reference trajectories ${\widehat{\xi }}_{{}_{ref}}\in$, which will be tracked by the closed-loop system consisting of the plant and control law. As shown in (1), a copy of these closed-loop dynamics is used by the NMPH optimization problem, where the variables used by NMPH are denoted by a $\tilde{\phantom{x}}$ to visually differentiate them from the actual system variables. For instance, within (1), $\tilde{x}$ represents the predicted system state trajectory, and $\tilde{\xi }$ is the predicted output trajectory.

The online NMPH optimization problem to bring the system from a current state x to a terminal stabilization setpoint $x_{ss}$ is shown in Equation (1) [27]. Let $t_n$, $n=0,1,2,\cdots$ represent successive sampling times. At every sampling instant, the optimization treats the following problem for $\tilde{x}$ and ${\widehat{\xi }}_{{}_{ref}}$, running for as long as $\parallel x_{ss}-x\left(t_n\right)\parallel \ge \mathsf{\Delta }$, where $\mathsf{\Delta }\in {\mathbb{R}}^{+}$ is a user-specified tolerance:

$$\begin{array}{cc}\hfill & \underset{\tilde{x},{\widehat{\xi }}_{{}_{ref}}}{argmin}\left(J\left(\tilde{x},{\widehat{\xi }}_{{}_{ref}}\right)=E\left(\tilde{x}\left(t_n+T\right)\right)+{\int }_{t_n}^{t_n+T}L\left(\tilde{x}\left(\tau \right),{\widehat{\xi }}_{{}_{ref}}\left(\tau \right)\right)d\tau \right)\hfill \end{array}$$

(1)

$$\begin{array}{cccc}\hfill \mathrm{subject}\mathrm{to}& \tilde{x}\left(t_n\right)=x\left(t_n\right),\hfill & \hfill & \hfill \end{array}$$

(1a)

$$\begin{array}{cccc}\hfill & \dot{\tilde{x}}\left(\tau \right)=f\left(\tilde{x}\left(\tau \right),\tilde{u}\left(\tau \right)\right),\hfill & \hfill & \hfill \end{array}$$

(1b)

$$\begin{array}{cccc}\hfill & \tilde{u}\left(\tau \right)=g\left(\tilde{x}\left(\tau \right),{\widehat{\xi }}_{{}_{ref}}\left(\tau \right)\right),\hfill & \hfill & \hfill \end{array}$$

(1c)

$$\begin{array}{cccc}\hfill & \tilde{x}\left(\tau \right)\in \mathcal{X},\tilde{u}\left(\tau \right)\in \mathcal{U},\tilde{\xi }\left(\tau \right),{\widehat{\xi }}_{{}_{ref}}\left(\tau \right)\in \mathcal{Z},\hfill & \hfill & \hfill \end{array}$$

(1d)

$$\begin{array}{cc}\hfill & {\mathcal{O}}_i\left(\tilde{x}\right)\le 0,i=1,2,\dots ,p,\hfill \\ \hfill \mathrm{for}& \tau \in [t_n,t_n+T].\hfill \end{array}$$

(1e)

where $\mathcal{X}\subseteq X$, $\mathcal{U}\subseteq U$, and $\mathcal{Z}\subseteq X$ are the constraint sets for the state, input, and output trajectories, respectively, and each ${\mathcal{O}}_i\left(\tilde{x}\right)\le 0$ in (1e) is an inequality constraint corresponding to a detected static or dynamic obstacle within the environment [27]. The stage cost L and terminal cost E functions in (1) are assigned as follows:

$$\begin{array}{cc}\hfill L\left(\tilde{x}\left(\tau \right),{\widehat{\xi }}_{{}_{ref}}\left(\tau \right)\right)& = \parallel \tilde{x}\left(\tau \right)-x_{ss}{\parallel }_{W_x}^2+{\parallel \tilde{\xi }\left(\tau \right)-{\widehat{\xi }}_{{}_{ref}}\left(\tau \right)\parallel }_{W_{\xi }}^2\hfill \end{array}$$

(2a)

$$\begin{array}{cc}\hfill E\left(\tilde{x}\left(t_n+T\right)\right)& = \parallel \tilde{x}\left(t_n+T\right)-x_{ss}{\parallel }_{W_T}^2\hfill \end{array}$$

(2b)

where the errors in (2a) and (2b) are weighted by matrices $W_x\in {\mathbb{R}}^{n_x\times n_x}$, $W_{\xi }\in {\mathbb{R}}^{n_{\xi }\times n_{\xi }}$, and $W_T\in {\mathbb{R}}^{n_x\times n_x}$, which in this work will be adaptively tuned using DRL algorithms.

The optimization problem in (1) begins with measuring the current state of the physical system $x\left(t_n\right)$ at time $t_n$. The cost function $J\left(\tilde{x},{\widehat{\xi }}_{{}_{ref}}\right)$ is then minimized over the prediction horizon $[t_n,t_n+T]$ subject to constrains (1b), (1c), and (1e) to provide a prediction of the values of $\tilde{x}$ and ${\widehat{\xi }}_{{}_{ref}}$. Finally, either the estimated reference trajectory ${\widehat{\xi }}_{{}_{ref}}$ or the predicted output trajectory $\tilde{\xi }$ (as both converge to each other) is input into the closed-loop system for tracking. This process is repeated in real time at a user-specified rate until the plant reaches the desired terminal setpoint. Details about the NMPH approach can be found in our recent works [25,26].

In this work, the nonlinear backstepping control law is used within the NMPH optimization problem as a constraint in (1c). The detailed development and implementation of the BSC technique within NMPH, as well as its advantages over the earlier FBL-based design [25], are described in our recent work [26].

The NMPH trajectory planning algorithm receives terminal points from a modular global motion planner [27]. The global motion planner generates terminal points within unexplored areas of an incrementally built-up volumetric map of the environment [28,29]. These terminal points, along with the current pose of the vehicle, the constraints representing the closed-loop system dynamics and environmental obstacles (which are extracted from the volumetric map), and the entries of the weighting matrices (which in the present design are adjusted online by a DRL algorithm) are sent to the NMPH optimization problem in order to calculate optimal trajectories between the vehicle’s current pose and the next terminal point. The results are then used as reference trajectories by the vehicle’s low-level flight controller.

2.2. Deep Reinforcement Learning Overview

This section covers the preliminaries of reinforcement learning, then describes the DDPG and SAC algorithms used within the adaptive NMPH frameworks.

2.2.1. Reinforcement Learning Preliminaries

A reinforcement learning (RL) system is composed of an agent that interacts with an environment in a sampling-based manner. Assuming the environment is fully observed, at each time sample the agent observes the environment state $s\in \mathcal{S}$, applies the action $a\in \mathcal{A}$ decided by a policy, and receives a scalar reward $r:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$, where $\mathcal{S}$ and $\mathcal{A}$ are the environment state space and the action space, respectively. In our work, we consider continuous action spaces with a real-valued vector $a\in {\mathbb{R}}^n$. The main components of an RL framework are depicted in Figure 1.

The agent’s policy can be deterministic (denoted by $\mu \left(s\right)$), or stochastic (denoted by $\pi (·|s)$). In deep RL, we parameterize the policy and represent it using a universal function approximator realized by a neural network. The parameters (representing the weights and biases of the policy’s neural network) are denoted by $\theta$, and the corresponding policies for the deterministic and stochastic cases are denoted by ${\mu }_{\theta }\left(s\right)$ and ${\pi }_{\theta }(·|s)$, respectively.

We consider a stochastic environment with transition probability function $p:\mathcal{S}\times \mathcal{R}\times \mathcal{S}\times \mathcal{A}\to [0,1]$, where $p(s^{\prime },r|s,a)$ is the probability of transition from the current state s and action a to the next state $s^{\prime }$ with reward $r\in \mathcal{R}$. In addition, we define the ‘return’ as the expected weighted sum of future rewards $R={\sum }_{t=0}^{\infty }{\gamma }^{t}r(s,a)$, where $r(s,a)$ is the reward function and $0\le \gamma \le 1$ is the discounting factor. The main objective in RL is to find a policy that maximizes the expected sum of rewards $J=\underset{\tau \sim \pi }{\mathbb{E}}\left[R\right]$, where $\tau =(s_0,a_0,s_1,a_1,\dots )$ is the trajectory sequence of states and actions in the RL system.

The state-action value function (also known as the Q-function) specifies the expected return of an agent after performing an action a at a state s by following a policy $\pi$ or $\mu$. The Q-function can be described by a Bellman equation [14].

Many recent advances in deep reinforcement learning consider a replay buffer (also known as an experience buffer or experience replay) during the learning process. The replay buffer is a memory that collects the previous experience tuples $(s,a,r,s^{\prime })\in \mathcal{B}$, in which the agent uses them to increase the computational efficiency and speed up learning [30].

We will now review the DDPG and SAC deep reinforcement learning algorithms used within our proposed adaptive NMPH frameworks.

2.2.2. Deep Deterministic Policy Gradient

Deep deterministic policy gradient (DDPG) [20] is a model-free deep reinforcement learning technique that is designed for applications with deterministic action spaces. It uses stored experiences in a replay buffer to concurrently learn a Q-function and a policy. DDPG is classified as an actor-critic technique, where the actor is a policy network that receives the state of the environment and provides continuous action to the system, while the critic is a Q-function network that inputs a state and action pair and outputs a Q-value.

DDPG seeks to find the optimal action-value function $Q^{*}(s,a)$ followed by the optimal action $a^{*}\left(s\right)$, where $a^{*}\left(s\right)={arg\; max}_a{Q}^{*}(s,a)$. As a deep reinforcement learning approach, DDPG uses universal function approximators represented by neural networks to learn $Q^{*}(s,a)$ and $a^{*}\left(s\right)$. Consider a neural network approximator $Q_{\varphi }(s,a)$ (also known as a Q-network) with parameters $\varphi$, where the objective is to make the approximator as close as possible to the optimal action-value function written in the form of a Bellman equation. The associated mean square Bellman error (MSBE [31]) function is defined as follows:

$$J_Q(\varphi ,\mathcal{B})=\underset{(s,a,r,s^{\prime })\sim \mathcal{B}}{\mathbb{E}}\left[{\left(Q_{\varphi }(s,a)-\left(r+\gamma \underset{a^{\prime }}{max}{Q}_{\varphi }(s^{\prime },a^{\prime })\right)\right)}^2\right]$$

(3)

where a random batch of data $(s,a,r,s^{\prime })$ from the replay buffer $\mathcal{B}$ is used for each update. The goal is to minimize the loss in (3) by performing a gradient descent of the MSBE $J_Q(\varphi ,\mathcal{B})$.

As shown in (3), the neural network parameters represented by $\varphi$ are used for both the action-value function approximator $Q_{\varphi }(s,a)$ and the network that estimates $Q_{\varphi }(s^{\prime },a^{\prime })$, which uses the next states and actions. Unfortunately, this makes it impossible for the gradient descent to converge. To tackle this issue, a time delay is added to the network parameters $\varphi$ for $Q_{\varphi }(s^{\prime },a^{\prime })$. The adjusted network is called the target Q-network $Q_{{\varphi }_{\mathrm{targ}}}(s^{\prime },a^{\prime })$ with parameters ${\varphi }_{\mathrm{targ}}$. A copy of the Q-network $Q_{\varphi }(s^{\prime },a^{\prime })$ is used for the target Q-network $Q_{{\varphi }_{\mathrm{targ}}}(s^{\prime },a^{\prime })$, where the latter uses the weighted average of the model parameters ${\varphi }_{\mathrm{targ}}\leftarrow \rho {\varphi }_{\mathrm{targ}}+(1-\rho )\varphi$ to stabilize Q-function learning [32]. It should be noted that the parameters of the target Q-network are not trained. However, they are periodically synchronized with the original Q-network’s parameters.

The MSBE function given in (3) contains a maximization term for the Q-value. One way to perform this maximization is to apply the optimal action $a^{*}\left(s\right)$. This can be achieved by creating another approximator for the policy ${\mu }_{\theta }\left(s\right)$ with parameters $\theta$ and maximizing the associated Q-function with regard to the replay buffer $\mathcal{B}$. This new policy also requires a time delay to stabilize its learning. Therefore, a target policy ${\mu }_{{\theta }_{\mathrm{targ}}}\left(s\right)$ is introduced to maximize $Q_{{\varphi }_{\mathrm{targ}}}$. The Bellman equation, MSBE, and policy learning function are respectively given by

$$\begin{array}{ccc}\hfill y(r,s^{\prime })& =& r+\gamma \stackrel{\mathrm{target}\mathrm{Q}-\mathrm{network}}{\overbrace{Q_{{\varphi }_{\mathrm{targ}}}(s^{\prime },\underset{\mathrm{target}\mathrm{policy}\mathrm{network}}{\underbrace{{\mu }_{{\theta }_{\mathrm{targ}}}(s^{\prime })}})}}\hfill \end{array}$$

(4)

$$\begin{array}{ccc}\hfill J_Q(\varphi ,\mathcal{B})& =& \underset{(s,a,r,s^{\prime })\sim \mathcal{B}}{\mathbb{E}}\left[{\left(\underset{\mathrm{Q}-\mathrm{network}}{\underbrace{Q_{\varphi }(s,a)}}-y(r,s^{\prime })\right)}^2\right]\hfill \end{array}$$

(5)

$$\begin{array}{ccc}\hfill J_{\mu }(\theta ,\mathcal{B})& =& \underset{s\sim \mathcal{B}}{\mathbb{E}}\left[Q_{\varphi }\left(s,{\mu }_{\theta }\left(s\right)\right)\right]\hfill \end{array}$$

(6)

Practically, for a random sample $B=\left\{(s,a,r,s^{\prime })\right\}$ from the replay buffer $\mathcal{B}$ with cardinality $\left|B\right|$, Equations (5) and (6) can be expressed as

$$\begin{array}{ccc}\hfill J_Q(\varphi ,B)& =& \frac{1}{\left|B\right|}\sum _{(s,a,r,s^{\prime })\in B}{\left(Q_{\varphi }(s,a)-y(r,s^{\prime })\right)}^2\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill J_{\mu }(\theta ,B)& =& \frac{1}{\left|B\right|}\sum _{s\in B}{Q}_{\varphi }\left(s,{\mu }_{\theta }\left(s\right)\right)\hfill \end{array}$$

(8)

During training, Ornstein–Uhlenbeck noise is added to the action vector to enhance the exploration of the DDPG policy [31]. The pseudo-code summarizing the DDPG process is given in Algorithm 1.

Algorithm 1 Deep Deterministic Policy Gradient.

1: Initialize: $\theta$, $\varphi$, $\mathcal{B}\leftarrow \varnothing$

2: Set ${\theta }_{\mathrm{targ}}\leftarrow \theta$, ${\varphi }_{\mathrm{targ}}\leftarrow \varphi$

3: repeat

4:       Observe the state s

5:       Find and apply noise to the action $a={\mu }_{\theta }\left(s\right)+{\eta }_{{}_{\mathrm{OU}-\mathrm{noise}}}$

6:       Apply a by the agent

7:       Observe the next state $s^{\prime }$ and calculate the reward r

8:       Store $(s,a,r,s^{\prime })$ in the replay buffer $\mathcal{B}$

9:       for a given number of episodes do

10:           Obtain a random sample $B=\left\{(s,a,r,s^{\prime })\right\}$ from $\mathcal{B}$

11:           Compute Bellman function $y(r,s^{\prime })$

12:           Update the Q-function by applying gradient descent to MSBE: ${\nabla }_{\varphi }{J}_Q(\varphi ,B)$

13:           Update the policy by applying gradient ascent to (8): ${\nabla }_{\theta }{J}_{\mu }(\theta ,B)$

14:           Update the parameters of the target networks: $$\begin{gathered} \left\{\begin{array}{c}{\varphi }_{\mathrm{targ}}\leftarrow \rho {\varphi }_{\mathrm{targ}}+(1-\rho )\varphi \\ {\theta }_{\mathrm{targ}}\leftarrow \rho {\theta }_{\mathrm{targ}}+(1-\rho )\theta \hfill \end{array}\right. \end{gathered}$$

15: until convergence

The hyperparameters used for the DDPG algorithm are the number of training episodes, target update factor $\left(\rho \right)$, actor and critic network learning rates, replay buffer size, random batch size, and discount factor value. The sensitivity to the hyperparameter values and the interaction between the Q-value and policy approximator ${\mu }_{\theta }\left(s\right)$ make analyzing the stability and convergence of DDPG difficult tasks [33], especially when using high-dimensional nonlinear universal function approximators [34]. Moreover, DDPG is expensive in terms of its sample complexity, which is measured by the number of training samples needed to complete the learning process.

An alternative approach that overcomes the issues of the DDPG algorithm is soft actor-critic (SAC) [22,34], a probabilistic DRL algorithm, which is considered next.

2.2.3. Soft Actor-Critic

Soft actor-critic (SAC) is a model-free deep reinforcement learning technique that obtains a stochastic policy by maximizing its expected return and entropy [22]. Maximizing the expected entropy in the policy leads to broader exploration in complicated domains, which enhances the sampling efficiency, increases robustness, and guards against convergence to a local maximum [31]. SAC is a probabilistic framework that builds on Soft Q-learning within an actor-critic formulation.

SAC involves simultaneously learning two Q-functions $Q_{{\varphi }_1}$, $Q_{{\varphi }_2}$ using two different Q-networks, as well as a stochastic policy ${\pi }_{\theta }$ using a policy network. Both Q-functions use a modified MSBE (known as soft-MSBE), to be presented in (10), where the minimum Q-value of both functions is used to update the policy [21]. SAC employs a ‘target network’ associated with each Q-network to enhance the stability of the learning process, where both target Q-networks are copies of the corresponding Q-network, but employ weighted averaging on the network parameters during training. Because of the policy’s stochastic nature, SAC uses the current policy to obtain the next state-action values without needing to have a target policy [31]. In addition, the stochastic nature of the exploration process means it’s not necessary to artificially introduce noise, as was done in the deterministic DDPG.

The objective of SAC is to maximize the sum of the expected return and entropy. The Bellman equation within its Q-value function thus includes the expected entropy of the policy as follows:

$$Q_{\pi }(s,a)\approx r+\gamma \left(Q_{\pi }(s_{\mathcal{B}}^{\prime },a_{\pi }^{\prime })-\alpha log\pi \left(a_{\pi }^{\prime }\right|s_{\mathcal{B}}^{\prime })\right)$$

(9)

where $\alpha$ is the coefficient that regulates the trade-off between the expected entropy and return, $s_{\mathcal{B}}^{\prime }$ indicates that the replay buffer is used to obtain the expectation of the future states, and $a_{\pi }^{\prime }\sim \pi (·|s^{\prime })$ indicates that the current policy is used to obtain future actions. For simplicity of notation, we will denote $s_{\mathcal{B}}^{\prime }$ by $s^{\prime }$ and $a_{\pi }^{\prime }$ by $a^{\prime }$ in the sequel.

Two Bellman residuals are used within SAC [22], referred to as soft-MSBEs. In addition to the policy network ${\pi }_{\theta }$, each soft-MSBE includes a Q-network and two target Q-networks in its calculation as follows:

$$J_Q({\varphi }_i,\mathcal{B})=\underset{(s,a,r,s^{\prime },a^{\prime })\sim \mathcal{B}}{\mathbb{E}}\left[{\left(Q_{{\varphi }_i}(s,a)-y(r,s^{\prime },a^{\prime })\right)}^2\right],i=1,2$$

(10)

and their Bellman equation forms are

$$y(r,s^{\prime },a^{\prime })=r+\gamma \left(\underset{j=1,2}{min}{Q}_{{\varphi }_{\mathrm{targ},j}}(s^{\prime },a^{\prime })-\alpha log{\pi }_{\theta }\left(a^{\prime }\right|s^{\prime })\right),a^{\prime }\sim {\pi }_{\theta }(·|s^{\prime })$$

(11)

Similar to DDPG, the Q-functions are updated using gradient descent, while gradient ascent is utilized to update the policy network.

The policy should maximize the state-value function $V_{\pi }\left(s\right)$, defined as follows:

$$V_{\pi }\left(s\right)=\underset{a\sim \pi }{\mathbb{E}}\left[Q_{\pi }(s,a)-\alpha log\pi \left(a\right|s)\right]$$

(12)

which represents the expected return when starting from a state s and following a policy $\pi$.

For the optimal value of the action, we can employ reparameterization [22,31] to obtain a continuous action from a deterministic function that represents the policy. The function is expressed by the state and additive Gaussian noise as follows:

$$a_{\theta }(s,\xi )=tanh\left({\mu }_{\theta }\left(s\right)+{\sigma }_{\theta }\left(s\right)\xi \right),\xi \sim \mathcal{N}\left(0,\mathrm{diag}(1,\dots ,1)\right).$$

(13)

The policy optimization can be performed by maximizing the Q-function, which implicitly maximizes the entropy of the trajectory. Using the computed value of the action from (13), the function to be maximized is

$$J_{\pi }(\theta ,\mathcal{B})=\underset{s\sim \mathcal{B},\xi \sim \mathcal{N}}{\mathbb{E}}\left[\underset{j=1,2}{min}{Q}_{{\varphi }_j}(s,a_{\theta }(s,\xi ))-\alpha log{\pi }_{\theta }\left(a_{\theta }(s,\xi )\right|s)\right]$$

(14)

and the optimum policy can be obtained by finding ${arg\; max}_{\theta }{J}_{\pi }(\theta ,\mathcal{B})$ using gradient ascent. For a random sample $B=\left\{(s,a,r,s^{\prime },a^{\prime })\right\}$ from the buffer $\mathcal{B}$, Equations (10) and (14) can be expressed as follows:

$$\begin{array}{ccc}\hfill J_Q({\varphi }_i,B)& =& \frac{1}{\left|B\right|}\sum _{(s,a,r,s^{\prime })\in B}{\left(Q_{{\varphi }_i}(s,a)-y(r,s^{\prime },a^{\prime })\right)}^2,i=1,2\hfill \end{array}$$

(15)

$$\begin{array}{ccc}\hfill J_{\mu }(\theta ,B)& =& \frac{1}{\left|B\right|}\sum _{s\in B}\left(\underset{j=1,2}{min}{Q}_{{\varphi }_j}\left(s,a_{\theta }(s,\xi )\right)-\alpha log{\pi }_{\theta }\left(a_{\theta }(s,\xi )|s\right)\right)\hfill \end{array}$$

(16)

The pseudo-code for the SAC algorithm is provided in Algorithm 2.

Algorithm 2 Soft Actor-Critic.

1: Initialize: $\theta$, ${\varphi }_i$, $\alpha$, $\mathcal{B}\leftarrow \varnothing ,i=1,2$

2: Set ${\varphi }_{\mathrm{targ},i}\leftarrow {\varphi }_i$

3: repeat

4:       Observe the state s

5:       Find the action $a\sim {\pi }_{\theta }(·|s)$, and apply it through the agent

6:       Observe the next state $s^{\prime }$ and the reward r

7:       Find the next action $a^{\prime }\sim {\pi }_{\theta }(·|s^{\prime })$

8:       Store $(s,a,r,s^{\prime },a^{\prime })$ in the replay buffer $\mathcal{B}$

9:       for a given number of episodes do

10:           Obtain a random sample $B=\left\{(s,a,r,s^{\prime },a^{\prime })\right\}$ from $\mathcal{B}$

11:           Compute Bellman functions $y(r,s^{\prime },a^{\prime })$ in (11) and find the soft-MSBEs (10)

12:           Apply gradient descent on the soft-MSBEs: ${\nabla }_{{\varphi }_i}{J}_Q({\varphi }_i,B)$

13:           Reparametrize the action: $a_{\theta }(s,\xi )=tanh\left({\mu }_{\theta }\left(s\right)+{\sigma }_{\theta }\left(s\right)\xi \right)$

14:           Apply gradient ascent on the policy: ${\nabla }_{\theta }{J}_{\mu }(\theta ,B)$

15:           Apply gradient descent to tune $\alpha$: ${\nabla }_{\alpha }J\left(\alpha \right)$

16:           Update target networks: ${\varphi }_{\mathrm{targ},i}\leftarrow \rho {\varphi }_{\mathrm{targ},i}+(1-\rho ){\varphi }_i$

17: until convergence

3. Adaptive Trajectory Planning Framework

In this section, we present the DRL-based adaptive framework used to adjust the gains of the NMPH trajectory planning algorithm. First, we will describe the agent and environment involved in the DRL problem; then, we will present two adaptive NMPH architectures based on the DDPG and the SAC algorithm, respectively.

3.1. Agent and Environment Representations

Figure 2 shows the main components of the adaptive NMPH system. The environment is an autonomous drone that flies within an incrementally built-up 3D volumetric map of the surroundings. The drone uses the NMPH algorithm for planning local trajectories between the current pose and a terminal setpoint provided by the exploration algorithm presented in [27]. As covered in Section 2.1, the NMPH optimization process (blue box in Figure 2) contains models of the nonlinear system dynamics and nonlinear control law, as well as constraints representing actuation limits and environmental obstacles. The onboard flight control system tracks the optimum reference trajectories generated by the NMPH.

From an RL perspective, at each episode the drone is commanded to fly through k terminal setpoints. Hence, each episode consists of k iterations. Following each iteration, three observations are sent to the agent: initial velocity $v_o$, angle $\phi$ between the initial velocity vector $v_o$ and the vector $\overrightarrow{r}=p_{ss}-p_o$ running from the initial point $p_o$ to the terminal point $p_{ss}$, and the distance $|\overrightarrow{r}|$.

A sketch of the observations $\{v_o,\phi ,|\overrightarrow{r}\left|\right\}$ for one iteration is given in Figure 3.

Our objective is to tune the NMPH parameters online by using reinforcement learning to maximize the total reward. This reward is a function of the tracking performance by the drone of the reference path generated by the NMPH algorithm, which consists of three indicators:

• Trajectory tracking reward, which reflects how well the flight trajectory matches the generated reference. The trajectory tracking reward is calculated as follows:

  $$r_{traj}=\left\{\begin{array}{cc}-\frac{r_{t,max}}{r_{t,th}}{e}_{t,\mathrm{RMS}}+r_{t,max},\hfill & \mathrm{for}{e}_{t,\mathrm{RMS}}\le r_{t,th}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

  (17)

  where $e_{t,\mathrm{RMS}}$ is root-mean-square (RMS) error between the generated and flight trajectories, and $r_{t,max}$ and $r_{t,th}$ are the maximum and threshold values of the trajectory tracking reward, respectively.
• Terminal setpoint reward, which reflects how close the ending point of the flight trajectory is to the terminal setpoint of the reference trajectory. The terminal setpoint reward is calculated as follows:

  $$r_{ss}=\left\{\begin{array}{cc}-\frac{r_{s,max}}{r_{s,th}}{e}_{{}_{ss}}+r_{s,max},\hfill & \mathrm{for}{e}_{{}_{ss}}\le r_{s,th}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

  (18)

  where $e_{ss}=\parallel p_{ss}-{\widehat{\xi }}_{ref}^{pos}(t_n+T)\parallel$ is the error between the terminal point and the final point of the reference trajectory generated by the NMPH, and $r_{s,max}$, $r_{s,th}$ are the maximum and threshold values of this reward, respectively.
• Completion reward, which reflects how far the drone travels along its prescribed flight trajectory in the associated time interval. This is given by the following:

  $$r_c=\left\{\begin{array}{cc}-\frac{r_{c,max}}{r_{c,th}}{e}_{{}_c}+r_{c,max},\hfill & \mathrm{for}{e}_{{}_c}\le r_{c,th}\hfill \\ -5,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

  (19)

  where $e_c=\parallel p_{ss}-p{|}_{{}_{t_n+T}}\parallel$ is the error between the drone’s position at $t_n+T$ and the flight trajectory’s endpoint, while $r_{c,max}$, $r_{c,th}$ are the maximum and threshold values of the completion reward, respectively. We place more importance on this factor by reducing the total reward ($r_c<0$) whenever the error $e_c$ exceeds the assigned threshold value $r_{c,th}$. Consequently, the overall algorithm will give priority to ensuring the drone reaches the desired setpoint in the allotted timeframe.

3.2. DRL-Based Adaptive NMPH Architecture

The objective of adaptive NMPH is to integrate deep learning—in this case, an actor-critic method (DDPG or SAC)—within the NMPH optimization problem to adaptively tune the NMPH parameters and thus provide the best possible reference flight trajectories for the drone.

The structures of the NMPH-DDPG and NMPH-SAC algorithms are illustrated in Figure 4 and Figure 5, respectively. Both DRL structures contain two parts, the actor and the critic. The actor contains the policy network, which selects the action that maximizes the total reward (a function of the state of the vehicle and environment) and subsequently improves the policy based on feedback from the critic. A target policy network is used in DDPG to obtain a stable learning process, while SAC does not need a target network because of its probabilistic nature. The critic is responsible for policy evaluation; within DDPG, it consists of a Q-network and a target Q-network, while in SAC, it is composed of two Q-networks, two target Q-networks, and an optimization problem for $\alpha$ tuning. Both DDPG and SAC employ a replay buffer to store previous experiences, which are used to refine the actor and critic networks. The policy evaluation and improvement processes within DDPG and SAC are explained in Section 2.2.1 and Section 2.2.2 and depicted in Figure 4 and Figure 5, respectively.

The action produced by the actor is a vector of positive values representing the entries of the weighting matrices used in the NMPH optimization problem. Using these, NMPH calculates its stage and terminal cost functions used to perform its optimization and generates the estimated reference trajectory ${\widehat{\xi }}_{{}_{ref}}$. This result is used by the drone’s flight control system, and the vehicle’s resulting trajectory is used to calculate the observations $\{v_o,\phi ,|\overrightarrow{r}\left|\right\}$ and the total reward $r_t=r_{{}_{traj}}+r_{{}_{ss}}+r_c$ sent to the replay buffer to be used in the learning process.

4. Implementation and Evaluation

This section evaluates the effectiveness of tuning the NMPH parameters in real time via two DRL algorithms (DDPG and SAC). It also assesses the sample complexity and stability of both methods.

The overall architecture is implemented within the robot operating system (ROS) [35], which handles the interactions between the various subsystems, including physics simulation, optimization calculations, and DRL algorithm. The AirSim open-source simulator [36] is used to simulate the physics of the drone and provides photo-realistic environment data. For optimization, the ACADO Toolkit [37] is used to solve the NMPH’s optimization problem in real time. The TensorFlow [38] and Keras [39] libraries are used to train the deep neural networks within the DDPG and SAC algorithms. In addition, the TensorLayer library [40] was used to tailor the SAC algorithm to our application. TensorLayer is a TensorFlow-based package that offers various RL and DRL modules for learning system implementations.

As stated in Section 3.2, three observations of the system are fed back to the individual neural networks: $v_o$, $\phi$, and $|\overrightarrow{r}|$. DDPG is very sensitive to hyperparameters when the action space has a high dimension, in which case achieving stable learning becomes challenging. Therefore, we employ only three actions corresponding to the weights of the NMPH optimization dealing with position states. The learning process for the three weight factors $\{w_1=w_x,w_2=w_y,w_3=w_z\}$ is performed using DDPG and SAC in parallel for comparison purposes.

Each episode is composed of a sequence of iterations, where each iteration represents a trajectory between two endpoints (terminal points). At the start of each iteration, the velocity vector of the drone $v_o$, the angle $\phi$ between the velocity and endpoint-to-endpoint vectors, and the distance $|\overrightarrow{r}|$ between endpoints are calculated, followed by the errors $\{e_{t,\mathrm{RMS}},e_{ss},e_c\}$ and the total reward at the end of the iteration. All this data is stored in the replay buffer. In order to cover a wider portion of the state and action spaces of the system, the initial velocity is randomly selected at the beginning of each episode.

The structures of the actor-critic DRL (policy and Q-networks) for DDPG and SAC algorithms are presented in Figure 6 and Figure 7, respectively. Each network is composed of an input layer, multiple hidden layers, and an output layer. Figure 6 and Figure 7 depict our neural network designs in terms of the layer structure of each network and the number of nodes in each layer. The policy networks in the actor are responsible for generating actions that maximize the total reward based on observations of the environment, while the Q-networks in the critic compute a Q-value that is used for policy improvement. For DDPG, four networks are used: a policy network, a Q-network (depicted in Figure 6), a target policy network, and a target Q-network. The target networks are replicas of the policy and Q-networks with a delay added to their parameters. Meanwhile, SAC consists of five networks: a policy network, two Q-networks, and two target Q-networks. The SAC’s policy and Q-network structures are shown in Figure 7.

Figure 8 shows the average episodic reward during the training processes of the DDPG and SAC architectures. In this comparison, each framework is learning to optimize the values of only three actions, which represent the entries of the weight matrix corresponding to the position states within the NMPH optimization problem.

To enhance DDPG performance in terms of sample complexity and its sensitivity to hyperparameters, we propose and apply a ‘pre-exploration’ technique, which traverses the RL problem spaces before the training process is started. Pre-exploration is performed by applying a set of predefined actions and considering a random system state for each action. The collected experiences of the pre-exploration process are then stored in the replay buffer, which is used during the training process. It was found that using this technique helps DDPG to improve convergence and stability over the case without pre-exploration, as can be seen from Figure 8. Conversely, a number of episodes must be spent for pre-exploration, which delays the learning process in the real-time adaptation. Note that the results shown in Figure 8 also show that SAC generally outperforms DDPG (either with or without pre-exploration) in terms of learning speed. In addition, during the training process, SAC showed noticeably better learning stability relative to DDPG with regard to the process of selecting the hyperparameter values for each algorithm.

To test the performance of the SAC approach in a higher-dimensional setting, the number of actions was increased to 12 to estimate the weight matrix entries corresponding to the position, velocity, and acceleration states $\{w_x,w_y,w_z,w_{\psi },w_{\dot{x}},w_{\dot{y}},w_{\dot{x}},w_{\dot{\psi }},w_{\ddot{x}},w_{\ddot{y}},w_{\ddot{x}},w_{\ddot{\psi }}\}$ within the NMPH optimization problem. Figure 9 shows the resulting training curve of SAC; DDPG failed to complete the learning process in this case. The effect of increasing the number of NMPH parameters being tuned can be seen by comparing the SAC training curves in Figure 8 and Figure 9 in terms of the average episodic reward. In the 12-parameter trial, SAC has better training performance than in the 3-parameter case, which is because the former covers a larger action space and consequently provides better solutions of the NMPH optimization problem.

To test the trajectory planning performance of NMPH with and without the proposed adaptation scheme, four different flight tests were performed within the AirSim simulation environment. For the second case, the weighting matrices within NMPH used fixed parameters, which were used as the initial values in the DRL-based adaptation method.

Table 1. Comparison between the conventional NMPH design (fixed values of the NMPH parameters) and the adaptive NMPH-SAC approach, for different flight trials.

	Average Error	Zigzag Pattern	Square Pattern	Ascending Square Pattern	Random Setpoints (Exploration)
Fixed NMPH	$e_{t,\mathrm{RMS}}$	$0.11353$	$0.09758$	$0.10741$	$0.09646$
parameters	$e_{ss}$	$0.08659$	$0.07547$	$0.07663$	$0.07339$
	$e_c$	$0.12033$	$0.06426$	$0.07413$	$0.07739$
Adaptive	$e_{t,\mathrm{RMS}}$	$0.08877$	$0.08495$	$0.09212$	$0.06749$
NMPH-SAC	$e_{ss}$	$0.01029$	$0.00919$	$0.01046$	$0.01150$
	$e_c$	$0.04400$	$0.04419$	$0.04952$	$0.05874$

provides a comparison between the conventional NMPH design with fixed parameter values and the adaptive NMPH-SAC design. The comparison is based on the average of the error metrics discussed in Section 3.1, namely $e_{t,\mathrm{RMS}}$, $e_{{}_{ss}}$, and $e_{{}_c}$. Each flight trajectory consists of ten trials, and each trial includes five iterations. The initial velocity and drone orientation were selected randomly at the beginning of each trial. The first trial uses a zigzag pattern, which consists of five paths, each with length of 5.6 m. For the second trial (square pattern), the side length was 5 m. For the third trial (ascending square pattern), the elevation gain was set to 1 m. The fourth trial involved a set of position setpoints provided by a graph-based exploration algorithm (see [27] for the complete details). As shown in Table 1, the flight performance obtained with the adaptive NMPH is much better than the one from the non-adaptive (conventional) NMPH. The reason for this is that real-time adaptation of NMPH parameters works better than using a single set of fixed values when performing a variety of different flying trajectories.

To show how the values of the NMPH parameters are adjusted online using SAC, Figure 10 and Figure 11 present the results of a flight through 20 randomly generated setpoints. Figure 10 depicts the values of the observations $v_o$, $\phi$, and $|\overrightarrow{r}|$ at the beginning of each iteration, and Figure 11 shows the changing values of the NMPH weighting matrix entries. An animation of this test showing the vehicle’s flight trajectory and corresponding online calculation outputs is available as a supplementary video file.

5. Conclusions and Future Work

This paper presented a DRL-based adaptive scheme to tune the optimization parameters of our previously proposed NMPH trajectory planning approach. The overall design aims to provide the best-performing flight trajectory generation for an aerial drone across a wide range of flight patterns and environments by tuning these parameters in real-time flights instead of selecting them a priori. The adaptation scheme was implemented through two different actor-critic DRL algorithms—the deterministic DDPG and the probabilistic SAC.

The two variants of DRL-based NMPH were trained and tested on an aerial drone in a simulation environment. The results showed a marked improvement in flight performance when using the adaptive NMPH-DDPG and NMPH-SAC over the conventional NMPH. Comparisons between DDPG and SAC showed that the latter outperforms the former in terms of learning speed, ability to handle a larger set of tuning parameters, and overall flight performance.

The pros, cons, and limitations of this study are summarized as follows:

• Pros:

  -

  The proposed design is able to dynamically adjust the parameters of the optimization problem online during flight, which is preferable to tuning them before flight and evaluating the resulting performance afterwards.

  -

  The DRL model can adapt the gains of the optimization problem in response to changes in the vehicle, such as new payload configurations or replaced hardware components.

• Cons:

  -

  DRL algorithms employ a large number of hyperparameters. While SAC is less sensitive to hyperparameters than DDPG, finding the best combination of these parameters to achieve fast training is a challenging task.

• Limitations:

  -

  The present study was performed entirely within a simulation environment and does not include hardware testing results.

Future work will include implementing NMPH-SAC onboard our hardware drone and testing its performance in a variety of real-world environments, as well as using the DRL algorithms for disturbance and parameter estimation.