Multi-Queue-Based Offloading Strategy for Deep Reinforcement Learning Tasks

Abstract

With the boom in mobile internet services, computationally intensive applications such as virtual and augmented reality have emerged. Mobile edge computing (MEC) technology allows mobile devices to offload heavy computational tasks to edge servers, which are located at the edge of the network. This technique is considered an effective approach to help reduce the burden on devices and enable efficient task offloading. This paper addresses a dynamic real-time task-offloading problem within a stochastic multi-user MEC network, focusing on the long-term stability of system energy consumption and energy budget constraints. To solve this problem, a task-offloading strategy with long-term constraints is proposed, optimized through the construction of multiple queues to maintain users’ long-term quality of experience (QoE). The problem is decoupled using Lyapunov theory into a single time-slot problem, modeled as a Markov decision process (MDP). A deep reinforcement learning (DRL)-based LMADDPG algorithm is introduced to solve the task-offloading decision. Finally, Experiments are conducted under the constraints of a limited MEC energy budget and the need to maintain the long-term energy stability of the system. The results from simulation experiments demonstrate that the algorithm outperforms other baseline algorithms in terms of task-offloading decisions.

1. Introduction

Over the past few years, the number of applications has grown exponentially, and the variety of innovative applications has exploded [1]. It has become increasingly challenging for computing resources to meet the demands of application devices, which cannot process tasks locally with low latency. The traditional way of offloading tasks in cloud computing suffers from high latency, network congestion, and long transmission distances, which no longer meet the demands of compute-intensive and latency-sensitive tasks [2]. MEC provides computing power closer to the user by pushing computing resources to the edge of the network, enabling faster and more responsive task processing [3,4,5]. Users can place tasks on edge servers for processing by means of task offloading, thus not only relying on local device computation [6]. In contrast to traditional cloud computing, MEC is less costly for data transfer and easier for task offloading [7].

Although MEC is gradually moving in a prominent direction, there are still many challenges. In mobile edge computing scenarios, the dynamic and random tasks to be processed are difficult to predict accurately. There is a need to consider how to offload tasks to MEC edge servers while maintaining system performance benefits [8]. When performing task offloading, the offloading incurs a series of overheads. The energy consumption and latency in performing task offloading are particularly important. Therefore, we need to find a task-offloading strategy to optimize energy consumption and latency.

In order to find a suitable task-offloading strategy, current research delves deeply and has conducted a large number of studies on computational task-offloading strategies. Typically, this involves solving mixed-integer nonlinear programming (MINLP) problems. Since decoupling these problems is very complex, most studies have focused on how to reduce the complexity of the algorithms, and many heuristic algorithms have been proposed [9,10,11,12,13,14,15]. However, finding a superior task-offloading strategy through heuristic algorithms requires complex and repetitive iteration. In a realistic scenario, if the parameters change, the MINLP problem needs to be solved all over again, resulting in computational redundancy. Therefore, the costs incurred by using traditional optimization algorithms in MEC environments with frequent dynamic changes are very high. However, the emergence of deep reinforcement learning now provides a state-of-the-art solution to address online computational offloading strategy.

In addition to the need to optimize performance, long-term system stability and staying within the MEC’s energy budget need to be considered. Failure to consider local energy stability and the limited energy budget of the MEC may result in performance degradation of the offloading policy during a long-term task-offloading process, resulting in a reduced ability to make accurate decisions in real time. Yet most DRL-based research approaches nowadays give little consideration to the stability of local system energy consumption in the long term and the impact of energy budget constraints in MEC. There is no overall optimization of the system in the context of a long-term constraint, and most of the studies only introduce events that trigger task failures, such as [16,17,18] by introducing packet loss events.

In this paper, we study the dynamic task-offloading strategy in dynamic multi-user MEC scenarios and propose a strategy to solve task offloading with long-term constraints. The long-term constraints are represented by building multiple queues. And the LMADDPG algorithm is designed using a joint approach to its optimization, which can take full advantage of Lyapunov optimization and DRL. In a dynamic stochastic MEC scenario, an optimal task-offloading policy is found to ensure minimizing the task-offloading cost, which maximizes the user’s QoE. The main contributions are as follows:

• We propose a real-time task-offloading problem with long-term local system energy consumption and energy budget constraints for MEC in a dynamic MEC scenario. The objective is to maximize the QoE while ensuring that the constraints can be satisfied in the long run. An optimal task-offloading strategy is found to minimize the weighted sum of delay and energy consumption in an unknown dynamic scenario.
• We propose a task-offloading strategy with long-term constraints. The long-term constraint states are represented by creating multiple queues that are jointly optimized with the optimization objective. Unlike other studies that directly add adverse behaviors to the penalty term, we model the problem as a problem with long-term constraints. The problem is decoupled using Lyapunov optimization and transformed into a MINLP problem with a single time slot, thus facilitating the solving of real-time optimization problems.
• We describe the problem as an MDP and then design an LMADDPG algorithm based on the union of Lyapunov optimization and deep reinforcement learning, which solves the real-time task-offloading problem by establishing the advantages of deep reinforcement learning.
• We testify the LMADDPG algorithm experimentally, proving its ability to find an optimal task-offloading strategy to ensure minimization of cost under long-term constraints, and comparing it with other baseline algorithms.

The rest of the paper is structured as follows: Section 2 discusses related work. Section 3 describes the system modeling. Section 4 describes the problem description and problem transformation process. Section 5 details the algorithmic design of the LMADDPG algorithm. Section 6 describes the experimental parameter settings and analyses the experimental simulation results. Section 7 concludes the paper.

2. Related Work

MEC has strong computing power, which can provide a platform for user devices to support them in offloading tasks to their servers for processing. In terms of server layout, MEC servers are located a short distance from the end devices. More emerging applications can use MEC to process tasks [19].

Hao et al. [20] proposed a MURL algorithm to minimize long-term average latency. Wu et al. [21] proposed a DAR-AC algorithm to optimize computational performance and energy consumption. Zhao et al. [22] proposed a branch-bounding method to minimize energy consumption. Zhuang et al. [23] proposed an OTDDPG algorithm to optimize energy consumption and delay. Xiao et al. [24] performed a modeling transformation of collaborative offloading into a MINLP problem formulated to minimize execution delay. Li et al. [25] successfully achieved more flexible energy savings in MEC environments by quantifying the correlation between statistical quality of service guarantees and task-offloading policies. Kim et al. [26] proposed a migration optimization scheme for user mobility by using integer linear programming and heuristic solution algorithms aimed at minimizing service provider costs and user latency. Lim et al. [27] considered optimizing latency, energy consumption, and the packet loss rate with DRL-OS based on the D3QN algorithm. Cao et al. [28] proposed the NSGA-II algorithm to solve the problem of minimizing the overall latency and energy consumption, taking into account time-varying networks and limited computational resources in realistic scenarios. The above work provides important reference criteria for the optimization objectives of task-offloading strategies to cope with realistic task offloading in different scenarios.

Wang et al. [29] investigated server failures, where mobility and power constraints caused tasks running on them to fail as well, and solved the problem by designing a new model. Jiang et al. [30] considered the case of limited edge server computational power and found a task-offloading strategy to effectively safeguard the QoE of end-users under the condition of limited edge server computational power. The above work has led us to recognize the need to consider the limitations of edge servers, which can cause task execution to fail if the server fails.

As the complexity of offloading tasks gradually increases, deep reinforcement learning (DRL) can be utilized to manage the burgeoning application tasks. In different scenarios, it can be optimized by learning strategies to find an excellent task-offloading strategy. Qiu et al. [31] proposed a DC-DRL algorithm to solve the problem, which can be trained in two ways: distributed and centralized training. Zou et al. [32] proposed a dual-offloading framework for realistic application scenarios by designing simulation experiments to simulate realistic task-offloading scenarios for dynamic regional resource scheduling. This is solved using an asynchronous advantageous participant–critic (A3C) algorithm to reduce energy and time costs. Alam et al. [33] proposed an autonomic management framework by considering the challenges surrounding mobility, heterogeneity, and the geographic distribution of mobile devices, which were eventually solved using DRL. Ren et al. [34] coordinated the offloading of software through multiple DRL agents on multiple edge nodes, which can deal with dynamic environments and enable the cost of task offloading to be optimized. Cao et al. [35] used deep reinforcement learning for optimization and a modified algorithm of DDPG for solving.

The aforementioned literature used DRL to solve the task-offloading problem and investigated various optimization objectives. The long-term optimization problem is crucial when solving the task-offloading problem. Some studies have already started to focus on long-term optimization. Some researchers have formulated the problem as an MDP and proposed algorithms to solve it [36,37]. In addition, some studies performed the solution by combining Lyapunov optimization with deep reinforcement learning [38,39]. However, most studies do not consider the design of representing long-term constraints by constructing multiple queues when using DRL methods to solve the task-offloading problem.

Unlike previous studies, this paper builds on existing research and investigates the problem of the dynamic task-offloading strategy with long-term constraints for multi-user MEC dynamic scenarios. In a scenario with a dynamically changing system environment, the performance constraint—where the long-term system energy consumption is stable and does not exceed the long-term MEC energy budget—is considered, which is described as an optimization problem with long-term constraints. A task-offloading strategy with long-term constraints is proposed to co-optimize with the optimization objective by constructing multiple queues to represent the long-term constraint states. The optimization problem with long-term constrained task offloading is constructed as a MINLP problem. The MINLP problem is decoupled by using Lyapunov optimization. On this basis, the problem is described as a MDP. A LMADDPG algorithm is proposed for the solution. The algorithm represents the long-term constraints in the form of multiple queues and makes use of Lyapunov for optimization, which ensures that the long-term constraints find an optimal task-offloading policy to ensure that the task-offloading cost is minimized.

3. System Model

3.1. System Overview

As Figure 1 shows, in this paper, a scenario with multiple edge servers (ESs) and multiple types of user equipment (UE) is considered, and each user has its energy consumption queue and MEC energy queue. Each edge server has limited resources, and each user needs to execute multiple tasks, which are processed by task offloading, where an edge server is selected to offload part of the tasks and leave some to be executed locally.

Let $N=\left\{1,2,\dots ,N\right\}$ denote the set of user devices, for each user device $i\in N$, computational tasks will be executed at each time slot. And these tasks need to be handed off to the edge server by way of offloading, or they can be executed on local devices. The task queue has a queuing form, which needs to queue until the end of the previous tasks before the subsequent tasks can be executed, and the latest processing end time of the whole task is its execution time. The set of edge servers is $S=\left\{1,2,\dots ,S\right\}$, $j\in S$. In a continuous time, the edge server assists the user device in computing for a duration, T, set to a number of consecutive time slots, which we define as time slots $T=\left\{1,2,\dots ,T\right\}$, where $t\in T$ is considered a time slot.

The aim of this paper is to consider the optimization of task-offloading costs under the constraints of guaranteeing long-term task energy stability, not exceeding the MEC energy budget. The optimal user QoE can be achieved by determining the best offloading strategy for each time slot. Therefore, the impact of task offloading to the ES for execution or local execution on the QoE needs to be considered.

3.2. Task Transfer and Computational Model

It is assumed that the wireless access system of the edge servers uses orthogonal frequency division multiple access (OFDMA). The network deployment uses the same common bandwidth B. $r_{\mathrm{ul}}\left(t\right)$ indicates the uplink transmission rate and $r_{\mathrm{dl}}\left(t\right)$ indicates the downlink transmission rate. The offloading method uses partial offloading, where the user device selects only one edge server at a time for offloading. $f_i^{local}$ represents the local computational capacity for the ith UE (user equipment), and $f_j^{edge}$ represents the computational capacity for the jth ES. $P_{ij}$ represents the offload ratio of user i for the jth edge server, and $dat{a}_i$ represents the task data size. $c_i\left(t\right)$ represents the CPU cycles required to execute the task, $c_i\left(t\right)=\sigma ·dat{a}_i$, where $\sigma$ represents the number of CPU cycles required to process 1 bit of data. Task processing methods include local execution and offloading to ES for execution.

(1)

Local execution

The local computational delay $T_{ij}^{local}\left(t\right)$ is as follows:

$$T_{ij}^{local}\left(t\right)=\frac{\left(1-P_{ij}\right)c_i\left(t\right)}{f_i^{local}}$$

(1)

The local energy consumption $E_{ij}^{local}\left(t\right)$ is as follows:

$$E_{ij}^{local}\left(t\right)=\kappa \left(1-P_{ij}\right)c_i\left(t\right){\left(f_i^{local}\right)}^2$$

(2)

where $\kappa$ denotes the energy consumption coefficient.

(2)

Offloading execution

The transmission delay $T_{ij}^{tran}\left(t\right)$ is as follows:

$$T_{ij}^{tran}\left(t\right)=\frac{dat{a}_i\left(t\right)}{r_{\mathrm{ul}}\left(t\right)}+\frac{dat{a}_i\left(t\right)}{r_{dl}\left(t\right)}$$

(3)

Using Shannon’s formula for uplink and downlink, respectively, $r_{\mathrm{ul}}\left(t\right)$ and $r_{\mathrm{dl}}\left(t\right)$ are denoted as follows:

$$r_{ul}\left(t\right)={\mathrm{Blog}}_2\left(1+\frac{p_{ue}{h}_{ul}}{{\Gamma }^2}\right)$$

(4)

$$r_{dl}\left(t\right)={B\log }_2\left(1+\frac{p_{edge}{h}_{dl}}{{\Gamma }^2}\right)$$

(5)

where $p_{ue},p_{edge}$ represent the transmission power of the UE and ES, respectively. $h_{ul},h_{dl}$ are the channel gains of the uplink and downlink, respectively, and ${\Gamma }^2$ is the noise power.

The edge server computational delay $T_{ij}^{edge}\left(t\right)$ is as follows:

$$T_{ij}^{edge}\left(t\right)=\frac{P_{ij}{c}_i\left(t\right)}{f_j^{edge}}$$

(6)

The edge offloading delay $T_{ij}^{ES}\left(t\right)$ is as follows:

$$T_{ij}^{ES}\left(t\right)=T_{ij}^{tran}\left(t\right)+T_{ij}^{edge}\left(t\right)$$

(7)

The transmission energy consumption $E_{ij}^{tran}\left(t\right)$ is as follows:

$$E_{ij}^{tran}\left(t\right)=p_{ue}·\frac{dat{a}_i\left(t\right)}{r_{\mathrm{ul}}\left(t\right)}+p_{edge}·\frac{dat{a}_i\left(t\right)}{r_{dl}\left(t\right)}$$

(8)

where $p_{ue}$ and $p_{edge}$ represent the transmission power of the UE and ES, respectively.

The edge server computing energy consumption $E_{ij}^{edge}\left(t\right)$ is as follows:

$$E_{ij}^{edge}\left(t\right)=\kappa P_{ij}{c}_i\left(t\right){\left(f_j^{edge}\right)}^2$$

(9)

The edge server energy consumption $E_{ij}^{ES}\left(t\right)$ is as follows:

$$E_{ij}^{ES}\left(t\right)=E_{ij}^{tran}\left(t\right)+E_{ij}^{edge}\left(t\right)$$

(10)

3.3. Queue Model

3.3.1. Energy Queue

To ensure long-term system energy consumption stabilization, we introduce N energy consumption queues ${\left\{K_i\left(t\right)\right\}}_{i=1}^N$ for each UE. We set $K_i\left(0\right)=0$. The queue is dynamically updated as follows:

$$K_i\left(t+1\right)=max\left\{K_i\left(t\right)+E_{ij}^{local}\left(t\right)-{\gamma }_i,0\right\}$$

(11)

where $E_{ij}^{local}\left(t\right)$ is the energy consumption of the ith UE, and ${\gamma }_i$ is the energy consumption threshold of the ith UE. When the energy consumption queue does not show an infinite increasing trend, i.e., the UE’s energy consumption $E_{ij}^{local}\left(t\right)$ is less than or equal to the energy consumption threshold ${\gamma }_i$ can be satisfied.

3.3.2. MEC Energy Queue

To ensure that the MEC energy budget is not exceeded when offloading for long-term tasks, we introduce N MEC energy queues ${\left\{M_i\left(t\right)\right\}}_{i=1}^N$. Each UE has a MEC energy queue, and the UEs incur different MEC energy consumption values under different offloading policies. We set $M_i\left(0\right)=0$. The queue is dynamically updated as follows:

$$M_i\left(t+1\right)=max\left\{M_i\left(t\right)+E_{ij}^{edge}\left(t\right)-{\varsigma }_i,0\right\}$$

(12)

where $E_{ij}^{edge}\left(t\right)$ is the energy consumed on the edge server and ${\varsigma }_i$ is the average energy budget. When the ith user uses too much energy at the edge server, the MEC energy queue at time slot $t+1$ will expand and, thus, the queue can be used to measure the energy constraint of the MEC.

4. Problem Description and Transformation

The task-offloading cost is defined as $S_i\left(t\right)$, which is the weighted sum of task latency and user energy consumption, as follows:

$$S_i\left(t\right)={\lambda }_{1}max\left\{T_{ij}^{local}\left(t\right),T_{ij}^{ES}\left(t\right)\right\}+{\lambda }_2\left(E_{ij}^{local}\left(t\right)+E_{ij}^{ES}\left(t\right)\right)$$

(13)

where ${\lambda }_1,{\lambda }_2$ denote the respective corresponding weights.

The aim of this paper is to consider the optimization of task-offloading costs under the constraints of guaranteeing long-term task energy stability, not exceeding the MEC energy budget. The optimal user QoE can be achieved by determining the best offloading strategy for each time slot. Task-offloading optimization with long-term constraints needs to be considered. Thus, the problem form is expressed as follows:

$$P1:minS=\underset{T\to \infty }{lim}\frac{1}{T}\sum _{t=0}^{T-1}\sum _{i=1}^N{S}_i\left(t\right)$$

(14)

$$\begin{array}{c}s.t.\hfill \\ C1:\underset{T\to \infty }{lim}\frac{1}{T}{\displaystyle \sum _{t=1}^{T-1}}E\left[E_{ij}^{local}\left(t\right)\right]\le {\gamma }_i,\forall i\hfill \\ C2:\underset{T\to \infty }{lim}\frac{1}{T}{\displaystyle \sum _{t=1}^{T-1}}E\left[E_{ij}^{edge}\left(t\right)\right]\le {\varsigma }_i,\forall i\hfill \end{array}$$

where $C1$ denotes the energy constraint of the local energy queue and $C2$ denotes the energy constraint of the MEC energy queue.

Making optimal task-offloading decisions under a long-term constraint is very difficult due to environment and task unknowns.

Lyapunov Optimization

The MINLP optimization problem is decoupled into a single time-slot deterministic problem through Lyapunov optimization, as the optimization objective, as well as the constraints of $P1$, have both long-term optimizations. In order to have control over the system energy queue and the MEC energy queue, we define $N\left(t\right)\stackrel{\Delta }{=}\{K\left(t\right),M\left(t\right)\}$, where $K\left(t\right)={\left\{K_i\left(t\right)\right\}}_{i=1}^N$, and $M\left(t\right)={\left\{M_i\left(t\right)\right\}}_{i=1}^N$, $N\left(t\right)={\left\{N_i\left(t\right)\right\}}_{i=1}^N$. According to the Lyapunov optimization theory, the Lyapunov function $L\left(N_i\left(t\right)\right)$, is denoted as follows:

$$L\left(N_i\left(t\right)\right)\stackrel{\Delta }{=}\frac{1}{2}\left(\sum _{i=1}^N{K}_i{\left(t\right)}^2+\sum _{i=1}^N{M}_i{\left(t\right)}^2\right)$$

(15)

The change in value will be referred to as the Lyapunov drift function $\Delta \left(N_i\left(t\right)\right)$, denoted as follows:

$$\Delta \left(N_i\left(t\right)\right)\stackrel{\Delta }{=}E\left\{L\left(N_i\left(t+1\right)\right)-L\left(N_i\left(t\right)\right)\left|N_i\left(t\right)\right.\right\}$$

(16)

Due to $N\left(t\right)\stackrel{\Delta }{=}\{K\left(t\right),M\left(t\right)\}$; therefore, $\Delta \left(N_i\left(t\right)\right)$ can be derived from $K_i\left(t\right),M_i\left(t\right)$.

Theorem 1.

Given the dynamic relationship of the queues in the system, when $\mathrm{E}\left\{L\left(N_i\left(t\right)\right)\right\}<\infty$, and there exist constants $\mu >0,\epsilon >0$, we can obtain an upper bound on the Lyapunov drift $\Delta \left(N\left(t\right)\right)$ as follows:

$$\Delta \left(N_i\left(t\right)\right)\le \mu +\epsilon \sum _{i=1}^N{N}_i\left(t\right)$$

(17)

Proof of Theorem 1.

Using ${\{max\left\{x,0\right\}\}}^2\le x^2$, for the energy consumption queue, squaring Equation (11), we have the following:

$$\begin{array}{cc}\hfill K_i{(t+1)}^2& ={\left\{max\left\{K_i\left(t\right)+E_{ij}^{local}\left(t\right)-{\gamma }_i,0\right\}\right\}}^2\hfill \\ \hfill & \le {\left(K_i\left(t\right)+E_{ij}^{local}\left(t\right)-{\gamma }_i\right)}^2\hfill \end{array}$$

(18)

The drift function is as follows:

$$\Delta \left(K_i\left(t\right)\right)\stackrel{\Delta }{=}E\left\{L\left(K_i\left(t+1\right)\right)-L\left(K_i\left(t\right)\right)\left|K_i\left(t\right)\right.\right\}$$

(19)

This can be obtained by substituting Equation (18) into Equation (19), as follows:

$$\begin{array}{cc}\hfill \Delta \left(K_i\left(t\right)\right)& =\frac{1}{2}\sum _{i=1}^{N}E\left\{K_i{(t+1)}^2\left|K_i\left(t\right)\right.\right\}-\frac{1}{2}\sum _{i=1}^{N}E\left\{K_i{\left(t\right)}^2\left|K_i\left(t\right)\right.\right\}\hfill \\ \hfill & \le \frac{1}{2}\sum _{i=1}^{N}E\left\{{\left(K_i\left(t\right)+E_{ij}^{local}\left(t\right)-{\gamma }_i\right)}^2-K_i{\left(t\right)}^2\left|K_i\left(t\right)\right.\right\}\hfill \\ \hfill & \le \frac{1}{2}\sum _{i=1}^{N}E\left\{{\left(E_{ij}^{local}\left(t\right)-{\gamma }_i\right)}^2\left|K_i\left(t\right)\right.\right\}\hfill \\ \hfill & +\sum _{i=1}^{N}E\left\{K_i\left(t\right)\left(E_{ij}^{local}\left(t\right)-{\gamma }_i\right)\left|K_i\left(t\right)\right.\right\}\hfill \\ \hfill & \le {\mu }_1+\sum _{i=1}^{N}E\left\{K_i\left(t\right)\left(E_{ij}^{local}\left(t\right)-{\gamma }_i\right)\left|K_i\left(t\right)\right.\right\}\hfill \end{array}$$

(20)

where ${\mu }_1$ is a constant, ${\mu }_1\stackrel{\Delta }{=}\frac{1}{2}{\displaystyle \sum _{i=1}^N}{\left(E_{ij,max}^{local}\left(t\right)-{\gamma }_i\right)}^2$, holds for all $i\in N$ since all $E_{ij}^{local}\left(t\right)\le E_{ij,max}^{local}\left(t\right)$.

For the MEC energy queue, we square Equation (12) as follows:

$$\begin{array}{cc}\hfill M_i{\left(t+1\right)}^2& ={\left\{max\left\{M_i\left(t\right)+E_{ij}^{edge}\left(t\right)-{\varsigma }_i,0\right\}\right\}}^2\hfill \\ \hfill & \le {\left(M_i\left(t\right)+E_{ij}^{edge}\left(t\right)-{\varsigma }_i\right)}^2\hfill \end{array}$$

(21)

The drift function of the MEC energy queue is calculated as follows:

$$\Delta \left(M_i\left(t\right)\right)\stackrel{\Delta }{=}E\left\{L\left(M_i\left(t+1\right)\right)-L\left(M_i\left(t\right)\right)\left|M_i\left(t\right)\right.\right\}$$

(22)

This can be obtained by substituting Equation (21) into Equation (22):

$$\begin{array}{cc}\hfill \Delta \left(M_i\left(t\right)\right)& =\frac{1}{2}\sum _{i=1}^{N}E\left\{M_i{\left(t+1\right)}^2\mid M_i\left(t\right)\right\}-\frac{1}{2}\sum _{i=1}^{N}E\left\{M_i{\left(t\right)}^2\mid M_i\left(t\right)\right\}\hfill \\ \hfill & \le \frac{1}{2}\sum _{i=1}^{N}E\left\{{\left(M_i\left(t\right)+E_{ij}^{edge}\left(t\right)-{\varsigma }_i\right)}^2-M_i{\left(t\right)}^2\mid M_i\left(t\right)\right\}\hfill \\ \hfill & \le \frac{1}{2}\sum _{i=1}^{N}E\left\{{\left(E_{ij}^{edge}\left(t\right)-{\varsigma }_i\right)}^2\mid M_i\left(t\right)\right\}\hfill \\ \hfill & +\sum _{i=1}^{N}E\left\{M_i\left(t\right)\left(E_{ij}^{edge}\left(t\right)-{\varsigma }_i\right)\mid M_i\left(t\right)\right\}\hfill \\ \hfill & \le {\mu }_2+\sum _{i=1}^{N}E\left\{M_i\left(t\right)\left(E_{ij}^{edge}\left(t\right)-{\varsigma }_i\right)\mid M_i\left(t\right)\right\}\hfill \end{array}$$

(23)

where ${\mu }_2$ is a constant, ${\mu }_2\stackrel{\Delta }{=}\frac{1}{2}{\displaystyle \sum _{i=1}^N}{\left(E_{ij,max}^{edge}\left(t\right)-{\varsigma }_i\right)}^2$, holds for all $i\in N$ since all $E_{ij}^{edge}\left(t\right)\le E_{ij,max}^{edge}\left(t\right)$.

Summarizing Equations (20) and (23) yields the following:

$$\begin{array}{cc}\hfill \Delta \left(N_i\left(t\right)\right)& =\Delta \left(K_i\left(t\right)\right)+\Delta \left(M_i\left(t\right)\right)\hfill \\ \hfill & \le \mu +\sum _{i=1}^{N}E\left\{K_i\left(t\right)\left(E_{ij}^{local}\left(t\right)-{\gamma }_i\right)\mid N_i\left(t\right)\right\}\hfill \\ \hfill & +\sum _{i=1}^{N}E\left\{M_i\left(t\right)\left(E_{ij}^{edge}\left(t\right)-{\varsigma }_i\right)\mid N_i\left(t\right)\right\}\hfill \end{array}$$

(24)

where $\mu ={\mu }_1+{\mu }_2$.    □

This is optimized to ensure long-term task energy consumption stability without exceeding the constraints of the MEC energy budget. This is achieved using the Lyapunov drift plus penalty, minimizing its upper bound. The expression is given as follows:

$$\Delta \left(N_i\left(t\right)\right)+VE\left\{\sum _{i=1}^N{S}_i\left(t\right)\left|N_i\left(t\right)\right.\right\}$$

(25)

where V is a hyperparameter and $V>0$; the first term is the Lyapunov drift function containing the energy consumption queue and the MEC energy queue, and the second term $VE\left\{{\displaystyle \sum _{i=1}^N}{S}_i\left(t\right)\left|N_i\left(t\right)\right.\right\}$ is the penalty term. The optimal QoE is found by minimizing the upper bound. According to Theorem 1, Equation (24) can be obtained by substituting into Equation (25):

$$\begin{array}{c}\Delta \left(N_i\left(t\right)\right)+VE\left\{{\displaystyle \sum _{i=1}^N}{S}_i\left(t\right)\left|N_i\left(t\right)\right.\right\}\hfill \\ \le \mu +{\displaystyle \sum _{i=1}^N}E\left\{K_i\left(t\right)\left(E_{ij}^{local}\left(t\right)-{\gamma }_i\right)\left|N_i\left(t\right)\right.\right\}\hfill \\ +{\displaystyle \sum _{i=1}^N}E\left\{M_i\left(t\right)\left(E_{ij}^{edge}\left(t\right)-{\varsigma }_i\right)\left|N_i\left(t\right)\right.\right\}\hfill \\ +VE\left\{{\displaystyle \sum _{i=1}^N}{S}_i\left(t\right)\left|N_i\left(t\right)\right.\right\}\hfill \end{array}$$

(26)

Therefore, the long-term optimization problem $P1$ is decoupled into a single time-slot MINLP subproblem, transforming the $P1$ problem into the following:

$$\begin{array}{cc}\hfill P2:minR\left(t\right)& ={\displaystyle \sum _{i=1}^N}E\left\{K_i\left(t\right)\left(E_{ij}^{local}\left(t\right)-{\gamma }_i\right)\mid N_i\left(t\right)\right\}\hfill \\ \hfill & +{\displaystyle \sum _{i=1}^N}E\left\{M_i\left(t\right)\left(E_{ij}^{edge}\left(t\right)-{\varsigma }_i\right)\mid N_i\left(t\right)\right\}\hfill \\ \hfill & +VE\left\{{\displaystyle \sum _{i=1}^N}{S}_i\left(t\right)\mid N_i\left(t\right)\right\}\hfill \end{array}$$

(27)

When all $P2$ problems are solved, the $P1$ problem can be solved. Decisions within the current time slot are based on the current state without regard to the historical state.

5. Deep Reinforcement Learning-Based Solutions

5.1. Markov Decision Process

In the environment designed in this paper, the UE is viewed as the DRL-Agent. In the face of dynamically changing MEC environments, it is often difficult to go to the state transfer probability matrix P. Thus, it is not possible to rely on the full quaternion $\left(S, a, R, P\right)$, which contains state, action, reward, and state transfer probability, to characterize the MDP. Therefore, in this paper, we turn to the use of the ternary $\left(S, a, R\right)$, which contains state, action, and reward, without considering state transfer probability.

(1) State space: Reinforcement learning involves the use of one’s own powerful learning ability to improve one’s decision-making by learning the information stored in the experienced replay buffer, which updates the strategy based on its improved decision-making. Therefore, defining an appropriate state space is crucial for overall performance improvement. In defining the state space, the complete environment needs to be added to it. For user device i, define the state space as follows:

$$s\left(t\right)=\left(s_1\left(t\right),s_2\left(t\right),\dots ,s_i\left(t\right)\right)$$

(28)

$$s_i\left(t\right)=\left(P\left(t\right),r_{\mathrm{ul}}^j\left(t\right),r_{\mathrm{dl}}^j\left(t\right),C_i^{loacl},C_j^{edge}\right)$$

(29)

where $C_i^{loacl},C_j^{edge}$ represent the computational resources of UE and ES, respectively.

(2) Action space: When the DRL-Agent acquires state $s\left(t\right)$, it selects an action from the action space that determines the target server $E{S}_j$ and the offload ratio $P_{ij}$ for offloading tasks. This strategy aims to achieve a balanced allocation of tasks by processing some of them locally and offloading others to edge servers. For the user device i, define the action as follows:

$$a\left(t\right)=\left(a_1\left(t\right),a_2\left(t\right),\dots ,a_i\left(t\right)\right)$$

(30)

$$a_i\left(t\right)=[E{S}_j,P_{ij}]$$

(31)

(3) Reward function: The aim of this paper is to consider the optimization of task-offloading costs under the constraints of guaranteeing long-term task energy stability, not exceeding the MEC energy budget. However, the aim of reinforcement learning is to obtain the highest reward value, and the task-offloading cost is inversely proportional to the reward value. Therefore, the reward function for time slot t is as follows:

$$r\left(t\right)=reward\left(s\left(t\right),a\left(t\right)\right)=-R\left(t\right)$$

(32)

5.2. DRL-Based Algorithm Design

5.2.1. Deep Reinforcement Learning Algorithms

Traditional reinforcement learning algorithms typically use a Q-value table to select out the best action. However, as the complexity of the problem changes, it eventually makes the size of the Q-value table explode, resulting in a significant increase in storage and computational cost and, thus, is no longer advantageous in high-dimensional spaces. However, Deep Q-Network (DQN) introduces a neural network to approximate the Q-function, thus avoiding the need to directly store Q-values and being able to handle more complex problems. However, a single Q-learning or DQN cannot be used to implement complex interactions between multiple ‘intelligences’. In this case, the use of the multi-agent deep deterministic policy gradient (MADDPG) algorithm offers significant advantages. The MADDPG algorithm facilitates co-learning among ‘intelligences’ and enhances overall performance through policy sharing, making it more effective in addressing multi-intelligence collaborative decision-making problems.

The MADDPG algorithm achieves better performance by utilizing deep learning and policy gradient methods that allow multiple ‘intelligences’ to learn and make decisions collaboratively in the environment. Its distributed learning feature makes the algorithm scalable and adaptable to large-scale multi-intelligent body systems. The algorithm uses the structure of the two networks joined so that the algorithm can be used in a dynamically changing environment, and can autonomously learn an advantageous strategy, through the learning of the strategy constantly updated and adjusted, in the face of dynamic tasks and different scenarios can be reasonably adjusted to the optimal strategy. In addition, the algorithm can further optimize and adjust the optimal strategy through the interaction between multiple ‘intelligences’, so that each intelligence can fully improve its own decision-making level.

5.2.2. Actor–Critic Network

The actor–network will maximize the reward value accumulated over time by receiving states as inputs and then outputting actions, and it learns the best action to choose in each state by continuously adjusting the network parameters. This network contains an evaluation network ${\mu }_i\left(s_i,{\omega }_i\right)$, and a target network ${\mu }_i^{\prime }\left(s_i,{{\omega }^{\prime }}_i\right)$, where ${\omega }_i,{\omega }_i^{\prime }$ denote the respective parameters of the two networks. The critic–network is used to evaluate the action’s output by the actor–network and provide feedback to improve the strategy. The network receives states and actions as its inputs and outputs the payoff values of the selected actions. The critic–network then provides feedback to the actor–network by accurately estimating these values so that the actor–network can use the feedback to optimize its strategy. The network also includes an evaluation network $Q_i\left(s_i,a_i\left|{\theta }_i\right.\right)$ and a target network $Q_i^{\prime }\left(s_i,a_i\left|{{\theta }^{\prime }}_i\right.\right)$, where ${\theta }_i,{\theta }_i^{\prime }$ denote the parameters of each of the two networks.The updating of the network parameters between the two networks is done by gradient descent.

5.2.3. LMADDPG Algorithm Design

Each user device is individually defined as an intelligence, and each intelligence has two queues each, i.e., the system energy queue and the MEC energy queue, which are used to represent the long-term constraints. The use of the actor–critic network in this algorithm is used for learning to optimize the task-offloading decision. In order to reduce the data relevance of the input experience during the training process and to improve the data utilization, the LMADDPG algorithm employs an experience replay mechanism, which improves the performance of the algorithm by creating an experience replay buffer D. During training, samples are randomly selected from the experience replay buffer D. Each data sample includes information about the state, action, reward, and next state. It is assumed that Z data, each denoted as $\left(s_j,a_j,r_j,s_{j+1}\right)$, are randomly drawn from D for each training session.The corresponding loss values are computed using these randomly drawn data, and the computed loss values are used to update the parameters of the network. The critic–network at each intelligence interacts with each other through the ‘intelligences’ to update the network parameters by minimizing the computed loss function. Define the objective value as follows:

$$y_i^j=r_i^j+{\left.\gamma Q^{\prime }\left(s^{\prime j},a_1^{\prime j},\dots ,a_N^{\prime j}\mid {\omega }_i\right)\right|}_{{a^{\prime }}_i={\mu }_i^{\prime }\left(s_i^{\prime j}\right)}$$

(33)

where $\gamma$ is the discount factor.

The loss function its expression is as follows:

$$L_i\left({\omega }_i\right)=\frac{1}{Z}\sum _{j=1}^Z{\left(y_i^j-Q_i\left(s^j,a_1^j,\dots ,a_N^j\left|{\omega }_i\right.\right)\right)}^2$$

(34)

To update the actor–network using the policy gradient, the actor–network is used to compute the action in the current state, and then the gradient of the action value function computed by the critic–network is used to update the actor–network parameters with the following formula:

$${\nabla }_{{\theta }_i}J\left({\mu }_k\right)\approx \frac{1}{Z}\sum _{j=1}^Z{\nabla }_{{\theta }_i}{\mu }_k\left(s_i^j\right){\nabla }_{a_i}{Q}_i\left(s^j,a_1^j,\dots ,a_N^j\left|{\omega }_i\right.\right){|}_{a_i={\mu }_i\left(s_i^{\prime i}\right)}$$

(35)

Use the soft update strategy to update the target actor–network and target critic–network parameters:

$${\omega }_i^{\prime }=\tau {\omega }_i+\left(1-\tau \right){\omega }_i^{\prime }$$

(36)

$${\theta }_i^{\prime }=\tau {\theta }_i+\left(1-\tau \right){\theta }_i^{\prime }$$

(37)

The reward function is optimized using Lyapunov optimization based on the multiple queues created. Each intelligent body obtains the reward value by executing an action after acquiring the environment information and accumulates experience data using the experience playback mechanism. Each intelligent body updates its policy network with the accumulated data, using a deep deterministic policy gradient approach to optimize its policy to maximize the expected cumulative reward. The queue information is updated at each training time slot. When updating the policy network, the intelligence also uses the target policy network to stabilize the training process by updating the parameters of the target network through a soft update approach. During the training process, the ‘intelligences’ learn from each other and optimize their respective strategies through collaboration and competition to achieve the global optimal solution. In this way, the LMADDPG algorithm enables multiple ‘intelligences’ to learn to work together effectively in a collaborative environment to achieve finding an optimal task-offloading decision under the performance constraint of stable long-term system energy consumption and not exceeding the long-term MEC energy budget. The overall execution process is shown in Algorithm 1.

Algorithm 1 LMADDPG task-offloading algorithm

1: Initialization: Initialize the parameters of each agent’s actor and critic evaluations and target networks. Initialize the replay buffer D, learning rate, the discount factor, the maximum learning epoch, steps. 2: for  $epoch=1$  to  M  do 3:    Initialize a random process N for action exploration 4:    Initialize $K\left(0\right)=0$, $M\left(0\right)=0$ 5:    for $t=1$ to T do 6:        For each agent i, select action $a\left(t\right)$, $a\left(t\right)$ based on the current observed state: $a\left(t\right)={\mu }_i\left(s_i\right|{\omega }_i)+N\left(t\right)$ 7:        Execute actions $a\left(t\right)=(a_1\left(t\right),a_2\left(t\right),\cdots ,a_i\left(t\right))$, obtain the reward $r_i\left(t\right)$ based on Lyapunov drift-plus-penalty function, and the subsequent new state $s_i^{\prime }\left(t\right)$ 8:        For each agent i, input the new state $s_i^{\prime }\left(t\right)$ to agent i 9:        Store the agent’s information into the experience replay buffer D as a tuple of four elements $(s,a,r,s^{\prime })$ 10:       $s_i=s_i^{\prime }$ 11:       for Each agent $i=1$ to N do 12:          Sample a random mini-batch of Z samples from the replay buffer D 13:          $y_i^j=r_i^j+\gamma Q^{\prime }\left({s^{\prime }}^j,{a^{\prime }}_1^j,\cdots ,{a^{\prime }}_N^j\left|{\omega }_i\right.\right){|}_{{a^{\prime }}_i={{\mu }^{\prime }}_i\left(s_i^{\prime j}\right)}$ 14:          Update the critic–network according to the following loss function: $L_i\left({\omega }_i\right)=\frac{1}{Z}{\displaystyle \sum _{j=1}^Z}{\left(y_i^j-Q_i\left(s^j,a_1^j,\cdots ,a_N^j\left|{\omega }_i\right.\right)\right)}^2$ 15:          Update the actor–network using the sampled gradient based on Equation (36) 16:          Update $K\left(t\right)$ using Equation (11) 17:          Update $M\left(t\right)$ using Equation (12) 18:      end for 19:      Update target network parameters for each agent with: ${\omega }_i^{\prime }=\tau {\omega }_i+(1-\tau ){\omega }_i^{\prime }$ ${\theta }_i^{\prime }=\tau {\theta }_i+(1-\tau ){\theta }_i^{\prime }$ 20:    end for 21: end for

5.2.4. Algorithm Complexity Analysis

In the LMADDPG algorithm, each intelligent body uses the actor–critic network architecture. The number of ‘intelligences’ is N. ${\mathrm{L}}^a$ denotes the number of network layers of actor–network and ${\mathrm{L}}^c$ denotes the number of network layers of critic–network. ${\mathrm{I}}_{l^a-1},{\mathrm{I}}_{l^a}$ denotes the dimensions of inputs and outputs of the $l^a$ layer of actor–network and ${\mathrm{I}}_{l^c-1},{\mathrm{I}}_{l^c}$ denotes the dimensions of inputs and outputs of the $l^c$ layer respectively. Therefore, the algorithm complexity of LMADDPG can be calculated as $O\left(2N·\left({\displaystyle \sum _{l^a=1}^{L^a}}{\mathrm{I}}_{l^a-1}{\mathrm{I}}_{l^a}+{\displaystyle \sum _{l^c=1}^{L^c}}{\mathrm{I}}_{l^c-1}{\mathrm{I}}_{l^c}\right)\right)$.

6. Simulation Experiment and Analysis

6.1. Experimental Parameters

This research conducted simulation experiments within a Python 3.10.7 environment on a Windows 11 operating system. In this paper, the simulation scenario is set up as a multi-user multi-MEC scenario, where each UE can transmit data to and from ES through channel. The size of each task $dat{a}_i$, follows a uniform distribution. The computational power of the UE of the mobile device is set up to be $[1.2,1.8]$ Ghz, and that of the edge server is set up to be $[6,7]$ Ghz. The task size is $[1.5,2]$ Mb. The main parameters are shown in

Table 1. Experimental parameter.

Parameters	Value
Number of users N	5
Number of edge servers S	7
Bandwidth B	40 Mhz
Lyapunov weights V	30
User equipment transmission power $p_{ue}$	0.01 W
Edge server transmission power $p_{edge}$	0.1 W
Gaussian noise power ${\Gamma }^2$	−174 dBm
The local computing capability $f_i^{local}$	$[1.2,1.8]$ Ghz
The MEC computing capability $f_j^{edge}$	$[6,7]$ Ghz
Actor–network learning rate	0.001
Critic–network learning rate	0.001
Size of mini-batch	32
Training epochs	1000
Time slots T	500
Optimizer	Adam

:

6.2. Comparative Experiments

6.2.1. Comparative Algorithms Overview

In order to prove the superiority and reliability of this paper’s algorithm, this paper’s algorithm is compared with five other baseline algorithms.

(1)

DDQN-based algorithm [40]. Double DQN algorithm is an improved algorithm based on DQN. DDQN improves the performance by using two independent neural networks for selecting the action and evaluating the value of the action.

(2)

D3QN-based algorithm [41]. D3QN estimates the function of state values and advantage separately by introducing the Double Q-Learning mechanism while using the Dueling Network architecture. It enables the Agent to learn the task-offloading strategy more accurately.

(3)

MAPPO-based algorithm [42]. MAPPO is a multi-intelligence body algorithm. A centralized approach is used during training to acquire information from multiple ‘intelligences’ and optimize the decision making of the ‘intelligences’. In the execution process the decision is based on local observation information only. The algorithm deals with the problem of policy learning in a multi-intelligent body environment through a proximal policy optimization approach.

(4)

Local Computing Only (LOC) scheme. All the tasks are all computed locally without task offloading.

(5)

Random scheme (Random). Directly randomly generate task-offloading strategies for task offloading.

6.2.2. Simulation Results and Analysis

Firstly, the convergence of the algorithm designed in this paper is evaluated, and this paper evaluates the convergence through two objects. The LMADDPG algorithm proposed in this paper performs task offloading through actor–critic network, so the first object for evaluation is the loss function values of the two networks. DRL-Agent will probabilistically choose the action in each step and, thus, the image will fluctuate continuously. Where Figure 2 represents the actor–network loss value and Figure 3 represents the critic–network loss value. As can be seen from Figure 2 and Figure 3, with the increase in the number of iterations, the loss values of the two networks are decreasing and both of them begin to converge gradually, indicating that both networks are functioning properly.

The LMADDPG algorithm reward value is used as the second evaluation object. In Figure 4, the convergence judgment of the LMADDPG algorithm is demonstrated. The algorithm, which undergoes a learning process, has a strong tendency to jitter in the initial phase due to the randomness of early reinforcement learning. However, the reward value obviously grows as the number of iterations increases. Through the learning process, an optimal task-offloading decision is finally learned, which leads to the stabilization of the reward value, i.e., the LMADDPG algorithm gradually reaches convergence. Moreover, the algorithm of this paper can reach convergence in a very short number of iterations, which can be attributed to its policy update. This update integrates the deterministic behavioral policy with the exploitation of the Q-value function. The effectiveness of the LMADDPG algorithm is based on the proof of convergence of the two evaluated objects described above.

Then, the energy consumption is evaluated for the system energy queue and the MEC energy queue. In Figure 5 experiments are conducted for four different algorithms for the system energy queue, where the horizontal coordinate is the time slot and the vertical coordinate is the energy consumption. In this process, as the time slots increase, all algorithms through optimization simultaneously keep the system energy consumption queue stable and do not show an explosive rising trend. It means that in the environment of long-term task offloading, the system energy consumption queue will not be oversized. Instead, it shows a gradual reduction trend and eventually remains in a stable state. This is because the algorithm can maintain a relatively stable queue length after learning. In addition, comparing the LMADDPG algorithm with other baseline algorithms, the LMADDPG algorithm maintains the smallest queue length, i.e., the best optimization.

In Figure 6, the energy consumption of four different algorithms MEC energy queues are evaluated. It can be seen that the MEC energy queues of the optimized algorithms do not have an infinite rising trend. As the time slot progresses, it becomes evident that the MEC energy queue of the LMADDPG algorithm exhibits a more stable behavior compared to the other algorithms. Although the performance of LMADDPG will be lower than other algorithms in the initial process of task offloading, the performance of LMADDPG is more advantageous in the long-term process. The LMADDPG algorithm outperforms the other baseline algorithms more in keeping the MEC energy stable during long-term optimization.

In addition, the effect of penalty coefficient adjustment average task-offloading cost is experimentally compared. The impact of Lyapunov penalty coefficient V on the performance of LMADDPG is further demonstrated in Figure 7, where the delay and energy weighted sum of tasks will be compared by adjusting the penalty coefficient V, setting $V\in \left[1,10,20,30,40,50,60\right]$. As V increases, the average cost is decreasing, which is due to the skewing of the fairness of the optimization strategy when the penalty factor is increasing. This trend is due to the fact that the higher the value of V gives more weight to the weighted delay and energy cost, which drives the algorithm towards cost optimization. Therefore, in practical scenarios, a suitable V value needs to be set to balance the average system energy consumption as well as the stability between the average MEC energy and the average cost.

Figure 8 shows the performance comparison between the LMADDPG algorithm and the MADDPG algorithm using multi-queue optimization. Both algorithms perform task-offloading decisions separately and calculate the average task-offloading cost, i.e., the average cost. The proposed LMADDPG algorithm is able to reach stability in a very short number of iterations, converges better than the MADDPG algorithm, and produces a much smaller average cost than the MADDPG algorithm. This shows that the LMADDPG outperforms MADDPG in terms of performance.

Finally, the advantages of the algorithms over the baseline algorithms are verified, and the computational delay and energy weighted sums are compared for the different baseline algorithms. The results are shown in Figure 9, where the task-offloading strategy of LMADDPG is compared with other algorithms and the weighted sum of delay and energy consumption is calculated. It is found that our algorithm reduces the cost and finds the optimal task-offloading strategy faster compared to the other five baseline algorithms. Compared to other baseline algorithms, our algorithm is more advantageous.

7. Conclusions

In this paper, we propose a multi-queue-based real-time task-offloading strategy for deep reinforcement learning. We establish multiple queues to represent the long-term constraint states and co-optimize with the optimization objective. The optimization problem with long-term constraints is decoupled into subproblems to be solved in a single time slot using Lyapunov optimization, which describes the problem as an MDP. We propose a DRL-based LMADDPG algorithm to solve the task-offloading decision problem. During the training process, all the ‘intelligences’ share a unified strategy network in order to utilize the experience of all the ‘intelligences’ during the training process. However, in the execution phase, each intelligence independently executes its own policy to interact with the environment. Finally, an optimal task-offloading strategy is found, which can effectively maintain long-term system energy consumption and MEC energy stability. The simulation results prove the effectiveness of the improved algorithm and it is more advantageous in comparison with other baseline algorithms.