Deep-Reinforcement-Learning-Based Low-Carbon Economic Dispatch for Community-Integrated Energy System under Multiple Uncertainties

Abstract

A community-integrated energy system under a multiple-uncertainty low-carbon economic dispatch model based on the deep reinforcement learning method is developed to promote electricity low carbonization and complementary utilization of community-integrated energy. A demand response model based on users’ willingness is proposed for the uncertainty of users’ demand response behavior; a training scenario set of a reinforcement learning agent is generated with a Latin hypercube sampling method for the uncertainties of power, load, temperature, and electric vehicle trips. Based on the proposed demand response model, low-carbon economic dispatch of the community-integrated energy system under multiple uncertainties is achieved by training the agent to interact with the environment in the training scenario set and reach convergence after 250 training rounds. The simulation results show that the reinforcement learning agent achieves low-carbon economic dispatch under 5%, 10%, and 15% renewable energy/load fluctuation scenarios, temperature fluctuation scenarios, and uncertain scenarios of the number of trips, time periods, and mileage of electric vehicles, with good generalization performance under uncertain scenarios.

1. Introduction

The increasing economic level and energy demand will lead to the problem of fossil energy depletion and ecological environment degradation, and the development of low-carbon energy and the complementary use of energy has become a strategic choice for all countries in the world.

In the context of energy decarbonization, the use of renewable energy sources such as wind and solar is of great significance. However, renewable energy is characterized by randomness, so the integrated energy system containing renewable energy generation needs to have the ability to cope with the randomness of renewable energy.

In the community-integrated energy system (CIES) that includes renewable energy power generation, in addition to the demand for electricity, there are often demands for natural gas and cooling supply [1,2]. In a community-integrated energy system with electricity–gas–cooling coupling, the complementary characteristics of the three energies can be fully utilized to promote the consumption of renewable energy. However, the diversity of various energy devices, the complexity of joint control of equipment, and various uncertainties in the energy system bring challenges to the dispatch of a community-integrated energy system.

In existing work, optimal dispatch of energy systems can be achieved with a variety of methods. Yang Li, Yuanyuan Zhang, and Shenbo Yang et al. [3,4,5] proposed a hierarchical stochastic dispatch method for an integrated energy system based on Stackelberg game theory. For non-convex problems in an integrated energy system, Han Gao et al. [6] proposed a new optimization method based on Benders decomposition, whose sub-problems can be solved in parallel to accelerate the computation further. To address the complexity of multiple supplies and demands in an integrated energy system, X.J. Luo et al. [7] proposed a multi-energy system management strategy that includes three core algorithms for demand-side rolling optimization, supply-side rolling optimization, and feedback correction. To address the uncertainty of an integrated energy system, Peng Li, Guang Liu, Rujing Yan, and Xiaoqing Li et al. [8,9,10,11] applied robust optimization methods to solve the uncertainty in an integrated energy system based on multi-energy load coupling and proposed a stochastic robust optimal operation strategy for an integrated energy system.

With the development of artificial intelligence, deep reinforcement learning technology has had some applications in the optimal dispatch of energy systems. It can realize the joint optimal dispatch of various energy devices within an integrated energy system through the continuous interaction between the agent and the environment, with good adaptive learning capability.

Salman Sadiq Shuvo et al. [12] proposed a discrete action deep reinforcement learning method for managing smart devices based on the A2C (Advantage Actor–Critic) algorithm to optimize power costs. The method manages flexible loads as a discrete power staging control. Renzhi Lu et al. [13] considered uncontrollable, shiftable, and curtailable loads in the system and approximated the optimal policy using a discrete-action DQN (Deep Q-Network) method. Mifeng Ren et al. [14] built on the DQN method with a model-free discrete-action Dueling-double deep Q-learning neural network (Dueling-DDQN) algorithm for joint dispatch of air conditioners, electric vehicles, and energy storage devices in a home energy management system model. Bo-Chen Lai et al. [15] proposed a multi-agent reinforcement-learning-based community energy management system model in which the appliances are classified into three categories: uncontrollable appliances, shiftable appliances, and power-curtailable appliances, and a discrete-action Multi-agent Q-Learning algorithm is used for optimal dispatch based on the DQN approach. The control actions of the energy units are designed as a hierarchical regulation. To achieve continuous control of energy units in residential energy systems, Yujian Ye et al. [16] proposed a new real-time management strategy for residential energy systems based on the continuous-action Deep Deterministic Policy Gradient (DDPG) deep reinforcement learning approach to achieve multi-dimensional continuous state control of multiple energy units in energy systems to minimize energy costs for users. Hongyuan Ding et al. [17] classified smart home loads into HVAC, shiftable, uncontrollable, and thermal loads and proposed a continuous-action PD-DDPG (Primal-Dual Deterministic Policy Gradient) deep reinforcement learning method to optimize the control of home energy system devices based on the DDPG method. Lin Xue et al. [18] proposed a model–data–event-based low-carbon economic scheduling framework for the community-integrated energy system, and used an improved DDPG algorithm that takes into account generation and load uncertainty for real-time scheduling. Yue Qiu et al. [19] proposed a mathematical model of the local integrated energy system that takes into account supply- and load-side flexible resources and used an improved twin delayed Deep Deterministic Policy Gradient (TD3) algorithm to achieve operational optimization under renewable energy generation, electrical load, and thermal load uncertainty. Seong-Hyun Hong et al. [20] propose an energy management system (EMS) algorithm based on secure reinforcement learning to achieve more robust energy management considering generation and load uncertainties.

In summary, the application of reinforcement learning methods to a variety of demand response management has been realized in existing work. However, the uncertainty of demand response is usually not considered. In a community-integrated energy system, the user’s demand response behavior is the result of the user’s trade-offs and should be subject to uncertainty. Inspired by existing work, this paper develops a demand response model that takes into account user behavioral uncertainty and proposes a deep-reinforcement-learning-based low-carbon and economic dispatch method for community-integrated energy systems under multiple uncertainties.

The main contributions of this paper are as follows.

(1) A community-integrated energy system simulation model is developed to simulate the energy flow within the energy system. The simulation model includes energy units such as gas turbines, electric vehicles, and air conditioning systems, as well as demand response resources. The demand response resources are categorized into two categories in the simulation model, including curtailable load and shiftable load.

(2) To address the uncertainty of user participation in demand response behavior and consider real-time energy prices, a demand response model based on the degree of users’ willingness is proposed to take into account the uncertainties of curtailable load and shiftable load.

(3) A Soft Actor–Critic deep-reinforcement-learning-based approach is proposed for the dispatch of community-integrated energy systems, which achieves a better dispatch scheme through the stronger exploratory capability of the Soft Actor–Critic algorithm, and adapts to the uncertainties of renewable energy generation, outdoor temperature, and electric vehicle trips through the training of the agent to improve the applicability of this method under multiple uncertainties.

2. Plant Model

The community-integrated energy system model based on deep reinforcement learning includes an environment model and an agent model. The environment model is the context of reinforcement learning in low-carbon economic dispatch, which consists of various component models such as renewable energy generation, energy demand, etc., and the energy market and carbon trading market. The agent is the decision-making subject of reinforcement learning, which learns the optimal dispatch strategy by continuously interacting with the environment, observing the environment, taking actions, and obtaining rewards. The general framework of the model is shown in Figure 1.

2.1. Environmental Model

The CIES environmental model integrates the electricity–gas coupling model on the energy supply side, the electricity–cooling coupling model on the energy supply side, the demand response model on the energy demand side, the energy storage device model on the energy storage side, etc. The internal components of CIES, together with the electricity market, the natural gas market, and the carbon trading market, constitute the environmental model.

2.1.1. Electricity–Gas Coupling Model on the Energy Supply Side

(1)

Gas turbine model

The gas turbine [21] converts natural gas into electricity as follows:

$$H_t=a_{\mathrm{g}}{(P_t^{\mathrm{GT}})}^2+b_{\mathrm{g}}{P}_t^{\mathrm{GT}}+c_{\mathrm{g}}$$

(1)

$$G_t=H_t/G^{HV}$$

(2)

where $H_t$ means the heat consumption of the gas turbine; $a_{\mathrm{g}}$, $b_{\mathrm{g}}$, and $c_{\mathrm{g}}$ are the heat consumption coefficients; $G_t$ means the natural gas consumed using the gas turbine; $G^{HV}$ means the thermal value.

The gas turbine is operated to satisfy the constraints as shown in Equations (3) and (4).

$$p_{\min }^{\mathrm{GT}}<p_t^{\mathrm{GT}}<p_{\max }^{\mathrm{GT}}$$

(3)

$$-\Delta P^{\mathrm{GTmax}}\le P_t^{\mathrm{GT}}-P_{t-1}^{\mathrm{GT}}\le \Delta P^{\mathrm{GTmax}}$$

(4)

where $p_{\max }^{\mathrm{GT}}$ and $p_{\min }^{\mathrm{GT}}$ are the maximum and minimum power output of the gas turbine, respectively; $\Delta P^{\mathrm{GTmax}}$ is the maximum climbing power of the gas turbine, respectively.

(2)

P2G equipment model

The P2G equipment [22] converts electrical energy to natural gas as follows:

$$G_t^{\mathrm{P}2\mathrm{G}}=P_t^{\mathrm{P}2\mathrm{G}}{\eta }^{\mathrm{P}2\mathrm{G}}/G^{HV}$$

(5)

where $G_t^{\mathrm{P}2\mathrm{G}}$ means the natural gas generated; $P_t^{\mathrm{P}2\mathrm{G}}$ means the electric power consumed; ${\eta }^{\mathrm{P}2\mathrm{G}}$ means the conversion efficiency of the P2G equipment.

2.1.2. Electric–Cooling Coupling Model on the Energy Supply Side

The electricity–cooling coupling on the energy supply side is realized through the cooling storage air conditioner, which needs to meet the constraints as follows [23]:

$$0\le H_t^{\mathrm{ACc}}\le H_{\max }^{\mathrm{ACc}}$$

(6)

where $H_t^{\mathrm{ACc}}$ means the cooling created by the chiller at time t; $H_{\max }^{\mathrm{ACc}}$ means the maximum cooling creation of the chiller at time t.

Indoor temperature changes can be described in Equations (7) and (8).

$$T_t^{\mathrm{in}}=\epsilon T_{t-1}^{\mathrm{in}}+(1-\epsilon )(T_t^{\mathrm{out}}-(H_t^{\mathrm{ACd}}-Q_t)/A)$$

(7)

$$H_t^{\mathrm{ACd}}=H_t^{\mathrm{ACc}}+H_t^{\mathrm{ACr}}-H_t^{\mathrm{ACs}}$$

(8)

where $H_t^{\mathrm{ACd}}$ is the cooling provided to the room using the air conditioning at time t; $Q_t$ is the heat gained by the building through solar radiation, indoor installations, etc., in addition to the heat transfer from the temperature difference at time t; $\epsilon$ is the air inertia coefficient, set to 0.95; $H_t^{\mathrm{ACs}}$ and $H_t^{\mathrm{ACr}}$ mean the cooling charging and discharging volume of the storage tank at time t, respectively.

The indoor temperature constraints to be met using air conditioning dispatch are shown in Equations (9) and (10).

$$T_0^{\mathrm{in}}=T^{\mathrm{origin}}$$

(9)

$$T_{\min }^{\mathrm{in}}\le T_t^{\mathrm{in}}\le T_{\max }^{\mathrm{in}}$$

(10)

where $T^{\mathrm{origin}}$ is the initial temperature; $T_t^{\mathrm{in}}$ means the indoor temperature at time t; $T_{\max }^{\mathrm{in}}$ and $T_{\min }^{\mathrm{in}}$ mean the maximum and minimum appropriate temperature, respectively.

The electric power of the air conditioner at time t can be expressed as follows:

$$P_t^{\mathrm{AC}}=\frac{H_t^{\mathrm{ACc}}}{{\mu }^{\mathrm{ACc}}}+{\mu }^{\mathrm{ACs}}{H}_t^{\mathrm{ACs}}+{\mu }^{\mathrm{ACr}}{H}_t^{\mathrm{ACr}}$$

(11)

where ${\mu }^{\mathrm{ACc}}$ is the energy efficiency ratio of the chiller; ${\mu }^{\mathrm{ACs}}$ and ${\mu }^{\mathrm{ACr}}$ are the energy efficiency ratios of the cooling charging and discharging, respectively.

2.1.3. Energy Storage Device Model on the Energy Storage Side

The energy storage device model involves constraints on the battery, the gas storage tank, and the cooling storage tank [11,24] as shown in Equations (12)–(15).

$$E_{\min }^{\mathrm{ES}}\le E_t^{\mathrm{ES}}\le E_{\max }^{\mathrm{ES}}$$

(12)

$$0\le P_t^{\mathrm{ESch}}\le P_{\max }^{\mathrm{ESch}}$$

(13)

$$0\le P_t^{\mathrm{ESdis}}\le P_{\max }^{\mathrm{ESdis}}$$

(14)

$$E_t^{ES}=E_{t-1}^{\mathrm{ES}}+\Delta t{P}_t^{\mathrm{ESch}}{\eta }^{\mathrm{ESch}}-\frac{\Delta t{P}_t^{\mathrm{ESdis}}}{{\eta }^{\mathrm{ESdis}}}$$

(15)

where $E_t^{\mathrm{ES}}$ is the residual capacity at time t; $E_{\max }^{\mathrm{ES}}$ and $E_{\min }^{\mathrm{ES}}$ are the maximum and minimum storage capacity, respectively; $P_t^{\mathrm{ESch}}$ and $P_t^{\mathrm{ESdis}}$ are the charging and discharging energy at time t; $P_{\max }^{\mathrm{ESch}}$ and $P_{\max }^{\mathrm{ESdis}}$ are the maximum charge and discharge energy per unit time, respectively; ${\eta }^{\mathrm{ESch}}$ and ${\eta }^{\mathrm{ESdis}}$ are the charging and discharging efficiency, respectively.

2.1.4. Demand Response Model Based on User’s Willingness on the Energy Demand Side

In this paper, we consider the uncertainty of user participation in demand response and establish the demand response model based on the user’s willingness (DRUW). The dispatch decisions for CIES are implemented with the community-integrated energy management system (CIEMS). After CIEMS issues the incentive signal, users consider whether to execute a demand response from their own interests according to the incentive signal. Since the response behavior of users involves their own interests, the demand response behavior of users is inseparable from the real-time energy price and demand response incentive compensation mechanism.

Define the willingness degree of a user to participate in load curtailment and load shifting demand response as follows.

$$\left\{\begin{array}{c}{\theta }_t^{\mathrm{RL}}={\theta }_t^{\mathrm{RLB}}{\beta }_t^{\mathrm{RL}}=\frac{c^{\mathrm{RL}}{\beta }_t^{\mathrm{RL}}}{{\epsilon }^{\mathrm{dam}}{\rho }_t^{\mathrm{pu}}}\\ {\theta }_t^{\mathrm{SL}}={\theta }_t^{\mathrm{SLB}}{\beta }_t^{\mathrm{SL}}=\frac{c^{\mathrm{SL}}{\beta }_t^{\mathrm{SL}}}{{\epsilon }^{\mathrm{dam}}(1/{\rho }_t^{\mathrm{pu}})}\end{array}\right.$$

(16)

where ${\theta }_t^{\mathrm{RL}}$ and ${\theta }_t^{\mathrm{SL}}$ are the willingness degree of a user to participate in load curtailment at time t and load shifting to time t, respectively; ${\theta }_t^{\mathrm{RLB}}$ and ${\theta }_t^{\mathrm{SLB}}$ are the benchmark willingness degrees of a user to participate in load curtailment at time t and load shifting to time t, respectively; ${\beta }_t^{\mathrm{RL}}$ and ${\beta }_t^{\mathrm{SL}}$ are the demand response incentive factors for load curtailment at time t and load shifting to time t, respectively, and take the values of [1,2]. ${\epsilon }^{\mathrm{dam}}$ is the response damping coefficient for a user’s demand response to electric/gas energy prices, which depends on the user and on the energy source; ${\rho }_t^{\mathrm{pu}}$ is the normalized energy price at time t; $c^{\mathrm{RL}}$ is the unit curtailed electric/gas load compensation price; $c^{\mathrm{SL}}$ is the unit shifted electric/gas load compensation price.

Based on the user’s willingness degree, define their demand response probability as shown in Equation (17) and Figure 2.

$${\omega }_t=\left(\begin{array}{ll}0\hfill & {\theta }_t<{\theta }_{\min }\hfill \\ ({\theta }_t-{\theta }_{\min })/({\theta }_{\max }-{\theta }_{\min })\hfill & {\theta }_{\min }<{\theta }_t<{\theta }_{\max }\hfill \\ 1\hfill & {\theta }_t>{\theta }_{\max }\hfill \end{array}\right.$$

(17)

where ${\omega }_t$ means the probability of a user’s participation in the response at time t; ${\theta }_{\min }$ means the lower limit of the response uncertainty interval of the user; ${\theta }_{\max }$ means the upper limit of the response uncertainty interval of the user.

As shown in Figure 2, the horizontal coordinate is the user’s willingness degree, and the vertical coordinate is the probability of the user’s participation in the demand response. When the user’s willingness degree is lower than the lower limit of the response uncertainty interval at a certain moment, the user does not participate in the response, and the response probability is 0. When the user’s willingness degree is higher than the upper limit of the response uncertainty interval, the user’s response probability is 1. When the user’s willingness degree is between the upper and lower limits of the response uncertainty interval, the user’s response probability is proportional to the willingness degree and takes values between 0 and 1.

(1)

Curtailable electric/gas load model in DRUW

$$P_t^{\mathrm{RL}}={\delta }_t^{\mathrm{RL}}{P}_t^{\mathrm{RL}0}$$

(18)

where ${\delta }_t^{\mathrm{RL}}$ means the binary state variable of the electric/gas load curtailment response at time t; $P_t^{\mathrm{RL}}$ means the actual power curtailment of electric/gas load at time t; $P_t^{\mathrm{RL}0}$ means the expected power curtailment of electric/gas at time t.

The dispatch of curtailable electric/gas load needs to satisfy the continuous curtailment time constraint and the total curtailment times’ constraint as shown in Equations (19) and (20).

$$T_{\min }^{\mathrm{RL}}\le {\displaystyle \sum _{t=\tau }^{\tau +T_{\max }^{\mathrm{RL}}-1}{\delta }_t^{\mathrm{RL}}}\le T_{\max }^{\mathrm{RL}}$$

(19)

$${\displaystyle \sum _{t=1}^T{\delta }_t^{\mathrm{RL}}}\le N_{\max }^{\mathrm{RL}}$$

(20)

where $T_{\max }^{\mathrm{RL}}$ and $T_{\min }^{\mathrm{RL}}$ mean the upper and lower limits of the continuous curtailment time, respectively; $N_{\max }^{\mathrm{RL}}$ means the upper limit of the total curtailment times.

(2)

shiftable electric/gas load model in DRUW

$$P_{ts’}^{\mathrm{SL}}={\delta }_{ts’}^{\mathrm{SL}}{P}_{ts}^{\mathrm{SL}0}$$

(21)

where $ts$ means the load onset moment before the shiftable electric/gas load participates in the dispatch; $ts’$ means the load onset moment after the shiftable electric/gas load participates in the dispatch; ${\delta }_{ts’}^{\mathrm{SL}}$ means the binary state variable of the shiftable electric/gas load response at time $ts’$; $P_{ts’}^{\mathrm{SL}}$ means the load shifted to period $ts’$; $P_{ts}^{\mathrm{SL}0}$ means the shifted electric/gas load at time $ts$.

The power distribution vector of the shiftable load before it participates in the dispatch is as follows:

$$L_{\mathrm{before}}^{\mathrm{SL}}=(0,\cdots ,P_{ts}^{\mathrm{SL}},P_{ts+1}^{\mathrm{SL}},\cdots ,P_{ts+td}^{\mathrm{SL}},\cdots ,0)$$

(22)

where $td$ means the dispatch time of the shiftable electric/gas load; $P_{ts}^{\mathrm{SL}}$ means the shiftable electric/gas load in the period $ts$ before it participates in the dispatch.

The power distribution vector after the participation of shiftable load in the dispatch is as follows:

$$L_{\mathrm{after}}^{\mathrm{SL}}=(0,\cdots ,P_{ts’}^{\mathrm{SL}},P_{ts’+1}^{\mathrm{SL}},\cdots ,P_{ts’+td}^{\mathrm{SL}},\cdots ,0)$$

(23)

Shiftable electrical/gas loads need to meet dispatch interval constraints as follows:

$$t{s}^{\min }<ts’<ts’+td<t{e}^{\max }$$

(24)

where $t{s}^{\min }$ and $t{e}^{\max }$ mean the lower and upper limits of the allowable dispatch interval for shiftable load, respectively.

2.1.5. Community Electric Vehicle Model on the Energy Demand Side

Electric vehicles rely on onboard batteries to participate in power dispatch, and while participating in dispatch, they have to meet the trip power demand of users, so they need to meet the constraints as shown in Equations (25) and (26) on top of the battery-related constraints.

$${\delta }_t^{\mathrm{EVch}}+{\delta }_t^{\mathrm{EVdis}}\le {\delta }_t^{\mathrm{V}2\mathrm{G}}$$

(25)

$$E_{tg-1}^{\mathrm{EV}}-E_{\min }^{\mathrm{EV}}\ge s_{tg}^{\mathrm{EV}}{\zeta }^{\mathrm{EV}}$$

(26)

where ${\delta }_t^{\mathrm{V}2\mathrm{G}}$ is a binary state variable, which indicates the connection state of the electric vehicle to the grid, and the constraint is to restrict the electric vehicle to charge and discharge only when it is connected to the grid, and cannot be in the charge and discharge state at the same time; ${\delta }_t^{\mathrm{EVch}}$ and ${\delta }_t^{\mathrm{EVdis}}$ are the charging and discharging states of the electric vehicle at time t, respectively; $tg$ means the electric vehicle trip moment; $E_{tg-1}^{\mathrm{EV}}$ means the electric vehicle pre-trip storage; $E_{\min }^{\mathrm{EV}}$ indicates the minimum capacity of an electric vehicle; $s_{tg}^{\mathrm{EV}}$ means the electric vehicle trip mileage at the period $tg$; ${\zeta }^{\mathrm{EV}}$ means the electric vehicle power consumption per unit mileage.

2.2. Agent Model

2.2.1. Markov Decision Process for CIES Dispatch

The dispatch process of CIES can be represented with a Markov Decision Process (MDP) [25,26,27], and the MDP can be described with a five-tuple (S, A, P, γ, R): where S means the observation space; A means the agent action space; P is the state transfer probability, i.e., the probability of executing action a₁ in state s₁ and the state transforming to s₂; R means the reward given by the environment after the agent makes the action; and γ means the discount factor, which means the degree of influence of the reward obtained in future periods on the cumulative reward.

The community-integrated energy system dispatch cycle is 24 h, and the agent needs to make dispatch actions from the first period after observing the environmental state until the last period of the dispatch cycle, making a total of 24 decisions of dispatch actions. During the dispatch cycle, the state shifts once for each dispatch action made by the agent [28], as shown in Figure 3.

Taking into account the discount factor γ, the return obtained by the agent at time t of the dispatch cycle can be described as follows [29]:

$$U_t=R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}\cdots +{\gamma }^{23-t}{R}_{24}$$

(27)

The training goal of an agent is to learn an optimal policy that maximizes the return in the dispatch cycle.

2.2.2. SAC Deep Reinforcement Learning Algorithm

The SAC (Soft Actor–Critic) algorithm is a deep reinforcement learning algorithm based on the maximum entropy theory. Compared with the deterministic policy algorithm that selects the action with the largest action value in each step of the policy, the SAC algorithm introduces an entropy term in the policy objective and selects the action with the largest sum of action value and entropy term in each step of the policy, which has the characteristics of policy randomization, stronger generalization, robustness, and exploration ability compared with general deep reinforcement learning algorithms, and can avoid premature convergence to local optima [30,31]. The optimal strategy after introducing the entropy term is as follows:

$${\pi }_{\mathrm{soft}}^{\ast }=\arg {\max }_{\pi }{\mathrm{E}}_{(s_t,a_t)~{\rho }_{\pi }}[{\displaystyle \sum _{t=0}^T{\gamma }^{t}R(s_t,a_t)}+\alpha H(\pi (•|s_t))]$$

(28)

where $\alpha$ is the temperature parameter, which is the weighting factor of the entropy term.

The entropy term in Equation (28) is expressed as

$$H(\pi (•|s_t))=-E_{\pi }\log \pi (a’|s_t)$$

(29)

In addition to the policy function, the SAC algorithm also introduces entropy terms in the action value function and state value function, as shown in Equations (30) and (31), respectively.

$$Q_{\mathrm{soft}}(s_t,a_t)=E_{s_{t+1},a_{t+1}}[R\left(s_t,a_t\right)+\gamma Q(s_{t+1},a_{t+1})-\alpha \log (\pi (a_{t+1}|s_{t+1}))]$$

(30)

$$V_{\mathrm{soft}}(s_t)=E_{a_t}[Q\left(s_t,a_t\right)-\alpha \log \pi (a_t|s_t)]$$

(31)

2.2.3. Agent Observation Space

The observation space is the CIES state information needed by the agent in the decision-making process, which can be expressed as follows:

$$S=[P_{\mathrm{t}}^{\mathrm{Ren}},L_t^{\mathrm{e}},L_t^{\mathrm{g}},P_{t-1}^{\mathrm{GT}},{\rho }_t^{\mathrm{e}},{\rho }_t^{\mathrm{g}},T_t^{\mathrm{in}},E_t^{\mathrm{AC}},E_t^{\mathrm{ES}},E_t^{\mathrm{GS}},E_t^{\mathrm{EV}},{\delta }_t^{\mathrm{V}2\mathrm{G}},{\delta }_t^{\mathrm{e},\mathrm{RL},\mathrm{aseq}},{\delta }_t^{\mathrm{g},\mathrm{RL},\mathrm{aseq}},{\delta }_t^{\mathrm{e},\mathrm{RL},\mathrm{aall}},{\delta }_t^{\mathrm{g},\mathrm{RL},\mathrm{aall}}]$$

(32)

where $P_{\mathrm{t}}^{\mathrm{Ren}}$ means renewable energy power output; $L_t^{\mathrm{e}}$ and $L_t^{\mathrm{g}}$ mean electric load and natural gas load; ${\rho }_t^{\mathrm{e}}$ and ${\rho }_t^{\mathrm{g}}$ mean real-time electricity price and natural gas price; $E_t^{\mathrm{AC}}$, $E_t^{\mathrm{ES}}$, $E_t^{\mathrm{GS}}$, and $E_t^{\mathrm{EV}}$ mean air conditioner, battery, gas tank, and electric vehicle capacity; ${\delta }_t^{\mathrm{V}2\mathrm{G}}$ is the binary variable of whether the electric vehicle is connected to the grid or not; ${\delta }_t^{\mathrm{e},\mathrm{RL},\mathrm{aseq}}$ and ${\delta }_t^{\mathrm{g},\mathrm{RL},\mathrm{aseq}}$ mean the period of time for which the electric/gas load has been continuously curtailed, respectively; ${\delta }_t^{\mathrm{e},\mathrm{RL},\mathrm{aall}}$ and ${\delta }_t^{\mathrm{g},\mathrm{RL},\mathrm{aall}}$ mean the total number of times the electric/gas load has been curtailed, respectively.

2.2.4. Agent Action Space

The action space of an agent is the control variable that needs to be optimized to achieve CIES dispatch, which can be expressed as follows:

$$A=[P_t^{\mathrm{dc},\mathrm{ES}},P_t^{\mathrm{dc},\mathrm{GS}},P_t^{\mathrm{dc},\mathrm{EV}},P_t^{\mathrm{GT}},P_t^{\mathrm{P}2\mathrm{G}},H_t^{\mathrm{ACc}},H_t^{\mathrm{ACrs}},{\beta }_t^{\mathrm{e},\mathrm{RL}},{\beta }_t^{\mathrm{g},\mathrm{RL}},{\beta }_t^{\mathrm{e},\mathrm{SL}},{\beta }_t^{\mathrm{g},\mathrm{SL}},P_t^{\mathrm{e},\mathrm{RL}},P_t^{\mathrm{g},\mathrm{RL}}]$$

(33)

where $P_t^{\mathrm{dc},\mathrm{ES}}$, $P_t^{\mathrm{dc},\mathrm{GS}}$, and $P_t^{\mathrm{dc},\mathrm{EV}}$ means the discharge/charge power of the battery, gas storage tank, and electric vehicle, when taking a positive value for discharging, and vice versa for charging; $H_t^{\mathrm{ACrs}}$ means the discharge/charge volume of the cooling storage tank; ${\beta }_t^{\mathrm{e},\mathrm{RL}}$, ${\beta }_t^{\mathrm{g},\mathrm{RL}}$, ${\beta }_t^{\mathrm{e},\mathrm{SL}}$, and ${\beta }_t^{\mathrm{g},\mathrm{SL}}$ mean the incentive factors for curtailable electric/gas load and shiftable electric/gas load, respectively; $P_t^{\mathrm{e},\mathrm{RL}}$ and $P_t^{\mathrm{g},\mathrm{RL}}$ mean the expected curtailments in curtailable electric load and gas load, respectively.

2.2.5. Agent Reward Function

Low-carbon economic dispatch refers to dispatch with the objective function of maximizing net benefits when considering the equivalent economic costs associated with carbon emissions and minimizing carbon dioxide emissions during dispatch operations. The objective function of CIES optimal dispatch is to maximize the CIES net revenue. Based on the objective function of CIES dispatch, the reward function of the agent can be expressed as follows:

$$RE{W}_t=I_t^{\mathrm{sell},\mathrm{e}}+I_t^{\mathrm{sell},\mathrm{g}}-C_t^{\mathrm{CO}2}-C_t^{\mathrm{GTpol}}-C_t^{\mathrm{buy}}-C_t^{\mathrm{RL}}-C_t^{\mathrm{SL}}-C{F}_t$$

(34)

where $I_t^{\mathrm{sell},\mathrm{e}}$ and $I_t^{\mathrm{sell},\mathrm{g}}$ mean CIES revenue from electricity/gas sales, respectively; $C_t^{\mathrm{CO}2}$ means CIES carbon trading costs; $C_t^{\mathrm{GTpol}}$ means gas turbine pollution emission costs; $C_t^{\mathrm{buy}}$ means CIES purchased electricity/gas costs; $C_t^{\mathrm{RL}}$ means dispatch costs for load curtailment; $C_t^{\mathrm{SL}}$ means dispatch costs for load shifting; $C{F}_t$ means action out-of-limit penalty costs.

(1)

CIES revenue from electricity sales

$$I_t^{\mathrm{sell},\mathrm{e}}=P_t^{\mathrm{sell},\mathrm{in},\mathrm{e}}{\rho }_t^{\mathrm{sell},\mathrm{in},\mathrm{e}}\Delta t+P_t^{\mathrm{sell},\mathrm{out},\mathrm{e}}{\rho }_t^{\mathrm{sell},\mathrm{out},\mathrm{e}}\Delta t$$

(35)

where $P_t^{\mathrm{sell},\mathrm{in},\mathrm{e}}$ and $P_t^{\mathrm{sell},\mathrm{out},\mathrm{e}}$ mean the power sold using CIES to the users and energy market at time t, respectively; ${\rho }_t^{\mathrm{sell},\mathrm{in},\mathrm{e}}$ and ${\rho }_t^{\mathrm{sell},\mathrm{out},\mathrm{e}}$ mean the price of electricity sold using CIES to the users and energy market at time t, respectively.

(2)

CIES revenue from gas sales

$$I_t^{\mathrm{sell},\mathrm{g}}=P_t^{\mathrm{sell},\mathrm{g}}{\rho }_t^{\mathrm{sell},\mathrm{g}}\Delta t$$

(36)

where $P_t^{\mathrm{sell},\mathrm{g}}$ means the volume of gas sold using CIES to users at time t; ${\rho }_t^{\mathrm{sell},\mathrm{g}}$ means the price of gas sold using CIES at time t.

(3)

Cost of gas turbine pollution emissions

The pollution emission of gas turbines is mainly considered in this paper for sulfur oxides, ${\mathrm{SO}}_{\mathrm{X}}$, and nitrogen oxides, ${\mathrm{NO}}_{\mathrm{X}}$, as follows:

$$C_t^{\mathrm{GTpol}}=P_t^{\mathrm{GT}}{m}^{{\mathrm{GTSO}}_{\mathrm{X}}}{c}^{{\mathrm{SO}}_{\mathrm{X}}}\Delta t+P_t^{\mathrm{GT}}{m}^{{\mathrm{GTNO}}_{\mathrm{X}}}{c}^{{\mathrm{NO}}_{\mathrm{X}}}\Delta t$$

(37)

where $m^{{\mathrm{GTSO}}_{\mathrm{X}}}$ and $m^{{\mathrm{GTNO}}_{\mathrm{X}}}$ are the pollutant emission coefficients; $c^{{\mathrm{SO}}_{\mathrm{X}}}$ and $c^{{\mathrm{NO}}_{\mathrm{X}}}$ are the pollutant emission unit cost coefficients of ${\mathrm{SO}}_{\mathrm{X}}$ and ${\mathrm{NO}}_{\mathrm{X}}$, respectively.

(4)

Cost of Carbon trading

The carbon allowances that CIES needs to purchase are as follows:

$$E_t^{\mathrm{I}}=({\displaystyle \sum _{t=1}^T{P}_t^{\mathrm{GT}}{m}^{\mathrm{GTCO}2}}+{\displaystyle \sum _{t=1}^T{P}_t^{\mathrm{buy}}{m}^{\mathrm{gridCO}2}}-{\displaystyle \sum _{t=1}^{24}{k}^{\mathrm{P}2\mathrm{G}}{P}_t^{\mathrm{P}2\mathrm{G}}})-(e^{\mathrm{GT}}{\displaystyle \sum _{t=1}^T{P}_t^{\mathrm{GT}}}+e^{\mathrm{grid}}{\displaystyle \sum _{t=1}^T{P}_t^{\mathrm{buy}}})$$

(38)

where $m^{\mathrm{GTCO}2}$ and $m^{\mathrm{gridCO}2}$ are the carbon emission coefficients of the gas turbine and grid, respectively; $k^{\mathrm{P}2\mathrm{G}}$ is the CO₂ absorption coefficient of the P2G equipment; $e^{\mathrm{GT}}$ and $e^{\mathrm{grid}}$ are the unit carbon emission allowances for the gas turbine and grid, respectively.

The cost of purchasing CO2 allowances [32] can be expressed as follows:

$$C_t^{\mathrm{CO}2}=\left\{\begin{array}{ll}\rho E_t^{\mathrm{I}}\hfill & E_t^{\mathrm{I}}\le l\hfill \\ \rho (1+v)(E_t^{\mathrm{I}}-l)+\rho l\hfill & l\le E_t^{\mathrm{I}}\le 2l\hfill \\ \rho (1+2v)(E_t^{\mathrm{I}}-2l)+\rho (2+v)l\hfill & 2l\le E_t^{\mathrm{I}}\le 3l\hfill \\ \rho (1+3v)(E_t^{\mathrm{I}}-3l)+\rho (3+3v)l\hfill & 3l\le E_t^{\mathrm{I}}\le 4l\hfill \\ \rho (1+4v)(E_t^{\mathrm{I}}-4l)+\rho (4+6v)l\hfill & E_t^{\mathrm{I}}\ge 4l\hfill \end{array}\right.$$

(39)

where $\rho$ is the carbon trading base price; $v$ is the price growth rate; and $l$ is the stepped interval of carbon price growth.

(5)

Cost of purchasing electricity/gas

$$C_t^{\mathrm{buy}}=P_t^{\mathrm{buy}}{\rho }_t^{\mathrm{buy}}\Delta t$$

(40)

where $P_t^{\mathrm{buy}}$ means the amount of electricity/gas purchased using CIES; ${\rho }_t^{\mathrm{buy}}$ means the price of electricity/gas purchased using CIES.

(6)

Cost of dispatch curtailable electric/gas load

$$C_t^{\mathrm{RL}}=c^{\mathrm{RL}}{\beta }_t^{\mathrm{RL}}{\delta }_t^{\mathrm{RL}}{P}_t^{\mathrm{RL}}\Delta t$$

(41)

(7)

Cost of dispatch shiftable electric/gas load

$$C_t^{\mathrm{SL}}=c^{\mathrm{SL}}{\beta }_t^{\mathrm{SL}}{\delta }_t^{\mathrm{SL}}{P}_t^{\mathrm{SL}}\Delta t$$

(42)

3. Model Training

3.1. Construction of the Training Scenario Set

To enhance the generalization performance of the agent in the presence of uncertainties in source, load, weather, and electric vehicle trips, the training scenario set of the agent was generated using a Latin hypercube sampling method based on the idea of stratification to generate 300 scenario sets [33,34], each including renewable energy power output, electric/gas load, outdoor temperature, and electric vehicle trip plan.

Wind power output depends mainly on natural wind speed and is described with Weibull distribution [35]; PV output and outdoor temperature depend mainly on solar radiation and are described with Beta distribution [36]; electric/gas load and electric vehicle trip plan are described with normal distribution [37,38].

3.2. Construction of the Training Scenario Set

On the basis of the described reinforcement learning model, the agent was trained using a comprehensive set of training scenarios. To analyze the effect of the maximum entropy strategy on the model, the models with temperature parameters of 0.05 and 0 were trained, and the return values and carbon trading costs at 2000 episodes are shown in Figure 4.

As can be seen from Figure 4, the model with a temperature parameter of 0 converges faster because it does not consider the entropy term; the model with a temperature parameter of 0.05 considers the entropy term and begins to converge only after 250 episodes, and the training curve at the convergence stage fluctuates more, but the model is more capable of finding the optimal solution and converges to a higher return value. The model with a temperature parameter of 0.05 achieved a higher low-carbon goal.

4. Case Studies

4.1. Experimental Setup

The parameters of each power component of CIES are shown in

Table 1. Parameters of each power component of CIES.

Parameter	Value	Parameter	Value	Parameter	Value
$p_{\max }^{\mathrm{GT}}$(kW)	100	$E_{\max }^{\mathrm{ES},\mathrm{AC}}$(kWh)	50	${\eta }^{\mathrm{ESch},\mathrm{BA}}$	95%
$p_{\min }^{\mathrm{GT}}$(kW)	10	$E_{\min }^{\mathrm{ES},\mathrm{AC}}$(kWh)	0	${\eta }^{\mathrm{ESdis},\mathrm{BA}}$	95%
$\Delta P^{\mathrm{GTmax}}$(kW/h)	70	$E_{\max }^{\mathrm{ES},\mathrm{EV}}$(kWh)	20	${\eta }^{\mathrm{ESch},\mathrm{GS}}$	95%
$m^{{\mathrm{GTSO}}_{\mathrm{X}}}$	0.0098	$E_{\min }^{\mathrm{ES},\mathrm{EV}}$(kWh)	6	${\eta }^{\mathrm{ESdis},\mathrm{GS}}$	95%
$m^{{\mathrm{GTNO}}_{\mathrm{X}}}$	0.543	$P_{\max }^{\mathrm{ESch},\mathrm{BA}}$(kW)	30	${\eta }^{\mathrm{ESch},\mathrm{EV}}$	95%
$a_{\mathrm{g}}$	0.11	$P_{\max }^{\mathrm{ESdis},\mathrm{BA}}$(kW)	30	${\eta }^{\mathrm{ESdis},\mathrm{EV}}$	95%
$b_{\mathrm{g}}$	2	$P_{\max }^{\mathrm{ESch},\mathrm{GS}}$(m3)	50	${\mu }^{\mathrm{ACc}}$	2.6
$c_{\mathrm{g}}$	0	$P_{\max }^{\mathrm{ESdis},\mathrm{GS}}$(m3)	50	${\mu }^{\mathrm{ACs}}$	0.0045
$E_{\max }^{\mathrm{ES},\mathrm{BA}}$(kWh)	100	$P_{\max }^{\mathrm{ESch},\mathrm{AC}}$(kW)	20	${\mu }^{\mathrm{ACr}}$	0.0038
$E_{\min }^{\mathrm{ES},\mathrm{BA}}$(kWh)	10	$P_{\max }^{\mathrm{ESdis},\mathrm{AC}}$(kW)	20	$H_{\max }^{\mathrm{ACc}}$	25
$E_{\max }^{\mathrm{ES},\mathrm{GS}}$(m3)	150	$P_{\max }^{\mathrm{ESch},\mathrm{EV}}$(kW)	8	${\zeta }^{\mathrm{EV}}$	0.241
$E_{\min }^{\mathrm{ES},\mathrm{GS}}$(m3)	10	$P_{\max }^{\mathrm{ESdis},\mathrm{EV}}$(kW)	8

. The real-time electric/gas prices are shown in

Table 2. Parameters of real-time electricity/gas price.

Time Period	Electricity Price (USD/kWh)	Natural Gas Prices (USD/m3)
Peak section	0.143	0.043
Flat section	0.114	0.036
Valley section	0.086	0.029

. The parameters of the agent are shown in

Table 3. Parameters of SAC agent.

Time Steps per Episode	Learning Rate	Discount Factor	Batch Size	Replay Buffer Size	Soft Update Factor
24	0.0003	0.998	256	1,000,000	0.005

. The experimental data are shown in Figure 5.

In the DRUW, let the users’ response damping coefficient be 1.0; then, the benchmark willingness curve of users’ 24 h response to electric/gas demand is shown in Figure 6.

To analyze the impact of the DRUW parameters proposed in this paper on the CIES dispatch, four experimental cases are established as shown in

Table 4. Experimental cases of DRUW.

Case Index	Realization DRUW	Response Damping Coefficient	Demand Response Uncertainty Interval
1	×	—	—
2	√	1.0	[0.5,0.6]
3	√	1.3	[0.5,0.6]
4	√	1.0	[0.5–0.7]

.

4.2. Simulation Results

The dispatch results for the four cases are shown in

Table 5. CIES dispatch results under DRUW cases.

Case Index	Electric Load Curtailment (kWh)	Gas Load Curtailment (m3)	Demand Response Compensation Costs (USD)	Cost of Electricity/Gas Purchase (USD)	CIES Net Revenue (USD)
1	0	0	0	183.34	1287.00
2	101.00	39.66	7.87	170.83	1368.32
3	2.29	3.35	2.13	187.55	1303.95
4	28.18	1.34	3.17	180.49	1316.06

.

As can be seen from Table 5, the CIES net revenues for case 2–case 4, where DRUW is implemented, are all higher than case 1, where DRUW is not implemented, due to the fact that the user’s demand response participation allows for load curtailment during peak load periods and load shifting during low periods.

Comparing the results of case 3 and case 2, it can be seen that the user’s participation in the electric/gas demand response decreases due to an increase in its response damping coefficient, resulting in a decrease in both the amount of electric/gas load curtailment and demand response compensation costs. In addition, the CIES needs to purchase energy from the market to meet the system energy supply/demand balance, the cost of purchased electric/gas increases by USD 16.72, and the net revenue decreases.

Comparing the results of case 4 and case 2, it can be seen that the uncertainty of user participation in the electric/gas demand response increases due to the increase in the user’s demand response uncertainty interval and the increase in the upper limit of the interval, resulting in a decrease in both the amount of electric/gas load curtailment and demand response compensation costs. In addition, the increase in CIES’s need to purchase energy from the market to meet the system energy supply/demand balance increases the cost of purchased electricity/gas by USD 9.66 and decreases the net revenue.

It can be seen that an increase in both the response damping coefficient and the uncertainty interval of the demand response leads to a decrease in demand response participation, which in turn leads to a decrease in CIES’s net revenue.

4.3. Generalization Performance Analysis under Source and Load Uncertain Scenarios

The Monte Carlo method is used to generate source and load uncertain scenario sets with fluctuation rates of 5%, 10%, and 15%, respectively, and the scenario set includes renewable energy power output and electric/gas load, as shown in Figure 7.

The three uncertain scenarios in Figure 7 are optimally dispatched using the trained agent, and the dispatch results are shown in Figure 8, and the economic indicators related to the dispatch results are shown in

Table 6. Economic indicators of CIES dispatch results under the source and load uncertain scenarios.

Source and Load Fluctuation Rate	Cost of Electricity Purchase (USD)	Cost of Gas Purchase (USD)	Carbon Trading Costs (USD)	Demand Response Compensation Costs (USD)	CIES Net Revenue (USD)
5%	65.92	101.41	3.41	8.06	1403.97
10%	54.89	101.14	2.82	7.69	1392.19
15%	62.76	102.08	3.37	7.76	1392.87

. As can be seen from Figure 8 and Table 6, when there is uncertainty in the source and load, the agent can make corresponding decisions for the uncertain environment, i.e., the optimal dispatch of CIES is achieved considering the uncertainty in the source and load.

4.4. Generalization Performance Analysis under Outdoor Temperature Uncertain Scenarios

The uncertain scenario sets of outdoor temperature with 5%, 10%, and 15% fluctuation rates were generated with the Monte Carlo method, as shown in Figure 9a. The dispatch results of indoor temperature are shown in Figure 10. As can be seen from Figure 9b, the indoor temperature in all three uncertain scenarios is limited to the required 25.5–27.5 °C during the 24 h dispatch cycle, satisfying the indoor temperature constraint.

4.5. Generalization Performance Analysis under Uncertain Scenarios of Electric Vehicle Trips

Uncertainty about when electric vehicles will connect to and leave the grid due to the uncertainty of community users’ daily trip plans, the agent needs to realize the optimal dispatch of EVs under the uncertain trip scenario. Let the number of EVs involved in the dispatch be 20, and consider the uncertainty of trip number, trip time, and trip distance. Establish three EV trip uncertain scenarios, as shown in

Table 7. Electric vehicle trip uncertain scenario.

Scenario Index	Number of Trips	Trip Time	Trip Mileage
1	1	7:00–16:00	24
2	1	9:00–18:00	20
3	2	7:00–13:00 16:00–18:00	12 16

. The results of EV dispatch under three uncertain scenarios are shown in Figure 10 and

Table 8. Dispatch results under the uncertain scenarios of electric vehicle trips.

Scenario Index	Net Charging Volume (kWh)	Surplus Power Storage (kWh)	CIES Net Revenue (USD)
1	6.93	14.22	1368.92
2	5.00	15.18	1360.35
3	6.84	16.15	1355.24

.

As can be seen from Figure 10, in the three trip uncertain scenarios, the charging and discharging operations of the agent to the EVs only occur when electric vehicles are connected to the grid, and the charging hours in all three scenarios are concentrated in the low load hours from 0:00 a.m. to 1:00 a.m., which can reduce the charging cost; meanwhile, the EV discharging hours are concentrated in the peak load hours from 21:00 p.m. to 22:00 p.m., which can release the stored surplus power and improve the net revenue.

As can be seen from Table 8, the actual storage capacity of EVs in all three scenarios before their respective trip time can meet the trip power demand, reflecting the good generalization performance of the agent to the uncertain scenarios of EV trips.

5. Conclusions

The uncertainty of the demand response is rarely considered in existing applied research using reinforcement learning methods for energy system dispatch. However, in the community-integrated energy system, the user’s demand response behavior should be subject to uncertainty. In this paper, we develop a demand response model that takes into account the uncertainty of user behavior, and a multiple-uncertainty community-energy-system low-carbon economic dispatch model based on a deep reinforcement learning method is proposed. The proposed model considers the uncertainties of various factors such as renewable energy, electric/gas load, temperature, and electric vehicle trip, and proposes a demand response model based on the user’s willingness to the uncertainty of the user’s demand response behavior, which is combined with the SAC reinforcement learning method to realize the low-carbon economic dispatch of a community-integrated energy system under multiple uncertainties. The simulation results show the following:

(1) In the DRUW, the increase in both the response damping coefficient and demand response uncertainty interval leads to the decrease in demand response participation, resulting in the decrease in operating net revenue of the community-integrated energy system.

(2) The trained agent has good adaptability to multiple uncertainties in the community-integrated energy system and has good generalization performance in the scenarios with uncertainty in the user’s demand response behavior as well as uncertainty in source, load, outdoor temperature, and electric vehicle trips.

This paper’s model of demand response uncertainty and reinforcement-learning-based low-carbon economic dispatch in CIES with demand response uncertainty may positively influence future related research, but at the same time, there are some limitations and it is worthy of further improvement. For the user demand response uncertainty results, we mainly consider the two states of response and non-response, and in the future, a continuous response modeling the response can also be used for the demand response. A simple linear approximation is used for the description of the demand response uncertainty curve in this paper, which can be combined with Monte Carlo or other methods in the future to provide a more accurate modeling of the user’s demand response uncertainty.