Improvement of Linear and Nonlinear Control for PMSM Using Computational Intelligence and Reinforcement Learning
Round 1
Reviewer 1 Report
In this paper improvement of linear and nonlinear control for PMSM using computational intelligence and reinforcement learning is presented.
The paper is well-written and contains scientific insights. Following are some suggestions:
1. The article is focused on FOC techniques. Anyway, in the introduction, additional control techniques for PMSM should be mentioned (such as the Direct Flux Control Technique...).
2. The authors stated: " the advantage of RL-TD3 agent is that it does not use the mathematic model of the controlled process, but provides CCS after the training stages in order to improve the performance of PMSM–CS". The authors should better explain the advantages and the disadvantages. Which data are necessary to proceed with this optimization? Probably a disadvantage is the time needed to perform the optimization. How does this approach work under tolerance with respect to the model-based approach?
3. In Fig. 12, the time interval should probably be reduced, since the most interesting part is the transient (and also the steady-state, but it is reached very fast, after 0.1s). The suggestion is to highlight the transient.
Author Response
Dear reviewer, thanks for your recommendations and appreciations.
- The primary control strategy of PMSM is Direct Torque Control (DTC) [5], which is based on the direct control of torque and flux with the help of simple ON/OFF type controllers. For superior PMSM control performance, the FOC-type control strategy is used.
- First of all, we remind that RL-TD3 agent is used in this paper for two major purposes:
- A) In order to improve the control system of PMSM performance based on a controller (LQR, FL, Nonlinear, PCH), it can be used an RL-TD3 agent algorithm, which after the learning phase will be able to provide correction command signals overlapping the command signals of the controller (LQR, FL, Nonlinear, PCH).
- B) In case of using RL-TD3 agent algorithm for optimizing the vector of command parameters K for LQR-type controller (u = Kx).
The reward is used in the form:
(91)
The state vector in this case is x = [iq ω θ]T, and the command law can be expressed by the following relation:
u = [iq ωerror θ]·[k1 k2 k3]T (92)
The RL-TD3 agent used to implement the command law (92), it is implemented by using a neural network with one fully-connected layer. The vector of parameters K is obtained at the end of the learning period and represents the learnable parameters of the actor.
They can be obtained using the functions: actor = getActor (agent) and parameters = getLearnableParameters (actor).
The vector obtained is: k1 = 10, k2 = 307, and k3 = 0.3.
Indeed, in case B), the time for training the RL-TD3 agent is of the order of hours, while the time for optimization of the controller parameters using the computational intelligence algorithm is approximately 10 times less.
We specify that for the use of RL-TD3 agent, in the two cases A) and B), these elements are not introduced in the structure of the controlled process model, the interaction taking place as described in the article at the level of input and output signals based on calculation of a reward. The controlled system model is used for the synthesis of LQR, FL, Nonlinear, and PCH controllers. In case A) RL-TD3 agent provides correction command signals that overlap with the command of LQR, FL, Nonlinear, and PCH controllers. In case B) due to the particular form of the LQR-type controller command and the implementation using a neural network with one fully-connected layer, at the end of the training the weights of this layer represent the optimal values of the parameters of the LQR-type controller.
3. For each elements of Figure 12 are given details for transient stage, between 0.01s and 0.02s.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Dear Authors,
This paper comparatively presents the linear and nonlinear control of a Permanent Magnet Synchronous Motor (PMSM). These motors have are applied in Air conditioners, Refrigerators, Compressors, Robotics, Aerospace, etc... .This article presents four types of linear and nonlinear control algorithms for the control of a PMSM, together with the means of improving the PMSM control system performances by using Computational Intelligence (CI) algorithms and RL-TD3 agent algorithms. The authors have already published papers dealing with this topic (mentioned in references no. [37–41]). This paper contains theoretical part and experimental part. The authors present numerical simulation results carried out by applying up-to-date and advanced computational software and machine learning algorithms. The hardware has been developed in the MATLAB/Simulink programming environment based on Motor Control Block set (MCB) and Embedded-Coder Support-Package (ECSP). RL-TD3 is an RL (Reinforcement Learning) agent (It is improved variant of Deep Deterministic Policy Gradient - DDPG) has been considered as the most suitable RL agent for the control PMSM hardware. The response times of the RL-TD3 agent controller with correction command signals (CCS) are shorter than the other controllers (see Table 1). Figure 21 shows that the average reward is converged. In my opinion this paper should be more organized. The hardware and the software tools should be described in section 2 (Materials and Methods). The version of MATLAB software, SIMULINK and Reinforced Learning toolboxes and their versions should be described in section 2. The first part of section 8 should be moved to this section. It is recommended to reconsider this interesting paper for publication, after performing the following major revisions.
Comments and Suggestions for Authors
1) (Lines no. 63-78) The novelty of this paper should be strengthen and emphasized based on gaps of current literature and the Author’s papers published so far dealing with this topic (mentioned in references [37-41]).
2) The advantages, disadvantages and the applications of the Permanent Magnet Synchronous Motor should be described in the Introduction section.
3) A lots of abbreviations have been used in this paper. The abbreviations (such as: PMCM, FOC, LQR, GM, RL etc.) used inside the abstract and in the introduction of the paper should be defined at the end of the paper.
4) (Section 2 - LQR Control for PMSM – Equations (1) and (45)) the authors should describe the numerical libraries applied in order to solve the ordinary differential equations shown in equations (1) and (45).
5) (Section 8 – Experimental Results) the first part of this section should be moved to section 2 - Materials and Methods.
6) It is recommended to move the system photo shown in figure 29, to section 2 - Materials and Methods. The version of MATLAB software, SIMULINK and Reinforced Learning toolboxes and their versions should be described.
7) The conclusion section is very short. It should be extended.
8) (Reference no. 35, Line no. 795) the publication year of this paper should be modified. The correct year should be 2021 and not 2030 as was written. See the paper web site: https://0-link-springer-com.brum.beds.ac.uk/article/10.1007/s00521-020-05352-1
9) (References section) the digital object identifier (DOI) of the paper should appear (See the following reference example):
Author 1, A.B.; Author 2, C.D. Title of the article. Abbreviated Journal Name Year, Volume, page range, DOI.
Author Response
Dear reviewer, thanks for your recommendations.
1) We removed the papers mentioned in references [37-41] that belong to us and we added another references with Matlab toolboxes used for PMSM control system based on LQR, FL, Nonlinear, and PCH controllers, Computational Intelligence algorithms, and RL-TD3 agent implementation (was a requirement of Academic Editor).
From the point of view of the description equations of the operation of a PMSM, they are obviously nonlinear, but the approaches regarding the synthesis of the controllers can consist both in linearization and in using techniques specific to nonlinear control systems. Thus, from the first category, this article presents LQR algorithms and a control algorithm obtained by FL. Regarding the nonlinear approach, the article presents control algorithms obtained by using Lyapunov functions, but also Hamiltonian control algorithms obtained by rewriting the PMSM operating equations under a specific form. An important role in optimizing the parameter vectors in the control algorithms is the use of Computational Intelligence (CI) algorithms. Of these, four algorithms were chosen, which we consider representative of the category of CI-type algorithms, namely: Particle Swarm Optimization (PSO), Simulated Annealing (SA), Genetic Algorithm (GA), and Grey Wolf Optimization (GWO). Also, a special role in improving the performance of PMSM-CS is the use of Reinforcement Learning-Twin Delayed Deep Deterministic Policy Gradient (RL-TD3) agent algorithm. RL is a framework for learning the relationship between the states characteristic to the description of the system and the actions on it. The agent performs the maximization of a reward based on the actions on the system, which also take into account the observations on the system. Thus, we notice a similarity between the interaction of this RL-TD3 agent and that of a controller with the controlled process. One of the advantages of the RL-TD3 agent is that it does not use the mathematic model of the controlled process, but provides CCS after the training stages in order to improve the performance of PMSM-CS. Therefore, starting from the PMSM-CS performances obtained for a benchmark, this article presents four types of linear and nonlinear control algorithms for the control of a PMSM, together with the means of improving the PMSM-CS performances by using CI algorithms and RL-TD3 agent algorithms.
Thus, the main contributions of this article can be summarized by the synthesis of linear and nonlinear controllers for the control of a PMSM, together with their improved variants by using CI algorithms and RL-TD3 agent algorithms. Also, we can add the validation by numerical simulations of the proposed controllers and the presentation of the PCH-RL-TD3 as the most efficient agent. Additionally, the article presents the real-time implementation in embedded systems by completing the Software-in-the-Loop (SIL), Processor-in-the-Loop (PIL), and Hardware-in-the-Loop (HIL) stages of an LQR-CI algorithm and the superiority of its performances compared to the use of classical PI-type controllers.
2) PMSMs have a number of advantages when used in electric drives, and the purpose of this article is to obtain improved performance of its control system. The possible disadvantages given by the difficulty of construction, prices or the rare earths used in the construction of PMSM are not the subject of this article.
3) We added the nomenclature in the article as follows:
Nomenclature
PMSM Permanent Magnet Synchronous Motor;
FOC Field Oriented Control;
DTC Direct Torque Control;
PMSM–CS PMSM–Control System;
LQR Linear Quadratic Regulator;
FL Feedback Linearization;
PCH Port Controlled Hamiltonian;
CI Computational Intelligence;
PSO Particle Swarm Optimization;
SA Simulated Annealing;
GA Genetic Algorithm;
GWO Grey Wolf Optimization;
CCS Correction Command Signals;
RL Reinforcement Learning;
TD-3 Twin Delayed Deep Deterministic Policy Gradient;
MCB Motor-Control-Blockset;
ECSP Embedded-Coder Support-Package.
4) The numerical libraries applied in order to solve the ordinary differential equations shown in equations (1) to (45) are included in Matlab/Simulink programming environment and the type of the used solver is ode8 Dormand-Prince solver.
5) and 6) The presented article has a relatively large size and it was chosen to present in Sections 2, 3, 4, and 5 both the general elements and the synthesis of the type controllers: LQR, FL, Nonlinear, and PCH respectively. In Section 6, specific elements of computational intelligence and reinforcement learning agent algorithms were presented. The numerical simulations were presented in Section 7 and the PMSM control system based on LQR, FL, Nonlinear, and PCH controllers are implemented in Matlab/Simulink version 2021b and Simscape Electrical toolbox. Also, for the optimization of the controllers K parameters we used Optimization toolbox and for the RL-TD3 agent creation, implementation, and training we used Reinforcement Learning toolbox. In Section 8 the experimental results of the real-time implementation in embedded systems by completing the SIL, PIL, and HIL stages of an LQR-CI algorithm (for example) are presented and prove the superiority of its performances compared to the use of classical PI -type controllers.
For another shorter article in which the presentation of a single controller for the PMSM control system is chosen, the proposed form of organization of the article can be used. We hope for your understanding because another form of organization would mean rewriting the article…
7) We improved the conclusions section.
8) We modified the publication year of the paper (Reference no. 35).
9) We added the digital object identifier (DOI) of the papers from References section.
Author Response File: Author Response.pdf
Reviewer 3 Report
The reviewer is grateful for the opportunity to review this article. This article is about improving linear and nonlinear control for PMSM using computational intelligence and reinforcement learning. After reading the paper, everything seems well organized. The motivation behind this work, the gaps in the research, the novelties of the contributions are very clear to readers. The organization of the manuscripts, the writing in English, the graphical abstracts and the results figures and mathematical illustrations, are also made very clear. The figures are presented in high quality and colorful and artistic adopt open access publishing standards. For me, everything is presented qualitatively and adequately. This article deserves to be published as presented.
Author Response
Dear reviewer, thanks for your recommendations and appreciations.
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
Dear Authors,
Thank you very much. It is recommended to mention at end of the introduction section that detailed description of the hardware and MATLAB/Simulink, Numerical and Reinforcement Learning toolboxes are shown in sections 7 and 8. The conclusion section should be renumbered. It's number should be section 9 and not 5 as written. It is recommended to accept this contribution for publication after performing these minor revisions.
Author Response
Dear reviewer, thanks for your recommendations and appreciations.
We made all the required recommendations.
Author Response File: Author Response.pdf
