Difference-Based Mutation Operation for Neuroevolution of Augmented Topologies†
Round 1
Reviewer 1 Report
This paper proposes a difference-based mutation operator for the NEAT scheme.
The paper is overall well-written and enjoyable. The authors can find below my concerns:
1) in the Introduction (page 2, approx line 40), the authors must better specify the differences between this paper and their previous work (ref. [11])
2) the authors should better describe Figure 1. For example:
- it seems that there are 4 input nodes (In_1, In_2, In_3, In_4) instead of 2
- the connection weights in the lower part of Figure 1 (0, -1.335, -1.708 and so on) cannot be found in the upper part of the Figure. Similarly, the weights on the upper part of the Figure (0.5, 0.1, 0.7 and so on) cannot be found in the lower part of the Figure
- which of these nodes is "number 57"?
- the connections in the lower part of the Figure are unnumbered
3) the scaling/sampling factor F (see e.g. Equation 4) should be named differently because F already refers to the fitness function (see e.g. Equation 2)
4) Tables 3 and 4 refer to an "average accuracy", but the authors should specify such average on which/how many experiments has been evaluated
5) In Tables 3 and 4, it would be better to highlight (e.g., in bold) the most performing technique for each dataset
6) on page 10, line 264, the authors should avoid keras-like nomenclature: what do beta1 and beta2 refer to?
7) the comparison in Table 7 seems unfair to me: we have NEAT (that is, an optimised classifier) against non-optimised classifiers whose Sklearn default hyperparameter values have been used instead
8) In Table 7, it would be better to highlight (e.g., in bold) the most performing technique for each dataset
9) Lines 176 and 183, it would be better to put the mathematical symbol for "pi" rather than writing "PI"
10) At the beginning of Section 3, the authors mention that imbalanced datasets are difficult to solve for many machine learning algorithms. This is not properly true, in general, and from Table 7 we can see that, for example, a simple Random Forest behaves nicely on these datasets.
I suggest the authors to remove that sentence.
11) The list of abbreviations is incomplete. For example "RIP" (rotary inverted pendulum) is missing
12) The paper needs a thorough English editing. Some sentences have no verbs and some plural/singular forms are wrong.
Author Response
This paper proposes a difference-based mutation operator for the NEAT scheme.
The paper is overall well-written and enjoyable. The authors can find below my concerns:
Answer: Thank you for your detailed and valuable comments, we have made several corrections to the manuscript following them.
1) in the Introduction (page 2, approx line 40), the authors must better specify the differences between this paper and their previous work (ref. [11])
Answer: Added "with sensitivity analysis of scaling factor influence and modified population update scheme".
2) the authors should better describe Figure 1. For example:
- it seems that there are 4 input nodes (In_1, In_2, In_3, In_4) instead of 2
- the connection weights in the lower part of Figure 1 (0, -1.335, -1.708 and so on) cannot be found in the upper part of the Figure. Similarly, the weights on the upper part of the Figure (0.5, 0.1, 0.7 and so on) cannot be found in the lower part of the Figure
- which of these nodes is "number 57"?
- the connections in the lower part of the Figure are unnumbered
Answer: the lower part of Figure 1 was changed to correspond to the upper part.
3) the scaling/sampling factor F (see e.g. Equation 4) should be named differently because F already refers to the fitness function (see e.g. Equation 2)
Answer: used "fit" instead of "F" to avoid confusion.
4) Tables 3 and 4 refer to an "average accuracy", but the authors should specify such average on which/how many experiments has been evaluated
Answer: we have expanded the sentence describing the cross-validation before Table 3: "The 10-fold cross-validation was applied to estimate the efficiency of the classifier, which was the accuracy measure, i.e. the number of correctly classified instances relative to total sample size, averaged over 10 folds."
5) In Tables 3 and 4, it would be better to highlight (e.g., in bold) the most performing technique for each dataset
Answer: highlighted best accuracies for every dataset.
6) on page 10, line 264, the authors should avoid keras-like nomenclature: what do beta1 and beta2 refer to?
Answer: changed notation and described the meaning of both parameters in ADAM optimizer.
7) the comparison in Table 7 seems unfair to me: we have NEAT (that is, an optimized classifier) against non-optimized classifiers whose Sklearn default hyperparameter values have been used instead
Answer: The NEAT-DBM algorithm was not specifically tuned to solve this kind of problems, and the optimization, which is only the choice of scaling factor sampling parameter, resulted in the same value of 0.5 for both classification and control problems, which shows that this is a universal setting for NEAT. Considering this, and the fact that all the methods, same as NEAT-DBM had the same settings for all classification datasets, we believe that this comparison should be considered fair.
8) In Table 7, it would be better to highlight (e.g., in bold) the most performing technique for each dataset
Answer: highlighted best accuracies for every dataset.
9) Lines 176 and 183, it would be better to put the mathematical symbol for "pi" rather than writing "PI"
Answer: corrected.
10) At the beginning of Section 3, the authors mention that imbalanced datasets are difficult to solve for many machine learning algorithms. This is not properly true, in general, and from Table 7 we can see that, for example, a simple Random Forest behaves nicely on these datasets.
I suggest the authors to remove that sentence.
Answer: removed this part of the sentence.
11) The list of abbreviations is incomplete. For example "RIP" (rotary inverted pendulum) is missing
Answer: added Rotary Inverted Pendulum (RIP) to abbreviations.
12) The paper needs a thorough English editing. Some sentences have no verbs and some plural/singular forms are wrong.
Answer: made several minor corrections.
Other corrections: added two more methods for comparison on classification problems, namely K-NN and SVM, and performed statistical tests. Added new figure showing the block scheme for classifier testing. Improved the introduction section by explaining the contents of cited papers, as well as broadening the literature review, which now includes some recent studies on weights tuning in neuroevolution.
Reviewer 2 Report
- Results: Recommend to be Major revised
This paper proposes the novel search operation for the neuroevolution of augmented topologies, namely the difference-based mutation. The implemented neuroevolution algorithm allows backward connections and loops in the topology, and uses a set of mutation operators, including connections merging and deletion. The experimental results prove that the newly developed operator delivers significant improvements to the classification quality in several cases, and allow finding better control algorithms.
It is with some merits for Algorithms, however, it requires some major revisions.
Firstly, for Section 1, authors should provide the comments of the cited papers after introducing each relevant work. What readers require is, by convinced literature review, to understand the clear thinking/consideration why the proposed approach can reach more convinced results. This is the very contribution from authors. In addition, authors also should provide more sufficient critical literature review to indicate the drawbacks of existed approaches, then, well define the main stream of research direction, how did those previous studies perform? Employ which methodologies? Which problem still requires to be solved? Why is the proposed approach suitable to be used to solve the critical problem? We need more convinced literature reviews to indicate clearly the state-of-the-art development.
For Section 2, authors should introduce their proposed research framework more effective, i.e., some essential brief explanation vis-à-vis the text with a total research flowchart or framework diagram for each proposed algorithm to indicate how these employed models are working to receive the experimental results. It is difficult to understand how the proposed approaches are working. For the employed data set, please provide more details illustration.
For Section 3, authors should use more alternative models as the benchmarking models, authors should also conduct some statistical test to ensure the superiority of the proposed approach, i.e., how could authors ensure that their results are superior to others? Meanwhile, authors also have to provide some insight discussion of the results. Authors can refer the following references for conducting statistical test.
Support vector regression model based on empirical mode decomposition and auto regression for electric load forecasting. Energies, 2013, 6(4), 1887-1901.
Author Response
This paper proposes the novel search operation for the neuroevolution of augmented topologies, namely the difference-based mutation. The implemented neuroevolution algorithm allows backward connections and loops in the topology, and uses a set of mutation operators, including connections merging and deletion. The experimental results prove that the newly developed operator delivers significant improvements to the classification quality in several cases, and allow finding better control algorithms.
It is with some merits for Algorithms, however, it requires some major revisions.
Answer: Thank you for your detailed and valuable comments, we have made several corrections to the manuscript following them.
Firstly, for Section 1, authors should provide the comments of the cited papers after introducing each relevant work. What readers require is, by convinced literature review, to understand the clear thinking/consideration why the proposed approach can reach more convinced results. This is the very contribution from authors. In addition, authors also should provide more sufficient critical literature review to indicate the drawbacks of existed approaches, then, well define the main stream of research direction, how did those previous studies perform? Employ which methodologies? Which problem still requires to be solved? Why is the proposed approach suitable to be used to solve the critical problem? We need more convinced literature reviews to indicate clearly the state-of-the-art development.
Answer: We have revised the introduction by adding more detailed explanation of the contents of cited papers, and broadened the literature review by citing some recent studies considering the weight tuning in neuroevolutionary approaches.
For Section 2, authors should introduce their proposed research framework more effective, i.e., some essential brief explanation vis-à-vis the text with a total research flowchart or framework diagram for each proposed algorithm to indicate how these employed models are working to receive the experimental results. It is difficult to understand how the proposed approaches are working. For the employed data set, please provide more details illustration.
Answer: For a better explanation of the DBM approach, Figure 3 is added showing the identification of shared connection between several solutions demonstrated in Figure 2, as well as the resulting solution with the changed corresponding weight. Also, a new figure is added with block scheme diagram showing the main steps implemented to test the NEAT and NEAT-DBM on a set of classification problems.
For Section 3, authors should use more alternative models as the benchmarking models, authors should also conduct some statistical test to ensure the superiority of the proposed approach, i.e., how could authors ensure that their results are superior to others? Meanwhile, authors also have to provide some insight discussion of the results. Authors can refer the following references for conducting statistical test.
Support vector regression model based on empirical mode decomposition and auto regression for electric load forecasting. Energies, 2013, 6(4), 1887-1901.
Answer: Two more alternative classification methods are added to the comparison, namely K-NN and SVM, and the Mann-Whitney statistical test are performed to compare NEAT-DBM with all other approaches. The discussion section is expanded describing the results.
Other corrections: Highlighted best accuracies for every dataset, changed notation and described the meaning of both parameters in ADAM optimizer, added Rotary Inverted Pendulum (RIP) to abbreviations, the lower part of Figure 1 was changed to correspond to the upper part, expanded the sentence describing the cross-validation before Table 3.
Round 2
Reviewer 1 Report
The authors did a very good job in revising the manuscript that I do think now is ready for publication.
I have a final minor remark: at page 11, line 282, the authors list the competitors (DR, RF, HEFCA) and they do not list the newly added competitors (K-NN and SVM). Such new competitors will only be listed at lines 294-295. So I invite the authors to either merge these two sentences or introduce also the new competitors at line 282-283.
Reviewer 2 Report
Authors have completely addressed all my concerns.
