Automatic Evaluation of Neural Network Training Results

Abstract

This article is dedicated to solving the problem of an insufficient degree of automation of artificial neural network training. Despite the availability of a large number of libraries for training neural networks, machine learning engineers often have to manually control the training process to detect overfitting or underfitting. This article considers the task of automatically estimating neural network training results through an analysis of learning curves. Such analysis allows one to determine one of three possible states of the training process: overfitting, underfitting, and optimal training. We propose several algorithms for extracting feature descriptions from learning curves using mathematical statistics. Further state classification is performed using classical machine learning models. The proposed automatic estimation model serves to improve the degree of automation of neural network training and interpretation of its results, while also taking a step toward constructing self-training models. In most cases when the training process of neural networks leads to overfitting, the developed model determines its onset ahead of the early stopping method by 3–5 epochs.

1. Introduction

Training of an artificial neural network is an iterative process which involves finding parameters of the network in which it achieves the optimal performance [1,2,3]. There exists a plethora of software designed to solve the task of automatic estimation of model parameters: Auto-sklearn, Auto-WEKA, TPOT, H2O AutoML etc [4,5,6,7,8,9,10].

During training, the optimization algorithm tries to minimize the cost function of the network; however, this process does not always proceed smoothly. As such, deep learning engineers try to detect problems occurring during this process as early as possible [11,12,13]. Missing or ignoring training issues can degrade the performance of a network or render it completely unusable for the task at hand, making retraining necessary and, thus, incurring additional costs in time and resources [14].

This article proposes a method for evaluating the training process of a neural network. Three states of the training process are considered in this work: underfitting, overfitting, and optimal training. Thus, the task of evaluation of the network’s state is viewed as a multiclass classification problem [15].

We propose an algorithm for calculating feature descriptions of the loss and accuracy learning curves. The features calculated for these curves differ since the accuracy curve allows normalization of values to the [0; 1] interval, while the loss curve does not.

Use of the proposed approach allows one to increase the degree of automation in their network’s training process. The proposed approach allows for a much earlier stopping point when compared to the classic early stopping method, while not allowing overfitting and providing an additional decision reinforcement tool for novice deep learning engineers.

2. Overview of Existing Methods for Detecting Model Training Issues

1. Expert method: learning curve—a graph showing the change in a particular metric as the machine learning model is being trained. There are two main types of learning curves. Optimization curves are learning curves calculated on the basis of the metric by which model parameters are optimized, such as various types of error functions. Performance curves are learning curves calculated on the basis of the metric by which the model will be evaluated and selected later, such as accuracy, precision, recall, or a combined F₁-score.

The most popular example of a learning curve is the curve of a model’s loss over time. Loss measures a model’s error; consequently, a lower loss indicates higher performance of the model. In the long run, loss should decrease over time, indicating that the model is learning.

Another example of a learning curve is the accuracy graph, which reflects the performance of the model, whereby a higher accuracy indicates a more efficiently trained model. A model’s accuracy graph rising over time indicates that the model is improving as it accumulates experience. Over time, the graph of the performance metric reaches a plateau, which means that the model is no longer learning, and the performance limit for a given configuration has been reached.

One of the most widely used combinations of metrics is training and validation loss over time. Training loss shows how well the model matches the training data, and validation loss shows how well the model matches new data that were not available during training.

As mentioned above, the main issues that experts seek to identify during model training are model underfitting and overfitting. To detect overfitting and underfitting, machine learning experts apply a set of rules [16,17,18] according to an understanding of the mathematical process of model training.

Basic case—absence of problems during model training. The training process stops when the trend of the loss function on validation changes from downward to upward.

In the absence of issues in the training phase, the values of the model error function on training data will almost always be lower than those on test data. This means that we should expect some discontinuity between training and validation loss curves. This discontinuity is one of generalization. The optimal case is considered to be one in which the training loss curve decreases until a point of stability, the validation loss curve decreases until a point of stability, or the generalization gap is minimal (almost zero in the ideal case).

If the values of the loss function are high and do not decrease with the number of iterations for both test and training curves, this indicates insufficient complexity of the model used in relation to the data, which leads to model underfitting.

If the learning curve is linear with a derivative close to zero or shows noisy values around some constant high loss value, this indicates that the model failed to reveal patterns during training, or in other words, the fact of learning is almost (or completely) absent.

The values of training and validation loss functions continuously decreasing in the last epochs of training indicate premature cessation of training and the model’s inability to learn further.

If the values of the training loss function decrease over time, reaching acceptably low values, but the values of the validation loss function decrease only up to a certain point after which they begin to increase, then it can be concluded that the model reached an overfit.

It is worth noting that the learning curves can identify not only issues associated with the model, but also issues of unrepresentativeness of the data used to train or test the model. A representative dataset proportionally reflects the statistical characteristics of another dataset from the same subject area.

The issue of an unrepresentative training dataset occurs when the data available for training are insufficient for effective training of the model relative to the test dataset. This situation is also possible when the training dataset does not reflect the statistical parameters that are inherent to the data in a given subject area.

In such cases, the training and validation loss values decrease, but there is a large gap between the curves present, which means that the datasets used for training and validation belong to different probability distributions.

The issue of an unrepresentative validation dataset occurs if the dataset used to test the quality of model learning does not provide enough statistical information to assess the generalizability of the model. In such cases, the learning loss curve matches the basic case, while the validation loss curve shows highly noisy values in a region close to the learning loss curve.

It is possible that a validation loss may be much lower than training loss, reflecting the fact that the validation dataset is easier to predict than the training dataset. In this case, the validation dataset is too small and may be widely represented in the training dataset (i.e., there is an overlap between the training and validation datasets).

The expert method of neural network training assessment implies direct control of the network training process by an expert. In turn, this task imposes additional responsibilities and risks on the experts. The expert method depends on competence and experience of a decision maker, as well as increasing development time of a neural network. Additionally, a serious disadvantage of the method is a complete or partial lack of automation.

2. Keras API callbacks. Keras is a commonly used API for building and deploying deep learning models. A Keras API callback is a function that can be executed at different stages of the model’s training pipeline. Such functions can be used for various tasks such as controlling the model’s learning rate, which can affect the behavior of the training algorithm.

The main drawback of the described method is that the majority of parameters used to detect instances of a model’s faulty behavior during training are constants specified in advance by the researchers or a set of rules formulated by researchers in advance. It is worth noting that the various events which can be detected by this method are not classified. An event detected by a Keras callback, be it overfitting or underfitting, is assumed to be hypothetical (with the final decision being made by an expert overseeing the training process) and only valid for that specific stage of model training. Another disadvantage of these callbacks is the lack of premade automatic tools for detection of a supposed model underfit [19].

3. In [20], the authors proposed an approach to accelerate the search for optimal hyperparameters of the neural network through an early stopping of training. The criterion for such stopping is extrapolated learning curves.

In general, the process of extrapolation of a learning curve from an arbitrary number of initial values to final values is as follows: during optimization of the neural network’s parameters using the stochastic gradient descent algorithm, regular measurements of the model’s performance function are taken. Let y_1:n be the observed values of the performance function for the first n iterations of stochastic gradient descent. Then, while observing y_1:n values of the performance function, it is necessary to predict the performance y_m at step m, where m >> n. In this approach, such a problem is solved using a probabilistic model.

The basic approach is to model the partially observed learning curve y_1:n using a set of parametric functions {f₁, …, f_K}. In turn, each of these parametric functions f_k is described by a set of parameters θ_k. Assuming Gaussian noise ε ~ N(0; σ²), we can use each function f_k to model network’s performance at timestep t as y_t = f_k(t|θ) + ε. The unit observation probability y_t is, thus, defined as

$$p(y_t|{\theta }_k,{\sigma }^2)=N(y_t;f_k(t|{\theta }_k),{\sigma }^2).$$

(1)

The authors of [20] also presented a set of models of parametric curves, the shape of which coincides with the general ideas about the shape of performance curves. Usually, such curves represent an increasing, saturating function. In total, K = 11 different parametric functions are presented in the paper. It is worth noting that all the models presented cover only certain aspects of learning curves, but none of them can fully describe all of the possible curves. Therefore, there is a need to combine these models.

The stronger, combined model is a weighted linear combination of simple models:

$$f_{comb}(t|\xi )={\displaystyle \sum _{k=1}^K{w}_k{f}_k(t|{\theta }_k)},$$

(2)

where the new combined vector of parameters,

$$\xi =(w_1,\dots ,w_K,{\theta }_1,\dots ,{\theta }_K,{\sigma }^2),$$

(3)

includes the weight w_k for each model, the individual model parameters θ_k, and the noise variance σ²; then, y_t = f_comb(t|ξ) + ε.

Given such a model, it is necessary to model uncertainty and, hence, adopt a Bayesian perspective by predicting y_m values using the Monte Carlo method with Markov chains.

The early stopping method using extrapolation of learning curves provides one with an opportunity to estimate the value of an accuracy function y_m at step m; however, the decision about the model being subject to overfitting or the number of training epochs m being insufficient (i.e., underfitting) has to be made by an external mechanism or an expert [21,22]. Consequently, with this approach, the automation of the learning assessment process is only partial and requires additional methods to support itself.

4. The authors of [23] described three criteria for an early stopping of the training process. The first criterion of early stopping is the suggestion to stop training at the point when the generalization loss exceeds a certain threshold. Let E be the target (error) function of the training algorithm. E_tr(t) is the training error, an average over the training set measured after epoch t. E_va(t) is the error on the validation set. E_te(t) is the error on the test set. In real life, the generalization error is usually unknown, and only the validation error E_va(t) can be used to estimate it.

The value E_opt(t) is defined as the smallest error on the test set obtained before epoch t. The generalization error at epoch t is then defined as the relative increase in error on the test set compared to the minimum error at that point in time (as a percentage):

$$GL(t)=100\cdot \left(\frac{E_{va}(t)}{E_{opt}(t)}-1\right).$$

(4)

A high loss of generalization is one obvious possibility for stopping the training, as it directly indicates overfitting. The criterion itself can be described as

$$GL(t)>\alpha .$$

(5)

The second early stopping criterion differs from the first in that the generalization error is averaged over k previous epochs and is compared to the minimum error at those k epochs:

$$P_k(t)=1000\cdot \left(\frac{{\displaystyle \sum _{t^{\prime }=t-k+1}^t{E}_{tr}(t^{\prime })}}{k\cdot {\min }_{t^{\prime }=t-k+1}^t{E}_{tr}(t^{\prime })}-1\right).$$

(6)

It is assumed that, with large changes in the error function on a small interval (k of about five epochs), there is a greater chance of subsequently obtaining a smaller value of the generalization error. The second early stopping criterion is formulated as

$$\frac{GL(t)}{P_k(t)}>\alpha .$$

(7)

The third criterion of early stopping is where training is halted when the generalization error increases over s consecutive sequences of k epochs. The idea behind this definition is that, according to the authors’ assumption, when the validation error increases not only once, but over s consecutive sequences, such an increase indicates the beginning of final overfitting, no matter how large the actual increase is.

Choosing a particular stopping criterion, in essence, involves a tradeoff between training time and generalization error.

3. Proposed Model for Automatic Assessment of Network Training

This paper proposes a model for automatic evaluation of the training results of a neural network (Figure 1) based on quality metrics derived from its training process [24].

At each step of model training, the optimizer calculates given network performance metrics. For classification, the main metrics used are cost functions, such as binary or cross-entropy, and the rate of correct predictions. Thus, it is reasonable to consider these metrics as a time series with a step of one epoch of model training [25].

The metrics obtained from the optimizer are used to calculate a series of features that are passed as input to a classification network. The classifier then determines the state of a training process at a given epoch. Three states are defined for the model output: normal training, underfitting and overfitting. Examples of learning curves representing the dependence of the cost function on the number of training epochs are shown in Figure 2.

The classification result, i.e., an assessment of the learning state at the current epoch, is passed to a management system which, combined with the original performance metrics, determines whether to adjust the learning process, stop it, or continue without changes.

Thus, the task of automatic estimation of neural network training results is reduced to a task of multinomial classification into three classes.

4. Data Collection and Sampling

The main criteria for data selection were the types of neural network performance metrics. Only learning curves for the cost and accuracy functions that depend on the training epochs passed were selected [26].

After the data collection phase, the obtained learning curves were converted from graphical representation into time series.

It is worth noting that, due to the fact that training data were collected from multiple sources, it was not possible to collect a single coherent set of data. Therefore, two samples were generated—one including data from cost function learning curves and one including data from accuracy function learning curves.

5. Feature Descriptions of Learning Curves

It is useful to divide descriptions of the learning curves into two types: those calculated from the learning curves for both cost and accuracy functions, and those obtained from accuracy functions only.

This distinction is based on the fact that it makes sense to compute some of the proposed attributes using only normalized data. The time series obtained using learning curves of an accuracy function with values in the range [0, 1] are considered normalized. On the other hand, time series obtained from cost function curves are not normalized. We propose to introduce the common features described below for all considered learning curves, regardless of the type of function (cost or accuracy).

1. Standard deviation of the difference between the functions of a given metric on training and validation sets:

$$f_1=\sigma (F_{t_i}-F_{v_i}),i=\overline{1,N},$$

(8)

where F_t is the metric function for training, F_v is the metric function for validation, and N is the number of training epochs of a neural network model. Using this feature, we can infer how much the values of a given metric differ between training and validation, and which function has a larger average. If the values of the training function nearly match the values of the validation function, it will produce a value of f₁ that is close to zero.

2. Standard deviation of the training metric function:

$$f_2=\sigma (F_{t_i}),i=\overline{1,N}.$$

(9)

This feature can be used to understand how much the metric in question changes during training. A feature value close to zero can indicate that the neural network is not learning.

3. Standard deviation of the validation metric function:

$$f_3=\sigma (F_{v_i}),i=\overline{1,N}.$$

(10)

This feature can be used to understand how much the metric in question changes during validation. For cost functions, if the initial value is unsatisfactory with respect to the intended task, the lack of change during validation may indicate that the neural network model has not learned, i.e., is not able to detect patterns

4. Average value of a metric’s derivative function at extreme training epochs:

$$f_4=\frac{1}{N(1-k)}{\displaystyle \sum _{i=N(1-k)}^N{F}_{t_i}{}^{\prime }},$$

(11)

where k is the percentage of extreme training epochs at which the feature is calculated. The parameter k was empirically set to 10%. The sign of the feature serves as an indicator of a function’s tendency toward increasing or decreasing. The value of the derivative can be used to see how much a given metric changes by the end of training, which, together with the sign, gives us an idea of the model’s trend towards overfitting or underfitting due to an insufficient number of training epochs.

5. Average value of a metric’s derivative function at extreme validation epochs:

$$f_5=\frac{1}{N(1-k)}{\displaystyle \sum _{i=N(1-k)}^N{F}_{v_i}{}^{\prime }}.$$

(12)

6. Standard deviation of a metric’s function at extreme training epochs:

$$f_6=\sigma (F_{t_i}),i=\overline{N(1-n),N},$$

(13)

where n is the percentage of extreme training epochs at which the feature is calculated. The parameter k was empirically set to 20%. Using this feature, it is possible to infer how much the metric in question changes during the last training epochs. Values close to zero can indicate stabilization of the metric’s function at the end of training.

7. Standard deviation of a metric’s function at extreme training epochs:

$$f_7=\sigma (F_{v_i}),i=\overline{N(1-n),N}.$$

(14)

This feature can be used to infer how much the metric in question changes during the most recent validation epochs. A value close to zero may indicate that there is no overfitting.

8. A number of discrete basis functions from active perception theory [27,28,29] calculated for both metrics on training and validation:

$$f_8=-F_{t_i}-F_{t_{i+\frac{1}{4}N}}+F_{t_{i+\frac{1}{2}N}}+F_{t_{i+\frac{3}{4}N}},i=\overline{1,\frac{1}{4}N},$$

(15)

$$f_9=-F_{t_i}+F_{t_{i+\frac{1}{4}N}}+F_{t_{i+\frac{1}{2}N}}-F_{t_{i+\frac{3}{4}N}},i=\overline{1,\frac{1}{4}N},$$

(16)

$$f_{10}=F_{t_i}-F_{t_{i+\frac{1}{4}N}}+F_{t_{i+\frac{1}{2}N}}-F_{t_{i+\frac{3}{4}N}},i=\overline{1,\frac{1}{4}N},$$

(17)

$$f_{11}=-F_{v_i}-F_{v_{i+\frac{1}{4}N}}+F_{v_{i+\frac{1}{2}N}}+F_{v_{i+\frac{3}{4}N}},i=\overline{1,\frac{1}{4}N},$$

(18)

$$f_{12}=-F_{v_i}+F_{v_{i+\frac{1}{4}N}}+F_{v_{i+\frac{1}{2}N}}-F_{v_{i+\frac{3}{4}N}},i=\overline{1,\frac{1}{4}N},$$

(19)

$$f_{13}=F_{v_i}-F_{v_{i+\frac{1}{4}N}}+F_{v_{i+\frac{1}{2}N}}-F_{v_{i+\frac{3}{4}N}},i=\overline{1,\frac{1}{4}N}.$$

(20)

These features provide a relationship between the time series for which they are calculated. For example, features f₈ and f₁₁, which are calculated during the training and validation phases, respectively, show the tendency of learning curves to increase or decrease.

We also propose to introduce the following features for learning curves of the accuracy function specifically.

1. Difference between the initial and final values of the accuracy function during training:

$$f_{14}=F_{t_N}-F_{t_1}.$$

(21)

In addition to the standard deviation over the whole period of training, the value of this feature can be used to indicate how much the accuracy function has changed. A value close to zero can be interpreted as a lack of learning. More importantly, a positive sign for this feature’s value shows a trend toward improvement in the accuracy metric, and a negative sign shows a trend toward its deterioration.

2. Difference between the initial and final values of the accuracy function during validation:

$$f_{15}=F_{v_N}-F_{v_1},$$

(22)

interpreted similar to the previous feature.

3. Difference between final values of the accuracy function for training and validation:

$$f_{16}=F_{t_N}-F_{v_N}.$$

(23)

This feature can be used to determine how different the accuracy metrics are for training and validation at the end of training. With the value of the training accuracy function close to one, a value of around zero is an indication that the training was performed successfully.

4. Maximum value of the training accuracy function:

$$f_{17}=\max (F_t).$$

(24)

5. Maximum value of the validation accuracy function:

$$f_{18}=\max (F_v).$$

(25)

6. Difference between the initial and final values of the accuracy function at extreme training epochs:

$$f_{19}=F_{t_{{}_N}}-F_{t_{N(1-n)}}.$$

(26)

The parameter n was empirically set to 20%. A near-zero value of this feature, together with a near-zero standard deviation for the last training epochs, may indicate a stabilization of the accuracy function’s values.

7. Difference between the initial and final values of the accuracy function at extreme validation epochs:

$$f_{20}=F_{v_{{}_N}}-F_{v_{N(1-n)}}.$$

(27)

A near-zero value for this feature, together with a near-zero standard deviation for the last validation epochs, can indicate stabilization of the accuracy function for validation. If the value of the accuracy function is close to one, it can be used as an indicator that the learning process was performed successfully. If the value of the accuracy function is close to zero for validation, we can say that the model has not trained enough or that it is now in a stabilized overfit.

8. The area under the curve of the accuracy function for learning/validation (f₂₁). A near-zero value of this feature may indicate underfitting of the neural network.

9. The area under the learning curve of the accuracy function on validation (f₂₂). A near-zero value can indicate underfitting of the neural network, provided a maintained upward trend of the accuracy function, as well as overfitting in situations where the derivative of the accuracy function is negative for the last stages of validation.

Thus, the feature descriptions of learning curves consist of 13 features for the loss functions and 22 features for the accuracy function.

6. Classifying the State of the Model

On the basis of the generated data samples and their feature descriptions, we decided to create a classification algorithm based on two independent classifiers (Figure 3).

This algorithm takes learning curve feature descriptions of both cost and accuracy functions obtained during the feature extraction process as input.

Next, two classifiers determine the probabilities that the learning curve feature descriptions indicate one of the three training states: normal training, underfitting, and overfitting.

After this, the maximum value across obtained coefficients is selected, and a decision about the state of model training is made. Thus, the classification algorithm consists of three main stages:

• Direct classification of input feature descriptions of learning curves by two independent classifiers.
• Combination of classification results.
• Determination of the model’s state.

7. Results

In this selection experiment, after the feature extraction process and the SMOTE [30,31,32,33] data augmentation algorithm, we obtained the following distribution of training samples: cost function learning curve data sample size—387 objects (129 objects for each class); accuracy function learning curve data sample size—225 objects (75 objects for each class). The experiment consisted of performing cross-validation (

Table 1. Classifiers’ performance quality metrics.

Datasets	Learning Curves’ Cost Functions			Learning Curves’ Accuracy Functions
Models	Accuracy	Precision	Recall	Accuracy	Precision	Recall
Decision Tree	0.734	0.733	0.734	0.791	0.795	0.791
SVC (OvO)	0.336	0.554	0.336	0.640	0.701	0.640
SVC (OvO, polynomial features)	0.337	0.328	0.337	0.724	0.719	0.724
SVC (OvA)	0.354	0.392	0.408	0.782	0.806	0.796
SVC (OvA, polynomial features)	0.440	0.417	0.411	0.810	0.824	0.822
K-neighbors	0.612	0.615	0.612	0.769	0.770	0.769
Logistic regression	0.292	0.293	0.292	0.649	0.701	0.646
Gradient boosting	0.780	0.789	0.788	0.879	0.872	0.872
Random forest	0.860	0.862	0.860	0.880	0.876	0.876

).

From the obtained results, it is clear that, with respect to selected performance metrics, the most efficient classifier models for determining the state of the training process were the random forest classifier models.

To conduct a computational experiment comparing the proposed model with its counterparts, it was decided to create and train 20 artificial neural networks to solve a classification problem.

The criterion by which the models were compared with each other was the average value of differences in the number of training epochs after which the training process was stopped. In the case of the proposed model, the epoch at which training is halted was the epoch at which the model first outputs “overfit”.

$$Q=\frac{1}{M}{\displaystyle \sum _{i=1}^M{n}_m^{(i)}}-n_a^{(i)},$$

(28)

where M is the number of neural network models, n_m is the index of the last epoch computed by the proposed model, and n_a is the last epoch computed by the counterpart.

For each neural network model, a fixed maximum of 100 training epochs was specified. The system’s performance was compared to the “early stopping” method provided by the Keras API, without any manual configuration on our part.

The obtained results showed a value of Q = −0.15 for API Keras callbacks. The negative and near-zero value of the Q parameter indicates that, in most cases, the proposed model detected overfitting simultaneously with the “early stopping” method of API Keras and, occasionally, 3–5 epochs ahead of it.

However, it is worth noting that the API Keras callback method does not provide information to interpret the training results, unlike the proposed model.

Figure 4, Figure 5 and Figure 6 show examples of the results of the computational experiment. The vertical line indicates the epoch at which the training process was stopped by the “early stopping” API Keras method.

8. Discussion

We reviewed existing methods for detecting training issues in neural networks [34]. As a result, a number of conclusions could be made about the shortcomings of the reviewed methods. We then proposed a model for automatic evaluation of neural network training results.

This approach not only allows one to automate the process of identifying issues during training, but also provides a toolkit that allows nonexperts to train deep learning models without the need for additional consultation on the interpretation of their results.

The computational experiment showed that the performance metrics of the training state classification system were comparable to those of its counterparts.

In the future, the following improvements of the model are planned:

• Developing a smart control system for the training process which allows one to change the optimizer’s configuration during model training to influence its behavior.
• Extending the set of models supported by the proposed method.
• Developing an algorithm to identify complexity issues in a network’s architecture for a particular dataset using the proposed approach.