Correlation Assessment of the Performance of Associative Classifiers on Credit Datasets Based on Data Complexity Measures

Abstract

One of the four basic machine learning tasks is pattern classification. The selection of the proper learning algorithm for a given problem is a challenging task, formally known as the algorithm selection problem (ASP). In particular, we are interested in the behavior of the associative classifiers derived from Alpha-Beta models applied to the financial field. In this paper, the behavior of four associative classifiers was studied: the One-Hot version of the Hybrid Associative Classifier with Translation (CHAT-OHM), the Extended Gamma (EG), the Naïve Associative Classifier (NAC), and the Assisted Classification for Imbalanced Datasets (ACID). To establish the performance, we used the area under the curve (AUC), F-score, and geometric mean measures. The four classifiers were applied over 11 datasets from the financial area. Then, the performance of each one was analyzed, considering their correlation with the measures of data complexity, corresponding to six categories based on specific aspects of the datasets: feature, linearity, neighborhood, network, dimensionality, and class imbalance. The correlations that arise between the measures of complexity of the datasets and the measures of performance of the associative classifiers are established; these results are expressed with Spearman’s Rho coefficient. The experimental results correctly indicated correlations between data complexity measures and the performance of the associative classifiers.

1. Introduction

Numerous classification models have been developed in the scientific literature, with different theoretical foundations. One of the pioneers is the k-nearest neighbors (kNN) classifier [1], which is based on neighborhood criteria. Other well-known classifiers are based on probabilistic approaches, such as naïve Bayes [2], whereas others use information theory as part of their foundation (for example, ID3 [3] and C4.5 [4]).

From another perspective and focusing on finding hyperspaces where decision classes are separated, support vector machine (SVM) models have been developed [5]. Models of the human nervous system have also been used for classification tasks through the proposal of artificial neural networks (ANN) [6].

On the other hand, in Mexico, the associative approach has been developed, with the Alpha-Beta associative models [7] and their derivatives, for classification tasks. Among these derived models are the Smallest Normalized Distance Associative Memory (SNDAM) method [8] and the Hybrid Associative Classifier with Translation (HACT) method [9], also known as CHAT for its Spanish acronym, and its derivative, the CHAT-OHM classifier [10].

Another classification model within this approach is the Gamma classifier [11], with its versions of weight adjustments with differential evolution (CAG-ED) [12], its version for handling mixed and incomplete data (the Extended Gamma associative classifier, EG) [13], and the version of the latter which uses weight adjustment with differential evolution (Modified Associative Gamma Classifier, CAGM-ED) [14].

In addition to the previous models, the Naïve Associative Classifier (NAC) [15] stands out, with its disambiguation variants (NACe) [16] NACe-kVotes and NACe-kMajority [17] and its variant for the classification of data streams (Naïve Associative Classifier for Online Data, NACOD) [18].

Alpha-Beta associative models and their derivatives have shown satisfactory performance in numerous classification tasks from different fields. Among these, the medical [19], financial [20], social [14], productive [21], educational [22,23], and environmental [24] fields can be highlighted.

However, with so many different models available, how can one know which model will offer the best classification for a specific problem? The focus of our research is on the behavior of associative classifiers in the financial field. Several studies indicate that associative-based classifiers have shown good performance for finance-related datasets, outperforming several of the well-known supervised classifiers such as support vector machines, neural networks, decision trees, statistical-based classifiers, and others [15,16,25,26]. This is where the topic of interest for this paper appears and why we decided to focus on associative-based classifiers.

The algorithm selection problem (ASP) is a challenging learning task, focused on selecting proper classification algorithms for a given problem [27]. ASP has been the focus of various studies in the machine learning community, and for the supervised classification domain, meta-learning has shown ample achievements in solving the ASP. However, even though numerous studies have been conducted using diverse classifiers in the literature, to the knowledge of the authors of this study, there has been no comparative study that critically analyzes, summarizes, and evaluates the performance of associative classifiers derived from Alpha-Beta models, that allows the identification of the measures of data complexity that influence the good or bad performance of these models, considering their correlation with the data complexity measures.

This paper addresses a problem of which the solution would be beneficial for researchers in artificial intelligence, pattern recognition, and related areas: How can we know, in advance, in which dataset(s) a specific classifier will exhibit good (or bad) performance? The innovative and scientific contribution of this paper consists in the analysis of the performance of associative classifiers, considering their correlation with the measures of data complexity, corresponding to six categories based on specific aspects of the datasets.

In this study, we intended to make use of different complexity measures that will help to understand how some of the mentioned associative-based classifiers behave in the financial field, based on the data of the problem to be solved. The experimental results correctly indicate the existing correlations between complexity measures and the performance of the associative classifiers.

2. Background

Ho and Basu attribute the complexity of supervised classification problems to a combination of three main factors: the ambiguity of the classes, the scarcity and dimension of the data, and the complexity of the limit that separates the classes [28].

To determine the presence of such factors, several data descriptors have been introduced, based mainly on the geometric and statistical properties of the data. The data complexity measures proposed by Ho and Basu [28] are dedicated to computing the complexity of the boundary needed to separate binary classification problems (later extended for multiclass problems. Ho and Basu split their measures into three categories: individual feature value overlap measures; class separability measures; and geometry, topology, and density measures of multiple measures.

Ho and Basu’s work was instrumental in determining the complexity of a given dataset using data descriptors. Furthermore, Mansilla and Ho [29] studied the competence domains of the XCS classification system (the learning classifier system type) in the area of complexity measures and found that difficult problems for this system are characterized by a large number of samples located in the decision boundaries, the presence of a high number of adherence subsets, a high dispersion of class samples, and a high level of nonlinearity in the shapes of classes and boundaries.

A couple of years later, Sánchez et al. [30] analyzed how the complexity of the training data affects the kNN classifier and showed that the performance of this classifier is very susceptible to class overlapping and class density; in addition, they found that high dimensionality also affects the performance of kNN. Sanchez et al. also pointed out that measures are domain-dependent. That is, the performance of the classifier will depend on the characteristics of the problem.

In their studies, Luengo and Herrera presented two rules that are “simple, interpretable, and precise” to describe the good and bad performance of the FH-GBML method [31]. Two years later, they conducted another study [32]. They computed five data complexity measures and class separability measures to obtain ranges of such metrics for which the method performed significantly well or poorly.

The competence domains of the semi-naïve Bayes network classifiers (BNC) have also been studied by Flores, Gómez and Martínez [33]. They define the general attributes of ideal datasets for classification with BNCs. They also provide an automatic method to find the best BNC (in terms of prediction errors) given a particular dataset.

One year later, Luengo and Herrera [34] proposed a method of automatically extracting the competence domains for classifiers using measures of data complexity. The objective was to recognize whether the classification problem was good or bad for each classifier and obtain intervals of good performance for the classifiers based on data complexity measures. Their results showed that the domains of competence obtained were general enough to find the right datasets for the classifiers.

Years later, Morán-Fernández et al. [35] studied micro-array data, showing that the classification performance can be predicted by complexity measures and observed that attribute selection can reduce the complexity of the data.

In 2018 Barella [36] and collaborators studied data complexity measures for imbalanced classification tasks. They showed that several of the existing data complexity measures do not consider the difficulty of imbalanced classification problems. Therefore, they improved some data complexity measures to assess classes one by one and focus on the minority class. However, they also showed that data complexity measures mainly approximate majority class difficulty, which can be misleading, especially for imbalanced tasks with high class overlap. In addition, Barella suggested evaluating the difficulty of both classes in understanding the full classification problem, although minority class difficulty is often the main challenge in imbalanced tasks.

A year later, Lorena et al. [37] reviewed the main measures of data complexity in the literature and concluded that these measures allow one to characterize the complexity of a classification problem considering a geometrical approach and the distribution of data within or between classes. In 2020 Khan and collaborators [38] produced a survey about meta-learning for classifier selection. Using this approach, they deeply analyzed the problem of recommending a classification algorithm based on meta-learning. Their in-depth analysis showed that meta-learning was successful in the automatic algorithm recommendation process.

A recent study by Maillo et al. [39] analyzed redundancy and complexity measures for big data. The authors emphasized that it is possible to obtain similar or better results in several big data tasks by using a small set of quality data.

Since no classifier can regularly obtain the best performance for each classification problem, an effect proved in the no free lunch theorems [40], the analysis of data complexity measures allows us to understand the situations in which a certain classifier is successful and in which it fails.

3. Materials and Methods

This section describes the most important data complexity measures and some associative classifiers derived from the Alpha-Beta associative models that will be tested against different datasets for the development of this area of research.

3.1. Data Complexity Measures

Some of the most widely used data complexity measures are those proposed by Ho and Basu [28] and later extended in [30,41] to deal with multiclass classification problems. Ho and Basu [28] split the data complexity measures into three groups: (1) individual characteristic value overlap measures; (2) class separability measures; and (3) geometry, topology, and manifold density measurements. Likewise, Sotoca et al. [42] divided complexity measures into the following groups: (1) overlap measures, (2) class separability measures, and (3) geometry and density measurements.

Subsequently, Lorena et al. [37] classified complexity measures into six categories:

• Feature-based measures;
• Linearity measures;
• Neighborhood measures;
• Network measures;
• Dimensionality measures; and
• Class imbalance measures.

In this work, we use five of the six types of measures mentioned above. To compute the measures, we assume that we have a learning dataset “T” that contains pairs of instances $\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}}\right)$, where ${\mathrm{x}}_{\mathrm{i}}=\left({\mathrm{x}}_{\mathrm{i}1},\dots ,{\mathrm{x}}_{\mathrm{im}}\right)$ and ${\mathrm{y}}_{\mathrm{i}}\in \left\{1,\dots ,{\mathrm{n}}_{\mathrm{c}}\right\}$. That is, each instance $x_i$ is described by $m$ attributes and also has a class label $y_i$ of one of the $n_c$ classes.

Most data complexity measures assume that the attributes are numeric and complete (no missing values). Categorical values need to be codified into numeric ones, and missing values must be deleted or imputed. An additional assumption is that linearly separable problems are simpler than nonlinear classification problems.

Table 1. Complexity measures used.

Measure	Equation
Maximum Fisher’s Discriminant Ratio (F1)—Complementary formulation [37]	$\mathrm{F}1=\frac{1}{1+{\max }_{\mathrm{i}=1}^{\mathrm{m}}{\mathrm{r}}_{ƒ\mathrm{i}}}$
Volume of overlap region (F2)	$\mathrm{F}2={{\displaystyle \prod }}_{\mathrm{i}}^{\mathrm{m}}\frac{\mathrm{overlap}\left(\mathrm{fi}\right)}{\mathrm{range}\left(\mathrm{fi}\right)}={{\displaystyle \prod }}_{\mathrm{i}}^{\mathrm{m}}\frac{\max \{0,\min \max \left(\mathrm{fi}\right)-\max \min \left(\mathrm{fi}\right)\}}{\max \max \left(\mathrm{fi}\right)-\min \min \left(\mathrm{fi}\right)}$, where $\min \max \left(\mathrm{fi}\right)=\min (\max \left({\mathrm{fi}}^{\mathrm{c}1}\right),\max \left({\mathrm{fi}}^{\mathrm{c}2}\right))$, $\max \min \left(\mathrm{fi}\right)=\max (\min \left({\mathrm{fi}}^{\mathrm{c}1}\right),\min \left({\mathrm{fi}}^{\mathrm{c}2}\right)),\max \max \left(\mathrm{fi}\right)=\max (\max \left({\mathrm{fi}}^{\mathrm{c}1}\right),\max \left({\mathrm{fi}}^{\mathrm{c}2}\right)),$ $\min \min \left(\mathrm{fi}\right)=\min (\min \left({\mathrm{fi}}^{\mathrm{c}1}\right),\min \left({\mathrm{fi}}^{\mathrm{c}2}\right))$ and max$\left({\mathrm{fi}}^{\mathrm{cj}}\right)$ and min$\left({\mathrm{fi}}^{\mathrm{cj}}\right)$ are the maximum and minimum values of each feature in the class cj ∈ {1, 2}, respectively.
Maximum Individual Feature Efficiency (F3)—Complementary formulation [37]	$\mathrm{F}3={\min }_{\mathrm{i}=1}^{\mathrm{m}}\frac{{\mathrm{n}}_{\mathrm{o}}\left(\mathrm{fi}\right)}{\mathrm{n}}$, where ${\mathrm{n}}_{\mathrm{o}}\left(\mathrm{fi}\right)={{\displaystyle \sum }}_{\mathrm{j}}^{\mathrm{n}}\mathrm{I}({\mathrm{x}}_{\mathrm{ji}}>\max \min \left(\mathrm{fi}\right)\wedge {\mathrm{x}}_{\mathrm{ji}}<\min \max \left(\mathrm{fi}\right))$
Sum of the Error Distance by Linear Programming (L1)	$\mathrm{L}1=1-\frac{1}{1+SumErrorDist}=\frac{SumErrorDist}{1+SumErrorDist}$, where $SumErrorDist=\frac{1}{\mathrm{n}}{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{\mathsf{\epsilon }}_{\mathrm{j}}$. The ${\mathsf{\epsilon }}_{\mathrm{j}}$ values are determined by optimizing an SVM
Error rate of linear classifier (L2)	$\mathrm{L}2=\frac{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}\mathrm{I}\left(\mathrm{h}\left({\mathrm{x}}_{\mathrm{i}}\right)\ne {\mathrm{y}}_{\mathrm{i}}\right)}{\mathrm{n}}$ where h(x) is the linear classifier.
Nonlinearity of a Linear Classifier (L3)	$\mathrm{L}3=\frac{1}{\mathrm{l}}{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{l}}\mathrm{I}({\mathrm{h}}_{\mathrm{T}}({\acute{\mathrm{x}}}_{\mathrm{i}})\ne {\acute{\mathrm{y}}}_{\mathrm{i}})$ where ${\mathrm{h}}_{\mathrm{T}}\left(\mathrm{x}\right)$ is the linear classifier induced using the original dataset T, and l in the number of interpolated examples ${\acute{\mathrm{x}}}_{\mathrm{i}}$ and ${\acute{\mathrm{y}}}_{\mathrm{i}}$ are the corresponding labels.
Fraction of Borderline Points (N1)	$\mathrm{N}1=\frac{1}{\mathrm{n}}{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}\mathrm{I}(\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{x}}_{\mathrm{j}}\right)\in \mathrm{MST}\wedge {\mathrm{y}}_{\mathrm{i}}\ne {\mathrm{y}}_{\mathrm{j}})$, where MST is a Minimum Spanning Tree obtained over the interpolated data.
Ratio of Intra/Extra Class Nearest Neighbor Distance (N2)	$\mathrm{N}2=1-\frac{1}{1+\mathrm{intra}\_\mathrm{extra}}=\frac{\mathrm{intra}\_\mathrm{extra}}{1+\mathrm{intra}\_\mathrm{extra}}$, where $\mathrm{intra}\_\mathrm{extra}=\frac{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}\mathrm{d}({\mathrm{x}}_{\mathrm{i}},\mathrm{NN}\left({\mathrm{x}}_{\mathrm{i}}\right)\in {\mathrm{y}}_{\mathrm{i}})}{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}\mathrm{d}({\mathrm{x}}_{\mathrm{i}},\mathrm{NN}\left({\mathrm{x}}_{\mathrm{i}}\right)\in {\mathrm{y}}_{\mathrm{j}}\ne {\mathrm{y}}_{\mathrm{i}})}$
Error Rate of the Nearest Neighbor Classifier (N3)	$\mathrm{N}3=\frac{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}\mathrm{I}\left(\mathrm{NN}\left({\mathrm{x}}_{\mathrm{i}}\right)\ne {\mathrm{y}}_{\mathrm{i}}\right)}{\mathrm{n}}$
Nonlinearity of the Nearest Neighbor Classifier (N4)	$\mathrm{N}4=\frac{1}{\mathrm{l}}{\displaystyle {\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{l}}}\mathrm{I}({\mathrm{NN}}_{\mathrm{T}}({\mathrm{x}}_{\mathrm{i}}^{,})\ne {\mathrm{y}}_{\mathrm{i}}^{,})$
Fraction of Hyperspheres Covering Data (T1)	$\mathrm{T}1=\frac{\#\mathrm{Hyperspheres}\left(\mathrm{T}\right)}{\mathrm{n}}$, where $\#\mathrm{Hyperspheres}\left(\mathrm{T}\right)$ is the number of hyperspheres needed to cover the dataset, considering that only points of the same class can be inside the same hypersphere.
Average Number of Features per Dimension (T2)	$\mathrm{T}2=\frac{\mathrm{m}}{\mathrm{n}}$

presents the data complexity measures used in this study. The measures used consisted of three feature-based measures (F1, F2, and F3), two linearity measures (L1 and L2), four neighborhood-based measures (N1, N2, N3, and N4), one network measure (T1), and one dimensionality measure (T2). We did not consider class-imbalance-related measures in this study. All the data complexity measures used here were presented in a review by Lorena et al. [37].

In addition, the complexity measures presented here are available for computation in KEEL software [43], which is an open software environment designed for experimentation with supervised classification.

3.2. Associative Classifiers

3.2.1. CHAT-OHM

The Hybrid Associative Classifier with Translation (HACT or CHAT by its Spanish acronym) is a classification algorithm proposed by Santiago-Montero [9], which has shown good results in supervised classification [25]. It has two phases: association (in which the classifier is trained) and recovery (in which the classes of the instances to be classified are obtained). This classifier assumes that the dataset does not contain absences of information and that all attributes are numeric. Furthermore, it assumes that the class values are consecutive integers.

This classification algorithm was refined in 2014, with the development of a “One-Hot” version (CHAT-OHM) [10]. The first modification arises in the training phase and consists of a new way of coding the classes. In this case, the classes are formed by binary vectors of size p (instead of vectors of size m as in HACT), and the components of said vectors have the form ${\mathrm{y}}_{\mathrm{i}}^{\mathsf{\mu }}=\{\begin{array}{cc}1& \mathrm{if}\mathrm{i}=\mathsf{\mu }\\ 0& \mathrm{otherwise}\end{array}$.

The second modification consists of using the majority vote rule in the classification process. To accomplish this, the CHAT-OHM classification phase has three steps:

• Translate the instance to classify $o$, considering the mean of the training instances, as $\hat{\mathrm{o}}\leftarrow \mathrm{o}-\hat{\mathrm{x}}$.
• For each class $\mathrm{k}\in \left\{1,2,\dots ,\mathrm{m}\right\}$, create a masking vector ${\mathrm{mv}}^{\mathrm{k}}$ of size $\mathrm{p}$, of which the components are ${\mathrm{mv}}_{\mathsf{\mu }}^{\mathrm{k}}=\{\begin{array}{cc}1& {\mathrm{if}\hat{\mathrm{x}}}^{\mathsf{\mu }}\in \mathrm{k}\\ 0& \mathrm{otherwise}\end{array}$ and obtain the output pattern ${\mathrm{z}}^{\mathrm{o}}$, as ${\mathrm{z}}^{\mathrm{o}}=\left[\mathsf{\alpha }{{\displaystyle \sum }}_{\mathsf{\mu }=1}^{\mathrm{p}}{\mathrm{y}}^{\mathsf{\mu }}\left({\hat{\mathrm{x}}}^{\mathsf{\mu }}{}^{\mathrm{T}}\right)\right]\hat{\mathrm{o}}$. Then, for each class obtain a counter vector ${\mathrm{cv}}^{\mathrm{k}}={\mathrm{z}}^{\mathrm{o}}\wedge {\mathrm{mv}}^{\mathrm{k}}$
• Compute the components of the output vector (class) for the instance to classify $\hat{\mathrm{o}}$ as ${\mathrm{y}}_{\mathrm{i}}^{\mathrm{o}}=\{\begin{array}{cc}1& \mathrm{if}{{\displaystyle \sum }}_{\mathrm{j}=1}^{\mathrm{p}}{\mathrm{cv}}_{\mathrm{j}}^{\mathrm{i}}=\underset{\mathrm{h}=1..\mathrm{m}}{\max }\left[{{\displaystyle \sum }}_{\mathrm{j}=1}^{\mathrm{p}}{\mathrm{cv}}_{\mathrm{j}}^{\mathrm{h}}\right]\\ 0& \mathrm{otherwise}\end{array}$. Thus, the class of the input instance ${\mathrm{x}}^{\mathsf{\mu }}$ will be returned if and only if the obtained vector has a one in the $\mathsf{\mu }$-th component and zero in the remaining components.

Although HACT does not handle mixed or incomplete data, it has achieved good performance in the financial field [25].

3.2.2. Extended Gamma

López-Yáñez introduced the Gamma associative classifier as a model designed to predict time series [44]. It has been applied to several supervised classification tasks [11,21,24]. However, the same as HACT, it assumes that the dataset is numeric and complete. To address this drawback, the Extended Gamma (EG) classifier was proposed [13].

Let be $\mathrm{X}$ and P the training and test sets, respectively, from a universe of data U, where each instance $\mathrm{x}\in \mathrm{X},\mathrm{p}\in \mathrm{P}$ is described by a set of features $\mathrm{A}=\left\{{\mathrm{A}}_1,{\mathrm{A}}_2,\cdots ,{\mathrm{A}}_{\mathrm{m}}\right\}$; and each attribute ${\mathrm{A}}_{\mathrm{i}}$ has associated a definition domain $\mathrm{dom}\left({\mathrm{A}}_{\mathrm{i}}\right)$, which can be numeric or categoric. If the value of an attribute ${\mathrm{A}}_{\mathrm{i}}$ in an instance $\mathrm{x}$ is missing, it is denoted as ${\mathrm{x}}_{\mathrm{i}}$ = ‘?’. Extended Gamma considers the data to belong to a set of classes $\mathrm{K}=\left\{{\mathrm{K}}_1,\dots ,{\mathrm{K}}_{\mathrm{c}}\right\}$.

EG has the same training phase as its predecessor, and for the classification phase, it replaces the gamma similarity operator with the extended gamma similarity operator, ${\mathsf{\gamma }}_{\mathrm{ext}}$. To determine the class value of an instance, its average similarity with respect to all classes is computed as:

$${\mathrm{c}}_{{\mathrm{k}}_{\mathrm{l}}}=\frac{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{{\displaystyle \sum }}_{\mathrm{j}=1}^{\mathrm{m}}{\mathrm{w}}_{\mathrm{j}}{\ast \mathsf{\gamma }}_{\mathrm{ext}}\left({\mathrm{x}}_{\mathrm{j}}^{\mathrm{i}},{\mathrm{p}}_{\mathrm{j}}\right)}{\mathrm{n}}$$

(1)

where ${\mathrm{x}}_{\mathrm{j}}^{\mathrm{i}}$ is the value of the j-th attribute in the i-th instance belonging to the class ${\mathrm{k}}_{\mathrm{l}}$, ${\mathrm{w}}_{\mathrm{j}}$ is the weight associated with the j-th attribute, and n is the number of instances in the training set having ${\mathrm{k}}_{\mathrm{l}}$ as the class value. If there is a unique maximum in the values of ${\mathrm{c}}_{{\mathrm{k}}_{\mathrm{l}}}$, the process ends. Otherwise, an iterative process is carried out [13].

The definition of the extended gamma similarity operator allows for the handling of mixed continuous and categorical attributes and missing values. When the value of a feature is unknown, it is assumed that the values are similar, and in the case of categorical features, the patterns will only be similar if the values of the features are equal. For the case of continuous attributes, a definition equivalent to the original operator definition is used.

Extended Gamma has been applied effectively to solve problems of a social nature with mixed and incomplete data, such as electoral predictions [14].

3.2.3. Naïve Associative Classifier

Another associative classifier that is able to deal with mixed and incomplete data is the Naïve Associative Classifier (NAC), which has also been applied to solve finance-related problems [15,26]. This classifier has the properties of being transparent and transportable [15].

The training of NAC includes the computation of the standard deviation of each numeric feature. Such values will be used below for classification. To do so, NAC uses a mixed and incomplete similarity operator to compute the similarity between patterns. This operator can deal with both continuous and categorical attribute values and incomplete information in the data. As was the case for Extended Gamma, NAC computes the average similarities to each class $k_l$ for the instance to classify p. If there is a unique maximum, the corresponding class is returned. Otherwise, ties are broken randomly.

NAC has several variants: NACe, which includes a procedure to disambiguate classes [16]; NACe kVotes, and NACe MajoritykVotes [17], which use neighborhoods in the disambiguation process; and NACOD, which allows the handling of data streams [18]. NAC has shown good performance in solving finance-related problems and has been found to be robust to data imbalance [20].

3.2.4. Assisted Classification for Imbalanced Datasets

The Assisted Classification for Imbalanced Datasets (ACID) classifier was also designed [19] for mixed and incomplete classification problems. In the training phase, ACID includes an optional procedure to compute attribute weights via differential evolution. ACID also deals with class disjoints, by performing clustering, and with class imbalance. Additionally, it uses data aggregation to diminish the influence of class overlapping and noisy or mislabeled instances [19].

In this section, we have detailed the most frequently explored data complexity measures in the literature and the associated classifiers. We found no works that analyzed how data complexity, beyond the analysis of data imbalance, affects the performance of the associative classifiers of the Alpha-Beta approach.

4. Results and Discussion

This section describes the datasets used in this study to obtain the complexity measures described in the previous chapter, as well as some of the performance measures obtained from the described classifiers.

4.1. Datasets

Several datasets containing credit data were used. All of these are related to some of the credit process phases: granting, promotion and recovery. Some of the datasets were obtained from the UCI Machine Learning Repository [45]. The datasets that were used in the experiments are:

• Australian credit approval data (Australian, https://archive.ics.uci.edu/ml/datasets/statlog+(australian+credit+approval) accessed on 15 June 2021);
• German credit data (German, https://archive.ics.uci.edu/ml/datasets/statlog+(german+credit+data), accessed on 15 June 2021)
• Japanese credit approval data (Japanese, https://archive.ics.uci.edu/ml/datasets/Credit+Approval, accessed on 15 June 2021);
• Iranian (Iranian), which was introduced in [46] and was shared by Hassan Sabzevari.
• Polish bankruptcy data (Polish_year1 to Polish_year5, https://archive.ics.uci.edu/ml/datasets/Polish+companies+bankruptcy+data, accessed on 15 June 2021);
• The PAKDD 2009 dataset (the PAKDD, https://www.cs.purdue.edu/commugrate/data/credit_card/, accessed on 15 June 2021); and
• Qualitative bankruptcy data (Qualitative, https://archive.ics.uci.edu/ml/datasets/qualitative_bankruptcy, accessed on 15 June 2021).

These datasets (

Table 2. Description of the datasets used.

Dataset	Instances	Attributes		Missing Values	Imbalance Ratio
		Continuous	Categorical
Australian	690	8	6	No	1.25
German	1000	7	13	No	2.33
Iranian	1002	28	0	Yes	19.04
Japanese	690	6	9	Yes	1.21
Polish_year1	7027	64	0	Yes	24.93
Polish_year2	10,173	64	0	Yes	24.43
Polish_year3	10,503	64	0	Yes	20.22
Polish_year4	9792	64	0	Yes	18.01
Polish_year5	5910	64	0	Yes	13.41
Qualitative	250	0	6	No	1.34
The PAKDD	20,000	10	9	Yes	4.12

) included numerical and categorical attributes, and missing data (four mixed and eight with missing values). They also contained class imbalances, due to 8 of the 11 having IR > 1.5. The number of attributes ranged between six and 64 features, whereas the number of instances was between 250 and 20,000. All datasets corresponded to good and default clients; they only contained two classes.

4.2. Performance Measures

In the following, we address the performance measures [47] we used in this study, which have been proven to be robust for imbalanced problems [48]. The measures were based on a two-class confusion matrix, as shown in Figure 1.

Three basic performance measures for two-class problems are precision, recall (also known as sensitivity and the true positive rate), and the true negative rate. Other measures, such as the F-score, geometric mean, and the area under the ROC curve, are based on those measures.

Table 3. Performance measures used.

Measure	Equation
Precision	$\mathrm{P}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$
Recall	$\mathrm{R}=\mathrm{TPR}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$
True Negative Rate	$\mathrm{TNR}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$
F-score	$\mathrm{F}=2\ast \frac{\mathrm{P}\ast \mathrm{R}}{\mathrm{P}+\mathrm{R}}$
Geometric Mean	$\mathrm{GM}=\sqrt{\mathrm{P}\ast \mathrm{R}}$
Area under the ROC curve	$\mathrm{AUC}=\frac{\mathrm{TPR}+\mathrm{TNR}}{2}$

summarizes the performance measures used in this study.

In this study, the F-score, the geometric mean, and the area under the ROC curve were considered in order to evaluate the performance of the classifiers.

4.3. Complexity Measures in the Selected Datasets

Table 4. Complexity measure values for the datasets used.

Dataset	F1	F2	F3	L1	L2	L3	N1	N2	N3	N4	T1	T2
Australian	2.28	0.00	0.03	0.30	0.14	0.14	0.32	0.56	0.20	0.16	1.00	39.43
German	2.45	0.01	0.03	0.28	0.14	0.13	0.32	0.57	0.20	0.17	1.00	36.48
Iranian	0.36	0.64	0.01	0.83	0.24	0.33	0.46	0.85	0.31	0.30	1.00	40.00
Japanese	0.35	0.00	0.21	0.10	0.05	0.50	0.12	0.31	0.07	0.37	1.00	28.57
Qualitative	9.51	0.00	0.78	0.60	0.00	0.00	0.03	0.10	0.01	0.00	0.95	33.33
The PAKDD	0.11	0.05	0.00	0.39	0.20	0.50	0.26	0.15	0.06	0.45	1.00	842.11
Polish_year1	0.03	0.00	0.03	0.08	0.04	0.50	0.09	0.57	0.06	0.45	1.00	87.84
Polish_year2	0.01	0.00	0.02	0.08	0.04	0.50	0.11	0.63	0.08	0.47	1.00	127.16
Polish_year3	0.04	0.00	0.03	0.09	0.05	0.50	0.12	0.64	0.08	0.47	1.00	131.29
Polish_year4	0.06	0.00	0.02	0.11	0.05	0.50	0.13	0.63	0.09	0.45	1.00	122.40
Polish_year5	0.18	0.00	0.04	0.14	0.07	0.50	0.15	0.71	0.10	0.39	1.00	73.88

shows the results of the complexity measures computed for the selected datasets. We used KEEL software [43] to obtain these measures. KEEL software automatically stores all data in numeric arrays; therefore, it includes the imputation of the missing values and the conversion of the categorical values of the datasets into numeric values. In addition, KEEL uses the original definition for F1 and F3 measures [28], and therefore, in Table 4, small values of F1 and F3 denote complex problems (inverse relation). Consequently, all remaining complexity measures are proportional to the values; that is, small values correspond to less complex problems.

As can be seen, for measure F1 the simplest dataset was the qualitative dataset, and the most difficult ones were the Polish datasets. The F2 measure did not offer much information since it assigned 0.0 to all datasets but the Iranian (0.64), German (0.01), and the PAKDD (0.05) datasets. For measures L1 and L2, the most complex dataset was the German. The L3 measure gave a high complexity to the Japanese, the PAKDD, and the Polish datasets, all of which had values of 0.5.

Regarding measures N1, N2m and N3, again, the German database offered the greatest complexity. According to measure N4, the most complex databases were Polish_year2 and Polish_year3. The T1 measure did not offer much information since it assigned complexities equal to 1.00 for all datasets except the qualitative one. Finally, the T2 measure considered that the most complex dataset was the PKDD09 set.

4.4. Performance of the Classifiers

To establish the performance of the associative classifiers under study, we used a five-fold cross-validation procedure because some datasets were imbalanced. As for performance measures, we used the AUC, F-score, and the geometric mean, as described in Section 4.2.

Table 5. Area under the ROC curve of the compared classifiers. The best results are indicated in bold.

Dataset	ACID	CHAT-OHM	EG	NAC
Australian	0.86	0.59	0.84	0.83
Japanese	0.85	0.66	0.82	0.83
German	0.57	0.64	0.68	0.69
Iranian	0.56	0.64	0.68	0.50
Qualitative	0.93	0.95	0.99	0.99
The PAKDD	0.77	0.58	0.61	0.61
Polish_year1	0.58	0.52	0.76	0.50
Polish_year2	0.56	0.50	0.71	0.50
Polish_year3	0.56	0.53	0.74	0.50
Polish_year4	0.54	0.55	0.75	0.53
Polish_year5	0.65	0.62	0.79	0.62

,

Table 6. F-score of the compared classifiers. The best results are indicated in bold.

Dataset	ACID	CHAT-OHM	EG	NAC
Australian	0.86	0.59	0.84	0.83
Japanese	0.86	0.66	0.82	0.84
German	0.58	0.63	0.67	0.67
Iranian	0.60	0.58	0.63	0.49
Qualitative	0.93	0.95	0.99	0.99
The PAKDD	0.78	0.57	0.59	0.59
Polish_year1	0.58	0.51	0.64	0.50
Polish_year2	0.55	0.50	0.62	0.53
Polish_year3	0.57	0.52	0.64	0.51
Polish_year4	0.55	0.53	0.64	0.54
Polish_year5	0.66	0.57	0.68	0.61

and

Table 7. Geometric mean of the compared classifiers. The best results are indicated in bold.

Dataset	ACID	CHAT-OHM	EG	NAC
Australian	0.86	0.59	0.83	0.83
Japanese	0.85	0.65	0.82	0.83
German	0.49	0.63	0.68	0.68
Iranian	0.26	0.63	0.63	0.00
Qualitative	0.93	0.95	0.99	0.99
The PAKDD	0.75	0.57	0.61	0.61
Polish_year1	0.42	0.51	0.76	0.05
Polish_year2	0.39	0.50	0.71	0.04
Polish_year3	0.39	0.53	0.74	0.10
Polish_year4	0.32	0.55	0.75	0.20
Polish_year5	0.57	0.61	0.79	0.55

present the performance results of each classifier according to the measures mentioned above.

According to the AUC results, the best-performing classifier was EG, which performed the best in seven of the 11 datasets. The ACID and NAC classifiers showed good behavior for the Australian, Japanese, German, and qualitative datasets. Nevertheless, their performance for the remaining datasets was poor, except for the PAKDD dataset, in which ACID showed the best performance. On the other hand, CHAT-OHM demonstrated relatively good performance for the qualitative and German datasets. Equivalent results were shown for the F-score measure.

As for geometric mean, EG was the best in eight of the compared datasets. ACID remained the best in the Australian, Japanese, and PAKDD datasets. In addition, for the qualitative datasets, it displayed satisfactory results. However, the performance was poor for the remaining datasets because the classifier was biased toward one class, with geometric mean values ranging from 0.26 to 0.57.

NAC exhibited a behavior similar to ACID (except for the German dataset, in which NAC performed well), showing bias towards the majority class, with values ranging from 0.0 (Iranian) to 0.55 (Polish_year5). CHAT-OHM obtained average results for most of the compared datasets.

In summary, the ACID classifier showed a good performance for most of the databases, and we observed that it exceeded 0.5 for almost all the measures, except for some databases with a greater number of instances. On the other hand, the CHAT-OHM classifier achieved excellent classification performance for the “qualitative-bankruptcy” database. Nevertheless, in the other datasets it failed to obtain performance percentages greater than 0.7, which made it one of the classifiers with the lowest ranking average for this set of databases.

EG was another classifier with good performance (mostly exceeding percentages of 0.6), making it the classifier with the best results in this study. The EG classifier had an almost perfect classification performance for the qualitative database and relatively good performance for the Australian and Japanese databases.

Finally, in terms of the classification results, the NAC classifier, similarly to the EG, exhibited an almost perfect classification performance with the qualitative-bankruptcy database and a very good performance with the Australian and Japanese databases. However, we can also observe that it showed a certain weakness in the case of databases such as the Polish datasets, in which it classified practically an entire class incorrectly. Moreover, its performance was slightly less efficient compared to the EG classifier.

4.5. Correlation Analysis

Another aim of this study was to analyze in-depth and establish the relationship between complexity measures and associative classifiers. This aspect can be investigated in detail by calculating the correlations that arise between the measures of complexity of the datasets and the measures of performance of the associative classifiers. To accomplish this, intensive use of Spearman’s Rho coefficient was made. Spearman’s Rho is a nonparametric measure of dependence in which the mean rank of the observations is calculated [49], and the differences are squared and incorporated into the formula. In other words, a classification is assigned to the observations of each variable, and the dependency relationship between two given variables is studied. The interpretation of Spearman’s coefficient depends on its values. It ranges between −1 and +1, indicating negative or positive associations. Respectively, values close to zero indicate no correlation but not independence. To carry out the correlation tests, two hypotheses were used in all cases:

H0: 

There is no correlation between the measures of data complexity and the performance of the analyzed classifiers.

H1: 

There is a correlation between the measures of data complexity and the performance of the analyzed classifiers.

We set a significance value $\alpha =0.05$, for a 95% confidence level.

Table 8. Correlation analysis of the data complexity measures and the performance measures.

Complexity Measure	Ro	F-Score				Geometric Mean				AUC
		ACID	CHAT-OHM	EG	NAC	ACID	CHAT-OHM	EG	NAC	ACID	CHAT-OHM	EG	NAC
F1	cc	0.864	0.991	0.764	0.782	0.682	0.945	0.482	0.800	0.655	0.936	0.482	0.800
	p	0.001	0.000	0.006	0.004	0.021	0.000	0.133	0.003	0.029	0.000	0.133	0.003
F2	cc	0.282	0.391	−0.018	0.227	0.164	0.364	−0.364	0.291	0.109	0.327	−0.364	0.245
	p	0.401	0.235	0.958	0.502	0.631	0.272	0.272	0.385	0.750	0.326	0.272	0.467
F3	cc	0.418	0.336	0.436	0.055	0.173	0.400	0.564	0.027	0.291	0.473	0.564	0.064
	p	0.201	0.312	0.180	0.873	0.612	0.223	0.071	0.937	0.385	0.142	0.071	0.853
N1	cc	0.236	0.364	0.218	0.409	0.273	0.282	−0.109	0.427	0.173	0.218	−0.109	0.427
	p	0.484	0.272	0.519	0.212	0.417	0.401	0.750	0.190	0.612	0.519	0.750	0.190
N2	cc	−0.564	−0.246	0.036	−0.100	−0.345	−0.200	−0.127	−0.173	−0.445	−0.273	−0.127	−0.082
	p	0.071	0.467	0.915	0.770	0.298	0.555	0.709	0.612	0.170	0.417	0.709	0.811
N3	cc	−0.036	0.246	0.409	0.309	0.055	0.173	0.145	0.245	−0.045	0.109	0.145	0.318
	p	0.915	0.467	0.212	0.355	0.873	0.612	0.670	0.467	0.894	0.750	0.670	0.340
N4	cc	−0.809	−0.909	−0.818	−0.691	−0.645	−0.845	−0.618	−0.709	−0.645	−0.836	−0.618	−0.700
	p	0.003	0.000	0.002	0.019	0.032	0.001	0.043	0.015	0.032	0.001	0.043	0.016
L1	cc	0.682	0.809	0.509	0.800	0.673	0.755	0.136	0.827	0.573	0.691	0.136	0.809
	p	0.021	0.003	0.110	0.003	0.023	0.007	0.689	0.002	0.066	0.019	0.689	0.003
L2	cc	0.236	0.327	0.045	0.327	0.218	0.273	−0.291	0.355	0.127	0.218	−0.291	0.336
	p	0.484	0.326	0.894	0.326	0.519	0.417	0.385	0.285	0.709	0.519	0.385	0.312
L3	cc	−0.727	−0.853	−0.811	−0.863	−0.769	−0.748	−0.642	−0.863	−0.727	−0.705	−0.642	−0.863
	p	0.011	0.001	0.002	0.001	0.006	0.008	0.033	0.001	0.011	0.015	0.033	0.001
T1	cc	−0.670	−0.563	−0.326	−0.251	−0.363	−0.530	−0.381	−0.326	−0.451	−0.600	−0.381	−0.251
	p	0.024	0.071	0.328	0.456	0.273	0.093	0.247	0.328	0.164	0.051	0.247	0.456
T2	cc	−0.573	−0.764	−0.600	−0.355	−0.255	−0.773	−0.436	−0.318	−0.300	−0.809	−0.436	−0.345
	p	0.066	0.006	0.051	0.285	0.450	0.005	0.180	0.340	0.370	0.003	0.180	0.298

shows the correlations obtained. In these results, the text “cc” represents the value of the correlation coefficient, whereas “p” is the probability value of the test. If $p\le \alpha$, we reject the null hypothesis H0 and accept the alternative hypothesis H1; that is, in these scenarios, there is a correlation between the data complexity measures and the performance achieved by the classifiers. These results are shown in bold in the tables. In bold and italics are the results in which the null hypothesis was rejected within a 90% confidence level instead of 95%.

The results obtained for the ACID classifier allowed us to reject the null hypothesis for the measures F1, N2, N4, L1, L3, and T1. The measures F1, N4, and L3 significantly correlated with the performance measures of the F-score, geometric mean, and AUC for the ACID classifier using Spearman’s rho coefficient, with 95% confidence. On the other hand, the L1 measure showed a significant correlation with the F-score performance measure, with the geometric mean with 95% confidence, and with the AUC measure with 90%. The N2 measure showed significant correlations with the F-score performance measure with 90% confidence, and T1 showed a significant correlation with the F-score performance measure, with 95% confidence.

The results obtained for the CHAT-OHM classifier allowed us to reject the null hypothesis for the measurements F1, N4, L1, L3, T1, and T2. Furthermore, the measures F1, N4, L1, L3, and T2 significantly correlated with the performance measures of the F-score, geometric mean, and AUC for the CHAT-OHM classifier using Spearman’s rho coefficient, with 95% confidence.

On the other hand, the T1 measure showed a significant correlation with the performance measures of the F-score, geometric mean, and AUC with 90% confidence.

The results obtained for the EG allowed us to reject the null hypothesis for the measures F1, F3, N4, L3, and T2. The N4 and L3 measures showed a significant correlation with the F-score, geometric mean, and AUC performance measures for the EG classifier using Spearman’s rho coefficient, with 95% confidence. On the other hand, the F1 measure showed a significant correlation with the F-score performance measure with 95% confidence; the F3 measure showed a significant correlation with the geometric mean and AUC measures with 90%, and finally, the T2 measure showed a significant correlation with the F-score measure with 90% confidence

Finally, the results obtained for the NAC allowed us to reject the null hypothesis for the measures F1, N4, L1, and L3. The measures F1, N4, L1, and L3 showed a significant correlation with the performance measures of the F-score, geometric mean, and AUC for the NAC classifier using Spearman’s rho coefficient, with 95% confidence.

It is important to note that F1 showed a stronger correlation than the other feature-based complexity measures. Finding the reasons behind this behavior requires deeper research, including numerical simulations using synthetic data, to be able to bound the factors that may intervene. However, in our opinion, this may be due to the formulation of the feature discriminant ratio. As a result, less complex problems have features that can be used to separate the classes, whereas complex problems are overlapping.

Similarly, N4 and L3 showed a stronger correlation than the other neighborhood-based and network-based measures, respectively. Again, further research is needed to fully understand the reason behind the high correlation between these measures. Our intuition tells us that N4 was again highly correlated because it considers the nonlinearity of the nearest neighbor classifier, focusing on class overlapping, and L3 also uses the notion of nonlinearity for a linear classifier. In our future work, we intend to address this behavior in depth.

The experimental results correctly indicated the existing correlations between complexity measures and the performance of the associative classifiers. The comparison was evaluated and studied, resulting in positive correlation coefficients in regard to different complexity measures for each associative classifier.

The comparative results indicate that our correlation study will depend entirely on the complexity of the problems to be solved and how each associative classifier was created, which leads us to return to the no-free-lunch theorem (NFL), as the ACID, CHAT, and EG classifiers are the ones that presented the highest correlations in terms of performance measures versus complexity measures in the selected datasets. In contrast, the NAC classifier turned out to be the classifier that presented the lowest correlation in its performance against complexity measures.

4.6. Comparison with Other Supervised Classifiers

In this subsection, we present the comparison of the associative-based classifiers with respect to other supervised classifiers in the literature. For this purpose, we again used the area under the ROC curve.

To execute the literature algorithms, with the exception of the nearest neighbor approach, we again used the KEEL environment because it enabled the implementation of naïve Bayes (NB), multilayer perceptron with backpropagation (MLP), and support vector machines (SVM). It is important to mention that all supervised classifiers (associative–based and others) used the same partitions of the five-fold cross-validation procedure, and no algorithm was particularly tuned. No weighing scheme was performed for associative classifiers, and for the literature classifiers, we used the default parameters offered by KEEL. However, those default parameters corresponded to the best values reported in previous studies, and therefore, we can consider the algorithms from the literature to be somewhat optimized. For the nearest neighbor (NN) classifier, we used EPIC software [50,51] with the HEOM dissimilarity function [52]. For the associative-based classifiers, we also used EPIC software.

Table 9. Area under the ROC curve for associative-based and other supervised classifiers.

Dataset	ACID	CHAT-OHM	EG	NAC	NB	NN	MLP	SVM
Australian	0.86	0.59	0.84	0.83	0.85	0.71	0.85	0.85
Japanese	0.85	0.66	0.82	0.83	0.86	0.72	0.86	0.86
German	0.57	0.64	0.68	0.69	0.68	0.52	0.53	0.66
Iranian	0.56	0.64	0.68	0.50	0.61	0.64	0.58	0.50
Polish_year1	0.58	0.52	0.76	0.50	0.61	0.54	0.50	0.50
Polish_year2	0.56	0.50	0.71	0.50	0.55	0.50	0.50	0.50
Polish_year3	0.56	0.53	0.74	0.50	0.61	0.52	0.50	0.50
Polish_year4	0.54	0.55	0.75	0.53	0.63	0.54	0.50	0.50
Polish_ year5	0.65	0.62	0.79	0.62	0.76	0.55	0.53	0.50
Qualitative	0.93	0.95	0.99	0.99	0.98	1.00	0.99	1.00
The PAKDD	0.77	0.58	0.61	0.61	0.51	0.53	0.52	0.50
Times Best	2	0	6	1	1	0	1	2

shows the results regarding the AUC for the compared algorithms. Best results for each dataset are highlighted in bold.

We did not include classifier ensembles such as the well-known random forest [53] in the comparison for two main reasons. First, we considered that, since associative classifiers are base (single) classifiers, it would be unfair to compare the algorithms with respect to classifier committees composed of several supervised classifiers. Second, neither KEEL nor EPIC software includes classifier committees in their selection of supervised classifiers.

The results presented in Table 9 showed that the best-performing algorithm was Extended Gamma (EG), which exhibited the best results in five of the datasets, followed by ACID and SVM (which showed the best results in two datasets). The poor results of CHAT-OHM and NN resulted from the class overlapping of the datasets, and those of NAC resulted from the lack of feature weighting. It was shown in [26] that automatic feature weighting improves the performance of NAC. In addition, the ACID classifier also includes an optional feature weighting procedure [19], which was not applied here.

Regarding the interpretability of the supervised classifiers compared in this study, in our opinion, both Extended Gamma (EG) and Naïve Associative Classifier (NAC) was highly interpretable [15]. ACID was also interpretable [19], although it was not as straightforward as EG and NAC. However, these three classifiers were transparent and transportable. On the other hand, the CHAT-OHM classifier obtained an association matrix, which is not easy to interpret. Regarding the other supervised classifiers compared, the nearest neighbor classifier was interpretable, the naïve Bayes was somewhat interpretable, and MLP and SVM were not easy to interpret due to their training process.

5. Conclusions

In this paper, we used meta-learning, in the context of ASP, to analyze the correlations that arise between the measures of complexity of the datasets and the measures of performance of the associative classifiers under study. We observed correlations between complexity measures and the performance of the associative classifiers.

The comparison was evaluated and studied, resulting in positive correlation coefficients in different complexity measures for each associative classifier. The comparative results indicate that our correlation studies will depend entirely on the complexity of the problems to be solved and the way in which each associative classifier was created, which leads us to return to the no-free-lunch theorem (NFL), and the ACID, CHAT, and EG classifiers are the ones that presented the highest correlations in terms of performance measures versus complexity measures in the selected datasets. In contrast, the NAC classifier turned out to be the classifier that presented the lowest correlation in its performance with complexity measures.

In future work, we intend to explore why some complexity measures showed a stronger correlation than others for the compared associative classifiers.