Metric Learning with Dynamically Generated Pairwise Constraints for Ear Recognition

Abstract

The ear recognition task is known as predicting whether two ear images belong to the same person or not. More recently, most ear recognition methods have started based on deep learning features that can achieve a good accuracy, but it requires more resources in the training phase and suffer from time-consuming computational complexity. On the other hand, descriptor features and metric learning play a vital role and also provide excellent performance in many computer vision applications, such as face recognition and image classification. Therefore, in this paper, we adopt the descriptor features and present a novel metric learning method that is efficient in matching real-time for ear recognition system. This method is formulated as a pairwise constrained optimization problem. In each training cycle, this method selects the nearest similar and dissimilar neighbors of each sample to construct the pairwise constraints and then solves the optimization problem by the iterated Bregman projections. Experiments are conducted on Annotated Web Ears (AWE) database, West Pommeranian University of Technology (WPUT), the University of Science and Technology Beijing II (USTB II), and Mathematical Analysis of Images (AMI) databases.. The results show that the proposed approach can achieve promising recognition rates in ear recognition, and its training process is much more efficient than the other competing metric learning methods.

1. Introduction

As an important person authentication technique, biometric recognition has been widely applied in surveillance applications, forensics, and criminal investigations. Since the biometric traits are unique, universal, and permanent, biometric recognition is more secure and reliable than traditional person authentication approaches.

Among the existing biometric recognition methods, there have been many kinds of biometric traits, e.g., face, fingerprint, palm-print, iris, signature, voice, key-stroke, and gait. Compared with the other traits, the human ear has a stable structure at different ages [1]. Additionally, the ear is insensitive to variations, such as make-up, glasses, and facial expression [2]. The ear image is also easy to acquire with little person awareness and user cooperation [3]. Furthermore, it has been proven that the left and right ears of the same person have some similarities but are not strictly symmetric [4]. Therefore, ear recognition has received increasing research interest.

Recently, for the ear recognition task, deep features [5,6,7] and learning methods have exploited and achieved better recognition rates than traditional handcrafted approaches. One of the most common methods is the metric learning method, which aims to learn the distance between two instances, where the distances between similar instances are shorter than those between dissimilar instances. Metric learning plays a crucial role in machine learning and has been successfully applied into many biometric recognition tasks, such as face recognition. Most of the existing metric learning methods learn the distance metric from the pairwise or triplet constraints [8]. The pairwise constraints make the distances of similar pairs shorter than a given threshold, while the distances of dissimilar pairs longer than this threshold. On the other hand, the triplet constraints make the distance of similar samples shorter than that of dissimilar samples. As the quantity of pairwise constraints is very large, $O(N^2)$ pairs can be constructed from $N$ samples, the existing metric learning methods usually select part of the pairwise constraints for training. Davis et al. [9] proposed to select the pairwise constraints randomly. Wang et al. [10] proposed a strategy to construct the pairs from the training samples. For each training sample, its nearest similar and dissimilar samples are used to construct the similar and dissimilar pairs.

The previous metric learning methods construct the pairwise constraints as a preprocessing step and use the fixed pairwise constraints in training. This strategy, however, suffers from evident drawbacks. As the number of training pairs is fixed, and some pairs are never used in training, the trained models will under-fit the non-used training pairs. To address the aforementioned limitation, we propose a novel method to learn the distance metric from online generated pairwise constraints for an ear recognition system. First, it extracts the local phase quantization (LPQ), histogram of oriented gradient (HOG), and Gabor features of ear images. Then it uses the discriminant correlation analysis (DCA) method to fuse different features and reduce the feature dimension. Finally, it learns the distance metric based on the extracted feature. In this method, we learn the distance metric for several cycles. In each cycle, we construct the pairwise constraints by the trained distance in the last cycle and learn the distance metric based on these pairwise constraints. As the pairwise constraints are updated in each cycle, there are larger amount of training pairs utilized in our proposed method than the previous metric learning methods. We conduct the experiments on several ear image datasets, i.e., AWE, AMI, WPUT, and USTB databases, to evaluate our proposed method. The results show that our proposed method outperforms the state-of-the-art ear recognition approaches and most of the other competing metric learning methods in recognition accuracy and training time.

The rest of this paper is organized as follows: Section 2 introduces the related work about the commonly used ear recognition methods and metric learning algorithms. Section 3 demonstrates the proposed approach based on metric learning. Section 4 presents the experimental results, comparison on accuracy and training time, and offers a discussion of the results. Finally, Section 5 draws the conclusion of this paper.

2. Related Works

In this section, we give a brief review on the related works from two aspects, i.e., 2D ear recognition methods and metric learning algorithms.

2.1. 2D Ear Recognition

The existing works on 2D ear recognition system mainly focus on two aspects, i.e., feature extraction and classification. Three kinds of features, i.e., geometric, appearance-based, and deep neural network features, are mainly used in the ear recognition methods. The geometric features include maximum ear height line (EHL) [11,12], inner and outer helixes [11], tragus [13], etc. Lakshmanan [14] has presented a geometric approach based only the middle portion of the ear that adopted the reference line, normal lines, and angles. Some methods use the combination of these features [15,16] which can facilitate the discriminability of the geometric features. For the appearance-based features, it mainly includes intensity, directional and spatial-temporal information. Many of works have holistically presented features such as Eigen-ear and Eigen-face [2], independent component analysis (ICA) [17], and active shape model (ASM) to detect the outer ear contour [18], 1D and 2D Gabor filters [19,20], and locale features, such as local binary pattern (LBP) [21], HOG [22], scale invariant feature transform (SIFT) [23], and speeded up robust features (SURF) [24]. Recently, Nanni and Lumini [25] adopted the sequential forward floating selection (SFFS) to select the best features from sub-windows in an ear image. Yuan and Mu [26] presented a brief review of ear recognition and proposed a fusion method for ear recognition based on local features information. More recently, deep learning and convolutional neural networks (CNNs) have received more attention and achieved better performance for ear recognition problem; Pedro et al. [27] presented a personal neural network for ear recognition and explained a brief review for deep learning ear recognition. On the other hand, Omara et al. [5,28] exploited the pre-trained VGG-M model for extracting ear features, then fusion CNN features, and proposed a pairwise SVM classifier for ear classification. Emersic et al. [6] provided various CNN ear recognition models with limited training data based on the augmentation process of the training images to overcome the limited amount of labeled training data, and Dodge et al. [7] illustrated various deep CNNs as feature extractors and also adopted a shallow classifier for ear recognition. However, it is known that the deep learning methods have high feature dimensions and need more training computational complexity. Compared to deep features, local descriptors features need low computational complexity in both feature extraction and training because they have low feature dimensions.

For ear classification, we can notice that famous and typical ear recognition methods for 2D ear images focus on the feature extraction process and adopt a simple nearest neighbor, or typical SVM methods for ear recognition problems. Different distances are mainly adopted as the matching criterion, such as the Euclidean distance [15,29], Hamming distance [12,13,30], radial basis function (RBF) [17], back propagation (BP) network [11], and neural networks [31] and, recently, discriminative classifiers, such as approximate nearest neighbor (ANN) [32], support vector machine (SVM) [33,34], and pairwise SVM, have drawn much attention [5]. However, due to the lack of training images of ears and the multiple class properties of ear databases, typical matching processes may not lead to the desired performance whereas the Mahalanobis matrix takes the correlation of various features as the elements of the off-diagonal, and it is scale invariant. Therefore, an appropriate similarity distance metric should be considered in our proposed method. Called a Mahalanobis distance metric, it is learning over prior information to measure the similarity or dissimilarity between different instances. Therefore, we investigate an alternative method for ear recognition problem; metric learning extends the similarity measurement to take the advantage of prior information as a label over standard similarity measures. Metric learning has been proposed to learn Mahalanobis distance metrics for k-nearest neighbor classification and has better interclass generalization performance, which can be applied to handle the multiple classes of ear image databases better.

2.2. Metric Learning

Metric learning plays a crucial role for various applications in computer vision and machine learning applications, such as signature verification [35], data classification [36], and person re-identification [37]. The existing metric learning can be divided into two categories, i.e., pairwise constrained metric learning and triplet constrained metric learning. The pairwise constrained metric learning methods include information-theoretic metric learning (ITML) [9], logistic discriminant metric learning (LDML) [38], distance metric learning with eigenvalue (DML-eig) [39], and sparse metric learning (SML) [40], etc. The ITML method formulates the problem as minimizing the LogDet divergence instance between the learned distance metric and prior distance metric subject to the pairwise constraints [9]. It can be solved by the iterated Bregman projection algorithm. Guilllaumin et al. [38] proposed the LDML method. It defines the probabilities of each sample pair to be similar and dissimilar, respectively, and formulates the problem as maximizing the log-likelihood of all training pairs. Ying et al. proposes the DML-eig method formulated their metric learning as an eigenvalue optimization problem with pairwise constraints [39]. Kostinger et al. [41] proposes the KISSME method to learn the distance metric from the equivalence constraints. Its training is a one-pass process and doesn’t need any iteration.

In addition to the pairwise constrained metric learning methods, some other methods learn the metric based on the triplet constraint. This causes each sample to be close to its similar sample and far from its dissimilar sample. Large margin nearest neighbor (LMNN) learns the distance metric by a convex problem with triplet constraints [42]. It separates the similar and dissimilar neighbors of each sample by a large margin. Shen et al. propose the BoostMetric [43], MetricBoost [44], and FrobMetric [45] methods. They parameterize the distance metric as the linear combination of rank-one matrices and learn the combination parameters based on triplet constraints. The construction of pairwise or triplet constraints is crucial to the performance of metric learning methods. Among the existing metric learning methods, many methods construct the constraints by randomly selecting the pairs or triplets. Wang et al. [10] propose a nearest neighbor strategy to construct the pairs and triplets and proved that this strategy can lead to higher recognition accuracy than the random selection strategy. However, the pairs and triplets are fixed in their training process. In our work, we train the model for many cycles $(m)$, and we dynamically update the pairwise constraints by the nearest neighbor strategy in each cycle as shown in Figure 1, where $T$ is the distance threshold. Thus, our proposed method can incorporate more pairwise constraints in training for learning the distance function.

3. Proposed Method

In this section, we illustrate the proposed method, ear recognition based on the LogDet divergence (ERLD) method, for an ear recognition system. Following a similar preprocessing phase for all approaches, where the geometrical properties of the ears are mostly flat, the proposed method first resizes all images to a fixed size of 100 × 100 pixels and applies histogram equalization to the resized image, since various ear databases such as the AWE database have different ear image sizes and the local descriptors require aligning images before the feature extraction phase. Then, for feature extractions, it uses the local phase quantization (LPQ), histogram of oriented gradient (HOG), and Gabor feature techniques as used in [46] to represent the ear images. Afterward, it adopts a discriminant correlation analysis (DCA) algorithm [47] to fuse and reduce the feature dimension. Finally, it learns a Mahalanobis distance metric by using the proposed dynamically pairwise constraints based on the extracted feature and applies KNN for ear matching and classification. The sketch of the proposed method is illustrated in Figure 2. In details for description LPQ, HOG, and Gabor features are as follows: In the LPQ descriptor, the proposed approach uses a window size of 5 × 5, and the block size is 18 × 18 pixels without block overlap. In the HOG descriptor, the proposed approach adopts a cell of 8 × 8 pixels, and the block size is 16 × 16 pixels with an overlap of eight pixels. Finally, for Gabor features, the number of orientations is fixed to eight with five scales and 40 filters and a down-sampling value of 64. In addition, discriminant correlation analysis (DCA) is a feature-level fusion algorithm and it considers the class structure in feature fusion. It also includes the class associations in the correlation analysis of the feature sets and performs an effective feature fusion based on maximizing the pair-wise correlations across the two feature vectors. DCA also reduces the between-class correlations and restricting the correlations to be within classes (for more details of DCA, see [47]). Therefore, our approach exploits the advantages of the DCA algorithm for the feature fusion and limitation dimension problem.

3.1. Problem Formulation

Given n training samples $\{\left(x_i,x_j\right)|i=1,2,.....n\}$ where $x_i\in R^d$, is the feature of the $i^{th}$ training sample, the squared Mahalanobis distance between $x_{i}and{x}_j$ is defined as Equation (1):

$$D_A\left(x_i,x_j\right)={\left(x_i-x_j\right)}^{T}A\left(x_i-x_j\right)$$

(1)

where $A$ is the distance metric which is a positive semi-definite matrix.

Denote by $S=\{\left(x_i,x_j\right)|x_{i}and{x}_j$ are from the same person} the set of similar pairs and $D=\{\left(x_i,x_j\right)|x_{i}and{x}_j$ are from different people} the set of dissimilar pairs. We hope the pairs satisfy the following constraints shown in Equation (2):

$$D_A=\{\begin{array}{ll}\le U& \forall \left(x_i-x_j\right)\in S\\ \ge L& \forall \left(x_i-x_j\right)\in D\end{array}$$

(2)

where U and L are the upper and lower thresholds.

Similar with ITML [9], we formulate the objective function as that of minimizing the LogDet divergence function to make the learned distance metric $A$ to be close to the given prior distance metric $P$. Thus, the problem is formulated as Equation (3):

$$\begin{array}{cc}\hfill \underset{A,\xi }{{\displaystyle min}}& B_{ld}(A,P)+\gamma B_{ld}({\xi }_0,\xi )\hfill \\ \hfill s.t& D_A(x_i,x_j)\le {\xi }_{i,j},\forall (x_i,x_j)\in S\hfill \\ & D_A(x_i,x_j)\ge {\xi }_{i,j},\forall (x_i,x_j)\in D\hfill \\ & A\ge 0,{\xi }_{i,j}\ge 0\hfill \end{array}$$

(3)

where $B_{ld}(A,P)=tr(A,P^{-1})-\log \det (A,P^{-1})-d$ is the LogDet divergence function between two matrices $A$ and $P$ [9,48]. $tr(.)$ is the trace of the matrix, and $d$ is the dimension of training samples. Both the proposed method and ITML [9] are fast and scalable because they do not require any eigenvalue computations or semi-definite programming. However, the constraints of the ITML model are fixed and strict, which may require more iteration to achieve convergence. Therefore, the ITML method suffers from suboptimal performance since it may not learn the most appropriate from the fixed pairwise constraints which are used to improve the accuracy of a k-nearest neighbor classifier. On the other hand, the proposed method tries to solve the previous issues by using $m$ cycles to select the most appropriate constraints for learning the distance faster without requiring more learning iterations. In other words, the proposed method can achieve better performance and improve the accuracy of the KNN classifier, and is faster than the ITML method based on updating the pairwise constraints through several cycles to learn the distance metric, as shown in Figure 1.

3.2. Training Algorithm

3.2.1. Construction of the Pairwise Constraints

For constructing the pairwise constraints, most of previous metric learning methods construct the pairwise constraints as a preprocessing step and use the fixed pairwise constraints in training, such as ITML [9]. To address the previous issues, the proposed metric learning algorithm adopts dynamically generated pairwise constraints based on a cyclic projection method to seek an optimized kernel matrix under linear inequality constraints. In detail, suppose that there are $n$ training samples $\{\left(x_i,x_j\right)|i=1,2,.....n\}$, where $x_i\in R^d$. The proposed method initializes the similar and dissimilar pair sets in the first cycle, and then updates the pair sets in each of the rest cycles. Denoted by $S_k,D_k$ and $A_k$ the sets of similar and dissimilar pairs and the learned distance metric in the $k^{th}$ cycle, respectively. In the first cycle, we initialize $P$ as the identity matrix, compute the Euclidean distance of every two training samples, and then find the nearest similar and dissimilar neighbors of each sample to initialize $S_{0}and{D}_0$, respectively. In the $k^{th}$ cycle, we use $A_{k-1}$ as the prior distance metric to compute the Mahalanobis distances of every two training samples. Based on the distances, we find the nearest similar and dissimilar neighbors of each sample to sets $S_{k}and{D}_k$, respectively.

3.2.2. Optimization of the Distance Metric

In each training cycle, we set the prior distance metric $P$ as $A_{k-1}$, and solve Equation (3) with the training pairs $S_{k}and{D}_k$. Following [9], we initialize $A_k$ as $A_{k-1}$, and learn the distance metric by repeatedly compute the Bregman projections as follows:

$$A_k=A_k+{\beta }_{ij}{A}_k(x_i-x_j){(x_i-x_j)}^T{A}_k$$

(4)

where $x_i-x_j$ is a training pair in $S_{k}or{D}_k$, and ${\beta }_{ij}$ is the Lagrange multiplier corresponding to pair $x_i-x_j$.

3.2.3. The proposed metric learning algorithm

As described in Section 3.2.1 and Section 3.2.2, we can summarize the algorithm of the proposed method as Algorithm 1. Since $A,S$ and $D$ refer to the distance matrix, the similar and dissimilar pairs, respectively. For training the proposed model, the model trains for many cycles $(m)$, and we dynamically update the pairwise constraints by the nearest neighbor strategy in each cycle to incorporate more pairwise constraints in training for learning the distance function and to ensure the Mahalanobis distance is optimized. This means, in each cycle, the proposed metric learning algorithm generates a new pairwise constraint to optimize the Mahalanobis metric distance step by step from $A_k$ to $A_{k+1}$, and $A_k$ is the iterative Mahalanobis metric at cycle $k$. Therefore, the proposed algorithm computes the convergence factor between the prediction and target (desired Mahalanobis) metrics, i.e., $D_{A_{k+1}}(x_i,x_j)-D_{A_k}(x_i,x_j)\prec \eta$, where $\eta$ is a threshold. If the convergence factor achieves the required threshold, the proposed algorithm then adopts the present Mahalanobis distance as a target metric for training the model. On other hand, if the convergence factor does not achieve the required threshold, then the proposed algorithm updates the pairwise constraints and Mahalanobis metric as shown in Figure 1 and Algorithm 1 until it achieves the convergence factor. Where ${\beta }_{ij}$ is the Lagrange multiplier corresponding to pair $(x_i,x_j),\gamma$ is a regularization parameter that balances the regularization function $B_{ld}(A,P)$ and $B_{ld}({\xi }_0,\xi )$, and $\lambda ,{\xi }_{i,j}$ are the initial and desired distances, respectively, that are used to classify the instances as similar and dissimilar.

Algorithm 1. The training algorithm of the proposed method.

Input: Training set $\{\left(x_i,x_j\right)|i=1,2,.....n\}$, cycle number $(m)$ Output: Learned distance metric $A$. Step 1. Initialize the prior distance metric $A_0$ as the identity matrix. Step 2. For $k=1tom$      2.1. $S_k\leftarrow \phi ,D_k\leftarrow \phi$      2.2. Compute the distances of every two training samples with the distance metric $A_{k-1}$. Step 3. For each training sample $x_i$:      3.1. Find the nearest similar and dissimilar neighbors of $x_i$ as $x_p$ and $x_q$.      3.2. $S_k\leftarrow S_k\cup (x_i,x_p),D_k\leftarrow D_k\cup (x_i,x_q)$      3.3. End for (Step 3) Step 4. $A_k\leftarrow A_{k-1}$      4.1. Repeat      4.2. Pick a pair $(x_i,x_j)$ in $S_k\cup D_k$      4.3. $P\leftarrow {(x_i-x_j)}^{T}A(x_i-x_j)$.      4.4. $\delta \leftarrow 1,if(x_i,x_j)\in S_{k}and\delta \leftarrow -1,if(x_i,x_j)\in D_k$.      4.5. $\alpha \leftarrow min({\lambda }_{ij},\delta /2(1/P-\gamma /{\xi }_{ij}\left)\right)$      4.6. $\beta \leftarrow \delta \alpha /(1-\delta \alpha P)$      4.7. ${\xi }_{ij}\leftarrow \gamma {\xi }_{ij}/(\gamma +\delta \alpha {\xi }_{ij})$      4.8. ${\lambda }_{ij}\leftarrow {\lambda }_{ij}-\alpha$      4.9. $A_k\leftarrow A_k+\beta A_k(x_i-x_j){(x_i-x_j)}^T{A}_k$   Until convergence Step 5. $k\leftarrow k+1$      5.1. End for Step 6. Return $A$

4. Datasets and Experimental Results

To evaluate our proposed method, the experiments are conducted on four public and complicated ear databases, i.e., Annotated Web Ears (AWE) database [46], West Pommeranian University of Technology (WPUT) [49], the University of Science and Technology Beijing II (USTB II) [11] and Mathematical Analysis of Images (AMI) [50] databases.

4.1. Datasets

First, the AWE database [46] includes 100 subjects and it is collected from the internet with high degree of variability and various illuminations, and with different scales and rotations. All AWE images were manually inspected, and each person has 10 images that were selected for inclusion in the dataset. Second, the WPUT database [49] was introduced in 2010 and consists of 3345 images of 475 persons with 1388 duplicates, among which each person has 4–10 images; we are used only four images for each subject. The WPUT images are taken from men and women, and under different indoor lightning conditions and head rotation angles ranging from approximately 90° for profile to 75°, and occlusions include earrings, hats, tattoos, etc., for more difficulty. Third, the USTB II database [11] contains 308 images of 77 persons, which are taken under different illumination and camera views. Each person has four images. The first image is the frontal ear image under standard illumination, the second and the third images are taken with +30 and −30 rotations respectively, and the fourth image is taken under weak illumination. Finally, the AMI database [50] has 700 images from 100 persons, all of subjects in the age range of 19–65 years. AMI ear images were collected from students, teachers, and staff at Universidad de Las Palmas de Gran Canaria (ULPGC), Las Palmas, Spain, and taken in an indoor environment. Each person has seven images; five of them were right side profile (right ear) and the sixth image of right profile was taken but with a different camera, and the last image was taken from a left side profile (left ear). Figure 3 shows the original ear images for one subject from each database.

4.2. Experiment Settings

The experiments are conducted on four complicated public ear databases, AWE, WPUT, USTB2, and AMI databases. For comprising and evaluating the proposed metric learning method with state-of-the-art ear recognition methods and metric learning algorithms, we partition each of the AWE, USTB2, AMI, and WPUT databases into two sets, one for training and the other for testing. The training dataset randomly contains 60% of the ear images from each database and the remainder, i.e., 40%, from each database for testing. We also use ten pairs in each cycle, which the number of cycles is 10 for the learning distance function of the proposed method. All experiments are performed with MATLAB R2016a software and on a computer with 3.4 GHz Intel Xeon E3-1230 CPU V2 and 32 GB for RAM.

4.3. Comparison of Accuracy with Existing Ear Recognition Systems

In this subsection, we compare the proposed method with the state-of-the-art ear recognition methods, the Euclidean distance, and the traditional Mahalanobis distance with the inverse covariance matrix as the distance metric on the recognition accuracy, as shown in

Table 1. Comparison on recognition rate (%) for the AWE, USTB2, AMI, and WPUT databases.

Methods		AWE	USTB2	AMI	WPUT
Emersic et al. [46]		49.6 ± 6.80	90.9 ± 6.50	--	--
Raghavendra et al. [51]		--	--	86.36	--
Benzaoui et al. [34]		--	90.9 ± 6.50	47.72	--
Earnest et al. [52]		90.60	--	--	EER = 9.40
Samuel et al. [7]		85.00	--	--	--
Ours	Mahalanobis distance	72.22 ± 1.93	91.71 ± 5.11	95.50 ± 1.19	57.00 ± 1.83
	Metric learning	98.13 ± 0.72	95.85 ± 3.90	96.65 ± 1.36	93.42 ± 2.06

. For our experiments, all experiments are performed 10 times for illustrating the robustness and efficiency of the proposed method and the results illustrate the mean and standard deviation of the system performance for each database. Table 1 shows the comparable results of our proposed methods to existing ear recognition systems from the literature. The results show the superior of our proposed method than other methods by large margin, due to this large margin on the results of the proposed method that is based on learning a Mahalanobis distance for ear recognition system, we firstly evaluate the ear recognition system based on different metric learning state-of-the-art methods as will illustrate in the following subsection.

4.4. Comparison of Metric Learning Methods on Accuracy and Training Time

To illustrate our proposed method’s ability to learn a distance function, we compare it to some existing metric learning methods in the literature. To the best of our knowledge, we firstly exploit metric learning methods to evaluate the accuracy on the ear recognition system; different databases with the same setting and protocol are used in this evaluation. Moreover, we compare the proposed metric learning method with the state-of-the-art metric learning methods, i.e., ITML [9], LMNN [42], LDML [38], and LDMLT [53] on recognition rate and training time as shown in

Table 2. The average of recognition rate (%) and training time for different metric learning methods in the AWE database.

Methods	Time	K = 1	K = 2	K = 3	K = 4	K = 5
ITML [9]	59.68 s	83.38 ± 5.62	72.23 ± 7.22	70.68 ± 8.35	69.70 ± 7.96	69.5 ± 8.25
LMNN [42]	148.75 s	99.25 ± 0.72	99.25 ± 0.72	99.23 ± 0.82	99.23 ± 0.83	98.88 ± 1.11
LDML [38]	18.29 s	99.10 ± 0.01	99.10 ± 0.01	99.10 ± 0.01	99.05 ± 0.01	99.10 ± 0.01
LDMLT [53]	9.68 s	98.95 ± 0.86	98.95 ± 86	99.18 ± 0.8	99.35 ± 0.86	99.35 ± 0.85
Ours	2.13 s	98.13 ± 0.72	98.13 ± 0.72	98.70 ± 0.39	98.75 ± 0.50	98.80 ± 0.61

,

Table 3. The average of recognition rate (%) and training time for different metric learning methods in the USTB2 database.

Methods	Time	K = 1	K = 2	K = 3	K = 4	K = 5
ITML [9]	17.67 s	93.33 ± 4.51	80.33 ± 3.65	75.37 ± 3.97	73.17 ± 4.28	69,51 ± 4.97
LMNN [42]	19.70 s	94.07 ± 3.72	94.07 ± 3.72	78.78 ± 3.22	78.78 ± 3.22	57.32 ± 8.78
LDML [38]	13.25 s	93.09 ± 3.59	93.09 ± 3.59	92.44 ± 3.49	91.87 ± 3.68	91.71 ± 3.73
LDMLT [53]	15.64 s	94.96 ± 2.7	94.96 ± 2.7	94.88 ± 2.66	94.88 ± 2.66	94.63 ± 3.05
Ours	11.77 s	95.85 ± 3.90	95.85 ± 3.90	95.77 ± 3.82	95.37 ± 4.08	95.37 ± 4.35

,

Table 4. The average of recognition rate (%) and training time for different metric learning methods in the WPUT database.

Methods	Time	K = 1	K = 2	K = 3	K = 4	K = 5
ITML [9]	140.12 s	91.80 ± 2.20	91.80 ± 2.20	91.95 ± 2.21	91.85 ± 2.16	91.72 ± 2.12
LMNN [42]	3346.9 s	93.75 ± 1.96	93.75 ± 1.96	88.64 ± 2.15	88.64 ± 2.15	78.32 ± 3.19
LDML [38]	95.29 s	93.07 ± 0.02	93.07 ± 0.02	88.16 ± 0.02	83.89 ± 0.03	79.91 ± 0.05
LDMLT [53]	173.36 s	92.45 ± 2.17	92.45 ± 2.17	92.37 ± 2.18	92.2 ± 2.15	91.92 ± 2.14
Ours	12.15 s	93.42 ± 2.06	93.42 ± 2.07	93.30 ± 2.04	92.88 ± 2.03	92.36 ± 2.19

and

Table 5. The average of recognition rate (%) and training time for different metric learning methods in the AMI database.

Methods	Time	K = 1	K = 2	K = 3	K = 4	K = 5
ITML [9]	54.91 s	97.64 ± 0.66	96.29 ± 1.31	95.89 ± 1.69	96.04 ± 1.72	96.19 ± 1.76
LMNN [42]	58.15 s	99.12 ± 1.03	99.12 ± 1.03	98.07 ± 1.47	98.07 ± 1.47	95.21 ± 2.22
LDML [38]	5.43 s	98.79 ± 1.09	98.79 ± 1.09	98.32 ± 1.17	97.86 ± 1.31	97.32 ± 1.75
LDMLT [53]	1.21 s	98.04 ± 1.61	98.04 ± 1.61	98.32 ± 1.21	98.32 ± 1.14	98.39 ± 1.07
Ours	0.74 s	96.96 ± 1.36	96.96 ± 1.36	97.12 ± 1.43	97.61 ± 1.53	97.76 ± 1.38

for the AWE, USTB2, WPUT, and AMI databases, respectively. We follow the same previous protocol and repeat our experiments 10 times and report the mean recognition accuracy and standard deviation by using the K-nearest neighbor (K-NN) classifier with K = 1, 2, …5. It may be observed that our proposed method can achieve better performances than the other competing methods in the WPUT and USTB II databases in Table 3 and Table 4, even if the WPUT and USTB II databases are small-scale, meaning that the metric learning methods based on triplet constraints cannot achieve good performance with a small ear training dataset. Moreover, the training results time on four databases show the superior of the proposed method. On the other hand, when the ear training dataset increased, such as in the AWE and AMI databases, the recognition rates of our proposed method are slightly lower than those of LMNN and LDML. That is because LDMLT and LMNN are based on triplet constraints which provide more information of the data, but it requires higher computational complexity. While our proposed method, which is based on dynamically pairwise constraints, can provide good information and can apply to real-time application as shown on the training time in Table 2, Table 3, Table 4 and Table 5 for the AWE, USTB2, WPUT, and AMI databases, respectively.

4.5. Discussion

The metric learning methods in the literature include two types of constraints, pairwise and triplet constraints. For pairwise constraints, the metric learning methods learn the distance metric by selecting a random part of the pairwise constraints while the quantity of all constraints is very large: $O(N^2)$ pair for $N$ samples or instances. In this regard, ITML [9] and LDML [38] methods randomly select the pairwise constraints for each iteration. Then, it uses the fixed constraints to update the metric learning matrix, while this fixed constraint may not be provided a critical metric learning matrix for solving the problem, and not achieve the desired convergence that leads to training more iterations for adopting a good result. On other hand, the metric learning methods make each instance $x_i$ to be close to its similar instance $x_j$ and far from dissimilar instance $x_k$ for building triplet constraints $(x_i,x_j,x_k)$. These methods, such as LMNN [42] and LDMLT [53], use a number of cycles to update the triplet constraints every cycle, and to achieve the critical learning matrix that is play a vital role in classification problem. However, the metric learning methods LMNN and LDMLT have complicated triplet constraints and need more computational complexity for training. Therefore, the proposed metric learning method exploits the advantages of pairwise constraints besides updating it by the nearest neighbor strategy through several cycles to achieve a critical metric learning matrix and obtain promising results with a short time as shown in previous subsections. Moreover, we can notice that the training time of our proposed method is much shorter than the other metric learning methods. We analyzed the computational complexity of our proposed method in training. From Equation (3), the computational complexity of each iteration is $O(d^2)$. However, as there are m cycles and $O(n)$ pairs in each cycle, the computational complexity of our proposed method is $O(mn{d}^2)$.

5. Conclusions

Recently, most researchers have gained more attention working on unconstrained ear recognition, which is a very challenging problem. Most previous ear recognition methods are based on the feature extraction phase and used traditional classifiers for ear matching. More recently, metric learning methods have achieved a promising performance on various applications, such as face recognition, signature verification, data classification, and person re-identification. Therefore, in this paper, we firstly propose a novel pairwise constrained metric learning method for unconstrained ear recognition. In this method, we update the pairwise constraint using the nearest neighbor strategy in each training cycle and learn the distance metric via minimizing the LogDet divergence of the learned metric and prior metric. This method can incorporate more pairwise constraints to learn the distance metric, which can improve the recognition performance with efficient training time. The experiments are conducted on challenging and complicated ear databases, such as AWE, AMI, WPUT, and USTB II databases. The results show that our proposed method can achieve favorable recognition accuracy compared with the state-of-the-art ear recognition methods, and its training time is much faster than the other competing metric learning methods. In the future, we will investigate applying this strategy to dynamically build the triplet constraint and propose more metric learning methods for ear recognition system. In addition, we will try to propose other methods for identical twins based on the fusion of handcrafted and deep features by using ear images. Moreover, we will investigate a simple convolutional neural network for ear recognition with more occlusion.