Symmetric Face Normalization

Abstract

Image registration is an important process in image processing which is used to improve the performance of computer vision related tasks. In this paper, a novel self-registration method, namely symmetric face normalization (SFN) algorithm, is proposed. There are three contributions in this paper. Firstly, a self-normalization algorithm for face images is proposed, which normalizes a face image to be reflection symmetric horizontally. It has the advantage that no face model needs to be built, which is always severely time-consuming. Moreover, it can be considered as a pre-processing procedure which greatly decreases the parameters needed to be adjusted. Secondly, an iterative algorithm is designed to solve the self-normalization algorithm. Finally, SFN is applied to the between-image alignment problem, which results in the symmetric face alignment (SFA) algorithm. Experiments performed on face databases show that the accuracy of SFN is higher than 0.95 when the translation on the x-axis is lower than 15 pixels, or the rotation angle is lower than 18°. Moreover, the proposed SFA outperforms the state-of-the-art between-image alignment algorithm in efficiency (about four times) without loss of accuracy.

1. Introduction

The image registration technique has been considered as an important research field in computer vision. It focuses on the problem of aligning several images of the same object. This technique has played an important role in many applications such as medical imaging, biometrics, image mosaicking, video stabilization, etc. A lot of algorithms and techniques have been proposed for these different applications. In this paper, we only pay attention to the face alignment problem.

There have been a lot of face registration techniques, which can be roughly classified into two categories, i.e., model-based algorithms and model-free algorithms. In model based registration methods, a face model is always built based on a large number of face images. For example, active appearance model (AAM) is one of the most powerful model-based algorithms proposed by Cootes et al. [1,2]. It decouples and models the shape and the texture of the deformable object, and is able to generate a variety of instances photorealistically. Because of the outstanding modeling ability, AAM has been widely applied in fields of human eyes modeling, object tracking, facial recognition, gait analysis, and medical image segmentation and analysis [3,4,5,6,7]. Then, shape regression based methods were proposed [8,9,10]. These methods begin with an initial estimate of the landmark locations which are then refined in an iterative manner. Subsequently, a deep learning framework was introduced to greatly improve the face alignment performance [11,12,13,14]. Regardless of the satisfied performance, model based algorithms suffer from the serious dependence of a large amount of training data.

On the other hand, model free algorithms consider the image registration as a between-image matching problem that does not need a great number of training samples. For example, to address the transformation error between gallery and probe image pairs, a deformable sparse representation model (SRC) is proposed [15]. The key idea of their work is that an aligning procedure is introduced to avoid the error caused by misalignment. This method is effective in cases that accuracy of face detection is not guaranteed; however, the additional alignment also introduces great computational complexity. Then, an illumination subspace [16,17] is also proposed to be learned independently and able to extend SRC with help of transfer learning. However, the most serious difficulty of model free algorithms is the heavy computational complexity.

The common problem in both model-based and model-free algorithms is that neither of them makes use of the symmetrical character of images. There exists redundant information in symmetric images such as reflection, rotation, translation, etc. In fact, there is a long history of the research of symmetry. The earliest research work on symmetrical image may date back to 1993 in which Masuda et al. [18] proposed a method to extract the properties of rotational symmetry and reflectional symmetry in 2D images. Then, Liu et al. [19] developed a method to detect dihedral and frieze symmetry as well as asymmetric sub-patterns in planar. Then, the outstanding work conducted by Loy and Eklundh [20] introduced scale-invariant feature transform (SIFT) features along with a voting scheme to cluster neighboring symmetrical features. Lee and Liu [21] generalized reflection symmetry detection to a glide reflection symmetry detection problem by estimating a set of contiguous local straight reflection axes. Kondra et al. [22] presented a multi-scale strategy to detect symmetry. First, SIFT correlation measures are computed at different directions of the same scale. Then, these SIFT correlation measures are used to select symmetrical regions. Similarly, symmetrical regions of other scales are also found by this procedure. Patraucean et al. [23] conducted a detection-and-validation procedure to find symmetrical regions. Candidate patches are first detected using a Hough-like voting scheme. Then, these regions are validated using a principled statistical procedure inspired from a contrario theory, which minimizes the number of false positives. Michaelsen et al. [24] performed symmetry detection combinatorial gestalt algebra technique as well as the SIFT descriptors. Cicconet et al. [25] introduced wavelet filtering to conduct pairwise voting. Cai et al. [26] attempted to clutter symmetrical features adaptively. Wang et al. [27] matched symmetry areas by establishing the correspondence of locally affine invariant edge-based features rather than SIFT. It outperformed other methods because it is insensitive to illumination variations and suitable for textureless objects. Cicconet et al. [28] proposed to detect symmetry by using a pairwise convolutional approach, which was based on the products of complex-valued wavelet convolutions. Funk and Liu [29] applied symmetry detection to reCAPTCHA solutions. However, most of these works are based on feature-based methods, whose accuracy depends on the localization of features.

To boost the efficiency of the model free registration methods, this paper proposes a symmetric face normalization (SFN) algorithm. There are three contributions in this paper. Firstly, a self-normalization algorithm for face images is proposed, which normalizes a face image to be reflection symmetric horizontally. It has an advantage that no face model needs to be built, which is always severely time-consuming. Moreover, it can be considered as a pre-processing procedure which greatly decreases the parameters needed to be adjusted. Secondly, an iterative algorithm is designed to solve the self-normalization algorithm. Finally, SFN is also applied to the between-image matching problem, which results in the symmetric face alignment (SFA) algorithm.

Related Work

Image registration has a long history, which can date back to the 1972 [30]. However, the most representative algorithm occured in 1981, namely the Lucas–Kanade algorithm [31,32]. It models the image registration task as an optimization problem that can be solved using the gradient descent algorithm efficiently. However, the same object embeded in two images is always disturbed by noise, which may affect the accuracy of registration. Therefore, researchers [33] proposed the DSRC algorithm where registration problem was modeled as

$$\underset{p,e}{min}\parallel E\parallel s.t.T\left(W(x,p)\right)=I\left(x\right)+E\left(x\right),$$

(1)

where $\parallel \parallel$ denotes $L2$ norm ($L1$ norm is also allowed [33]), I and T are two images to be aligned, W is the warping function with p as the parameter vector, and E denotes the noise image. DSRC performs registration well; however, it does not consider the symmetry information in face images.

2. Symmetric Face Normalization Algorithm

The objective of this paper is to find the appropriate parameters for similarity transformation so that the object becomes symmetry with respective to the y-axis.

2.1. Symmetric Face Normalization

Suppose $W(\mathbf{x},p)$ is the image warping function with p as parameters. For simplification, the discussion of this paper is limited to the similarity transformation. However, other transformations are also applicable, such as scale transformation and shearing transformation. Given an image I and its reflectional symmetric image $R_I\left(\mathbf{x}\right)$, this paper models the symmetric face normalization problem as following optimization problem

$$minE=\sum _{\mathbf{x}\in I}{(I\left(W(\mathbf{x},p)\right)-R_I\left(W(\mathbf{x},p)\right))}^2,$$

(2)

where

$$\begin{array}{ccc}\hfill W(\mathbf{x},p)& =& \left(\begin{array}{ccc}a& -b& x_0\\ b& a& 0\end{array}\right)\left(\begin{array}{c}x\\ y\\ 1\end{array}\right),\hfill \end{array}$$

(3)

$$\begin{array}{ccc}\hfill a^2+b^2& =& 1,\hfill \end{array}$$

(4)

where $p=[a,b,x_0]$ is the parameter vector of the warping function, which is constrained to be similarity transformation by Equation (4). It should be mentioned that, since we only consider the symmetry with respect to the y-axis, thus the translation on the y-axis is not involved. In addition, Equation (4) indicates that there is no scale variation in the transformation W.

Subsequently, it is found that the reflectional symmetric operator $R_I$ can be equivalent to the composition of two operators that

$$\begin{array}{ccc}\hfill R_I\left(W(\mathbf{x},p)\right)& =& I(S\circ W(\mathbf{x},p\left)\right),\hfill \end{array}$$

(5)

where S is the symmetric warping operator which can be formulated as

$$\begin{array}{ccc}\hfill S\left(x\right)& =& \left(\begin{array}{ccc}-1& 0& w+1\\ 0& 1& 0\end{array}\right)\left(\begin{array}{c}x\\ y\\ 1\end{array}\right)\hfill \end{array}$$

(6)

$$\begin{array}{cc}=& \left(\begin{array}{c}w-x+1\\ y\end{array}\right),\hfill \end{array}$$

(7)

where w is the width of the image I. As a result, objective (2) is transformed to

$$E=\sum _{\mathbf{x}\in I}{(I\left(W(\mathbf{x},p)\right)-I(S\circ W(\mathbf{x},p)))}^2,$$

(8)

where the symmetric transformation $S\circ W(\mathbf{x},p)$ can be obtained as

$$\begin{array}{ccc}\hfill S\circ W(\mathbf{x},p)& =& \left(\begin{array}{ccc}-1& 0& w+1\\ 0& 1& 0\end{array}\right)\left(\begin{array}{c}x+by+c\\ -bx+y\end{array}\right)\hfill \\ & =& \left(\begin{array}{c}-x-by+w+1-c\\ -bx+y\end{array}\right).\hfill \end{array}$$

(9)

Subsequently, the optimal parameter p can be obtained with the help of the Lucas–Kanade algorithm

$$\begin{array}{ccc}minE\hfill & =\hfill & {\sum }_x{(I\left(W(\mathbf{x},p+▵p)\right)-I\left(S\left(W(\mathbf{x},p+▵p)\right)\right)}^2\hfill \\ & =\hfill & {\sum }_x{(I\left(W(\mathbf{x},p)\right)-I\left(S\left(W(\mathbf{x},p)\right)\right)+\nabla I·\frac{\partial W}{\partial p}·▵p-\nabla I·\frac{\partial S\circ W}{\partial p}▵p)}^2\hfill \\ & =\hfill & {\sum }_x{(▵I{|}_p+\nabla I(\frac{\partial W}{\partial p}-\frac{\partial S\circ W}{\partial p})▵p)}^2\hfill \\ & =\hfill & {\sum }_x{(▵I{|}_p+J▵p)}^2,\hfill \end{array}$$

(10)

where

$$\begin{array}{ccc}\hfill \frac{\partial W}{\partial p}& =& \left(\begin{array}{cc}y& 1\\ -x& 0\end{array}\right),\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill \frac{\partial S\circ W}{\partial p}& =& \left(\begin{array}{cc}-y& -1\\ -x& 0\end{array}\right).\hfill \end{array}$$

(12)

The Jacobi matrix J and the Hessian matrix H can be calculated respectively by

$$\begin{array}{ccc}\hfill J& =& \nabla I(\frac{\partial W}{\partial p}-\frac{\partial S\circ W}{\partial p}),\hfill \end{array}$$

(13)

$$\begin{array}{ccc}\hfill H& =& J^{T}J\hfill \end{array}$$

(14)

$$\begin{array}{cc}=& {(y,1)}^T\nabla I_x^T\nabla I_x(y,1).\hfill \end{array}$$

(15)

Finally, by setting the derivative $\frac{\partial E}{\partial ▵p}=0$, the update of parameters p can be obtained as

$$\begin{array}{ccc}\hfill J^{T}J▵p& =& J^T{▵I|}_p,\hfill \end{array}$$

(16)

$$\begin{array}{ccc}\hfill ▵p& =& H^{-1}{J}^T{▵I|}_p.\hfill \end{array}$$

(17)

Thus, the parameter p can be updated by

$$\begin{array}{c}\hfill p=p+▵p.\end{array}$$

(18)

The pseudocode of the symmetric face normalization is shown in Algorithm 1.

Algorithm 1 Symmetric Face Normalization Algorithm

1: Initialize $p=[1,0,0]$ 2: repeat 3:  Warp the image I with W and parameters p to obtain $I\left(W\right(\mathbf{x},p\left)\right)$ 4:  Compute $▵I=I\left(W\right(\mathbf{x},p\left)\right)-I\left(S\right(W(\mathbf{x},p)\left)\right)$ 5:  Compute image gradient $\nabla I$ 6:  Compute $J=\nabla I(\frac{\partial W}{\partial p}-\frac{\partial S\circ W}{\partial p})$ 7:  Compute $H={(y,1)}^T\nabla I_x^T\nabla I_x(y,1)$ 8:  Compute $▵p=H^{-1}{J}^T{▵I|}_p$ 9:  Update $p=p+▵p$ 10: until$|▵p|<ϵ$

2.2. Symmetric Face Alignment

In this subsection, we combine the symmetric normalization with the DSRC in problem (1) to solve the face alignment problem. Suppose I is a probe image subject to an unknown expression and T denotes a template image with the same identity with I. The goal of alignment is to find a transformation of T such that the difference between I and $T\left(W\right(\mathbf{x},p\left)\right)$ is minimized, i.e.,

$$\begin{array}{c}\hfill \underset{p}{min}\sum _{\mathbf{x}}{\parallel T\left(W(\mathbf{x},p)\right)-I\left(\mathbf{x}\right)\parallel }_2,\end{array}$$

(19)

where W denotes the transformation of T with parameters p, and ${\parallel .\parallel }_2$ means $L_2$ norm. The summation is over the mesh on T.

To solve the optimization problem in problem (19), it is necessary to linearize the optimization problem. Assume that a current estimate of p is known. We can iteratively solve for increments to the parameters $▵p$ [32]. As a result, an equivalent problem in (19) is deduced as

$$\begin{array}{c}\hfill \underset{▵p}{min}\sum _{\mathbf{x}}{\parallel T\left(W(\mathbf{x},p+▵p)\right)-I\left(\mathbf{x}\right)\parallel }_2.\end{array}$$

(20)

The optimization iteratively updates p in each step by

$$\begin{array}{c}\hfill p=p+▵p,\end{array}$$

(21)

until a threshold is reached that $\parallel ▵p\parallel \le ϵ$. Here, $ϵ$ is a threshold preset manually.

It is remarkable that if we constrain the transformation T to be similarity transformation; then, it can be described as

$$\begin{array}{ccc}\hfill T\left(\mathbf{x}\right)& =& \left(\begin{array}{ccc}a& b& c\\ -b& a& d\end{array}\right)\left(\begin{array}{c}x\\ y\\ 1\end{array}\right)\hfill \end{array}$$

(22)

$$\begin{array}{cc}=& s\left(\begin{array}{cc}cos\theta & sin\theta \\ -sin\theta & cos\theta \end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right)+\left(\begin{array}{c}c\\ d\end{array}\right).\hfill \end{array}$$

(23)

However, based on the symmetric normalization above, it is assumed that two parameters have been estimated by symmetric normalization in Section 2.1. In other words, the $\theta$ and c are fixed in (23). Therefore, the transformation T is reduced to

$$\begin{array}{ccc}\hfill T\left(\mathbf{x}\right)& =& s\left(\begin{array}{c}x\\ y\end{array}\right)+\left(\begin{array}{c}0\\ d\end{array}\right).\hfill \end{array}$$

(24)

As a result, only two parameters are left, i.e., scale s and vertical translation d.

3. Experimental Evaluation

In this section, experiments are designed to evaluate two aspects of the proposed algorithms, i.e., efficiency and accuracy. For efficiency evaluation, the configuration of the experimental platform is PC with Intel Core i5 of Intel, Santa Clara, CA, USA, 1.7 GHz CPU, 8 GB of memory. Algorithms in this paper are coded by Matlab 2016, MathWorks of Natick, MA, USA.

Two databases are involved in the following experiments, i.e., FERET [34] and Cohn–Kanade [35].

• The FERET database contains 15 sessions of face pictures. It was collected during 1993 and 1996. The dataset includes 14,126 images of 1199 individuals, which are separated into 1564 sets. Moreover, there are duplicate sets which contain images of same person that are already in the database but collected on a different day. This time difference is as long as two years to allow enough distinction. In the following experiments, only “fb” with a smile expression is chosen. Example images are shown in Figure 1.
• The Cohn–Kanade Database is collected for research in automatic facial image analysis and collected with 97 subjects. It contains 486 video sequences, each of which begins with a neutral expression and ends with a peak expression. In this paper, both neutral and peak expressions are involved for our test. Example images are shown in Figure 2.

For comparison, we mainly evaluate the performance of four algorithms, i.e., SRC, symmetric face normalization (SFN) + SRC, deformable SRC (DSRC), and SFN + SFA + SRC. Particularly, SRC is performed on images without alignment. It can be considered as the baseline. SFN + SRC applies symmetric normalization on faces before SRC. This is effective when minor disturbance occurs. DSRC employs alignment with affine transformation before SRC. It is robust to complex transformation; however, it triggers more computational cost. SFN + SFA + SRC performs SFN and SFA in turn before SRC.

3.1. Symmetric Face Normalization

In this subsection, we evaluate the accuracy of the SFN algorithm on single faces. For this purpose, the FERET dataset is involved. The two outer eye corners of each image in FERET dataset are manually labeled, c.f. Figure 2. Then, each face is deformed by 2D similarity transformation, i.e., translation, rotation, and scale. The translation ranges from [−12, 12] pixels with a step of two pixels, the rotation is up to 60 degrees, and the scale ranges from [0.6, 1.4].

Normalization results can be evaluated by the mean error of the two outer eye corners in the x-axis, i.e.,

$$e=mean(e_{xleft},e_{xright}),$$

(25)

where $e_{xleft}$ and $e_{xright}$ are the errors of the two outer eye corners in x-axis, respectively.

Since there is unavoidable noise in labeling, it is accepted that there is minor mis-normalization. Therefore, it is considered to be successful normalization if $\parallel e\parallel \le 0.5$. Given the number of test images is N, the normalizing accuracy is evaluated as

$$\begin{array}{c}\hfill accuracy=\frac{1}{N}\sum _i^N{\delta }_i,{\delta }_i=\left\{\begin{array}{cc}1,\hfill & \parallel e\parallel \le 0.5,\hfill \\ 0,\hfill & \parallel e\parallel >0.5.\hfill \end{array}\right.\end{array}$$

(26)

The normalizing accuracy results are shown in Figure 3. Three conclusions can be drawn as follows: firstly, the symmetric normalization algorithm is sensitive to translation on the x-axis and rotation, and robust to translation on the y-axis and variation in scale. This result is consistent with the definition of “symmetric” in this paper which only affects parameters in connection with the x-axis, such as translation on the x-axis and rotation. Secondly, there is hardly any effect of the variation of scale and translation on the y-axis to the accuracy of symmetric face normalization. It is easy to understand that symmetry has nothing to do with translation on the y-axis and variation in scale. Finally, the robustness of the symmetric face normalization is limited in a range of parameter values. For example, the accuracy of SFN is higher than 0.95 when the translation on the x-axis is lower than 15 pixels, or the rotation angle is lower than 18${}^{\circ }$.

3.2. Symmetric Face Alignment (SFA)

In this subsection, we evaluate the robustness and efficiency of symmetric face alignment algorithm proposed in Section 2.2. To this end, all images with neutral expression in the FERET dataset are normalized using the SFN algorithm, which leads to symmetric faces. Then, each face is deformed by a 2D similarity transformation, i.e., translation, and scale. The translation ranges from [−12, 12] pixels with a step of 2 pixels, the rotation is up to 60 degrees, and the scale ranges from [0.6, 1.4]. Then, the SFA algorithm is applied to each image and its transformed one.

The results are evaluated by the alignment error ${\parallel e\parallel }_1$. More specifically, let $e_0$ be the alignment error obtained by aligning a probe image from the manually labeled position on the gallery image. It is considered to be successful alignment if ${\parallel e\parallel }_1-{\parallel e_0\parallel }_1\le 0.01{\parallel e_0\parallel }_1$. Given the number of probe images is N, the aligning accuracy is evaluated as

$$\begin{array}{c}\hfill SuccessfulRate=\frac{1}{N}\sum _i^N{\delta }_i,{\delta }_i=\left\{\begin{array}{cc}1,\hfill & \mathrm{alignment}\mathrm{is}\mathrm{successful},\hfill \\ 0,\hfill & \mathrm{alignment}\mathrm{is}\mathrm{failed}.\hfill \end{array}\right.\end{array}$$

(27)

Comparison is performed among DSRC, SFN + DSRC and SFN + SFA. Particularly, DSRC aligns each image pair without symmetric normalization. SFN + DSRC normalizes the face image with SFN before aligning each image pair with DSRC, while SFN + SFA first normalizes the face image with SFN and then aligns each image pair with SFA. The aligning results are shown in Figure 4. From the results, we can find that SFN + SFA obtains the best performance. The most possible reason comes from the more robust symmetric normalization algorithm, which greatly reduces the number of parameters needed to optimize in the alignment problem. However, it can also be observed that the difference between these three algorithms is not much.

At the same time, the efficiency of the two algorithms is compared by means of the average running time (ART), i.e., the average time spent on aligning one image. The ART results are shown in

Table 1. Efficiency comparison between DSRC, SFN + DSRC and SFN + SFA.

	DSRC	SFN + DSRC	SFN + SFA
ART (s)	1.21 ± 0.11	1.19 ± 0.95	0.32 ± 0.05

. It indicates that the efficiency of SFN + SFA (0.32 s) is about four times that of the DSRC (1.21 s).

4. Conclusions

In this paper, we have presented a symmetric face normalization algorithm. The key idea is to normalize a face image by using the intrinsic characteristic, i.e., symmetry. As a result, an image is normalized with respect to the symmetric axis of face. The proposed symmetric normalization algorithm has two advantages. On the one hand, it is able to normalize a face image without the help of any other image. On the other hand, it is able to decrease the number of parameters in the later alignment procedure. This is important in cases where an image needs to be aligned in the pre-process. Subsequently, an extension of SFN, symmetric face alignment (SFA), was proposed to make use of symmetry in the alignment between two images. Experiments showed that the proposed algorithm significantly enhanced the efficiency of between-image alignment without loss of accuracy.