Next Article in Journal
Application of a Deep Neural Network to Phase Retrieval in Inverse Medium Scattering Problems
Next Article in Special Issue
Multi-Model Approach and Fuzzy Clustering for Mammogram Tumor to Improve Accuracy
Previous Article in Journal
Stabilization of the Computation of Stability Constants and Species Distributions from Titration Curves
Previous Article in Special Issue
Classification of Categorical Data Based on the Chi-Square Dissimilarity and t-SNE
Article

Integrated Multi-Model Face Shape and Eye Attributes Identification for Hair Style and Eyelashes Recommendation

1
Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool L69 3GJ, UK
2
Department of Computer Engineering, University of Diyala, Baqubah 32010, Iraq
*
Author to whom correspondence should be addressed.
Academic Editor: Shengkun Xie
Received: 4 March 2021 / Revised: 22 April 2021 / Accepted: 26 April 2021 / Published: 27 April 2021

Abstract

Identifying human face shape and eye attributes is the first and most vital process before applying for the right hairstyle and eyelashes extension. The aim of this research work includes the development of a decision support program to constitute an aid system that analyses eye and face features automatically based on the image taken from a user. The system suggests a suitable recommendation of eyelashes type and hairstyle based on the automatic reported users’ eye and face features. To achieve the aim, we develop a multi-model system comprising three separate models; each model targeted a different task, including; face shape classification, eye attribute identification and gender detection model. Face shape classification system has been designed based on the development of a hybrid framework of handcrafting and learned feature. Eye attributes have been identified by exploiting the geometrical eye measurements using the detected eye landmarks. Gender identification system has been realised and designed by implementing a deep learning-based approach. The outputs of three developed models are merged to design a decision support system for haircut and eyelash extension recommendation. The obtained detection results demonstrate that the proposed method effectively identifies the face shape and eye attributes. Developing such computer-aided systems is suitable and beneficial for the user and would be beneficial to the beauty industrial.
Keywords: cosmetic; deep learning; facial image; decision support system; eyelash extension; haircut recommendation; convolutional neural networks cosmetic; deep learning; facial image; decision support system; eyelash extension; haircut recommendation; convolutional neural networks

1. Introduction

The human face plays an important role in daily life. Pursuing beauty, especially facial beauty, is the nature of human beings. As the demand for aesthetic surgery has increased widely over the past few years, an understanding of beauty is becoming increasingly important for medical settings. The physical beauty of faces affects many social outcomes, such as mate choices and hiring decisions [1,2]. Thus, the cosmetic industry has produced various products that target to enhance different parts of the human body, including hair, skin, eye, eyebrow, and lips. Not surprisingly, research topics based on the face features have a long track record in psychology, and many other scientific fields [3]. During recent decades, computer vision systems have played a major role in obtaining an image from a camera to process and analyse it in a manner similar to natural human vision system [4]. Computer vision algorithms have recently attracted increasing attention and been considered one of hottest topics due to its significant role in healthcare, industrial and commercial applications [4,5,6,7,8,9]. Facial image processing and analysis are essential techniques that help extract information from images of human faces. The extracted information such as locations of facial features such as eyes, nose, eyebrows, mouth, and lips, can play a major role in several fields, such as medical purposes, security purposes, cosmetic industry, social media applications, and recognition [5]. Several techniques have been developed to localise these parts and extract them for analysis [10]. The landmarks used in computational face analysis often resemble the anatomical soft tissue landmarks that are used by physicians [11]. Recently, advanced technologies such as artificial intelligence and machine/deep learning algorithms have helped the beauty industry in several ways, from providing statistical bases for attractiveness and helping people alert their looks to developing products which would tackle specific needs of customers [12,13]. Furthermore, cloud computing facilities and data centre services have gained a lot of attention due to their significant role for customers’ access to such products by building web-based and mobile applications [14].
In the literature, there have been many facial attribute analysis methods presented to recognise whether a specific facial attribute is present in a given image. The main aim of developing facial attribute analysis methods was to build a bridge between feature representations required by real-world computer vision tasks and human-understandable visual descriptions [15,16]. Deep learning-based facial attribute analysis methods can generally be grouped into two categories: holistic methods which exploit the relationships among attributes to get more discriminative cues [17,18,19] and part-based methods that emphasise facial details for detecting localisation features [20,21]. Unlike the existing facial attribute analysis methods, which focus on recognising whether a specific facial attribute is present in a given face image or not, our proposed method suggests that all concerned attributes are present but in more than one label.
Furthermore, many automated face shape classification systems were presented in the literature. Many of these published face classification methods consider extracting the face features manually then passing them to three classifiers for classification, including linear discriminant analysis (LDA), artificial neural networks (ANN), and support vector machines (SVM) [22,23], k-nearest neighbours [24], and probabilistic neural networks [25]. Furthermore, Bansode et al. [26] proposed a face shape identification method based on three criteria which are region similarity, correlation coefficient and fractal dimensions. Recently, Pasupa et al. [27] presented a hybrid approach combining VGG convolutional neural network (CNN) with SVM for face shape classification. Moreover, Emmanuel [28] adopted pretrained Inception CNN for classifying face shape using features extracted automatically by CNN. The work presented by researchers showed progress in face shape interpretation; however, existing face classification systems require more effort to achieve better performance. The aforementioned methods perform well only on images taken from subjects looking straight towards the camera and their body in a controlled position and acquired under a clear light setting.
Recently, many approaches were developed for building fashion recommender systems [29,30]. Conducting a deep search in the literature seeking the existing recommendation systems leads to find out two hairstyle recommendation systems [27,31]. The system developed by Liang and Wang [31] considers many attributes such as age, gender, skin colour, occupation and customer rating for recommending hairstyle. However, the recommended haircut style might not fit the beauty experts’ recommendations based on face shape attributes. Furthermore, the hairstyle recommender system presented by [27] is not general and focuses only on women. To the best of our knowledge, our proposed eyelashes and hairstyle recommendation system is the first study conducted to automatically make a recommendation of a suitable eyelash type based on computer vision techniques.
Facial attributes such as eye features, including shape, size, position, setting, face shape, and contour determine what makeup technique, eyelashes extension and haircut styles should be applied. Therefore, it is important for beautician to know clients’ face and eye features before stepping into the makeup-wearing, haircut style and eyelashes extension applying. The face and eye features are essential for beauty expert because different types of face shape and eye features are critical information to decide what kind of eye shadows, eyeliners, eyelashes extension, haircut style and colour of cosmetics are best suited to a particular individual. Thus, automation of facial attribute analysis tasks based on developing computer-based models for cosmetic purposes would help to easing people’s life and reducing time and effort spent by beauty experts. Furthermore, virtual consultation and recommendation systems based on facial and eye attributes [27,31,32,33,34,35,36] have secured a foothold in the market and individuals are opening up to the likelihood of substituting a visit to a physical facility with an online option or use a specific-task software. Use of the innovation of virtual recommendation systems has numerous advantages; including accessibility, enhanced privacy and communication, cost saving, and comfort. Hence, our work’s motivation is to automate the identification of eye attributes and face shape and subsequently produce a recommendation to the user for the appropriate eyelashes and suitable hairstyle.
This research work includes the development of system with friendly graphical user interface that can analyse eye and face attributes from an input image. Based on the detected eye and face features, the system suggests a suitable recommendation of eyelashes and hairstyles for both men and women. The proposed framework integrates three main models: face shape classification model, gender prediction model for predicting the gender of user, and eye attribute identification model to make a decision on the input image. Machine and deep learning approaches with various facial image analysis strategies including facial landmark detection, facial alignment, and geometry measurements schemes are designed to establish and realise the developed system. The main contributions and advantages of this work are summarised as follows:
  • We introduce a new framework merging two types of virtual recommendation, including hairstyle and eyelashes. The overall framework is novel and promising.
  • The proposed method is able to extract many complex facial features, including eye attributes and face shape, accomplishing the extraction of features and detection simultaneously.
  • The developed system could help the worker in beauty centres and reduce their workload by automating their manual work and can produce subjective results, particularly with a large dataset.
  • Our user-friendly interface system has been evaluated on a dataset provided with a diversity of lighting, age, and ethnicity.
  • The labelling of a publicly available dataset with three eye attributes, five face shape classes, and gender has been conducted by a beauty expert to run our experiments.
  • In the face shape classification method, we developed a method based on merging the handcrafted features with deep learning features.
  • We presented a new geometrical measurement method to determine the eye features, including eye shape, position, and setting, based on the coordinates of twelve detected eye landmarks.
The remainder of this paper is presented as follows: in Section 2, the materials and proposed framework are described and explained; results of the proposed system are revealed and discussed in Section 3, and finally, the work has been concluded in Section 4.

2. Development of Hairstyle and Eyelash Recommendation System

2.1. Data Preparation: Facial and Eye Attributes

A publicly available dataset, MUCT [37], is used to conduct our experiments and carry out the evaluation on the developed face shape and eye attribute identification system. The MUCT database consists of 3755 faces captured from 276 subjects with 76 manual landmarks. The database was created to provide more diversity of lighting, age and gender with an obstacle (some subjects wear glasses) than other available landmarked 2D face databases. The resolution of images is 480 × 640 . Five webcams were used to photograph each subject. The five webcams are located in five different positions but not on the left of the subject. Each subject was photographed with two or three lighting setting producing ten different lighting setups in all five webcams. To achieve diversity without too many images, not every subject was photographed with every lighting setup.
The MUCT dataset does not provide face shape and eye attribute labels. Therefore, we recruited a beauty expert to provide the ground truth by labelling the images in the dataset. To formulate our research problem, we initially set up the attributes required to be identified. Face shape and eye specifications/attributes required to be detected in this research work are defined as follow:
  • Eye Shape: Two attributes (Almond or Round),
  • Eye Setting: Three attributes (Wide, Close, or Proportional),
  • Eye Position/Pitch: Three attributes (Upturned, Downturned, or Straight),
  • Face Shape: Five attributes (Round, Oval, Square, Rectangle (Oblong), Heart Shape).
Identifying these attributes would help automatically report eyelashes and haircut style recommendations that meet the eye and face attributes. Illustrations of these attributes are shown in Figure 1 and Figure 2.
The process of determining the facial shape and eye attributes manually by beauty experts can be carried out in several stages, such as outlining the face, measuring the width and length of the face, jaw, forehead, cheekbones, the gap between eyes, and outer and inner corner of the eye. According to the beauty experts, the face shape can be recognised as five categories, namely: oval, square, round, oblong, and heart shapes [41]. The guidelines provided by beauty experts Derrick and Brooke [41,42] have been followed to label the face shape of 276 subjects into five classes. Furthermore, to provide the ground truth of eye attributes, the detailed guideline and described criteria presented in [43] has been followed by a beauty expert. The distribution of data labelled manually is depicted by exploratory data analysis in Figure 3 and Figure 4 describing the distribution of attributes over 276 subjects and per one camera, respectively.

2.2. Developed Multi-Model System

To develop a personalised and automatic approach to finding the appropriate haircut and/or eyelash style for men and women, we develop a multi-model system comprising three separate models, as shown in Figure 5. Each individual model is dedicated to achieving a specific task, including face shape identification, eye attribute identification, and gender identification. The diagram shows that the eye features extracted from an input image by the developed eye feature identification model are merged with identified face shape class. The gender of the individual in the input image is detected by the gender identification model. Thus, the face and eye attributes are combined with identified gender information. If the gender in the input image is a male, then a hairstyle recommendation for men is given; otherwise, both hairstyle and eyelashes recommendations are reported. The description of the three models are explained as follows:

2.2.1. Face Shape Identification Model

To identify the face shape, our developed model [44] that was designed by merging hand-crafted features with automatically learned features was trained and tested on data from MUCT. MUCT data has been randomly split into 3255 images for training, and 500 images was retained for testing. The developed model (shown in Figure 6) achieves the classification as follows: (1) detect the face region and crop it using a model trained on the histogram of oriented gradients (HOG) features with Support Vector Machine (SVM) model as classifier [45], (2) the detected face is aligned using the detected face landmarks (68 landmarks) by the ensemble of regression tree method (ERT) [46], and (3) Finally, the aligned images are used for training and evaluating Inception V3 convolutional neural network [47] along with hand-engineered HOG features and landmarks to classify the face into one of five classes. For further details about the face shape identification model, please refer to our developed system in [44].

2.2.2. Eye Attributes Identification

The geometry of face/eye measurements based on the coordinates of detected eye landmarks are determined to develop this model. Thus, the eye specifications (shape, setting, position) are obtained. First, the face region is detected, and the landmarks of the eye region are localised using the methods presented in our face shape classification system [44,45,46]. We further customised the landmark predictor to return just the locations of the eyes using dlib library. Only six landmarks per eye—12 in two eyes—are taken into consideration for predicting eye attributes, as shown in Figure 7. Given 12 eye landmark points P n = P 0 , P 1 , P 2 , . . . , P n where n = 0 , 1 , 2 , . . . , 11 , the eye specifications can be calculated using geometrical eye measurements as follows:
  • Eye Setting: The setting of eye S e can be determined by calculating the ratio of the distance between two eyes L 1 to the eye width W e y e . The width of the eye W e y e can be found by measuring the distance from the start and the end of an individual eye. The distance between two facial landmark points d can be obtained using the Euclidean distance.
    d ( X , Y ) = ( x 2 x 1 ) 2 + ( y 2 y 1 ) 2
    W e y e 1 = d ( P 0 , P 3 ) W e y e 2 = d ( P 6 , P 9 )
    L 2 = W e y e 1 + W e y e 2 2
    L 1 = d ( P 3 , P 6 )
    S e = L 1 L 2
    The threshold values for category classification are empirically chosen and estimated as follow:
    C l a s s o f e y e s e t t i n g = C l o s e , if S e < 1.57 P r o p o r t i o n a l , if 1.57 S e 1.6 W i d e , Otherwise
    From Equation ( 6), the threshold value used to determine (Close) label is calculated by finding the median of subjects’ eye setting measurements. Thus, the eye setting value of a subject that is less than the median is labelled as (Close). Likewise, the middle number between the median and the highest number (the maximum) is harnessed to differentiate between (Proportional) and (Wide) labels. For example, the eye setting value of a subject that is greater than the median and less than the middle number is labelled as (Proportional); otherwise, it is labelled as (Wide).
  • Eye Shape: The criterion for discriminating round eye from almond eyes is that, for round eyes, the white sclera area below the iris is visible [43] as illustrated in Figure 8. Our proposed criteria exploits eye-aspect-ratio rule to determine the eye shape as follows:
    S h = d ( P 1 , P 5 ) + d ( P 2 , P 4 ) 2 d ( P 0 , P 3 )
    The threshold values for category classification are empirically chosen and estimated as follow:
    C l a s s o f e y e s h a p e = A l m o n d , if S h 0.28 R o u n d , Otherwise
    From Equation (8), the threshold value used to differentiate between (Almond) and (Round) labels is calculated by finding the median of subjects’ eye shape measurements. Thus, the eye shape value that is less than the median is labelled as (Almond); otherwise, it is labelled as (Round).
  • Eye Position: Eye position categorised into up-turned, down-turned and straight can be determined using geometrical calculations by measuring the slope angle (degree). The slope angle is defined as the angle measured between a horizontal line and a slanted line (the line drawn between landmark points ( P 3 , P 6 ) and ( P 0 , P 3 ) for e y e 1 , ( P 6 , P 9 ) for e y e 2 ). The equation of eye position identification can be defined as follows:
    P o = θ e y e 1 + θ e y e 2 2
    where θ represents the angle between the line drawn between landmark points ( P 3 , P 6 ) and ( P 0 , P 3 ) for e y e 1 , ( P 6 , P 9 ) for e y e 2 and each point is represented as ( x , y ) .
    θ e y e = t a n 1 m 1 m 2 1 + m 1 m 2
    where m 1 and m 2 are the slope of each line formulated to calculate the angle θ which is defined as follows:
    m = y 2 y 1 x 2 x 1
    The threshold values for category classification are empirically chosen and estimated as follow:
    C l a s s o f e y e p o s i t i o n = U p t u r n e d , if P o > 6.9 S t r a i g h t , if 0 P o 6.9 D o w n t u r n e d , Otherwise
    From Equation (12), the threshold value used to determine (Up-turned) label is calculated by finding the median of subjects’ eye position measurements. Thus, the eye position value that is greater than the median is labelled as (Up-turned). Likewise, the value between zero and the median is labelled as (Straight). Any value of less than zero (negative) is identified as (Down-turned).

2.2.3. Gender Identification

Gender identification is a pre-requisite step in the our developed framework. If the gender in the input image is male then the recommendation engine reports only the suitable hairstyle for men, otherwise; the system recommends for hairstyle and eyelash extension for women. The deep learning approach developed in [48] has been adopted to achieve this task. Convolutional neural network composed of three convolutional layers, each followed by a rectified linear operation and pooling layer is implemented. The first two layers also follow the normalisation using local response normalisation. The first convolutional layer contains 96 filters of 7 × 7 pixels, the second convolutional layer contains 256 filters of 5 × 5 pixels, the third and final convolutional layer contains 384 filters of 3 × 3 pixels. Finally, two fully connected layers are added, each containing 512 neurons.
The dataset of the Adience Benchmark [48] is used to train the gender identification model. The Adience Benchmark is a collection of unfiltered face images collected from Flickr. It contains 26,580 images of 2284 unique subjects that are unconstrained, extreme blur (low-resolution), with occlusions, out-of-plane pose variations, and have different expressions. Images are first rescaled to 256 × 256 , and a crop of 227 × 227 is fed to the network. All MUCT images composed of 52.5% female were used for testing.

2.2.4. Rule-Based Hairstyle and Eyelash Recommendation System

The right shape of artificial eyelash extensions can be used as a way to correct and balance facial symmetry and open the eyes wider. Eyelash Extensions can produce an illusion of various eye effects giving an appearance of makeup on the eye without applying it. It also can create an appearance of a lifted eye, or even a more feline look being produced [49]. For the best hairstyle, instead of choosing the latest fashion, it should be attempted instead to pick a style that fits face shape. The correct haircut that fits the face shape will frame and balance it expertly while showing the best features for a flattering and complementary appearance [50,51].
The guidelines from specialised websites [49,50,51] are followed to implement the rules of the recommendation engine for hairstyle and eyelashes in our proposed system. The rule-based inference engine system focuses on knowledge representation and applies rules to obtain new information by existing knowledge. When the data (eye attributes, face shape, gender) matches rule conditions, the inference engine can interpret the facts in the knowledge base by applying logical rules and deduce the suitable recommendation using if-then operations. For instance,
IF (Gender is Male) AND (Face Shape is Round) THEN Haircut Recommendation is Side part, French crop, pompadour.
Figure 9 and Figure 10 present the graphical user interface (GUI) of the developed system showing an input image (man and woman) with the automatically detected landmarks, automatically extracted facial/eye attributes and the recommended haircut and eyelash extension.

3. Results and Discussions

A new approach based on merging three models in one framework has been proposed for the simultaneous detection of the face shape, eye attributes, and gender from webcam images. The designed framework achieves the detection by extracting hand-crafted features, learning features automatically, and exploiting the face geometry measurements. To evaluate the performance of the proposed system, different measurements, including accuracy (Acc.), sensitivity (Sn.), specificity (Sp.), Matthews Correlation Coefficient (MCC), Precision (Pr), and F 1 score, have been calculated. These metrics are defined as follows:
A c c u r a c y ( A c c ) = t p + t n t p + t n + f p + f n
S p e c i f i c i t y ( S p ) = t n t n + f p × 100
S e n s i t i v i t y ( S n ) = t p t p + f n × 100
M a t t h e w s C o r r e l a t i o n C o e f f i c i e n t ( M C C ) = t p × t n f p × f n ( t p + f p ) ( t p + f n ) ( t n + f p ) ( t n + f n )
F 1 S c o r e = 2 ( P r e c i s i o n × R e c a l l ) P r e c i s i o n + R e c a l l
where P r e c i s i o n = t p t p + f p , R e c a l l = S e n s i t i v i t y , t p : True Positive, t n : True Negative, f p : False Positive, f n : False Negative
Table 1 depicts the confusion matrix of face shape classes, reporting an identification accuracy of 85.6% on 500 test images (100 image per each class). To compare our face shape classification model with the existing face shape classification methods, Table 2 presents a comparison in terms of classification accuracy. It can be noticed that our developed face shape classification system [44] outperforms the other methods in the literature.
Initially, images belong to one subject were excluded from data because of failure detecting the eye landmarks of this subject in the eye attributes identification system due to extreme eye closing and obstacle. Thus, the number of subjects that will be considered for evaluation is 275. For the gender identification, the gender detection model has achieved prediction performance of 0.9018 , 0.8923 , 0.9103 , 0.8992 , and 0.8958 in terms of accuracy, sensitivity, specificity, precision, and f 1 score measures; respectively, on 275 subjects. Likewise, the gender detection model achieved 0.9064 , 0.8989 , 0.9136 , 0.9088 , and 0.9038 ; respectively, using the same metrics in terms of 748 image per camera. Table 3 presents the confusion matrix obtained from gender identification system. The first part of Table 3 shows the gender prediction in terms of the number of subjects, whereas the second part explores the prediction in terms of images per camera. In terms of comparison with the existing methods, authors in [48] achieved an accuracy of 0.868 on the Adience dataset [48]. On the same dataset, Duan et al. [52] reported accuracy of 0.778 using convolutional neural network (CNN) and extreme learning machine (ELM) method. On Pascal VOC [53] dataset, De et al. [54] obtained accuracy of 0.8739 based on a deep learning approach.
There are five webcams used to photograph each subject. In the following experiments, we name the webcams with five letters (A–E) including, Cam A, Cam B, Cam C, Cam D, and Cam E. Symbols X and Y (used in Cam X vs. Cam Y) refer to any individual webcam from the five cameras. Our developed system has provided flavour results on eye attributes identification. Table 4, Table A1 and Table A2 in Appendix A report eye setting attributes detection performance using various evaluation metrics and state the obtained results from the proposed system by comparing the eye setting attribute prediction performance in one camera with ground truth using confusion matrices, and compare the prediction performance in each camera with another camera using confusion matrices, respectively. Likewise, Table 5, Table A3 and Table A4 report eye position classes detection performance using various evaluation metrics and state the obtained results from the proposed system by comparing the eye position classes prediction performance in one camera with ground truth using confusion matrices, and comparing the prediction performance in each camera with another camera using confusion matrices, respectively. In the same way, Table 6, Table A5 and Table A6 present eye shape labels prediction performance.
From the reported results of eye attributes, it can be noticed that the model achieves the highest performance on eye shape detection, reporting accuracy of 0.7535, sensitivity of 0.7168, specificity of 0.7861, precision of 0.7464, F1 score of 0.7311 and MCC of 0.5043 while the lowest results were obtained from eye position class identification giving 0.6869, 0.7075, 0.8378, 0.6917, 0.6958, and 0.5370 using the same evaluation metrics, respectively. Likewise, the model reveals the results of 0.7021, 0.6845, 0.8479, 0.7207, 0.6926, and 0.5501 on eye setting detection in terms of the same aforementioned metrics. Although the system shows many misclassified prediction of eye attributes, it could be considered to be tolerated prediction cases (skew is small). By “tolerated prediction”, we mean that most misclassified cases are resulted from predicting the class into average labels, which is clear in all the reported confusion matrices. For example, in Table A3, the major predictions of images in Cam E were misclassified as straight rather than upturned label giving: downturned label: 149, straight: 64, upturned: 0. This proves that the predicted measurements of attributes are closer to the ground truth value. This also can be clearly noticed in the reported confusion matrix of the eye setting shown in Table A1.
To study the correlation of prediction among different webcams, all the evaluation metrics mentioned earlier, as well as confusion matrices, have been utilised to depict the variance and closeness of prediction. For instance, Table 4 presents the correlation between cameras (such as Cam A vs. Cam B and so on) in terms of evaluation metrics. Whilst, Table A2 reports the confusion matrices explaining the correlation of prediction among different webcams, which shows good prediction agreement among images captured from different cameras’ positions. Matthews Correlation Coefficient (MCC) is a correlation measurement that returns a value between +1 and −1, where −1 value refers to a total disagreement between two assessments and +1 represents the perfect detection (agreement). The range of obtained MCC value (between two webcams) was (0.3250–0.9145), which indicates of strong correlation of prediction among various webcam positions.
In comparison with the existing methods considering eye attributes detection, Borza et al. [55] and Aldibaja [56] presented eye shape estimation method. Yet, they have not reported the accuracy of the shape detection and instead reported eye region segmentation performance. To the best of our knowledge, our proposed method for eye attributes detection seeking the suitable eyelashes recommendation is the first work carried out automatically based on computer vision approaches. Thus, we could not find more related works in the literature to conduct further comparison with the results obtained from our framework. To further investigate our model’s performance, we validated the developed framework on external data represented by sample images of celebrities, as shown in Table 7. Thus, we have demonstrated that the system generalises well on another data and performs well under variant conditions such as camera position, obstacle (wearing glasses), light condition, gender and age. Yet, on the other hand, it should be considered that the proposed framework brings up some limitations, including: (1) the developed system has not been evaluated on diverse ethnic and racial groups, (2) the presented research work has not considered eye situation characteristics analysis such as mono-lid, hooded, crease, deep-set, and prominent. However, these limitations are due to a lack of labelled data captured from variant races and ethnicities. These challenges can be overcome when larger annotated data becomes available. Moreover, the threshold parameter values used for converting scalar value into a nominal label of eye attribute are selected empirically. For future studies, studying the possibility of using automatic parameter tuning strategies (heuristic mechanisms) for determining and setting optimum threshold values could be investigated.

4. Conclusions

In this paper, a decision support system for appropriate eyelashes and hairstyle recommendation was developed. The developed framework was evaluated on dataset provided with diversity of lighting, age, and ethnicity. The developed system, based on integrating three models, has proven efficient performance in the identification of face shape and eye attributes. Face and eye attributes measurement tasks are usually carried out by an expert person manually before applying the treatment of individual eyelashes extension and hairstyle. Measuring face and eye characteristics costs time and effort. The proposed system could alleviate the need for additional time and effort made by expert before each treatment. Furthermore, face shape identification automatically is a challenging duty due to the complexity of face and the possible variations in rotation, size, illumination, hairstyle, age, and expressions. Moreover, the existence of face occlusion from hats and glasses also adds difficulties to the classification process. Our model trained on diverse images for automatic face shape classification would be viable in many other decision support systems such as eyeglasses style recommendation systems. In future work, we aim to extend our developed GUI desktop software and build a cloud-based decision support system using a mobile application. The extended system targets users’ facial and eye analysis via cloud server programs based on the images captured and uploaded from mobile phones to the server. Furthermore, the detected facial and eye attributes could be harnessed to develop a makeup recommendation system. Another important aspect that could be considered to be potential future work is studying the algorithm’s computational complexity to achieve the best and fastest convergence in the developed framework.

Author Contributions

T.A. designed and performed the experiments. W.A.-N. conceived the idea of research work and ran the project. B.A.-B. contributed to the data analysis and paper writing up. All authors have read and agreed to the published version of the manuscript.

Funding

Theiab Alzahrani was funded by the Kingdom of Saudi Arabia government.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study through a data collection process conducted by the authors who created this data and made it publicly available for scientific research purposes.

Data Availability Statement

The data presented in this study are openly available in http://www.milbo.org/muct, accessed on 26 April 2021.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations and Abbreviations

Abbreviations

Abbreviations

Shape-AEye shape-Almond
Shape-REye shape-Round
Setting-AVGEye setting-Average
Setting-CEye setting-Close
Setting-WEye setting-Wide
Position-DEye position-Downturned
Position-SEye position-Straight
Position-UEye position-Upturned
Gender-FGender-Female
Gender-MGender-Male
F-SquareFace shape-Square
F-RectangleFace shape-Rectangle
F-RoundFace shape-Round
F-HeartFace shape-Heart
F-OvalFace shape-Oval

Appendix A

Table A1. Confusion matrices depict eye setting attributes prediction in each webcam versus the ground truth.
Table A1. Confusion matrices depict eye setting attributes prediction in each webcam versus the ground truth.
PredictedDesired
CloseAverageWide
Cam AClose111721
Average2316261
Wide358257
Cam BClose112943
Average1915265
Wide646251
Cam CClose122957
Average1316858
Wide229254
Cam DClose110822
Average2115267
Wide658250
Cam EClose108753
Average2517170
Wide446246
All CamerasClose56341816
Average101805321
Wide212371258
Table A2. Confusion matrices depict eye setting attributes prediction in each individual camera versus another camera.
Table A2. Confusion matrices depict eye setting attributes prediction in each individual camera versus another camera.
Cam X vs. Cam YCam B
CloseAverageWide
Cam AClose154291
Average5312865
Wide279237
Cam C
Cam AClose165190
Average5013462
Wide986223
Cam D
Cam AClose147370
Average4413369
Wide370245
Cam E
Cam AClose145381
Average3815256
Wide376239
Cam C
Cam BClose169400
Average4712564
Wide874221
Cam D
Cam BClose156494
Average3813266
Wide059244
Cam E
Cam BClose153542
Average3113372
Wide279222
Cam D
Cam CClose160577
Average3312284
Wide161223
Cam E
Cam CClose161549
Average2514074
Wide072213
Cam E
Cam DClose148442
Average3513768
Wide385226
Table A3. Confusion matrices depict eye position attributes prediction in each webcam versus the ground truth.
Table A3. Confusion matrices depict eye position attributes prediction in each webcam versus the ground truth.
PredictedDesired
DownturnedStraightUpturned
Cam ADownturned161781
Straight5219265
Upturned028171
Cam BDownturned156700
Straight5619884
Upturned130153
Cam CDownturned155852
Straight5618074
Upturned233161
Cam DDownturned165680
Straight4819581
Upturned035156
Cam EDownturned149750
Straight6420565
Upturned018172
All CamerasDownturned7863763
Straight276970369
Upturned3144813
Table A4. Confusion matrices depict eye position attributes prediction in each individual camera versus another camera.
Table A4. Confusion matrices depict eye position attributes prediction in each individual camera versus another camera.
Cam X vs. Cam YCam B
DownturnedStraightUpturned
Cam ADownturned152871
Straight7417956
Upturned072127
Cam C
Cam ADownturned158802
Straight8316957
Upturned161137
Cam D
Cam ADownturned152871
Straight8116068
Upturned077122
Cam E
Cam ADownturned152880
Straight7218354
Upturned063136
Cam C
Cam BDownturned148762
Straight9216482
Upturned270112
Cam D
Cam BDownturned145810
Straight8716982
Upturned174109
Cam E
Cam BDownturned146800
Straight7718774
Upturned167116
Cam D
Cam CDownturned145961
Straight8615272
Upturned276118
Cam E
Cam CDownturned146942
Straight7717162
Upturned169126
Cam E
Cam DDownturned135980
Straight8916174
Upturned075116
Table A5. Confusion matrices depict eye shape attributes prediction in each webcam versus the ground truth.
Table A5. Confusion matrices depict eye shape attributes prediction in each webcam versus the ground truth.
PredictedDesired
RoundAlmond
Cam ARound256104
Almond80308
Cam BRound256101
Almond80311
Cam CRound248103
Almond88309
Cam DRound252100
Almond84312
Cam ERound24288
Almond94324
All CamerasRound1254496
Almond4261564
Table A6. Confusion matrices depict eye shape attributes prediction in each individual camera versus another camera.
Table A6. Confusion matrices depict eye shape attributes prediction in each individual camera versus another camera.
Cam X vs. Cam YCam B
RoundAlmond
Cam ARound33822
Almond19369
Cam C
Cam ARound33525
Almond16372
Cam D
Cam ARound34020
Almond12376
Cam E
Cam ARound32040
Almond10378
Cam C
Cam BRound32730
Almond24367
Cam D
Cam BRound33423
Almond18373
Cam E
Cam BRound31740
Almond13378
Cam D
Cam CRound32823
Almond24373
Cam E
Cam CRound31338
Almond17380
Cam E
Cam DRound32032
Almond10386

References

  1. Shahani-Denning, C. Physical attractiveness bias in hiring: What is beautiful is good. Hofstra Horiz 2013, 14–17. Available online: https://www.hofstra.edu/pdf/orsp_shahani-denning_spring03.pdf (accessed on 26 April 2021).
  2. Kertechian, S. The impact of beauty during job applications. J. Hum. Resour. Manag. Res. 2016, 2016, 598520. [Google Scholar] [CrossRef]
  3. Jain, A.K.; Li, S.Z. Handbook of Face Recognition; Springer: New York, NY, USA, 2011; Volume 1. [Google Scholar]
  4. Nixon, M.S. Feature Extraction & Image Processing for Computer Vision; Academic Press: Cambridge, MA, USA, 2012. [Google Scholar]
  5. Garcia, C.; Ostermann, J.; Cootes, T. Facial image processing. EURASIP J. Image Video Process. 2008, 207, 1–33. [Google Scholar]
  6. Zivkovic, M.; Bacanin, N.; Venkatachalam, K.; Nayyar, A.; Djordjevic, A.; Strumberger, I.; Al-Turjman, F. COVID-19 cases prediction by using hybrid machine learning and beetle antennae search approach. Sustain. Cities Soc. 2021, 66, 102669. [Google Scholar] [CrossRef] [PubMed]
  7. Vandenhende, S.; Georgoulis, S.; Proesmans, M.; Dai, D.; Van Gool, L. Revisiting Multi-Task Learning in the Deep Learning Era. arXiv 2020, arXiv:2004.13379. [Google Scholar]
  8. De Rosa, G.H.; Papa, J.P.; Yang, X.S. Handling dropout probability estimation in convolution neural networks using meta-heuristics. Soft Comput. 2018, 22, 6147–6156. [Google Scholar] [CrossRef]
  9. Al-Bander, B.; Alzahrani, T.; Alzahrani, S.; Williams, B.M.; Zheng, Y. Improving fetal head contour detection by object localisation with deep learning. In Annual Conference on Medical Image Understanding and Analysis; Springer: Cham, Switzerland, 2019; pp. 142–150. [Google Scholar]
  10. Wu, Y.; Ji, Q. Facial landmark detection: A literature survey. Int. J. Comput. Vis. 2019, 127, 115–142. [Google Scholar] [CrossRef]
  11. Koestinger, M.; Wohlhart, P.; Roth, P.M.; Bischof, H. Annotated Facial Landmarks in the Wild: A Large-scale, Real-world Database for Facial Landmark Localization. In Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain, 6–13 November 2011; pp. 2144–2151. [Google Scholar]
  12. Priya, M. Artificial Intelligence and the New Age Beauty and Cosmetics Industry. Available online: https://www.prescouter.com/2019/06/artificial-intelligence-and-the-new-age-beauty-and-cosmetics-industry/ (accessed on 1 April 2020).
  13. Edwards, C. AI in the Beauty Industry: The Tech Making the Future Beautiful. Available online: https://www.verdict.co.uk/ai-in-the-beauty-industry/ (accessed on 1 April 2020).
  14. Panda, S.K.; Jana, P.K. Load balanced task scheduling for cloud computing: A probabilistic approach. Knowl. Inf. Syst. 2019, 61, 1607–1631. [Google Scholar] [CrossRef]
  15. Zheng, X.; Guo, Y.; Huang, H.; Li, Y.; He, R. A survey of deep facial attribute analysis. Int. J. Comput. Vis. 2020, 1–33. [Google Scholar] [CrossRef]
  16. Liu, Z.; Luo, P.; Wang, X.; Tang, X. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 3730–3738. [Google Scholar]
  17. Rudd, E.M.; Günther, M.; Boult, T.E. Moon: A mixed objective optimization network for the recognition of facial attributes. In European Conference on Computer Vision; Springer: Cham, Switzerland, 2016; pp. 19–35. [Google Scholar]
  18. Hand, E.M.; Chellappa, R. Attributes for improved attributes: A multi-task network utilizing implicit and explicit relationships for facial attribute classification. Proc. Thirty First Aaai Conf. Artif. Intell. 2017, 31, 4068–4074. [Google Scholar]
  19. Cao, J.; Li, Y.; Zhang, Z. Partially shared multi-task convolutional neural network with local constraint for face attribute learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 4290–4299. [Google Scholar]
  20. Kalayeh, M.M.; Gong, B.; Shah, M. Improving facial attribute prediction using semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 6942–6950. [Google Scholar]
  21. Mahbub, U.; Sarkar, S.; Chellappa, R. Segment-based methods for facial attribute detection from partial faces. IEEE Trans. Affect. Comput. 2018, 11, 601–613. [Google Scholar] [CrossRef]
  22. Sunhem, W.; Pasupa, K. An approach to face shape classification for hairstyle recommendation. In Proceedings of the 2016 Eighth International Conference on Advanced Computational Intelligence (ICACI), Chiang Mai, Thailand, 14–16 February 2016; pp. 390–394. [Google Scholar]
  23. Sarakon, P.; Charoenpong, T.; Charoensiriwath, S. Face shape classification from 3D human data by using SVM. In Proceedings of the 7th 2014 Biomedical Engineering International Conference, Fukuoka, Japan, 26–28 November 2014; pp. 1–5. [Google Scholar]
  24. Zafar, A.; Popa, T. Face and Eye-ware Classification using Geometric Features for a Data-driven Eye-ware Recommendation System. In Graphics Interface Conference; ACM Library: New York, NY, USA, 2016; pp. 183–188. [Google Scholar]
  25. Rahmat, R.F.; Syahputra, M.D.; Andayani, U.; Lini, T.Z. Probabilistic neural network and invariant moments for men face shape classification. In IOP Conference Series: Materials Science and Engineering; IOP Publishing: Bristol, UK, 2018; Volume 420, p. 012095. [Google Scholar]
  26. Bansode, N.; Sinha, P. Face Shape Classification Based on Region Similarity, Correlation and Fractal Dimensions. Int. J. Comput. Sci. Issues (IJCSI) 2016, 13, 24. [Google Scholar]
  27. Pasupa, K.; Sunhem, W.; Loo, C.K. A hybrid approach to building face shape classifier for hairstyle recommender system. Expert Syst. Appl. 2019, 120, 14–32. [Google Scholar] [CrossRef]
  28. Tio, E.D. Face Shape Classification Using Inception V3. 2017. Available online: https://github.com/adonistio/inception-face-shape-classifier (accessed on 1 June 2019).
  29. Kang, W.C.; Fang, C.; Wang, Z.; McAuley, J. Visually-aware fashion recommendation and design with generative image models. In Proceedings of the 2017 IEEE International Conference on Data Mining (ICDM), New Orleans, LA, USA, 18–21 November 2017; pp. 207–216. [Google Scholar]
  30. Lodkaew, T.; Supsohmboon, W.; Pasupa, K.; Loo, C.K. Fashion Finder: A System for Locating Online Stores on Instagram from Product Images. In Proceedings of the 2018 10th International Conference on Information Technology and Electrical Engineering (ICITEE), Bali, Indonesia, 24–26 July 2018; pp. 500–505. [Google Scholar]
  31. Liang, Z.; Wang, F. Research on the Personalized Recommendation Algorithm for Hairdressers. Open J. Model. Simul. 2016, 4, 102–108. [Google Scholar] [CrossRef]
  32. Wang, Y.F.; Chuang, Y.L.; Hsu, M.H.; Keh, H.C. A personalized recommender system for the cosmetic business. Expert Syst. Appl. 2004, 26, 427–434. [Google Scholar] [CrossRef]
  33. Scherbaum, K.; Ritschel, T.; Hullin, M.; Thormählen, T.; Blanz, V.; Seidel, H.P. Computer-suggested facial makeup. In Computer Graphics Forum; Wiley Online Library: Hoboken, NJ, USA, 2011; Volume 30, pp. 485–492. [Google Scholar]
  34. Chung, K.Y. Effect of facial makeup style recommendation on visual sensibility. Multimed. Tools Appl. 2014, 71, 843–853. [Google Scholar] [CrossRef]
  35. Alashkar, T.; Jiang, S.; Fu, Y. Rule-based facial makeup recommendation system. In Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA, 30 May–3 June 2017; pp. 325–330. [Google Scholar]
  36. Zhao, Z.; Zhou, L.; Zhang, T. Intelligent recommendation system for eyeglass design. In International Conference on Applied Human Factors and Ergonomics; Springer: Cham, Switzerland, 2019; pp. 402–411. [Google Scholar]
  37. Milborrow, S.; Morkel, J.; Nicolls, F. The MUCT landmarked face database. Pattern Recognit. Assoc. S. Afr. 2010, 201, 1–6. Available online: http://www.milbo.org/muct (accessed on 26 April 2021).
  38. Presleehairstyle. Wedding Veils: Denver Wedding Hairstylist. Available online: https://www.presleehairstyle.com/blog/2017/5/31/wedding-veils-denver-wedding-hairstylist-braiding-specialist-and-updo-artist (accessed on 1 February 2020).
  39. Aumedo. Make Up Eyes: Make-Up Tips for Every Eye Shape. Available online: https://www.aumedo.de/make-up-tipps-fuer-jede-augenform/ (accessed on 1 February 2020).
  40. Moreno, A. How to Pick the Best Lashes for Your Eye Shape. Available online: https://www.iconalashes.com/blogs/news/how-to-pick-the-best-lashes-for-your-eye-shape (accessed on 1 May 2020).
  41. Derrick, J. Is Your Face Round, Square, Long, Heart or Oval Shaped? Available online: https://www.byrdie.com/is-your-face-round-square-long-heart-or-oval-shaped-345761 (accessed on 1 June 2019).
  42. Shunatona, B. A Guide to Determining Your Face Shape (If That’s Something You Care about). Available online: https://www.cosmopolitan.com/style-beauty/beauty/a22064408/face-shape-guide/ (accessed on 1 February 2020).
  43. WikiHow. How to Determine Eye Shape. Available online: https://www.wikihow.com/Determine-Eye-Shape#aiinfo (accessed on 1 February 2020).
  44. Alzahrani, T.; Al-Nuaimy, W.; Al-Bander, B. Hybrid feature learning and engineering based approach for face shape classification. In Proceedings of the 2019 International Conference on Intelligent Systems and Advanced Computing Sciences (ISACS), Taza, Morocco, 26–27 December 2019; pp. 1–4. [Google Scholar]
  45. Dalal, N.; Triggs, B. Histograms of oriented gradients for human detection. In Proceedings of the International Conference on Computer Vision & Pattern Recognition (CVPR’05), San Diego, CA, USA, 20–25 June 2005; Volume 1, pp. 886–893. [Google Scholar]
  46. Kazemi, V.; Sullivan, J. One millisecond face alignment with an ensemble of regression trees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 1867–1874. [Google Scholar]
  47. Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 2818–2826. [Google Scholar]
  48. Levi, G.; Hassner, T. Age and gender classification using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Boston, MA, USA, 7–12 June 2015; pp. 34–42. [Google Scholar]
  49. Creasey, C. Eye Shapes and Eyelash Extensions—What Works Best for You? Available online: https://www.sydneyeyelashextensions.com/eye-shapes-and-eyelash-extensions-what-works-best-for-you/ (accessed on 1 March 2020).
  50. Brewer, T. The Most Flattering Haircuts for Your Face Shape. Available online: https://www.thetrendspotter.net/haircuts-for-face-shape/ (accessed on 1 March 2020).
  51. Mannah, J. The Best Men’s Hairstyles for Your Face Shape. Available online: https://www.thetrendspotter.net/find-the-perfect-hairstyle-haircut-for-your-face-shape/ (accessed on 1 March 2020).
  52. Duan, M.; Li, K.; Yang, C.; Li, K. A hybrid deep learning CNN–ELM for age and gender classification. Neurocomputing 2018, 275, 448–461. [Google Scholar] [CrossRef]
  53. Everingham, M.; Eslami, S.A.; Van Gool, L.; Williams, C.K.; Winn, J.; Zisserman, A. The pascal visual object classes challenge: A retrospective. Int. J. Comput. Vis. 2015, 111, 98–136. [Google Scholar] [CrossRef]
  54. de Araujo Zeni, L.F.; Jung, C.R. Real-time gender detection in the wild using deep neural networks. In Proceedings of the 2018 31st Conference on Graphics, Patterns and Images (SIBGRAPI), Parana, Brazil, 29 October–1 November 2018; pp. 118–125. [Google Scholar]
  55. Borza, D.; Darabant, A.S.; Danescu, R. Real-time detection and measurement of eye features from color images. Sensors 2016, 16, 1105. [Google Scholar] [CrossRef] [PubMed]
  56. Aldibaja, M.A.J. Eye Shape Detection Methods Based on Eye Structure Modeling and Texture Analysis for Interface Systems. Ph.D. Thesis, Toyohashi University of Technology, Toyohashi, Japan, 2015. [Google Scholar]
Figure 1. Categories of face shape [38].
Figure 1. Categories of face shape [38].
Computation 09 00054 g001
Figure 2. Eye attributes. Adapted from [39,40].
Figure 2. Eye attributes. Adapted from [39,40].
Computation 09 00054 g002
Figure 3. Distribution of attributes over 276 subject.
Figure 3. Distribution of attributes over 276 subject.
Computation 09 00054 g003
Figure 4. Distribution of attributes over 751 image (per one camera).
Figure 4. Distribution of attributes over 751 image (per one camera).
Computation 09 00054 g004
Figure 5. Block diagram of proposed method including three models: face shape identification model, eye feature identification model, gender identification model.
Figure 5. Block diagram of proposed method including three models: face shape identification model, eye feature identification model, gender identification model.
Computation 09 00054 g005
Figure 6. The developed face shape classification system [44].
Figure 6. The developed face shape classification system [44].
Computation 09 00054 g006
Figure 7. The six landmarks taken into consideration to predict the eye attributes using geometrical measurements.
Figure 7. The six landmarks taken into consideration to predict the eye attributes using geometrical measurements.
Computation 09 00054 g007
Figure 8. Round vs. almond shape.
Figure 8. Round vs. almond shape.
Computation 09 00054 g008
Figure 9. GUI of facial and eye attributes predication showing man example.
Figure 9. GUI of facial and eye attributes predication showing man example.
Computation 09 00054 g009
Figure 10. GUI of facial and eye attributes predication showing woman example.
Figure 10. GUI of facial and eye attributes predication showing woman example.
Computation 09 00054 g010
Table 1. Confusion matrix representing the five classes of face shape.
Table 1. Confusion matrix representing the five classes of face shape.
Predicted
OvalSquareRectangleRoundHeart
Oval59111695
Square099010
Rectangle209215
Round114913
Heart607087
Table 2. The performance of proposed face shape classification methodology compared to the existing methods in the literature.
Table 2. The performance of proposed face shape classification methodology compared to the existing methods in the literature.
MethodAccuracy
Inception V3 [28]84.4%
Region Similarity, Correlation and Fractal Dimensions [26]80%
Active Appearance Model (AAM), segmentation, and SVM [22]72%
Hybrid approach VGG and SVM [27]70.3%
3D face data and SVM [23]73.68%
Geometric features [24]80%
Probability Neural Network and Invariant Moments [25]80%
Our model85.6%
Table 3. Confusion matrix representing the gender identification. The first part of matrix represents the identification in terms of number of subjects; whereas the second part explores the prediction in terms of images per one camera. F: Female, M: Male.
Table 3. Confusion matrix representing the gender identification. The first part of matrix represents the identification in terms of number of subjects; whereas the second part explores the prediction in terms of images per one camera. F: Female, M: Male.
Predicted
FMFM
F1321334933
M1411637329
Table 4. Performance evaluation of proposed system for eye setting attributes detection. Acc.: accuracy, Sn.: Sensitivity, Sp.: Specificity, Pr.: Precision, F 1 : F 1 score, MCC: Matthews Correlation Coefficient, GT: Ground Truth, Cam: Camera.
Table 4. Performance evaluation of proposed system for eye setting attributes detection. Acc.: accuracy, Sn.: Sensitivity, Sp.: Specificity, Pr.: Precision, F 1 : F 1 score, MCC: Matthews Correlation Coefficient, GT: Ground Truth, Cam: Camera.
Webcam SettingAcc.Sn.Sp.Pr. F 1 MCC
Cam A vs. GT0.70860.69000.85030.72360.70020.5563
Cam B vs. GT0.68850.66940.84250.70830.67670.5305
Cam C vs. GT0.72730.71290.86250.75400.71660.5958
Cam D vs. GT0.68450.66550.83890.70240.67530.5218
Cam E vs. GT0.70190.68490.84530.71500.69390.5459
Average (all Cams)0.70210.68450.84790.72070.69260.5501
Cam A vs. B0.69390.70090.84460.68710.69270.5381
Cam A vs. C0.69790.71420.84730.69320.70030.5500
Cam A vs. D0.70190.70330.84770.69740.70010.5479
Cam A vs. E0.71660.71920.85590.71950.71870.5740
Cam B vs. C0.68850.68920.84380.68430.68620.5300
Cam B vs. D0.71120.70370.85380.71040.70660.5611
Cam B vs. E0.67910.67610.83760.69090.68190.5200
Cam C vs. D0.67510.66910.83560.68110.67320.5112
Cam C vs. E0.68720.68400.84180.70380.69100.5349
Cam D vs. E0.68320.68450.83870.69140.68710.5255
Table 5. Performance evaluation of proposed system for eye position attributes detection. Acc.: accuracy, Sn.: Sensitivity, Sp.: Specificity, Pr.: Precision, F 1 : F 1 score, MCC: Matthews Correlation Coefficient, GT: Ground Truth.
Table 5. Performance evaluation of proposed system for eye position attributes detection. Acc.: accuracy, Sn.: Sensitivity, Sp.: Specificity, Pr.: Precision, F 1 : F 1 score, MCC: Matthews Correlation Coefficient, GT: Ground Truth.
Webcam SettingAcc.Sn.Sp.Pr. F 1 MCC
Cam A vs. GT0.70050.71720.84530.70720.70930.5574
Cam B vs. GT0.67780.70250.83270.68080.68670.5235
Cam C vs. GT0.66310.68090.82610.67040.67240.5015
Cam D vs. GT0.68980.70890.83950.69570.69860.5412
Cam E vs. GT0.70320.72810.84560.70440.71210.5614
Average (all Cams)0.68690.70750.83780.69170.69580.5370
Cam A vs. B0.61230.61690.79610.63080.62300.4196
Cam A vs. C0.62030.63120.80200.63230.63180.4338
Cam A vs. D0.58020.58810.78040.59500.59130.3716
Cam A vs. E0.62970.63630.80530.64740.64120.4469
Cam B vs. C0.56680.58290.77160.57070.57600.3489
Cam B vs. D0.56550.57800.76930.57150.57460.3443
Cam B vs. E0.60030.60990.78690.60740.60860.3956
Cam C vs. D0.55480.56380.76700.56980.56660.3335
Cam C vs. E0.59220.59930.78590.60900.60350.3896
Cam D vs. E0.55080.56120.76210.56510.56300.3250
Table 6. Performance evaluation of proposed system for eye shape attributes detection. Acc.: accuracy, Sn.: Sensitivity, Sp.: Specificity, Pr.: Precision, F 1 : F 1 score, MCC: Matthews Correlation Coefficient, GT: Ground Truth.
Table 6. Performance evaluation of proposed system for eye shape attributes detection. Acc.: accuracy, Sn.: Sensitivity, Sp.: Specificity, Pr.: Precision, F 1 : F 1 score, MCC: Matthews Correlation Coefficient, GT: Ground Truth.
Webcam SettingAcc.Sn.Sp.Pr. F 1 MCC
Cam A vs. GT0.75400.71110.79380.76190.73560.5072
Cam B vs. GT0.75800.71710.79540.76190.73880.5146
Cam C vs. GT0.74470.70660.77830.73810.72200.4865
Cam D vs. GT0.75400.71590.78790.75000.73260.5055
Cam E vs. GT0.75670.73330.77510.72020.72670.5075
Average (all Cams)0.75350.71680.78610.74640.73110.5043
Cam A vs. B0.94520.93890.95100.94680.94280.8902
Cam A vs. C0.94520.93060.95880.95440.94230.8904
Cam A vs. D0.95720.94440.96910.96590.95510.9145
Cam A vs. E0.93320.88890.97420.96970.92750.8685
Cam B vs. C0.92780.91600.93860.93160.92370.8553
Cam B vs. D0.94520.93560.95400.94890.94220.8902
Cam B vs. E0.92910.88800.96680.96060.92290.8598
Cam C vs. D0.93720.93450.93950.93180.93310.8739
Cam C vs. E0.92650.89170.95720.94850.91920.8532
Cam D vs. E0.94390.90910.97470.96970.93840.8885
Table 7. Performance evaluation of proposed system on external image data. Red cells refer to incorrect detection while last column shows the true label.
Table 7. Performance evaluation of proposed system on external image data. Red cells refer to incorrect detection while last column shows the true label.
Celebrity IDFace ShapeEye ShapeEye SettingEye PositionCorrection
Olivia WildeSquareRoundWideUpturnedAlmond
RihannaOvalAlmondWideStraightUpturned
Cara DelevingneOvalAlmondWideStraightUpturned
George ClooneyOvalAlmondWideDownturnedRound
AdeleRoundAlmondWideDownturnedStraight
Scarlett JohanssonRoundAlmondWideStraightHeart
Johnny DeppRectangleAlmondAverageDownturnedSquare
Chloe Grace MoretzHeartAlmondWideUpturnedAverage
Camilla BelleRoundAlmondWideDownturnedAverage
Will SmithOvalAlmondWideStraight-
Hilary SwankRectangleAlmondAverageStraightWide
Sarah Jessica ParkerRectangleAlmondWideStraight-
Selena GomezRoundRoundWideStraight-
Naomi CampbellRoundRoundAverageUpturnedHeart
Meryl StreepRectangleRoundAverageStraightClose
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Back to TopTop