Automatic Facial Palsy Detection—From Mathematical Modeling to Deep Learning

Abstract

Automated solutions for medical diagnosis based on computer vision form an emerging field of science aiming to enhance diagnosis and early disease detection. The detection and quantification of facial asymmetries enable facial palsy evaluation. In this work, a detailed review of the quantification of facial palsy takes place, covering all methods ranging from traditional manual mathematical modeling to automated computer vision-based methods. Moreover, facial palsy quantification is defined in terms of facial asymmetry indices calculation for different image modalities. The aim is to introduce readers to the concept of mathematical modeling approaches for facial palsy detection and evaluation and present the process of the development of this separate application field over time. Facial landmark extraction, facial datasets, and palsy grading systems are included in this research. As a general conclusion, machine learning methods for the evaluation of facial palsy lead to limited performance due to the use of handcrafted features, combined with the scarcity of the available datasets. Deep learning methods allow the automatic learning of discriminative deep facial features, leading to comparatively higher performance accuracies. Datasets limitations, proposed solutions, and future research directions in the field are also presented.

1. Introduction

Facial palsy is a common neuromuscular disorder causing facial weakness and the disability of facial expressions [1]. Palsy patients lose control of the affected side of their face, experiencing the dropping or stiffness of muscles and disorders of taste buds. Statistics regarding facial palsy report 25 incidents annually per 100,000 people, or approximately one patient out of 60 people in their lifetime, while an average of 40,000 palsy patients are reported in the United States every year [2]. Even though palsy does not cause patients to be in physical pain, they experience phycological stress, external discomfort, and depression, since palsy affects their appearance, facial movements, feeding functions, and, thus, their daily lives [3]. Therefore, the accurate diagnosis and exact evaluation of the degree of palsy are essential for the objective assessment of the facial nerve’s function in terms of monitoring the progress or resolution of palsy. The latter could help for evaluating the therapeutic processes and designing effective treatment plans.

Traditionally, the diagnosis of facial palsy is clinically performed by specialized neurologists who force patients to perform specific facial expressions for evaluating the condition of certain face muscles. The level of palsy is assessed by evaluating the symmetry between the right and left parts of the face in terms of various scoring standards and measuring distances between facial landmarks for both sides with a simple ruler [4]. The manual and empirical evaluation of palsy are, therefore, both labor intensive and subjective. Assessment based on visual inspection makes it hard to precisely quantify the severity of palsy, and it is not feasible to track improvements between subsequent rehabilitation interventions. Moreover, assessment relies on the degree of human expertise; thus, the clinical quantification of palsy may differ between different neurologists [5]. Hence, an objective quantitative evaluation value would be useful.

Automatic inspection approaches can alleviate these disadvantages and provide more consistent and objective facial palsy diagnosis and evaluation methods, providing neurologists with an efficient decision-supporting tool [6]. The automatic quantitative evaluation of facial palsy has been a subject of research for many years. Several approaches use optical markers attached to human faces to determine the degree of palsy [7,8], as well as full-face laser scanning [9,10] or electroneurography (ENoG) and electromyography (EMG) signals. The latter approaches, although very accurate, require specialized high-cost equipment and a constrained clinical environment and presuppose physical interventions, which are obtrusive and uncomfortable. Moreover, the patients themselves cannot perform these approaches on their own to monitor their progress at home.

Recent advancements in image analysis algorithms, combined with the increasingly affordable cost of high-resolution capturing devices, resulted in the development of efficient, simple, and cost-effective vision-based techniques for medical applications, reporting impressive state-of-the-art performances [11,12,13]. The diagnosis of various diseases is greatly assisted by facial abnormalities recognition using computer vision [14,15], dynamically incorporating facial recognition into artificial intelligence (AI)-based medicine [16,17]. Automatic image-based facial palsy could accelerate the diagnosis and progress evaluation of the disease, offering a non-invasive, simple, and time- and cost-saving method that could be used by the palsy patients themselves without the presence of a human expert.

Most image-based techniques are based on hand-crafted features associated with the affected facial regions, used as training data for machine learning and deep learning models. Such facial landmarks need to be accurate and discriminative for improving the performance of the models. Facial landmarking has been a subject of extensive research over the past few years, focusing on the extraction of specific landmarks for different medical conditions inferred from different facial regions [18,19]. Yet, facial landmarks’ annotation requires domain experts. Moreover, deep learning algorithms require large public datasets of facial palsy images for training and model performance comparisons, which are not easy to acquire due to medical confidentiality issues.

To this end, the contributions of this work are summarized as follows:

• Τhis work has collected all the necessary background information related to the detection of facial palsy. Specifically, it includes the following areas:

  a.

  The structure of typical facial assessment systems;

  b.

  Facial landmarks for palsy detection;

  c.

  Facial datasets;

  d.

  Facial palsy grading systems.

• This work aggregately provides the mathematical formulation of all facial palsy asymmetry indices as follows:

  a.

  2D images;

  b.

  3D images;

  c.

  Videos.

• This work reviews and compares the following mainstream techniques:

  a.

  Machine learning;

  b.

  Deep learning approaches for facial palsy detection and evaluation.

This work, to the best of our knowledge, is the first complete guide on facial palsy that integrates all the above relevant knowledge, ranging from mathematical modeling to deep learning, aiming to provide a mathematical foundation, establish baselines, and guide future research in this field. This work constitutes a literature review, providing a synthesis of the state of knowledge on the given topic.

The rest of the paper is structures as follows: Section 2 summarizes the necessary background information related to the detection of facial palsy; Section 3 includes the mathematical formulation of all facial palsy asymmetry indices; and Section 4 reviews facial palsy detection and evaluation methods. Discussion is provided in Section 5, while Section 6 concludes this paper.

2. Foundations

2.1. Overview of Typical Facial Palsy Assessment System

Image analysis methods have been widely utilized to either detect or quantify facial palsy. Therefore, based on their intended use, these techniques can be divided into (1) detection techniques, aiming to discriminate between unhealthy and healthy subjects by forming a binary classification problem, and (2) evaluation techniques, aiming to quantify the level of palsy, forming a multi-class classification problem. As a further division, two additional research sub-categories emerge: research based on static images and research based on videos. The research based on images is less comprehensive, since an image may contain less information than a video. Moreover, images need to depict specific patterns of facial movements, e.g., smile, eyebrow raising, etc., so as to identify facial paralysis. For this reason, techniques based on images may lead to less accurate results; however, they have lower demands and computational costs. The research based on videos can provide more accurate results; in this research, facial key point features are defined, and point tracking is performed to define facial asymmetries during facial movements.

For all cases, the outline of a basic system for facial palsy assessment is illustrated in Figure 1. The first preprocessing step usually includes color and geometric transformations, resizing, denoising, etc., to increase both the quality and the quantity of the image dataset. Then, facial detection is performed for the input image. Facial landmarks are extracted, constituting the key point facial symmetry features used to detect facial palsy. Finally, a classifier is trained to evaluate input images based on the discriminative power of the extracted features.

Deep learning approaches are also considered to be an alternative to handcraft feature extraction methods. In what follows, the mathematical modeling of both hand-crafted features-based and deep learning-based approaches for facial symmetry determination are presented.

2.2. Facial Landmarks

Facial palsy detection and evaluation are based on the quantification of facial symmetry/asymmetry. The first step is to determine reference points on human faces (facial landmarks) and then develop appropriate mathematical modeling methods to numerically calculate symmetry. The most crucial step for defining efficient asymmetry indices is the appropriate selection of landmark reference points, which need to be clear, reproducible, and evenly distributed of the face surface, especially where palsy affects symmetry, such as the eyes and mouth. Moreover, an ideal number of landmark points needs to be considered; a big number of points may increase the accuracy of palsy evaluation, though this would lead to greater computational costs. We note that 2D techniques focus on frontal face views. Most of the approaches for facial palsy detection rely on static images, where facial landmarks are determined and geometric features are calculated. Based on the above information, it is obvious that there are many challenges related to the determination of efficient facial landmarks. Moreover, landmark localization and extraction can be inaccurate, especially when using video sequences or in-the-wild benchmark image datasets; in these cases, landmark extraction and tracking can be rather challenging [20]. The latter is also verified by the vivid interest of researchers in this specific field of efficient facial landmark extraction, both for palsy detection and for other tasks, e.g., facial expression detection [21,22,23].

A general overview of used facial reference points is provided in Figure 2a. In this case, the 27 reference points (10 for each side and 7 in the middle vertical line, including the upper (tr) and lower (gn) centers of the hair line and gnathion, respectively; the eyebrow upper center (eup) and upper lip (lup); the lower center of the lower lip (ldo); the center of the lips (st); the upper (nasion, n) and lower (sn) points of the center of the nose; the upper (aup) and lower (ado) corners of the ear; the inner (en) and outer (ex) corners of the eye; the corner of the mouth (ch); the outmost points of the zygomatic arc (zy); the nasal wing (an) and angle of the mandible (m); and the center of the pupil (p). The latter system is mainly used in facial surgery [24].

For facial palsy detection, however, specific points are required to highlight the dominant facial characteristics that are affected. More specifically, facial landmark detection plays an important role in the computer-aided diagnosis and evaluation of facial palsy, since all machine learning methods rely on facial landmarks to learn facial symmetry, texture, and shape features from patients’ images. Moreover, palsy patients have facial asymmetry symptoms in specific face areas, such as one-sided eye narrowing and deformation of the lip contour. Therefore, specific facial landmarks can concentrate on the most affected areas, both to reduce computations and improve detection algorithms’ performance, due to the exclusion of redundant information. A well-known facial reference point system is that of the 68 points, found in the 300-W dataset [25], as illustrated in Figure 2b. The latter annotation system is also used in the Annotated Facial Landmarks for Facial Palsy (AFLFP) dataset [5], which represents the first annotated public facial landmark dataset used in facial palsy detection tasks. Variations have been proposed, and various datasets with faces are available, including indoor and outdoor face images, which have various expressions (neutral, surprise, smile, scream, squint, disgust, brow raise, etc.), along with their proposed facial landmark annotations. Figure 2c illustrates a variation in the 98 reference points. It should be noted here that all images in this work were original and made by the authors to serve the scope of the survey. The face model depicted in the figures is a render of an originally designed head sculpture model implemented in ZBrush via Maxon; lines, points, letters and numbers were added afterwards to the render of the head model created in Adobe Illustrator.

2.3. Facial Datasets

Table 1. Basic details of popular facial datasets.

Dataset [Ref.]	Information	Frontal Facial Landmark Points
Labeled Face Parts in-the-wild (LFPW) [26]	A total of 1287 annotated images from Flickr, Google, and Yahoo.	35
HELEN [27]	A total of 2330 annotated images from Flickr.	194
Annotated Faces in-the-wild (AFW) [28]	A total of 205 annotated images, including 468 faces.	6
Annotated Facial Landmarks in-the-wild (AFLW) [29]	A total of 25,000 annotated images from Flickr.	21
300 Faces in-the-Wild (300-W) [25]	A total of 600 annotated images captured in unconstrained settings.	68
Extended Multi Modal Verification for Teleservices and Security applications (XM2VTS) [30]	A total of 30 h of annotated video recordings of 295 subjects, including frontal and profile views.	22
CMU MultiPIE [31]	A total of 750,000 images of 337 people from different viewpoints, expressions, and illuminations. A small subset annotated.	68
Annotated Facial Landmarks for Facial Palsy (AFLFP) [5]	Annotated video images of 16 facial expressions of 88 subjects (palsy patients and healthy individuals).	68
Hi4D-ADSIP 3D dynamic facial articulation database [32]	Three-dimensional dynamic facial articulation dataset, with scans with high temporal and spatial resolutions, containing 3360 facial sequences captured from 80 healthy volunteers.	84
YouTube Facial Palsy (YPF) [33]	A total of 32 videos of 22 patients acquired from YouTube and labeled by clinic experts.	-
Massachusetts Eye and Ear Infirmary (MEEI) [34]	Images and videos of patients with flaccid and non-flaccid facial palsy	-

includes information regarding a selection of the most popular facial palsy datasets. It should be noted that all datasets, apart from the AFLFP, YFP, and MEEI, only include face images of healthy subjects. AFLFP is the first annotated image dataset for facial palsy. YFP and MEEI are also recent facial palsy datasets; however, they do not specify the annotations of the facial landmark points and are used in deep learning-based approaches for the automatic quantitative evaluation of facial palsy.

2.4. Facial Palsy Grading Systems

Facial palsy diagnosis and rehabilitation require a comprehensive standard grading system to measure the extent of facial paralysis and its progression over time. A standard grading system is not yet available; however, several grading systems for evaluating facial palsy have been proposed, based on either traditional approaches, such as subjective judgements made by clinicians, or automated approaches, such as computer aided objective facial measurements using image analysis and classification [35].

Table 2. Facial paralysis grading systems.

Grading system, Year [Ref.]	Information	Traditional (T)/Computer Vision-Based (CV)
Botman and Jongkees, 1955 [36]	Five-category gross scale from 0 (normal) to IV (total paralysis).	T
Peitersen, 2002 [37]	Five-degree gross grading system from I (no palsy) to VI (complete palsy).	T
Smith, 1980 [38]	Unweighted regional system of four categories (entire face, forehead, eye function, and mouth function) leading to scores from 0 (no function) to 4 (normal).	T
Adour and Swanson, 1971 [39]	Weighted system that measures the percentage of divisions of the facial nerve (frontal, eye, mouth).	T
Janssen, 1963 [40]	Weighted scale expressed in percentage, considering four categories (repose, forehead, eye closure, oral branch).	T
Yanagihara, 1977 [41]	Unweighted system of 10 graded areas of facial functions.	T
Stennert, 1977 [42]	Specific regional negative scale for motor function and secondary defects, ranging from 0 (normal) to −10 (total palsy).	T
House and Brackmann (HB), 1983 [43]	Six category scale from I (normal) to VI (no movement).	T
Burres and Fisch, 1986 [44]	Distance measurement of facial landmarks while resting and between five expressions of the affected side and normal side.	T
Nottingham, 1994 [45]	Measuring the movement of four landmarks in three facial expressions.	T
Yanagihara, 2004 [46]	Unweighted scores of 10 facial expressions, scoring from 0 (complete paralysis) to 40 (normal).	T
Synnybook (SFGS), 2000 [47]	Measures three components of resting symmetry, voluntary movement, and synkinesis, scoring up to 100 (normal).	T
Stennert-Limberg (SLFS), 2012 [48]	Sum of scores of the normal side in rest in four face regions, with scores of motility evaluations while moving.	T
Maximum Static Response Array (MSRA), 1994 [49]	The measurement of displacement from a standard model during facial expressions.	CV
Automated Face Image Analysis (AFIA), 2007 [50]	Tracking the movements of the lips.	CV
Jane and Tomas, 2010 [51]	Implementing an HB scoring system to measure symmetric smile.	CV
Glasgow Facial Palsy Scale, 2012 [52]	HB system combined with regional grades.	CV
CEM Algorithm, 2012 [53]	Mouth parameter analysis using distances from the center of the nose to the edges of the mouth.	CV

includes information regarding the most popular traditional and computer-based grading systems for facial paralysis evaluation.

The House and Brackmann (HB) facial paralysis scale is ranked first among the most commonly used grading scales in the literature. It is used for the evaluation of facial palsy at the nerve trunk and consists of five impairment grades (Grades II to VI): mild dysfunction, moderate dysfunction, moderately severe dysfunction, severe dysfunction, and total paralysis. In this scale, one more grade is assigned to normal function (Grade I). Details of the HB scale are included in

Table 3. HB facial paralysis grading scale.

HB Scale	Information
I—normal	Normal facial function.
II—mild dysfunction	Overall: small weakness noticed on close inspection. At rest: normal symmetry. In motion: moderate-to-good function in forehead, complete eye closure with no effort, and slight mouth asymmetry.
III—moderate dysfunction	Overall: clear non-disfiguring variation between the two sides. At rest: normal symmetry. In motion: slight-to-moderate movement in forehead, complete eye closure with effort, and weak mouth with utmost effort.
IV—moderately severe dysfunction	Overall: clear weakness and asymmetry. At rest: normal symmetry and tone. In motion: none in forehead, incomplete eyes closure, and mouth asymmetry with utmost effort.
V—severe dysfunction	Overall: hardly noticeable motion. At rest: asymmetry. In motion: none in forehead, incomplete eye closure, and minor mouth movement.
VI—total paralysis	No movement

.

In general, several grading methods used to evaluate the severity of facial paralysis have been introduced, each one with its own respected use case. The landmark point approach is the most used method based on the measurement of the distance of facial landmarks. The distance is compared and evaluated based on the used system. This method is time consuming and probably less accurate than computer-based approaches, since accuracy depends on the extracted data.

Computer vision methods, aided by machine/deep learning algorithms, have been employed to overcome the aforementioned problems and achieve better evaluation techniques. However, computer vision-based approaches rely heavily on the size and diversity of the existing datasets. Due to personal information restrictions around those topics, there are lots of restrictions on public dataset usage. Once the dataset is acquired, computer vision methods extract facial descriptors to build a model so that they can generalize on test data. Predictions emerge after these models have been put into classifiers, and the output can be either static (classification) or continuous (regression) depending on the solution that each method provides.

3. Facial Asymmetry Indices

Different image modalities can be employed to evaluate facial palsy. As already mentioned, the most popular approach is based on digital images and videos; however, alternatives able to provide extended information have also been introduced. The recent advances in image capturing devices, as well as their reduced costs and ease of acquisition, combined with recent advancements in image analysis algorithms, have greatly assisted computer vision-based research.

Digital imaging is widely used to assess facial morphology from 2D images. One limitation, though, is its inability to include depth measurements and, consequently, provide a more detailed anatomic facial morphology. Stereo-photogrammetry is used as an alternative to provide 3D facial surface images with accurately measured volumetric distances. Video sequences are also used for palsy detection, since they can capture temporal variation and movements, revealing motion details not perceived in static images. Yet, the nature of video imaging data remains very similar to that of digital imaging data. It should be noted here that traditional facial palsy diagnostic techniques also include magnetic resonance tomography (MRI) and computed tomography (CT) images; however, such datasets are associated with high costs. Therefore, they are scarce and not considered here. In what follows, the mathematical modeling of asymmetry indices in different image modalities is reviewed.

3.1. Asymmetry Indices on 2D Images

A face is a (relatively) symmetric structure of its right side with respect to its left side [54]. It should be noted that only 2% of faces in the general population are really symmetric. The calculation of an asymmetry index in 2D images is mainly aided by reference lines on faces, which can be either horizontal, vertical, or centers of bilateral points. Several approaches have been developed and are analyzed in the following section, while approaches that do not use any reference lines or angles are also reported. It should be noted that distances are measured in either millimeters or pixels. In general, the higher the asymmetry index, the bigger the asymmetry.

3.1.1. Distance Symmetry from a Vertical Reference Line

In this approach, the distance from the bilateral points to the reference median sagittal line is measured (Figure 3a). More specifically, firstly, the vertical reference line is defined, and then the height and distance differences from the bilateral points to the line are measured. Based on this approach, the asymmetry index (AI) [55] for a pair of bilateral facial points is calculated from the following equation:

$$\mathrm{AI}=\left|\frac{{\mathrm{d}}_{\mathrm{R}}-{\mathrm{d}}_{\mathrm{L}}}{{\mathrm{d}}_{\mathrm{R}}+{\mathrm{d}}_{\mathrm{L}}}\right|$$

(1)

where ${\mathrm{d}}_{\mathrm{R}}$ and ${\mathrm{d}}_{\mathrm{L}}$ are the distances from the vertical reference line for each side of the face, i.e., right and left, respectively.

Figure 3. Measuring asymmetries in 2D images by using distances: (a) from a vertical reference line; (b) from a horizontal reference line.

Figure 3. Measuring asymmetries in 2D images by using distances: (a) from a vertical reference line; (b) from a horizontal reference line.

Based on Equation (1), an overall landmark-specific asymmetry index (${\mathrm{AI}}_{overall}$) for an l-number of landmark points can be defined as the average of l Ais as follows:

$${\mathrm{AI}}_{overall}=\frac{1}{l}{\displaystyle \sum }_{\mathrm{i}=1}^l\left|\frac{{\mathrm{d}}_{\mathrm{Ri}}-{\mathrm{d}}_{\mathrm{Li}}}{{\mathrm{d}}_{\mathrm{Ri}}+{\mathrm{d}}_{\mathrm{Li}}}\right|$$

(2)

An advantage of AI is that it avoids the overweighing of big distances, since it is a difference-sum ratio, ranging from zero for perfect symmetry to an ever-increasing value for increased asymmetries. A disadvantage is that since it is an absolute number, it does not denote which face point is out of symmetry when multiple landmark points are considered, as well as the face side and the direction in which the points divert from absolute symmetry.

3.1.2. Distance Symmetry from a Horizontal Reference Line

This approach considers a horizontal reference line defined by bilateral points that are hardly affected by palsy or other facial asymmetries, such as the pupils, and calculates the distance difference of other bilateral points from the reference line [56]. Distances from vertical and horizontal lines can be used in a complementary manner and combined [57]. The latter approach is illustrated in Figure 3b, with the sagittal line and the bipupillary forming the vertical y- and horizontal x-axis, respectively.

While a symmetry index, AI, has solely been defined for the vertical reference line, there is no calculation of a respective index for the horizontal line. However, their combination is studied in [57], and a symmetry index, namely z-score, is introduced; firstly, for all pairs of bilateral points, an asymmetry value is defined as the addition of their horizontal x-coordinates or the absolute difference of their vertical y-coordinates. For distinct landmarks on a reference line, only one asymmetry can be defined as the absolute value of the corresponding coordinate.

More specifically, for bilateral points, both horizontal y- and vertical x-symmetry is considered, being defined as follows:

$$\mathrm{Horizontal} y-\mathrm{symmetry}=\left|c_y-c_{y\_b}\right|$$

(3)

$$\mathrm{Vertical} x-\mathrm{symmetry}=\left|c_x+c_{x\_b}\right|$$

(4)

where c_y is the y-coordinate of a landmark, c_{y_b} is the y-coordinate of its bilateral landmark, c_x is the x-coordinate of a landmark, and c_{x_b} is the x-coordinate of its bilateral landmark. For single landmark points located on reference lines, only the vertical x-symmetry value is calculated as follows:

$$\mathrm{Single} \mathrm{point}\_\mathrm{Vertical} x-\mathrm{symmetry}=\left|c_x\right|$$

(5)

Finally, a symmetry value of z-score is defined for each point of one face from a set of n faces as the difference between each point’s symmetry and the mean symmetry of n faces divided by their standard deviation:

$$\mathrm{Asymmetry} \mathrm{index}\_\mathrm{z}-\mathrm{score}=\frac{1}{n}{\displaystyle \sum }_{i=1}^n\frac{x_{\mathrm{i}}-\overline{x_{\mathrm{l}}}}{{\mathsf{\sigma }}_{\mathrm{i}}}$$

(6)

where x_i is the value of symmetry for a single point or a pair of bilateral points for one direction estimated for one face, x_l is the mean of the same value for all n faces, and σ_i is their standard deviation. Overall symmetries, including vertical and horizontal symmetries for pairs of points or a single point, can be computed by calculating the average z-score for all points.

The advantage of using the z-score asymmetry index is that it does not signify an absolute metric, but it considers an orientation of asymmetries, meaning that a near-zero value indicates an average symmetry with a group of faces, while negative and positive valued indicate high symmetry and asymmetry with respect to the average, respectively. A disadvantage of z-score asymmetry index is that it is a relative metric, calculated with respect to a number of faces.

3.1.3. Distance Symmetry without a Reference Line

In this approach, i sets of bilateral points are connected with a line, and the center m_i of each line (Figure 4a) is defined as follows:

$$m_i=\left|\frac{x_{\mathrm{Ri}}-x_{\mathrm{Li}}}{2}\right|$$

(7)

where x_Ri and x_Li are the x-coordinates of each point in the right and left part of the face, respectively. Then, the horizontal distances between these N centers are calculated. The sum of all the absolute values of the aforementioned distances between these N centers is defined as a facial asymmetry index, namely the overall horizontal facial asymmetry (FA_hor) index, as follows:

$${\mathrm{FA}}_{\mathrm{hor}}={\displaystyle \sum }_{\begin{array}{c}\mathrm{\iota },\mathrm{j}=1,..,N\\ \mathrm{j}<\mathrm{i}\end{array}}\left|m_{\mathrm{i}}-m_{\mathrm{j}}\right|$$

(8)

The vertical facial asymmetry index (FA_ver) can be calculated in the same way, that is by measuring and summing the vertical deviation of landmark points, i.e., the sum of the differences in the horizontal locations of each facial landmark [58].

An index of central facial asymmetry (FA_cen or CFA) is also defined as the sum of Equation (8) between neighboring centers. Thus, many FA_cen can be derived for a single face, allowing the local investigation of symmetries on a face. Given a center of xϵX, a neighbor of x can be all centers w in X having a certain distance r from x as follows:

$${\mathrm{FA}}_{\mathrm{cen}}={\displaystyle \sum }_{\{\mathrm{w}\in \mathrm{X}|\mathrm{d}\left(\mathrm{x},\mathrm{w}\right)<\mathrm{r}\}}\left|m_{\mathrm{i}}-m_{\mathrm{j}}\right|$$

(9)

3.1.4. Angle Symmetry

In this approach, horizontal lines connecting bilateral landmark points are used, and the angles between them are measured [59]. Perfect symmetry would imply parallel lines. The angle θ between two non-parallel lines with inclinations inc₁ and inc₂ is defined as follows:

$$\tan \theta =\left|\frac{{\mathrm{inc}}_1-{\mathrm{inc}}_2}{1+{\mathrm{inc}}_1{\mathrm{inc}}_2}\right|$$

(10)

Vertical lines are also used to evaluate mid-face asymmetries, as well as the two basic auxiliary reference lines, i.e., median sagittal and bipupillary line [60]. In the case depicted in Figure 4b, the angle formed between the connecting line of the two midpoints, which is nearly vertical, and the vertical reference line is calculated using Equation (10). Popularly used angles extend from the middle of the bipuppilary line to the corners of the mouth.

The asymmetry index based on angles (AI_ang) is calculated as the sum of all g angles, including both horizontal and vertical cases. In perfect symmetry, the latter sum is equal to zero, as determined using the following equation:

$${\mathrm{AI}}_{\mathrm{ang}}={\displaystyle \sum }_{\mathrm{i}=1}^g{\theta }_{\mathrm{i}}$$

(11)

3.2. Asymmetry Indices on 3D Images

Recently, the diagnosis and evaluation of facial deformities has also been determined using 3D imaging. The primary advantage of 3D imaging is the capability to view faces from different angles and accurately measure deformations. For this reason, a 3D facial coordinate system is established [61]. The coordinate system is determined using three planes: the yz-vertical plane defined by the median sagittal line, the xy-horizontal plane that passes from the nasion, and the xz-plane passing from the nasion; thus, the nasion is considered to be the center of the coordinate system, as illustrated in Figure 5a. Facial landmarks are determined to be the same as those in 2D images, with each one having three coordinates (x,y,z). The nasion (n) is, therefore, located in coordinates (0,0,0). Then, two different approaches can be adapted to measure asymmetry: one is based on distances, and the other is based on angles.

3.2.1. Distance Asymmetry

In order to measure a facial asymmetry index (FA_3D), the distance from each landmark to the three planes is measured with respect to both face sides, i.e., left (L) and right (R), as follows:

$${\mathrm{FA}}_{3\mathrm{D}}=\sqrt{{({\mathrm{dx}}_{\mathrm{L}-}{\mathrm{dx}}_{\mathrm{R}})}^2+{({\mathrm{dy}}_{\mathrm{L}-}{\mathrm{dy}}_{\mathrm{R}})}^2+{({\mathrm{dz}}_{\mathrm{L}-}{\mathrm{dz}}_{\mathrm{R}})}^2}$$

(12)

where dx, dy, and dz are the distances of a landmark point from planes yz, xz, and xy, respectively.

3.2.2. Angle Asymmetry

Angle asymmetry can be defined using the cosine similarity of the landmark pair (left and right) vector and the normal vector of the mid-sagittal plane [62]. Considering that $\overrightarrow{n}$ is the normal vector of the midsagittal plane and $\overrightarrow{p}$ is the vector of the $p_{\mathrm{i}}$, the i-th facial landmark pair (right (R) and left (L) side) defined as follows:

$$\overrightarrow{p_{\mathrm{i}}}=\left(\left(p_{\mathrm{Rx},\mathrm{i}}-p_{\mathrm{Lx},\mathrm{i}}\right), \left(p_{\mathrm{Ry},\mathrm{i}}-p_{\mathrm{Ly},\mathrm{i}}\right), \left(p_{\mathrm{Rz},\mathrm{i}}-p_{\mathrm{Lz},\mathrm{i}}\right)\right)$$

(13)

then the angle asymmetry index (AI_ang3D) can be defined using the cosine similarity according to the following equation:

$${\mathrm{AI}}_{\mathrm{ang}3\mathrm{D}}=\cos \theta =\frac{\overrightarrow{n} \overrightarrow{p_{\mathrm{i}}}}{\Vert \overrightarrow{n}\Vert \Vert \overrightarrow{p_{\mathrm{i}}}\Vert }$$

(14)

Values for AI_ang3D near to 1 indicate symmetry, while values close to 0 indicate asymmetry. Figure 5b illustrates the calculation of angle asymmetry.

3.3. Asymmetry Indices on Video

Compared to previous methods, videos can also capture facial movements, clearly pointing out the use of each facial muscle. Facial landmark pairs are defined, and the amount of movement of each pair is evaluated with respect to each facial side (left and right) (Figure 6). For evaluating the amount of movement of each landmark pair, several facial expressions, i.e., smile, anger, etc., are captured and compared to neutral expression. When changing from neutral to an expression, landmark pairs on both sides of the face should move uniformly in cases of perfect symmetry.

An additional advantage of the use of videos and the calculation of the amount of movement is the ability to quantify the degree of improvement in palsy patients over time. We consider n_i to be the neutral expression and S_i to be the smile expression for the i-th facial landmark. The amount of movement of the i-th landmark pair, i.e., left and right (m_L,i and m_R,i), is calculated as follows:

$${\mathrm{m}}_{\mathrm{L},\mathrm{i}}=\sqrt{{({\mathrm{s}}_{\mathrm{Lx},\mathrm{i}}-{\mathrm{n}}_{\mathrm{Lx},\mathrm{i}})}^2+{({\mathrm{s}}_{\mathrm{Ly},\mathrm{i}}-{\mathrm{n}}_{\mathrm{Ly},\mathrm{i}})}^2+{({\mathrm{s}}_{\mathrm{Lz},\mathrm{i}}-{\mathrm{n}}_{\mathrm{Lz},\mathrm{i}})}^2}$$

(15)

$${\mathrm{m}}_{\mathrm{R},\mathrm{i}}=\sqrt{{({\mathrm{s}}_{\mathrm{Rx},\mathrm{i}}-{\mathrm{n}}_{\mathrm{Rx},\mathrm{i}})}^2+{({\mathrm{s}}_{\mathrm{Ry},\mathrm{i}}-{\mathrm{n}}_{\mathrm{Ry},\mathrm{i}})}^2+{({\mathrm{s}}_{\mathrm{Rz},\mathrm{i}}-{\mathrm{n}}_{\mathrm{Rz},\mathrm{i}})}^2}$$

(16)

The asymmetry movement index (AI_move,i) for the i-th landmark pair is then defined as follows:

$${\mathrm{AI}}_{\mathrm{move}, \mathrm{i}}=\left|{\mathrm{m}}_{\mathrm{L},\mathrm{i}}-{\mathrm{m}}_{\mathrm{R},\mathrm{i}}\right|$$

(17)

4. Machine Learning-Based Facial Palsy Detection and Evaluation

Traditional mathematical modeling was complemented by the advances in machine learning methods. In this section, the machine learning and deep learning approaches detailed in the bibliography for facial palsy detection and evaluation are reviewed to extract comparative information and draw a conclusion regarding the most efficient approaches.

Traditional machine learning methods are based on encoding facial palsy with facial asymmetry-related mathematical features. A portable automatic diagnosis system based on a smartphone application for classifying subjects to healthy or palsy patients was presented by Kim et al. [63]. Facial landmarks were extracted, and an asymmetry index was computed. Classification was implemented using Linear Discriminant Analysis (LDA) combined with Support Vector Machines (SVMs), resulting in 88.9% classification accuracy. Wang et al. [64] used Active Shape Models (ASMs) to locate facial landmarks, dividing the face in eight regions and Local Binary Patterns (LBPs) used to extract descriptors for recognizing patterns of facial movements in these regions, reaching the highest recognition rate of up to 93.33%. In [65], He et al. extracted features based on LBPs in the spatial–temporal domain in both facial regions and validated their method using biomedical videos, reporting an overall accuracy of up to 94% for the HB grading. In [51], the authors automatically measure the ability of palsy patients to smile using Active Appearance Models (AAMs) for feature extraction and facial expression synthesis, providing an average accuracy of 87%. McGrenary et al. [66] quantified facial asymmetry in videos using an artificial neural network (ANN).

Early research into facial asymmetry analysis was also studied by Quan et al. [67], who presented a method for automatically detecting and quantifying facial dysfunctions based on 3D face scans. The authors extracted a number of feature points that enabled the segmentation of faces in local regions, enabling specific asymmetry evaluation for regions of interest rather than the entire face. Gaber et al. [68] proposed an evaluation system for seven palsy categories based on an ensemble learning SVM classifier, reporting an accuracy of 96.8%. The authors proved that their proposed classifier was robust and stable, even for different training and testing samples. Zhuang et al. [69] implemented a performance evaluation between various feature extraction techniques and concluded that 2D static images with Histogram of Oriented Gradients (HOG) features tend to be more accurate. The authors proposed a framework in which landmark and HOG features were extracted, Principal Component Analysis (PCA) was employed separately to the features, and the results were used as inputs to an SVM classifier for classification into three classes, demonstrating performance of up to 92.2% for the entire face. The same research group, as shown in [70], demonstrated a video classification detection tool, namely the Facial Deficit Identification Tool for Videos (F-DIT-V), exploiting HOG features to find a 92.9% classification accuracy. Arora et al. [71] tested an SVM and a Logistic Regressor on generated facial landmark features, achieving 76.87% average accuracy with SVM. In [72], laser speckle contrast imaging was employed by Jiang et al. to monitor the facial blood flow of palsy patients. Then, faces were segmented into regions based on blood distribution features, and three HB score classifiers were tested for their classification performance: a neural network (NN), an SVM, and a k-NN, achieving an accuracy of up to 97.14%. A set of four classifiers (multi-layer perceptron (MLP), SVM, k-NN, multinomial logistic regression (MNLR)) was also comparatively tested in [73]. The authors explored regional information, extracting handcrafted features only in certain face areas of interest. Experimental results reported up to 95.61% correct facial palsy detection and 95.58% correct facial palsy assessment in three categories (healthy, slight palsy, and strong palsy).

All previous methods are based on hand-crafted features. Deep learning methods can automatically learn discriminative feature from the data, without the need to compute them in advance. Deep learning models have accomplished state-of-the-art performances in the field of medical imaging [74]. Based on the above, most of the recent works in vision-based facial palsy detection and evaluation employ deep features. Storey and Jiang [75] presented a unified multitask convolutional neural network (CNN) for the simultaneous object proposal, detection and asymmetry analysis of faces. Sajid et al. [76] introduced a CNN to classify palsy into five scales, resulting in a 92.6% average classification accuracy. Xia et al. [5] suggested a deep neural network (DNN) to detect facial landmarks in palsy. Hsu et al. [33] proposed a deep hierarchical network (DHN) to quantify facial palsy, including a YOLO2 detector for face detection, a fused neural architecture (line segment network—LSN) to detect facial landmarks, and an object detector, similar to Darknet, to locate palsy regions. Preliminary results of the same method were published in [77]. Guo et al. [78] investigated the unilateral peripheral facial paralysis classification using GoogLeNet, reaching a classification accuracy of up to 91.25% for predicting the HB degree.

Storey et al. [79] implemented a facial grading system from video sequences based on a 3D CNN model using ResNet as the backbone, reporting a palsy classification accuracy of up to 82%. Barrios Dell’Olio and Sra [80] proposed a CNN for detecting muscle activation and intensity in the users of their mobile augmented reality mirror therapy system. In [81], Tan et al. introduced a facial palsy assessment method, including a facial landmark detector, a feature extractor based on EfficientNet backbone and semi-supervised extreme learning to classify features, reporting an 85.5% accuracy. Abayomi-Alli et al. [82] trained a SqueezeNet network with augmented images and used the activations from the final convolutional layer as features to train a multiclass error-corrected output code SVM (ECOC-SVM) classifier, reporting an up to 99.34% mean classification accuracy. In [83], computed tomography (CT) images were used to train two geometric deep learning models, namely PointNet++ and PointCNN, for the facial part segmentation of healthy and palsy patients for facial monitoring and rehabilitation. Umirzakova et al. [84] suggested a light deep learning model for analyzing facial symmetry, using a foreground attention block for enhanced local feature extraction and a depth-map estimator to provide more accurate segmentation results.

Table 4. Methodologies for facial palsy (FP) detection.

Ref.	Objective	Methodology	Dataset	Performance	Conclusions/Limitations
[63]	Smartphone-based FP diagnostic system (five FP grades)	Linear regression model for facial landmark detection and SVM with linear kernel for classification	Private dataset of 36 subjects (23 noral−13 palsy patients) performing 3 motions	88.9% classification accuracy	Reproducibility under different experimental conditions, as well as repeatability of measurements over a period of time, were not implemented
[64]	Facial movement patterns recognition for FP (2 classes, i.e., normal and asymmetric)	Active Shape Models plus Local Binary Patterns (ASMLBP) for feature extraction and SVM for classification	Private dataset of 570 images of 57 subjects with 5 facial movements	Up to 93.33% recognition rate	High robustness and accuracy
[65]	Quantitative evaluation of FP (HB scale)	Multiresolution extension of uniform LBP and SVM for FP evaluation	Private dataset of 197 subject videos with 5 facial movements	~94% classification accuracy	Sensitive to out-plane facial movements, with significant natural bilateral asymmetry
[51]	Facial landmarks tracking and feedback for FP assessment (HB scale)	Active Appearance Models (AAMs) for facial expression synthesis	Private dataset of frontal images of neutral and smile expressions from 5 healthy subjects	87% accuracy	Preliminary results to demonstrate a proof of concept
[66]	FP assessment	ANN	Private dataset of 43 videos from 14 subjects	1.6% average MSE	Pilot study; general results follow the opinions of experts
[67]	Facial asymmetry measurement	Measuring 3D asymmetry index	Three-dimensional dynamic scans from Hi4D-ADSIP database (stroke)	-	Extraction of 3D feature points, as well as potential for detecting facial dysfunctions
[68]	FP classification of real-time facial animation units (seven FP grades)	Ensemble learning SVM classifier	Private dataset of 375 records from 13 patients and 1650 records from 50 control subjects	96.8% accuracy 88.9% sensitivity 99% specificity	Data augmentation for the imbalanced dataset issues
[69]	FP quantification	Combination of landmarks and intensity HoG-based features and a CNN model for classification	Private dataset of 125 images of left facial weakness, 126 images of right facial weakness, and 186 images of normal subjects	Up to 94.5% accuracy	The combination of landmarks and HoG intensity features produced the best, when compared to either landmarks or intensity features separately
[70]	FP classification (three classes)	HOG features and a voting classifier	Private dataset of 37 videos of left weakness, 38 of right and 60 of normal subjects	92.9% accuracy 93.6% precision 92.8% recall 94.2% specificity	Comparison with other methods revealed the reliability of HOG features
[71]	Facial metric calculation of face sides symmetry	Facial landmark features with cascade regression and SVM	Stroke faces dataset of 1024 images and 1081 images of healthy faces	76.87% accuracy	Machine learning problem-specific models can lead to improved performances
[72]	FP assessment (HB scale)	Laser speckle contrast imaging and NN classifiers	Private dataset of 80 FP patients	97.14% accuracy	Outperforms the state-of-the-art systems and other classifiers
[73]	FP classification (three classes)	Regional handcrafted features and four classifiers (MLP, SVM, k-NN, MNLR)	YouTube Facial Palsy (YFP) database	Up to 95.58% correct classification	Severity is higher classified in eyes and mouth regions
[75]	Face symmetry analysis (symmetrical-asymmetrical)	Unified multi-task CNN	AFLW database to fine tune the model and extended Cohn–Kanade (CK+) to learn face symmetry (18,786 images in total)	-	Lack of fully annotated training set, as well as the need for labeling or a synthesized training set
[76]	FP classification (five grades)	CNN (VGG-16)	Dataset from online sources augmented to 2000 images	92.6% accuracy 92.91% precision 93.14% sensitivity 93% F1 Score	Deep features combined with data augmentation can lead to robust classification
[5]	FP classification	FCN	AFLFP dataset	Normalized mean error (NME): 11.5% Mean average: 2.3% standard deviation	Comparative results indicate that deep learning methods are, overall, better than machine learning methods
[33]	Quantitative analysis of FP	Deep Hierarchical Network	YouTube Facial Palsy (YFP) database	5.83% NME	Line segment learning leads to an important part of deep features being able to improve the accuracy of facial landmark and palsy region detection
[77]	Quantitative analysis of FP	Hierarchical Detection Network	YouTube Facial Palsy (YFP) database	Up to 93% precision and 88% recall	Efficient for video-to-description diagnosis
[78]	Unilateral peripheral FP assessment (HB scale)	Deep CNN	Private dataset of 720 labeled images of four facial expressions	91.25% classification accuracy	Fine-tuning deep CNNs can learn specific representations from biomedical images
[79]	FP grading	Fully 3D CNN	Private FP dataset of 696 sequences with 17 subjects	82% classification accuracy	Very competent at learning spatio-temporal features
[80]	AR system for FP estimation	Light-Weight Facial Activation Unit model (LW-FAU)	Private dataset from 20 subjects	-	Lack of FP benchmark models and datasets
[81]	FP assessment (six classes)	FNPARCELM-CCNN method	YouTube Facial Palsy (YFP) database	85.5% accuracy	Semi-supervised methods can distinguish different degrees of FP, even with little-labeled data
[82]	FP detection and classification	Deep feature extraction with SqueezeNet and ECOC-SVM classifier	YouTube Facial Palsy (YFP) database	99.34% accuracy	Improvement in FP detection from a small dataset
[83]	Part segmentation	Point-Net++ and PointCNN	CT images of 33 subjects	99.19% accuracy 89.09% IOU	Geometric deep learning can be efficient
[84]	FP asymmetry analysis	Proposed deep architecture	YouTube Facial Palsy (YFP) database	93.8% IOU	Poor with bearded faces due to a lack of such training data images

summarizes basic information from all the aforementioned studies, including the followed methodology, dataset, and performance results. Details regarding the mathematical modeling of machine learning and deep learning classification models can be found in [85,86,87,88,89].

5. Discussion

From the information included in Table 4, useful conclusions can be drawn. The lack of available datasets designated for palsy detection and evaluation is obvious. Most research teams develop their own private sets to test their algorithms. The most used public dataset among the referenced works is the YFP dataset; however, it refers to a limited video dataset. The videos are converted into image sequences; however, low dysfunctions cannot be easily visible from only one image and, thus, a sequence of frames needs to be examined to draw conclusions. Moreover, the dataset is labeled but facial landmark points are not annotated. From Table 4, it can be observed that deep learning methods lead to better performance results compared to machine learning methods or methods relying on hand-crafted features.

It is obvious that the classification of facial palsy remains an important and challenging research topic in healthcare informatics. Representative facial feature learning of the many, diverse data that encode face symmetry mathematical modeling is essential for machine learning methods that depend on previous knowledge to extract features. Deep learning methods do not rely on hand-crafted features. However, it should be noted that most of the related works included in the literature use hand-crafted features instead of deep learning methods for the reason discussed in the following paragraphs. Yet, the selection of the appropriate method is not one-way; instead, it is based on the problem’s diversity, the number of available data, the desired accuracy, and the execution time of the algorithm, i.e., the problem under study.

On one hand, hand-crafted features used in machine learning models are efficient at classifying facial palsy; however, they are dependent on previous knowledge, and, thus, non-highly representative features may deteriorate the accuracy of classification. The major advantage of using landmark features is that they can provide exact measurements of deformed faces by calculating the distance between facial landmark points and comparing both facial sides, which is not feasible via deep learning models. For this reason, landmark-based geometrical feature learning is preferable and extensively used in maxillofacial surgery [90,91]. Moreover, geometrical feature extraction, combined with machine learning models, is less computational complex compared to deep learning approaches. An additional limitation lies in the exhausting identification and labeling of the level of palsy by domain experts, such as experienced neurologists. Yet, the interpretation of the level of palsy could be subjective and, therefore, may differ among different neurologists.

On the other hand, deep learning models are capable of automatically drawing conclusions for discriminative features and, thus, achieving a higher classification accuracy, as seen in Table 4. Computational costs involved in the process due to the great number of parameters related to deep architectures, compared to machine learning methods, remain a barrier. The interpretation of deep learning algorithms, due to their ‘black box’ nature, is an additional challenge. Despite the high reported accuracies of deep learning in medical image applications, their utilization in clinical practices remains limited [5]. The latter issue mainly occurs because deep models require a large set of training data, which is not available in medical cases such as facial palsy, leading to poor generalization and overfitting issues. It should be noted here that among the available public datasets, only one dataset, namely AFLFP, provides annotated facial landmark images of facial palsy patients. Researchers tend to use small and private in-house clinical data for their experimentations, obstructing the comparative evaluation of different algorithms using the same data, which lack sufficient sampling diversity, especially for large-scale applications.

Transfer learning, dropout, and data augmentation are some common solutions to overcome overfitting [76], as indicated in Table 4 in the Conclusions/Limitations column. Currently, a popular solution to enhance available facial palsy public datasets in terms of size and variability is to generate synthetic images. Several techniques are proposed in the literature, either based on image analysis, such as combining the left and right sides of the face from two photographs of the same subject performing different facial expressions to generate a new asymmetrical facial expression [51] or simple image transform approaches [82], as well as using generative adversarial networks (GANs) to automatically augment the training data with diverse face images [76,92]. In the same direction, researchers are also starting to focus on building mathematical models of the dynamics of facial expressions for realistic synthetic faces for the modeling of static images and animations [93].

6. Conclusions

The early and precise diagnosis, quantification, and process of rehabilitation of facial palsy are of great interest for neurologists and data scientists, aiming to meliorate human well-being in everyday life. Recently, facial palsy has tended to attract computer scientists for developing automated solutions as support tools for neurologists, providing a faster and more accurate way of performing facial palsy assessment. Traditional visual inspection and manual assessment of palsy are reinforced through the advancements in machine learning and deep learning algorithms. These advancements can enable the precise localization of face landmark points, consistent calculation of asymmetry indices, and building of a universal grading system for facial palsy.

This work, for the first time, includes all relevant aspects involved in facial palsy detection, ranging from mathematical modeling to deep learning approaches. It provides a complete guide on the mathematical modeling of facial palsy in terms of asymmetry indices in three different image modalities. Moreover, it provides popular facial databases that can be used in relevant research and available grading systems, and it reviews traditional machine learning and deep learning approaches for facial palsy evaluation. It can be concluded that deep learning models could be more accurate at classifying palsy; however, they require increased computations and large training datasets. Efficient landmark extraction and selection techniques, extended research into the feasibility and suitability of appropriate models for the problem under study, and the generation of appropriate labeled datasets, both natural and artificial, remain challenging.