Automatic Facial Palsy Detection—From Mathematical Modeling to Deep Learning
Abstract
:1. Introduction
- Τhis work has collected all the necessary background information related to the detection of facial palsy. Specifically, it includes the following areas:
- a.
- The structure of typical facial assessment systems;
- b.
- Facial landmarks for palsy detection;
- c.
- Facial datasets;
- d.
- Facial palsy grading systems.
- This work aggregately provides the mathematical formulation of all facial palsy asymmetry indices as follows:
- a.
- 2D images;
- b.
- 3D images;
- c.
- Videos.
- This work reviews and compares the following mainstream techniques:
- a.
- Machine learning;
- b.
- Deep learning approaches for facial palsy detection and evaluation.
2. Foundations
2.1. Overview of Typical Facial Palsy Assessment System
2.2. Facial Landmarks
2.3. Facial Datasets
2.4. Facial Palsy Grading Systems
3. Facial Asymmetry Indices
3.1. Asymmetry Indices on 2D Images
3.1.1. Distance Symmetry from a Vertical Reference Line
3.1.2. Distance Symmetry from a Horizontal Reference Line
3.1.3. Distance Symmetry without a Reference Line
3.1.4. Angle Symmetry
3.2. Asymmetry Indices on 3D Images
3.2.1. Distance Asymmetry
3.2.2. Angle Asymmetry
3.3. Asymmetry Indices on Video
4. Machine Learning-Based Facial Palsy Detection and Evaluation
5. Discussion
6. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Van Veen, M.M.; ten Hoope, B.W.T.; Bruins, T.E.; Stewart, R.E.; Werker, P.M.N.; Dijkstra, P.U. Therapists’ Perceptions and Attitudes in Facial Palsy Rehabilitation Therapy: A Mixed Methods Study. Physiother. Theory Pract. 2022, 38, 2062–2072. [Google Scholar] [CrossRef] [PubMed]
- Banita, B.; Tanwar, P. A Tour Toward the Development of Various Techniques for Paralysis Detection Using Image Processing. In Lecture Notes in Computational Vision and Biomechanics; Springer: Berlin/Heidelberg, Germany, 2018; pp. 187–214. [Google Scholar]
- Hotton, M.; Huggons, E.; Hamlet, C.; Shore, D.; Johnson, D.; Norris, J.H.; Kilcoyne, S.; Dalton, L. The Psychosocial Impact of Facial Palsy: A Systematic Review. Br. J. Health Psychol. 2020, 25, 695–727. [Google Scholar] [CrossRef] [PubMed]
- McKernon, S.; House, A.D.; Balmer, C. Facial Palsy: Aetiology, Diagnosis and Management. Dent. Update 2019, 46, 565–572. [Google Scholar] [CrossRef]
- Xia, Y.; Nduka, C.; Yap Kannan, R.; Pescarini, E.; Enrique Berner, J.; Yu, H. AFLFP: A Database With Annotated Facial Landmarks for Facial Palsy. IEEE Trans. Comput. Soc. Syst. 2023, 10, 1975–1985. [Google Scholar] [CrossRef]
- Guo, Z.; Dan, G.; Xiang, J.; Wang, J.; Yang, W.; Ding, H.; Deussen, O.; Zhou, Y. An Unobtrusive Computerized Assessment Framework for Unilateral Peripheral Facial Paralysis. IEEE J. Biomed. Health Inform. 2018, 22, 835–841. [Google Scholar] [CrossRef] [PubMed]
- Demeco, A.; Marotta, N.; Moggio, L.; Pino, I.; Marinaro, C.; Barletta, M.; Petraroli, A.; Palumbo, A.; Ammendolia, A. Quantitative Analysis of Movements in Facial Nerve Palsy with Surface Electromyography and Kinematic Analysis. J. Electromyogr. Kinesiol. 2021, 56, 102485. [Google Scholar] [CrossRef] [PubMed]
- Baude, M.; Hutin, E.; Gracies, J.-M. A Bidimensional System of Facial Movement Analysis Conception and Reliability in Adults. Biomed. Res. Int. 2015, 2015, 1–8. [Google Scholar] [CrossRef] [PubMed]
- Petrides, G.; Clark, J.R.; Low, H.; Lovell, N.; Eviston, T.J. Three-Dimensional Scanners for Soft-Tissue Facial Assessment in Clinical Practice. J. Plast. Reconstr. Aesthetic Surg. 2021, 74, 605–614. [Google Scholar] [CrossRef]
- Azuma, T.; Fuchigami, T.; Nakamura, K.; Kondo, E.; Sato, G.; Kitamura, Y.; Takeda, N. New Method to Evaluate Sequelae of Static Facial Asymmetry in Patients with Facial Palsy Using Three-Dimensional Scanning Analysis. Auris Nasus Larynx 2022, 49, 755–761. [Google Scholar] [CrossRef]
- Amsalam, A.S.; Al-Naji, A.; Yahya Daeef, A.; Chahl, J. Computer Vision System for Facial Palsy Detection. J. Tech. 2023, 5, 44–51. [Google Scholar] [CrossRef]
- Lou, J.; Yu, H.; Wang, F.-Y. A Review on Automated Facial Nerve Function Assessment from Visual Face Capture. IEEE Trans. Neural Syst. Rehabil. Eng. 2020, 28, 488–497. [Google Scholar] [CrossRef]
- Boochoon, K.; Mottaghi, A.; Aziz, A.; Pepper, J.-P. Deep Learning for the Assessment of Facial Nerve Palsy: Opportunities and Challenges. Facial Plast. Surg. 2023, 39, 508–511. [Google Scholar] [CrossRef]
- Meintjes, E.M.; Douglas, T.S.; Martinez, F.; Vaughan, C.L.; Adams, L.P.; Stekhoven, A.; Viljoen, D. A Stereo-Photogrammetric Method to Measure the Facial Dysmorphology of Children in the Diagnosis of Fetal Alcohol Syndrome. Med. Eng. Phys. 2002, 24, 683–689. [Google Scholar] [CrossRef]
- Wachtman, G.S.; Cohn, J.F.; VanSwearingen, J.M.; Manders, E.K. Automated Tracking of Facial Features in Patients with Facial Neuromuscular Dysfunction. Plast. Reconstr. Surg. 2001, 107, 1124–1133. [Google Scholar] [CrossRef] [PubMed]
- Rajpurkar, P.; Chen, E.; Banerjee, O.; Topol, E.J. AI in Health and Medicine. Nat. Med. 2022, 28, 31–38. [Google Scholar] [CrossRef] [PubMed]
- Wen, Z.; Huang, H. The Potential for Artificial Intelligence in Healthcare. J. Commer. Biotechnol. 2023, 27, 217–224. [Google Scholar] [CrossRef]
- Deng, W.; Fang, Y.; Xu, Z.; Hu, J. Facial Landmark Localization by Enhanced Convolutional Neural Network. Neurocomputing 2018, 273, 222–229. [Google Scholar] [CrossRef]
- Tang, X.; Guo, F.; Shen, J.; Du, T. Facial Landmark Detection by Semi-Supervised Deep Learning. Neurocomputing 2018, 297, 22–32. [Google Scholar] [CrossRef]
- Chrysos, G.G.; Antonakos, E.; Snape, P.; Asthana, A.; Zafeiriou, S. A Comprehensive Performance Evaluation of Deformable Face Tracking “In-the-Wild”. Int. J. Comput. Vis. 2018, 126, 198–232. [Google Scholar] [CrossRef]
- Peng, T.; Li, M.; Chen, F.; Xu, Y.; Zhang, D. Learning Efficient Facial Landmark Model for Human Attractiveness Analysis. Pattern Recognit. 2023, 138, 109370. [Google Scholar] [CrossRef]
- Huang, Y.; Huang, H. Stacked Attention Hourglass Network Based Robust Facial Landmark Detection. Neural Netw. 2023, 157, 323–335. [Google Scholar] [CrossRef] [PubMed]
- Bakkialakshmi, V.S.; Sudalaimuthu, T.; Umamaheswari, B. Emo-Spots: Detection and Analysis of Emotional Attributes through Bio-Inspired Facial Landmarks. In Lecture Notes in Electrical Engineering; Springer: Berlin/Heidelberg, Germany, 2023; pp. 103–115. ISBN 9789811981357. [Google Scholar]
- Berlin, N.F.; Berssenbrügge, P.; Runte, C.; Wermker, K.; Jung, S.; Kleinheinz, J.; Dirksen, D. Quantification of Facial Asymmetry by 2D Analysis—A Comparison of Recent Approaches. J. Cranio-Maxillofac. Surg. 2014, 42, 265–271. [Google Scholar] [CrossRef] [PubMed]
- Sagonas, C.; Tzimiropoulos, G.; Zafeiriou, S.; Pantic, M. 300 Faces In-the-Wild Challenge: The First Facial Landmark Localization Challenge. In Proceedings of the 2013 IEEE International Conference on Computer Vision Workshops, Sydney, Australia, 1–8 December 2013; pp. 397–403. [Google Scholar]
- Belhumeur, P.N.; Jacobs, D.W.; Kriegman, D.J.; Kumar, N. Localizing Parts of Faces Using a Consensus of Exemplars. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 2930–2940. [Google Scholar] [CrossRef] [PubMed]
- Le, V.; Brandt, J.; Lin, Z.; Bourdev, L.; Huang, T.S. Interactive Facial Feature Localization. In Lecture Notes in Computer Science; Including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics; Springer: Berlin/Heidelberg, Germany, 2012; pp. 679–692. ISBN 9783642337116. [Google Scholar]
- Zhu, X.; Ramanan, D. Face Detection, Pose Estimation, and Landmark Localization in the Wild. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, 16–21 June 2012; pp. 2879–2886. [Google Scholar]
- Kostinger, M.; Wohlhart, P.; Roth, P.M.; Bischof, H. Annotated Facial Landmarks in the Wild: A Large-Scale, Real-World Database for Facial Landmark Localization. In Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain, 6–13 November 2011; pp. 2144–2151. [Google Scholar]
- Messer, K.; Matas, J.; Kittler, J.; Luettin, J.; Maitre, G. XM2VTSDB: The Extended M2VTS Database. Proc. Second Int. Conf. Audio Video-Based Biom. Pers. Authentication 1999, 964, 965–966. [Google Scholar]
- Gross, R.; Matthews, I.; Cohn, J.; Kanade, T.; Baker, S. Multi-PIE. Image Vis. Comput. 2010, 28, 807–813. [Google Scholar] [CrossRef] [PubMed]
- Matuszewski, B.J.; Quan, W.; Shark, L.-K.; McLoughlin, A.S.; Lightbody, C.E.; Emsley, H.C.A.; Watkins, C.L. Hi4D-ADSIP 3-D Dynamic Facial Articulation Database. Image Vis. Comput. 2012, 30, 713–727. [Google Scholar] [CrossRef]
- Hsu, G.-S.J.; Kang, J.-H.; Huang, W.-F. Deep Hierarchical Network With Line Segment Learning for Quantitative Analysis of Facial Palsy. IEEE Access 2019, 7, 4833–4842. [Google Scholar] [CrossRef]
- Greene, J.J.; Guarin, D.L.; Tavares, J.; Fortier, E.; Robinson, M.; Dusseldorp, J.; Quatela, O.; Jowett, N.; Hadlock, T. The Spectrum of Facial Palsy: The MEEI Facial Palsy Photo and Video Standard Set. Laryngoscope 2020, 130, 32–37. [Google Scholar] [CrossRef] [PubMed]
- Samsudin, W.S.W.; Sundaraj, K. Evaluation and Grading Systems of Facial Paralysis for Facial Rehabilitation. J. Phys. Ther. Sci. 2013, 25, 515–519. [Google Scholar] [CrossRef]
- Botman, J.W.M.; Jongkees, L.B.W. The Result of Intratemporal Treatment of Facial Palsy. ORL 1955, 17, 80–100. [Google Scholar] [CrossRef]
- Peitersen, E. Bell’s Palsy: The Spontaneous Course of 2500 Peripheral Facial Nerve Palsies of Different Etiologies. Acta Otolaryngol. 2002, 122, 4–30. [Google Scholar] [CrossRef]
- Smith, I.M.; Murray, J.A.M.; Cull, R.E.; Slattery, J. A Comparison of Facial Grading Systems. Clin. Otolaryngol. 1992, 17, 303–307. [Google Scholar] [CrossRef] [PubMed]
- Adour, K.K.; Swanson, P.J. Facial Paralysis in 403 Consecutive Patients: Emphasis on Treatment Response in Patients with Bell’s Palsy. Trans. Am. Acad. Ophthalmol. Otolaryngol. 1971, 75, 1284–1301. [Google Scholar] [PubMed]
- Janssen, F.P. Over de Postoperatieve Facialis Verlamming. Ph.D. Thesis, University of Amsterdam, Amsterdam, The Netherlands, Verlag nicht ermittelbar. 1963. [Google Scholar]
- Yanagihara, N. Grading of Facial Palsy. In Proceedings of the Third International Symposium on Facial Nerve Surgery, Zurich, Switzerland, 9–12 August 1976; pp. 533–535. [Google Scholar]
- Stennert, E. Facial Nerve Paralysis Scoring System. In Proceedings of the Third International Symposium on Facial Nerve Surgery, Zurich, Switzerland, 9–12 August 1976; pp. 543–547. [Google Scholar]
- House, J.W.; Brackmann, D.E. Facial Nerve Grading System. Otolaryngol. Neck Surg. 1985, 93, 146–147. [Google Scholar] [CrossRef]
- Burres, S.; Fisch, U. The Comparison of Facial Grading Systems. Arch. Otolaryngol. Head Neck Surg. 1986, 112, 755–758. [Google Scholar] [CrossRef] [PubMed]
- Murty, G.E.; Diver, J.P.; Kelly, P.J.; O’Donoghue, G.M.; Bradley, P.J. The Nottingham System: Objective Assessment of Facial Nerve Function in the Clinic. Otolaryngol. Neck Surg. 1994, 110, 156–161. [Google Scholar] [CrossRef]
- Berg, T.; Jonsson, L.; Engström, M. Agreement between the Sunnybrook, House-Brackmann, and Yanagihara Facial Nerve Grading Systems in Bell’s Palsy. Otol. Neurotol. 2004, 25, 1020–1026. [Google Scholar] [CrossRef] [PubMed]
- Satoh, Y.; Kanzaki, J.; Yoshihara, S. A Comparison and Conversion Table of ‘the House–Brackmann Facial Nerve Grading System’ and ‘the Yanagihara Grading System’. Auris Nasus Larynx 2000, 27, 207–212. [Google Scholar] [CrossRef]
- Kecskés, G.; Jóri, J.; O’Reilly, B. Current Diagnostic, Pharmaceutics and Reconstructive Surgical Methods in the Management of Facial Nerve Palsy. Ph.D. Thesis, University of Szeged, Szeged, Hungary, 2012. [Google Scholar]
- Johnson, P.; Brown, H.; Kuzon, W. Simultaneous Quantification of Facial Movements: The Maximal Static Response Assays of Facial Nerve Function. Ann. Plast. Surg. 1994, 32, 171–179. [Google Scholar] [CrossRef]
- Rogers, C.R.; Schmidt, K.L.; VanSwearingen, J.M.; Cohn, J.F.; Wachtman, G.S.; Manders, E.K.; Deleyiannis, F.W.B. Automated Facial Image Analysis. Ann. Plast. Surg. 2007, 58, 39–47. [Google Scholar] [CrossRef]
- Delannoy, J.R.; Ward, T.E. A Preliminary Investigation into the Use of Machine Vision Techniques for Automating Facial Paralysis Rehabilitation Therapy. In Proceedings of the IET Irish Signals and Systems Conference (ISSC 2010), Cork, Ireland, 23–24 June 2010; pp. 228–232. [Google Scholar]
- Kecskés, G.; Jóri, J.; O’Reilly, B.F.; Viharos, L.; Rovó, L. Clinical Assessment of a New Computerised Objective Method of Measuring Facial Palsy. Clin. Otolaryngol. 2011, 36, 313–319. [Google Scholar] [CrossRef]
- Anguraj, K.; Kandiban, R.; Jayakumar, K.S. Facial Paralysis Diseases Level Detection Using CEM Algorithm for Clinical Applications. Eur. J. Sci. Res. 2012, 77, 543–548. [Google Scholar]
- Penke, L.; Bates, T.C.; Gow, A.J.; Pattie, A.; Starr, J.M.; Jones, B.C.; Perrett, D.I.; Deary, I.J. Symmetric Faces Are a Sign of Successful Cognitive Aging. Evol. Hum. Behav. 2009, 30, 429–437. [Google Scholar] [CrossRef]
- Nakamura, T.; Okamoto, K.; Maruyama, T. Facial Asymmetry in Patients with Cervicobrachial Pain and Headache. J. Oral Rehabil. 2008, 28, 1009–1014. [Google Scholar] [CrossRef]
- Gosla-Reddy, S.; Nagy, K.; Mommaerts, M.Y.; Reddy, R.R.; Bronkhorst, E.M.; Prasad, R.; Kuijpers-Jagtman, A.M.; Bergé, S.J. Primary Septoplasty in the Repair of Unilateral Complete Cleft Lip and Palate. Plast. Reconstr. Surg. 2011, 127, 761–767. [Google Scholar] [CrossRef] [PubMed]
- Bashour, M. An Objective System for Measuring Facial Attractiveness. Plast. Reconstr. Surg. 2006, 118, 757–774. [Google Scholar] [CrossRef]
- Scheib, J.E.; Gangestad, S.W.; Thornhill, R. Facial Attractiveness, Symmetry and Cues of Good Genes. Proc. R. Soc. London. Ser. B Biol. Sci. 1999, 266, 1913–1917. [Google Scholar] [CrossRef]
- Yamashita, Y.; Nakamura, Y.; Shimada, T.; Nomura, Y.; Hirashita, A. Asymmetry of the Lips of Orthognathic Surgery Patients. Am. J. Orthod. Dentofac. Orthop. 2009, 136, 559–563. [Google Scholar] [CrossRef]
- Yu, C.-C.; Bergeron, L.; Lin, C.-H.; Chu, Y.-M.; Chen, Y.-R. Single-Splint Technique in Orthognathic Surgery: Intraoperative Checkpoints to Control Facial Symmetry. Plast. Reconstr. Surg. 2009, 124, 879–886. [Google Scholar] [CrossRef] [PubMed]
- Huang, C.S.; Liu, X.Q.; Chen, Y.R. Facial Asymmetry Index in Normal Young Adults. Orthod. Craniofac. Res. 2013, 16, 97–104. [Google Scholar] [CrossRef]
- Kim, J.; Jeong, H.; Cho, J.; Pak, C.; Oh, T.S.; Hong, J.P.; Kwon, S.; Yoo, J. Numerical Approach to Facial Palsy Using a Novel Registration Method with 3D Facial Landmark. Sensors 2022, 22, 6636. [Google Scholar] [CrossRef]
- Kim, H.; Kim, S.; Kim, Y.; Park, K. A Smartphone-Based Automatic Diagnosis System for Facial Nerve Palsy. Sensors 2015, 15, 26756–26768. [Google Scholar] [CrossRef]
- Wang, T.; Dong, J.; Sun, X.; Zhang, S.; Wang, S. Automatic Recognition of Facial Movement for Paralyzed Face. Biomed. Mater. Eng. 2014, 24, 2751–2760. [Google Scholar] [CrossRef]
- He, S.; Soraghan, J.J.; O’Reilly, B.F.; Xing, D. Quantitative Analysis of Facial Paralysis Using Local Binary Patterns in Biomedical Videos. IEEE Trans. Biomed. Eng. 2009, 56, 1864–1870. [Google Scholar] [CrossRef]
- McGrenary, S.; O’Reilly, B.F.; Soraghan, J.J. Objective Grading of Facial Paralysis Using Artificial Intelligence Analysis of Video Data. In Proceedings of the 18th IEEE Symposium on Computer-Based Medical Systems (CBMS’05), Dublin, Ireland, 23–24 June 2005; pp. 587–592. [Google Scholar]
- Quan, W.; Matuszewski, B.J.; Shark, L.-K. Facial Asymmetry Analysis Based on 3-D Dynamic Scans. In Proceedings of the 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Seoul, Republic of Korea, 14–17 October 2012; pp. 2676–2681. [Google Scholar]
- Gaber, A.; Taher, M.F.; Wahed, M.A.; Shalaby, N.M.; Gaber, S. Classification of Facial Paralysis Based on Machine Learning Techniques. Biomed. Eng. Online 2022, 21, 65. [Google Scholar] [CrossRef] [PubMed]
- Zhuang, Y.; McDonald, M.; Uribe, O.; Yin, X.; Parikh, D.; Southerland, A.M.; Rohde, G.K. Facial Weakness Analysis and Quantification of Static Images. IEEE J. Biomed. Health Inform. 2020, 24, 2260–2267. [Google Scholar] [CrossRef] [PubMed]
- Zhuang, Y.; Uribe, O.; McDonald, M.; Yin, X.; Parikh, D.; Southerland, A.; Rohde, G. F-DIT-V: An Automated Video Classification Tool for Facial Weakness Detection. In Proceedings of the 2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), Chicago, IL, USA, 19–22 May 2019; pp. 1–4. [Google Scholar]
- Arora, A.; Sinha, A.; Bhansali, K.; Goel, R.; Sharma, I.; Jayal, A. SVM and Logistic Regression for Facial Palsy Detection Utilizing Facial Landmark Features. In Proceedings of the 2022 Fourteenth International Conference on Contemporary Computing, Noida, India, 4–6 August 2022; ACM: New York, NY, USA; pp. 43–48. [Google Scholar]
- Jiang, C.; Wu, J.; Zhong, W.; Wei, M.; Tong, J.; Yu, H.; Wang, L. Automatic Facial Paralysis Assessment via Computational Image Analysis. J. Healthc. Eng. 2020, 2020, 1–10. [Google Scholar] [CrossRef] [PubMed]
- Parra-Dominguez, G.S.; Garcia-Capulin, C.H.; Sanchez-Yanez, R.E. Automatic Facial Palsy Diagnosis as a Classification Problem Using Regional Information Extracted from a Photograph. Diagnostics 2022, 12, 1528. [Google Scholar] [CrossRef] [PubMed]
- Zhang, Y.; Gorriz, J.M.; Dong, Z. Deep Learning in Medical Image Analysis. J. Imaging 2021, 7, 74. [Google Scholar] [CrossRef]
- Storey, G.; Jiang, R. Face Symmetry Analysis Using a Unified Multi-Task CNN for Medical Applications. In Advances in Intelligent Systems and Computing; Springer: Berlin/Heidelberg, Germany, 2019; pp. 451–463. [Google Scholar]
- Sajid, M.; Shafique, T.; Baig, M.; Riaz, I.; Amin, S.; Manzoor, S. Automatic Grading of Palsy Using Asymmetrical Facial Features: A Study Complemented by New Solutions. Symmetry 2018, 10, 242. [Google Scholar] [CrossRef]
- Hsu, G.-S.J.; Huang, W.-F.; Kang, J.-H. Hierarchical Network for Facial Palsy Detection. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA, 18–22 June 2018; pp. 693–6936. [Google Scholar]
- Guo, Z.; Shen, M.; Duan, L.; Zhou, Y.; Xiang, J.; Ding, H.; Chen, S.; Deussen, O.; Dan, G. Deep Assessment Process: Objective Assessment Process for Unilateral Peripheral Facial Paralysis via Deep Convolutional Neural Network. In Proceedings of the 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), Melbourne, Australia, 18–21 April 2017; pp. 135–138. [Google Scholar]
- Storey, G.; Jiang, R.; Keogh, S.; Bouridane, A.; Li, C.-T. 3DPalsyNet: A Facial Palsy Grading and Motion Recognition Framework Using Fully 3D Convolutional Neural Networks. IEEE Access 2019, 7, 121655–121664. [Google Scholar] [CrossRef]
- Barrios Dell’Olio, G.; Sra, M. FaraPy: An Augmented Reality Feedback System for Facial Paralysis Using Action Unit Intensity Estimation. In Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology, Online, 10–14 October 2021; ACM: New York, NY, USA, 2021; pp. 1027–1038. [Google Scholar]
- Tan, X.; Yang, J.; Cao, J. Facial Nerve Paralysis Assessment Based on Regularized Correntropy Criterion SSELM vc and Cascade CNN. In Proceedings of the 2021 55th Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, USA, 31 October–3 November 2021; pp. 1043–1047. [Google Scholar]
- Abayomi-Alli, O.O.; Damaševičius, R.; Maskeliūnas, R.; Misra, S. Few-Shot Learning with a Novel Voronoi Tessellation-Based Image Augmentation Method for Facial Palsy Detection. Electronics 2021, 10, 978. [Google Scholar] [CrossRef]
- Nguyen, D.-P.; Berg, P.; Debbabi, B.; Nguyen, T.-N.; Tran, V.-D.; Nguyen, H.-Q.; Dakpé, S.; Dao, T.-T. Automatic Part Segmentation of Facial Anatomies Using Geometric Deep Learning toward a Computer-Aided Facial Rehabilitation. Eng. Appl. Artif. Intell. 2023, 119, 105832. [Google Scholar] [CrossRef]
- Umirzakova, S.; Ahmad, S.; Mardieva, S.; Muksimova, S.; Whangbo, T.K. Deep Learning-Driven Diagnosis: A Multi-Task Approach for Segmenting Stroke and Bell’s Palsy. Pattern Recognit. 2023, 144, 109866. [Google Scholar] [CrossRef]
- Bensoussan, A.; Li, Y.; Nguyen, D.P.C.; Tran, M.-B.; Yam, S.C.P.; Zhou, X. Machine Learning and Control Theory. In Handbook of Numerical Analysis; Elsevier: Amsterdam, The Netherlands, 2022; pp. 531–558. ISBN 9780323850599. [Google Scholar]
- Sukumaran, A.; Abraham, A. Automated Detection and Classification of Meningioma Tumor from MR Images Using Sea Lion Optimization and Deep Learning Models. Axioms 2021, 11, 15. [Google Scholar] [CrossRef]
- Berner, J.; Grohs, P.; Kutyniok, G.; Petersen, P. The Modern Mathematics of Deep Learning. In Mathematical Aspects of Deep Learning; Cambridge University Press: Cambridge, UK, 2022; pp. 1–111. [Google Scholar]
- Dutta, N.; Subramaniam, U.; Padmanaban, S. Mathematical Models of Classification Algorithm of Machine Learning. In Proceedings of the International Meeting on Advanced Technologies in Energy and Electrical Engineering, Tunis, Tunisia, 28–29 November 2019; Hamad bin Khalifa University Press (HBKU Press): Doha, Qatar, 2020. [Google Scholar]
- Pedrammehr, S.; Hejazian, M.; Chalak Qazani, M.R.; Parvaz, H.; Pakzad, S.; Ettefagh, M.M.; Suhail, A.H. Machine Learning-Based Modelling and Meta-Heuristic-Based Optimization of Specific Tool Wear and Surface Roughness in the Milling Process. Axioms 2022, 11, 430. [Google Scholar] [CrossRef]
- Ma, Q.; Kobayashi, E.; Fan, B.; Nakagawa, K.; Sakuma, I.; Masamune, K.; Suenaga, H. Automatic 3D Landmarking Model Using Patch-based Deep Neural Networks for CT Image of Oral and Maxillofacial Surgery. Int. J. Med. Robot. Comput. Assist. Surg. 2020, 16, e2093. [Google Scholar] [CrossRef] [PubMed]
- Li, J.; Erdt, M.; Janoos, F.; Chang, T.; Egger, J. Medical Image Segmentation in Oral-Maxillofacial Surgery. In Computer-Aided Oral and Maxillofacial Surgery; Elsevier: Amsterdam, The Netherlands, 2021; pp. 1–27. ISBN 9780128232996. [Google Scholar]
- Zhang, S.; Wang, T.; Peng, Y.; Dong, J. A Hierarchically Trained Generative Network for Robust Facial Symmetrization. Technol. Health Care 2019, 27, 217–227. [Google Scholar] [CrossRef] [PubMed]
- Pourebadi, M.; Riek, L.D. Facial Expression Modeling and Synthesis for Patient Simulator Systems: Past, Present, and Future. ACM Trans. Comput. Healthc. 2022, 3, 1–32. [Google Scholar] [CrossRef]
Dataset [Ref.] | Information | Frontal Facial Landmark Points |
---|---|---|
Labeled Face Parts in-the-wild (LFPW) [26] | A total of 1287 annotated images from Flickr, Google, and Yahoo. | 35 |
HELEN [27] | A total of 2330 annotated images from Flickr. | 194 |
Annotated Faces in-the-wild (AFW) [28] | A total of 205 annotated images, including 468 faces. | 6 |
Annotated Facial Landmarks in-the-wild (AFLW) [29] | A total of 25,000 annotated images from Flickr. | 21 |
300 Faces in-the-Wild (300-W) [25] | A total of 600 annotated images captured in unconstrained settings. | 68 |
Extended Multi Modal Verification for Teleservices and Security applications (XM2VTS) [30] | A total of 30 h of annotated video recordings of 295 subjects, including frontal and profile views. | 22 |
CMU MultiPIE [31] | A total of 750,000 images of 337 people from different viewpoints, expressions, and illuminations. A small subset annotated. | 68 |
Annotated Facial Landmarks for Facial Palsy (AFLFP) [5] | Annotated video images of 16 facial expressions of 88 subjects (palsy patients and healthy individuals). | 68 |
Hi4D-ADSIP 3D dynamic facial articulation database [32] | Three-dimensional dynamic facial articulation dataset, with scans with high temporal and spatial resolutions, containing 3360 facial sequences captured from 80 healthy volunteers. | 84 |
YouTube Facial Palsy (YPF) [33] | A total of 32 videos of 22 patients acquired from YouTube and labeled by clinic experts. | - |
Massachusetts Eye and Ear Infirmary (MEEI) [34] | Images and videos of patients with flaccid and non-flaccid facial palsy | - |
Grading system, Year [Ref.] | Information | Traditional (T)/Computer Vision-Based (CV) |
---|---|---|
Botman and Jongkees, 1955 [36] | Five-category gross scale from 0 (normal) to IV (total paralysis). | T |
Peitersen, 2002 [37] | Five-degree gross grading system from I (no palsy) to VI (complete palsy). | T |
Smith, 1980 [38] | Unweighted regional system of four categories (entire face, forehead, eye function, and mouth function) leading to scores from 0 (no function) to 4 (normal). | T |
Adour and Swanson, 1971 [39] | Weighted system that measures the percentage of divisions of the facial nerve (frontal, eye, mouth). | T |
Janssen, 1963 [40] | Weighted scale expressed in percentage, considering four categories (repose, forehead, eye closure, oral branch). | T |
Yanagihara, 1977 [41] | Unweighted system of 10 graded areas of facial functions. | T |
Stennert, 1977 [42] | Specific regional negative scale for motor function and secondary defects, ranging from 0 (normal) to −10 (total palsy). | T |
House and Brackmann (HB), 1983 [43] | Six category scale from I (normal) to VI (no movement). | T |
Burres and Fisch, 1986 [44] | Distance measurement of facial landmarks while resting and between five expressions of the affected side and normal side. | T |
Nottingham, 1994 [45] | Measuring the movement of four landmarks in three facial expressions. | T |
Yanagihara, 2004 [46] | Unweighted scores of 10 facial expressions, scoring from 0 (complete paralysis) to 40 (normal). | T |
Synnybook (SFGS), 2000 [47] | Measures three components of resting symmetry, voluntary movement, and synkinesis, scoring up to 100 (normal). | T |
Stennert-Limberg (SLFS), 2012 [48] | Sum of scores of the normal side in rest in four face regions, with scores of motility evaluations while moving. | T |
Maximum Static Response Array (MSRA), 1994 [49] | The measurement of displacement from a standard model during facial expressions. | CV |
Automated Face Image Analysis (AFIA), 2007 [50] | Tracking the movements of the lips. | CV |
Jane and Tomas, 2010 [51] | Implementing an HB scoring system to measure symmetric smile. | CV |
Glasgow Facial Palsy Scale, 2012 [52] | HB system combined with regional grades. | CV |
CEM Algorithm, 2012 [53] | Mouth parameter analysis using distances from the center of the nose to the edges of the mouth. | CV |
HB Scale | Information |
---|---|
I—normal | Normal facial function. |
II—mild dysfunction | Overall: small weakness noticed on close inspection. At rest: normal symmetry. In motion: moderate-to-good function in forehead, complete eye closure with no effort, and slight mouth asymmetry. |
III—moderate dysfunction | Overall: clear non-disfiguring variation between the two sides. At rest: normal symmetry. In motion: slight-to-moderate movement in forehead, complete eye closure with effort, and weak mouth with utmost effort. |
IV—moderately severe dysfunction | Overall: clear weakness and asymmetry. At rest: normal symmetry and tone. In motion: none in forehead, incomplete eyes closure, and mouth asymmetry with utmost effort. |
V—severe dysfunction | Overall: hardly noticeable motion. At rest: asymmetry. In motion: none in forehead, incomplete eye closure, and minor mouth movement. |
VI—total paralysis | No movement |
Ref. | Objective | Methodology | Dataset | Performance | Conclusions/Limitations |
---|---|---|---|---|---|
[63] | Smartphone-based FP diagnostic system (five FP grades) | Linear regression model for facial landmark detection and SVM with linear kernel for classification | Private dataset of 36 subjects (23 noral−13 palsy patients) performing 3 motions | 88.9% classification accuracy | Reproducibility under different experimental conditions, as well as repeatability of measurements over a period of time, were not implemented |
[64] | Facial movement patterns recognition for FP (2 classes, i.e., normal and asymmetric) | Active Shape Models plus Local Binary Patterns (ASMLBP) for feature extraction and SVM for classification | Private dataset of 570 images of 57 subjects with 5 facial movements | Up to 93.33% recognition rate | High robustness and accuracy |
[65] | Quantitative evaluation of FP (HB scale) | Multiresolution extension of uniform LBP and SVM for FP evaluation | Private dataset of 197 subject videos with 5 facial movements | ~94% classification accuracy | Sensitive to out-plane facial movements, with significant natural bilateral asymmetry |
[51] | Facial landmarks tracking and feedback for FP assessment (HB scale) | Active Appearance Models (AAMs) for facial expression synthesis | Private dataset of frontal images of neutral and smile expressions from 5 healthy subjects | 87% accuracy | Preliminary results to demonstrate a proof of concept |
[66] | FP assessment | ANN | Private dataset of 43 videos from 14 subjects | 1.6% average MSE | Pilot study; general results follow the opinions of experts |
[67] | Facial asymmetry measurement | Measuring 3D asymmetry index | Three-dimensional dynamic scans from Hi4D-ADSIP database (stroke) | - | Extraction of 3D feature points, as well as potential for detecting facial dysfunctions |
[68] | FP classification of real-time facial animation units (seven FP grades) | Ensemble learning SVM classifier | Private dataset of 375 records from 13 patients and 1650 records from 50 control subjects | 96.8% accuracy 88.9% sensitivity 99% specificity | Data augmentation for the imbalanced dataset issues |
[69] | FP quantification | Combination of landmarks and intensity HoG-based features and a CNN model for classification | Private dataset of 125 images of left facial weakness, 126 images of right facial weakness, and 186 images of normal subjects | Up to 94.5% accuracy | The combination of landmarks and HoG intensity features produced the best, when compared to either landmarks or intensity features separately |
[70] | FP classification (three classes) | HOG features and a voting classifier | Private dataset of 37 videos of left weakness, 38 of right and 60 of normal subjects | 92.9% accuracy 93.6% precision 92.8% recall 94.2% specificity | Comparison with other methods revealed the reliability of HOG features |
[71] | Facial metric calculation of face sides symmetry | Facial landmark features with cascade regression and SVM | Stroke faces dataset of 1024 images and 1081 images of healthy faces | 76.87% accuracy | Machine learning problem-specific models can lead to improved performances |
[72] | FP assessment (HB scale) | Laser speckle contrast imaging and NN classifiers | Private dataset of 80 FP patients | 97.14% accuracy | Outperforms the state-of-the-art systems and other classifiers |
[73] | FP classification (three classes) | Regional handcrafted features and four classifiers (MLP, SVM, k-NN, MNLR) | YouTube Facial Palsy (YFP) database | Up to 95.58% correct classification | Severity is higher classified in eyes and mouth regions |
[75] | Face symmetry analysis (symmetrical-asymmetrical) | Unified multi-task CNN | AFLW database to fine tune the model and extended Cohn–Kanade (CK+) to learn face symmetry (18,786 images in total) | - | Lack of fully annotated training set, as well as the need for labeling or a synthesized training set |
[76] | FP classification (five grades) | CNN (VGG-16) | Dataset from online sources augmented to 2000 images | 92.6% accuracy 92.91% precision 93.14% sensitivity 93% F1 Score | Deep features combined with data augmentation can lead to robust classification |
[5] | FP classification | FCN | AFLFP dataset | Normalized mean error (NME): 11.5% Mean average: 2.3% standard deviation | Comparative results indicate that deep learning methods are, overall, better than machine learning methods |
[33] | Quantitative analysis of FP | Deep Hierarchical Network | YouTube Facial Palsy (YFP) database | 5.83% NME | Line segment learning leads to an important part of deep features being able to improve the accuracy of facial landmark and palsy region detection |
[77] | Quantitative analysis of FP | Hierarchical Detection Network | YouTube Facial Palsy (YFP) database | Up to 93% precision and 88% recall | Efficient for video-to-description diagnosis |
[78] | Unilateral peripheral FP assessment (HB scale) | Deep CNN | Private dataset of 720 labeled images of four facial expressions | 91.25% classification accuracy | Fine-tuning deep CNNs can learn specific representations from biomedical images |
[79] | FP grading | Fully 3D CNN | Private FP dataset of 696 sequences with 17 subjects | 82% classification accuracy | Very competent at learning spatio-temporal features |
[80] | AR system for FP estimation | Light-Weight Facial Activation Unit model (LW-FAU) | Private dataset from 20 subjects | - | Lack of FP benchmark models and datasets |
[81] | FP assessment (six classes) | FNPARCELM-CCNN method | YouTube Facial Palsy (YFP) database | 85.5% accuracy | Semi-supervised methods can distinguish different degrees of FP, even with little-labeled data |
[82] | FP detection and classification | Deep feature extraction with SqueezeNet and ECOC-SVM classifier | YouTube Facial Palsy (YFP) database | 99.34% accuracy | Improvement in FP detection from a small dataset |
[83] | Part segmentation | Point-Net++ and PointCNN | CT images of 33 subjects | 99.19% accuracy 89.09% IOU | Geometric deep learning can be efficient |
[84] | FP asymmetry analysis | Proposed deep architecture | YouTube Facial Palsy (YFP) database | 93.8% IOU | Poor with bearded faces due to a lack of such training data images |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Vrochidou, E.; Papić, V.; Kalampokas, T.; Papakostas, G.A. Automatic Facial Palsy Detection—From Mathematical Modeling to Deep Learning. Axioms 2023, 12, 1091. https://0-doi-org.brum.beds.ac.uk/10.3390/axioms12121091
Vrochidou E, Papić V, Kalampokas T, Papakostas GA. Automatic Facial Palsy Detection—From Mathematical Modeling to Deep Learning. Axioms. 2023; 12(12):1091. https://0-doi-org.brum.beds.ac.uk/10.3390/axioms12121091
Chicago/Turabian StyleVrochidou, Eleni, Vladan Papić, Theofanis Kalampokas, and George A. Papakostas. 2023. "Automatic Facial Palsy Detection—From Mathematical Modeling to Deep Learning" Axioms 12, no. 12: 1091. https://0-doi-org.brum.beds.ac.uk/10.3390/axioms12121091