- Research article
- Open access
- Published:
A novel investigation of the effect of iterations in sliding semi-landmarks for 3D human facial images
BMC Bioinformatics volume 21, Article number: 208 (2020)
Abstract
Background
Landmark-based approaches of two- or three-dimensional coordinates are the most widely used in geometric morphometrics (GM). As human face hosts the organs that act as the central interface for identification, more landmarks are needed to characterize biological shape variation. Because the use of few anatomical landmarks may not be sufficient for variability of some biological patterns and form, sliding semi-landmarks are required to quantify complex shape.
Results
This study investigates the effect of iterations in sliding semi-landmarks and their results on the predictive ability in GM analyses of soft-tissue in 3D human face. Principal Component Analysis (PCA) is used for feature selection and the gender are predicted using Linear Discriminant Analysis (LDA) to test the effect of each relaxation state. The results show that the classification accuracy is affected by the number of iterations but not in progressive pattern. Also, there is stability at 12 relaxation state with highest accuracy of 96.43% and an unchanging decline after the 12 relaxation state.
Conclusions
The results indicate that there is a particular number of iteration or cycle where the sliding becomes optimally relaxed. This means the higher the number of iterations is not necessarily the higher the accuracy.
Background
Geometric Morphometrics approach differs from traditional morphometrics because it was predicated on the statistical theory of shape and utilizes geometric information collected through the landmark coordinates. This produces various powerful and flexible statistical procedures for shape investigation [1], which are directly interpreted using graphic visualizations [2]. The data obtained during landmarks acquisition follow homology rules, such that, all landmarks must be the same number and be positioned in the same order throughout the specimen. These landmarks can be replicated from subject to subject based on common geometry, common function, and common morphology [3]. With the introduction of Thin Plate-Spline (TPS) and Iterative Closest Point (ICP), the corresponding landmarks of the beginning and target form appear precisely in the corresponding positions and landmark correspondence can now be repeatedly registered in the neighborhood of a landmark [4, 5].
GM of landmarks of 2D or 3D coordinates are the most commonly applied in morphometrics [5, 6]. Through the projection from a reference form, semi-landmarks could be placed semi-automatically by estimating the positions on the target surface [7]. This makes the application of semi-landmarks in geometric analysis of curves and surfaces easy, providing more precise quantification structure which are not available for classical landmark-based [8]. In landmark-based geometric analysis, after the differences due to location, scale and orientation are removed, with the help of Generalized Procrustes Analysis (GPA) [9, 10], shape can be defined as the information remaining in a configuration of landmark points [11, 12]. This produces a set of subjects in partial Procrustes superimposition with respect to a common reference form [13, 14].
Facial landmarking is a crucial step in the facial analysis for biometrics and numerous other applications. Because 3D data contain more information and are less sensitive to illumination and occlusion than that of 2D, the use of 3D data to improve facial analysis has become a trend in computer vision [15]. Consequently, in the course of extracting facial features for facial analysis, the problem of landmarks has been extensively studied in faces. In [16], a literature of some algorithms were presented based on occlusion detection which may be provoked by hair or external objects; such as hats, glasses, scarves, or by the subject’s hands. Hence, due to the decrease in performance when facial area is partially occluded, facial changes are important factors to be considered by face recognition systems [16]. A novel automatic method for facial landmark localization was proposed in [17] to improve performance recognition in human face. The method relies on geometrical properties of 3D facial surface that work on complete face by displaying different emotions in the presence of occlusions. A total of 8 anatomical landmarks (subnasale, pronasale, alare, nasion, endocanthion, exocanthion, inner and outer eyebrows) were selected one-by-one for the study. Under constant geometrical condition, the method double-checks to ensure alare, nasion, and pronasale are correctly localized, else the process starts afresh. A novel fusion pipeline was presented in [18] to address the problem of extreme head pose estimation from intensity images in a monocular setup. The method integrates and updates the covariance of Kalman Filter in every frame. A set of key-points is used to carry out tracking technique and extracts points in the head region. The method proved suitability cases with occlusions and extreme head rotations by relying on the alignment of facial landmarks in each frame.
Geometric analysis of curves and surfaces were made possible through the application of semi-landmarks. Where many structures cannot be quantified and larger areas of many biological objects cannot be captured using classical landmarks, semi-landmarks now provide a more precise quantification [19]. More so, it has been proven in [20] that only anatomical landmarks are insufficient to investigate shape variation of some biological patterns, thus, the method of sliding semi-landmarks was introduced. The sliding semi-landmarks were developed to be placed on surfaces or curves [8, 11] by minimizing bending energy [21, 22] or Procrustes distance [6, 23] which should be homologous throughout the subjects [24].
To perform the sliding of landmarks in 3D, several software packages are currently available. To name but a few, Edgewarp [25], EVAN toolbox (http://evan-society.org), Viewbox [26], Mathematica [23], and R packages: geomorph [27] and Morpho [28], etc.
Sliding semi-landmarks have been used in the study of bones surface such as articular and the diaphysis [22] and curves, providing descriptors of outlines and crests [29, 30]. In [6], sliding semi-landmarks were used to investigate craniofacial and dental variation in human by minimizing bending energy and Procrustes distance. Sliding semi-landmarks that are relaxed against a symmetrized mean were used in [31] to circumvent the problem of asymmetry caused by manual semi-landmarks, using bending energy to investigate nasal soft tissue reconstruction. To evaluate difference software packages for semi-landmark, sliding semi-landmarks were used in [20] to analyze the workflow complexity and time consumption to complete the sliding task and accounted for the duration to perform each task. However, asides this work, to the best of our knowledge, no study has presented any research in accounting for the time (duration) to perform the sliding of semi-landmark task for soft-tissue facial analysis. Furthermore, no study has investigated the effect of iteration in sliding semi-landmark for accuracy prediction.
This work aims to investigate whether the number of iteration in sliding semi-landmarks has effect on the predictive result or classification accuracy in geometric morphometric analyses of soft-tissue landmark-based in 3D human face. This is done by projecting the surface semi-landmarks from the template object to the target objects and iteratively sliding the semi-landmarks to a point relaxed. Here we used five relaxation states (one, six, twelve, twenty-four, and thirty) to ensure convergence and optimum smoothness. PCA was used as dimensionality reduction and feature selection due to the many number of facial points. The results for each relaxation state are further analyzed to predict the classification accuracy using LDA; and the visualization was performed using relative warp of the principal components. Figure 1 shows the architectural diagram of the application of multi-point warping for sliding iterations in 3D.
Methods
Dataset description
The dataset consists of 80 (40 males and 40 females) randomly selected sujects in wavefront obj format from Stirling/ESRC 3D Face Database [32]. The 3D facial scans which are intended to facilitate research in face recognition, expression recognition, sexual dimorphism, and perception were acquired in neutral position.
Template mesh and target warping
To build the template, we manually located 16 anatomical points on a 3D mesh (Fig. 2), following the landmark standard in [33, 34]. The landmarks description can be found in [35]. The modeling of the template and sliding of semi-landmarks was performed in Viewbox 4.0 [26] using geometric morphometric tools based on the methodology in [35,36,37,38]. The anchor anatomical points were not subjected to sliding but were used to build the warping fields for the minimization of bending energy. We chose the pronasale to begin the sliding process because of its invariance to facial expression, pose correction and easy detection [39, 40]. Using this point, 484 semi-landmark were automatically generated which overlapped on the pronasale point and later uniformly spread on the facial surface with 1.5 mm radius. This was chosen to accomodate all the 500 points.
By applying the TPS warping, the semi-landmarks slid along the curves and surfaces of the mesh on each target by minimizing the bending energy. The process went through different iterative steps based on the five relaxation states (Fig. 3) to optimally and homologously relax the sliding points. Minimization of the bending energy makes the sliding points homologous to the template configuration. See the studies in [35, 36] for detailed implementation of the sliding and warping tasks.
Time estimation
The time for initial surface projection, sliding of each set of iterations against Procrustes consensus was measured using a timer. All analyses were run on a desktop computer (Dell Optiplex 7010) with a Intel® Core i5–3470 CPU, 8Gb of memory, running on Windows 7 Professional (64 bit).
Feature selection, visualization and Clasification
The data analysis, visualization and classification were performed using PAST 3.0 [41]. The features are selected by dimensionality reduction using Principal Components Analysis (PCA). The PCA yielded 97PCs in total and we chose the first 30PCs which have the highest ranking eigenvectors for each set which accounted for more than 95% variance. But for the visualization of the change in variation, the first PC of the relative warp which accounted for the highest variation was plotted. Linear Discriminant Analysis (LDA) was used to classify the selected PCs since it is easy to implement and no parameter tuning or adjustment required. It has been successfully used in the previous studies to classify gender in morphometrics [42,43,44].
Results
Time estimation
The times measured for surface projection and iterative sliding for relaxation for each set against Procrustes mean shape are presented in Table 1. The time patterns indicate that the higher the number of iterations the longer the processing time.
PCA visualization and classification
The first 2PCs of the PCA explain more than 50% of the variance for each sliding set: one iteration (PC1 = 34.15%, PC2 = 19.79%); six iterations (PC1 = 34.15%, PC2 = 19.79%); twelve iterations (PC1 = 34.41%, PC2 = 18.42%); twenty-four iterations (PC1 = 33.99%, PC2 = 18.30%); and thirty iterations (PC1 = 33.58%, PC2 = 18.46%). To visualise the pattern of variation in each relaxation state of the iterative sliding, the relative warp of the first principal component of each sliding set is presented (Fig. 4a-3e); alongside with the scatterplot of the PC1 vs PC2 (Fig. 4f), showing the distribution of specimens in morphological space. The selected PC scores are subjected to LDA to predict the gender in PAST software for each sliding set as shown in Table 2 and the gender were maximally discriminated.
Discussion
In this study, sliding semi-landmarks iteration was investigated. The approach minimized bending energy on a set of five iterative states (one, six, twelve, twenty-four, and thirty). The sliding task was necessary for the comparison of shapes and forms. Because the use of manual semi-landmarks may not be appropriate to measure shape variation when surfaces and curves on the target are not homologous. Each iterative state task was performed separately. The duration for subsequent iterative relaxation operations for each iterative state was computed and presented. The timing follows a consecutively progressive patterns such that the initial surface projection takes 480 s, one iteration takes 144 s, six iterations takes 3040, twelve iterations takes 96 min, twenty-four iterations takes 160 min, and thirty iterations takes 320 min. To the best of our knowledge, duration in sliding semi-landmark for soft-tissue 3D in human face has not been proposed in any literature asides the work presented in [20], where time difference was computed and analysed between two software packages (Morpho and Edgewarp) for the same sliding tasks.
The principal components between relaxation state one and relaxation state six show no observable difference in variation (PC1 = 34.15%, PC2 = 19.79%), both contains exactly the same percentage variances throughout the PC variables. Meanwhile, there is a slight difference in percentage variance in other relaxation states; though a noticeable decline occurs in PC1 from twelve relaxation state to thirty relaxation state. The relative warp visualisations show no observable difference between one and six relaxation states. But there is observable difference among twelve, twenty-four and thirty relaxation states. The distribution of specimens in morphological space shows a strong overlapping for one and six relaxation states. Both the black triangles and the blue square boxes sit on each other in the morphospace. But there is observable spread among the rest relaxation states.
Using LDA, the gender was maximally discriminated [45]. This was applied to test the accuracy of each relaxation state and possibly answer the hypothesis question, “does the number of iteration in sliding semi-landmarks have effect on the predictive result or classification accuracy?” The first two states (one and six) have exactly equal accuracy (94.64%); meaning that, the number of cycles has no effect on both states. Same is observed for twenty-four and thirty states, having exactly the same accuracy (92.86%). We noticed stability at twelve relaxation state with highest accuracy of 96.43% and a constant decline after the twelve relaxation state. This is an indication that the higher the number of cycles does not in any way indicate the higher the classification accuracy. It also means that there is a particular number of iteration or cycle where the sliding becomes optimally relaxed.
The model performance was measured using precision, sensitivity and specificity while the dataset was divided into 70% training and 30% testing, as no parameter turning is required in LDA. Table 3 presented the performance metric reports with 12 iteration state having the highest sensitivity (0.961) and specificity (0.966). This is an indication that the sliding process is best relaxed at twelve iterations when the bending energy was minimized.
Conclusions
In conclusion, this study investigates the predictive ability of sliding semi-landmarks using various iterative states and duration of time requires in performing the sliding tasks for each state. However, it is observed that the classification accuracy was affected by the number of iterations but not in progressive pattern (i.e. the higher the number of iterations is not necessarily the higher the accuracy). This study is based on Stirling/ERSC dataset which is European population, therefore the methods and results proposed may be tested in other populations. Furthermore, there is a noticeable challenge in the annotation of the eyeball in Viewbox 4.0 when the eyes are opened. Though, it does not affect the annotation of endocanthion and exocanthion. This will be looked into in our future studies.
Availability of data and materials
Raw three-dimensional digitized data and principal component analysis scores for each iteration state. Table S1: Three-dimensional raw data for each iteration state, Table S2: PCs scores for each iteration state. Note that the 3D face dataset is not permitted to be shared by third party according to the license agreement.
Abbreviations
- GPA:
-
General Procrustes Analysis
- GM:
-
Geometric Morphometric
- PCA:
-
Principal Component Analysis
- LDA:
-
Linear Discriminant Analysis
- TPS:
-
Thin Plate-Spline
- PAST:
-
Paleontological Statistics Software
References
Adams DC, Rohlf FJ, Slice DE. A field comes of age: geometric morphometrics in the 21st century. Hystrix. 2013;24(1):7.
Rohlf FJ, Marcus LF. A revolution morphometrics. Trends Ecol Evol. 1993;8(4):129–32.
Bookstein FL. A statistical method for biological shape comparisons. J Theor Biol. 1984;107(3):475–520.
Klingenberg CP. Visualizations in geometric morphometrics: how to read and how to make graphs showing shape changes. Hystrix, The Italian J Mammalogy. 2013;24:1.
Bookstein FL. Principal warps: thin-plate splines and the decomposition of deformations. IEEE Trans Pattern Anal Mach Intell. 1989;11(6):567–85.
Perez SI, Bernal V, Gonzalez PN. Differences between sliding semi-landmark methods in geometric morphometrics, with an application to human craniofacial and dental variation. J Anat. 2006;208(6):769–84.
Klingenberg CP, Barluenga M, Meyer A. Shape analysis of symmetric structures: quantifying variation among individuals and asymmetry. Evolution. 2002;56(10):1909–20.
Gunz P, Mitteroecker P, Bookstein FL. Semilandmarks in three dimensions. In: Modern morphometrics in physical anthropology. Springer; 2005. p. 73–98.
Rohlf FJ. Relative warp analysis and an example of its application to mosquito. Contributions to morphometrics, vol. 8; 1993. p. 131.
Rohlf FJ, Slice D. Extensions of the Procrustes method for the optimal superimposition of landmarks. Syst Biol. 1990;39(1):40–59.
Fred LB. Landmark methods for forms without landmarks- morphometrics of group differences in outline shape. Med Image Anal. 1997:1(3).
Bookstein FL. Morphometric tools for landmark data: geometry and biology: Cambridge University press; 1997.
Rohlf FJ, Loy A, Corti M. Morphometric analysis of Old World Talpidae (Mammalia, Insectivora) using partial-warp scores. Syst Biol. 1996;45(3):344–62.
Slice DE. Modern morphometrics in physical anthropology: Springer Science & Business Media; 2006.
Chen L, Mu Z, Zhang B, Zhang Y. Ear recognition from one sample per person. PLoS One. 2015;10(5):e0129505.
Dagnes N, Vezzetti E, Marcolin F, Tornincasa S. Occlusion detection and restoration techniques for 3D face recognition: a literature review. Mach Vis Appl. 2018;29(5):789–813.
Vezzetti E, Marcolin F, Tornincasa S, Ulrich L, Dagnes N. 3D geometry-based automatic landmark localization in presence of facial occlusions. Multimed Tools Appl. 2017:1–29.
Barros JMD, Mirbach B, Garcia F, Varanasi K, Stricker D. Fusion of Keypoint Tracking and Facial Landmark Detection for Real-Time Head Pose Estimation. In: Applications of Computer Vision (WACV), 2018 IEEE winter conference on: IEEE; 2018. p. 2028–37.
Oxnard CE. One biologist's view of morphometrics. Annu Rev Ecol Syst. 1978;9(1):219–41.
Botton-Divet L, Houssaye A, Herrel A, Fabre A-C, Cornette R. Tools for quantitative form description; an evaluation of different software packages for semi-landmark analysis. PeerJ. 2015;3:e1417.
Cornette R, Baylac M, Souter T, Herrel A. Does shape co-variation between the skull and the mandible have functional consequences? A 3D approach for a 3D problem. J Anat. 2013;223(4):329–36.
Fabre AC, Goswami A, Peigné S, Cornette R. Morphological integration in the forelimb of musteloid carnivorans. J Anat. 2014;225(1):19–30.
Philipp Mitteroecker PG, Windhagerc S, Schaefer K. A brief review of shape, form, and allometry in geometric morphometrics, with applications to human facial morphology. Hystrix Italian J Mammalogy. 2013;24(1):59–66.
Dean A, James RF. E SD: geometric morphometrics: ten years of progress following the ‘revolution’. Italian J Zoology. 2004;71(1):5–16.
Bookstein FL, Green WD. Edgewarp: A flexible program package for biometric image warping in two dimensions. In: Visualization in Biomedical Computing 1994. Rochester: International Society for Optics and Photonics; 1994. p. 135–47. https://doi.org/10.1117/12.185174.
Viewbox 4 - Cephalometric Software [http://dhal.com/viewboxindex.htm].
Adams DC. Otárola-Castillo E: geomorph: an R package for the collection and analysis of geometric morphometric shape data. Methods Ecol Evol. 2013;4(4):393–9.
Schlager S. Morpho: calculations and visualisations related to geometric Morphometrics. R package version. 2013;023:3.
Alvarez A, Ercoli MD, Prevosti FJ. Locomotion in some small to medium-sized mammals: a geometric morphometric analysis of the penultimate lumbar vertebra, pelvis and hindlimbs. Zoology. 2013;116(6):356–71.
De Groote I, Lockwood CA, Aiello LC. A new method for measuring long bone curvature using 3D landmarks and semi-landmarks. Am J Phys Anthropology. 2010;141(4):658–64.
Schlager S, Rüdell A. Analysis of the human osseous nasal shape—population differences and sexual dimorphism. Am J Phys Anthropol. 2015;157(4):571–81.
Stirling-ESRC 3D Face Database [http://pics.stir.ac.uk/ESRC/3d_images.htm].
Caple J, Stephan CN. A standardized nomenclature for craniofacial and facial anthropometry. Int J Legal Med. 2016;130(3):863–79.
Kesterke MJ, Raffensperger ZD, Heike CL, Cunningham ML, Hecht JT, Kau CH, Nidey NL, Moreno LM, Wehby GL, Marazita ML. Using the 3D facial norms database to investigate craniofacial sexual dimorphism in healthy children, adolescents, and adults. Biol Sex Differ. 2016;7(1):23.
Agbolade O, Nazri A, Yaakob R, Ghani AA, Cheah YK. 3-dimensional facial expression recognition in human using multi-points warping. BMC bioinformatics. 2019;20(1):619.
Agbolade O, Nazri A, Yaakob R, Ghani AAA, Cheah YK. Landmark-based homologous multi-point warping approach to 3D facial recognition using multiple datasets. PeerJ Computer Science. 2020;6:e249.
Agbolade O, Nazri A, Yaakob R, Ghani AA, Cheah YK: 3D Homologous Multi-Points Warping Application to Sexual Dimorphism in Human Face. In: 2019 3rd International Conference on Imaging, Signal Processing and Communication (ICISPC). Singapore: IEEE; 2019. p. 166–71. https://doi.org/10.1109/ICISPC.2019.8935694.
Agbolade O, Nazri A, Yaakob R, Ghani AA, Cheah YK. Homologous Multi-Points Warping: An Algorithm for Automatic 3D Facial Landmark. In: 2019 IEEE International Conference on Automatic Control and Intelligent Systems (I2CACIS). Malaysia: IEEE; 2019. p. 79–84. https://doi.org/10.1109/I2CACIS.2019.8825072.
Colombo A, Cusano C, Schettini R. 3D face detection using curvature analysis. Pattern Recogn. 2006;39(3):444–55.
Creusot C, Pears N, Austin J. 3D face landmark labelling. In: Proceedings of the ACM workshop on 3D object retrieval. ACM; 2010. p. 27–32.
Hammer Ø, Harper D, Ryan P. Paleontological statistics software: package for education and data analysis. Palaeontol Electron. 2001;4.
Gilani SZ, Rooney K, Shafait F, Walters M, Mian A. Geometric facial gender scoring: objectivity of perception. PLoS One. 2014;9(6):e99483.
Bekios-Calfa J, Buenaposada JM, Baumela L. Revisiting linear discriminant techniques in gender recognition. IEEE Trans Pattern Anal Mach Intell. 2011;33(4):858–64.
Greenacre M, Vrba E. Graphical display and interpretation of antelope census data in African wildlife areas, using correspondence analysis. Ecology. 1984;65(3):984–97.
Nacua SS, Torres MAJ, Demayo CG. Landmark-based geometric morphometrics in visualizing body shape dimorphism in the endemic cyprinid, Puntius tumba (Herre, 1924), from Lake Lanao, Philippines. In: Environmental Engineering and Applications (ICEEA), 2010 International Conference on: IEEE; 2010. p. 86–90.
Acknowledgments
We acknowledge Stirling/ESRC (University of Stirling) for prompt agreement to use their dataset. Furthermore, credit goes to the Computer Laboratory of the Faculty of Computer Science & Information Technology, Universiti Putra Malaysia.
Funding
This work was supported by Putra Geran UPM (9538100) and Fundamental Research Grant Scheme (5524959). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Author information
Authors and Affiliations
Contributions
OA and AN conceived the idea and data acquisition; OA developed the methodology; AAG prepared original draft; YKC reviewed and edit the manuscript; OA and AN prepared the visualization and performed analysis; RY supervised the entire work and approved the manuscript. We also declared that all authors have read and approved the final manuscript.
Corresponding authors
Ethics declarations
Ethics approval and consent to participate
The use of human subject was approved by the committee in charge of Stirling database headed by Peter Hancock (p.j.b.hancock@stir.ac.uk). See Attached license agreement files (by email). The informed consent to participate was obtained by written and signed.
Consent for publication
Many participants indicated their consent for publication. See the attached consent agreement files (by email) granted by Peter Hancock (p.j.b.hancock@stir.ac.uk). For the purpose of method demonstration, only subject with ID F1002 was visualized in the study who has indicated her consent by written and signed for publication.
Competing interests
The authors declare no competing interest or state.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Additional file 1.
Three-dimensional raw data for each iteration state.
Additional file 2.
PCs scores for each iteration state.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Nazri, A., Agbolade, O., Yaakob, R. et al. A novel investigation of the effect of iterations in sliding semi-landmarks for 3D human facial images. BMC Bioinformatics 21, 208 (2020). https://doi.org/10.1186/s12859-020-3497-7
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12859-020-3497-7