111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US007436988B2
`
`c12) United States Patent
`Zhang et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,436,988 B2
`Oct. 14, 2008
`
`(54) 3D FACE AUTHENTICATION AND
`RECOGNITION BASED ON BILATERAL
`SYMMETRY ANALYSIS
`
`(75)
`
`Inventors: Liyan Zhang, Jiangsu (CN); Anshuman
`Razdan, Phoenix, AZ (US); Gerald
`Farin, Paradise Valley, AZ (US)
`
`(73) Assignee: Arizona Board of Regents, Tempe, AZ
`(US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 289 days.
`
`(21) Appl. No.: 11/145,033
`
`(22) Filed:
`
`Jun.3,2005
`
`(65)
`
`Prior Publication Data
`
`US 2006/0078172 Al
`
`Apr. 13, 2006
`
`Related U.S. Application Data
`
`(60) Provisional application No. 60/577,367, filed on Jun.
`3, 2004.
`
`(51)
`
`Int. Cl.
`G06K 9100
`(2006.01)
`(52) U.S. Cl. ....................................... 382/118; 382/154
`(58) Field of Classification Search ................. 382/118,
`382/154
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`5,680,481 A *
`5,867,588 A *
`6,002,782 A *
`6,381,346 B1 *
`
`10/1997 Prasad eta!. ................ 382/190
`2/1999 Marquardt .................. 382/118
`12/1999 Dionysian ................... 382/118
`4/2002 Eraslan ....................... 382/118
`
`2002/0106114 A1 *
`2005/0044056 A1 *
`
`8/2002 Yan et al ..................... 382/118
`2/2005 Ray eta!.
`..................... 706/52
`
`OTHER PUBLICATIONS
`
`Wu, Y.; Pan, G.; Wu, Z., Face Authentication Based on Multiple
`Profiles Extracted from Range Data, Jun. 9-11, 2003, Springer Ber(cid:173)
`lin/ Heidelberg, vol. 2688/2003, pp. 515-522.*
`Chen, W.; Okamoto, N.; Minami, T., Automatic Personal Identifica(cid:173)
`tion based on Human Face Profiles, Electrical and Computer Engi(cid:173)
`neering, 1998, IEEE Candadian Conference on, May 24-28, 1998,
`vol. 1, pp. 53-56.*
`
`(Continued)
`
`Primary Examiner-Matthew C. Bella
`Assistant Examiner-Anthony Mackowey
`(74) Attorney, Agent, or Firm-Scully, Scott, Murphy &
`Presser, P.C.
`
`(57)
`
`ABSTRACT
`
`There is provided a novel approach for automatic human face
`authentication. Taking a 3D triangular facial mesh as input,
`the approach first automatically extracts the bilateral symme(cid:173)
`try plane of the face surface. The intersection between the
`symmetry plane and the facial surface, namely the Symmetry
`Profile, is then computed. By using both the mean curvature
`plot of the facial surface and the curvature plot of the sym(cid:173)
`metry profile curve, three essential points of the nose on the
`symmetry profile are automatically extracted. The three
`essential points uniquely determine a Face Intrinsic Coordi(cid:173)
`nate System (FICS). Different faces are aligned based on the
`FICS. The Symmetry Profile, together with two transversal
`profiles, namely the Forehead Profile and the Cheek Profile
`compose a compact representation, called the SFC represen(cid:173)
`tation, of a 3D face surface. The face authentication and
`recognition steps are finally performed by comparing the SFC
`representation of the faces.
`
`1 Claim, 11 Drawing Sheets
`(7 of 11 Drawing Sheet(s) Filed in Color)
`
`\\
`· ... ,,'\
`..
`
`:
`
`~­,,,
`
`·\\
`
`-~
`
`p
`I
`--~'\
`·,., ~---,.?·-·
`D
`
`:(~~
`
`... ..))
`
`.. --··r
`
`___..._/
`
`GTL 1005
`IPR of U.S. Patent 9,007,420
`
`0001
`
`

`
`US 7,436,988 B2
`Page 2
`
`OTHER PUBLICATIONS
`
`Yu, K.; Jiang, X.Y.; Bunke, H., Robust Facial Profile Recognition,
`Image Processing, 1996, Proceedings., International Conference on,
`Sep. 16-19, 1996, vol. 3, pp. 491-494.*
`Beumier, C. andAcheroy, M., Automatic face authentication from 3D
`surface. British Machine Vision Conference, Sep. 14-17, 1998, pp.
`449-458, University of Southampton, UK. (beumier98automatic.
`pdf).
`Blanz, V. and Vetter, T., A Morphable Model for the Synthesis of 3D
`Faces. Computer Graphics Proc., 1999, pp. 187-194, Siggraph '99,
`Los Angeles, CA, USA. (Blanz_morphable_model_99).
`Bronstein, A., Bronstein, M., and Kimmel, R., Expression-invariant
`3D face recognition, Proc. Audio & Video-based Biometric Person
`Authentication (AVBPA), 2003, pp. 62-69, Lecture Notes in Comp.
`Science
`2688,
`Springer-Verlag
`Berlin
`Heidelberg.
`(Bronstein_expressinvariance_faces2003.pdf).
`Chang, K., Bowyer, K. and Flynn, P., Multi-Modal 2D and 3D
`Biometrics for Face Recognition. The proceedings of the IEEE inter(cid:173)
`national Workshop on Analysis and Modeling of Faces and Gestures
`(AMFG), Oct. 2003, Nice, France, IEEE. (KChang_amfg03.pdf).
`Chua, C., Han, F. and Ho, Y., 3D human face recognition using point
`signature. 4th ICAFGR, Mar. 26-30, 2000, Gernoble, France
`(point_signature_chua_OO.pdf).
`Due, B. Fischer, S. and Bigun, J., Face Authentication with Gabor
`Information on Deformable Graphs, IEEE Transactions on Image
`Processing, Apr. 1999. pp. 504-516, vol. 8, No. 4, IEEE. (face(cid:173)
`authentication -with -gabor_99 .pdf).
`Gordon, G., Face recognition based on depth and curvature feature,
`Proceeding of the IEEE Computer Society Conference on Computer
`Vision and Pattern Recognition, 1992, pp. 808-810,
`IEEE.
`(gordon_cvpr92.pdf).
`Lao, S., Sumi, Y., Kawade, M. and Tomita, F., 3D template matching
`for pose invariant face recognition using 3D facial model built with
`isoluminance line based stereo vision, International Conference on
`Pattern Recognition
`(ICPR), 2000, pp.
`II:911-916.,
`IEEE.
`(Lao_3D_poselnvariance_faceRecog_2000.pdf).
`
`Lee, Y., Park, K., Shim, J. and Yi, T., 3D Face Recognition Using
`Statistical Multiple Features for the Local Depth Information, 16th
`Interface,
`Jun. 2003:
`International Conference on Vision
`( slat_multi_feature_3dFaceRecog_2003 .pdf).
`Lu, X., Colbry, D. and Jain, A., Matching 2.5D scans for face recog(cid:173)
`nition, International Conference on Pattern Recognition (ICPR),
`2004, pp. 362-366. (2_5_face.pdf).
`Medioni, G. and Waupotitsch, R., Face recognition and modeling in
`3-D, Proceedings of the IEEE International Workshop on Analysis
`and Modeling of Faces and Gestures (AMFG), Oct. 2003, pp. 232-
`233, IEEE. (Face Modeling and Recognition in 3-D_2003.pdf).
`Chen, X., Flynn, P. and Bowyer, K., Vi sable-light and Infrared Face
`Recognition, The proceedings of Workshop on Multimodal User
`Authentication, Dec. 2003, pp. 48-55, Santa Barbara, CA, USA
`(ChenMMUA_2003 .pdf).
`Pan, G., Wu, Y., Wu, Z. and Liu, W., 3D Face Recognition by Profile
`and Surface Matching, Proceedings of the International Joint Con(cid:173)
`ference on Neural Networks, 2003, IEEE. (face_profile_surface.
`pdf).
`Tanaka, H. and Ikeda, M. and Chiaki, H., Curvature-based surface
`recognition using spherical correlation principal directions for
`curved object recognition, Third IEEE International Conference on
`Automatic Face and Gesture Recognition, 1998, pp. 372-377, IEEE.
`( curvatureBased_face_recognition_98_ Tanaka. pdf).
`Wang, Y., Chua, C. and Ho, Y., Facial feature detection and face
`recognition from 2D and 3D images. Pattern Recognition Letters,
`2002,
`pp.
`1191-1202, vol.
`23, Elsevier Science B.V.
`(Chua_facialFeatureDetection_faceRecog_2D3D_02_wang.
`pdf).
`Xu, C., Wang, Y., Tan, T. and Quan, L., Automatic 3D Face Recog(cid:173)
`nition Combining Global Geometric Features with Local Shape
`Variation Information, Proceedings of the Sixth IEEE International
`Conference on Automated Face and Gesture Recognition, May 2004,
`pp. 308-313, IEEE. (wang_2_3dFace_featureDetection_2004.
`pdf).
`* cited by examiner
`
`0002
`
`

`
`00 = N
`
`00
`\c
`0'1
`~ w
`-....l
`rJl
`d
`
`....
`....
`0 ....
`....
`.....
`rFJ =(cid:173)
`
`('D
`('D
`
`FIG.1
`
`llQ
`Device
`
`102
`
`System
`
`11Q
`Kamal
`
`~ ...
`:-+- ....
`0
`
`(')
`
`QO
`0
`0
`N
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`I
`
`.11!i
`
`I
`
`105
`
`107
`
`Client Side
`
`/100
`
`0003
`
`

`
`U.S. Patent
`
`Oct. 14, 2008
`
`Sheet 2 of 11
`
`US 7,436,988 B2
`
`3D
`Face
`
`Date of Birth
`
`Date of
`Enrollment
`
`History of
`Authentication
`
`Raw Data
`(mesh)
`
`3D
`Data
`
`Compressed
`Data
`
`Forehead
`
`Cheek Surfaces
`
`FIG. 2
`
`0004
`
`

`
`U.S. Patent
`
`Oct. 14, 2008
`
`Sheet 3 of 11
`
`US 7,436,988 B2
`
`.,,,.,t:·:·,/·0·
`
`FlG.5
`
`0005
`
`

`
`U.S. Patent
`
`Oct. 14, 2008
`
`Sheet 4 of 11
`
`US 7,436,988 B2
`
`bilateral symmetry
`from
`
`Compute symmetry profile,
`transversal
`profile and
`cheek profile
`
`Store profiles as compact
`representation of
`surface
`
`I
`
`Use mean curvature plot of facial
`surface and curvature plot of
`try profile to
`points
`se and determine FICS
`
`206
`
`208
`
`Use CS to align compact
`representation of face surface with
`stored compact representations of
`face surfaces
`
`Compare compact representation of
`face surface with aligned compact
`representations of face surfaces
`
`FIG.6
`
`0006
`
`

`
`U.S. Patent
`
`Oct. 14, 2008
`
`Sheet 5 of 11
`
`US 7,436,988 B2
`
`F!G. 78
`
`F~G. SA
`
`FlG. 88
`
`FlG .. 8C
`
`/
`
`~~:::,,,
`
`'
`
`,....-':!\
`
`.··(\··
`(("'.,..· ..
`•
`I : .I
`' . ..,..,_
`
`. - /
`
`,J \IJ
`
`F!G. 8H
`
`F!G. 8f
`
`FIG. 8G
`
`0007
`
`

`
`U.S. Patent
`
`Oct. 14, 2008
`
`Sheet 6 of 11
`
`US 7,436,988 B2
`
`\
`\
`
`)
`
`)
`-l
`
`)
`-· .................... ~·
`
`FIG. 98
`
`0008
`
`

`
`U.S. Patent
`
`Oct. 14, 2008
`
`Sheet 7 of 11
`
`US 7,436,988 B2
`
`rn
`
`~
`
`>:;(
`0
`0
`w...
`
`<(
`
`~
`'<"""
`
`0
`!..! .•..
`
`0009
`
`

`
`U.S. Patent
`
`Oct. 14, 2008
`
`Sheet 8 of 11
`
`US 7,436,988 B2
`
`P.,
`
`>~
`
`FIG. 13C
`
`FIG, i2
`
`0010
`
`

`
`U.S. Patent
`
`Oct. 14, 2008
`
`Sheet 9 of 11
`
`US 7,436,988 B2
`
`riG, '158 (part 1)
`
`0011
`
`

`
`U.S. Patent
`
`Oct. 14, 2008
`
`Sheet 10 of 11
`
`US 7,436,988 B2
`
`\
`
`1
`
`0012
`
`

`
`U.S. Patent
`
`Oct. 14, 2008
`
`Sheet 11 of 11
`
`US 7,436,988 B2
`
`- - \h'<.-lght<:¥.l
`Symtn.'l!ry Pn}fil0 E$
`-
`--. Fo.l\?nsoo Pt%!~ Lt,.
`~ ChE:::~k P~~JfHe
`
`f.;~ ..
`
`~~
`·w·
`.$:
`'~· 3{L0
`;-r.·
`'~ (~. r:: .
`. !'l:
`~).. 2{}.0
`~ v
`.. ·~:
`
`I
`
`0013
`
`

`
`US 7,436,988 B2
`
`1
`3D FACE AUTHENTICATION AND
`RECOGNITION BASED ON BILATERAL
`SYMMETRY ANALYSIS
`
`RELATED APPLICATION DATA
`
`This application is based on and claims the benefit of U.S.
`Provisional Patent Application No. 60/577,367 filed on Jun.
`3, 2004, the disclosure of which is incorporated herein by this
`reference.
`
`U.S. GOVERNMENT FINANCIAL ASSISTANCE
`
`Financial assistance for this project was provided by the
`United States Govermnent, National Science Foundation
`Grant No. 0312849. The United States Govermnent may own
`certain rights to this invention.
`
`BACKGROUND
`
`This invention relates to automatic face authentication and
`recognition. More particularly, it relates to a method and
`system for authentication and recognition of faces using a
`three-dimensional facial surface representation and facial
`bilateral syrmnetry plane extraction to derive a profile curve
`and a coordinate system for aligning facial representations for
`comparison.
`Automatic face authentication refers to using facial images
`or scans to verifY an identity claim of a known individual.
`Automatic face authentication has long been an active
`research area for its wide potential applications, such as law
`enforcement, security access, and man-machine interaction.
`Authentication involves performing verification based on a
`one-to-one search to validate the identity claim of an indi(cid:173)
`vidual (i.e., access control for a building, room, or for making
`a transaction at an ATM terminal). Automatic face recogni(cid:173)
`tion refers to using facial images or scans to identifY an
`unknown individual within a database of known individuals.
`Recognition in one-to-many searches is based on comparison
`to a database of known individuals (e.g., law enforcement,
`surveillance, and recently driver licenses). Face authentica(cid:173)
`tion is in one sense a simpler process than face recognition:
`comparisons are made only to the claimed identity, and a
`threshold of similarity is used to accept or reject the claim. In
`another sense, authentication is more difficult, because of the
`need to determine this threshold rather than using a "best
`match" criterion as in many face recognition applications.
`With face authentication, the group of invalid IDs (imposters)
`is, by definition, not in the reference database. Therefore, face
`authentication methods must successfully operate in 1-to-1
`comparisons, without knowledge of possible errors in claims
`(i.e., who else might the individual be).
`Several approaches have been promoted to recognize and
`authenticate an individual or a group of people. Access con(cid:173)
`trol applications authenticate by physical appearance (by
`guard personnel, receptionist); by something the individual
`knows (pins, passwords); by something the individual has
`(lock/key, card, badge, token); by biometric evidence (a
`unique physiological or behavioral characteristic of the indi- 60
`vidual); or by a combination of both "what one has" (i.e., a
`card) and "what one knows" (i.e., theirpasscode ). Most work(cid:173)
`place entry points are typically controlled by a badge/card or
`by physical appearance. All of these methods, except biomet(cid:173)
`rics, are fallible and can be circumvented, lost, or stolen. 65
`Interest in authentication using biometrics is therefore grow(cid:173)
`ing dramatically.
`
`2
`Biometric access control uses measurable physiological or
`behavioral traits to automatically authenticate a person's
`identity. Biometric characteristics must be distinctive of an
`individual, easily acquired and measured, and comparable for
`purposes of security validation. The characteristic should
`change little over time (i.e., with age or voluntary change in
`appearance) and be difficult to change, circumvent, manipu(cid:173)
`late, or reproduce by other means. Typically, high-level com(cid:173)
`puter based algorithms and database systems analyze the
`10 acquired biometric features and compare them to features
`known or enrolled in the database. The mainstream biometric
`technologies use morphological feature recognition such as
`fingerprints, hand geometry, iris and retina scanning, and two
`dimensional (2D) face authentication. Each of these except
`15 face authentication is either intrusive or fails in some cases
`(e.g., about 10% of population do not have good enough
`fingerprints).
`There has been a large body of literature on 2D face rec(cid:173)
`ognition and authentication. For an overview, seeR. Chel-
`20 lappa, C. Wilson, and S. Sirohey. Human and machine rec(cid:173)
`ognition offaces: A survey, Proceedings of the IEEE, 83(5):
`705-740 (1995). Among various approaches, Principal
`Components Analysis (PCA) to face imaging, popularly
`called eigenfaces, is now a cornerstone in face recognition.
`25 For a more detailed explanation ofPCA, see Turk, M., Pent(cid:173)
`land, A, Face recognition using eigenfaces, Proc. CVPR,
`1991, pp 586-591. 2D face authentication, though less intru(cid:173)
`sive than other biometric technologies, has simply not
`attained the degree of accuracy necessary in a security setting.
`30 2D face recognition methods are in general unable to over(cid:173)
`come the problems resulting from illumination, expression or
`pose variations, facial hair and orientation.
`The emerging trend noted by many researchers in the field
`of face recognition is the 3D technology, which offers several
`35 additional advantages to 2D face recognition. 3D technology
`is expected to be more accurate and able to overcome the
`problems of 2D methods, because 3D information is view(cid:173)
`point and lighting condition independent. There are several
`strategies in 3D face recognition. Some researchers try to
`40 segment the 3D face surface into meaningful physiological
`points, lines and regions based on curvature analysis at each
`point. For example, Hallinan eta!. utilized curvature proper(cid:173)
`ties to segment a face surface into regions, and a set of twelve
`features were extracted for face recognition. P. Hallinan, G.
`45 G. Gorden, A. L. Yuille, et a!., Two and three-dimensional
`patterns of the face, A. K. Peters (ed), A K Peters Ltd (1999).
`Moreno eta!., used a HK segmentation (based on the analysis
`of signs of mean and Gaussian curvatures at each point) to
`isolate regions of pronounced curvature, and up to eighty-six
`50 descriptors were obtained from the segmented regions. A. B.
`Moreno, A. Sanchez, J. F. Velez, eta!., Face recognition using
`3D surface-extracted descriptors, Proceedings of the 7'h Irish
`Machine Vision & Image Processing Conference, Antrim, N.
`Ireland, 2003. For the presence of noise, expression variance
`55 and incomplete scanning, however, curvature based feature
`extraction is not robust enough for face recognition. For
`example, Moreno eta!. reported only a 78% correct recogni(cid:173)
`tion rate.
`In some methods, 3D face modeling has been used as an
`enhancement of2D analysis methods. For example, Blanz et
`a! used a 3D morphable face model, which is learned from a
`set of textured 3D scans ofheads, to encode images. V. Blanz,
`T. Vetter. Face Recognition Based on Fitting a 3D Morphable
`Model, IEEE Transactions on Pattern Analysis and Machine
`Intelligence, 25(9) (2003). Recognition is performed based
`on the model coefficients created in the process of fitting the
`morphable model to images. Lee et a!. also presented a
`
`0014
`
`

`
`3
`model-base face recognition approach under a similar frame(cid:173)
`work. M. W. Lee, S. Ranganath, Pose-invariant face recogni(cid:173)
`tion using a 3D deformable model, Pattern Recognition,
`36:1835-1846 (2003). In their method, the deformable 3D
`face model is a composite of an edge model, a color region
`model and a wireframe model. This strategy is a 2D solution
`in nature, for the media to be compared in these studies is still
`2D intensity images. A problem with this strategy is that
`fitting the morphable model to images is a computational
`expensive process. As reported by Lee eta!., 4.5 minutes are
`needed for fitting the model to an image on a workstation with
`a 2 GHz Pentium4 processor.
`Chang eta!. used both 2D and 3D face information for the
`recognition task. K. I. Chang, K. W. Bowyer, P. J. Flynn, Face
`Recognition Using 2D and 3D Facial Data, The Proceedings
`of Workshop in Multimodal User Authentication, pp. 25-32,
`Santa Barbara, Calif., USA (2003). In their experiments, a
`PCA-based approach was tnned for face recognition from 2D
`intensity images and 3D range images, respectively. Their
`comparison result is that 3D outperforms 2D. By combining 20
`the 2D distance metric and the 3D distance metric, a 2D-plus-
`3D criterion is used during the decision process of face rec(cid:173)
`ognition. In their experiments, pose variations that occur dur(cid:173)
`ing the acquisition process are manually normalized. The
`recognition rate of the combination scheme was reported to 25
`be higher than 98% nnder the condition that the 2D and 3D
`images are taken in a front view and the subjects are imaged
`in a normal facial expression. The scheme of Chang et a!.,
`however, requires manual normalization and has to use nor(cid:173)
`mal facial expressions.
`The work of Bronstein et a!, focused on developing a
`representation of the facial surface, invariant to different
`expressions and postures of the face. A. Bronstein, M. Bron(cid:173)
`stein, and R. Kimmel, Expression invariant 3D face recogni(cid:173)
`tion, Audio and Video Based Biometric PersonAuthetication, 35
`pp. 62-69 (2003). Before using the basic idea of PCA, they
`calculate the geometric invariants of a face surface by using
`multidimensional scaling (MDS). For a discussion ofMDS,
`see Schwartz, E. L., Shaw, A., Wolfson, E., A numerical
`solution to the generalized mapmaker's problem: flattening 40
`nonconvex polyhedral surfaces, IEEE Trans. PAMI, 11:
`1005-1008 (1989). Bronstein eta!. did not report the recog(cid:173)
`nition rate, though they claimed that their algorithm can rec(cid:173)
`ognize the difference of twins. Although they did not discuss
`in detail the computation cost of their method, it appears to be 45
`high because MDS needs to calculate the geodesic distances
`between each pair of points on the surface, as well as the eigen
`decomposition of a large matrix.
`Chua et a!. analyzed over four expressions of each person
`to determine the rigid parts of the face. C., F. Han, Y. Ho, 3D 50
`Human Face Recognition Using Point Signature, 4'h IEEE
`International Conference on Automatic Face and Gesture
`Recognition, Grenoble, France, (2000). These rigid parts are
`modeled by point signatures for face indexing. Their method,
`however, was tested on only six individuals.
`Taking a different approach, Beumier et a!. developed an
`integrated 3D face acquisition and comparison system. C.
`Beumier, M. Acheroy, Automatic 3D face authentication.
`Image and Vision Computing, 18:315-321 (2000). The struc(cid:173)
`tured light was used to capture the facial surface. For facial 60
`surface comparison, they abandoned feature extraction but
`calculated the global matching error of the facial surfaces. An
`Iterative Condition Mode (ICM) optimization was performed
`to determine the rotation and translation transform that mini(cid:173)
`mizes the global matching error sampled at fifteen profiles. In 65
`order to speed up the global matching process, they further
`extracted the central profile with maximal protrusion (due to
`
`30
`
`55
`
`US 7,436,988 B2
`
`4
`the nose). The central profile and a mean lateral profile were
`used to compare two faces in the curvature space. The main
`advantages of this method are its high speed and low storage
`needs. But as the authors pointed out, the optimization pro(cid:173)
`cedure used for the 3D face comparison can fail due to noise,
`local minima or bad initial parameters. The reported Equal
`Error Rate, i.e, the rate at which false acceptances (i.e., incor(cid:173)
`rectly accepting an imposter claim) and false rejects (i.e.,
`incorrectly rejecting a valid claim) are equal (the two rates
`10 tend to be inversely rated), is 9%. So they resorted to manual
`refinement for surface matching. Cartoux et a!. presented a
`similar approach to extract the symmetry profile by looking
`for the bilateral symmetry axis of Gaussian curvature values
`of the facial surface. J.Y. Cartoux, J. T. Lapreste, M. Richetin,
`15 Face authentification or recognition by profile extraction
`from range images, IEEE Computer Society Workshop on
`Interpretation of 3D Scenes, pp 194-199 (1989).
`There is a need, therefore, for an improved method and
`system for authentication and recognition using 3D facial
`data that is computationally faster and more accurate in
`matching facial features.
`Additional objects and advantages of the invention will be
`set forth in the description that follows, and in part will be
`apparent from the description, or may be learned by practice
`of the invention. The objects and advantages of the invention
`may be realized and obtained by means of the instrumentali(cid:173)
`ties and combinations pointed out in the appended claims.
`
`SUMMARY
`
`To achieve the foregoing objects, and in accordance with
`the purposes of the invention as embodied and broadly
`described in this document, there is provided a novel system
`and method for automatic face authentication. The system
`and method utilize a 3D triangular facial mesh surface as an
`input, and automatically extract a bilateral symmetry plane of
`the face surface. The intersection between the symmetry
`plane and the facial surface, namely the Symmetry Profile, is
`then computed. By using both the mean curvature plot of the
`facial surface and the curvature plot of the symmetry profile
`curve, three essential points of the nose on the symmetry
`profile are automatically extracted. The three essential points
`uniquely determine a Face Intrinsic Coordinate System
`(FICS). Different faces are aligned based on the FICS. The
`Symmetry Profile, together with two transversal profiles,
`namely the Forehead Profile and the Cheek Profile, comprise
`a compact representation, called the SFC representation, of a
`3D face surface. The face authentication and recognition
`steps are performed by comparing the SFC representation of
`the faces. The system and method of our invention provide
`greatly enhances accuracy in comparison to 2D methods and
`overcomes the barrier of computational complexity in com(cid:173)
`paring 3D facial data.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The patent or application file contains at least one drawing
`executed in color. Copies of this patent or patent application
`publication with color drawings will be provided by the
`Office upon request and payment of the necessary fee.
`The accompanying drawings, which are incorporated in
`and constitute a part of the specification, illustrate the pres(cid:173)
`ently preferred embodiments of the invention and, together
`with the general description given above and the detailed
`description of the preferred methods and embodiments given
`below, serve to explain the principles of the invention.
`
`0015
`
`

`
`US 7,436,988 B2
`
`6
`FIG.l4 illustrates distance measurements between profiles
`of two faces that are registered for comparison according to
`the present invention, wherein the two profiles differ in
`length.
`FIG. 15 shows scans of multiple expressions for an indi(cid:173)
`vidual. FIG. 15A shows the textured 3D meshes directly
`output by the scanner, and FIG. 3B shows the corresponding
`clean faces without texture.
`FIG. 16 shows similarity metric distribution in authentica-
`10 tion tests conducted according to our invention.
`FIG. 17 shows ROC curves of different metrics in authen(cid:173)
`tication tests conducting according to our invention.
`FIG. lS shows two scans of part of the tested individuals
`used in face recognition experiment according to our inven-
`15 tion. FIG. lSA shows the scans in the database, and FIG. lSB
`shows the corresponding scans to be recognized.
`FIG. 19 shows performance results in the face recognition
`tests.
`
`20
`
`5
`FIG. 1 is a functional block diagram of an exemplary
`computer system for archiving, cataloging, query, compari(cid:173)
`son and retrieval of information relating to 3D facial features
`in accordance with the present invention.
`FIG. 2 shows an exemplary XML schema according to the
`present invention to support the archiving, cataloging, query,
`comparison and retrieval of 3D facial features according to
`the present invention.
`FIG. 3 shows an example of a 3D camera scan (FIG. 3A)
`and its corresponding triangle mesh geometry (FIG. 3B).
`FIG. 4 illustrates the enrollment process for archiving 3D
`facial data in the database of the system of FIG. 1.
`FIG. 5 illustrates the general operation of the system of
`FIG. 1 to authenticate a facial image or scan of an individual
`by comparing it with archived 3D data to verify the identity
`claim of the individual.
`FIG. 6 is a diagram of the steps of a preferred method for
`authenticating a face according to the present invention using
`a three-dimensional facial surface representation and facial
`bilateral symmetry plane extraction to derive a profile curve
`and a coordinate system for aligning facial representations for
`comparison.
`FIG. 7 shows results of the estimated mirror plane (FIG.
`7A) and the extracted symmetry plane (FIG. 7B) for four
`facial meshes.
`FIG. SA-SH illustrate the MarkSkirt process used for cor(cid:173)
`rect for errors in calculation of the symmetry plan than can be
`caused by irregular boundaries of a facial mesh. FIG. SA
`shows a 3D facial mesh with an incomplete boundary. FIG.
`SB shows the mirrored mesh corresponding to the mesh of
`FIG. SA. FIG. SC shows the alignment of the meshes ofF I GS.
`SA and SB by using the ICP algorithm directly, which is not
`expected. FIG. SD shows the region between the boundary
`and the dashed curve on the mirrored mesh, which region is
`called the "skirt." FIG. SE illustrates the alignment of the
`non-skirt region on Sm and the original mesh S, which repre(cid:173)
`sents the expected correct alignment. FIGS. SF-SH illustrate
`an example where part of the forehead is missing due to the
`occlusion of hair. The vertices colored red in FIG. SF are the
`SkirtVertices skirt(sm). FIG. SG demonstrates the computed
`symmetry plane without the MarkSkirt process, and FIG. SH
`shows the result with the MarkSkirt process.
`FIG. 9 shows mean curvature plots (FIG. 9A) of the four
`facial surfaces of FIG. 7 and the corresponding colored sym(cid:173)
`metry profile (FIG. 9B) of each facial surface.
`FIG. 10 shows the curvature plots and the three essential
`points of the symmetry profile of the face surface of FIG.
`lOA. FIG. lOB shows the curvature distribution with respect
`to the arc length. FIG. lOC is generated by attaching a line
`segment along the normal direction at each point of the sym(cid:173)
`metry profiles. FIG.lOD shows the three essential points P Nn
`P NB and P NL extracted for the face surface.
`FIG.ll shows curvature plots and the three essential points
`of the symmetry profile for another example of a face surface,
`shown in FIG. llA.
`FIG. 12 illustrates the Face Intrinsic Coordinate System
`(FICS), in which they-axis and z-axis lie in the symmetry
`plane, and the x-axis perpendicular to the symmetry plane.
`FIG. 13 shows the SFC representation of an example facial
`surface, wherein FIG. 13A shows the front view of the facial
`surface with the profiles, FIG. 13B shows the isometric view
`of the profiles and FIG. 13C shows the isometric view of the 65
`SFC representation registered with a second SFC representa-
`tion.
`
`DESCRIPTION
`
`FIG. 1 illustrates in schematic block diagram form an
`exemplary computer network system 100 for storing,
`archiving, query and retrieval of information relating to 3D
`25 facial features according to the present invention. The com(cid:173)
`puter network system 100 includes a server computer system
`102 and a client computer system 104, which are connected
`by data connections 105,107 to a computernetworkl06, e.g.,
`an intranet, the Internet and/or the World Wide Web, so that
`30 the client computer system 104 and the server computer sys(cid:173)
`tem 102 can communicate. As will be readily apparent to
`persons skilled in the art, the client computer system 104 is
`intended to be representative of a plurality of client computer
`systems 104, each of which may communicate with the server
`35 computer system 102 via the network 106, whether sequen(cid:173)
`tially or simultaneously. The computer network system 100
`advantageously makes use of standard Internet protocols
`including TCP/IP and HTTP. TCP/IP is a common transport
`layer protocol used by a worldwide network of computers.
`40 Although the client 104 and the server computer 102 are
`coupled together via the Internet, the invention may also be
`implemented over other public or private networks or may be
`employed through a direct connection and any such commu(cid:173)
`nication implementation is contemplated as falling within the
`45 scope of the present invention.
`The server computer system 102, which has a conventional
`architecture, includes: a central processing unit (CPU), which
`may be implemented with a conventional microprocessor;
`means for temporary storage of information, which may be
`50 implemented with random access memory (RAM); and
`means for permanent storage of information, which may be
`implemented with read only memory (ROM); and means for
`mass storage of information, which may be implemented by
`hard drive or any other suitable means of mass storage known
`55 in the art.
`It will be obvious to someone of ordinary skill in the art that
`the invention can be used in a variety of other system archi(cid:173)
`tectures. As described herein, the exemplary system architec(cid:173)
`ture is for descriptive purposes only. Although the description
`60 may refer to terms commonly used in describing particular
`computer system architectures the description and concepts
`equally apply to other computer network systems, including
`systems having architectures dissimilar to that shown in FIG.
`1.
`
`Still referring to FIG. 1, the server computer system 102
`includes a Web server 103, and a database server 105. The
`Web server 103 manages network resources and handles all
`
`0016
`
`

`
`US 7,436,988 B2
`
`7
`application operations between the browser-based clients 104
`and the server side applications. The database server 105
`includes a database management system (DBMS), a collec(cid:173)
`tion of programs that enables the storing, modification and
`extraction of information from databases 114. The Web
`server 103 facilitates communication and data exchange
`between the client 104 and database 114.
`One or more data acquisition devices 130 can be used to
`generate raw 3D data from a face and to input the raw 3D data
`to the server 102. Some examples of suitable