USOO6108437A
`Patent Number:
`11
`(45) Date of Patent:
`
`6,108,437
`Aug. 22, 2000
`
`5,905,807 5/1999 Kado et al. ............................. 382/118
`FOREIGN PATENT DOCUMENTS
`O 805 416 11/1997 European Pat. Off..
`OTHER PUBLICATIONS
`IEEE Transactions. On Neural Networks, vol. 8, No. 1, Jan.
`1997, “Face Recognition/Detection by Probabilistic Deci
`Sion-Based Neural Network”, Shuang-Hung Lin, Sun-Yuan
`Kung, and Long Ji Lin, pp. 114-132.
`IEEE, “Remembering the Past: The Role of Embedded
`Memory in Recurrent Neural Network Architectures', C.
`Lee Giles, Tsungnan Lin, Gill G. Horne, pp. 34-43.
`IEEE Transactions on Signal Processing; vol. 45, No. 11,
`Nov. 1997; “A Delay Damage Model Selection Algorithm
`for NARX Neural Networks”; Tsung-Nan Lin, et al. pp.
`2719-2730.
`Primary Examiner Jon Chang
`ASSistant Examiner Vikkram Bali
`Attorney, Agent, or Firm-Eric B. Janofsky
`57
`ABSTRACT
`A face recognition System is provided comprising an input
`process or circuit, Such as a Video camera for generating an
`image of a perSon. A face detector proceSS or circuit deter
`mines if a face is present in a image. A face position
`registration proceSS or circuit determines a position of the
`face in the image if the face detector proceSS or circuit
`determines that the face is present. A feature extractor
`process or circuit is provided for extracting at least two
`facial features from the face. A voting proceSS or circuit
`compares the extractor facial features with a database of
`extracted facial features to identify the face.
`
`90 Claims, 12 Drawing Sheets
`
`United States Patent (19)
`Lin
`
`54 FACE RECOGNITION APPARATUS,
`METHOD, SYSTEMAND COMPUTER
`READABLE MEDIUM THEREOF
`
`75 Inventor: Shang-Hung Lin, San Jose, Calif.
`73 Assignee: Seiko Epson Corporation, Tokyo,
`Japan
`
`21 Appl. No.: 09/026,970
`22 Filed:
`Feb. 20, 1998
`Related U.S. Application Data
`60 Provisional application No. 60/066,282, Nov. 14, 1997.
`(51) Int. Cl." ....................................................... G06K 9/00
`52)
`... 382/118; 382/217
`58 Field of Search ..................................... 382/117, 118,
`382/157, 291, 116, 155, 156, 158, 159,
`190, 209, 199, 217, 218; 340/825.34; 348/77,
`78; 380/23
`
`56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,012,522 4/1991 Lambert.
`5,164,992 11/1992 Turk et al..
`5,450,504 9/1995 Calia.
`5,497,430 3/1996 Sadovnik et al..
`5,625,704 4/1997 Prasad ..................................... 382/118
`5,634,087 5/1997 Mammone et al..
`5,675,663 10/1997 Koerner et al..
`5,715,325 2/1998 Bang et al. ............................. 382/118
`5,835,616 11/1998 Lobo et al. ............................. 382/118
`5,850,470 12/1998 Kung et al. ............................. 382/157
`5,852,669 12/1998 Eleftheriadis et al.
`... 38.2/118
`5,892,837 4/1999 Luo et al. ............................... 382/117
`
`
`
`
`
`
`
`SYSTEM
`MEMORY
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`DATA
`STORAGE
`DEVICE
`
`SYSTEM
`BUS
`
`CAMERA
`INTERAC
`
`
`
`DATA
`STORAGE
`NTERACE
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`- as aw - - - -
`
`PERFECT CORP. EXH. 1005
`
`

`

`U.S. Patent
`
`Aug. 22, 2000
`
`Sheet 1 of 12
`
`6,108,437
`
`1O
`
`O
`2
`
`A
`
`FACE
`PROCESSOR
`SYSTEM
`
`40
`
`FACE
`RECOGNITION
`SERVER
`
`FIG. 1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`3Of
`
`3O3
`
`305
`
`LUMEN
`NORMALIZER
`
`EDGE
`EXTRACTOR
`
`DOWN
`SAMPLING
`CIRCUIT
`
`
`
`
`
`
`
`
`
`
`
`FIG.3
`
`
`
`
`
`
`
`
`
`FACIAL
`TEMPLATE
`CIRCUIT
`
`
`
`
`
`TEMPLATE
`MATCHING
`CIRCUIT
`
`3O7
`
`309
`
`
`
`
`
`PERFECT CORP. EXH. 1005
`
`

`

`U.S. Patent
`
`Aug. 22, 2000
`
`Sheet 2 of 12
`
`6,108,437
`
`31
`
`33
`
`35
`
`
`
`
`
`
`
`
`
`
`
`FACE
`DETECTOR
`
`
`
`FACE
`POSITION
`REGISTER
`
`
`
`
`
`FEATURE
`EXTRACTOR
`
`OUTPUT
`
`
`
`
`
`FIG. 1A
`
`
`
`VOTING
`CIRCUIT
`
`
`
`
`
`FACE
`RECOGNITION
`SERVER
`
`
`
`CROPPING
`CIRCUIT
`
`LUMEN
`NORMALIZATION
`CIRCUIT
`
`DOWN
`SAMPLING
`CIRCUIT
`
`FEATURE
`VECTOR
`CALCULATOR
`
`FIG.7A
`
`PERFECT CORP. EXH. 1005
`
`

`

`U.S. Patent
`
`Aug. 22, 2000
`
`Sheet 3 of 12
`
`6,108,437
`
`2O1
`
`219
`
`INPUT
`IMAGE
`
`OUTPUT
`MESSAGE
`
`2O3
`
`
`
`217
`
`DETECT
`FACE
`
`EXCEPTION
`HANDLING
`
`205
`
`FACE
`DETECTED
`p
`
`NO
`
`FACE
`LOCATION
`REGISTRATION
`
`FACE
`REGISTERED
`
`YES - 211
`
`EXTRACT
`FACIAL
`FEATURES
`
`COMPARE
`TO
`DATABASE
`
`IDENTIFY
`FACE
`
`213
`
`215
`
`FIG.2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`PERFECT CORP. EXH. 1005
`
`

`

`U.S. Patent
`
`Aug. 22, 2000
`
`Sheet 4 of 12
`
`6,108,437
`
`N
`
`LOWPASS
`FILTER
`
`DOWN
`SAMPLING
`1 : N
`
`N/n
`
`M/n
`
`FIG.4
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`5O1
`
`503
`
`505
`
`LUMEN
`NORMALIZER
`
`
`
`EDGE
`EXTRACTOR
`
`
`
`DOWN
`SAMPLNG
`CIRCUIT
`
`EYE
`TEMPLATE
`CRCUIT
`
`
`
`
`
`
`
`
`
`TEMPLATE
`MATCHING
`CIRCUIT
`
`
`
`FACE
`ALIGNER
`
`
`
`507
`
`509
`
`5f f
`
`FIG.5
`
`PERFECT CORP. EXH. 1005
`
`

`

`U.S. Patent
`
`Aug. 22, 2000
`
`Sheet S of 12
`
`6,108,437
`
`IMAGE
`
`
`
`LUMEN
`NORMALIZATION
`
`FIG.6 FIG-6A
`
`617
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`DOWN
`SAMPLNG
`RATIO 1 : N
`
`APPLY EYE
`TEMPLATE
`
`TEMPLATE
`MATCHING
`
`
`
`
`
`ERROR
`PROCESSING
`
`RESET (L)
`
`
`
`FIG.6A
`
`PERFECT CORP. EXH. 1005
`
`

`

`U.S. Patent
`
`Aug. 22, 2000
`
`Sheet 6 of 12
`
`6,108,437
`
`SETN = F(L)
`
`DOWN
`SAMPLING
`RATIO 1 : N
`
`637
`
`L = L + 1
`
`APPLY EYE
`MIRROR OF
`EYE TEMPLATE
`
`
`
`
`
`TEMPLATE
`MATCHING
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`NO
`
`635
`
`
`
`639
`
`ERROR
`PROCESSING
`
`YES
`
`631
`
`
`
`
`
`633
`
`ARE EYES
`ALIGNED
`p
`
`NO
`
`ROTATE
`MAGE
`
`FIG.6B
`
`RETURN
`
`PERFECT CORP. EXH. 1005
`
`

`

`U.S. Patent
`
`Aug. 22, 2000
`
`Sheet 7 of 12
`
`6,108,437
`
`IMAGE
`
`701
`
`RESET - J
`
`
`
`CROP
`FEATURE U
`
`LUMEN
`NORMALIZATION
`
`DOWN
`SAMPLING
`
`7O
`7
`
`
`
`70 9
`
`
`
`
`
`
`
`
`
`
`
`GENERATE
`X,Y
`DIRECTIONAL
`PROJECTION
`
`LAST
`FEATURE
`
`
`
`
`
`
`
`RETURN
`
`713
`
`
`
`
`
`FIG.7B
`
`PERFECT CORP. EXH. 1005
`
`

`

`U.S. Patent
`
`Aug. 22, 2000
`
`Sheet 8 of 12
`
`6,108,437
`
`1 OOA
`
`OOB
`
`COMPUTER
`1
`
`COMPUTER
`2
`
`
`
`
`
`COMPUTER
`N
`
`COMPUTER
`3
`
`FIG. 13
`
`PERFECT CORP. EXH. 1005
`
`

`

`U.S. Patent
`
`Aug. 22, 2000
`
`Sheet 9 of 12
`
`6,108,437
`
`! 16
`
`616
`
`606
`
`Z06
`
`G06
`
`Z 16
`
`G16
`
`£16
`
`#7
`
`?.
`
`HEZIN9OOBH
`HEZIN9OOEH
`HEZ?N9OOETH
`HEZ?N5)OOETH
`
`
`
`
`
`
`
`| 06
`
`806
`
`LITTOH|OETIS
`
`HEZ?N5)OOETH
`
`HEAHES
`
`PERFECT CORP. EXH. 1005
`
`

`

`U.S. Patent
`
`Aug. 22, 2000
`
`Sheet 10 of 12
`
`6,108,437
`
`fOOf
`
`FOR R = 1,O
`
`GENERATE
`IMAGE
`
`GENERATE
`FACIAL
`FEATURES
`1. . .J
`
`FORS = 1,T
`
`GENERATE
`VIRTUAL
`FACIAL
`FEATURES
`
`f009
`
`AVERAGEO
`FACIAL
`FEATURES
`AND TVIRTUAL
`FEATURES
`
`1Off
`
`STORE
`RESULTS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG 10
`
`PERFECT CORP. EXH. 1005
`
`

`

`U.S. Patent
`
`Aug. 22, 2000
`
`Sheet 11 of 12
`
`6,108,437
`
`- 100
`
`MEMORY
`
`DATA
`STORAGE
`DEVICE
`
`SYSTEM
`BUS
`
`CAMERA
`NTERFACE
`
`DATA
`STORAGE
`INTERFACE
`
`EXTERNAL
`STORE
`
`FIG 11
`
`
`
`
`
`
`
`
`
`
`
`
`
`PERFECT CORP. EXH. 1005
`
`

`

`U.S. Patent
`
`Aug. 22, 2000
`
`Sheet 12 of 12
`
`6,108,437
`
`8 || ||
`
`XHSIC]
`
`EIAIHCl
`
`
`
`
`
`
`
`
`
`
`
`EOLAECI
`
`CJHVOEWEX)
`
`PERFECT CORP. EXH. 1005
`
`

`

`1
`FACE RECOGNITION APPARATUS,
`METHOD, SYSTEM AND COMPUTER
`READABLE MEDIUM THEREOF
`
`Applicant claims priority under 35 U.S.C. S 119 (e) to
`provisional application Ser. No. 60/066,282, filed on Nov.
`14, 1997, entitled Face Recognition Apparatus, Method,
`System and Computer Readable Medium Thereof.
`BACKGROUND OF THE INVENTION
`1. Field of the Invention
`This invention relates generally to a method and System
`for machine recognition of an image. More particularly, this
`invention relates to an apparatus, a method, a System and a
`computer readable medium for automatic recognition a
`human face and then an identification of the recognized face
`from a database of previously stored face data.
`2. Description of the Related Art
`Machine vision has many commercial applications and
`therefore, has attracted much attention in recent years. Many
`machine vision techniques and methods have been devel
`oped for detecting various image patterns and objects
`including deformable objects Such as human faces. The
`ability to recognize a human face is an important machine
`Vision problem. Face recognition applications are numerous,
`as well as diverse.
`Face recognition applications can be used, for example,
`by Security agencies, law enforcement agencies, the airline
`industry, the border patrol, the banking and Securities indus
`tries and the like. Examples of potential applications include
`but are not limited to entry control to limited access areas,
`access to computer equipment, access to automatic teller
`terminals, identification of individuals and the like.
`Present face recognition methods used in perSon identi
`fication Systems, typically employ a face recognizer which
`determines the identity of a human face image. In order to
`identify the image, the System compares the face to other
`faces Stored in a database. Systems used in many visual
`monitoring and Surveillance incorporate a process whereby
`the System determines the position of the human eyes from
`an image or an image Sequence containing the human face.
`Once the position of the eyes is determined, all of other
`important facial features, Such as the position of the nose and
`the mouth, are determined by methods that (1) use correla
`tion templates, (2) spatial image invariants, (3) View-based
`eigen Spaces, etc. The use of one of these methods enables
`the System to recognize a face from a given face database,
`and can also be used to perform a variety of other related
`taskS.
`One problem associated with conventional recognition
`techniques is Somewhat inaccurate identification of the
`human face under analysis. In other words, the face is either
`misidentified or not identified at all. Moreover, Some of
`these techniques require Significant computation time to
`recognize and identify a face. The conventional Solution to
`this problem has been accepting slow recognition time or
`using more power and therefore expensive computation
`equipment.
`
`OBJECTS OF THE INVENTION
`Therefore, it is an object of the present invention to
`overcome the aforementioned problems.
`It is a further object of the present invention to provide a
`method, System, apparatus and computer readable medium
`for accurately recognizing and identifying a human face
`from a face database.
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6,108,437
`
`2
`It is an additional object of the present invention to
`provide a method, System, apparatus and computer readable
`medium for which extracts at least two facial features and
`then Selects one of the features for use in the identification.
`
`SUMMARY OF THE INVENTION
`According to this invention, a face recognition System is
`provided comprising an input process or circuit, Such as a
`Video camera for generating an image of a perSon. A face
`detector process or circuit determines if a face is present in
`a image. A face position registration process or circuit
`determines a position of the face in the image if the face
`detector proceSS or circuit determines that the face is present.
`A feature extractor process or circuit is provided for extract
`ing at least two facial features from the face. A Voting
`process or circuit compares the extractor facial features with
`a database of extracted facial features to identify the face.
`According to another aspect of the present invention, the
`face detector is provided with a lumen normalization circuit
`for lumen normalizing the output by the input circuit. An
`edge extractor extracts edge information from an output of
`the lumen normalization circuit; A down Sampling circuit is
`provided for down Sampling at a predetermined ratio an
`output Signal from the edge extraction circuit. A facial
`template circuit for applies a plurality of facial templates to
`an output of the down Sampling circuit. A facial template
`matching circuit for determines which one of the plurality of
`facial templates corresponds to the output of the down
`Sampling circuit.
`According to an additional aspect of the present
`invention, the face position registration circuit is provided
`with a lumen normalization circuit for lumen normalizing an
`output by the face detector circuit. An edge extractor extracts
`edge information from an output of the lumen normalization
`circuit, and a down Sampling circuit down Samples at a first
`predetermined ratio an output Signal from the edge extrac
`tion circuit. An eye template circuit applies a plurality of eye
`templates to an output of the down Sampling circuit, and an
`eye template matching circuit determines which one of the
`plurality of eye templates corresponds to the output of the
`down Sampling circuit. A face alignment circuit aligns the
`face of the image output by the input unit in accordance with
`the eye matching circuit.
`According to a further aspect of the present invention, the
`feature extractor is provided with a cropping circuit for
`cropping at least the first facial feature and the Second facial
`feature output by the face position registration circuit. A
`lumen normalization circuit is provided for lumen normal
`izing the first facial feature and the Second facial feature
`output by the cropping circuit. An edge extractor extracts
`edge information from the first facial feature and the Second
`facial feature output by the lumen normalization circuit. A
`down Sampling circuit down Samples at a first predetermined
`ratio the first facial feature and at a Second predetermined
`ratio the Second facial feature output Signal by the edge
`extraction circuit. A feature vector calculating circuit for
`calculating a first feature vector of the first facial feature
`from the down Sampling circuit and a Second feature vector
`of the Second facial feature output from Said down Sampling
`circuit.
`According to a still further aspect of the present invention,
`the Voting circuit comprises a first recognizer circuit for
`comparing the first feature vector with corresponding first
`feature vectors of a database of candidates. A Second rec
`ognizer circuit for compares the Second feature vector with
`corresponding Second feature vectors of the database of
`
`PERFECT CORP. EXH. 1005
`
`

`

`6,108,437
`
`3
`candidates, and a Selection circuit of Selects one of the
`candidates in the database in accordance with one of the first
`recognizer circuit and the Second recognizer circuit.
`Other objects and attainments together with a fuller
`understanding of the invention will become apparent and
`appreciated by referring to the following description and
`claims taken in conjunction with the accompanying draw
`IngS.
`
`4
`accomplish Specific tasks. Although a particular computer
`may only have some of the units illustrated in FIG. 12 or
`may have additional components not shown, most comput
`erS will include at least the units shown.
`Specifically, computer 100 shown in FIG. 12 includes a
`random access memory (RAM) 106 for temporary storage of
`information Such as instructions and data, a read only
`memory (ROM) 104 for permanent storage of the comput
`er's configuration and basic operating commands and an
`input/output (I/O) adapter 110 for connecting peripheral
`devices such as a disk unit 113 and printer 114 to the bus
`108, via cables 115 and 112, respectively. Disk unit 113,
`among other functions, Stores the face recognition applica
`tion Software and a database of face information for pro
`cessing by CPU 102. Image input device 132 is also con
`nected to bus 108 via interface 130 for acquiring an image
`containing a face. Image input device 132 can convert an
`image to data usable by CPU 102. Image input device 132
`may be implemented by a Video camera, a digital Still
`camera, a flat bed Scanner and the like. Alternatively, the
`image input device may be replaced by a communication
`channel to another computer or computer network having
`previously Stored images thereon. A user interface adapter
`116 is also provided for connecting input devices, Such as a
`keyboard 120, and other known interface devices including
`mice, speakers and microphones to the bus 108. Visual
`output is provided by a display adapter 118, which connects
`the bus 108 to a display device 122 such as a video monitor.
`Computer 100 has resident thereon, and is controlled and
`coordinated by, an operating System.
`FIG. 11 is another illustration of computer 100, which
`includes system memory or RAM 106, CPU 102 and camera
`or image input device 132 interconnected via a system bus
`108 well known in the computer arts. Also interconnected to
`the system bus 108 is system-addressable storage device
`110A and data storage interface 110B capable of accepting,
`reading and writing information to a type of removable
`media 113A and external store 113B as representative stor
`age mediums in communication with representative proceSS
`ing System 100. Accordingly, in this representative proceSS
`ing System, programming instructions corresponding to the
`face recognition processing may be partially or fully con
`tained within external store 113A, removable media 113B,
`or system memory 106 or ROM 104 as is well known in the
`art
`Moreover, buffer 223 and buffer 210 may be constructed
`within System memory 106 as an integral part thereof or may
`comprise discrete circuitry as is well-known in the comput
`ing arts.
`Removable media 113A may include a floppy disk,
`CD-ROM, ROM cartridge or other apparatus suitable for
`holding information in computer readable form. Similarly,
`external Store 113B may include another processing System,
`a computer readable Storage component, a collection or
`network of processing Systems and/or Storage components,
`or Similar device or devices in communication with pro
`cessing System 2000 to exchange information including the
`above-mentioned thread instructions. Further, in this
`embodiment, processing System 100 is indicated as being a
`general purpose personal computer. However, one of ordi
`nary skill with knowledge of the presently preferred embodi
`ment according to the present invention should know that
`the particular processing System could alternatively include,
`for example, a Special-purpose dedicated micro-controlled
`Subsystem or Similar processing device as long as it has
`Sufficient resources for at least execution of the techniques
`described and charted hereinbelow and has sufficient inter
`
`15
`
`35
`
`40
`
`25
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`In the drawings wherein like reference symbols refer to
`like parts:
`FIG. 1 is a Schematic representation of a preferred
`embodiment in accordance with the present invention;
`FIG. 1A is a Schematic representation of the face detector
`system of FIG. 1;
`FIG. 2 is a flow chart of the preferred embodiment of FIG.
`1;
`FIG. 3 is a block diagram of the face recognition proceSS
`or circuit of the preferred embodiment of FIG. 1;
`FIG. 4 is a block diagram of the down Sampling proceSS
`or circuit of the preferred embodiment of FIG. 1;
`FIG. 5 is a block diagram of the face location registration
`process or circuit of the preferred embodiment of FIG. 1;
`FIGS. 6A and 6B as shown together in FIG. 6 are flow
`charts of the face location registration circuit of FIG. 5;
`FIG. 7A is a block diagram of the feature extraction
`process or circuit of the preferred embodiment of FIG. 1;
`FIG. 7B is a flow chart of the feature extractor of the
`preferred embodiment of FIG. 1;
`FIG. 8 is a exemplary diagram illustrating the method of
`generating a feature Vector,
`FIG. 9 is a block diagram of the voting process or circuit
`of the preferred embodiment of FIG. 1;
`FIG. 10 is a flow chart of a routine for generating face data
`in the face recognition Server of the preferred embodiment
`of FIG. 1;
`FIG. 11 illustrates in more detail a representative process
`ing System;
`FIG. 12 is a block diagram of a computer System, for
`example, a personal computer System on which a face
`recognition application program in accordance with the
`present invention can operate; and
`FIG. 13 is a Schematic diagram of distribute processing
`implementation of the present invention.
`DESCRIPTION OF THE PREFERRED
`EMBODIMENTS
`The present invention is preferably practiced in the con
`text of an application program performed by a computer
`such as the x86 compatible computer (a.k.a. IBM
`55
`compatible), Apple Macintosh computer, Sun engineering
`work Station and the like. The present invention consists of
`primarily three major functions, namely image acquisiaton,
`face database processing and face recognition processing, as
`will be explained in detail herein below.
`A representative hardware environment is depicted in
`FIG. 12, which illustrates a typical hardware configuration
`of a computer 100 in accordance with the subject invention.
`The computer or processing system 100 is controlled by a
`central processing unit (CPU) 102, which may be at least
`one conventional microprocessor, a number of other units,
`all interconnected via a system bus 108, are provided to
`
`45
`
`50
`
`60
`
`65
`
`PERFECT CORP. EXH. 1005
`
`

`

`S
`facing resources to communicate and eXchange information
`with the image input member or camera 10. In this System,
`image acquisition, face recognition processing and database
`processing are performed on a Single computer.
`In another approach, as shown in FIG. 13, a distributed
`processing model may be implemented. In this approach,
`various components of the face recognition System are
`processed on different computers 100A, 100B, 100C .
`. .
`100N. These computers communicate with each other
`through a well know networking System. AS can be appre
`ciated by one of ordinary skill in the art, these computers
`may reside at the same location or at different locations. In
`this approach, for example, image acquisition may be pro
`cessed by computer 100A, face recognition may be per
`formed on computer 100B and a database processing may be
`performed on computer 100B.
`Details of each of these components will be presented
`herein below.
`Reference is now made to FIGS. 1 and 1A which show the
`general configuration of the face recognition System in
`accordance with the preferred embodiment of the present
`invention. AS will be appreciated by one of ordinary skill in
`the art, the functional blocks shown in these and other
`figures are divided arbitrarily in this specification for con
`Venience of description only. Other arrangements are also
`possible.
`FIG. 1 illustrates, a subject 1 positioned in the view of
`camera 10 for acquiring an image thereof. Camera 10 may
`also be implemented by a multi-frame video camera capable
`of capturing multiple, Sequential images over a period of
`time. Alternatively, a Scanner (not shown), other Suitable
`Video input device or digitizer may be Substituted for camera
`10 to perform face recognition from a previously recorded
`image, Such as a photograph. In another embodiment, a
`Video tape recorder can be employed as a Video input device
`to input video images recorded by a Video camera used by,
`for example, a Surveillance System. These devices may
`provide analog or digital image Signals. Of course, analog
`image Signals need to be converted by an analog to digital
`converter using conventional techniques. Such an image is
`represent by MxN pixels and is stored in memory of the
`System.
`After Subject 1 is appropriately positioned, Subject 1 may
`manually initiate camera 10 to acquire an image or take a
`picture thereof by using interactive panel 20. Alternatively,
`the face recognition System may comprise a proximity
`Sensor (not shown) for automatically acquiring an image
`when subject 1 is properly positioned in the field of vision
`of camera 10. Interactive panel 20 may consist of a display
`panel (such as a CRT or LCD display) and keyboard either
`an actual keyboard or Softkeys located on the display panel.
`Interactive panel 20 may also display the image taken by
`camera 10. In Simpler applications, interactive panel 20 may
`consist of lights, Sound making apparatuses and/or Switches.
`The output of camera 10 is provided to face processor
`System 30 for recognition and identification. Face processor
`System 30 may be implemented by programming a general
`purpose computer described above with appropriate appli
`cation Software Stored in a computer readable medium,
`discrete components, application specific integrated circuits
`and the like or any combination thereof.
`Face processor system 30 extracts a set of features from
`the image output by camera 10 and compares these features
`with a database of features Stored in face recognition Server
`40. Upon identification of the face, face processor system 30
`can perform a variety of activities depending on the appli
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6,108,437
`
`6
`cation of this System. For example, upon identification,
`Subject 1 could be allowed access into a building or a
`computer.
`Face processor System 30 can also provide communica
`tion and control with interactive panel 20 and camera 10.
`Specifically, as noted above, Subject 1, can interact with
`interactive panel 20 to initiate image acquisition. Face
`processor System 30, upon receipt of an initiation signal
`from panel 20, provides a Signal to camera 10 to acquire an
`image. Face processor System 30 could also provide excep
`tion signals to interactive panel, for example, notifying the
`subject to realign himself before camera 10 to take another
`image. A detailed explanation of the face processor System
`30 will be provided hereinbelow.
`Face processor System 30 interfaces with face recognition
`server 40. Face recognition server 40 stores a database
`containing facial characteristics of potential candidates for
`matching. Face recognition Server 40 compares the features
`extracted by face processor System 30 and finds, if possible,
`Statistically a matching face. Of course, if the database is of
`the appropriate size, the database may be Stored in the
`processor System 30 instead of the Separate face recognition
`Server 40.
`FIG. 1A Shows the general configuration of face processor
`system 30, and FIG. 2 is a flow chart of the method
`implemented by this system. Face processor system 30
`comprises a face detector 31 for determining if the image
`from camera 10 (Step 201) contains an image of a face (Step
`203). If no face image is present (step 205), face processor
`System 30 performs exception handling, by for example,
`displaying a message on interactive panel 20. This message
`may include instructions for Subject 1 to reposition himself
`for retaking the picture by camera 10. Once face detector 31
`determines that a face is present in the image, face position
`registrator 33, determines the location of the face in the
`image (step 207). Face position registrator 33 also deter
`mines if the face is properly aligned. If there is
`misalignment, the image is realigned, by for example rota
`tion. Misalignment of the face may occur if Subject 1 is not
`position perpendicular to camera 10 or if his head tilted. If
`the face can not be detected or is significantly out of
`alignment Such that the face could not be properly registered
`(step 209), face position registrator 33 initiates the exception
`handling process (step 217). Again, a message can be sent to
`interactive panel 20 to display a message with instructions
`for Subject 1 to align his head for retaking the picture by
`camera 10.
`Once face position registrator 33 determines the location
`of the face, facial features of the face are extracted by feature
`extractor 35 (step 211). Feature extractor 35 extracts a
`variety of facial features by generating feature vectors, each
`representing a respective feature. In the preferred embodi
`ment the features extracted include feature 1-an eye,
`feature 2-nose and both eyes, and feature-3 mouth, nose
`and both eyes. Of course other facial features and combi
`nations may be used. Feature extractor circuit 35 assigns a
`confidence level to each of the extracted features using a
`neural network or Statistical analysis approach. In the pre
`ferred embodiment a Probabilistic Decision-Based Neural
`Network (PDBNN) is implemented. Such a PDBNN is
`described in, for example, "Face Recognition/Detection by
`Probabilistic Decision-Based Neural Network”, IEEE
`Transaction on Neural Networks, Vol. 8, No. 1, pages
`116-132, January 1997, the entire contents of which are
`incorporated herein by reference. The confidence level indi
`cates the degree of degree of certainty that feature extractor
`35 extracted the appropriate feature. In the preferred
`
`PERFECT CORP. EXH. 1005
`
`

`

`7
`embodiment, three confidence levels are assigned, one for
`each of features 1, 2 and 3, respectively. Face recognition
`server 40 stores the same number of features extracted by
`feature extractor 35 for each potential candidate. In the
`preferred embodiment, face recognition Server 40 Stores
`three facial features for each potential candidate. A detailed
`description will be provided hereinbelow with respect to
`FIG. 10 on how to develop the database. Voting circuit 37
`provides each of the feature vectors extracted by feature
`extractor 35 to face recognition server 40 (step 213). Face
`recognition server 40 selects the stored feature vectors with
`the highest level of confidence based Statistical analysis
`techniques (step 215). Of course in an alternate embodiment,
`the face recognition Server may be part of face processor
`System 30 instead of as a separate processor.
`There are a variety of ways the result the identification
`may be utilized by output device 38 depending on the
`application of the face recognition System. For example in a
`Simple application, if an authorized Subject is recognized, a
`door can be unlock to allow the perSon access to a building.
`Alternatively, if an authorized Subject is recognized, that
`Subject is allowed access to a computer. This System can
`replace or Supplement conventional password Schemes. In
`another embodiment, a recognized Subject can be authorized
`to purchase goods and/or Services at an actual Store or on the
`Internet. In a more complex application, the preferred
`embodiment can identify Subjects from images taken by a
`Surveillance camera and provide Such identification to the
`appropriate perSons.
`FIG.3 provides a detailed explanation of face detector 31.
`AS shown therein, the image acquired by camera 10 is
`processed by lumen normalizer 301 to perform lumen nor
`malization or histogram equalization to compensate for any
`wide variation in illumination of the image of human being
`100. This circuit can increase the dynamic range of the
`image. One Such lumen normalization method is discussed
`in European Patent Application 0.805,416 A2. The image is
`then processed by edge extractor 303 to determine the edge
`of the face in the image.
`The image is then processed by down Sampling circuit to
`reduce the number of pixels processed without decreasing
`the effectiveness of this System while reducing the amount
`of computational power required. In general the down
`Sampling reduces the number of pixels in a ratio of 1:n. In
`the preferred embodiment, the down Sampling ratio is 1:8.
`FIG. 4 shows in more detail the down sampling circuit.
`Initially, image data represented by MxN pixels (401) is
`filtered by a low pass filter (403) using conventional tech
`niques. At this point the image is raster Scanned, and every
`n" pixel is read and then rewritten into buffer 210 (405). As
`a result of this circuit, new data consisting of M/nxN/n
`pixels are stored in the buffer 223 (407).
`Turning back to FIG. 3, after the image is down Sampled,
`predefined facial templates are applied to the down Sampled
`image by facial template circuit 307. These predefined
`templates represent facial characteristics, and may be Stored
`in a read only memory (ROM), main computer memory, disk
`memory and the like or any combination thereof. The facial
`templates were developed using neural network techniques
`as discussed above or Statistical analysis methods. Template
`matching circuit 309 compares the facial templates to the
`downsized image and determines if a match occurs. In other
`words, template matching circuit 309 determines whether a
`face exists in the image. The position of the face is recorded
`in memory. If a face exists processing proceeds; otherwise
`the exception handling procedure is executed, as shown in
`FIG. 2.
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6,108,437
`
`8
`Turning back to FIG. 1A and as noted above, after the
`image is processed by face detector 31, the image, in
`accordance with the position of the face Stored in memory,
`is then processed by face position registrator 33. FIG. 5
`shows in more detail face position registrator 33. Face
`position registrator 33 comprises lumen normalizer 501,
`edge ext