United States Patent (19)
`Coffin et al.
`
`USOO5991429A
`Patent Number:
`11
`(45) Date of Patent:
`
`5,991,429
`Nov. 23, 1999
`
`54 FACIAL RECOGNITION SYSTEM FOR
`SECURITY ACCESS AND IDENTIFICATION
`
`76 Inventors: Jeffrey S. Coffin, 6009 Fairfield La.,
`Sykesville, Md. 21784; Darryl Ingram,
`5444 Ring Dove La., Columbia, Md.
`21044
`
`Moghaddam, Baback and Pentland, Alex; “Face Recogni
`tion Using View-Based and Modular Eigenspaces”-M.I.T.
`Media Laboratory Perceptual Computing Section Technical
`Report No. 301; Appeared in Automatic Systems for the
`Identification and Inspection of Humans, SPIE VO. 2277,
`Jul. 1994.
`
`21 Appl. No.: 08/759,708
`22 Filed:
`Dec. 6, 1996
`51) Int. Cl. ................................ G06K 9/32; G06K 9/46
`52 U.S. Cl. .............................................................. 382/118
`58 Field of Search ..................................... 382/115, 116,
`382/118, 124, 218; 345/968; 434/155; 902/3,
`6; 340/573; 707/3, 6, 104
`
`56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`3,611,290 10/1971 Luisi et al. ........................... 340/146.3
`3,805,238 4/1974 Rothfjell .............................. 340/146.3
`4,135,147
`1/1979 Riganati et al. ..
`340/146.3
`4,300,160 11/1981 Pusch et al. ......
`... 358/113
`4,521,861
`6/1985 Logan et al.......
`... 364/517
`4,525,859 7/1985 Bowles et al. ...
`... 382/5
`4,646,352 2/1987 Asai et al. ....
`... 382/5
`4,699,149 10/1987 Rice ......
`... 128/664
`4,858,000 8/1989 Lu ............................................. 358/84
`4,876,725 10/1989 Tomko
`... 38.2/4
`4,975,969 12/1990 Tal .....
`..., 382/2
`5,027,413
`6/1991 Barnard .
`... 38.2/1
`5,072,120 12/1991 Siewick .........
`... 250/330
`5,107,117 4/1992 Ennenga et al. ..
`... 250/334
`5,163,094 11/1992 Prokoski et al. ............................ 382/2
`5,450,504 9/1995 Calia .............
`... 382/118
`5,553,155 9/1996 Kuhns et al. .
`... 382/115
`5,615,277 3/1997 Hoffman ...........
`... 382/115
`5,627,616
`5/1997 Sergeant et al. .......................... 354/81
`5,689,247 11/1997 Welner ............................... 340/825.31
`5,835,616 11/1998 Lobo et al. ............................. 382/118
`OTHER PUBLICATIONS
`Gilbert, Jeffrey M. and Yang, Woodward, “A Real-Time
`Face Recognition System Using Custom VLSI Hard
`ware”-Harvard University, Nov. 1993.
`
`Primary Examiner Leo H. Boudreau
`ASSistant Examiner Brian P. Werner
`Attorney, Agent, or Firm-Laubscher & Laubscher
`57
`ABSTRACT
`A method and apparatus for identifying individuals for the
`purposes of determining clearance acceSS or Surveillance is
`characterized by enrolling an image of a perSon's face either
`Voluntarily or Secretly to be later used for comparison when
`the perSon Voluntarily desires clearance or is covertly
`detected. The System can recognize or identify individuals
`regardless of whether the individual is wearing eye glasses
`or attempted disguises. In one embodiment, the System
`allows an authorized operator to enroll an image of a perSon
`through a facial Scan for Subsequent clearance access. The
`System records the camera positioning, captures an image,
`Scales the image and records data from a region of interest
`within the Scanned image to a database for later comparison.
`Enrollment data and the corresponding image information
`are then associated with a personal identification number
`assigned to the person. Upon presentment before the system,
`and entering the assigned personal identification number,
`another facial Scan is taken of the person to be compared
`with the data from the regions of interest from the enroll
`ment database of images to confirm the identity of the
`individual. In a Second embodiment, the System operator
`injects an image Secretly taken of an individual for later
`Surveillance and identification. In a Surveillance mode, the
`System automatically detects a perSon's presence, positions
`the camera through analysis of the image, captures an image,
`and then processes the image to determine if the perSon is
`enrolled in the enrollment database.
`
`6 Claims, 9 Drawing Sheets
`
`- 85
`Synchronize Servo
`Subsystern to
`Home Position
`
`66
`
`68
`
`59
`
`73
`
`Enter P.M.
`Nurnber
`
`67
`
`werity PIN
`x
`Y
`
`Record Lockup
`
`s
`
`Download Stored
`CarmeroPositions
`
`Position
`Corner Bused on
`Downloaded
`Information
`
`m
`
`Shecce
`Withic
`72^Location
`
`move
`Cornerossed
`Or Face/Ceriter
`
`
`
`Costure Inge
`
`73
`
`74
`
`the Head
`Locote
`ln Memory Space
`
`75 --
`— -
`Scaling
`
`alla
`
`Extract a
`Region of Interest
`template
`
`Facig|Moiching
`Engine
`
`77,
`
`78
`
`Correlatic
`Process
`
`y - Y -
`Receive
`Sorted Mitch
`List
`
`80,
`& Correlate List
`with P.N
`
`a
`
`Quiput
`Results
`
`VWGoA EX1024
`U.S. Patent No. 9,955,551
`
`

`

`U.S. Patent
`
`Nov. 23, 1999
`
`Sheet 1 of 9
`
`5,991,429
`
`F G
`
`1
`
`2
`
`Enter PassWord
`
`3
`Operator
`Password
`Walid?
`
`
`
`
`
`Select
`Mode
`
`
`
`Enrollment
`Mode
`
`Data Enroll
`
`
`
`New Operator
`Enrol
`
`1O
`
`ACCess Mode
`
`Image Enroll
`
`
`
`Return to
`Main Menu
`
`

`

`U.S. Patent
`
`Nov. 23, 1999
`
`Sheet 2 of 9
`
`5,991,429
`
`
`
`11
`
`F G 2
`
`
`
`
`
`12
`
`Enrollment
`
`Select
`Mode
`
`
`
`
`
`Clearance/Access
`Mode
`
`Surveillance
`Mode
`
`
`
`Dot d
`
`
`
`
`
`Select
`Enrollment
`Mode
`
`
`
`
`
`Data Enrol
`
`
`
`
`
`
`
`Image Based on
`Servo Camera
`
`Covert Image
`
`
`
`Return to
`Main Menu
`
`

`

`U.S. Patent
`
`Nov. 23, 1999
`
`Sheet 3 of 9
`
`5,991,429
`
`Open Data
`Input Screen
`
`F G 3
`
`No
`
`
`
`
`
`
`
`
`
`23
`
`S This NeW
`Enrollment?
`
`Yes
`
`Locate Record
`
`
`
`Open Edit Screen
`
`Edit Data
`
`28
`Open Append
`Screen
`
`29
`Select New
`Record
`
`3O
`
`Input
`Data
`
`32
`
`Yes
`
`Image
`Enrollment
`
`
`
`
`
`
`
`
`
`
`
`
`
`Yes
`
`Close Screen
`
`31
`Capture
`Image?
`33
`Return to
`Data input
`
`Main Menu
`
`

`

`U.S. Patent
`
`Nov. 23, 1999
`
`Sheet 4 of 9
`
`5,991,429
`
`F G, 4
`
`Facial
`Matching
`Engine
`
`Write ServO
`Location to
`Database
`
`
`
`Write Region of
`interest and
`Image
`Information to
`Database
`
`Enrollments of
`Same PerSon?
`
`Enrollment
`Of Different
`Person?
`
`Return to
`Main Menu
`
`Open image
`Enrollment
`Screen
`
`Synchronize Servo
`Subsystem to
`Home Position
`
`
`
`
`
`
`
`Go to Correct
`Record
`
`Position Camera
`Using Service
`Control Panel
`
`is the Face
`Centered?
`
`
`
`Select Region
`Of interest
`
`Scale and
`Normalize
`
`
`
`Image Walidity?
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Nov. 23, 1999
`
`Sheet 5 of 9
`
`5,991,429
`
`
`
`Open Image
`Enrollment
`Screen
`
`Enter dentificotion
`Doto on COvert
`Copture
`
`52
`
`53
`
`54
`
`Inject Covert 1
`Imoge
`
`Width Detection
`
`Select Region -
`of Interest
`
`Scole and
`Normalize
`
`58
`Image Volid?
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Focial Matching
`Engine
`
`Write Height
`Extraction to
`DO tobose
`
`Write Region
`of Interest to
`Dotobose
`
`
`
`
`
`
`
`Enrollments of
`Some Person?
`
`NO
`Return to
`Main Menu
`
`64
`
`F.G. 5
`
`
`
`
`
`
`
`63
`
`
`
`
`
`More
`Enrollments O
`Different
`PerSO
`
`

`

`U.S. Patent
`
`Nov. 23, 1999
`
`Sheet 6 of 9
`
`5,991,429
`
`Synchronize Servo
`Subsystem to
`Home Position
`
`Enter P.N.
`Number
`
`67
`
`65
`
`66
`
`68
`
`Record Lockup
`
`
`
`Downlood Stored
`Comero Positions
`
`POSition
`COmero Bosed on
`Downlooded
`Informotion
`
`
`
`
`
`is the Foce
`Within the
`Locotion?
`
`Capture Imoge
`
`73
`
`74
`
`F G 6
`
`LO Cote the Hedd
`in Memory Space
`
`75
`
`Extrocto
`Region of Interest
`Templote
`
`Focial Matching
`Engine
`
`Correlotion
`Process
`
`Receive
`Sorted Motoh
`List
`
`Correlote List
`with P.N.
`
`Output
`Results
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Nov. 23, 1999
`
`Sheet 7 of 9
`
`5,991,429
`
`Synchronize Se
`Subsystem to
`Home POSition
`
`
`
`
`
`Capture image
`
`Comparison
`PrOCeSS
`
`
`
`Enter P.I.N.
`Number
`
`
`
`Record Lookup
`
`83
`
`85
`
`86
`
`OCate the
`Head in Memory
`Space
`
`Receive Sorted
`Match List.
`
`LOCate Center
`of Face
`
`Correlate List
`With P.I.N.
`
`Scaling
`
`Output
`Results
`
`DOWnload Stored
`Camera POSitions
`
`Normalize
`
`87
`
`POSition
`Camera Based on
`DOWnloaded
`Information
`
`
`
`Extract a Region
`Of Interest
`
`F G. 7
`
`
`
`
`
`is the Face
`Within the
`Location?
`
`-
`
`Carner a Based
`
`Facial
`Matching
`Engine
`
`Look up by
`Class
`
`

`

`U.S. Patent
`
`Nov. 23, 1999
`
`Sheet 8 of 9
`
`5,991,429
`
`109
`
`11 O
`
`111
`
`Comparison
`Process
`
`Receive
`SOrted
`Match List
`
`Evaluate List
`By Goodness
`Of Fit
`
`F G. 8
`
`102
`
`Detect Personal
`Presence
`
`
`
`Position
`Camera
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`LOCate Head
`in Memory
`Space
`
`Scaling
`
`Extract a Region
`of interest
`
`Facial
`Matching
`Engine
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov. 23, 1999
`Nov. 23, 1999
`
`Sheet 9 of 9
`Sheet 9 of 9
`
`5,991,429
`5,991,429
`
`S.
`920
`
`910
`
`900
`
`
`
`Computer
`
`5Sas
`a 5 5
`OD
`OR
`O > -
`a= 2
`(f)
`nn
`
`FIG.9
`
`

`

`1
`FACIAL RECOGNITION SYSTEM FOR
`SECURITY ACCESS AND IDENTIFICATION
`
`2
`factor that the perSon detected is the known terrorist or
`fugitive whose image data was enrolled into the System.
`
`5,991,429
`
`15
`
`BRIEF DESCRIPTION OF THE FIGURES
`Other objects and advantages of the invention will
`become apparent from a Study of the following Specification
`when viewed in the light of the accompanying drawings, in
`which:
`FIG. 1 is a flow chart illustrating the Steps for operating
`the facial recognition System to enroll data, images or new
`operators and to use the System to determine clearance
`CCCSS.
`FIG. 2 is a flow chart illustrating a second embodiment of
`the basic Steps for operating the facial recognition System
`which adds an identification mode for Surveillance and
`detection based upon covert images.
`FIG. 3 is a flow chart illustrating enrollment of demo
`graphic data either by entering new records or editing
`previously existing records.
`FIG. 4 is a flow chart illustrating enrollment of a facial
`image to be used with the clearance access mode.
`FIG. 5 is a flow chart illustrating enrollment of a facial
`image to be used in the Surveillance detection mode.
`FIG. 6 is a flow chart illustrating the steps for identifying
`an individual for clearance acceSS based upon a Scanned
`facial image and an assigned personal identification number.
`FIG. 7 is a flow chart illustrating a second embodiment of
`the Steps for identifying an individual for clearance access
`which uses centering of face Step and class Sorting and
`doping techniques, and
`FIG. 8 is a flow chart illustrating the steps for identifying
`an individual based upon Surveillance detection and com
`parison with enrollment information of a covert image;
`FIG. 9 is a block diagram of a facial recognition System
`according to one embodiment of the present invention.
`DETAILED DESCRIPTION
`The invention relates to a method and apparatus for
`identifying individuals for the purposes of determining
`clearance acceSS or for Surveillance through a facial recog
`nition system 900, as shown in FIG. 9. The facial recognition
`system 900 is comprised of a stand-alone camera 920 for
`taking a facial Scan or image of a perSon and a separate
`computer 910 for image processing and database manage
`ment.
`The facial recognition System is operated in the enroll
`ment mode and the clearance acceSS mode Solely at the
`direction of an authorized operator or System administrator.
`The options presented to the system administrator will be
`described with reference to FIG. 1.
`Specifically, in FIG. 1, upon Successful computer
`initialization, the operator is presented with a main menu at
`step 1 which allows for interaction with the system. The
`operator is first queried to enter a password at Step 2, the
`Successful completion of which logs the operator or System
`administrator into the System. The password may be desig
`nated to give the operator a certain level of authority which
`allows the operator to only perform certain functions. AS an
`example, a password may qualify an operator to use the
`system to identify and verify for clearance, but not allow the
`operator to enroll more perSons into the database.
`Depending upon the level of responsibility assigned to the
`operator password, an operator may have the option at Step
`4 to enroll a new operator at Step 5. Upon entering the
`
`25
`
`BACKGROUND OF THE INVENTION
`Biometric techniques for determining the identity of
`individuals, Such as in Security applications, have been well
`known and in use for Some time. To date, biometric tech
`niques have primarily been oriented towards fingerprint
`analysis rather than the Visual recognition of facial images.
`Identification using infrared thermal imaging has been a
`more recent phenomena. For example, Rice, in U.S. Pat. No.
`4,699,149, taught the scanning of subcutaneous blood ves
`Sels by measuring the inflection of incident radiation as a
`technique for determining a perSon's identification Signa
`ture. However, active heating of the area being Scanned was
`required in order to get an accurate Scan. More recently, the
`Prokcski et al. patent, U.S. Pat. No. 5,163,094, moved the
`infrared field forward by describing a method and apparatus
`for identifying individuals characterized by analyzing
`elemental shapes from thermally derived biosensor data.
`Thermal images are converted into digital representations by
`measuring the intensity of each pixel corresponding to the
`level of thermal energy for a corresponding portion of the
`image. Following Standardization, an image containing the
`elemental shapes can be compared and then correlated with
`unique Structural features of an individual.
`However, Several aspects of the Systems have not been
`dealt with: automatically positioning a camera or other,
`biosensor, enhancing identification accuracy through class
`Sorting, and identifying individual facial features from those
`Subjects wearing eyeglasses.
`SUMMARY OF THE INVENTION
`To overcome the above-noted issues, it is a primary object
`of the present invention to provide a method and apparatus
`for enrolling and identifying individuals for clearance acceSS
`based upon computer matching of a region of interest in a
`facial image Scan. In particular, the present invention utilizes
`a personal identification number which is assigned to each
`enrolled individual. The personal identification number
`identifies a location in a database containing camera posi
`tioning information, clearance data and the reference facial
`image Scan. Upon entering the personal identification num
`ber and Standing before the System, a Scan image is taken
`45
`and processed for comparison with the enrollment image.
`According to another object of the invention, a method
`and apparatus are provided that can Survey a premises and
`immediately detect and identify particular unwanted
`perSons, Such as known terrorists or fugitives.
`A further object of this invention is to provide a facial
`image processing method used in identification both for
`clearance access and Surveillance. The method involves
`capturing an image, locating the head in memory Space,
`Scaling and normalizing the image and extracting a region of
`interest. In a further embodiment of the invention, the center
`of the face is detected for further accuracy even if the perSon
`has facial hair or wears glasses.
`Correlation techniques compare previously processed
`image information with a presently Scanned image to con
`firm or generate the identity of an individual. In a further
`embodiment, the correlation technique in a clearance acceSS
`mode correlates a Sorted list of possible matches with an
`entered personal identification number. The System can also
`Sort the information by a previously defined class. In a
`further embodiment, the correlation technique in a Surveil
`lance mode can generate a goodness of fit or likelihood
`
`35
`
`40
`
`50
`
`55
`
`60
`
`65
`
`

`

`5,991,429
`
`15
`
`25
`
`3
`relevant data, a new password will be assigned to a new
`operator consistent with the appropriate level of authority
`and responsibility for that operator. After enrollment, the
`System may be returned to the Selection mode at Step 4
`within the main menu.
`Again depending upon the level of responsibility assigned
`to the operator password, the operator may have the option
`at Step 4 to enter the enrollment mode or the access mode.
`In the enrollment mode at Step 6, the operator can choose to
`enroll new data at Step 7 or new images of perSons already
`in the database or unknown to the system at step 9. When the
`operator enrolls new data at Step 7, the System provides the
`option at Step 8 of adding a new image.
`After perSons are enrolled in the System database, the
`System may be placed into the acceSS mode at Step 10 to Scan
`facial images to determine or confirm their identity with the
`reference data and images. This can be used in applications
`Such as the entrance point to a Secured laboratory or com
`pany. If the operator wishes to exit the access mode, the
`System provides the option to return to the Selection mode at
`Step 4 within the main menu.
`The options presented to the System administrator in a
`Second embodiment will be described with reference to FIG.
`2. This embodiment provides for an additional feature of
`operation in a Surveillance or covert mode.
`The main menu is presented at Step 11. Depending upon
`the level of responsibility attributed to the password, the
`operator has the option to Select at Step 12 the enrollment
`mode (step 13), the access mode (step 15) or the
`surveillance/identification mode (step 14). In the enrollment
`mode at Step 13, the operator can choose to enroll new data
`(step 16) or new images (step 18) regarding a person already
`in the database or unknown to the System. If the operator
`enrolls new data at Step 16, the System provides he option of
`adding a new image at Step 17. If a new image is to be
`enrolled, the operator is prompted to choose an image type
`at Step 18. An image used for the clearance acceSS mode is
`generated from the Servo camera connected with the System
`at Step 19. The operator may also inject an image taken
`Secretly or for use in a covert operation for Surveillance at
`step 20. After enrollment, the system returns at step 21 to the
`Selection mode at Step 12.
`The operator may choose the clearance/access mode (step
`15). After persons are enrolled in the System database, the
`operator places the System in the acceSS mode to Scan facial
`images to determine or confirm their identity with the
`reference data and images. This can be used in applications
`Such as the entrance point to a Secured laboratory or com
`pany. If the operator wishes to exit the access mode at Step
`50
`15, the system can be returned at step 21 to the selection
`mode at Step 12 within the main menu.
`The operator has the further choice to place the System in
`the Surveillance identification mode (step 14). After persons
`are enrolled in the System database, the operator uses the
`Surveillance mode to detect the presence of known
`offenders, Such as terrorists or fugitives, as desired by the
`System design. If the operator wishes to exit the Surveillance
`mode at step 14, the system will then return at step 21 to the
`Selection mode at Step 12 within the main menu.
`The enrollment of data as set forth in step 6 of FIG. 1 will
`be described in further detail with reference to FIG. 3. Upon
`entering the open data input Screen at Step 22, the operator
`then chooses at Step 23 whether to add a new person to the
`System or edit demographic data of a perSon already present
`in the database. To edit an existing data record, the operator
`queries the System to locate the record at Step 24 and opens
`
`4
`an edit Screen at Step 25 according to the database configu
`ration used. Any conventional database configuration can be
`used in the present invention. The operator can then edit the
`desired data at Step 26 and close the data input Screen at Step
`27.
`If the operator chooses to add a new record, the System
`opens an append record Screen at Step 28 and prompts the
`operator to Select a new record at Step 29. The operator can
`then add all relevant data at step 30, such as the full name,
`address or Social Security number, and clearance level. The
`operator can then close the data input Screen at Step 27.
`If the operator chooses Lo capture a new image at Step 31,
`the System automatically prompts the operator to enroll a
`new image in to the System at Step 32 by entering the image
`enrollment mode, which is described in FIG. 4. If the image
`enrollment is complete or the operator chooses not to enter
`the image enrollment mode, the System then allows the
`operator to return to the data input Screen at Step 33 or return
`to the main menu 34.
`In a further embodiment, the operator may wish to enter
`into the System data and images of perSons who are not
`expected to utilize the facial recognition System for acceSS or
`Surveillance in order to further improve the reliability of the
`clearance access or Surveillance modes shown in FIGS. 6, 7
`and 8. The effect of “doping” the class by adding these
`records will be further discussed in reference to the clear
`ance acceSS mode.
`The enrollment of an image for use in the clearance access
`mode will be described in further detail with reference to
`FIG. 4.
`Upon opening the image enrollment Screen at Step 35, a
`Signal is generated from the System to cue a servo motor
`Subsystem 930 for the camera to a default position at step 36.
`The servo motor Sub-assembly is described in further detail
`in U.S. Pat. No. 5,768,647 the discussion of which is hereby
`incorporated by reference. The camera that Scans the facial
`image is Supported by the Servo motor assembly which can
`adjust upward and downward in order to compensate for
`differences in Subject height. The operator then Selects the
`proper record in the database corresponding to image to be
`stored at step 37. The subject stands before the camera at a
`predetermined distance as the operator positions the camera
`by Sending commands through the System to the Servo motor
`assembly at step 38. The operator continues to adjust the
`angles of the camera until the face of the Subject is centered
`in an operator computer Screen displaying a digital repre
`sentation of the output of the camera at step 39. Upon the
`operator's command, the System then captures at Step 40 an
`image of the Subject and displays the digital representation
`as a still image on the operator's computer Screen. At this
`point, if the face is not reasonably centered on the Screen or
`the Servo motor on the camera assembly is Set incorrectly,
`the operator can opt to discard the image, reposition the
`Servo and capture another image at Step 40.
`If the operator Selects enroll image at Step 41, the System
`then processes the image. The image is first adjusted through
`a width detection unit at Step 42. A region of interest is then
`Selected at Step 43. The region of interest is a portion of the
`facial image to be Scanned and utilized for later comparison.
`If the Scanned image is produced by a thermal infrared
`camera, the region of interest is comprised of a Set of two
`thermal differences in a Small area acroSS the face. The size
`of the region of interest is determined according to the level
`or Sensitivity required by the facial recognition System. A
`larger region of interest results in greater accuracy, but
`requires a larger Scan area. Persons with eye glasses and a
`
`35
`
`40
`
`45
`
`55
`
`60
`
`65
`
`

`

`S
`beard or mustache may limit the possible size of the region
`of interest. Accordingly, a Smaller, Suboptimal region of
`interest may be necessary.
`The System then Scales, Shifts and normalizes the region
`of interest at step 44 to account for inevitable differences in
`distance from the camera and head tilt. By Scaling, shifting
`and normalizing the image to compensate for these
`differences, the facial recognition System can identify per
`Sons in the clearance acceSS mode even if their stance before
`the camera has changed since their image was enrolled.
`The System determines whether the information gathered
`from the image Scan is acceptable for processing. If the
`image is found to be invalid at Step 45, the operator returns
`the System to capture another image at Step 40.
`The region of interest is then input to the facial matching
`engine at Step 46. One skilled in the art will appreciate that
`many facial matching engines are available. One example of
`a facial matching engine, developed by Lockheed Martin,
`utilizes a modified Huff transform. Dr. Pentland at the
`Massachusetts Institute of Technology laboratories devel
`oped a facial matching engine which compares images
`through Eigen values. Dr. Woodward Yang at Harvard
`University has developed a facial matching engine in which
`a set of different correlations are made to produce a Sorted
`table. Another product, Fast Pattern Recognizer developed
`by Fast Pattern Recognizer, Inc. uses an optical correlator
`System. In that System, a correlation engine performs a
`mathematical transform and an optical correlation.
`Additionally, other Systems are available which compare
`data through Artificial Intelligence and fuzzy logic algo
`rithms.
`The operation and type of data stored by these different
`engines varies. For example, Some facial matching engines
`directly compare visual, infrared, or other types of images.
`Other facial matching engines produce templates, vector
`maps or other types of Signatures from imageS which are
`then used for comparison. The term image information is
`used to refer to whatever kind of data is used and/or stored
`by the particular facial matching engine Selected for the
`System. It will also be appreciated that Some facial matching
`engines perform a comparison between image information,
`while others do not, requiring the user to perform the
`comparison Separately using conventional techniques.
`The servo motor location for the camera is written to the
`database location associated with the person to be enrolled
`at step 47. The location of the region of interest and the
`image information are written to the database at Step 48.
`The operator may then choose to enroll more image
`information for the same person at step 49. The system
`WorkS nominally with image information from three images
`of a perSon in clearance acceSS mode. If the operator chooses
`to enroll more image information for the Same perSon, the
`System immediately begins to Scan images for the operator
`to capture at Step 40 without repositioning the camera.
`Position is fixed based on the Settings from the first image
`enrollment. This is done because the height of the perSon
`whose image is to be recaptured should not change more
`than an amount accepted within the System tolerances.
`System tolerances are Selected to account for height changes
`Such as those caused by a person wearing high heels rather
`than flat shoes.
`The operator may choose to enroll image information for
`a different person at step 50. If the operator does choose to
`enroll image information for an additional person for the
`database, the System accesses the proper database record at
`Step 37 and allows the operator to adjust and position the
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`5,991,429
`
`6
`camera using the Service control panel at Step 38 to prepare
`for capturing an image. When the image information enroll
`ment process is complete, the System returns at Step 51 to the
`main menu.
`The enrollment of image information for use in the
`covert/Surveillance mode will be described in further detail
`with reference to FIG. 5. An image information enrollment
`Screen is opened at Step 52 by the operator in order to enter
`demographic data regarding the person whose image is to be
`covertly captured at step 53. This information should include
`but not be limited to the name, height, weight, hair color,
`alias, and criminal record. The operator then injects the
`covertly obtained image into the system at step 54. Most
`likely, this image will be one that was obtained without the
`perSon's consent. This image can be taken using the System
`camera or may be obtained from another Source. For
`example, a frame capture from a Video tape can be used. In
`an embodiment in which the System uses infrared facial
`thermograms to compare image data with a reference image,
`the covert image must be taken from an infrared camera of
`Sufficient Spatial and thermal resolution, Such as a Flir
`PRISM DS starting-array IR imager.
`Once the image is entered into the System, the computer
`automatically begins processing it. The width detection
`process occurs at Step 55, and a region of interest is Selected
`at step 56. The region of interest for a covertly obtained
`image is the same as the region of interest for a Voluntarily
`obtained image. The System then Scales, rotates and normal
`ized the region of interest at step 57 to account for inevitable
`differences in distance from the camera and head-tilt and
`angle.
`The system then determines whether the information
`gathered from the image Scan is acceptable at Step 58. If the
`image is found to be invalid, the System allows the operator
`to inject another covert image at Step 54. The region of
`interest is input to the facial matching engine at Step 59. The
`person's height is written to the database at step 60. Height
`therefore becomes an indicator to be compared with new
`height using a new position when in Surveillance mode. The
`height is determined based on the height of the camera, the
`distance of the person to the camera, and the angle of the
`camera used to position the head in the center of the image
`Space. The location of the region of interest and the image
`information from the facial matching engine are written to
`the database at Step 61.
`The operator may then choose to obtain more enrollment
`image information for the same person at Step 62. The
`System works nominally with image information from three
`covertly obtained images of a person for comparison pur
`poses. The operator then may select the option to enroll
`image information for a different person at step 63. If the
`operator Seeks to enroll image information for a different
`perSon into the database, the System returns to the image
`information enrollment screen at step 52. If or when the
`image information enrollment proceSS is complete, the Sys
`tem returns to the main menu at Step 64.
`After perSons are enrolled Such that the System has
`members in its database to Search, the System can be utilized
`in the clearance access mode. The operation of the System in
`the clearance access mode will be described in further detail
`with reference to FIG. 6.
`Upon entering clearance acceSS mode, and at periodic
`intervals of non-use thereafter, the camera Servo Subsystem
`is Synchronized to the home, or default, position at Step 65.
`The System then waits for a perSon desiring clearance. The
`person who desires clearance is first prompted to enter the
`
`

`

`7
`assigned personal identification number (“PIN”) at step 66.
`If the identification number does not correspond to one
`entered into the clearance access database, the System
`re-queries for a PIN at step 67. Upon the entry of an
`acceptable PIN, the System then gathers the data correspond
`ing to the PIN at step 68. This data may include, for
`example, the name, address, clearance level, etc., that was
`entered as described in FIG. 3, step 30.
`The System then prepares the camera to Scan a new image
`of the person to compare with the reference data. The
`information regarding camera positioning is downloaded at
`step 69. The camera is then automatically positioned based
`upon the downloaded information at step 70.
`If the camera positioning is not acceptable for image
`capture at Step 71, the camera position is automatically
`adjusted at Step 72 until the face is centered. The image is
`captured at Step 73, at which point the System begins
`processing the image. The System extracts the information
`pertaining to the head at Step 74 and Scales the data to a
`standard format at step 75. A region of interest is then
`extracted from the head data at step 76. The method of
`extraction set forth in U.S. Pat. No. 5,163,094, the disclosure
`of which is incorporated herein by reference, may be
`employed. However, it is contemplated that any extraction
`technique can be used. The region of interest is input to the
`facial matching engine at Step 77.
`The System then compares the image information for the
`perSon Seeking clearance access with the Stored image
`information of all perSons enrolled in the database in the
`comparison process at Step 78. The comparison process can
`be part of the facial matching engine or any conventional
`proceSS which can quickly c