(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2010/0172551A1
`Gilley et al.
`(43) Pub. Date:
`Jul. 8, 2010
`
`US 20100172551A1
`
`(54) ORGANIZING IMAGES BY CORRELATING
`FACES
`
`(75) Inventors:
`
`Greg Gilley, Los Altos, CA (US);
`Joshua Fagans, Redwood City, CA
`(US); Nikhil Bhatt, Cupertino, CA
`(US); Jeff Albouze, Boulder Creek,
`CA (US); Simeon Leifer, Santa
`Clara, CA (US); Timothy B.
`Martin, Sunnyvale, CA (US);
`Gregory Charles Lindley,
`Sunnyvale, CA (US)
`
`Correspondence Address:
`FSH & RICHARDSON P.C.
`PO BOX 1022
`MINNEAPOLIS, MN 55440-1022 (US)
`
`(73) Assignee:
`
`APPLE INC., Cupertino, CA (US)
`
`(21) Appl. No.:
`
`12/567,541
`
`(22) Filed:
`
`Sep. 25, 2009
`
`Related U.S. Application Data
`(63) Continuation-in-part of application No. 12/436,069,
`filed on May 5, 2009.
`Provisional application No. 61/142,606, filed on Jan.
`5, 2009.
`
`(60)
`
`Publication Classification
`
`(51)
`
`Int. C.
`(2006.01)
`G06K 9/00
`(52)
`U.S. Cl. ........................................................ 382/118
`(57)
`ABSTRACT
`Methods and systems are presented for organizing images. In
`one aspect, a method can include generating a correlation
`value indicating a likelihood that a face included in a test
`image corresponds to a face associated with a base image,
`determining that a correlation threshold exceeds the correla
`tion value and that the correlation value exceeds a non-cor
`relation threshold, generating a similarity score based on one
`or more exposure values and one or more color distribution
`values corresponding to the test image and the base image,
`combining the similarity score with the correlation value to
`generate a weighted correlation value, and determining that
`the test image and the base image are correlated when the
`weighted correlation value exceeds the correlation threshold.
`
`
`
`Correlation System
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 1 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 1 of 16
`
`US 2010/0172551A1
`
`
`
`
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 2 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 2 of 16
`
`US 2010/0172551A1
`
`
`
`a)s
`
`
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 3 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 3 of 16
`
`US 2010/0172551A1
`
`so
`
`
`
`increase Threshold
`314
`
`TOO Few
`
`
`
`
`
`
`
`
`
`
`
`
`
`Receive Images
`302
`
`Set Threshold to
`Default
`
`304
`
`Correlate images
`306
`
`DeCrease Threshold
`316
`
`Number of Buckets
`
`Too Many
`
`Acceptable
`
`Singletons
`310
`
`Too Many
`
`Acceptable
`
`Receive User input
`312
`
`FIG. 3A
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 4 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 4 of 16
`
`US 2010/0172551A1
`
`so
`
`
`
`Set Threshold at
`Moderate Value
`352
`
`Increase Bucket Size
`354
`
`Increase Threshold
`356
`
`FIG. 3B
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 5 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 5 of 16
`
`US 2010/0172551A1
`
`370
`
`Receive Correlation
`SCOre
`372
`
`
`
`A )
`
`NO
`
`Correlate
`
`86
`
`
`
`Do Not Correlate
`388
`
`No
`
`Below
`Correlation?
`374
`YeS
`
`
`
`NonCOrrelation?
`
`YeS
`
`
`
`
`
`
`
`Nu, No
`
`Within Time
`Threshold?
`
`
`
`YeS
`
`Compute Similarity
`SCOre
`
`380
`
`Weight Correlation
`SCOre
`382
`
`
`
`
`
`
`
`Above
`Correlation?
`384
`
`Yes
`
`FIG. 3C
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 6 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 6 of 16
`
`US 2010/0172551A1
`
`o
`
`Receive Detection
`Confidence
`
`402
`
`Test Sharpness and
`Skin Tone
`
`404
`
`
`
`
`
`
`
`Confidence High?
`
`Weight Tests Less
`Relative to
`Confidence
`408
`
`Weight Tests More
`Relative to
`Confidence
`410
`
`Detection Valid?
`
`
`
`
`
`Eliminate Detection
`414
`
`FIG. 4A
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 7 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 7 of 16
`
`US 2010/0172551A1
`
`
`
`1so
`
`Generate Possible
`Correlations
`452
`
`Examine POSSible
`COrrelations
`454
`
`Unlikely
`COrrelations?
`456
`
`Eliminate Unlikely
`COrrelations
`458
`
`FIG. 4B
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 8 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 8 of 16
`
`US 2010/0172551A1
`
`
`
`N
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 9 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 9 of 16
`
`US 2010/0172551A1
`
`
`
`seen nann
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 10 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 10 of 16
`
`US 2010/0172551A1
`
`
`
`
`
`
`
`089-, uue! HINW
`
`• • • • • • • • • • • • • • •*
`
`
`
`
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 11 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 11 of 16
`
`US 2010/0172551A1
`
`
`
`:
`:
`:
`
`.
`
`:
`
`V
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 12 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 12 of 16
`
`US 2010/0172551A1
`
`
`
`S
`
`?
`e
`C
`2
`Z
`O
`
`:::::::::::::::::::
`a SS
`
`O
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 13 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 13 of 16
`
`US 2010/0172551A1
`
`(~~~~…
`
`
`
`
`
`
`
`80/
`
`8 | /
`
`
`
`Suunq|\/ queuuS
`
`90/
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 14 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 14 of 16
`
`US 2010/0172551A1
`
`
`
`?
`
`Z
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 15 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 15 of 16
`
`US 2010/0172551A1
`
`Œ5
`
`spuOdse}}
`
`puodsex,
`
`Jes?n
`
`
`
`?ae quaseuae
`
`
`
`
`
`º N
`
`£ 13)|Ong
`
`Correlation
`
`V
`
`(1.e4
`
`
`
`
`
`
`
`
`
`
`
`
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 16 of 28
`
`

`

`Patent Application Publication
`
`Jul. 8, 2010 Sheet 16 of 16
`
`US 2010/0172551A1
`
`
`
`eueduuOO
`
`
`
`
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 17 of 28
`
`

`

`US 2010/0172551 A1
`
`Jul. 8, 2010
`
`ORGANIZING MAGES BY CORRELATING
`FACES
`
`0001. This application is a continuation-in-part of U.S.
`patent application Ser. No. 12/436,069, filed on May 5, 2009,
`entitled "Organizing Images. By Correlating Faces, and also
`claims the benefit of U.S. Provisional Application No.
`61/142,606, filed Jan. 5, 2009, entitled “Organizing Digital
`Images. By Correlating Graphical Faces, the disclosures of
`which are considered part of and are incorporated by refer
`ence in the disclosure of this application.
`
`TECHNICAL FIELD
`0002 The present disclosure relates to organizing images,
`Such as digital images, by correlating one or more faces
`represented in the images.
`
`BACKGROUND
`0003 Photography is a popular pastime. Images, includ
`ing digital images produced through digital photography, are
`prevalent. Images also can be produced by graphical editing
`Software (e.g., image editing or drawing Software), Video
`equipment, rendering Systems, Scanners, tomography scan
`ners, and radio telescopes.
`0004 Images often represent people and, in particular, the
`faces of people. Facial detection relates to determining
`whether images represent one or more faces. Facial recogni
`tion (or recognition) relates to identifying a person repre
`sented in an image. Recognition can be accomplished by
`comparing selected facial features from a graphical face to a
`facial database. Facial recognition algorithms can identify
`faces by extracting landmarks corresponding to facial fea
`tures from the image. For example, an algorithm may analyze
`the relative position, size, and shape of the eyes, nose, cheek
`bones, and jaw. Recognition algorithms include eigenface,
`fisherface, the Hidden Markov model, and neuronal moti
`vated dynamic link matching.
`
`SUMMARY
`0005. This disclosure describes technologies and tech
`niques for organizing images, such as digital images, by
`correlating faces represented in images.
`0006 Systems implementing techniques described here
`enable users to organize images such as media stored on a
`computer-readable storage device. Images can be automati
`cally assigned to a particular storage location, logically and/
`or physically, based on the correlation of one or more faces
`appearing in the media. For example, an album corresponding
`to a particular individual can be automatically populated with
`digital images that represent that individual. The degree to
`which the images included in an album must be correlated can
`be varied, such as based on user specifications.
`0007. In general, in one aspect, the disclosed techniques
`can be implemented to include generating a correlation value
`indicating a likelihood that a face included in a test image
`corresponds to a face associated with a base image, determin
`ing that a correlation threshold exceeds the correlation value
`and that the correlation value exceeds a non-correlation
`threshold, generating a similarity Score based on one or more
`exposure values and one or more color distribution values
`corresponding to the test image and the base image, combin
`ing the similarity score with the correlation value to generate
`
`a weighted correlation value, and determining that the test
`image and the base image are correlated when the weighted
`correlation value exceeds the correlation threshold.
`0008. The techniques also can be implemented to include
`determining an elapsed time between the base image and the
`test image and computing the similarity score when the
`elapsed time does not exceed a time threshold. Further, the
`techniques can be implemented to include determining the
`elapsed time based on time and date metadata associated with
`the base image and the test image. Additionally, the tech
`niques can be implemented Such that generating a similarity
`score further includes assigning a weighting factor to at least
`one of the one or more exposure values and one or more color
`distribution values.
`0009. The techniques also can be implemented such that
`the correlation threshold comprises a static value. Further, the
`techniques can be implemented Such that wherein generating
`a similarity score further includes adjusting the similarity
`score based on an elapsed time between the base image and
`the test image. Also, the techniques can be implemented to
`include determining that the base image and the test image
`include multiple faces and computing a color distribution
`comparison between a corresponding face region of the base
`image and the test image for use in generating the similarity
`score. Additionally, the techniques can be implemented to
`include determining that a correlation between the base
`image and the test image represents a forbidden association
`and prohibiting the correlation.
`0010. In general, in another aspect, the techniques can be
`implemented to include presenting an image in a user inter
`face, receiving user input identifying a face region in the
`image depicting a face that was not automatically detected,
`altering one or more facial detection criteria, and performing
`a facial detection process for the identified face region using
`the altered facial detection criteria.
`0011. The techniques also can be implemented to include
`receiving user input identifying the face region through a
`graphical user interface. Further, the techniques can be imple
`mented Such that altering one or more facial detection criteria
`further includes suspending the application of one or more
`facial detection criteria.
`0012. The techniques also can be implemented such that
`altering one or more facial detection criteria further includes
`reducing a value associated with a facial detection criterion
`by a predetermined amount. Further, the techniques can be
`implemented to include determining that a face was not
`detected in the identified face region by the facial detection
`process and repeating the facial detection process with incre
`mentally reduced facial detection criteria until a face is
`detected in the identified face region. Additionally, the tech
`niques can be implemented Such that the facial detection
`criteria comprise one or more of a sharpness value, a skin
`tone value, and a head-tilt angle.
`0013. Other implementations of this aspect include corre
`sponding Systems, apparatus, and computer programs, con
`figured to perform the actions of the methods, encoded on
`computer storage devices.
`0014. The techniques described in this specification can be
`implemented to realize one or more of the following potential
`advantages. Correlating faces can be implemented to permit
`automatically sorting images into one or more storage struc
`tures, such as folders or albums. Further, the subject matter
`can be implemented to permit identifying new individuals for
`whom a graphical face definition or profile has not been
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 18 of 28
`
`

`

`US 2010/0172551 A1
`
`Jul. 8, 2010
`
`defined. Additionally, the subject matter can be implemented
`to permit refining the criteria used to correlate faces. Such as
`a definition or profile, based on the association of additional
`images with an individual.
`00.15
`Organizing images by the people represented in the
`media provides several potential advantages. For example,
`Such an organizational scheme can be intuitive for users of an
`image system, enabling users to quickly understand the func
`tioning of the system. Further, the burden of manually orga
`nizing many images can be substantially eliminated or
`reduced. In addition, images can be accurately grouped based
`on a person represented in the images. Accurately grouping
`images can provide improved accessibility, organization and
`usability of the images by users.
`0016. The details of one or more implementations are set
`forth in the accompanying drawings and the description
`below. Other features and advantages will be apparent from
`the description and drawings, and from the claims.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0017 FIG. 1 shows an exemplary overview of correlating
`graphical faces.
`0018 FIG. 2 shows an exemplary system for correlating
`graphical faces.
`0019 FIG. 3A is a flow chart showing an exemplary cor
`relation process.
`0020 FIG. 3B is a flow chart showing an exemplary pro
`cess for setting an adaptive correlation threshold.
`0021 FIG. 3C is a flow chart showing an exemplary pro
`cess for testing a recognition.
`0022 FIG. 4A is a flow chart showing an exemplary detec
`tion elimination process.
`0023 FIG. 4B is a flow chart showing an exemplary cor
`relations elimination process.
`0024 FIG. 5 is a diagram of an exemplary user interface
`for displaying correlated faces.
`0025 FIG. 6A is a diagram of an exemplary user interface
`for receiving user input.
`0026 FIG. 6B is a diagram of an exemplary user interface
`for receiving user input.
`0027 FIG. 6C is a diagram of an exemplary user interface
`for receiving user input identifying a face location.
`0028 FIG. 6D is a diagram of an exemplary user interface
`for receiving user input to associate a name with a face.
`0029 FIG. 7 is a diagram of an exemplary user interface
`for creating albums.
`0030 FIG. 8 is a diagram of an exemplary user interface
`for receiving user input.
`0031
`FIG. 9 is a diagram showing exemplary correlation
`techniques.
`0032 FIG. 10 is a diagram showing selection of an album
`COV.
`0033. Like reference symbols indicate like elements
`throughout the specification and drawings.
`
`DETAILED DESCRIPTION
`0034. Organizing images by correlating faces includes
`associating representations of the same person across images.
`FIG. 1 shows an exemplary overview of correlating graphical
`faces. In general, images can be transmitted from a camera to
`a computer. The computer can include a correlation system.
`The correlation system can correlate the images by grouping
`images of the same individual.
`
`0035 Digital camera 102 is a camera for capturing digital
`images. The digital camera 102 can capture video or still
`images, or both, by recording images via an electronic image
`sensor. The digital camera 102 can be a stand-alone digital
`camera or can be incorporated into any of several devices. For
`example, the digital camera 102 can be incorporated into a
`personal digital assistant (PDA), a mobile phone, or a web
`cam. Further, the digital camera 102 can utilize removable
`storage media to store captured image data. For example, the
`digital camera 102 can store images in a removable flash
`memory device (e.g., CompactFlash and Secure Digital
`cards). The digital camera 102 can record images in various
`formats, e.g., raw formatorin accordance with the standard of
`the joint photographic experts group (JPEG). The digital
`camera 102 also can include one or more interfaces for com
`municating with a computer. For example, the digital camera
`102 can include a wired interface, such as universal serial bus
`(USB) or FireWire, and/or a wireless interface, such as infra
`red or radio frequency.
`0036. The image 104 is a digital image produced by a
`digital camera (e.g., the digital camera 102). The image 104
`represents a face. The image 104 also can include represen
`tations of other items. For example, the image 104 can be an
`image of a person standing in front of a landmark, and can
`represent buildings, trees, the sky, etc. Further, the image 104
`can be stored in various formats, such as a compressed format
`(e.g., JPEG), or an uncompressed format (e.g., RAW). The
`image 104 can include graphical data and metadata. The
`metadata can conform to the exchangeable image file format
`(Exif) specification. The Exif data can include aperture, expo
`Sure time, date and time, location and other information about
`the image 104.
`0037. The digital camera 102 can capture images. For
`example, the images 106, 108, and 110 can be other images
`like the image 104. Each of the images 106, 108, and 110 also
`can represent a graphical face.
`0038. The computer system 112 can be configured to
`receive images transferred from a digital camera, Such as the
`digital camera 102. The computer system 112 can be a com
`puter (e.g., a desktop, a laptop, a server, etc.), a mobile tele
`phone (e.g., a Blackberry), a PDA, or other such device. The
`computer system 112 can include hardware configured to
`execute one or more software applications. Further, the com
`puter system 112 can include electronic storage media, Such
`as random access memory (RAM), flash memory, and a hard
`drive. Additionally, the computer system 112 can include one
`or more communications interfaces capable of receiving digi
`tal information, Such as images. The one or more communi
`cations interfaces can be wired or wireless.
`0039. The correlation system 114 can be configured to
`correlate graphical faces represented in images. The correla
`tion system 114 can be implemented in Software executing on
`a computer system (e.g., the computer system 112), hardware
`included in the computer system, or a combination thereof.
`The correlation system 114 can access images (e.g., digital
`images), analyze data associated with the images, and corre
`late two or more images determined to represent a common
`graphical face. For example, digital images can be analyzed
`and images including a face corresponding to the same person
`can be associated with one another to form an image group.
`0040. The correlation system 114 can process the images
`104, 106, 108, and 110, and generate one or more image
`groups. The image group 116 includes digital images, where
`each of the digital images represents a face corresponding to
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 19 of 28
`
`

`

`US 2010/0172551 A1
`
`Jul. 8, 2010
`
`the same individual. An image group, such as the image group
`116 can be a very large group of images (e.g., hundreds or
`thousands of images), a small group of images (e.g., ones or
`tens of images), or even a single image. Further, the same
`image can belong to multiple image groups. For example, an
`image including a person named Jack and another person
`named Mary can be included both in an image group associ
`ated with Jack and a different image group associated with
`Mary.
`0041. The image group 118 includes only one image. The
`image group 118 may include only one image for a variety of
`reasons, such as because the individual is represented in only
`one image, the individual whose face appears in the image has
`not yet been defined in the correlation system 114, or because
`the individual in the image is not recognized as someone
`already defined in the correlation system 114. Similarly, the
`image group 120 also can include only one image.
`0042. In operation, the images 104,106, 108, 110 can be
`transmitted from the camera 102 to the computer system 112.
`The correlation system 114 can run on the computer system
`112 and process the images 104,106, 108 and 110, such as in
`response to a command received from a user or automatically
`in response to the transmission of images. The correlation
`system 114 can determine that image 106 and image 110
`represent faces corresponding to the same person, and thus
`group image 106 and image 110 together in image group 116.
`Image group 116 also can include images already stored on
`the computer system 112 that represent the same person.
`Further, the correlation system 114 may not recognize one or
`more faces in an image. Such as image 104 and image 108,
`and thus can create image groups containing only a single
`image. Such as image group 118 and image group 120. Images
`containing an unrecognized face can be referred to as single
`tons. In some implementations, the correlation system 114
`can group some or all of the singletons in an image group.
`0043 FIG. 2 shows an exemplary system 200 for correlat
`ing graphical faces represented in images. Correlating can
`involve various hardware and Software components. Input,
`Such as images and identification information, can be
`received and used to correlate faces represented in digital
`images. Further, the results of the correlating process can be
`iteratively improved by receiving additional input and per
`forming additional correlation.
`0044 As shown in FIG. 2, the system 200 can include a
`digital camera 202. The digital camera 202 can be similar to
`the digital camera 102 in FIG. 1.
`0045. A computer system 204 can be similar to the com
`puter system 112 in FIG.1. In addition, the computer system
`204 can include hardware and software to receive and trans
`mit information over a network (e.g., the network 206). The
`computer system 204 also can be configured to transmit infor
`mation to and receive information from a storage device (e.g.,
`the storage 210).
`0046. A correlation system 212 can be configured to cor
`relate faces appearing in digital images. The correlation sys
`tem 212 can include Software and/or hardware components.
`Further, the correlation system 212 can be a stand alone
`Software application or can be implemented as part of a larger
`Software application. The correlation system 212 can be con
`figured to run in the background. Alternatively or addition
`ally, the correlation system 212 can include a user interface
`214 that enables a user to actively perform one or more
`management and/or processing functions.
`
`0047. The user interface 214 can be, e.g., a graphical user
`interface (GUI), a command line interface, a tactile interface,
`or a network interface for receiving input from a remote user.
`The user interface 214 can include mechanisms for input and
`output of commands and data, including digital images. For
`example, a user can input commands and other information to
`manipulate the correlation system 212. The user interface 214
`also can output information to the user through one or more
`interface Screens.
`0048. The user input module 216 can be configured to
`receive, store, and process input from an external Source. For
`example, the user input module 216 can receive input from a
`user identifying a person in a digital image. The identification
`can be used Subsequently in identifying other images of the
`same person. In addition, input can be received from other
`applications having relevant information. For example, data
`from an address book application can be used. The data can
`include images, names, email addresses, etc. The data can be
`used to seed the system with names. When a user is identify
`ing people, the system can Suggest names from the address
`book. Furthermore, the data can be used to seed the system
`with named faces. The data can include associations between
`names and faces. Advantageously, the data can be used to
`begin suggesting names to users before they have positively
`identified anyone (e.g., a user can have an image of a contact
`in an address book and the information can be used to Suggest
`images representing the contact).
`0049. User input also can be received through a network
`206 (e.g., the Internet) from a remote server 208 (e.g., a
`website). For example, if a user uploads images to a Social
`network site (e.g., Facebook), the user and the user's friends
`can identify faces in the images. The identifications can be
`received as input and used in correlating images. Differing
`levels of confidence can be assigned to input received through
`the network (e.g., identifications by a user can be more con
`fidently relied upon than identifications by the user's friends).
`0050 A preprocessing module 218 can be configured to
`process an image prior to passing the image to a facial analy
`sis engine 220. The preprocessing module 218 can perform
`functions including applying rules and heuristics that can
`improve the performance of the facial analysis engine 220.
`For example, the preprocessing module 218 can apply a rule
`stating that an individual can only appear in an image once.
`0051. The facial analysis engine 220 can be configured to
`detect and recognize one or more faces represented in images.
`The facial analysis engine 220 can receive an image and
`settings, and can detect whether the image represents a face
`corresponding to a person. The settings can include param
`eters that affect the operation of the facial analysis engine
`220. Such as thresholds indicating, e.g., the level of confi
`dence the facial analysis engine 220 should have about a
`detection before providing information about the detection
`(e.g., to the post-processing module 222).
`0.052
`Further, the facial analysis engine 220 also can cor
`relate faces by comparing a graphical face to other graphical
`faces to determine if they belong to the same person. The
`facial analysis engine 220 can be configured to use algorithms
`Such as eigenface or fisherface, or techniques such as a neural
`network to perform facial recognition. Additionally, the facial
`analysis engine 220 can be implemented as a library of facial
`analysis tools, such as the tools provided by OMRON Corp.
`of Schaumburg, Ill. in the OKAO Vision suite.
`0053. The post-processing module 222 can be configured
`to receive information from the facial analysis engine 220.
`
`UNIFIED PATENTS EXHIBIT 1007
`Page 20 of 28
`
`

`

`US 2010/0172551 A1
`
`Jul. 8, 2010
`
`Further, the post-processing module 222 can apply rules
`based on the input received from the engine 220 to improve
`the performance of the engine 220. For example, the post
`processing module 222 can analyze an image to determine
`whether the portion of the image asserted by the facial analy
`sis engine 220 to be a face includes a skin tone or appears
`sharp. If one or both of the conditions is failed, the module
`222 can disregard the detection of a face. Skin tone and
`sharpness are discussed more fully below with respect to FIG.
`4. Further, the module 222 can resubmit information to the
`engine 220 for processing again with new settings based on
`the post processing.
`0054 An induction module 224 can use inductive reason
`ing to make educated guesses as to whom a face may belong.
`For example, if the induction module 224 believes that two
`images represent the face of the same person (e.g., as received
`from the post-processing module 222), and a user identifies
`the person in one of the images (e.g., as received from the user
`input module 216), the induction module 224 can induce that
`the other image is also of the same person.
`0055. A correlation module 226 can be configured to use
`the information provided by one or more of the other modules
`included in the correlation system 212 to correlate faces
`appearing in digital images. The correlation process can
`include organizing images into groups based on the faces
`represented in the images. In particular, a group can be
`formed that contains all of the images accessible to the system
`that are known or believed to belong to a certain individual.
`0056 FIG. 3A is a flow chart showing an exemplary cor
`relation process 300. The correlation process 300 iteratively
`adjusts a correlation threshold to produce an acceptable num
`ber of groups of images. The groups can be referred to as
`"buckets. Each bucket can contain one or more images
`believed to represent the same person. If the correlation
`threshold is set very high, the process 300 will not correlate
`two images unless it is virtually certain that the images rep
`resent the face of the same person. Thus, if the correlation
`threshold is set very high, more buckets with fewer images in
`each will be generated as Some images of the same individual
`likely will not be correlated. If the correlation threshold is set
`very low, the process 300 can frequently correlate faces that
`may not correspond to the same person. Thus, setting the
`correlation threshold to a low value can produce fewer buck
`ets with more images in each. However, the likelihood is
`increased that two images will be correlated that do not rep
`resent a face corresponding to the same individual. Ideally, a
`bucket will include as many images as possible of the same
`person and few or no images that do not represent that person.
`0057 Images representing faces can be received (302).
`Further, the threshold can be set to a default value (304). The
`default value can be an intermediate value. Once the default
`value has been set, images can be correlated, such that images
`believed to represent the face of the same person are associ
`ated with a single bucket (306). It then can be determined
`whether the number ofbuckets generated is acceptable (308).
`If the number of buckets is acceptable (e.g., there are not
`Substantially as many buckets as images and not very few
`buckets relative to the number of images), the process 300 can
`determine whether too many singletons are remaining (310).
`Singletons can be images containing one or more faces for
`which the process 300 cannot identify a corresponding per
`son. Having too many singletons can be a sign that the thresh
`
`old value is set too high and that faces thus are not being
`correlated appropriately with other images containing faces
`of the same person.
`0058 If it is determined that there are too few buckets, the
`threshold value can be increased (314). Increasing the thresh
`old value requires the process 300 to have additional confi
`dence that faces in a bucket belong to the same person. Once
`the threshold value is increased, the process 300 once again
`performs correlation of the images (306).
`0059. If it is determined that there are too many buckets,
`the threshold value can be decreased (316). In addition, if the
`process 300, at 310, determines that there are too many
`singletons (e.g., indicating the degree of confidence required
`to assign an image to a bucket is too high), the threshold value
`also can be decreased (316). Decreasing the threshold value
`allows the process 300 to correlate images based on a lower
`degree of confidence. Once the threshold value is decreased,
`the process 300 once again performs correlation of the images
`(306).
`0060. If it is determined that there are an acceptable num
`ber of singletons (e.g., relative to the total number of images),
`the process 300 can continue by receiving user input (312).
`For example, the user input can include information about
`whose faces are represented in one or more images.
`0061
`FIG. 3B is a flow chart showing an exemplary pro
`cess 350 for setting an adaptive correlation threshold. The
`correlation threshold can be automatically adjusted in
`response to one or more factors, including the number of
`images representing an individual's face that have been asso
`ciated with a bucket. A threshold value for correlation confi
`dence can be set on aperbucket basis. Initially, the correlation
`threshold can be set at a moderate value (352). For instance,
`the correlation threshold can be set to a value that will facili
`tate the identification of initial matches. As matches are made,
`the bucket size increases (354). For example, more images
`can be added to the bucket as new images are added to the
`system or more faces in images can be identified as represent
`ing a given individual, e.g., as