a2, United States Patent
`US 10,417,275 B2
`(0) Patent No.:
`
`Frigon Sep. 17, 2019 (45) Date of Patent:
`
`
`US010417275B2
`
`ARTIFICIAL INTELLIGENCE ASSISTED
`TAGGING OFUSERS IN DIGITAL MEDIA
`ONLINE
`
`(58) Field of Classification Search
`CPC wie ceeeeeeee GOGE 16/58; GOGE 16/583
`See application file for complete search history.
`
`(54)
`
`1)
`
`(72)
`
`(73)
`
`Applicant: Angel ‘lechnologies, LLC, San
`Francisco, CA (US)
`
`Inventor: Mark Frederick Frigon, San
`Francisco, CA (US)
`
`Assignee: Angel Technologies, LLC, San
`Francisco, CA (US)
`
`Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`USS.C. 154(b) by 0 days.
`
`(21)
`
`Appl. No.: 15/933,531
`
`(22)
`
`(65)
`
`(63)
`
`(60)
`
`1)
`
`(52)
`
`Filed:
`
`Mar. 23, 2018
`
`Prior Publication Data
`
`US 2018/0218008 Al
`
`Aug. 2, 2018
`
`Related U.S. Application Data
`
`Continuation of application No. 14/053,626, filed on
`Oct. 15, 2013, now Pat. No. 9,959,291, which is a
`continuation of application No. 09/991,324,filed on
`Nov. 15, 2001, now Pat. No. 8,954,432.
`
`Provisional application No. 60/248,994,filed on Nov.
`15, 2000.
`
`Int. Cl.
`G06F 1630
`GU6E 16/583
`G06F 16/58
`G06F 3/0482
`US. Cl.
`CPC oe. GO6F 16/583 (2019.01); GO6F 3/0482
`(2013.01); G06F 16/58 (2019.01)
`
`(2019.01)
`(2019.01)
`(2019.01)
`(2013.01)
`
`(56)
`
`References Cited
`
`
`U.S. PATENT DOCUMENTS
`
`4,847,604 A
`5,063,603 A
`5,553,281 A
`5,721,851 A
`5,802,361 A
`5,893,110 A
`5,937,413 A
`6,014,644 A
`6,029,141 A
`
`7/1989 Doyle
`11/1991 Burt
`9/1996 Brown
`2/1998 Cline
`9/1998 Wang
`4/1999 Weber
`8/1999 Tlyun
`1/2000 Erickson
`2/2000 Bezos
`(Continued)
`
`OTHER PUBLICATIONS
`
`Article entitled “Direct annotation: a drag-and-drop strategy for
`labeling photos”, dated. Jul. 21, 2000, by Schneiderman et al.*
`(Continued)
`
`Primary Examiner — Mahesh H Dwivedi
`(74) Attorney, Agent, or Firm — Nixon Peabody LLP
`
`(57)
`
`ABSTRACT
`
`A system, computer program, and method for storing and
`sharing images such as photographs via a communications
`network (16) and for permitting the identification of objects
`within the images. The inventionallowsthe identification of
`objects such as persons within the photos without requiring
`the person submitting the photos to type in identification
`information for each and every photo in a photo album. The
`invention also allows users to automatically share their
`photos with others and to automatically search for photos
`and/or certain people in photos.
`
`12 Claims, 9 Drawing Sheets
`
`14
`
`&
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 001
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 001
`
`

`

`US 10,417,275 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`6,038,333 A
`6,097,853 A
`6,216,133 B1
`6,236,978 BL
`6,288,717 Bl
`6,332,193 BI
`6,353,452 B1
`6,411,949 BI
`6,442,573 B1
`6,496,594 B1
`Bl
`Bl
`Bl
`Bl
`
`6,542,956
`6,545,660
`6,556,709
`6,587,593
`
`6,616,701
`6,687,878
`6,829,607
`6,931,419
`6,973,201
`7,010,751
`7,028,253
`7,068,309
`7,209,149
`7,243,079
`7,576,752
`8,954,432
`9,959,291
`2001/0037721
`2002/0039447
`
`2002/0054059
`2002/0055955
`2002/0059201
`2002/0069222
`2002/0093678
`
`B2
`Bl
`Bl
`Bl
`Bl
`B2
`Bl
`B2
`B2
`Bl
`Bl
`B2
`B2
`Al
`Al
`
`Al
`Al
`Al
`Al
`Al
`
`me
`
`me
`
`3/2000
`8/2000
`4/2001
`5/2001
`9/2001
`12/2001
`3/2002
`6/2002
`8/2002
`12/2002
`
`4/2003
`4/2003
`4/2003
`7/2003
`
`9/2003
`2/2004
`12/2004
`8/2005
`12/2005
`3/2006
`4/2006
`6/2006
`4/2007
`7/2007
`8/2009
`2/2015
`5/2018
`11/2001
`4/2002
`
`5/2002
`5/2002
`5/2002
`6/2002
`7/2002
`
`Wang
`Gu
`Masthoff
`Tuzhilin
`Dunlde
`Glass
`Hamada.
`Schaffer
`Schiller
`Prokoski ...........
`
`Mayle
`Shen
`Kumar
`Matsuoka ............
`
`AGIB 5/1176
`382/118
`
`.. GO6T 5/50
`382/260
`
`Doyle
`Eintracht
`Tafoya
`Linguist
`Colmenarez
`Shneiderman
`Lieberman
`Toyama
`Jogo
`Manolis
`Benson
`Frigon
`Frigon
`Hasegawa
`Shniberg ........... GO6F 17/30259
`382/224
`Schneiderman
`Lloyd-Jones
`Work
`McNeely
`Skidgel 0.0.0... GO6F 17/30265
`358/1.15
`
`2002/0103813 Al
`2002/0184203 Al
`2003/0076363 Al
`
`8/2002 Frigon
`12/2002 Nastar
`4/2003 Murphy
`
`OTHER PUBLICATIONS
`
`Article entitled “FotoFile: A Consumer Multimedia Organization
`and Retrieval System”, by Kuchinskyet al., dated May 20, 1999,*
`Bakic, V.et al., “Menu Selection by Facial Aspect,” Vision Interface
`99, May 21, 1999. pp. 203-209 (7 pages).
`Ghidary, S.S. et al., “Human detection and localization at indoor
`environment by home robot,” IEEE International Conference on
`Systems, Man and Cybernetics, Oct. 11, 2000 (10 pages).
`TIan, C-C. et al., “Cast T'ace Detection via Morphology-Based
`Pre-processing,” ResearchGate, Jan. 1997 (24 pages),
`Kang,H. et al., “Visualization Methods for Personal Photo Collec-
`tions: Browsing and Scarching in PhotoFinder,” IEEE International
`Conference on Multimedia and Expo, Aug. 2, 2000 (6 pages).
`Rex, F.. “Online Photo Delivery,” Upside, Jun. 2000, pp. 331-338
`(6 pages).
`Stewart, J. et al., “Single Display Groupware: A Model for Co-
`present Collaboration,” May 20, 1999, pp. 286-293 (8 pages).
`Chang, S-F, “Content-Based Indexing and Retrieval of Visual
`Information,” INTE Signal Processing Magazine,vol. 14, No. 4, Jul.
`1997, pp. 45-48 (4 pages).
`Meyer, E.A. ct al., “Borcalis Image Server,” Computcr Networks
`and ISDN Systems, May 28, 1996, pp. 1123-1137 (15 pages).
`Swain, M.J., “Searching for Multimedia on the World Wide Web,”
`Proceedings of the 1999 IFT. Conferences on Multimedia Com-
`puting and Systems, Jun. 1999, pp. 32-37 (6 pages).
`Wei, G. ct al., “Face Detection for Image Annotation,” Pattern
`Recognition Letters, 20, 1999, pp. 1313-1321 (© pages).
`Kuchinsky, et al., “FotoFile: A Consumer Multimedia Organization
`and Retrieval System,” May 1999, ACM, pp. 496-503 (8 pages).
`Schneiderman et al., “Direct Annotation: A Drag-and-Drop Strategy
`for Labeling Photos,” Jun. 2000 (8 pages).
`
`* cited by examiner
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 002
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 002
`
`

`

`U.S. Patent
`
`Sep. 17, 2019
`
`Sheet 1 of 9
`
`US 10,417,275 B2
`
`
`Ke
`
`Figure 7
`
`database
`
`240
`
`250
`
`identifications
`dstabase
`
`|
`
`images
`database
`
`FiELOS
`Jeroenvronnsnmrenenannusennnsasssinnsnted
`i imege 12
`oe
`}
`Location
`Foannes
`cseonsessnann
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 003
`
`Users
`dalabase
`alabtatlaiaMtteabbatelee,
`"FIELDS
`User i
`seanevesevonenuatentenvenesesonsasects
`‘Name
`|Ema” a ct
`| EmPei7 i
`
`3senborceeensiedecswenteetien
`
`Rat
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 003
`
`

`

`U.S. Patent
`
`Sep. 17, 2019
`
`Sheet 2 of 9
`
`US 10,417,275 B2
`
`Figure 3
`
`rr aos
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 004
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 004
`
`

`

`U.S. Patent
`
`Sep. 17, 2019
`
`Sheet 3 of 9
`
`US 10,417,275 B2
`
`Figure 4
`
`‘\ CONTACTS
`© John Doe
`
`& Jane Doe
`
`O#7
`
`© O O
`
`s
`
`QO #6
`
`38
`
`suit )
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 005
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 005
`
`

`

`U.S. Patent
`
`Sep. 17, 2019
`
`Sheet 4 of 9
`
`US 10,417,275 B2
`
`Figure 5
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 006
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 006
`
`

`

`U.S. Patent
`
`Sep. 17, 2019
`
`Sheet 5 of 9
`
`US 10,417,275 B2
`
`Figure 6
`~808
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 007
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 007
`
`

`

`U.S. Patent
`
`Sep. 17, 2019
`
`Sheet 6 of 9
`
`US 10,417,275 B2
`
`Figure 7
`
`<. PEOPLEIN IMAGE
`
`a2
`
`Jane Doe
`
`58
`
`PEOPLE IN IMAGE
`as
`Newer,
`
`dim Due
`Sue Doe
`
`Chris Doe
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 008
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 008
`
`

`

`U.S. Patent
`
`Sep. 17, 2019
`
`Sheet 7 of 9
`
`US 10,417,275 B2
`
`Figure 8
`
`CLT,
`
`OBTAIN'A REQUESTFoR EO?
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 009
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 009
`
`

`

`U.S. Patent
`
`Sep. 17, 2019
`
`Sheet 8 of 9
`
`US 10,417,275 B2
`
`Figure 9
`
`Cursor not over
`
`
`
`
`
`;
`_ Photos including Jane
`
`$2a
`
`After Clicked
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 010
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 010
`
`

`

`U.S. Patent
`
`Sep. 17, 2019
`
`Sheet 9 of 9
`
`US 10,417,275 B2
`
`Figure 10
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 011
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 011
`
`

`

`US 10,417,275 B2
`
`1
`ARTIFICIAL INTELLIGENCE ASSISTED
`TAGGING OF USERSIN DIGITAL MEDIA
`ONLINE
`
`
`
`CROSS REFERENCE TO RELATED
`
`
`APPLICATIONS
`
`This application is a continuation of U.S. patent applica-
`tion Ser. No. 14/053,626, filed Oct. 15, 2013, now allowed,
`which is a continuation of U.S. patent application Ser. No.
`09/991,324, filed Nov. 15, 2001, now U.S. Pat. No. 8,954,
`432, which claims priority to and the benefit of U.S. Pro-
`visional Patent Application No. 60/248,994, filed Nov. 15,
`2000, the contents of which are each herebyincorporated by
`reference herein in their entireties.
`
`
`
`BACKGROUND OF THE INVENTION
`
`Field of the Invention
`
`Thepresent invention relates to computer software. More
`particularly, the invention relates to a method and apparatus
`for storing, and sharing images such as photographs via a
`communications network and for permitting the identifica-
`tion of objects and the location of the objects within the 2
`images. The invention enables users to supply and/or receive
`information about the existence of objects within images.
`Portions of this patent document contain material that is
`subject to copyright protection. The copyright owner has no
`objection to the facsimile reproduction by anyone of the
`patent documentor the patent disclosure as it appears in the
`Patent and Trademark Office file or records, but otherwise
`reserves all copyrights whatsoever.
`
`30
`
`Background
`
`Digital photography allows people to quickly andcasily
`take, print, and/or share photos without film and the asso-
`ciated film developing costs. The popularity of digital pho-
`tography has increased dramatically due to the increased
`availability of digital cameras and the widespread accep-
`tance of the Internet. Many Internet users have created
`websites that organize their digital photographs into online
`photo albums that can be accessed byother Internet users.
`These web pages permit a large number of people to view
`the photos and allow the author to both add and remove
`photos from an album.
`Although creating web pages for online photo albums
`offer advantages over traditional photo albums, their cre-
`ation requires programming expertise such as skill in Hyper-
`text Markup Language (HTML). Many people do not have
`the technical knowledge and/or time to build their own
`online photo albums. Sharing of photos from online photo
`albumsis also problematic and. requires that each photo be
`manually sent, e-mailed, or downloaded. If the recipient
`wishes to use the photos in their own web pages, each photo
`must be then uploaded to an additional site. These steps are
`time consuming, duplicative, and also require programming
`knowledge.
`To address these problems, several websites that special-
`ize in online photo albums have been created. These web-
`sites allowusers without programming skills to quickly and
`easily create and maintain online photo albums. Users are
`only required to upload photos that they wish to add to an
`online photo album. Thesites, through programming on the
`webserver, or “server-side scripting”, then create all neces-
`sary HTML programming to add the uploaded photos to the
`
`35
`
`40
`
`45
`
`i2
`5
`
`35
`
`60
`
`65
`
`2
`users’ photo albums. Manyof these sites also offer image
`enhancement capabilities (e.g. sharpen, brighten, reduce
`“red-eye”, etc.), as well as photo printing and developing
`functions. Photo-specific sites have also eased the process of
`sharing photos by providing a meansfor users to quickly and
`easily e-mail links to photos or photos themselves to one
`another and, in some cases, contribute photos directly to
`other’s photo albums.
`While websites that specialize in the creation of online
`photo albums offer many advantages,
`they suffer from
`several limitations. For example, existing websites do not
`offer users the ability to identify objects within photos. Ifa
`user is Viewing a photo and does not knowcertain individu-
`als in the picture, the user is unable to use current systems
`to determine their identities. Typically, photo album sites
`offer the ability to describe uploaded photos through the use
`ofcaptions or other descriptive fields. These fields may be
`used. to describe or name individuals within the photo. lor
`example, a user mayupload a photo and giveit a caption of
`“Stephanie rollerblading with Tauny”. This caption alone is
`not sufficient information to ensure the viewer can correctly
`determine the identities of the individuals within the photo,
`because to do so would require prior knowledge of which
`person is “Stephanie” and which is “Tauny”. A viewer who
`does not know these individuals may have difficulty or be
`unable to correctly identify the individuals within the photo.
`Turthermore, less descriptive captions such as ‘“The amily
`at the beach” wouldoffer little help unless the viewer knows
`each individual
`in the photo. Thus,
`in order to provide
`sufficient information required to ensure a viewer can iden-
`tify individuals within a photo, copious descriptions must be
`input and maintained by users. Typing such captions or
`descriptions is very time consuming and tedious, especially
`when a large number of photos are uploaded. Furthermore,
`when the same person appears in multiple photos,
`the
`submitter is still required. to retype the identification imfor-
`mation into a caption for cach photo.
`Processes have been developed to identify individuals
`within photos. For example, one process requires people to
`wear badges containing identification information such as
`unique color codings when their photos are taken. When a
`computer processes a photo containing a badge, the com-
`puter can access a database to determine which badge was
`assigned to which person and thenidentifies the person as
`existing in this photo. This process has many drawbacks
`including the dependence on physical badges. If the badges
`cannot be seen in the photos, the people wearing the badges
`cannot be identified. Moreover, this process is not functional
`for the vast majority of photographs: any photos that were
`taken outside of this tightly controlled environment.
`Another problem with photo-specific websites is that they
`have no search capabilities that permit users to search for
`photos of specific individuals. Users wishing, to view all
`photographs that contain a particular person must manually
`access all possible photo albumsto find the desired photos.
`For example, if a user wishes to find all pictures of a
`particular friend,
`the user may have to browse through
`severalofhis/her own albums wherethat friend may appear
`(e.g. “High School pictures”, “Reunion Weekend”, etc.), as
`well as access and browse through each of the friend’s
`albums, or any third-party’s albums where the friend may
`appear. With a reasonable number of photos, this becomes
`time-prohibiting if not impossible. Search features could be
`designed to search aforementioned descriptivefields (c.g. a
`caption), however,as there is no naming standard, people are
`described differently by different people (e.g. first name, last
`name, full name, nickname, etc.) and many names would be
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 012
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 012
`
`

`

`US 10,417,275 B2
`
`a 5
`
`IcS
`
`3
`used to describe multiple people (e.g. Mom, Dad, John,etc.).
`As a result,
`it would be difficult
`to have a one to one
`relationship between descriptions of people and people
`themselves. Consequently, searches would be unreliable,
`likely returning many photos oflittle relevance and failing
`to return photos that would be of relevance. Tor example, a
`search for the term “John Smith” would fail to return photos
`in which this person was referred to as “John”or, perhaps,
`“Dad”. However,
`this same search likely would return
`photos of other individuals who share the same name “John
`Smith”.
`While reliable database searchability for digital images
`over the Internet is available, it has not been implemented
`with photo albums. Particularly, several companies which
`sell and license images have put catalogs of digital photo-
`graphs on the Internet. Such websites generally describe
`images in terms of themes(e.g. sunset, dark or sad) or broad
`descriptions of objects therein (e.g. beach, farmland, or
`water), without
`identifying the objects explicitly. For
`example. a picture of a man and womanon the beach at
`sunset, which could be licensed to a beachresort for print or
`web use, could be identified as “beach”, “sunset”, and/or
`“man and woman”. While such descriptive terms can pro-
`vide a reliable mechanism for searching for images match-
`i]a
`ing the terms, they do not uniquely identify any objects in 2
`the images, nor do they have the capability of identifying
`where these objects appear within the images. For example,
`a search for the term “clown” is likely to return many
`pictures of clowns, however, it is very unlikely that every
`clown pictured would be the sameperson, just as a search for
`the term “beach” is unlikely to return pictures of the same
`beach. Even a search for the term “President” will return
`several photos of different people. Furthermore, the objects
`that appear within a photo are not identified as being in a
`particular area within the photo, they merely are described
`as being in the photo. For example, websites that return a
`photograph from a “President” search do not have the
`capability of identifying that the Presidentis in, perhaps, the
`lowerright cornerofthe picture. Such technologies have the
`capability to describe the nature or themeof a photo but not
`to uniquely identify who or what is in the photo or where in
`the photo it appears.
`Anotherrelated limitation ofprior art websites is that the
`captions described above contain static information only,
`such as names, and do not provide a wayto obtain addilional
`information about a person or object such as contact infor-
`mation or to locate additional photos of the person or object.
`Another problem with photo-specific websites is that
`users Wishing to share their photos must proactively e-mail
`or otherwise send their photos to others (i.e., a photo is not
`automatically shared whena user is described in a photo).
`For example, even if a user describes a photos as containing
`“Jane@doe.com” and “John@doe.com”, existing website
`will not share the photos with these individuals. Through a
`separate process,
`the user must still email this photo or
`otherwise share the phota with, in this case, Jane and John.
`Although photo-specific websites ease this process,it is still
`time consuming for users with large photo albums and/or a
`large number of friends to send their photos to all such
`persons.
`
`5
`
`5
`
`60
`
`SUMMARY OF THE INVENTION
`
`An embodimentof the invention comprises a system and
`method for identifying and displaying objects (in most
`cases, a person) within a digital image over a system of
`interconnected computers (e.g. client/server, peer-to-peer,
`
`4
`etc... .). For example, a user can identify where a person
`exists in an image by selecting the person’s name(e.g. from
`a list) and clicking on the person’s location in the image. By
`identifying people within images,
`the system is able to
`search for images where a specified person(s) appears as
`well as generate output that recognizes where in the image
`a person appears. This output is also capable of creating
`hyperlinks and calls to client-side scripts from the otherwise
`“static” image file. In an embodiment of the invention the
`system can also utilize artificial intelligence algorithms to
`locate other images where the identified object has not yet
`been defined bya user of the system.
`While, one embodiment of this invention utilizes the
`Internet as the medium for communication and web pages
`for displaying data, the scope of this invention is not limited
`specifically to the Internet and/or to web pages. The inven-
`tion contemplates the use of any display output in a com-
`puter environment (e.g. any types of document). The terms
`person, people, and/or object are used interchangeably and
`meant to refer to any object(s) (e.g. a pet, a possession, a
`place, or a thing) a user wishes to identify within a digital
`image. ‘his most frequently will, in fact, be a person. When
`shown within the bounds of an image, a person is considered
`akin to any type of object. Thus, the reader should note that
`the invention can be utilized to identify any type of object
`within an image.
`Depending on the computer environment, different media
`will be used for output and display to the clicnt. On the
`Internet, the most accessible and most common output is
`Hypertext Markup Language (“HTML”). However,
`the
`functionality of HTML can be duplicated or configured in
`other non-HTML media. For the sake of both brevity and
`clarity, I will outline an embodimentof the present invention
`that utilizes HTML. This embodiment of the present inven-
`tion (i.e. utilizing the Internet and HTML)is logical due to
`the broad reach of the Internet and wide acceptance of
`HTML. Howeverthe scope of the invention is not intended
`to be limited to the use HTMLorthe Internet.
`
`DESCRIPTION OF THE DRAWINGS
`
`An embodiment of the invention is described in detail
`below with reference to the attached drawing figures,
`wherein:
`FIG. 1 is a schematic diagram of computer and commu-
`nicalions equipment
`that may be used to implement an
`embodiment ofthe present invention;
`FIG. 2 is a schematic diagram of databases that may be
`used to implementcertain aspects of the invention;
`FIG.3 is a flow chart illustrating a high-level overview of
`the steps of an embodiment of the invention.
`VIG. 4 is a schematic diagram of an identifying page that
`may be used when identifying objects within an image in
`accordance with an embodiment of the invention;
`FIG. 5 further describes the general details relating to the
`process for requesting, generating, and transmitting an iden-
`tifying page to the requestor(e.g., the client computer).
`FIG.6 is a flow diagram illustrating a general overviewof
`the process for using the identifying page to obtain identi-
`fying information (e.2.,
`information associated with
`object(s) in an image) from the user.
`FIG.7 is a schematic diagram of a display page that may
`be used to display images and information about objects that
`have beenidentified within the image in accordance with an
`embodiment ofthe invention.
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 013
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 013
`
`

`

`US 10,417,275 B2
`
`5
`FIG.8 is a flow chart illustrating a high-level view ofthe
`process for providing display page to a requesting user in
`accordance with an embodimentof the invention.
`FIG. 9 illustrates various ways to display identification
`information,
`including the various states a user interface
`configured to operate in accordance with an embodiment of
`the invention may have.
`FIG. 10 is a flow chart describing the process for gener-
`aling a set of search results in response to a request.
`‘The figures do not limit the invention to the specific
`embodiments disclosed and described herein. The drawings
`are not necessarily to scale, emphasis instead being placed
`upon clearly illustrating the principles of the invention.
`
`0
`
`DETAILED DESCRIPTION
`
`6
`those operating with Microsoft’s PocketPC™ operating
`software, mobile phones, or even Internet appliancesthat are
`specifically designed for accessing the Internet such as
`Webl'V'™. Each client computer 100 may also include or
`can access a conventional Internet connection such as a
`modem, Digital Subscriber Line (DSL) converter, or Inte-
`grated Service Digital Network (ISDN) converter and a web
`browser that permits it to access and view data over the
`Internet.
`The communications network 150 maybethe Internet but
`can also be any other communications network such as a
`local area network, a wide area network, a wireless network,
`an intranet or a virtual private network. Hereinafter, the term
`Internet or communication network refers to any type of
`interconnection fabric that provides computers coupled to
`the network with a mechanism for transmitting and/or
`receiving data.
`The computer program or programs embodying one or
`more aspects of the invention are stored in or on computer-
`readable medium residing on or accessible by host computer
`200 and provide a mechanismfor instructing, host computer
`200 to operate the invention as described herein. ‘the com-
`putcr programs typically comprise ordered listings of
`executable instructions for implementing logical functions
`in host computer 200 and user computers 100 coupled with
`host computer 200. The computer programs can be embod-
`ied in any computer-readable medium for use by or in
`connection with an instruction execution system, apparatus,
`or device, such as a computer-based system, processor-
`containing system, or other systemthat can fetch the instruc-
`tions from the instruction execution system, apparatus, or
`device, and execute the instructions. In the context of this
`application, a “computer-readable medium” can be any
`means that can contain, store, communicate, propagate or
`transport the program for use byor in connection with the
`instruction execution system, apparatus, or device. The
`computer-readable medium can be, for cxample, but not
`limited to, an electronic, magnetic, optical, electro-mag-
`netic, infrared, or semi-conductor system, apparatus, device,
`or propagation medium. Examples of the computer-readable
`medium would include the following: an electrical connec-
`tion having one or more wires, a portable computerdiskette,
`a random access memory (RAM), a read-only memory
`(ROM), an erasable, programmable,
`read-only memory
`
`(EPROMorFlash memory), an optical fiber, and a portable
`compact disk read-only memory (CDROM). The computer-
`readable medium could even be paper or another suitable
`medium upon which the programis printed, as the program
`can be electronically captured, via for instance, optical
`scanning of the paper or other medium,
`then compiled,
`interpreted, or otherwise processed in a suitable manner, if
`necessary, and then stored in a computer memory.
`In an embodiment of the invention, host computer 200
`comprises server engine 220 which is programmed to oper-
`ate or host a website and serve as a repository for images and
`identification information for objects within the images as
`described in more detail below. The images may be photo-
`graphs, graphics, artwork, or any other digital image that
`contains or depicts one or more objects. The objects within
`the images may include people, animals, plants, buildings,
`places, or anything else shown in images. In one embodi-
`ment of the invention, the images and objects are referred to
`(but not limited to) as photographs and people, respectively.
`The reader should note that the term object as used herein
`includes anyidentifiable entity within the image data.
`Server engine 220 can access web page templates 210,
`databases 230, 240, and 250, and internal or external
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 014
`
`IcS
`
`The invention described below is a method and apparatus
`for obtaining information relating to the existenceofat least
`one object in an image. In the following description, numer-
`ous specific details are set forth to provide a more thorough
`description of the present invention. It will be apparent,
`however, to one skilled in the art, that the present invention
`may be practiced without these specific details. In other
`instances, well known features have not been described in
`detail so as not to obscure the present invention.
`General System Architecture
`Lmbodiments of the invention can be implemented in
`hardware, software, firmware, or a combination thereof. In
`one or more embodiments, however, the invention is imple-
`mented with one or more computer programs that operate
`one or more pieces of computer equipment broadly referred
`to by the numeral 101 in FIG. 1. Such computer equipment
`101 typically includes a server computer 200 (also referred
`to as a host computer) and any number of user computers
`100 (e.g., client computers) that may access the host com-
`puter 200 via a communications network 150. The computer
`programs and equipmentillustrated and described herein are
`merely examples of programs and equipment that may be
`used to implement an embodimentof the invention and may
`be replaced with other software and computer equipment
`without departing from the scope of the invention. For
`example, any computational device configured to display
`image data may be configured to implement one or more
`aspects of the invention.
`Ilost computer 200 may be any computing device such as
`a network computer running Windows 2000, Novel Net-
`ware, Unix,or any other network operating system. The host
`computer 200 may be connected to a firewall computerat the
`boundaries of network 150 to prevent tampering with infor-
`i2
`mation stored on or accessible by the host computer 200 and 5
`to another computing device that may operated by an
`administrator of the host computer 200. If the invention is
`implemented with the Internet, the host computer 200 may
`include conventional web hosting operating software, an
`Internet connection such as a modem, DSL converter or
`ISDN converter, and be assigned an IP address and corre-
`sponding domain nameso that the website hosted thereon
`can be accessed via the communications network 150.
`Client computer(s) 100 may be used by anyone wishing to
`upload and/or view photographs via host computer 200. The
`client computers 100 may be any device or group of devices
`that permits access to host computer 200 via communica-
`tions network 150, including, but not limited to, personal or
`laptop computers such as those manufactured and sold by
`Dell™, Compag™, Gateway™, or any other computer
`manufacturer, handheld personal assistants such as those
`operating with Palm’s PalmOS™ operating software or
`
`45
`
`5
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 014
`
`

`

`US 10,417,275 B2
`
`8
`The user, John Doe, may then upload a photo, “my_im-
`age.jpg”, which maybe assigned an image I.D. of “ABCD”
`in Images database 250. Images database 250 could then be
`populated to include the following record:
`
`5
`
`Image LD. Location
`
`Caption
`
`Uploaded by
`
`
`
`To identify the user, John Doe, with this image, my_im-
`age.jpg, host computer 200 inserts into Identifications data-
`base 240 a record with John Doe’s UserI.D.(i.e. “007”) in
`the User |.D. field and my_image.jpg’s Image ID. (ie.
`“ABCD”)in the Image LD.field. ‘This creates a relationship
`betweenthis user and this image. Assume nowit is desired
`to identify John Doe as appearing in a certain area within the
`image. In IITML, image coordinates are used to define an
`area within an image. By storing the image coordinates in
`this record, we can define specifically where within my_im-
`age.jpg that John Doe appears. Assuming John Doe appears
`in the circular area bound by the coordinates 50,50,25
`(x-axis location, y-axis location, radius of circle), the Iden-
`tifications database 240 entry would appear as follows:
`
`User
`LD.
`
`007
`
`Image
`LD.
`
`ABCD
`
`Shared by
`
`Coordinates
`
`G07
`
`50, 50, 25
`
`Similarly, another user, Jane Doe, who also appears in the
`photo uploaded by John Doe, can be identified in my_im-
`age.jpz by inserting an additional entry into the Identifica-
`tions database 240. The database entries could now be the
`following.
`
`User database
`
`User 1.D. Name
`
`Email
`
`Contacts
`
`007
`008
`
`John Doe
`Jane Doe
`
`jdoe@doe.com
`janedoe@doe.com
`
`John,Jane, Lisa
`Jane, John, Fuzzie
`
`30
`
`35
`
`40
`
`45
`
`35
`
`Image LD. Location
`
`Caption
`
`Uploaded by
`
`Images database
`
`7
`memory. Web page templates 210 can be any technology
`used to transcribe data from a data source into a web page
`or otherwise provide “dynamic” or customized documents,
`including but not
`limited to, Active Server Pages, Java
`Server Pages, Cold Fusion, PHP, CGI scripts, or even
`client-side scripts. Server engine 220 receives information
`and requests from user computer(s) 100 via communications
`network 150, processes web page templates 210 by tran-
`scribing data from data sources, including databases 230,
`16
`ABCD=Cxdir\my_image.jpg John and Jane at 007
`
`240, and 250, into templates 210, and sends the resulting
`the beach
`web page or document to requesting client computer 100 as
`described in more detail below.
`In order to implement the invention host system 200 will
`access a server system or database system configured to
`support the functionality described. Referring back to FIG.
`2, one or more databases could be utilized to store the data
`utilized by the system. I'or instance, the system mayutilize
`a users database 230, Identifications database 240, and
`Images database 250 as depicted in FIG. 2.
`Users database 230) receives, stores, and provides intfor-
`mation about people or any other objects identified within
`photos, including various people who access host computer
`200. Users database 230 can be populated to include, for
`each useror client system, a user identifier that is unique to
`that uscr or clicnt system, and other information such as the
`user’s name, e-mail address, home page address, and/ora list
`of contacts. The list of contacts is not necessary and maybe
`stored in a