(12) United States Patent
`Frigon
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,954,432 B2
`Feb. 10, 2015
`
`USOO8954432B2
`
`(54) USERS TAGGING USERS IN PHOTOS
`ONLINE
`
`(75) Inventor: Mark Frigon, Avon, CO (US)
`(73) Assignee: Mark Frigon, San Francisco, CA (US)
`(*) Notice:
`Subject to any disclaimer, the term of this
`past i.S.S. adjusted under 35
`.S.C. 154(b) by 0 days.
`(21) Appl. No.: 09/991,324
`
`(22) Filed:
`
`Nov. 15, 2001
`
`(65)
`
`O
`O
`Prior Publication Data
`US 2002/0103813 A1
`Aug. 1, 2002
`
`- I -
`
`9, 1996 Brown et al.
`5,553,281 A
`5,721,851 A * 2/1998 Cline et al. .................... 345,839
`5,802.361 A
`9/1998 Wang et al.
`5,893,110 A
`4/1999 Weber et al.
`6,014,644. A
`1/2000 Erickson ......................... 705/37
`6,038,333 A
`3/2000 Wang
`6,332,193 B1* 12/2001 Glass et al. ...
`T13, 170
`6,353,452 B1* 3/2002 Hamada et al.
`715,825
`32. R : 3: Shills s al. .
`Ei
`6,556,709 B1 * 4/2003 Kumar ...
`... 382,165
`6,616,701 B2 * 9/2003 Doyle ............
`T15,501.1
`6,687,878 B1* 2/2004 Eintracht et al.
`715,512
`6,931419 B1* 8/2005 Lindquist ...........
`707/792
`7,010,751 B2 * 3/2006 Shneiderman .....
`T15,764
`7,028,253 B1 * 4/2006 Lieberman et al. ........... 715/232
`7,243,079 B1* 7/2007 Manolis et al. ...
`TO5/26.81
`7,576,752 B1* 8/2009 Benson et al. ...
`... 345,619
`2002/0055955 A1* 5/2002 Lloyd-Jones et al. ... 707/512
`2002/0093678 A1* 7/2002 Skidgel et al. ...
`358,115
`2002fO1842O3 A1* 12/2002 Nastar et al. ...................... 707/3
`
`C Cal. .....
`
`Related U.S. Application Data
`(60) Pygal application No. 60/248,994, filed on Nov.
`s
`(51) Int. Cl.
`G06F 7/30
`(2006.01)
`G06F 3/0482
`(2013.01)
`(52) U.S. Cl.
`CPC. G06F 17/30247 (2013.01); G06F 17/30265
`(2013.0), Gosf/0482 (201301)
`USPC ........................................... 707/736; 705/319
`(58) Field of Classification Search
`USPC ........ 705/2, 3, 37, 26, 27, 319; 709/303, 233;
`707/104.1, 103 X, 3, 100, 1, 5, 736,
`707/103, 104
`See application file for complete search history.
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`OTHER PUBLICATIONS
`Kuchinsky et al., “FotoFile: A Consumer Multimedia Organization
`and Retrieval System”. May 1999, ACM, 496-503.*
`(Continued)
`
`Primary Examiner — Mahesh Dwivedi
`
`ABSTRACT
`(7)
`A system, computer program, and method for storing and
`sharing images such as photographs via a communications
`network (16) and for permitting the identification of objects
`within the images. The invention allows the identification of
`objects such as persons within the photos without requiring
`the person Submitting the photos to type in identification
`information for each and every photo in a photo album. The
`invention also allows users to automatically share their photos
`with others and to automatically search for photos and/or
`certain people in photos.
`
`4,847,604 A * 7/1989 Doyle ........................... 345,180
`5,063,603 A * 1 1/1991 Burt .............................. 382,115
`
`8 Claims, 9 Drawing Sheets
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Glient is
`
`Bowser
`
`Web pages ill
`Image
`
`2
`
`Image ID
`UserID
`- Ili,
`Client)
`
`identifications
`database
`
`
`
`s
`:
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 001
`
`

`

`US 8,954432 B2
`Page 2
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Wei et al., “Face Detection for Image Annotation', 1999, Pattern
`Recogntion Letters, 20 (1999), pp. 1313-1321.*
`Shneiderman et al., Direct Annotation: A Drag-and-Drop Strategy for
`Labeling Photos, University of Maryland, College Park, Jun. 2000.*
`Article entitled "Single Display Groupware: A Model for Co-present
`Collaboration”, by Stewart et al., dated May 20, 1999.*
`
`Swain, M.J., Searching of Multimedia on the World Wide Web,
`Proceedings of the 1999 IEEE Conference on Multimedia Comput
`ing and Systems, Jun. 1999, pp. 32-37, especially Figure 3, pp. 34.
`col. 1 and col. 2.
`Meyer et al., Borealis Image Server, Computer Networks and ISDN
`Systems, 1996, pp. 1123-11327, the entire document.
`Chang, Shih-Fu, Content-Based Indexing and Retrieval of Visual
`Information, IEEE Signal Processing Magazine, vol. 14, No. 4, Jul.
`1997, pp. 4548.
`* cited by examiner
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 002
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 1 of 9
`
`US 8,954432 B2
`
`
`
`Figure 1
`
`Templates
`
`identifications
`database
`
`made ID
`User ID
`Client D
`
`
`
`230
`
`240
`
`Users
`database
`
`Images
`database
`
`FELDS
`
`Identifications
`database
`
`FELDS
`
`dentifications 124
`
`Relationship
`e.g. in photo, took picture)
`Coordinate type
`e.g. circle, rectangle, polygon)
`Coordinates
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 003
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 2 of 9
`
`US 8,954432 B2
`
`Figure 3
`
`
`
`302
`
`306
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 004
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 3 of 9
`
`US 8,954432 B2
`
`Figure 4
`
`
`
`-EX
`R CONTACTS
`O John Doe
`O Jane Doe
`
`38
`
`SUBMI
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 005
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 4 of 9
`
`US 8,954432 B2
`
`Figure 5
`
`504
`
`506
`
`508
`
`TERING)
`
`
`
`
`
`:
`
`: Aft
`sRANSCR
`::: R
`RON 5A
`NO :::::
`::::::: FRO WDA
`iDENTIFYINGPAG
`MPLAT
`
`88xxxxxxx&x:::::::::::::::::::::::8:
`
`i 88: is:
`
`
`
`
`
`
`
`s
`
`512
`
`E.
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 006
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 5 Of 9
`
`US 8,954432 B2
`
`Figure 6
`
`
`
`
`
`606
`
`608
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 007
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 6 of 9
`
`US 8,954432 B2
`
`
`
`Figure 7
`
`-EX
`R PEOPLE INIMAGE
`
`42
`
`Jane Doe
`
`PEOPLE IN MAGE
`48
`44 N Jim Doe
`Sue DOe
`Chris DOe
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 008
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 7 Of 9
`
`US 8,954432 B2
`
`Figure 8
`
`
`
`814
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 009
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 8 of 9
`
`US 8,954432 B2
`
`900
`
`901
`
`Figure 9
`
`Cursor not over
`
`911
`
`
`
`
`
`
`
`
`
`
`
`
`
`922
`
`
`
`-
`
`
`
`E.
`
`923
`Photos including Jane
`
`
`
`926 -1
`Photos including John
`
`
`
`927
`
`After Clicked
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 010
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 9 Of 9
`
`US 8,954432 B2
`
`
`
`
`
`
`
`
`
`
`
`Figure 10
`
`: of
`
`1 OOO
`
`1004
`
`CORDS.
`
`1 OO6
`
`1008
`
`OO
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 011
`
`

`

`US 8,954,432 B2
`
`1.
`USERS TAGGING USERS IN PHOTOS
`ONLINE
`
`This non-provisional application takes priority from U.S.
`Provisional Application No. 60/248,994 filed on Nov. 15,
`2OOO.
`
`5
`
`BACKGROUND OF THE INVENTION
`
`2
`their identities. Typically, photo album sites offer the ability
`to describe uploaded photos through the use of captions or
`other descriptive fields. These fields may be used to describe
`or name individuals within the photo. For example, a user
`may upload a photo and give it a caption of "Stephanie roll
`erblading with Tauny’. This caption alone is not sufficient
`information to ensure the viewer can correctly determine the
`identities of the individuals within the photo, because to do so
`would require prior knowledge of which person is
`“Stephanie' and which is “Tauny'. A viewer who does not
`know these individuals may have difficulty or be unable to
`correctly identify the individuals within the photo. Further
`more, less descriptive captions such as “The Family at the
`beach would offer little help unless the viewer knows each
`individual in the photo. Thus, in order to provide sufficient
`information required to ensure a viewer can identify individu
`als within a photo, copious descriptions must be input and
`maintained by users. Typing such captions or descriptions is
`very time consuming and tedious, especially when a large
`number of photos are uploaded. Furthermore, when the same
`person appears in multiple photos, the Submitter is still
`required to retype the identification information into a caption
`for each photo.
`Processes have been developed to identify individuals
`within photos. For example, one process requires people to
`wear badges containing identification information Such as
`unique color codings when their photos are taken. When a
`computer processes a photo containing a badge, the computer
`can access a database to determine which badge was assigned
`to which person and then identifies the person as existing in
`this photo. This process has many drawbacks including the
`dependence on physical badges. If the badges cannot be seen
`in the photos, the people wearing the badges cannot be iden
`tified. Moreover, this process is not functional for the vast
`majority of photographs: any photos that were taken outside
`of this tightly controlled environment.
`Another problem with photo-specific websites is that they
`have no search capabilities that permit users to search for
`photos of specific individuals. Users wishing to view all pho
`tographs that contain a particular person must manually
`access all possible photo albums to find the desired photos.
`For example, ifa user wishes to find all pictures of a particular
`friend, the user may have to browse through several of his/her
`own albums where that friend may appear (e.g. “High School
`pictures”, “Reunion Weekend’, etc.), as well as access and
`browse through each of the friends albums, or any third
`party's albums where the friend may appear. With a reason
`able number of photos, this becomes time-prohibiting if not
`impossible. Search features could be designed to search
`aforementioned descriptive fields (e.g. a caption), however,
`as there is no naming standard, people are described differ
`ently by different people (e.g. first name, last name, full name,
`nickname, etc.) and many names would be used to describe
`multiple people (e.g. Mom, Dad, John, etc.). As a result, it
`would be difficult to have a one to one relationship between
`descriptions of people and people themselves. Consequently,
`searches would be unreliable, likely returning many photos of
`little relevance and failing to return photos that would be of
`relevance. For example, a search for the term “John Smith'
`would fail to return photos in which this person was referred
`to as “John' or, perhaps, “Dad'. However, this same search
`likely would return photos of other individuals who share the
`same name "John Smith'.
`While reliable database searchability for digital images
`over the Internet is available, it has not been implemented
`with photo albums. Particularly, several companies which sell
`and license images have put catalogs of digital photographs
`
`10
`
`15
`
`25
`
`30
`
`35
`
`1. Field of the Invention
`The present invention relates to computer software. More
`particularly, the invention relates to a method and apparatus
`for storing and sharing images Such as photographs via a
`communications network and for permitting the identifica
`tion of objects and the location of the objects within the
`images. The invention enables users to Supply and/or receive
`information about the existence of objects within images.
`Portions of this patent document contain material that is
`Subject to copyright protection. The copyright owner has no
`objection to the facsimile reproduction by anyone of the
`patent document or the patent disclosure as it appears in the
`Patent and Trademark Office file or records, but otherwise
`reserves all copyrights whatsoever.
`2. Background
`Digital photography allows people to quickly and easily
`take, print, and/or share photos without film and the associ
`ated film developing costs. The popularity of digital photog
`raphy has increased dramatically due to the increased avail
`ability of digital cameras and the widespread acceptance of
`the Internet. Many Internet users have created websites that
`organize their digital photographs into online photo albums
`that can be accessed by other Internet users. These web pages
`permit a large number of people to view the photos and allow
`the author to both add and remove photos from an album.
`Although creating web pages for online photo albums offer
`advantages over traditional photo albums, their creation
`requires programming expertise Such as skill in Hypertext
`Markup Language (HTML). Many people do not have the
`technical knowledge and/or time to build their own online
`photo albums. Sharing of photos from online photo albums is
`also problematic and requires that each photo be manually
`sent, e-mailed, or downloaded. If the recipient wishes to use
`the photos in their own web pages, each photo must be then
`uploaded to an additional site. These steps are time consum
`ing, duplicative, and also require programming knowledge.
`To address these problems, several websites that specialize
`in online photo albums have been created. These websites
`allow users without programming skills to quickly and easily
`create and maintain online photo albums. Users are only
`required to upload photos that they wish to add to an online
`50
`photo album. The sites, through programming on the web
`server, or “server-side Scripting, then create all necessary
`HTML programming to add the uploaded photos to the users
`photo albums. Many of these sites also offer image enhance
`ment capabilities (e.g. sharpen, brighten, reduce "red-eye'.
`etc.), as well as photo printing and developing functions.
`Photo-specific sites have also eased the process of sharing
`photos by providing a means for users to quickly and easily
`e-mail links to photos or photos themselves to one another
`and, in Some cases, contribute photos directly to other's photo
`albums.
`While websites that specialize in the creation of online
`photo albums offer many advantages, they suffer from several
`limitations. For example, existing websites do not offer users
`the ability to identify objects within photos. If a user is view
`ing a photo and does not know certain individuals in the
`picture, the user is unable to use current systems to determine
`
`40
`
`45
`
`55
`
`60
`
`65
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 012
`
`

`

`US 8,954,432 B2
`
`3
`on the Internet. Such websites generally describe images in
`terms of themes (e.g. Sunset, dark or sad) or broad descrip
`tions of objects therein (e.g. beach, farmland, or water), with
`out identifying the objects explicitly. For example, a picture
`of a man and woman on the beach at Sunset, which could be
`licensed to a beach resort for print or web use, could be
`identified as “beach', 'sunset', and/or “man and woman'.
`While such descriptive terms can provide a reliable mecha
`nism for searching for images matching the terms, they do not
`uniquely identify any objects in the images, nor do they have
`the capability of identifying where these objects appear
`within the images. For example, a search for the term "clown”
`is likely to return many pictures of clowns, however, it is very
`unlikely that every clown pictured would be the same person,
`just as a search for the term “beach is unlikely to return
`pictures of the same beach. Even a search for the term “Presi
`dent' will return several photos of different people. Further
`more, the objects that appear within a photo are not identified
`as being in a particular area within the photo, they merely are
`described as being in the photo. For example, websites that
`return a photograph from a “President' search do not have the
`capability of identifying that the President is in, perhaps, the
`lower right corner of the picture. Such technologies have the
`capability to describe the nature or theme of a photo but not to
`uniquely identify who or what is in the photo or where in the
`photo it appears.
`Another related limitation of prior art websites is that the
`captions described above contain static information only,
`Such as names, and do not provide a way to obtain additional
`information about a person or object such as contact infor
`mation or to locate additional photos of the person or object.
`Another problem with photo-specific websites is that users
`wishing to share their photos must proactively e-mail or oth
`erwise send their photos to others (i.e., a photo is not auto
`matically shared when a user is described in a photo). For
`example, even if a user describes a photos as containing
`"Jane(a).doe.com’ and “John(a).doe.com', existing website
`will not share the photos with these individuals. Through a
`separate process, the user must still email this photo or oth
`erwise share the photo with, in this case, Jane and John.
`Although photo-specific websites ease this process, it is still
`time consuming for users with large photo albums and/or a
`large number of friends to send their photos to all such per
`SOS,
`
`SUMMARY OF THE INVENTION
`
`An embodiment of the invention comprises a system and
`method for identifying and displaying objects (in most cases,
`a person) within a digital image over a system of intercon
`nected computers (e.g. client/server, peer-to-peer, etc . . . ).
`For example, a user can identify where a person exists in an
`image by selecting the person's name (e.g. from a list) and
`clicking on the person's location in the image. By identifying
`people within images, the system is able to search for images
`where a specified person(s) appears as well as generate output
`that recognizes where in the image a person appears. This
`output is also capable of creating hyperlinks and calls to
`client-side scripts from the otherwise “static' image file. In an
`embodiment of the invention the system can also utilize arti
`ficial intelligence algorithms to locate other images where the
`identified object has not yet been defined by a user of the
`system.
`While, one embodiment of this invention utilizes the Inter
`net as the medium for communication and web pages for
`displaying data, the scope of this invention is not limited
`specifically to the Internet and/or to web pages. The invention
`
`4
`contemplates the use of any display output in a computer
`environment (e.g. any types of document). The terms person,
`people, and/or object are used interchangeably and meant to
`refer to any object(s) (e.g. a pet, a possession, a place, or a
`thing) a user wishes to identify within a digital image. This
`most frequently will, in fact, be a person. When shown within
`the bounds of an image, a person is considered akin to any
`type of object. Thus, the reader should note that the invention
`can be utilized to identify any type of object within an image.
`Depending on the computer environment, different media
`will be used for output and display to the client. On the
`Internet, the most accessible and most common output is
`HypertextMarkup Language (“HTML). However, the func
`tionality of HTML can be duplicated or configured in other
`non-HTML media. For the sake of both brevity and clarity, I
`will outline an embodiment of the present invention that
`utilizes HTML. This embodiment of the present invention
`(i.e. utilizing the Internet and HTML) is logical due to the
`broad reach of the Internet and wide acceptance of HTML.
`However the scope of the invention is not intended to be
`limited to the use HTML or the Internet.
`
`10
`
`15
`
`DESCRIPTION OF THE DRAWINGS
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`An embodiment of the invention is described in detail
`below with reference to the attached drawing figures,
`wherein:
`FIG. 1 is a schematic diagram of computer and communi
`cations equipment that may be used to implement an embodi
`ment of the present invention;
`FIG. 2 is a schematic diagram of databases that may be
`used to implement certain aspects of the invention;
`FIG. 3 is a flow chart illustrating a high-level overview of
`the steps of an embodiment of the invention.
`FIG. 4 is a schematic diagram of an identifying page that
`may be used when identifying objects within an image in
`accordance with an embodiment of the invention;
`FIG. 5 further describes the general details relating to the
`process for requesting, generating, and transmitting an iden
`tifying page to the requester (e.g., the client computer).
`FIG. 6 is a flow diagram illustrating a general overview of
`the process for using the identifying page to obtain identify
`ing information (e.g., information associated with object(s) in
`an image) from the user.
`FIG. 7 is a schematic diagram of a display page that may be
`used to display images and information about objects that
`have been identified within the image in accordance with an
`embodiment of the invention.
`FIG. 8 is a flow chart illustrating a high-level view of the
`process for providing display page to a requesting user in
`accordance with an embodiment of the invention.
`FIG. 9 illustrates various ways to display identification
`information, including the various states a user interface con
`figured to operate in accordance with an embodiment of the
`invention may have.
`FIG.10 is a flow chart describing the process for generating
`a set of search results in response to a request.
`The figures do not limit the invention to the specific
`embodiments disclosed and described herein. The drawings
`are not necessarily to scale, emphasis instead being placed
`upon clearly illustrating the principles of the invention.
`
`DETAILED DESCRIPTION
`
`65
`
`The invention described below is a method and apparatus
`for obtaining information relating to the existence of at least
`one object in an image. In the following description, numer
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 013
`
`

`

`5
`ous specific details are set forth to provide a more thorough
`description of the present invention. It will be apparent, how
`ever, to one skilled in the art, that the present invention may be
`practiced without these specific details. In other instances,
`well known features have not been described in detail so as
`not to obscure the present invention.
`General System Architecture
`Embodiments of the invention can be implemented inhard
`ware, Software, firmware, or a combination thereof. In one or
`more embodiments, however, the invention is implemented
`with one or more computer programs that operate one or more
`pieces of computer equipment broadly referred to by the
`numeral 101 in FIG. 1. Such computer equipment 101 typi
`cally includes a server computer 200 (also referred to as a host
`computer) and any number of user computers 100 (e.g., client
`computers) that may access the host computer 200 via a
`communications network 150. The computer programs and
`equipment illustrated and described herein are merely
`examples of programs and equipment that may be used to
`implement an embodiment of the invention and may be
`replaced with other software and computer equipment with
`out departing from the scope of the invention. For example,
`any computational device configured to display image data
`may be configured to implement one or more aspects of the
`invention.
`Host computer 200 may be any computing device such as
`a network computer running Windows 2000, Novel Netware,
`Unix, or any other network operating system. The host com
`puter 200 may be connected to a firewall computer at the
`boundaries of network 150 to prevent tampering with infor
`mation stored on or accessible by the host computer 200 and
`to another computing device that may operated by an admin
`istrator of the host computer 200. If the invention is imple
`mented with the Internet, the host computer 200 may include
`conventional web hosting operating software, an Internet
`connection such as a modem, DSL converter or ISDN con
`Verter, and be assigned an IP address and corresponding
`domain name so that the website hosted thereon can be
`accessed via the communications network 150.
`Client computer(s) 100 may be used by anyone wishing to
`upload and/or view photographs via host computer 200. The
`client computers 100 may be any device or group of devices
`that permits access to host computer 200 via communications
`network 150, including, but not limited to, personal or laptop
`computers such as those manufactured and sold by DellTM,
`Compaq TM, GatewayTM, or any other computer manufacturer,
`handheld personal assistants such as those operating with
`Palms PalmOSTM operating software or those operating with
`Microsoft's PocketPCTM operating software, mobile phones,
`or even Internet appliances that are specifically designed for
`50
`accessing the Internet such as WebTVTM. Each client com
`puter 100 may also include or can access a conventional
`Internet connection Such as a modem, Digital Subscriber Line
`(DSL) converter, or Integrated Service Digital Network
`(ISDN) converter and a web browser that permits it to access
`and view data over the Internet.
`The communications network 150 may be the Internet but
`can also be any other communications network Such as a local
`area network, a wide area network, a wireless network, an
`intranet or a virtual private network. Hereinafter, the term
`Internet or communication network refers to any type of
`interconnection fabric that provides computers coupled to the
`network with a mechanism for transmitting and/or receiving
`data.
`The computer program or programs embodying one or
`more aspects of the invention are stored in or on computer
`readable medium residing on or accessible by host computer
`
`30
`
`35
`
`40
`
`45
`
`55
`
`60
`
`65
`
`US 8,954,432 B2
`
`10
`
`15
`
`25
`
`6
`200 and provide a mechanism for instructing host computer
`200 to operate the invention as described herein. The com
`puter programs typically comprise ordered listings of execut
`able instructions for implementing logical functions in host
`computer 200 and user computers 100 coupled with host
`computer 200. The computer programs can be embodied in
`any computer-readable medium for use by or in connection
`with an instruction execution system, apparatus, or device,
`Such as a computer-based system, processor-containing sys
`tem, or other system that can fetch the instructions from the
`instruction execution system, apparatus, or device, and
`execute the instructions. In the context of this application, a
`“computer-readable medium' can be any means that can con
`tain, Store, communicate, propagate or transport the program
`for use by or in connection with the instruction execution
`system, apparatus, or device. The computer-readable medium
`can be, for example, but not limited to, an electronic, mag
`netic, optical, electromagnetic, infrared, or semi-conductor
`system, apparatus, device, or propagation medium. Examples
`of the computer-readable medium would include the follow
`ing: an electrical connection having one or more wires, a
`portable computer diskette, a random access memory
`(RAM), a read-only memory (ROM), an erasable, program
`mable, read-only memory (EPROM or Flash memory), an
`optical fiber, and a portable compact disk read-only memory
`(CDROM). The computer-readable medium could even be
`paper or another Suitable medium upon which the program is
`printed, as the program can be electronically captured, via for
`instance, optical scanning of the paper or other medium, then
`compiled, interpreted, or otherwise processed in a Suitable
`manner, if necessary, and then stored in a computer memory.
`In an embodiment of the invention, host computer 200
`comprises server engine 220 which is programmed to operate
`or host a website and serve as a repository for images and
`identification information for objects within the images as
`described in more detail below. The images may be photo
`graphs, graphics, artwork, or any other digital image that
`contains or depicts one or more objects. The objects within
`the images may include people, animals, plants, buildings,
`places, or anything else shown in images. In one embodiment
`of the invention, the images and objects are referred to (but
`not limited to) as photographs and people, respectively. The
`reader should note that the term object as used herein includes
`any identifiable entity within the image data.
`Server engine 220 can access web page templates 210,
`databases 230, 240, and 250, and internal or external memory.
`Web page templates 210 can be any technology used to tran
`scribe data from a data source into a web page or otherwise
`provide "dynamic' or customized documents, including but
`not limited to, Active Server Pages, Java Server Pages, Cold
`Fusion, PHP, CGI scripts, or even client-side scripts. Server
`engine 220 receives information and requests from user com
`puter(s) 100 via communications network 150, processes
`web page templates 210 by transcribing data from data
`sources, including databases 230, 240, and 250, into tem
`plates 210, and sends the resulting web page or document to
`requesting client computer 100 as described in more detail
`below.
`In order to implement the invention host system 200 will
`access a server system or database system configured to Sup
`port the functionality described. Referring back to FIG.2, one
`or more databases could be utilized to store the data utilized
`by the system. For instance, the system may utilize a users
`database 230. Identifications database 240, and Images data
`base 250 as depicted in FIG. 2.
`Users database 230 receives, stores, and provides informa
`tion about people or any other objects identified within pho
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 014
`
`

`

`US 8,954,432 B2
`
`10
`
`15
`
`7
`tos, including various people who access host computer 200.
`Users database 230 can be populated to include, for each user
`or client system, a user identifier that is unique to that user or
`client system, and other information Such as the user's name,
`e-mail address, home page address, and/or a list of contacts.
`The list of contacts is not necessary and may be stored in a
`separate database.
`Images database 250 receives and stores information about
`photos (e.g., image data) uploaded or otherwise made acces
`sible to the network by users. Images database 250 can be
`populated as described below to include, for each photo, a
`photo identifier that is unique to that photo, and the location of
`the image file on the network. The Images database 250 may
`include descriptive information about the photo such as a
`caption or the date the photo was taken. While it is preferred
`to have photos uploaded to and stored on the host computer
`200, it is possible to have the photos stored elsewhere, being
`still accessible to users on the network, such as another web
`site or a personal computer connected to the network. In an
`embodiment of the invention, both of these processes will be
`referred to hereinafter as uploading. It is also possible to store
`the images directly into the database itself as binary objects,
`rather than simply storing information about the location of
`the image elsewhere on the network.
`25
`Identifications database 240 may receive, store, and pro
`vide information about relationships between users and pho
`tos. Accordingly, in one embodiment of the invention Identi
`fications database 240 links information in Users database
`230 with information in Images database 250. Identifications
`database 240 may be populated as described below to contain
`fields specifying what kind of relationship exists between a
`photo and a user (e.g. the user is in the photo, the user took the
`photo, the photo has been shared with the user, etc.). In one
`embodiment of the invention, Identifications database 240
`also can store information about the location within a photo,
`or the “coordinates', that a user or other person appears.
`For example, a user, John Doe, may access and sign-up
`with host computer 200 and be assigned a user I.D. of "007'
`in the Users database. The Users database 230 could then be
`populated to include the following record:
`
`8
`Doe appears. Assuming John Doe appears in the circular area
`bound by the coordinates 50.50.25 (X-axis location, y-axis
`location, radius of circle), the Identifications database 240
`entry would appear as follows:
`
`User
`I.D.
`
`OO7
`
`Image
`I.D.
`
`Shared by
`
`Coordinates
`
`ABCD
`
`OO7
`
`50, 50, 25
`
`Similarly, another user, Jane Doe, who also appears in the
`photo uploaded by John Doe, can be identified in my
`image.jpg by inserting an additional entry into the Identifica
`tions database 240. The database entries could now be the
`following:
`
`User database
`
`User I.D.
`
`Name
`
`Email
`
`Contacts
`
`OO7
`OO8
`
`John Doe
`Jane Doe
`
`jdoe(a)doe.com
`janedoe?a doe.com
`
`John, Jane, Lisa
`Jane, John, Fuzzie
`
`Image I.D.
`
`Location
`
`Images database
`Caption
`
`Uploaded by
`
`ABCD
`
`C:\diramy image.jpg John and Jane at
`the beach
`
`OO7
`
`Identifications database
`
`User I.D.
`
`Image I.D.
`
`Shared by
`
`Coordinates
`
`OO7
`O08
`
`ABCD
`ABCD
`
`OO7
`OO7
`
`50, 50, 25
`25, 75, 10
`
`As a result, two users have been identified within the image,
`my image.jpg, each defined by a record in the Identifications
`database 240.
`The host computer 200 may access information in the
`Identifica