US009959291B2
`
`US 9,959,291 B2
`(10) Patent No:
`a2 United States Patent
`Frigon
`(45) Date of Patent:
`May1, 2018
`
`
`(54) USERS TAGGING USERS IN MEDIA ONLINE
`
`(56)
`
`References Cited
`
`(72)
`
`(71) Applicant: Mark Frederick Frigon, San
`Francisco, CA (US)
`\
`.
`Inventor: Mark Frederick Frigon, San
`Francisco, CA (US)
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`USS.C. 154(b) by 835 days.
`
`(*) Notice:
`
`:
`(21) Appl. No.: 14/053,626
`.
`(22) Filed:
`Oct. 15, 2013
`
`(65)
`
`Prior Publication Data
`US 2014/0122459 Al
`May1, 2014
`US 2016/0364413 AY
`Dec. 15, 2016
`
`
`U.S. PATENT DOCUMENTS
`4,847,604 A
`7/1989 Doyle
`5,063,603 A
`11/1991 Burt
`5,553,281 A
`9/1996 Brown
`5,721,851 A
`2/1998 Cline
`5,802,361 A
`9/1998 Wang
`aie ‘
`woo Teun
`6.014644 A
`1/2000 Erickson
`6,029,141 A
`—-2/2000 Bezos
`6,038,333 A
`3/2000 Wang
`6,097,853 A
`8/2000 Gu
`6,216,133 BL
`4/2001 Masthoff
`6,236,978 Bl
`5/2001 ‘Luzhilin
`6,288,717 BL
`9/2001 Dunkle
`6,332,193 Bl
`12/2001 Glass
`6,353,452 Bl
`3/2002 Hamada
`6,411,949 Bl
`6/2002 Schaffer
`6,442,573 Bl
`8/2002 Schiller
`(Continued)
`
`OTHER PUBLICATIONS
`
`Related U.S. Application Data
`.
`Lo.
`.
`(63) eeeea application NOae filed on
`nowPat.
`ov.
`No.
`.
`>
`’
`oo
`
`Article entitled “Fotolile: A Consumer Multimedia Organization
`and Retrieval System”, by Kuchinsky et al., dated May 20, 1999,*
`:
`(Continued)
`Primary Examiner — Mahesh H Dwivedi
`(74) Attorney, Agent, or Firm — Nixon Peabody LLP
`(60) Provisional application No. 60/248,994,filed on Nov.
`15, 2000.
`(57)
`ABSTRACT
`>
`A system, computer program, and method for storing and
`Int. Cl.
`haring
`i
`h
`1
`hs
`vi
`icati
`sharing images such
`as photographs via a communications
`G06F 1730
`network (16) and for permitting the identification of objects
`G06F 3/0482
`within the images. ‘lhe invention allows the identification o
`IC
`ithin
`the
`images.
`‘The
`i
`ion
`allows
`the
`identification
`of
`(52) US. CL
`objects such as persons within the photos without requiring
`crc ...... GO6F 17/30247 (2013.01); GO6F 3/0482
`the person submitting the photos to type in identification
`.
`
`(2013.01); GO6F 17/30265 (2013.01)__th on sub he ph 1 dentifi
`
`
`
`
`
`(58) Field of Classification Search
`information[or each and every photo in aphoto abu. are
`CPC... GOGF 17/30247; GO6F 17/30265; GOGF
`mvention also allows users to automatica ly share their
`photos withothers and to automatically search for photos
`3/0482
`
`_. 107/736
`and/or certain people in photos.
`USPC _..
`See application file for complete search history.
`26 Claims, 9 Drawing Sheets
`
`(51)
`
`(2006.01)
`(2013.01)
`
`
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 001
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 001
`
`

`

`US 9,959,291 B2
`Page 2
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`6,542,936 BL
`6,545,660 BL*®
`
`4/2003 Mayle
`4/2003 Shen ween GO06EF3/048
`
`OTHER PUBLICATIONS
`Article entitled “Visualization Methods for Personal Photo Collec-
`tions: Browsing and Searching in PhotoFinder”, by Kang et al.,
`dated. Aug. 2, 2000.*
`Article entitled “Direct annotation: a drag-and-drop strategy for
`labeling photos”, by Schneiderman et al., dated Jul. 21, 2000.*
`Article entitled “Fast Hace Detection via Morphology-Based. Pre-
`processing”, by Han et al., dated Jan. 1997,*
`Article entitled “Menu Selection by Facial Aspect”, by Bakic et al.,
`dated May 21, 1999.*
`Article entitled “Human detection and localization at indoor envi-
`ronment by homerobot”, by Ghidaryet al., dated Oct. 11, 2000.*
`Rex, Erica, “Online Pholo Delivery”, Jun, 2000, Upside Magazine,
`wwwiupsidetoday.com (6 pages).
`Stewart et al., “Single Display Groupware: A Model for Co-present
`Collaboration”, dated May 20, 1999, pp. 286-293.
`Chang, Shih-Fu, “Content-Based Indexing and Retrieval of Visual
`Information”, IEEE Signal Processing Magazine, vol. 14, No. 4, Jul.
`1997, pp. 45-48.
`Kuchinskyet al., “FotoFile: A Consumer Multimedia Organization
`and Retrieval System”, May 1999, ACM, pp. 496-503.
`Meyer et al., “Borealis Image Server: Computer Networks and
`ISDN Systems”, May 1996, pp. 1123-1137.
`Shneiderman et al., “Direct Annotation: A Drag-and-Drop Strategy
`11/2001 Hasegawa
`2001/0037721 Al
`for Labeling Photos”, University of Maryland, College Park, Jun.
`5/2002 Schneiderman
`2002/0054059 Al
`2000, 8 pages.
`5/2002 Lloyd-Jones
`2002/0055955 Al
`Swain, M.J., “Searching of Multimedia on the World Wide Web”,
`§/2002 Work
`2002/0059201 Al
`Proceedings of the 1999 IEEE Conferences on Multimedia Com-
`6/2002 McNeely
`2002/0069222 Al
`
`
`2002/0093678 Al* ............ GO6E 17/30265_puting and Systems, Jun. 1999, pp. 32-37.7/2002 Skidgel
`Wei et al., “Face Detection for Image Annotation”, 1999, Pattern
`Recognition Letters, 20 (1999), pp. 1313-1321.
`
`6,536,709 Bl
`6,616,701 B2
`6,687,878 BL*
`
`4/2003, Kumar
`9/2003 Doyle
`2/2004 Eintracht
`
`..............
`
`6,829,607 Bl
`6,931,419 BI*
`
`12/2004 ‘lafoya
`8/2005 Lindquist............
`
`6,973,201 Bl
`7,010,751 B2
`7,028,253 B1*
`
`12/2005 Colmenarez
`3/2006 Shneiderman
`4/2006 Lieberman ........ GO6F 17/30265
`
`7,068,309 B2
`7,209,149 B2
`7,243,079 B1*
`
`7,576,752 BL*
`
`6/2006 Toyama
`4/2007 Jogo
`7/2007 Manolis ............ GO6F 17/30017
`
`8/2009 Benson... GO6F3/1415
`
`8/2002 Frigon
`2002/0103813 Al
`2002/0184203 A1* 12/2002 Nastar..
`2003/0076363 Al
`4/2003. Murphy
`
`
`see, G06Q 30/06
`
`* cited by examiner
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 002
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 002
`
`

`

`U.S. Patent
`
`May1, 2018
`
`Sheet 1 of 9
`
`US 9,959,291 B2
`
`i
`
`i
`
`iii
`
`rows
`FP iS
`wy"no"
`if
`re
`
`%
`
`
`
`
`
`i
`
`iii
`
`ele
`
`i
`
`ene
`
`eee,
`
`poresnasoene
`
`
`
` Wrensraamsanerarconmsncncenntenccetaad
`
`i}“Wirevnenvnvevrenesenneavinen
`
`
`
`rocansconnloonssennsesennancrasscanraeenncaeenennn:neve
`
`TYeaitennnnnnnnnnnrnmnnannnttnsoie Te
`
`
`
`NLCONTERCEERIOCRMEMEernnemnaneRGInEEReeceerereEeTenTaneeatiit%
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 003
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 003
`
`
`
`
`
`

`

`U.S. Patent
`
`May1, 2018
`
`Sheet 2 of 9
`
`US 9,959,291 B2
`
`Figure 3
`
`| oprain:IMAGE DATA |
`
`ATLEAST ONEOBJECT,
`
`e.g. A PHOTOGRAPH)
`COMPRISING
`
`FROM AUSER
`
`
`-ORTA
`
` N IDENTIFYING
`NEO!RIMATION
`
`
`
`
`WHERE THEIDENT]Pye
`
`
`“INFORMATION RELATES T
`
`
`THE EXISTENCE OF
`
`
`THE OBJECT
`
`
`
`& DATA
`INTHEIMAG
`
`
`
`
`
`STORING IDENTIEYING 804
`INFORMATION a
`ASSOCIATED Eo
`
`
`WITH THE IMAGE DATA |
`
`
`| PRESENTENTING THE IDENTIFYING
`
`ebennttntete“a Vocimeraeenateeenwesteentaeoncea
`
`
`sevemnnnaneannnannnnnnannnenteeterteyrvreeeeirenttonnnenannecenesrne
`
`|
`
`
`INFORMATION TO TH
`USER UPON SELECTION
`OF ATLEAST ONE
`OBJECT OR IMAGE.
`
`
`
`Bee
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 004
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 004
`
`

`

`| 3a# i''
`
` SANESTeennnoannotonteosomiasscenaEnotesennaiaBeetNCetoh
`
`
`
`
`|;Bs
`
`snonaaesensosnnannsoimnensinsenommenccoseite
`
`ous38
`SUBMIT }
`pore
`nonannaens
`
`O
`© #7
`
` cercanceDNDNRADIATAAANAORCARONaRHINEAARPHNETB
`PEN
`
`U.S. Patent
`
`May1, 2018
`
`Sheet 3 of 9
`
`US 9,959,291 B2
`
`Piguyseog
`
`
`© John Dee
`
`
`i
`
`3 Jane Doe
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 005
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 005
`
`

`

`U.S. Patent
`
`May1, 2018
`
`Sheet 4 of 9
`
`US 9,959,291 B2
`
`Figure 5
`
`|oe EQUESTFOR
`
`IDENTIFYING PAGE
`|
`te.g,, MAPPING PAGE)
`5
`TOSERVER .
`
`ré o
`
`DETERMINE DESIRED
`
`
`
`ENVIRONMENT
`
`
`RETRENVE IDENTIFYING FAGE
`E FOR ENVIRONMENTTy
`
`TE!MELAS
`
`
`RETHEIVE INFORMATION |.
`
`
`FROM IMAGES DATABASE |
`
`
`
`
`
`
`
`RETREIVE INFORMATION :
`FROM USERS DATABASE F
`(OPTIONALFILTERING)
`
`
`TRANSCRRIBE INFORMATION
`FROM DATABASES INTO
`
`
`G PAGE TEMPLATE |
`IDENPTERYIN
`
`
`oman
`ob
`
`
`
`IDENTIFYING PAGE
`TO CLIENT COMPUTER
`
`
`|
`
`rey
`-S2
`
`a
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 006
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 006
`
`

`

`U.S. Patent
`
`May1, 2018
`
`Sheet 5 of 9
`
`US 9,959,291 B2
`
`Figure 6
`
`
`
`-_ 683
`
`(L“
`SELECT OBJEOTIS)
`TO IDENTIFYIN IMAGE
`
`
`
`i
`
`
`
`" PROVIDE
`“COORDINATE
`
`INFORMATION
`
`
`
`
`
`NO
`
`
`nk
`IDENTIFY LOCATIONOF |
`|
`|
`OBIECTE) BY
`
`
`|
`pa} SELECTING THE
`USERS LOCATION
`
`|
`INTHEIMAGE
`
`
`y
`chats
`
`
`os
`
`SEND}IMAGE IMA,ACN
`USERINFORMATION, AN
`
`
`
`COORDINATE -
`oS
`
`TION OF SUPPLIED)
`
`TOSERVER -
`
`
`
`
`
`INFORMATION SENT BY G
`
`SEAVER STORES aya.
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 007
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 007
`
`

`

`U.S. Patent
`
`May1, 2018
`
`Sheet 6 of 9
`
`US 9,959,291 B2
`
`Figure 7
`
` gpisossoseceeereicn
`shockinnisntisichnincmntebseinety£3
`
`>age
`=~"y mn . “Tl eonn = Ss>O iT"
`
`
`
`
`
`sbSeaseres
`
`—
`
`
`
`
`
`e
`2
`
`s
`
`me
`
`;
`3
`
`|
`
`tA
`
`‘nen,
`
`FET? OE
`Sue Doe
`
`Cars Boe
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 008
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 008
`
`

`

`U.S. Patent
`
`May1, 2018
`
`Sheet 7 of 9
`
`US 9,959,291 B2
`
`IBTAIN A REQUESTFOR E/
`IMAGE DISPLAY PAGE -f
`
`DETERMINE DESIRED
`ENVIRONMENT
`
`» B02
`|/
`
`
`
`
`
`
`GBTAIN IMAGEAND
`_ NAMING INFORMATON
`
`Figure &
`
`
`
`
`lrHETREIVEDISPLAYPAGE OOF
`FEMPLATE FOR ENVIRON
`
`
`TRANSCRIBEIMAGEAND NAMING.
`
`
`
`INFORMATION INTO
`
`
`DySPLAYPAGETEMPLATE_
`
`
`VES
`a COORDINAINATE
`
`INTObsANSeae.
`‘NG
`
`INFORMATION
`
`
`
`TRANSCRIBE
`aon
`COORDINATE o
`
`
`
`TEMPLATE
`
`
`i
`PROVIDE THE IMAGE
`
`el
`DISPLAY PAGE
`ToaRECEIVING USER
`f
`i
`[INIT ATING THE REQUEST
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 009
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 009
`
`

`

`U.S. Patent
`
`May1, 2018
`
`Sheet 8 of 9
`
`US 9,959,291 B2
`
`
`
`
`
`figure &
`
`Cursor not aver
`
`dane Doe%
`
`
`
`
`
`After Clicked
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 010
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 010
`
`

`

`U.S. Patent
`
`May1, 2018
`
`Sheet 9 of 9
`
`US 9,959,291 B2
`
`Figure 10
`
`_ CUENT SENDS BEQUEST FOR
`IMAGES (OR LIST OF IMAGES)
`CONTAINING ONE
`OR MORE OBJECTS <
`
`DETERMINE GESIRED
`
`_OBTA)NSEARCH RESULTS |
`
` 1000
`
`
`
`
`-ENVIRONMENT
`TEMPLATE FOR
`“RETRE:VERECORDS
`|
`
`BROMO
`
`-IDENTIE
`
`FICATIONS DATABASE
`
`
`
`
`
`
`-"TRANSCRIGE DATA”
`
`
`NTOSEARCH RESULTS:
`| TEMPLATE _
`
`SENDLIST OF|IMAGES AND/OR.
`SEARCH RESULTS PAGE
`TO CUENT —
`
`eePOTG
`A
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 011
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 011
`
`

`

`US 9,959,291 B2
`
`1
`USERS TAGGING USERS IN MEDIA ONLINE
`
`
`
`CROSS-REFERENCE TO RELATED
`
`
`APPLICATIONS
`
`This is a continuation of application Ser. No. 09/991 ,324
`presenting under examination. The non-provisional applica-
`tion Ser. No. 09/991,324 takes priority from U.S. Provi-
`sional Application No. 60/248,994 filed on Nov. 15, 2000.
`
`
`
`BACKGROUND OF THE INVENTION
`
`This non-provisional application takes priority from U.S.
`Provisional Application No. 60/248,994 filed on Nov. 15,
`2000.
`
` FIELD OF THE INVENTION
`
`‘The present invention relates to computer software. More
`particularly, the invention relates to a method and apparatus
`for storing and sharing images such as photographs via a
`communications network and for permitting the identifica-
`tion of objects and the location of the objects within the
`images. The invention enables users to supply and/or receive
`information about the existence of objects within images.
`Portions of this patent document contain material thatis
`subject to copyright protection. The copyright owner has no
`objection to the facsimile reproduction by anyone of the
`patent documentor the patent disclosure as it appears in the
`Patent and Trademark Office file or records, but otherwise
`reserves all copyrights whatsoever.
`
`30
`
`BACKGROUND
`
`2
`“red-eye”, etc.), as well as photo printing and developing
`functions. Photo-specific sites have also eased the process of
`sharing photos by providing a meansfor users to quickly and
`easily e-mail links to photos or photos themselves to one
`another and, in some cases, contribute photos directly to
`other’s photo albums.
`While websites thal specialize in the creation of online
`photo albums offer many advantages,
`they suffer from
`several limitations. For example, existing websites do not
`offer users the ability to identify objects within photos. Ifa
`user is viewing a photo and does not knowcertain individu-
`als in the picture, the user is unable to use current systems
`to determine their identities. lypically, photo albumsites
`offer the ability to describe uploaded photos through the use
`of captions or other descriptive fields. These fields may be
`used to describe or name individuals within the photo. For
`example, a user mayupload a photo andgiveit a caption of
`“Stephanie rollerblading with Tauny”. This caption alone is
`not sufficient information to ensure the viewer cancorrectly
`determine the identities of the individuals within the photo,
`because to do so would require prior knowledge of which
`person is “Stephanie” and whichis “launy”. A viewer who
`docs not know these individuals may have difficulty or be
`unable to correctly identify the individuals within the photo.
`Furthermore, less descriptive captions such as ‘““The Family
`at the beach” would offerlittle help unless the viewer knows
`each individual
`in the photo. Thus,
`in order to provide
`sufficient information required to ensure a viewer can iden-
`tify individuals within a photo, copious descriptions must be
`input and maintained by users. Typing such captions or
`descriptions is very time consuming and tedious, especially
`when a large number of photos are uploaded. Furthermore,
`when the same person appears in multiple photos,
`the
`submitter is still required to retype the identification infor-
`mation into a caption for each photo.
`Processes have been developed to identify individuals
`within photos. For cxample, one process requires people to
`wear badges containing identification information such as
`unique color codings when their photos are taken. When a
`computer processes a photo containing a badge, the com-
`puter can access a database to determine which badge was
`assigned to which person and then identifies the person as
`existing in this photo. This process has many drawbacks
`including the dependence on physical badges. If the badges
`cannot be seen in the photos, the people wearing the badges
`cannot be identified. Morcover, this process is not functional
`for the vast majority of photographs: any photos that were
`taken outside of this lightly controlled environment.
`Another problem with photo-specific websitesis that they
`have no search capabilities that permit users to search for
`photos of specific individuals. Users wishing to view all
`photographs that contain a particular person must manually
`access all possible photo albumsto find the desired photos.
`Tor example, if a user wishes to find all pictures of a
`particular friend,
`the user may have to browse through
`several of his/her own albums wherethat friend may appear
`(e.g. “High School pictures”, “Reunion Weekend”, etc.), as
`well as access and browse through each ofthe friend’s
`albums, or any third-party’s albums where the friend may
`appear. With a reasonable numberof photos, this becomes
`time-prohibiting if not impossible. Search features could be
`designed to search aforementioned descriptive fields (e.g. a
`caption), however,as there is no naming standard, people are
`described differently bydifferent people (c.g. first name,last
`name,full name, nickname,etc.) and many names would be
`used to describe multiple people (e.g. Mom, Dad, John,etc.).
`As a result,
`it would be difficult to have a one to one
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 012
`
`35
`
`40
`
`45
`
`Digital photography allows people to quickly and easily
`take, print, and/or share photos without film and the asso-
`ciated film developing costs. The popularity of digital pho-
`tography has increased dramatically due to the increased
`availability of digital cameras and the widespread accep-
`tance of the Internet. Many Internet users have created
`websites that organize their digital photographs into online
`photo albums that can be accessed byother Internet users.
`These web pages permit a large number of people to view
`the photos and allow the author to both add and remove
`photos from an album.
`Although creating web pages for online photo albums
`offer advantages over traditional photo albums, their cre-
`alion requires programming expertise such as skill in Hyper-
`text Markup Language (H1ML). Many people do not have
`i2
`the technical knowledge and/or time to build their own 5
`online photo albums. Sharing of photos from online photo
`albumsis also problematic and requires that each photo be
`manually sent, e-mailed, or downloaded. If the recipient
`wishes to use the photos in their own web pages, each photo
`must be then uploaded to an additional site. These steps are
`time consuming, duplicative, and also require programming
`knowledge.
`To address these problems, several websites that special-
`ize in online photo albums have been created. These web-
`sites allow users without programmingskills to quickly and
`easily create and maintain online photo albums. Users are
`only required to upload photos that they wish to add to an
`online photo album. Thesites, through programming on the
`webserver, or “server-side scripting”, then create all neces-
`sary HTML programming to add the uploaded photos to the
`users’ photo albums. Many ofthese sites also offer image
`enhancement capabilities (e.g. sharpen, brighten, reduce
`
`35
`
`60
`
`65
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 012
`
`

`

`US 9,959,291 B2
`
`0
`
`a 5
`
`IcS
`
`3
`relationship between descriptions of people and people
`themselves. Consequently, searches would be unreliable,
`likely returning many photos oflittle relevance and failing
`to return photos that would be of relevance. For example, a
`searchfor the term “John Smith” would fail to return photos
`in which this person was referred to as “John”or, perhaps,
`“Dad”. However,
`this same search likely would return
`photos of other individuals who share the same name “John
`Smith”.
`While reliable database searchability for digital images
`over the Internet is available, it has not been implemented
`with photo albums. Particularly, several companies which
`sell and license images have put catalogs of digital photo-
`graphs on the Internet. Such websites generally describe
`images in terms of themes(e.g. sunset, dark or sad) or broad
`descriptions of objects therein (c.g. beach, farmland, or
`water), without
`identifying the objects explicitly. For
`example, a picture of a man and woman on the beach at
`sunset, which could be licensed to a beachresort for print or
`web use, could be identified as “beach”, “sunset”, and/or
`“man and woman”. While such descriptive terms can pro-
`vide a reliable mechanism for searching for images match-
`ing the terms, they do not uniquely identify any objects in
`the images, nor do they have the capability of identifying
`where these objects appear within the images. For example,
`a search for the term “clown” is likely to return many
`pictures of clowns, however, it is very unlikely thal every
`clownpictured would be the sameperson, just as a search for
`the term “beach”is unlikely to return pictures of the same
`beach. Even a search for the term “President” will return
`several photos of different people. Furthermore, the objects
`that appear within a photo are not identified as being in a
`particular area within the photo, they merely are described
`as being in the photo. For example, websites that return a
`photograph from a “President” search do not have the
`capability of identifying that the Presidentis in, perhaps, the
`lowerright corner ofthe picture. Such technologies have the
`capability to describe the nature or themeof a photo but not
`to uniquely identify who or what is in the photo or where in
`the photo it appears.
`Anotherrelated limitationofprior art websites is that the
`captions described above contain static information only,
`such as names, and do not provide a wayto obtain additional
`information about a person or object such as contact infor-
`mation or to locate additional photos of the person or object.
`Another problem with photo-specific websites is that
`users wishing to share their photos must proactively e-mail
`or otherwise send their photos to others (i.e., a photo is not
`automatically shared whena user is described in a photo).
`i2
`For example, evenif a user describes a photos as containing 5
`“Jane@doe.com” and “John@doe.com”, existing website
`will not share the photos with these individuals. Through a
`separate process,
`the user must still email this photo or
`otherwise share the photo with, in this case, Jane and John.
`Although photo-specific websites ease this process, it is still
`time consuming for users with large photo albums and/or a
`large number of friends to send their photos to all such
`persons.
`
`40
`
`45
`
`5
`
`SUMMARY OF THE INVENTION
`
`60
`
`An embodiment of the invention comprises a system and
`method for identifying and displaying objects (an most
`cases, a person) within a digital image over a system of
`interconnected computers (e.g. client/server, peer-to-peer,
`elec... .). For example, a user can identify where a person
`exists in an image byselecting the person’s name(e.g. from
`
`4
`a list) and clicking on the person’s location in the image. By
`identifying people within images,
`the system is able to
`search for images where a specified person(s) appears as
`well as generate output that recognizes where in the image
`a person appears. This output is also capable of creating
`hyperlinks and calls to client-side scripts from the otherwise
`“static” image file. In an embodiment of the invention the
`system can also utilize artificial intelligence algorithms to
`locate other images where the identified object has not yet
`been defined bya user of the system.
`While, one embodiment of this invention utilizes the
`Internet as the medium for communication and web pages
`for displaying data, the scope of this inventionis not limited
`specifically to the Internet and/or to web pages. The inven-
`tion contemplates the use of any display output in a com-
`puter environment (e.g. any types of document). The terms
`person, people, and/or object are used interchangeably and
`meant to refer to any object(s) (e.g. a pet, a possession, a
`place, or a thing) a user wishes to identify within a digital
`image. ‘his most frequently will, in fact, be a person. When
`shownwithin the boundsof an image, a personis considered
`akin to any type ofobject. ‘hus, the reader should note that
`the invention can be utilized to identify any type of object
`within an image.
`Depending on the computer environment, different media
`will be used for output and display to the client. On the
`Internet, the most accessible and most commonoutput is
`Hypertext Markup Ianguage (“HTMI.”’). However,
`the
`functionality of HTML can be duplicated or configured in
`other non-HTML media. For the sake of both brevity and
`clarity, I will outline an embodimentof the present invention
`that utilizes IITML. This embodiment of the present inven-
`tion (i.c. utilizing the Internct and HTML)is logical duc to
`the broad reach of the Internet and wide acceptance of
`HTML. Howeverthe scope of the invention is not intended
`to be limited to the use HTMLorthe Internet.
`
`DESCRIPTION OF THE DRAWINGS
`
`An embodiment of the invention is described in detail
`below with reference to the attached drawing figures,
`wherein:
`FIG. 1 is a schematic diagram of computer and commu-
`nications equipment that may be used to implement an
`embodiment of the present invention;
`FIG. 2 is a schematic diagram of databases that may be
`used to implement certain aspects of the invention;
`FIG. 3 is a flow chart illustrating a high-level overview of
`the steps of an embodiment of the invention.
`FIG.4 is a schematic diagram of an identifying page that
`may be used when identifying objects within an image in
`accordance with an embodiment of the invention;
`VIG. 5 further describes the genera! details relating to the
`process for requesting, gencrating, and transmitting an iden-
`tifying page to the requestor(e.g., the client computer).
`FIG.6 is a flowdiagram illustrating a general overviewof
`the process for using the identifying page to obtain identi-
`fying information (e.g., information associated with object
`(s) in an image) from the user.
`FIG.7 is a schematic diagram ofa display page that may
`be used to display images and information about objects that
`have beenidentified. within the image in accordance with an
`embodiment of the invention.
`FIG.8 is a flow chart illustrating a high-level view of the
`process for providing display page to a requesting user in
`accordance with an embodiment ofthe invention.
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 013
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 013
`
`

`

`US 9,959,291 B2
`
`5
`FIG. 9 illustrates various ways to display identification
`information,
`including the various states a user interface
`configured to operate in accordance with an embodimentof
`the invention may have.
`FIG. 10 is a flow chart describing the process for gener-
`ating a set of search results in response to a request.
`The figures do not limit the invention to the specific
`embodiments disclosed and described herein. The drawings
`are nol necessarily to scale, emphasis instead being placed
`upon clearly illustrating the principles of the invention.
`
`DETAILED DESCRIPTION
`
`6
`
`WebTV™., Each client computer 100 may also include or
`can access a conventional Internet connection such as a
`modem, Digital Subscriber Line (DSL) converter, or Inte-
`grated Service Digital Network (ISDN) converter and a web
`browser that permits it to access and view data over the
`Internet.
`The communications network 150 may be the Internet but
`can also be any other communications network such as a
`local area network, a wide area network, a wireless network,
`an intranet or a virtual private network. Hereinafter, the term
`Internet or communication network refers to any type of
`interconnection fabric that provides computers coupled to
`the network with a mechanism for transmitting and/or
`receiving data.
`The computer program or programs embodying one or
`more aspects of the invention are stored in or on compuler-
`readable mediumresiding onor accessible by host computer
`200 and provide a mechanism for instructing host computer
`200 to operate the invention as described herein. The com-
`puter programs typically comprise ordered listings of
`executable instructions for implementing logical functions
`in host computer 200 and user computers 100 coupled with
`host computer 200. The computer programs can be embod-
`ied in any computer-readable medium for use by or in
`connection with an instruction execution system, apparatus,
`or device, such as a computer-based system, processor-
`containing system, or other system that can fetch the instruc-
`tions from the instruction execution system, apparatus, or
`device, and execute the instructions. In the context of this
`application, a “computer-readable medium” can be any
`means that can contain, store, communicate, propagate or
`transport the program for use by or in connection with the
`instruction execution system, apparatus, or device. The
`computer-readable medium can be, for example, but not
`limited to, an electronic, magnetic, optical, electro-mag-
`netic, infrared, or semi-conductor system, apparatus, device,
`or propagation medium. Examples of the computer-readable
`medium would include the following: an electrical connec-
`tion having one or morewires, a portable computerdiskette,
`a random access memory (RAM), a read-only memory
`(ROM), an erasable, programmable,
`read-only memory
`
`(EPROMorFlash memory), an optical fiber, and a portable
`compact disk read-only memory (CDROM). The computer-
`readable medium could even be paper or another suitable
`medium upon which the programis printed, as the program
`can be electronically captured, via for instance, optical
`scanning of the paper or other medium, then compiled,
`interpreted, or otherwise processed in a suitable manner, if
`necessary, and then stored in a computer memory.
`In an embodiment of the invention, host computer 200
`comprises server engine 220 which is programmedto oper-
`ate or host a website and serve as a repository for images and
`identification information for objects within the images as
`described in more detail below. The images may be photo-
`graphs, graphics, artwork, or any other digital image that
`contains or depicts one or more objects. The objects within
`the images may include people, animals, plants, buildings,
`places, or anything else shown in images. In one embodi-
`ment of the invention, the images and objects are referred to
`(but not limited to) as photographs and people, respectively.
`The reader should note that the term object as used herein
`includes anyidentifiable entity within the image data.
`Server engine 220 can access web page templates 210,
`databases 230, 240, and 250, and internal or external
`memory. Web page templates 210 can be any technology
`used to transcribe data from a data source into a web page
`or otherwise provide “dynamic” or customized documents,
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 014
`
`a wa
`
`IcS
`
`30
`
`The invention described below is a method and apparatus
`for obtaining information relating to the existenceof at least
`one object in an image. In the following description, numer-
`ous specific details are set forth to provide a more thorough
`description of the present invention. It will be apparent,
`however, to one skilled in the art, that the present invention
`may be practiced without these specific details. In other
`instances, well known features have not been described in
`detail so as not to obscure the present invention.
`General System Architecture
`Embodiments of the invention can be implemented in
`i]a
`hardware, software, firmware. or a combination thereof. In :
`one or more embodiments, however, the invention is imple-
`mented with one or more computer programs that operate
`one or more pieces of computer equipment broadly referred
`to by the numeral 101 in FIG. 1. Such computer equipment
`101 typically includes a server computer 200 (also referred
`to as a host computer) and any number of user computers
`100 (e.g., client computers) that may access the host com-
`puter 200 via a communications network 150. The computer
`programsand equipmentillustrated and described herein are
`merely examples of programs and equipment that may be
`used to implement an embodimentof the invention and may
`be replaced with other software and computer equipment
`without departing from the scope of the invention. For
`example, any computational device configured to display
`image data may be configured to implement one or more
`aspects of the invention.
`Host computer 200 may be any computing device such as
`a network computer running Windows 2000, Novel Net-
`ware, Unix,or any other network operating system. The host
`computer 200 may be connectedto a firewall computer at the
`boundaries of network 150 to prevent tampering with infor-
`mation stored on or accessible by the host computer 200 and
`to another computing device that may operated by an
`aciministrator of the host computer 200. If the invention is
`i2
`implemented with the Internet, the host computer 200 may 5
`include conventional web hosting operating software, an
`Internet connection such as a modem, DSL converter or
`ISDN converter, and be assigned an IP address and corre-
`sponding domain nameso that the website hosted thereon
`can be accessed via the communications network 150.
`Client computer(s) 100 may be used by anyone wishing to
`upload and/or view photographs via host computer 200. The
`client computers 100 may be any device or group ofdevices
`that permits access to host computer 200 via communica-
`tions network 150, including, but not limited to, personal or
`laptop computers such as those manufactured and sold by
`Dell™, Compag™, Gateway™, or any other computer
`manufacturer, handheld personal assistants such as those
`operating with Palm’s PalmOS™ operating software or
`those operating with Microsoft’s PocketPC™ operating
`software, mobile phones, or even Internet appliancesthat are
`specifically designed for accessing the Internet such as
`
`40
`
`45
`
`Meta Platforms, Inc.
`Exhibit 1001
`Page 014
`
`

`

`US 9,959,291 B2
`
`8
`The user, John Doe, may then upload a photo, “my_im-
`age.jp2”, which maybe assigned an image ID. of “ABCD”
`in Images database 250. Images database 250 could then be
`populated to include the following record:
`
`1
`
`ID
`eee
`ABCD
`
`Cant
`epnon
`John and Jane al the
`beach
`To identify the user, John Doe, with this image, my_im-
`age.jpg, host computer 200 inserts into Identifications data-
`base 240 a record with John Doe’s User LD.(i.e. “007”) in
`the User I.D. field and my_image.jpg’s Image LD. (ve.
`“ABCD”)in the Image LD.field. ‘This creates a relationship
`between this user and this image. Assume now it is desired
`to identify John Doe as appearing in a certain area within the
`image. In HTML, image coordinates are used to define an
`area within an image. By storing the image coordinates in
`this record, we can define specifically where within my_im-
`age.jpg that John Doc appears. Assuming John Doc appears
`in the circular area bound by the coordinates 50,50,25
`(x-axis location, y-axis location, radius ofcircle), the Iden-
`tifications database 240 entry would appear as follows:
`
`7
`limited to, Active Server Pages, Java
`including but not
`Server Pag