`
`ps
`
`2mare
`Mh,
`ernie,
`accessenn
`ou/
`wane
`ene
`NrneearnIPLOENAAAS
`
`speHORNAAAARAN
`OR
`ances
`hn
`ay
`na
`
`ARRANERaEEEETT
`
`aa
`
`C3
`
`ONE
`JANAARENA
`NNN
`cnn
`we
`~
`
`daShay.
`CAROAAA
`
`feen,
`
`
`
`
`
`
`
`PiaCdADELPCELEELEEEECOOLELECCEDCEDLSCLOTELLPLLEPPLELELLCPTDEECOODEPELSIEAEPAIUEDVMATLEEECODEDCOEDEEREPETE
`
` VENFS
`
`
` PTEIEEmyggZ%gs%peerscereanncamerrnrrterercceenerereracegretigeeranstsssrinssiripnsnisstisensnrcnrearerdenisrrerenterbeneety,wernerse
`
`
`eT
`
`
`
`WhARNTAETAIIDDEIDETEDIPDPSDSSEDIIDIPOPPOOETOP!
`
`
` ncnéLirterrnerrnrenenEDETLEDbbeaoenihTERTTEECRLETOCISCCIObITT4%oS“e
`
`
`
`fa
`
`mG
`
`53
`
`can]
`
`te}
`
`<b
`
`St
`
`a
`
`eee
`
`a
`“SRAereSRRSREVE
`
`SeEVERTENREE
`
`ifuA
`
`‘444i4jHi4
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 001
`
`
`
`
`
`
`
`
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 001
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ChewGaeCe|20GCR7Su|AgeteeerEpeSe“2mee2b
`oOmeOt
`
`
`
` :Bno3!:33gsPoaes;:yyi.teL&PoiveeneeBeerLE“AaoonwagefooFeeatpefenact
`
`
`ceeitceetiratniccttetearit.wefronBoeeOF%SyeoE
`.aeeeosieA.ae)if
`
`wrFLieweebdEyboon:ESewe
`
`
`
`
`co
`
`oo‘°wr,4LA
`raemafrohAAL-my,~,co\“A
`
`
`-%tNco
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 002
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 002
`
`
`
`
`
` fis
`iad ae
`
`HRNNET
`
`NATE A
`at ae? AW
`
`shldeet
`
` fishedDientetenAhada
`
`Sone
`
`,S
`* “eaccconecnstell
`nt AARPEIERey
`f°sennonll
`
`*heavenanereetery
`
`free,
`
`intrpice
`C
`
`DAREEAAAthalOeEde
`
`|
`Sy
`X3 Jo
`ye
`¥
`tA
`&} Jane
`
`oy
`
`i
`
`
`gRnEeRRENNEPONASHINNINNENANIAEANANIENNNNNN
`AAT
`
`AANAANATOANATANNAANGATNNeTNTNNSa
`
`— EXT
`ane 2a
`
`if
`t
`SRIINRIENNNARNIANINNNNNANNIESSHONNNANNNNNN
`
`
`
`
`etLLAELOMDALAdAilEPApACCLLALAELOELAUTESTATEALLAELUATESALAALLSURUUSOESATCUOOLTOOLEOSUESLTVEITUSELLLOUILELILLPUTLUEEEELE
`
`8TNMAELOEAEtPEARLLOLRETLETAEEUDDUTEECOTALUTEEKIOTTERTOTOALSTEAERIDALLESTACOAAEELEE
`
`ELELATOEIEIOTODO,
`
`wt
`
`4,%
`
`t,Ceeeputsing,
`
`ARASHRNNNNNANNANNENT
`ye
`
`many
`es
`
`Stara
`
`UBMIT
`"agenneni”
`
`EE
`
`CORES
`SENRNANNNNRANANTESNAANINNNINN SoSOEENNETSONSIORNINSANDSNANNINGIRSEANRDN
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 003
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 003
`
`
`
`
`
`
`
`ae
`
`
`
`EQUEST FOR
`SENG
`
`IDENTIFYING PYAGEBP8
`
`{9.9.3SEAVERPAGE)
`
`
`L_. O SERVER_
`
`502
`
`(
`
`
`
`Bee
`
`
`
`¥
`
`t
`oe eee
`
`DETERMINE DESIRED
`ENVIRONMENT
`
`
`ciel ane or
`| RETREIVE IDENTIFYING PAGE Le
`
`TEMMPLATEFORENVIRONMENT |
`
`
`=
`nn DoMbtion| a
`“ey
`ACTER
`AR
`TOES 77808
`) RETAEIVE INFO IMATION|
`
`&FROM |IMAAGES DATAE aan
`
`waes
`
`FROM USERS DATABASE
`
`(OPTIONAL FILTERING)
`
`
`
`ainrmtinnneannndnasenannaasinnioay
`
`rm ee| RETREIVE INFORMATION | --808
`
`
`
`|
`|
`i
`
`
`TRANSCRIBE INFORMATION |
`FROM| DATABASES INTO
`b
`mr
`
`PAGE TEMPLATE [
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 004
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 004
`
`
`
`of
`
`mie
`IDENT!
`RECEIV!
`“
`SAL
`ad
`PEA
`E
`x~}
`JEN
`Ct
`
`t&
`
`A
`
`wt
`2
`
`"O
`
`gee fees,
`Ne
`}
`ELECT OBJE
`Pt
`x
`A
`DENTIEY IN
`|
`
`INFORMAT
`
`ASRS
`
`
`
`
`———ier
`
`ay
`
`is
`
`specienoiaaS=
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 005
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 005
`
`
`
`
`
`
`
`
`X
`.
`e
`
`oo
`ey
`k
`yo
`x
`:
`3
`x
`:
`2
`:
`3
`Skee
`wr
`
`Yo
`
`
`
`8s
`
`
`
`s:
`
`3
`,
`,
`‘
`x
`X
`
`wn
`a 8
`x
`r4
`SHE 3
`ae
`PEOPLE INIMAGE
`Rew
`PPM PEVEPSARS
`3
`RO
`
`RMONNNINSNANNING
`ASSENTAEEI e COATESx
`
`
`i
`geal! ow &
`g
`i
`.
`Bee
`
`Ras 5smaneannasasaNSURINSANRSSNSSHOUDEEEESASAEARENNSASNAST
`xx
`: ——— ————_—__—_—_—_—_—
`ON
`8
`‘
`s
`&
`3
`x
`&
`[os &
`:
`¢
`x “ Sey
`‘
`‘
`§
`;
`8
`Ys
`a
`og
`&
`& of
`s
`‘
`y
`‘
`42
`: ,
`:
`¢
`x
`poem
`yf
`§
`g
`g
`x
`8
`$
`s
`$
`lakes
`f
`o
`s
`A og
`U- a
`:
`.
`;
`¢
`i Jann Doe |
`
`,
`|
`x
`i.
`ere
`x
`8
`¢
`8
`x
`8
`\
`:
`‘
`‘
` s
`;
`\
`:
`Seg
`8
`8
`Pare Theis
`y
`x
`Stason
`8
`8
`Jae Le
`8
`S
`8
`i
`:
`3
`.
`x
`s
`RarmnemeaentaancneeencennnNAANNTN
`g
`8
`QoAAAANAMAANANTHNTN
`8
`;
`:
`s
`3
`g
`8
`a
`x
`8
`_
`‘
`s
`x
`3
`3
`oN
`8
`8
`SS
`‘s
`=
`=
`&
`,
`8
`‘
`Fes ON
`x
`8
`s
`i
`s
`§
`x
`x
`S
`s
`:
`cae ay \
`s
`&
`‘
`‘
`:
`s
`‘
`3
`yA
`Ss
`§
`:
`e
`:
`8
`8
`:
`4
`e
`>
`s
`x
`3
`$
`g
`S
`&
`x
`x
`3
`:
`5
`8
`:
`eag
`s
`‘
`¥
`s
`4
`8
`:
`3
`s
`:
`3
`,
`‘
`:
`3
`3
`Sout
`g
`8
`g
`aa:
`3
`4
`g
`s
`Nenwas
`4
`3
`s
`x
`A
`s
`3
`SaemmmanseenannrcanennnnenncANCONA
`8
`8
`x3
`3
`
`8 8&seossaasnantagonistaARiBANOTANHERSISSOERADEARESSENASUNONOAENEENITNNT
`
`x
`‘
`s
`3
`<
`R
`8
`
`3
`8
`s
`s
`8
`8
`8
` (
`s
`s
`8
`‘
`8
`5
`8
`=
`8
`s
`8
`s
`j
`
`pA
`pes eS,
`2;
`er
`gs
`$
`PEOPLE INIMAGE
`g
`fe
`Lio EN
`FS
`3 AS 3
`
`4
`So
`A
`
`
`
`“oar
`
`ts
`Pies
`ae
`Jim Doe
`See egy Mo zh
`UE LOG
`Oy gi
`ey
`oA et
`~
`Ws od
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 006
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 006
`
`€
`
`
`
`
`i
`ure & Feca
`
`a
`REE
`BOO
`OBTAIN A REQUEST FOR [7
`IMAGE DISPLAY PAGE |,
`atneiinmceimenl
`
`~~
`| DETERMINE DESIRED
`ENVIRONMENT
`
`a phe
`L
`
`
`
`BOs
`-
`;
`Ly
`RETAEIVE DISPLAY PAGE
`TEMPLATE FOR ENVIBONMENT |
`
`
`NAMING INFOSMATON
`i
`
`SegnogpannannantieviesiuaNNORNETEOEAREERERFMANAS
`eeecateaNRRSRSS ; =TreeeeeITEOEE OEE
`
`vOBTAINIMAGEAND
`
`| B08
`
`
`eli eeeeeee
`
`
`:| TRANSCRIBE IMAGE AND NAMING
`=:
`INFORMATION INT
`:
`DISPLAY PAGE TEMPLATE
`
`
`
`
`TRANSCRIBE
`|
`COORDINATE
`
`[INFORMAT)ONINTO:
`'
`SPLot PAGEo
`NO
`;
`7
`‘an
`t
`
`
`
`
`¥
`|
`RRAAROBONNENNHRANERALERemma
`,"PROVIDETHE IMAGE en
`DISPLAYPAGE |
`“LoTOA RECEIVINGUSER
`INTIATING
`REQUEST
`E
`
`ss ttopnionneranpomannaennnsrnraanwrrsseynnancananneananatpaneer
`
`po
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 007
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 007
`
`
`
`
`
`
`
`ELLLTLELSEPTLESESELDLLTYELLELLOOTLLLELIEESPPTELLOELLPLLLLISSLLEEOTALELD
`
`end\git2poererreereaabocnnnnn.
`coEnrontnenniaansapanonraratenetgjywOo<i
`
`
`
`
`
`
`
`iYipA4;4i3go!edoe§aya4ooaha.?CaeP|—CoiyJi};i’j4aGoeeeweifj:AmeabonA
`
` ZZZgXsma2@an|fopovalBY7Zg%“tn49e.44|8jL|meyygOaI,ZOBm,||gyg4pety3i| UP™oe1-4.tA‘EIBGMil2&64ranrendbeeee4iijiaseet;oyfjonLaSod|Be4/8as;,GFanoD:1;Daerrecrccercereeorenucrenereceararenerereareavennrecseiaih4eeEegGeyyyy]<0)4x!%8igos%4abeen4SySsLOLELLLLLLATTELITTLELLLETUTTLEchGUre.41iDSoeos\;aoS
`
`
`
`
`€-\,yIfPLLLLLERLLEELSELLLELLLLSLLLLLLLEELEELSLILLELLLPEELSftoo
`g=jNGitBiipyay|FLLRFEwe
`
`
`
`
`be00TLRUPEPELSOOTLULLTULLUDAEEDLDEULYELLEUISTLITELECEESEPLOLPLSSEDEDELSTOSTELEED
`RAREZmA3gInj%|oN,4fig4aeymF.;o.4§4.)OEsZZvgHieos5*.taay%og|2yg|im|aceg[4|®ro|&onZgoooxg ¥tHs_PsBaa“weByUgoii=4cgogx4474img
`offofi=SSx|
`ioF°rat&ee-,m4becbeecd“eat
`Omom|Z|
`4%g~
`4g9gipomeerrm“ng
`
`PLATESLSTSTENEIDELEDALEDLLSPULLPLLVELLEbygjceefproeemereneveerfine;fTLLEILTAEEOAOARIserene
`
`
`
`
`BLLLIILEZLYMLLLEDIRPLILOTALLREDALLLDLLITSOSISETTOLLPOTLE
`
`
`
`Oo
`
`web
`
`eR
`
`a5
`
`ee
`
`S x
`
`AANATONSA
`
`Afier (Alc
`
`i>
`kod
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 008
`
`sronniensanunnnnananmnnnnnnanniiniii,OOLIIEALEALLDLELLPEPETETTEEEEELLITSLETLSEPILTELELDDREEECLM2CIDDIISLLPLAISTPLLSITET8OILSIGSPYITLIVIPTIGISTESTILTECTSSETIELLISLSYSLIPASE
`
`e=g|%||j|
`
`Figure S
`wet
`
`2
`
`t 8
`
`00
`
`
`
`
`
`
`
`
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 008
`
`
`
`
`
`
`
`Figure 10
`
`
`
`CLIENT SENDS REQUESTFOR=| -t000
`
`L J
`r
`
`IMAGES (OR LIST OF IMAGES)
`CONTAINING ONE
`OR MORE OBJECTS -
`
`
`ENVIRONMENT
`
`
`DETERMINE DESIRED
`ENVIRONMENT
`
`(per ae
`- TEMPLATEFOR
`
`OBTAIN SEARCH RESULTS
`
`p-t004
`
`
`
`eeeeeET
`METREIVE RECORDS
`— f
`
`
`PROM
`IDENTIFICATIONS DATA
`
`
`|
`
`“TRANSCRIRE DATA ~
`
`|
`
`i
`
`4
`
`INTO SEARCH RESULTS
`
`TEMPLATE
`
`
`
`
`
`SEND LIST OF IMAGES AND/OR | —a¢
`~SEARCH RESULTS PAGE
`TO CLIENT
`
`
`
`
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 009
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 009
`
`
`
`UNITED STATES PATENT APPLICATION
`
`FOR
`
`TAGGING DIGITAL IMAGES
`
`INVENTOR:
`
`MARK FRIGON
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 010
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 010
`
`
`
`
`CROSS-REFERENCE TO RELATED APPLICATIONS
`
`This is a continuation of application Ser. Nr. 09/991,324 presenting under
`
`examination. The non-provisional application Ser. Nr. 09/991,324 takes priority from
`
`U.S. Provisional Application Number 60/248,994 filed on November15, 2000.
`
`
`BACKGROUNDOF THE INVENTION
`
`This non-provisional application takes priority from U.S. Provisional Application
`
`Number 60/248 ,994 filed on November 15, 2000.
`
`10
`
`
`FIELD OF THE INVENTION
`
`The present invention relates to computer software. More particularly, the
`
`invention relates to a method and apparatus for storing and sharing images such as
`
`photographs via a communications network and for permitting the identification of
`
`objects and the location of the objects within the images. The invention enables users to
`
`15
`
`supply and/or receive information about the existence of objects within images.
`
`Portions of this patent document contain material that is subject to copyright
`
`protection. The copyright owner has no objection to the facsimile reproduction by
`
`anyone of the patent documentor the patent disclosure as it appears in the Patent and
`
`Trademark Office file or records, but otherwise reserves all copyrights whatsoever.
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 011
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 011
`
`
`
`BACKGROUND
`
`Digital photography allows people to quickly and easily take, print, and/or share
`
`photos withoutfilm and the associated film developing costs. The popularity of digital
`
`photography has increased dramatically due to the increased availability of digital
`
`cameras and the widespread acceptance of the Internet. Many Internet users have created
`
`websites that organize their digital photographs into online photo albums that can be
`
`accessed by otherInternet users. These web pages permit a large number of people to
`
`view the photos and allow the author to both add and remove photos from an album.
`
`10
`
`Although creating web pages for online photo albumsoffer advantages over
`
`traditional photo albums, their creation requires programming expertise such as skill in
`
`Hypertext Markup Language (HTML). Manypeople do not havethe technical
`
`knowledge and/or time to build their own online photo albums. Sharing of photos from
`
`online photo albumsis also problematic and requires that each photo be manually sent, e-
`
`15
`
`mailed, or downloaded. If the recipient wishes to use the photos in their own web pages,
`
`each photo must be then uploaded to an additional site. These steps are time consuming,
`
`duplicative, and also require programming knowledge.
`
`To address these problems, several websites that specialize in online photo
`
`20
`
`albums have been created. These websites allow users without programming skills to
`
`quickly and easily create and maintain online photo albums. Users are only required to
`
`upload photos that they wish to add to an online photo album. Thesites, through
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 012
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 012
`
`
`
`programming on the webserver, or "server-side scripting’, then create all necessary
`
`HTMLprogramming to add the uploaded photos to the users’ photo albums. Many of
`
`these sites also offer image enhancementcapabilities (e.g. sharpen, brighten, reduce "red-
`
`eye”, etc.), as well as photo printing and developing functions. Photo-specific sites have
`
`also eased the process of sharing photos by providing a meansfor users to quickly and
`
`easily e-mail links to photos or photos themselves to one anotherand, in somecases,
`
`contribute photos directly to other's photo albums.
`
`While websites that specialize in the creation of online photo albums offer many
`
`advantages, they suffer from several limitations. For example, existing websites do not
`
`10
`
`offer users the ability to identify objects within photos. If a user is viewing a photo and
`
`does not know certain individuals in the picture, the user is unable to use current systems
`
`to determinetheir identities. Typically, photo album sites offer the ability to describe
`
`uploaded photos throughthe use of captions or other descriptive fields. These fields may
`
`be used to describe or name individuals within the photo. For example, a user may
`
`15
`
`upload a photo and give it a caption of "Stephanie rollerblading with Tauny”. This
`
`caption aloneis not sufficient information to ensure the viewercan correctly determine
`
`the identities of the individuals within the photo, because to do so would require prior
`
`knowledge of which person is “Stephanie” and which is “Tauny’”. A viewer who does
`
`not know these individuals may have difficulty or be unable to correctly identify the
`
`20
`
`individuals within the photo. Furthermore, less descriptive captions such as "The Family
`
`at the beach” would offerlittle help unless the viewer knowseach individual in the photo.
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 013
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 013
`
`
`
`Thus, in order to provide sufficient information required to ensure a viewer can identify
`
`individuals within a photo, copious descriptions must be input and maintained byusers.
`
`Typing such captions or descriptions is very time consuming andtedious, especially
`
`when a large numberof photos are uploaded. Furthermore, when the same person
`
`appears in multiple photos, the submitter is still required to retype the identification
`
`information into a caption for each photo.
`
`Processes have been developed to identify individuals within photos. For
`
`example, one process requires people to wear badges containing identification
`
`information such as unique color codings whentheir photos are taken. When a computer
`
`10
`
`processes a photo containing a badge, the computer can access a database to determine
`
`which badge was assigned to which person and then identifies the person as existing in
`
`this photo. This process has many drawbacksincluding the dependenceon physical
`
`badges. If the badges cannot be seen in the photos, the people wearing the badges cannot
`
`be identified. Moreover, this process is not functional for the vast majority of
`
`15
`
`photographs: any photos that were taken outside of this tightly controlled environment.
`
`Another problem with photo-specific websites is that they have no search
`
`capabilities that permit users to search for photos of specific individuals. Users wishing
`
`to view all photographsthat contain a particular person must manually access all possible
`
`photo albumsto find the desired photos. For example, if a user wishes to find all pictures
`
`20
`
`of a particular friend, the user may have to browse through several of his/her own albums
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 014
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 014
`
`
`
`where that friend may appear (e.g. "High School pictures”, "Reunion Weekend”, etc.), as
`
`well as access and browse through each of the friend’s albums, or any third-party’s
`
`albums where the friend may appear. With a reasonable numberof photos, this becomes
`
`time-prohibiting if not impossible. Search features could be designed to search
`
`aforementioned descriptive fields (e.g. a caption), however, as there is no naming
`
`standard, people are described differently by different people (e.g. first name, last name,
`
`full name, nickname, etc.) and many names would be used to describe multiple people
`
`(e.g. Mom, Dad, John,etc.), As a result, it would be difficult to have a one to one
`
`relationship between descriptions of people and people themselves. Consequently,
`
`10
`
`searches would be unreliable, likely returning many photos oflittle relevance andfailing
`
`to return photos that would be of relevance. For example, a search for the term “John
`
`Smith” would fail to return photos in which this person wasreferred to as “John”or,
`
`perhaps, “Dad”. However, this samesearch likely would return photos of other
`
`individuals who share the same name “John Smith’.
`
`15
`
`While reliable database searchability for digital images over the Internetis
`
`available, it has not been implemented with photo albums. Particularly, several
`
`companies whichsell and license images have put catalogs of digital photographs on the
`
`Internet. Such websites generally describe images in terms of themes(e.g. sunset, dark or
`
`sad) or broad descriptions of objects therein (e.g. beach, farmland, or water), without
`
`20
`
`identifying the objects explicitly. For example, a picture of a man and womanonthe
`
`beach at sunset, which could be licensed to a beach resort for print or web use, could be
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 015
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 015
`
`
`
`identified as “beach”, "sunset”, and/or "man and woman’. While such descriptive terms
`
`can provide a reliable mechanism for searching for images matching the terms, they do
`
`not uniquely identify any objects in the images, nor do they have the capability of
`
`identifying where these objects appear within the images. For example, a search forthe
`
`term “clown”is likely to return many pictures of clowns, however, it is very unlikely that
`
`every clown pictured would be the same person,just as a search for the term "beach”is
`
`unlikely to return pictures of the same beach. Even a search for the term “President” will
`
`return several photos of different people. Furthermore, the objects that appear within a
`
`photo are not identified as being in a particular area within the photo, they merely are
`
`10
`
`described as being in the photo. For example, websites that return a photograph from a
`
`"President” search do not havethe capability of identifying that the Presidentis in,
`
`perhaps, the lower right corner of the picture. Such technologies have the capability to
`
`describe the nature or theme of a photo but not to uniquely identify who or whatis in the
`
`photo or where in the photo it appears.
`
`15
`
`Anotherrelated limitation of prior art websites is that the captions described
`
`above contain static information only, such as names, and do not provide a way to obtain
`
`additional information about a person or object such as contact information or to locate
`
`additional photos of the person or object.
`
`Another problem with photo-specific websites is that users wishing to share their
`
`20
`
`photos must proactively e-mail or otherwise send their photos to others (i.e., a photo is
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 016
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 016
`
`
`
`not automatically shared when a user is described in a photo). For example, evenif a
`
`user describes a photos as containing “Jane@doe.com”and “John @doe.com”, existing
`
`website will not share the photos with these individuals. Through a separate process, the
`
`user must still email this photo or otherwise share the photo with, in this case, Jane and
`
`John. Although photo-specific websites ease this process, it is still time consuming for
`
`users with large photo albums and/ora large numberof friends to send their photosto all
`
`such persons.
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 017
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 017
`
`
`
`SUMMARYOF THE INVENTION
`
`An embodimentof the invention comprises a system and methodforidentifying
`
`and displaying objects (in most cases, a person) within a digital image over a system of
`
`interconnected computers(e.g. client/server, peer-to-peer, etc...). For example, a user can
`
`identify where a person exists in an image by selecting the person’s name(e.g. from a
`
`list) and clicking on the person’s location in the image. By identifying people within
`
`images, the system is able to search for images where a specified person(s) appears as
`
`well as generate output that recognizes where in the image a person appears. This output
`
`10
`
`is also capable of creating hyperlinks and calls to client-side scripts from the otherwise
`
`“static” image file. In an embodimentof the invention the system can also utilize
`
`artificial intelligence algorithms to locate other images where the identified object has not
`
`yet been defined by a user of the system.
`
`While, one embodiment of this invention utilizes the Internet as the medium for
`
`15
`
`communication and web pages fordisplaying data, the scope of this invention is not
`
`limited specifically to the Internet and/or to web pages. The invention contemplates the
`
`use of any display output in a computer environment(e.g. any types of document). The
`
`terms person, people, and/or object are used interchangeably and meantto refer to any
`
`object(s) (e.g. a pet, a possession, a place, or a thing) a user wishesto identify within a
`
`20
`
`digital image. This most frequently will, in fact, be a person. When shown within the
`
`boundsof an image, a person is considered akin to any type of object. Thus, the reader
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 018
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 018
`
`
`
`should note that the invention can be utilized to identify any type of object within an
`
`image.
`
`Depending on the computer environment, different media will be used for output
`
`and display to the client. On the Internet, the most accessible and most commonoutputis
`
`Hypertext Markup Language (“HTML”). However, the functionality of HTML can be
`
`duplicated or configured in other non-HTML media. For the sake of both brevity and
`
`clarity, I will outline an embodimentof the present invention that utilizes HTML. This
`
`embodiment of the present invention(i.e. utilizing the Internet and HTML)is logical due
`
`to the broad reach of the Internet and wide acceptance of HTML. Howeverthe scope of
`
`10
`
`the invention is not intended to be limited to the use HTMLorthe Internet.
`
`10
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 019
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 019
`
`
`
`DESCRIPTION OF THE DRAWINGS
`
`An embodimentof the invention is described in detail below with reference to the
`
`attached drawing figures, wherein:
`
`5
`
`Figure 1 is a schematic diagram of computer and communications equipmentthat
`
`may be used to implement an embodimentof the present invention;
`
`Figure 2 is a schematic diagram of databases that may be used to implement
`
`certain aspects of the invention;
`
`10
`
`Figure 3 is a flow chartillustrating a high-level overview of the steps of an
`
`embodimentof the invention.
`
`Figure 4 is a schematic diagram of an identifying page that may be used when
`
`15
`
`identifying objects within an image in accordance with an embodimentof the invention;
`
`Figure 5 further describes the general details relating to the process for requesting,
`
`generating, and transmitting an identifying page to the requestor (e.g., the client
`
`20
`
`computer).
`
`11
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 020
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 020
`
`
`
`Figure 6 is a flow diagram illustrating a general overview of the process for using
`
`the identifying page to obtain identifying information (e.g., information associated with
`
`object(s) in an image) from the user.
`
`Figure 7 is a schematic diagram of a display page that may be used to display
`
`images and information about objects that have been identified within the image in
`
`accordance with an embodiment of the invention.
`
`10
`
`Figure 8 is a flow chartillustrating a high-level view of the process for providing
`
`display page to a requesting user in accordance with an embodimentof the invention,
`
`Figure 9 illustrates various waysto display identification information, including
`
`the various states a user interface configured to operate in accordance with an
`
`15
`
`embodimentof the invention may have.
`
`Figure 10 is a flow chart describing the process for generating a set of search
`
`results in response to a request.
`
`12
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 021
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 021
`
`
`
`The figures do not limit the invention to the specific embodiments disclosed and
`
`described herein. The drawings are not necessarily to scale, emphasis instead being
`
`placed uponclearly illustrating the principles of the invention.
`
`13
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 022
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 022
`
`
`
`DETAILED DESCRIPTION
`
`The invention described below is a method and apparatus for obtaining
`
`informationrelating to the existence of at least one object in an image. In the following
`
`description, numerous specific details are set forth to provide a more thorough
`
`description of the present invention. It will be apparent, however, to one skilled in the
`
`art, that the present invention maybe practiced without these specific details.
`
`In other
`
`instances, well knownfeatures have not been describedin detail so as not to obscure the
`
`present invention.
`
`10
`
`General System Architecture
`
`Embodiments of the invention can be implemented in hardware, software,
`
`firmware, or a combination thereof.
`
`In one or more embodiments, however, the
`
`invention is implemented with one or more computer programs that operate one or more
`
`pieces of computer equipment broadly referred to by the numeral 101 in Figure 1. Such
`
`15
`
`computer equipment 101 typically includes a server computer 200 (also referred to as a
`
`host computer) and any number of user computers 100(e.g., client computers) that may
`
`access the host computer 200 via a communications network 150. The computer
`
`programs and equipmentillustrated and described herein are merely examples of
`
`programs and equipment that may be used to implement an embodimentof the invention
`
`20
`
`and may be replaced with other software and computer equipment without departing from
`
`14
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 023
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 023
`
`
`
`the scope of the invention. For example, any computational device configured to display
`
`image data may be configured to implement one or more aspects of the invention.
`
`Host computer 200 may be any computing device such as a network computer
`
`running Windows 2000, Novel Netware, Unix, or any other network operating system.
`
`The host computer 200 may be connected to a firewall computer at the boundaries of
`
`network 150 to prevent tampering with information stored on or accessible by the host
`
`computer 200 and to another computing device that may operated by an administrator of
`
`the host computer 200. If the invention is implemented with the Internet, the host
`
`10
`
`computer 200 may include conventional web hosting operating software, an Internet
`
`connection such as a modem, DSL converter or ISDN converter, and be assigned an IP
`
`address and corresponding domain nameso that the website hosted thereon can be
`
`accessed via the communications network 150.
`
`Client computer(s) 100 may be used by anyone wishing to upload and/or view
`
`15
`
`photographs via host computer 200. The client computers 100 may be any device or
`
`sroup of devices that permits access to host computer 200 via communications network
`
`150, including, but not limited to, personal or laptop computers such as those
`
`manufactured and sold by Dell™, Compaq™, Gateway™,or any other computer
`
`manufacturer, handheld personal assistants such as those operating with Palm’s
`
`20
`
`PalmOS™operating software or those operating with Microsoft’s PocketPC™operating
`
`software, mobile phones, or even Internet appliances that are specifically designed for
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 024
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 024
`
`
`
`accessing the Internet such as WebTV™., Each client computer 100 mayalso include or
`
`can access a conventional Internet connection such as a modem, Digital Subscriber Line
`
`(DSL) converter, or Integrated Service Digital Network (ISDN) converter and a web
`
`browserthat permits it to access and view data overthe Internet.
`
`The communications network 150 may be the Internet but can also be any other
`
`communications network such as a local area network, a wide area network, a wireless
`
`network, an intranet or a virtual private network. Hereinafter, the term Internet or
`
`communication network refers to any type of interconnection fabric that provides
`
`10
`
`computers coupled to the network with a mechanism for transmitting and/or receiving
`
`data.
`
`The computer program or programs embodying one or more aspects of the
`
`invention are stored in or on computer-readable medium residing on or accessible by host
`
`computer 200 and provide a mechanism for instructing host computer 200 to operate the
`
`15
`
`invention as described herein. The computer programs typically comprise ordered
`
`listings of executable instructions for implementing logical functions in host computer
`
`200 and user computers 100 coupled with host computer 200. The computer programs
`
`can be embodied in any computer-readable mediumforuse by or in connection with an
`
`instruction execution system, apparatus, or device, such as a computer-based system,
`
`20
`
`processor-containing system, or other system that can fetch the instructions from the
`
`instruction execution system, apparatus, or device, and execute the instructions.
`
`In the
`
`16
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 025
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 025
`
`
`
`context of this application, a “computer-readable medium” can be any meansthat can
`
`contain, store, communicate, propagate or transport the program for use by or in
`
`connection with the instruction execution system, apparatus, or device. The computer-
`
`readable medium can be, for example, but not limited to, an electronic, magnetic, optical,
`
`electro-magnetic, infrared, or semi-conductor system, apparatus, device, or propagation
`
`medium. Examples of the computer-readable medium would include the following: an
`
`electrical connection having one or more wires, a portable computer diskette, a random
`
`access memory (RAM), a read-only memory (ROM), an erasable, programmable, read-
`
`only memory (EPROM or Flash memory), an optical fiber, and a portable compact disk
`
`10
`
`read-only memory (CDROM). The computer-readable medium could even be paper or
`
`another suitable medium upon which the program is printed, as the program can be
`
`electronically captured, via for instance, optical scanning of the paper or other medium,
`
`then compiled, interpreted, or otherwise processed in a suitable manner,if necessary, and
`
`then stored in a computer memory.
`
`15
`
`In an embodimentof the invention, host computer 200 comprises server engine
`
`220 which is programmedto operate or host a website and serve as a repository for
`
`images and identification information for objects within the images as described in more
`
`detail below. The images may be photographs, graphics, artwork, or any otherdigital
`
`image that contains or depicts one or more objects. The objects within the images may
`
`20
`
`include people, animals, plants, buildings, places, or anything else shown in images. In
`
`one embodimentof the invention, the images and objects are referred to (but not limited
`
`17
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 026
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 026
`
`
`
`to) as photographs and people, respectively. The reader should note that the term object
`
`as used herein includes any identifiable entity within the image data.
`
`Server engine 220 can access web page templates 210, databases 230, 240, and
`
`250, and internal or external memory. Web page templates 210 can be any technology
`
`used to transcribe data from a data source into a web page or otherwise provide
`
`“dynamic” or customized documents, including but not limited to, Active Server Pages,
`
`Java Server Pages, Cold Fusion, PHP, CGIscripts, or even client-side scripts. Server
`
`engine 220 receives information and requests from user computer(s) 100 via
`
`10
`
`communications network 150, processes web page templates 210 by transcribing data
`
`from data sources, including databases 230, 240, and 250, into templates 210, and sends
`
`the resulting web page or documentto requesting client computer 100 as described in
`
`more detail below.
`
`In order to implementthe invention host system 200 will access a server system
`
`15
`
`or database system configured to support the functionality described. Referring back to
`
`Figure 2, one or more databases could be utilized to store the data utilized by the system.
`
`For instance, the system may utilize a users database 230, Identifications database 240,
`
`and Images database 250 as depicted in Figure 2.
`
`Users database 230 receives, stores, and provides information about people or any
`
`20
`
`other objects identified within photos, including various people who access host
`
`computer 200. Users database 230 can be populated to include, for each useror client
`
`18
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 027
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 027
`
`
`
`system, a user identifier that is unique to that user or client system, and other information
`
`such as the user’s name, e-mail address, home page address, and/ora list of contacts. The
`
`list of contacts is not necessary and may bestored in a separate database.
`
`Images database 250 receives and stores information about photos(e.g., image
`
`data) uploaded or otherwise made accessible to the network by users. Images database
`
`250 can be populated as described below to include, for each photo, a photo identifier
`
`that is unique to that photo, and the location of the imagefile on the network. The
`
`Images database 250 may include descriptive information about the photo such as a
`
`caption or the date the photo was taken. While it is preferred to have photos uploaded to
`
`10
`
`and stored on the host computer 200,it is possible to have the photos stored elsewhere,
`
`beingstill accessible to users on the network, such as another website or a personal
`
`computer connected to the network. In an embodimentof the invention, both of these
`
`processes will be referred to hereinafter as uploading. It is also possible to store the
`
`images directly into the database itself as binary objects, rather than simply storing
`
`1