prrereeneraneraerceensnrons,
`
`ps
`
`2mare
`Mh,
`ernie,
`accessenn
`ou/
`wane
`ene
`NrneearnIPLOENAAAS
`
`speHORNAAAARAN
`OR
`ances
`hn
`ay
`na
`
`ARRANERaEEEETT
`
`aa
`
`C3
`
`ONE
`JANAARENA
`NNN
`cnn
`we
`~
`
`daShay.
`CAROAAA
`
`feen,
`
`
`
`
`
`
`
`PiaCdADELPCELEELEEEECOOLELECCEDCEDLSCLOTELLPLLEPPLELELLCPTDEECOODEPELSIEAEPAIUEDVMATLEEECODEDCOEDEEREPETE
`
` VENFS
`
`
` PTEIEEmyggZ%gs%peerscereanncamerrnrrterercceenerereracegretigeeranstsssrinssiripnsnisstisensnrcnrearerdenisrrerenterbeneety,wernerse
`
`
`eT
`
`
`
`WhARNTAETAIIDDEIDETEDIPDPSDSSEDIIDIPOPPOOETOP!
`
`
` ncnéLirterrnerrnrenenEDETLEDbbeaoenihTERTTEECRLETOCISCCIObITT4%oS“e
`
`
`
`fa
`
`mG
`
`53
`
`can]
`
`te}
`
`<b
`
`St
`
`a
`
`eee
`
`a
`“SRAereSRRSREVE
`
`SeEVERTENREE
`
`ifuA
`
`‘444i4jHi4
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 001
`
`
`
`
`
`
`
`
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 001
`
`
`
`
`
`
`
`
`
`
`
`
`
`ps
`
`2mare
`Mh,
`ernie,
`accessenn
`ou/
`wane
`ene
`NrneearnIPLOENAAAS
`
`speHORNAAAARAN
`OR
`ances
`hn
`ay
`na
`
`ARRANERaEEEETT
`
`aa
`
`C3
`
`ONE
`JANAARENA
`NNN
`cnn
`we
`~
`
`daShay.
`CAROAAA
`
`feen,
`
`
`
`
`
`
`
`PiaCdADELPCELEELEEEECOOLELECCEDCEDLSCLOTELLPLLEPPLELELLCPTDEECOODEPELSIEAEPAIUEDVMATLEEECODEDCOEDEEREPETE
`
` VENFS
`
`
` PTEIEEmyggZ%gs%peerscereanncamerrnrrterercceenerereracegretigeeranstsssrinssiripnsnisstisensnrcnrearerdenisrrerenterbeneety,wernerse
`
`
`eT
`
`
`
`WhARNTAETAIIDDEIDETEDIPDPSDSSEDIIDIPOPPOOETOP!
`
`
` ncnéLirterrnerrnrenenEDETLEDbbeaoenihTERTTEECRLETOCISCCIObITT4%oS“e
`
`
`
`fa
`
`mG
`
`53
`
`can]
`
`te}
`
`<b
`
`St
`
`a
`
`eee
`
`a
`“SRAereSRRSREVE
`
`SeEVERTENREE
`
`ifuA
`
`‘444i4jHi4
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 001
`
`
`
`
`
`
`
`
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 001
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ChewGaeCe|20GCR7Su|AgeteeerEpeSe“2mee2b
`oOmeOt
`
`
`
` :Bno3!:33gsPoaes;:yyi.teL&PoiveeneeBeerLE“AaoonwagefooFeeatpefenact
`
`
`ceeitceetiratniccttetearit.wefronBoeeOF%SyeoE
`.aeeeosieA.ae)if
`
`wrFLieweebdEyboon:ESewe
`
`
`
`
`co
`
`oo‘°wr,4LA
`raemafrohAAL-my,~,co\“A
`
`
`-%tNco
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 002
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 002
`
`
`
`
`
` fis
`iad ae
`
`HRNNET
`
`NATE A
`at ae? AW
`
`shldeet
`
` fishedDientetenAhada
`
`Sone
`
`,S
`* “eaccconecnstell
`nt AARPEIERey
`f°sennonll
`
`*heavenanereetery
`
`free,
`
`intrpice
`C
`
`DAREEAAAthalOeEde
`
`|
`Sy
`X3 Jo
`ye
`¥
`tA
`&} Jane
`
`oy
`
`i
`
`
`gRnEeRRENNEPONASHINNINNENANIAEANANIENNNNNN
`AAT
`
`AANAANATOANATANNAANGATNNeTNTNNSa
`
`— EXT
`ane 2a
`
`if
`t
`SRIINRIENNNARNIANINNNNNANNIESSHONNNANNNNNN
`
`
`
`
`etLLAELOMDALAdAilEPApACCLLALAELOELAUTESTATEALLAELUATESALAALLSURUUSOESATCUOOLTOOLEOSUESLTVEITUSELLLOUILELILLPUTLUEEEELE
`
`8TNMAELOEAEtPEARLLOLRETLETAEEUDDUTEECOTALUTEEKIOTTERTOTOALSTEAERIDALLESTACOAAEELEE
`
`ELELATOEIEIOTODO,
`
`wt
`
`4,%
`
`t,Ceeeputsing,
`
`ARASHRNNNNNANNANNENT
`ye
`
`many
`es
`
`Stara
`
`UBMIT
`"agenneni”
`
`EE
`
`CORES
`SENRNANNNNRANANTESNAANINNNINN SoSOEENNETSONSIORNINSANDSNANNINGIRSEANRDN
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 003
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 003
`
`
`
`
`
`
`
`ae
`
`
`
`EQUEST FOR
`SENG
`
`IDENTIFYING PYAGEBP8
`
`{9.9.3SEAVERPAGE)
`
`
`L_. O SERVER_
`
`502
`
`(
`
`
`
`Bee
`
`
`
`¥
`
`t
`oe eee
`
`DETERMINE DESIRED
`ENVIRONMENT
`
`
`ciel ane or
`| RETREIVE IDENTIFYING PAGE Le
`
`TEMMPLATEFORENVIRONMENT |
`
`
`=
`nn DoMbtion| a
`“ey
`ACTER
`AR
`TOES 77808
`) RETAEIVE INFO IMATION|
`
`&FROM |IMAAGES DATAE aan
`
`waes
`
`FROM USERS DATABASE
`
`(OPTIONAL FILTERING)
`
`
`
`ainrmtinnneannndnasenannaasinnioay
`
`rm ee| RETREIVE INFORMATION | --808
`
`
`
`|
`|
`i
`
`
`TRANSCRIBE INFORMATION |
`FROM| DATABASES INTO
`b
`mr
`
`PAGE TEMPLATE [
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 004
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 004
`
`
`
`of
`
`mie
`IDENT!
`RECEIV!
`“
`SAL
`ad
`PEA
`E
`x~}
`JEN
`Ct
`
`t&
`
`A
`
`wt
`2
`
`"O
`
`gee fees,
`Ne
`}
`ELECT OBJE
`Pt
`x
`A
`DENTIEY IN
`|
`
`INFORMAT
`
`ASRS
`
`
`
`
`———ier
`
`ay
`
`is
`
`specienoiaaS=
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 005
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 005
`
`
`
`
`
`
`
`
`X
`.
`e
`
`oo
`ey
`k
`yo
`x
`:
`3
`x
`:
`2
`:
`3
`Skee
`wr
`
`Yo
`
`
`
`8s
`
`
`
`s:
`
`3
`,
`,
`‘
`x
`X
`
`wn
`a 8
`x
`r4
`SHE 3
`ae
`PEOPLE INIMAGE
`Rew
`PPM PEVEPSARS
`3
`RO
`
`RMONNNINSNANNING
`ASSENTAEEI e COATESx
`
`
`i
`geal! ow &
`g
`i
`.
`Bee
`
`Ras 5smaneannasasaNSURINSANRSSNSSHOUDEEEESASAEARENNSASNAST
`xx
`: ——— ————_—__—_—_—_—_—
`ON
`8
`‘
`s
`&
`3
`x
`&
`[os &
`:
`¢
`x “ Sey
`‘
`‘
`§
`;
`8
`Ys
`a
`og
`&
`& of
`s
`‘
`y
`‘
`42
`: ,
`:
`¢
`x
`poem
`yf
`§
`g
`g
`x
`8
`$
`s
`$
`lakes
`f
`o
`s
`A og
`U- a
`:
`.
`;
`¢
`i Jann Doe |
`
`,
`|
`x
`i.
`ere
`x
`8
`¢
`8
`x
`8
`\
`:
`‘
`‘
` s
`;
`\
`:
`Seg
`8
`8
`Pare Theis
`y
`x
`Stason
`8
`8
`Jae Le
`8
`S
`8
`i
`:
`3
`.
`x
`s
`RarmnemeaentaancneeencennnNAANNTN
`g
`8
`QoAAAANAMAANANTHNTN
`8
`;
`:
`s
`3
`g
`8
`a
`x
`8
`_
`‘
`s
`x
`3
`3
`oN
`8
`8
`SS
`‘s
`=
`=
`&
`,
`8
`‘
`Fes ON
`x
`8
`s
`i
`s
`§
`x
`x
`S
`s
`:
`cae ay \
`s
`&
`‘
`‘
`:
`s
`‘
`3
`yA
`Ss
`§
`:
`e
`:
`8
`8
`:
`4
`e
`>
`s
`x
`3
`$
`g
`S
`&
`x
`x
`3
`:
`5
`8
`:
`eag
`s
`‘
`¥
`s
`4
`8
`:
`3
`s
`:
`3
`,
`‘
`:
`3
`3
`Sout
`g
`8
`g
`aa:
`3
`4
`g
`s
`Nenwas
`4
`3
`s
`x
`A
`s
`3
`SaemmmanseenannrcanennnnenncANCONA
`8
`8
`x3
`3
`
`8 8&seossaasnantagonistaARiBANOTANHERSISSOERADEARESSENASUNONOAENEENITNNT
`
`x
`‘
`s
`3
`<
`R
`8
`
`3
`8
`s
`s
`8
`8
`8
` (
`s
`s
`8
`‘
`8
`5
`8
`=
`8
`s
`8
`s
`j
`
`pA
`pes eS,
`2;
`er
`gs
`$
`PEOPLE INIMAGE
`g
`fe
`Lio EN
`FS
`3 AS 3
`
`4
`So
`A
`
`
`
`“oar
`
`ts
`Pies
`ae
`Jim Doe
`See egy Mo zh
`UE LOG
`Oy gi
`ey
`oA et
`~
`Ws od
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 006
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 006
`
`€
`
`
`
`
`i
`ure & Feca
`
`a
`REE
`BOO
`OBTAIN A REQUEST FOR [7
`IMAGE DISPLAY PAGE |,
`atneiinmceimenl
`
`~~
`| DETERMINE DESIRED
`ENVIRONMENT
`
`a phe
`L
`
`
`
`BOs
`-
`;
`Ly
`RETAEIVE DISPLAY PAGE
`TEMPLATE FOR ENVIBONMENT |
`
`
`NAMING INFOSMATON
`i
`
`SegnogpannannantieviesiuaNNORNETEOEAREERERFMANAS
`eeecateaNRRSRSS ; =TreeeeeITEOEE OEE
`
`vOBTAINIMAGEAND
`
`| B08
`
`
`eli eeeeeee
`
`
`:| TRANSCRIBE IMAGE AND NAMING
`=:
`INFORMATION INT
`:
`DISPLAY PAGE TEMPLATE
`
`
`
`
`TRANSCRIBE
`|
`COORDINATE
`
`[INFORMAT)ONINTO:
`'
`SPLot PAGEo
`NO
`;
`7
`‘an
`t
`
`
`
`
`¥
`|
`RRAAROBONNENNHRANERALERemma
`,"PROVIDETHE IMAGE en
`DISPLAYPAGE |
`“LoTOA RECEIVINGUSER
`INTIATING
`REQUEST
`E
`
`ss ttopnionneranpomannaennnsrnraanwrrsseynnancananneananatpaneer
`
`po
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 007
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 007
`
`
`
`
`
`
`
`ELLLTLELSEPTLESESELDLLTYELLELLOOTLLLELIEESPPTELLOELLPLLLLISSLLEEOTALELD
`
`end\git2poererreereaabocnnnnn.
`coEnrontnenniaansapanonraratenetgjywOo<i
`
`
`
`
`
`
`
`iYipA4;4i3go!edoe§aya4ooaha.?CaeP|—CoiyJi};i’j4aGoeeeweifj:AmeabonA
`
` ZZZgXsma2@an|fopovalBY7Zg%“tn49e.44|8jL|meyygOaI,ZOBm,||gyg4pety3i| UP™oe1-4.tA‘EIBGMil2&64ranrendbeeee4iijiaseet;oyfjonLaSod|Be4/8as;,GFanoD:1;Daerrecrccercereeorenucrenereceararenerereareavennrecseiaih4eeEegGeyyyy]<0)4x!%8igos%4abeen4SySsLOLELLLLLLATTELITTLELLLETUTTLEchGUre.41iDSoeos\;aoS
`
`
`
`
`€-\,yIfPLLLLLERLLEELSELLLELLLLSLLLLLLLEELEELSLILLELLLPEELSftoo
`g=jNGitBiipyay|FLLRFEwe
`
`
`
`
`be00TLRUPEPELSOOTLULLTULLUDAEEDLDEULYELLEUISTLITELECEESEPLOLPLSSEDEDELSTOSTELEED
`RAREZmA3gInj%|oN,4fig4aeymF.;o.4§4.)OEsZZvgHieos5*.taay%og|2yg|im|aceg[4|®ro|&onZgoooxg ¥tHs_PsBaa“weByUgoii=4cgogx4474img
`offofi=SSx|
`ioF°rat&ee-,m4becbeecd“eat
`Omom|Z|
`4%g~
`4g9gipomeerrm“ng
`
`PLATESLSTSTENEIDELEDALEDLLSPULLPLLVELLEbygjceefproeemereneveerfine;fTLLEILTAEEOAOARIserene
`
`
`
`
`BLLLIILEZLYMLLLEDIRPLILOTALLREDALLLDLLITSOSISETTOLLPOTLE
`
`
`
`Oo
`
`web
`
`eR
`
`a5
`
`ee
`
`S x
`
`AANATONSA
`
`Afier (Alc
`
`i>
`kod
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 008
`
`sronniensanunnnnananmnnnnnnanniiniii,OOLIIEALEALLDLELLPEPETETTEEEEELLITSLETLSEPILTELELDDREEECLM2CIDDIISLLPLAISTPLLSITET8OILSIGSPYITLIVIPTIGISTESTILTECTSSETIELLISLSYSLIPASE
`
`e=g|%||j|
`
`Figure S
`wet
`
`2
`
`t 8
`
`00
`
`
`
`
`
`
`
`
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 008
`
`
`
`
`
`
`
`Figure 10
`
`
`
`CLIENT SENDS REQUESTFOR=| -t000
`
`L J
`r
`
`IMAGES (OR LIST OF IMAGES)
`CONTAINING ONE
`OR MORE OBJECTS -
`
`
`ENVIRONMENT
`
`
`DETERMINE DESIRED
`ENVIRONMENT
`
`(per ae
`- TEMPLATEFOR
`
`OBTAIN SEARCH RESULTS
`
`p-t004
`
`
`
`eeeeeET
`METREIVE RECORDS
`— f
`
`
`PROM
`IDENTIFICATIONS DATA
`
`
`|
`
`“TRANSCRIRE DATA ~
`
`|
`
`i
`
`4
`
`INTO SEARCH RESULTS
`
`TEMPLATE
`
`
`
`
`
`SEND LIST OF IMAGES AND/OR | —a¢
`~SEARCH RESULTS PAGE
`TO CLIENT
`
`
`
`
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 009
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 009
`
`
`
`UNITED STATES PATENT APPLICATION
`
`FOR
`
`TAGGING DIGITAL IMAGES
`
`INVENTOR:
`
`MARK FRIGON
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 010
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 010
`
`
`
`
`CROSS-REFERENCE TO RELATED APPLICATIONS
`
`This is a continuation of application Ser. Nr. 09/991,324 presenting under
`
`examination. The non-provisional application Ser. Nr. 09/991,324 takes priority from
`
`U.S. Provisional Application Number 60/248,994 filed on November15, 2000.
`
`
`BACKGROUNDOF THE INVENTION
`
`This non-provisional application takes priority from U.S. Provisional Application
`
`Number 60/248 ,994 filed on November 15, 2000.
`
`10
`
`
`FIELD OF THE INVENTION
`
`The present invention relates to computer software. More particularly, the
`
`invention relates to a method and apparatus for storing and sharing images such as
`
`photographs via a communications network and for permitting the identification of
`
`objects and the location of the objects within the images. The invention enables users to
`
`15
`
`supply and/or receive information about the existence of objects within images.
`
`Portions of this patent document contain material that is subject to copyright
`
`protection. The copyright owner has no objection to the facsimile reproduction by
`
`anyone of the patent documentor the patent disclosure as it appears in the Patent and
`
`Trademark Office file or records, but otherwise reserves all copyrights whatsoever.
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 011
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 011
`
`
`
`BACKGROUND
`
`Digital photography allows people to quickly and easily take, print, and/or share
`
`photos withoutfilm and the associated film developing costs. The popularity of digital
`
`photography has increased dramatically due to the increased availability of digital
`
`cameras and the widespread acceptance of the Internet. Many Internet users have created
`
`websites that organize their digital photographs into online photo albums that can be
`
`accessed by otherInternet users. These web pages permit a large number of people to
`
`view the photos and allow the author to both add and remove photos from an album.
`
`10
`
`Although creating web pages for online photo albumsoffer advantages over
`
`traditional photo albums, their creation requires programming expertise such as skill in
`
`Hypertext Markup Language (HTML). Manypeople do not havethe technical
`
`knowledge and/or time to build their own online photo albums. Sharing of photos from
`
`online photo albumsis also problematic and requires that each photo be manually sent, e-
`
`15
`
`mailed, or downloaded. If the recipient wishes to use the photos in their own web pages,
`
`each photo must be then uploaded to an additional site. These steps are time consuming,
`
`duplicative, and also require programming knowledge.
`
`To address these problems, several websites that specialize in online photo
`
`20
`
`albums have been created. These websites allow users without programming skills to
`
`quickly and easily create and maintain online photo albums. Users are only required to
`
`upload photos that they wish to add to an online photo album. Thesites, through
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 012
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 012
`
`
`
`programming on the webserver, or "server-side scripting’, then create all necessary
`
`HTMLprogramming to add the uploaded photos to the users’ photo albums. Many of
`
`these sites also offer image enhancementcapabilities (e.g. sharpen, brighten, reduce "red-
`
`eye”, etc.), as well as photo printing and developing functions. Photo-specific sites have
`
`also eased the process of sharing photos by providing a meansfor users to quickly and
`
`easily e-mail links to photos or photos themselves to one anotherand, in somecases,
`
`contribute photos directly to other's photo albums.
`
`While websites that specialize in the creation of online photo albums offer many
`
`advantages, they suffer from several limitations. For example, existing websites do not
`
`10
`
`offer users the ability to identify objects within photos. If a user is viewing a photo and
`
`does not know certain individuals in the picture, the user is unable to use current systems
`
`to determinetheir identities. Typically, photo album sites offer the ability to describe
`
`uploaded photos throughthe use of captions or other descriptive fields. These fields may
`
`be used to describe or name individuals within the photo. For example, a user may
`
`15
`
`upload a photo and give it a caption of "Stephanie rollerblading with Tauny”. This
`
`caption aloneis not sufficient information to ensure the viewercan correctly determine
`
`the identities of the individuals within the photo, because to do so would require prior
`
`knowledge of which person is “Stephanie” and which is “Tauny’”. A viewer who does
`
`not know these individuals may have difficulty or be unable to correctly identify the
`
`20
`
`individuals within the photo. Furthermore, less descriptive captions such as "The Family
`
`at the beach” would offerlittle help unless the viewer knowseach individual in the photo.
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 013
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 013
`
`
`
`Thus, in order to provide sufficient information required to ensure a viewer can identify
`
`individuals within a photo, copious descriptions must be input and maintained byusers.
`
`Typing such captions or descriptions is very time consuming andtedious, especially
`
`when a large numberof photos are uploaded. Furthermore, when the same person
`
`appears in multiple photos, the submitter is still required to retype the identification
`
`information into a caption for each photo.
`
`Processes have been developed to identify individuals within photos. For
`
`example, one process requires people to wear badges containing identification
`
`information such as unique color codings whentheir photos are taken. When a computer
`
`10
`
`processes a photo containing a badge, the computer can access a database to determine
`
`which badge was assigned to which person and then identifies the person as existing in
`
`this photo. This process has many drawbacksincluding the dependenceon physical
`
`badges. If the badges cannot be seen in the photos, the people wearing the badges cannot
`
`be identified. Moreover, this process is not functional for the vast majority of
`
`15
`
`photographs: any photos that were taken outside of this tightly controlled environment.
`
`Another problem with photo-specific websites is that they have no search
`
`capabilities that permit users to search for photos of specific individuals. Users wishing
`
`to view all photographsthat contain a particular person must manually access all possible
`
`photo albumsto find the desired photos. For example, if a user wishes to find all pictures
`
`20
`
`of a particular friend, the user may have to browse through several of his/her own albums
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 014
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 014
`
`
`
`where that friend may appear (e.g. "High School pictures”, "Reunion Weekend”, etc.), as
`
`well as access and browse through each of the friend’s albums, or any third-party’s
`
`albums where the friend may appear. With a reasonable numberof photos, this becomes
`
`time-prohibiting if not impossible. Search features could be designed to search
`
`aforementioned descriptive fields (e.g. a caption), however, as there is no naming
`
`standard, people are described differently by different people (e.g. first name, last name,
`
`full name, nickname, etc.) and many names would be used to describe multiple people
`
`(e.g. Mom, Dad, John,etc.), As a result, it would be difficult to have a one to one
`
`relationship between descriptions of people and people themselves. Consequently,
`
`10
`
`searches would be unreliable, likely returning many photos oflittle relevance andfailing
`
`to return photos that would be of relevance. For example, a search for the term “John
`
`Smith” would fail to return photos in which this person wasreferred to as “John”or,
`
`perhaps, “Dad”. However, this samesearch likely would return photos of other
`
`individuals who share the same name “John Smith’.
`
`15
`
`While reliable database searchability for digital images over the Internetis
`
`available, it has not been implemented with photo albums. Particularly, several
`
`companies whichsell and license images have put catalogs of digital photographs on the
`
`Internet. Such websites generally describe images in terms of themes(e.g. sunset, dark or
`
`sad) or broad descriptions of objects therein (e.g. beach, farmland, or water), without
`
`20
`
`identifying the objects explicitly. For example, a picture of a man and womanonthe
`
`beach at sunset, which could be licensed to a beach resort for print or web use, could be
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 015
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 015
`
`
`
`identified as “beach”, "sunset”, and/or "man and woman’. While such descriptive terms
`
`can provide a reliable mechanism for searching for images matching the terms, they do
`
`not uniquely identify any objects in the images, nor do they have the capability of
`
`identifying where these objects appear within the images. For example, a search forthe
`
`term “clown”is likely to return many pictures of clowns, however, it is very unlikely that
`
`every clown pictured would be the same person,just as a search for the term "beach”is
`
`unlikely to return pictures of the same beach. Even a search for the term “President” will
`
`return several photos of different people. Furthermore, the objects that appear within a
`
`photo are not identified as being in a particular area within the photo, they merely are
`
`10
`
`described as being in the photo. For example, websites that return a photograph from a
`
`"President” search do not havethe capability of identifying that the Presidentis in,
`
`perhaps, the lower right corner of the picture. Such technologies have the capability to
`
`describe the nature or theme of a photo but not to uniquely identify who or whatis in the
`
`photo or where in the photo it appears.
`
`15
`
`Anotherrelated limitation of prior art websites is that the captions described
`
`above contain static information only, such as names, and do not provide a way to obtain
`
`additional information about a person or object such as contact information or to locate
`
`additional photos of the person or object.
`
`Another problem with photo-specific websites is that users wishing to share their
`
`20
`
`photos must proactively e-mail or otherwise send their photos to others (i.e., a photo is
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 016
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 016
`
`
`
`not automatically shared when a user is described in a photo). For example, evenif a
`
`user describes a photos as containing “Jane@doe.com”and “John @doe.com”, existing
`
`website will not share the photos with these individuals. Through a separate process, the
`
`user must still email this photo or otherwise share the photo with, in this case, Jane and
`
`John. Although photo-specific websites ease this process, it is still time consuming for
`
`users with large photo albums and/ora large numberof friends to send their photosto all
`
`such persons.
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 017
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 017
`
`
`
`SUMMARYOF THE INVENTION
`
`An embodimentof the invention comprises a system and methodforidentifying
`
`and displaying objects (in most cases, a person) within a digital image over a system of
`
`interconnected computers(e.g. client/server, peer-to-peer, etc...). For example, a user can
`
`identify where a person exists in an image by selecting the person’s name(e.g. from a
`
`list) and clicking on the person’s location in the image. By identifying people within
`
`images, the system is able to search for images where a specified person(s) appears as
`
`well as generate output that recognizes where in the image a person appears. This output
`
`10
`
`is also capable of creating hyperlinks and calls to client-side scripts from the otherwise
`
`“static” image file. In an embodimentof the invention the system can also utilize
`
`artificial intelligence algorithms to locate other images where the identified object has not
`
`yet been defined by a user of the system.
`
`While, one embodiment of this invention utilizes the Internet as the medium for
`
`15
`
`communication and web pages fordisplaying data, the scope of this invention is not
`
`limited specifically to the Internet and/or to web pages. The invention contemplates the
`
`use of any display output in a computer environment(e.g. any types of document). The
`
`terms person, people, and/or object are used interchangeably and meantto refer to any
`
`object(s) (e.g. a pet, a possession, a place, or a thing) a user wishesto identify within a
`
`20
`
`digital image. This most frequently will, in fact, be a person. When shown within the
`
`boundsof an image, a person is considered akin to any type of object. Thus, the reader
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 018
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 018
`
`
`
`should note that the invention can be utilized to identify any type of object within an
`
`image.
`
`Depending on the computer environment, different media will be used for output
`
`and display to the client. On the Internet, the most accessible and most commonoutputis
`
`Hypertext Markup Language (“HTML”). However, the functionality of HTML can be
`
`duplicated or configured in other non-HTML media. For the sake of both brevity and
`
`clarity, I will outline an embodimentof the present invention that utilizes HTML. This
`
`embodiment of the present invention(i.e. utilizing the Internet and HTML)is logical due
`
`to the broad reach of the Internet and wide acceptance of HTML. Howeverthe scope of
`
`10
`
`the invention is not intended to be limited to the use HTMLorthe Internet.
`
`10
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 019
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 019
`
`
`
`DESCRIPTION OF THE DRAWINGS
`
`An embodimentof the invention is described in detail below with reference to the
`
`attached drawing figures, wherein:
`
`5
`
`Figure 1 is a schematic diagram of computer and communications equipmentthat
`
`may be used to implement an embodimentof the present invention;
`
`Figure 2 is a schematic diagram of databases that may be used to implement
`
`certain aspects of the invention;
`
`10
`
`Figure 3 is a flow chartillustrating a high-level overview of the steps of an
`
`embodimentof the invention.
`
`Figure 4 is a schematic diagram of an identifying page that may be used when
`
`15
`
`identifying objects within an image in accordance with an embodimentof the invention;
`
`Figure 5 further describes the general details relating to the process for requesting,
`
`generating, and transmitting an identifying page to the requestor (e.g., the client
`
`20
`
`computer).
`
`11
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 020
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 020
`
`
`
`Figure 6 is a flow diagram illustrating a general overview of the process for using
`
`the identifying page to obtain identifying information (e.g., information associated with
`
`object(s) in an image) from the user.
`
`Figure 7 is a schematic diagram of a display page that may be used to display
`
`images and information about objects that have been identified within the image in
`
`accordance with an embodiment of the invention.
`
`10
`
`Figure 8 is a flow chartillustrating a high-level view of the process for providing
`
`display page to a requesting user in accordance with an embodimentof the invention,
`
`Figure 9 illustrates various waysto display identification information, including
`
`the various states a user interface configured to operate in accordance with an
`
`15
`
`embodimentof the invention may have.
`
`Figure 10 is a flow chart describing the process for generating a set of search
`
`results in response to a request.
`
`12
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 021
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 021
`
`
`
`The figures do not limit the invention to the specific embodiments disclosed and
`
`described herein. The drawings are not necessarily to scale, emphasis instead being
`
`placed uponclearly illustrating the principles of the invention.
`
`13
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 022
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 022
`
`
`
`DETAILED DESCRIPTION
`
`The invention described below is a method and apparatus for obtaining
`
`informationrelating to the existence of at least one object in an image. In the following
`
`description, numerous specific details are set forth to provide a more thorough
`
`description of the present invention. It will be apparent, however, to one skilled in the
`
`art, that the present invention maybe practiced without these specific details.
`
`In other
`
`instances, well knownfeatures have not been describedin detail so as not to obscure the
`
`present invention.
`
`10
`
`General System Architecture
`
`Embodiments of the invention can be implemented in hardware, software,
`
`firmware, or a combination thereof.
`
`In one or more embodiments, however, the
`
`invention is implemented with one or more computer programs that operate one or more
`
`pieces of computer equipment broadly referred to by the numeral 101 in Figure 1. Such
`
`15
`
`computer equipment 101 typically includes a server computer 200 (also referred to as a
`
`host computer) and any number of user computers 100(e.g., client computers) that may
`
`access the host computer 200 via a communications network 150. The computer
`
`programs and equipmentillustrated and described herein are merely examples of
`
`programs and equipment that may be used to implement an embodimentof the invention
`
`20
`
`and may be replaced with other software and computer equipment without departing from
`
`14
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 023
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 023
`
`
`
`the scope of the invention. For example, any computational device configured to display
`
`image data may be configured to implement one or more aspects of the invention.
`
`Host computer 200 may be any computing device such as a network computer
`
`running Windows 2000, Novel Netware, Unix, or any other network operating system.
`
`The host computer 200 may be connected to a firewall computer at the boundaries of
`
`network 150 to prevent tampering with information stored on or accessible by the host
`
`computer 200 and to another computing device that may operated by an administrator of
`
`the host computer 200. If the invention is implemented with the Internet, the host
`
`10
`
`computer 200 may include conventional web hosting operating software, an Internet
`
`connection such as a modem, DSL converter or ISDN converter, and be assigned an IP
`
`address and corresponding domain nameso that the website hosted thereon can be
`
`accessed via the communications network 150.
`
`Client computer(s) 100 may be used by anyone wishing to upload and/or view
`
`15
`
`photographs via host computer 200. The client computers 100 may be any device or
`
`sroup of devices that permits access to host computer 200 via communications network
`
`150, including, but not limited to, personal or laptop computers such as those
`
`manufactured and sold by Dell™, Compaq™, Gateway™,or any other computer
`
`manufacturer, handheld personal assistants such as those operating with Palm’s
`
`20
`
`PalmOS™operating software or those operating with Microsoft’s PocketPC™operating
`
`software, mobile phones, or even Internet appliances that are specifically designed for
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 024
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 024
`
`
`
`accessing the Internet such as WebTV™., Each client computer 100 mayalso include or
`
`can access a conventional Internet connection such as a modem, Digital Subscriber Line
`
`(DSL) converter, or Integrated Service Digital Network (ISDN) converter and a web
`
`browserthat permits it to access and view data overthe Internet.
`
`The communications network 150 may be the Internet but can also be any other
`
`communications network such as a local area network, a wide area network, a wireless
`
`network, an intranet or a virtual private network. Hereinafter, the term Internet or
`
`communication network refers to any type of interconnection fabric that provides
`
`10
`
`computers coupled to the network with a mechanism for transmitting and/or receiving
`
`data.
`
`The computer program or programs embodying one or more aspects of the
`
`invention are stored in or on computer-readable medium residing on or accessible by host
`
`computer 200 and provide a mechanism for instructing host computer 200 to operate the
`
`15
`
`invention as described herein. The computer programs typically comprise ordered
`
`listings of executable instructions for implementing logical functions in host computer
`
`200 and user computers 100 coupled with host computer 200. The computer programs
`
`can be embodied in any computer-readable mediumforuse by or in connection with an
`
`instruction execution system, apparatus, or device, such as a computer-based system,
`
`20
`
`processor-containing system, or other system that can fetch the instructions from the
`
`instruction execution system, apparatus, or device, and execute the instructions.
`
`In the
`
`16
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 025
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 025
`
`
`
`context of this application, a “computer-readable medium” can be any meansthat can
`
`contain, store, communicate, propagate or transport the program for use by or in
`
`connection with the instruction execution system, apparatus, or device. The computer-
`
`readable medium can be, for example, but not limited to, an electronic, magnetic, optical,
`
`electro-magnetic, infrared, or semi-conductor system, apparatus, device, or propagation
`
`medium. Examples of the computer-readable medium would include the following: an
`
`electrical connection having one or more wires, a portable computer diskette, a random
`
`access memory (RAM), a read-only memory (ROM), an erasable, programmable, read-
`
`only memory (EPROM or Flash memory), an optical fiber, and a portable compact disk
`
`10
`
`read-only memory (CDROM). The computer-readable medium could even be paper or
`
`another suitable medium upon which the program is printed, as the program can be
`
`electronically captured, via for instance, optical scanning of the paper or other medium,
`
`then compiled, interpreted, or otherwise processed in a suitable manner,if necessary, and
`
`then stored in a computer memory.
`
`15
`
`In an embodimentof the invention, host computer 200 comprises server engine
`
`220 which is programmedto operate or host a website and serve as a repository for
`
`images and identification information for objects within the images as described in more
`
`detail below. The images may be photographs, graphics, artwork, or any otherdigital
`
`image that contains or depicts one or more objects. The objects within the images may
`
`20
`
`include people, animals, plants, buildings, places, or anything else shown in images. In
`
`one embodimentof the invention, the images and objects are referred to (but not limited
`
`17
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 026
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 026
`
`
`
`to) as photographs and people, respectively. The reader should note that the term object
`
`as used herein includes any identifiable entity within the image data.
`
`Server engine 220 can access web page templates 210, databases 230, 240, and
`
`250, and internal or external memory. Web page templates 210 can be any technology
`
`used to transcribe data from a data source into a web page or otherwise provide
`
`“dynamic” or customized documents, including but not limited to, Active Server Pages,
`
`Java Server Pages, Cold Fusion, PHP, CGIscripts, or even client-side scripts. Server
`
`engine 220 receives information and requests from user computer(s) 100 via
`
`10
`
`communications network 150, processes web page templates 210 by transcribing data
`
`from data sources, including databases 230, 240, and 250, into templates 210, and sends
`
`the resulting web page or documentto requesting client computer 100 as described in
`
`more detail below.
`
`In order to implementthe invention host system 200 will access a server system
`
`15
`
`or database system configured to support the functionality described. Referring back to
`
`Figure 2, one or more databases could be utilized to store the data utilized by the system.
`
`For instance, the system may utilize a users database 230, Identifications database 240,
`
`and Images database 250 as depicted in Figure 2.
`
`Users database 230 receives, stores, and provides information about people or any
`
`20
`
`other objects identified within photos, including various people who access host
`
`computer 200. Users database 230 can be populated to include, for each useror client
`
`18
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 027
`
`Meta Platforms, Inc.
`Exhibit 1002
`Page 027
`
`
`
`system, a user identifier that is unique to that user or client system, and other information
`
`such as the user’s name, e-mail address, home page address, and/ora list of contacts. The
`
`list of contacts is not necessary and may bestored in a separate database.
`
`Images database 250 receives and stores information about photos(e.g., image
`
`data) uploaded or otherwise made accessible to the network by users. Images database
`
`250 can be populated as described below to include, for each photo, a photo identifier
`
`that is unique to that photo, and the location of the imagefile on the network. The
`
`Images database 250 may include descriptive information about the photo such as a
`
`caption or the date the photo was taken. While it is preferred to have photos uploaded to
`
`10
`
`and stored on the host computer 200,it is possible to have the photos stored elsewhere,
`
`beingstill accessible to users on the network, such as another website or a personal
`
`computer connected to the network. In an embodimentof the invention, both of these
`
`processes will be referred to hereinafter as uploading. It is also possible to store the
`
`images directly into the database itself as binary objects, rather than simply storing
`
`1
Accessing this document will incur an additional charge of $.
After purchase, you can access this document again without charge.
Accept $ Charge
Still Working On It
This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.
Give it another minute or two to complete, and then try the refresh button.
A few More Minutes ... Still Working
It can take up to 5 minutes for us to download a document if the court servers are running slowly.
Thank you for your continued patience.
This document could not be displayed.
We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.
Your account does not support viewing this document.
You need a Paid Account to view this document. Click here to change your account type.
Your account does not support viewing this document.
Set your membership
status to view this document.
With a Docket Alarm membership, you'll
get a whole lot more, including:
- Up-to-date information for this case.
- Email alerts whenever there is an update.
- Full text search for other cases.
- Get email alerts whenever a new case matches your search.
One Moment Please
The filing “” is large (MB) and is being downloaded.
Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.
Your document is on its way!
If you do not receive the document in five minutes, contact support at support@docketalarm.com.
Sealed Document
We are unable to display this document, it may be under a court ordered seal.
If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.
Access Government Site
