(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2002/0055955A1
`Lloyd-Jones et al.
`(43) Pub. Date:
`May 9, 2002
`
`US 20020055955A1
`
`(54) METHOD OF ANNOTATING AN IMAGE
`
`(76) Inventors: Daniel John Lloyd-Jones, Canberia
`(AU); Alison Joan Lennon, Balmain
`(AU)
`
`Correspondence Address:
`FITZPATRICK CELLAHARPER & SCINTO
`30 ROCKEFELLER PLAZA
`NEW YORK, NY 10112 (US)
`
`(21) Appl. No.:
`(22) Filed:
`
`09/826,935
`Apr. 6, 2001
`
`
`
`Foreign Application Priority Data
`(30)
`Apr. 28, 2000 (AU).............................................. PO7177
`Publication Classification
`
`(51) Int. Cl." ..................................................... G06F 17/21
`(52) U.S. Cl. .............................................................. 707/512
`(57)
`ABSTRACT
`A method and apparatus for annotating an image (407) is
`disclosed. The image (407) and a plurality of icons (403) are
`displayed Such that each icon is associated with metadata. At
`least one of the icons is Selected depending on at least one
`subject of the image (407) and the metadata associated with
`the Selected icon is Stored as an annotation of the Subject of
`the image.
`
`Image Labeller Prototype
`
`XXXX
`XXXX
`XXXX XXXX
`XXXX
`Xxxxx xxxxx xxxxx XXXXX XXXXX
`
`(6) (6) () (6)
`
`XXXX XXXX XXXX XXXX
`XXXXX
`
`input loons
`
`() (C)
`Extract Submages
`input i
`Rectangles Display Submages
`
`515
`
`XXXXXXX
`
`XXXXX
`XXXXXXX
`
`Meta Platforms, Inc.
`Exhibit 1013
`Page 001
`
`

`

`Patent Application Publication
`
`May 9, 2002. Sheet 1 of 7
`
`US 2002/0055955A1
`
`100
`
`a
`
`Fig.
`
`Generate a list of
`metadata labels.
`
`Generate a plurality
`of icons.
`
`ASSociate each
`metadata With at least
`One of the icons
`
`Render an image.
`
`
`
`Select at least one of
`the icons and drop the
`icon on the image.
`
`Determine the location
`of a desired subject
`within the image.
`
`Store the location and
`metadata asSOciated
`with the desired subject.
`
`
`
`Any
`more images?
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`101
`
`103
`
`105
`
`107
`
`109
`
`111
`
`113
`
`Meta Platforms, Inc.
`Exhibit 1013
`Page 002
`
`

`

`Patent Application Publication May 9, 2002 Sheet 2 of 7
`
`US 2002/0055955A1
`
`215
`
`214
`
`Computer
`NetWork
`
`
`
`N-
`
`216
`
`220
`
`2OO
`
`?
`
`Video
`Display
`
`
`
`201
`
`210
`
`211
`
`To T-20
`
`D
`Storage Device
`
`204
`
`202
`
`N-203
`
`Meta Platforms, Inc.
`Exhibit 1013
`Page 003
`
`

`

`Patent Application Publication May 9, 2002. Sheet 3 of 7
`
`US 2002/0055955A1
`
`3O1
`
`
`
`300
`
`<2 Xml version = 1.02)
`<Entity 1 system 12Nov 1.jpg NDATA.jpeg)
`
`<Entity 2 system 12Nov 2.jpg NDATA.jpeg
`
`<image id= “1” subject= "Liza Hayward”
`position = *(2,5) (7, 15)” height = “10” width = “5”
`></imaged
`<image id="2’ subject= “Jenry Bill'></imaged
`
`Meta Platforms, Inc.
`Exhibit 1013
`Page 004
`
`

`

`Patent Application Publication
`
`May 9, 2002. Sheet 4 of 7
`
`US 2002/0055955A1
`
`
`
`
`
`
`
`XXXXX XXXXXXXXXX
`
`XXXXXXXXXXXXxxxxXXXX
`
`(@)
`
`Meta Platforms, Inc.
`Exhibit 1013
`Page 005
`
`

`

`Patent Application Publication
`
`May 9, 2002 Sheet 5 of 7
`
`US 2002/0055955A1
`
`
`
`XXXXXXX
`
`XXXXXpue/WKeH
`
`xxxxxxxxX×XXXXXXxxxx
`
`
`
`
`
`
`
`xxxxx xxxxx XXXXX XXXXX
`
`Meta Platforms, Inc.
`Exhibit 1013
`Page 006
`
`

`

`Patent Application Publication May 9, 2002 Sheet 6 of 7
`
`US 2002/0055955A1
`
`600
`
`s
`
`FIG. 6a:b
`
`Generate a list of
`metadata labels.
`
`601
`
`Generate a plurality of
`icons.
`
`603
`N
`
`ASSOCiate each
`metadata with at least
`One of the icons
`
`605
`
`N
`
`Render an image.
`
`6O7
`
`No
`
`Automatic
`subject
`Detection?
`
`Default sized bounding
`617 N. box is placed.
`— —
`Bounding box is
`619 - resized by the user.
`
`
`
`
`
`621
`
`
`
`Yes
`
`New
`bounding box
`to be manually
`placed ?
`
`
`
`
`
`
`
`Detect subjects
`automatically and place -611
`bounding boxes.
`
`Bounding boxes are
`resized by the user.
`
`613
`ra
`
`615
`
`
`
`New
`bounding box
`to be manually
`placed ?
`
`
`
`
`
`(c)
`
`527062.fm
`
`Meta Platforms, Inc.
`Exhibit 1013
`Page 007
`
`

`

`Patent Application Publication May 9, 2002 Sheet 7 of 7
`
`US 2002/0055955A1
`
`(C) A
`
`3. V
`
`Select at least one of the
`icons and drag the icon -N-
`on the image.
`H
`Emphasize the bounding
`box under the dragged -N.
`iCOn.
`
`623
`
`625
`
`
`
`NO
`
`
`
`Selected
`icon dropped ?
`
`Detect the emphasized
`bounding box.
`
`1N
`
`call sla is
`Store the metadata
`associated with the
`bounding box.
`
`
`
`631
`
`1N
`
`633
`
`Yes
`
`
`
`
`
`Any more
`metadata to be
`added ?
`
`NO
`
`Yes
`
`635
`
`Any more
`images 2
`
`
`
`FIG. 6a:b
`
`Meta Platforms, Inc.
`Exhibit 1013
`Page 008
`
`

`

`US 2002/0055955 A1
`
`May 9, 2002
`
`METHOD OF ANNOTATING AN IMAGE
`
`TECHNICAL FIELD OF THE INVENTION
`0001. The present invention relates generally to the field
`of multi-media and in particular, to a method and apparatus
`for generating metadata based on multi-media content. The
`invention also relates to a computer program product includ
`ing a computer readable medium having recorded thereon a
`computer program for generating metadata based on multi
`media content.
`
`BACKGROUND ART
`0002 The popularity of digital photography is ever
`increasing as digital cameras become more accessible and
`the production and editing capabilities of digital images
`improve. Further, the Internet use of digital images is
`increasing rapidly. Large databases of digital images are
`being assembled for both personal and commercial use. AS
`with conventional photography, the need to annotate and
`catalogue the ever increasing number of digital images is of
`paramount importance in order to allow ease of access and
`use. One method of facilitating the annotation of digital
`images is to generate “metadata” with the image. Metadata
`is information about the content of digital images or even
`Video. For example, an image depicting a beach Scene could
`include a short textual description Such as “a picture of a
`beach', the name of a perSon in the image or a date and time
`that the image was captured. Many Internet image Search
`Sites Search on metadata content descriptions to locate
`digital images for display.
`0003. Some digital cameras automatically generate a date
`and time, which is generally included in the file name of a
`digital image when the image is stored and/or displayed (e.g.
`12Nov 1.jpg). However, the automatically generated date
`and time Says nothing about the content and/or event
`depicted by the digital image and therefore provides only
`limited assistance in annotating, cataloguing and Searching
`for the digital image.
`0004 Conventionally, a text entry method of generating
`metadata for digital images has been used where a perSon
`Sorts through a database of digital images, using a computer,
`and Stores with each digital image a short textual label
`indicating a Subject and/or an event depicted by each digital
`image. However, the above conventional method is very
`labour intensive and thus time consuming.
`0005 The problem is further exacerbated when a person
`needs to Supply a Selection of digital images of a particular
`event and/or Subject by e-mail (electronic mail). This
`involves viewing all of the images in a database, Selecting
`the appropriate images and then grouping the images
`together by attaching them one-by-one to an e-mail for
`Sending.
`
`DISCLOSURE OF THE INVENTION
`0006. It is an object of the present invention to substan
`tially overcome, or at least ameliorate, one or more disad
`Vantages of existing arrangements.
`0007 According to one aspect of the present invention
`there is provided a method a method of annotating an image,
`Said method comprising the Steps of:
`
`0008 displaying the image and a plurality of icons,
`each icon being associated with metadata;
`0009 selecting at least one of said icons depending
`on at least one Subject of the image; and
`0010 storing the metadata associated with said
`Selected icon as an annotation of the Subject of the
`image.
`0011. According to another aspect of the present inven
`tion there is provided an apparatus for annotating an image,
`Said apparatus comprising:
`0012 displav means for displaving the image and a
`play
`playing
`9.
`plurality of icons, each icon being associated with
`metadata;
`0013 selection means for selecting at least one of
`Said icons depending on at least one Subject of the
`image; and
`0014 Storage means for Storing the metadata asso
`ciated with Said Selected icon as an annotation of the
`Subject of the image.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0015. A number of embodiments of the present invention
`will now be described with reference to the drawings, in
`which:
`0016 FIG. 1 is a flow diagram showing a method of
`annotating an image using metadata, in accordance With a
`preferred embodiment of the present invention;
`0017 FIG. 2 is a schematic block diagram of a general
`purpose computer upon which arrangements described can
`be practiced; and
`0018 FIG. 3 shows an example format of an XML file;
`0019 FIG. 4 shows a user interface that can be used to
`perform the method of FIG. 1;
`0020 FIG. 5 shows the user interface of FIG. 4 with
`bounding boxes overlaying the image according to the
`method of FIG. 1; and
`0021
`FIG. 6 is a flow diagram showing a method of
`annotating an image using metadata according to a further
`embodiment of the present invention.
`
`DETAILED DESCRIPTION INCLUDING BEST
`MODE
`0022. Where reference is made in any one or more of the
`accompanying drawings to Steps and/or features, which have
`the same reference numerals, those Steps and/or features
`have for the purposes of this description the same func
`tion(s) or operation(s), unless the contrary intention appears.
`0023 The principles of the method described herein have
`general applicability to generating metadata. However, for
`ease of explanation, the Steps of the method are described
`with reference to digital images. However, it is not intended
`that the present invention be limited to the described
`method. For example, the invention may have application to
`generating metadata regarding the Subject of a document, a
`Video clip or any computer generated page.
`
`Meta Platforms, Inc.
`Exhibit 1013
`Page 009
`
`

`

`US 2002/0055955 A1
`
`May 9, 2002
`
`0024 FIG. 1 is a flowchart 100 showing a method of
`annotating an image using metadata, in accordance with the
`preferred embodiment of the present invention. The method
`of FIG. 1 is preferably practiced using a conventional
`general-purpose computer System 200, Such as that shown in
`FIG. 2 wherein the processes of FIG. 1 may be imple
`mented as Software, Such as an application program execut
`ing within the computer System 200. In particular, the Steps
`of the method of annotating an image using metadata, are
`effected by instructions in the Software that are carried out
`by the computer. The Software may be divided into two
`Separate parts; one part for carrying out the annotation
`method; and another part to manage the user interface
`between the method and the user. The software may be
`Stored in a computer readable medium, including the Storage
`devices described below, for example. The Software is
`loaded into the computer from the computer readable
`medium, and then executed by the computer. A computer
`readable medium having Such Software or computer pro
`gram recorded on it is a computer program product. The use
`of the computer program product in the computer preferably
`effects an advantageous apparatus for annotating an image
`using metadata in accordance with the embodiments of the
`invention.
`0.025 The computer system 200 comprises a computer
`module 201, input devices such as a keyboard 202 and
`mouse 203, output devices including a printer 215 and a
`display device 214. A Modulator-Demodulator (Modem)
`transceiver device 216 is used by the computer module 201
`for communicating to and from a communications network
`220, for example connectable via a telephone line 221 or
`other functional medium. The modem 216 can be used to
`obtain access to the Internet, and other network Systems,
`such as a Local Area Network (LEN) or a Wide Area
`Network (WAN).
`0026. The computer module 201 typically includes at
`least one processor unit 205, a memory unit 206, for
`example formed from Semiconductor random acceSS
`memory (RAM) and read only memory (ROM), input/
`output (I/O) interfaces including a video interface 207, and
`an I/O interface 213 for the keyboard 202 and mouse 203
`and optionally a joystick (not illustrated), and an interface
`208 for the modem 216. A storage device 209 is provided
`and typically includes a hard disk drive 210 and a floppy
`disk drive 211. A magnetic tape drive (not illustrated) may
`also be used. A CD-ROM drive 212 is typically provided as
`a non-volatile source of data. The components 205 to 213 of
`the computer module 201, typically communicate via an
`interconnected bus 204 and in a manner which results in a
`conventional mode of operation of the computer system 200
`known to those in the relevant art. Examples of computers
`on which the embodiments can be practised include IBM
`PCS and compatibles, Sun SparcStations or alike computer
`systems evolved therefrom.
`0.027
`Typically, the application program of the preferred
`embodiment is resident on the hard disk drive 210 and read
`and controlled in its execution by the processor 205. Inter
`mediate Storage of the program and any data fetched from
`the network 220 may be accomplished using the Semicon
`ductor memory 206, possibly in concert with the hard disk
`drive 210. In Some instances, the application program may
`be supplied to the user encoded on a CD-ROM or floppy
`disk and read via the corresponding drive 212 or 211, or
`
`alternatively may be read by the user from the network 220
`via the modem device 216. Still further, the Software can
`also be loaded into the computer system 200 from other
`computer readable medium including magnetic tape, a ROM
`or integrated circuit, a magneto-optical disk, a radio or
`infra-red transmission channel between the computer mod
`ule 201 and another device, a computer readable card Such
`as a PCMCIA card, and the Internet and Intranets including
`e-mail transmissions and information recorded on websites
`and the like. An e-mail transmission is an electronic message
`Sent from one computer user to another over a digital
`network Such as the Internet or Intranet. The message is
`generally Sent using an e-mail application program and is
`generally in the form of a text file containing a textual
`message. However, an e-mail transmission can include other
`file types Such as jpg and .tif files, for example. The
`foregoing is merely exemplary of relevant computer read
`able mediums. Other computer readable mediums may be
`practiced without departing from the Scope and Spirit of the
`invention.
`0028. The method of annotating an image using metadata
`can alternatively be implemented in dedicated hardware
`Such as one or more integrated circuits performing the
`functions or Sub functions of FIG. 1. Such dedicated hard
`ware may include graphic processors, digital Signal proces
`Sors, or one or more microprocessors and associated memo
`CS.
`0029. The method of annotating an image using meta
`data, can now be described with reference to the flowchart
`100 of FIG. 1, where the method is performed using the
`computer system 200. The process begins at step 101, where
`a list of metadata labels is provided. The list of metadata
`labels is preferably provided automatically. For example, a
`list of people's names can be provided automatically by
`extracting the names from an existing database of names,
`Such as an e-mail address book. In this instance, names,
`asSociated e-mail addresses and other personal information
`can be extracted from the database. Alternatively, a list of
`metadata labels can be manually generated using any known
`Software database application, or metadata labels can be
`manually inputted by the user using the keyboard 202. At the
`next step 103, a plurality of icons are provided. Preferably,
`the number of icons is equal to the number of metadata
`labels in the list of metadata labels and the icons are
`preferably displayed on the video display 214. The process
`continues at the next step 105, where each label in the list of
`metadata labels is associated with at least one of the gen
`erated icons. For example, one of the icons could be asso
`ciated with the name “Jenny Smith' which was included in
`an imported e-mail address book. The association of each
`metadata label with an icon can be carried out arbitrarily.
`However, preferably each metadata label is associated with
`a generated icon.
`0030. At the next step 107, a digital image is rendered.
`The digital image is preferably rendered on a display, Such
`as the Video display 214. The digital image is preferably
`extracted from a database of imageS which is Stored on the
`hard disk drive 214 or accessed via the computer network
`220. The process continues at the next step 109, where one
`of the generated icons is Selected by the user according to the
`content of the image. For example, if the image depicts a
`person called “Liza Hayward”, then the icon associated with
`the name Liza Hayward can be selected. The icon is pref
`
`Meta Platforms, Inc.
`Exhibit 1013
`Page 010
`
`

`

`US 2002/0055955 A1
`
`May 9, 2002
`
`erably Selected by clicking on the appropriate icon using the
`mouse 203. The icon can preferably be dragged in a con
`ventional manner using the mouse 203 and positioned over
`a desired Subject of the image. Once the icon is positioned
`over the desired Subject of the image, the user can de-Select
`the mouse 203 in the conventional manner. This action is
`generally referred to as “dropping the icon on the image. At
`the next step 111, the location at which the Subject is
`rendered in the image is determined based on the initial
`placement of an icon on an image. Any known face or object
`detection method (e.g. colour region growing) can be used
`to detect where the Subject is located in the image based on
`the location of the initial placement of the icon. In accor
`dance with the preferred embodiment, a default sized bound
`ing box (e.g. 503 of FIG. 5) is presented to the user based
`on the location of the initial placement of the icon Such that
`the bounding box Substantially Surrounds the Subject (e.g.
`Liza Hayward). The bounding box is preferably re-sizeable
`by the user using the mouse 203 in a conventional manner.
`In this manner, the user can alter the size of the bounding
`box to totally surround the selected Subject or to cover only
`a portion of the Subject. Preferably, more than one icon can
`be selected. Alternatively, the bounding box can be auto
`matically resized to fit the Selected Subject. For example, if
`there is more than one perSon in the rendered image then
`another icon, which is associated with the name of the other
`perSon, can be Selected, dragged and positioned over the
`other perSon, as discussed above. This will result in another
`default sized bounding box (e.g. 505 of FIG. 5) being
`positioned over the Second Subject, which can preferably be
`re-sized by the user, as discussed above. In another example,
`a perSon in the image can be Standing with a mountain in the
`background. Therefore, after Selecting the icon associated
`with the person's name, an icon associated with the metadata
`"mountain” can be Selected and if desired a bounding box
`can be positioned adjacent to the mountain.
`0031. At the next step 113, the metadata (e.g. the name
`“Liza Hayward”) associated with the selected icons is stored
`as an association list, in a Storage device Such as the hard
`disk drive 210, and linked to the rendered image. The
`position (x,y) and size (width, height) of the bounding box
`(e.g. 503), associated with the subject, are also stored in the
`asSociation list, as at Step 113, Such that the metadata and
`bounding box information are linked together. Alternatively,
`two opposite co-ordinate points indicating the Size of the
`bounding box can be Stored in the association list. Therefore,
`the bounding box information (i.e. position, height and
`width) indicates the location and size of the subject within
`the rendered digital image. The association list preferably
`includes a tag indicating an association with the rendered
`image. For example, the list could include the file name (e.g.
`12Nov 1.jpg) of the corresponding image file for the ren
`dered image. Preferably the association list is an Extensible
`Mark-up Language (XML) file, Stored on a hard disk drive,
`such as the hard disk drive 210. XML is a computer
`language used for associating metadata with images. An
`example of the format of an XML file 301 is shown in FIG.
`3. As seen in FIG.3, the XML file includes the file name 300
`of an image file, the metadata 303, and bounding box
`information 305 associated with the image file. In a further
`implementation, the metadata and bounding box positional
`information associated with the Selected Subject can be
`Stored as part of the image file. For example, the .TIF image
`format allows metadata to be Stored as part of the image file.
`
`0032. The process continues at step 115, where a decision
`is made as to whether any more images require annotation.
`If more images require annotation, then the process returns
`to step 107, where another image is rendered. The process
`concludes when all of the images requiring annotation have
`been annotated.
`0033 FIG. 4 shows a preferred user interface 401 that
`can be used to perform the method shown in FIG. 1. The
`preferred user interface 401 includes an icon 415 labelled
`“Input Icon' which can be selected in order to enable the
`user to generate a database of metadata labels, as at Step 101
`of the method, by importing a pre-existing database from the
`computer network 220 or another application. The preferred
`user interface 401 also includes an icon 417 labelled “Import
`Images” which can be Selected in order to enable the user to
`access the digital images to be rendered. The user can
`preferably import the digital images from a pre-existing
`database which is stored on the hard disk drive 214 or
`accessed via the computer network 220 or another applica
`tion. The user interface 401 also includes a series of icons
`403, which are generated as at step 103 of the method. One
`of the icons 403 can be selected, as at step 109 of the
`method, in order to associate the Selected icon with a
`rendered image. The number of generated icons 403 is
`preferably equal to the number of metadata labels in the
`generated database, which has been previously imported or
`created by the user. The user interface 401 also includes a
`frame 407 for rendering the images that the user wishes to
`annotate. A forward and reverse button 409 and 411, respec
`tively, are provided below the frame 407 to enable the user
`to Successively render all of the images included on the
`database by selecting either forward or reverse with the
`mouse 203 in a conventional manner.
`0034.
`In accordance with a further embodiment, the user
`interface 401 includes three icons 419, 421 and 423 labelled
`“Extract Sub-images”, “Display Sub-images” and “Rect
`angles”, respectively. The user can select the icon 419
`labelled “Extract Subimages' resulting in an image being
`extracted from the rendered image, based on a bounding box
`which has been dropped on the rendered image using the
`method of FIG.1. Also, upon selecting the icon 419 labelled
`“Extract Sub-images”, the bounding box information (i.e.
`position, height and width) associated with the bounding
`box is Stored in the association list together with the meta
`data associated with the Subject Overlayed by the bounding
`box. After being extracted, the image in the bounding box
`can preferably be stored as an image file (e.g.jpg or .tif) in
`the Semiconductor memory 206. For example, an image,
`depicting Liza Hayward, based on the bounding box posi
`tioned over Liza Hayward in the rendered image, is
`extracted and Stored in the Semiconductor memory 206, and
`the position and size of the bounding box are Stored in the
`asSociation list together with the metadata associated with
`Liza Hayward, upon the user selecting the icon 419 labelled
`“Extract Sub-images”. As can be seen in FIG. 5, three
`bounding boxes 501, 503 and 505 have been displayed on
`the image being rendered in the frame 407 of the user
`interface 401 when the user selects the icon 421 labelled
`“Display Sub-images”. The user can then select the icon 421
`labelled “Display Sub-images” which results in the extracted
`image being displayed on the user interface 401. A user has
`then selected the icon 419 labelled “Extract Sub-images”
`followed by the icon 421 labelled “Display Subimages”
`which has resulted in three images 507, 509 and 511 being
`
`Meta Platforms, Inc.
`Exhibit 1013
`Page 011
`
`

`

`US 2002/0055955 A1
`
`May 9, 2002
`
`displayed on the user interface 401 based on the bounding
`boxes 501, 503 and 505. The process of extracting the
`Sub-images in accordance with the further embodiment at
`steps 109, 111 and 113 can be carried out sequentially or
`concurrently after dropping the selected icon 403.
`0035) Selecting the icon 423 labelled “Rectangles' pref
`erably results in the bounding boxes (e.g. 501, 503 and 505)
`being removed from the rendered image. However, the
`Selection of the icon 423 does not result in the removal of
`previously stored bounding box information and metadata
`asSociated with the image. Re-Selecting the icon 423 pref
`erably results in the display of all bounding boxes that have
`previously been associated with an image. For example, if a
`user selects the forward button 409 and the image displayed
`in FIG. 4 is selected from a database of images and rendered
`in the frame 407, then a user can select the icon 423 resulting
`in all previously stored bounding boxes (e.g. 501, 503, and
`505) being displayed. Re-selecting the icon 423 preferably
`results in the bounding boxes being removed from the
`image. However, the associated bounding box information
`and metadata previously Stored in the association list asso
`ciated with the image will not be affected.
`0036). In accordance with a still further embodiment of
`the present invention, once bounding box positional infor
`mation and metadata have been Stored for an image, pointing
`to a Subject in the image with a mouse pointer (not illus
`trated) in a conventional manner will result in the metadata
`associated with the subject being displayed. As seen in FIG.
`5, a user has selected Liza Hayward with the mouse pointer
`after the bounding box 503 has been dropped onto the
`image, resulting in the display of the “tip”515 including Liza
`Hayward's name.
`0037. In accordance with a still further embodiment of
`the present invention, the user interface 401 can include a
`“content” button (not illustrated) and a “like to send” button
`(not illustrated). The content button can be selected by a user
`before selecting one of the icons 403 in order to indicate that
`the icon to be subsequently selected will be referring to the
`content of the image. Alternatively, the user can Select the
`like to send button in order to indicate that the content of the
`image will be of interest to a Selected perSon. The icon
`Subsequently Selected, after Selecting the like to Send button,
`will be associated with the name of the person to whom the
`user wishes to Send the image. Selecting the content button
`preferably results in a tag (not illustrated) being included in
`the association list Stored at Step 113, indicating that the
`metadata associated with the Subsequently Selected icon
`refers to the content of the image. Similarly, Selecting the
`like to send button preferably results in a tag (not illustrated)
`being included in the association list Stored at Step 113,
`indicating that the metadata associated with the Subse
`quently Selected icon refers to a perSon to whom the user
`wants to Send the image to.
`0038. The icons that are provided at step 103 of the
`preferred method, are preferably default icons having a
`generic indicator displayed in them. For example, Stick
`figures depicting generic people can be displayed in the
`default icons, as seen in FIGS. 4 and 5. However, in
`accordance with the preferred embodiment, if the image
`rendered at step 107 is subsequently only associated with
`one subject at step 109, then the icon used to indicate the
`asSociation can have the generic indicator replaced by a
`
`Scaled and/or cropped version of the rendered image. For
`example, if the rendered image depicts a perSon Jenny and
`the icon associated with the metadata "Jenny' is Subse
`quently Selected, then the generic indicator of the Selected
`icon can be replaced with a Scaled and/or cropped version of
`the image depicting Jenny. Alternatively, the extracted
`image resulting from a bounding box being dropped onto
`Jenny in an image can be depicted in the icon associated
`with the metadata “Jenny”. Replacing the default icons with
`images improves the user friendliness of the icons 403.
`0039. In accordance with still a further embodiment of
`the present invention, the icons provided at Step 103 can
`have e-mail addresses associated with them at step 105. In
`this case, the database of metadata labels, created or
`imported by the user at Step 101, includes an e-mail address
`associated with each name included in the database. When
`an icon is subsequently selected, at step 109 of the method,
`the metadata associated with the Selected icon, describing
`the perSon's name and e-mail address, is Stored in the
`association list at step 113 of the method. Subsequently, if a
`user wishes to e-mail a number of image files to another
`perSon, the e-mail application can Search through the asso
`ciation lists associated with each image file annotated by the
`user and then Send the appropriate images to the perSon. For
`example, the e-mail application can Search through the
`asSociation lists and Send all of the images that have the
`metadata "Jenny' associated with them using Jenny's e-mail
`address. In a further example, if several of the lists have been
`tagged as “like to Send', as described above, then the e-mail
`application can Select those images and e-mail them to the
`address of the perSon indicated by the metadata included in
`the association lists associated with those images.
`0040. Once an image or document has been annotated
`using the method, a user can perform any number of
`functions with the annotated material. For example, a user
`can collate a number of annotated images for printing or
`faxing. Further, a number of annotated images can be
`displayed in Sequence based on the content of the images.
`Still further, the annotation can facilitate in later Searching
`for and recovering the image. Still further, the annotation
`can facilitate in the automatic presentation of images allow
`ing a user to Zoom in on a Subject of interest. Still further,
`the annotation can facilitate in the generation of “tool tips'
`Such that when a user does a conventional mouse roll-over
`of an annotated image, an unknown perSon in the image can
`be identified by pointing the mouse pointer at that perSon.
`Still further, the extraction and display of images in accor
`dance with the preferred implementation can be used to
`Subdivide and/or mosaic an image.
`0041 FIG. 6 is a flow diagram 600 showing a method of
`annotating an image using metadata, in accordance With a
`still further embodiment of the present invention. The pro
`cess of flow chart 600 begins at step 601, where a list of
`metadata labels is provided. The list of metadata labels is
`preferably provided automatically. For example, as dis
`cussed above, a list of people's names can be automatically
`provided by extracting the names from an existing database
`of names, Such as an e-mail address book. In this instance,
`names, associated e-mail addresses and other personal infor
`mation can be extracted from the database. Alternatively, a
`list of metadata labels can be manually generated using any
`known Software database application, or metadata labels can
`be manually inputted by the user using the keyboard 202. At
`
`Meta Platforms, Inc.
`Exhibit 1013
`Page 012
`
`

`

`US 2002/0055955 A1
`
`May 9, 2002
`
`the next step 603, a plurality of icons are provided. Prefer
`ably, the number of icons is equal to the number of metadata
`labels in the list of metadata labels and the icons are
`preferably displayed on the video display 214. The process
`continues at the next step 605, where each label in the list of
`metadata labels is associated with at least one of the gen
`erated icons, similarly to step 105 of FIG. 1, as discussed
`above.
`0.042 At the next step 607, a digi