(12) United States Patent
`US 6,976,229 B1
`Balabanovic et al.
`Dec. 13, 2005
`(45) Date of Patent:
`
`(10) Patent N0.:
`
`U8006976229B1
`
`(54)
`
`(75)
`
`METHOD AND APPARATUS FOR
`STORYTELLING WITH DIGITAL
`PHOTOGRAPHS
`
`Inventors: Marko Balabanovic, London (GB);
`Lonny L. Chu, Palo Alto, CA (US);
`Gregory J. Wolfi', Redwood City, CA
`(US)
`
`(73)
`
`Assignee: Ricoh C0., Ltd., Tokyo (JP)
`
`( * )
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21)
`
`Appl. No.: 09/465,982
`
`(22)
`
`Filed:
`
`Dec. 16, 1999
`
`(51)
`(52)
`
`(58)
`
`(56)
`
`G06F 3/00
`715/838; 715/719; 715/737;
`715/738; 715/780; 715/723
`Field of Search ................................ 345/835, 730,
`345/838, 731, 732, 719, 724, 723, 716, 717;
`707/500.1; 716/500.1, 512; 715/719, 737,
`715/738, 780, 723, 835, 730, 838, 731, 732,
`715/724, 716; 382/305; 358/442
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`............. 715/723
`4/1995 Klingler et al.
`5,404,316 A *
`.......
`715/716
`2/1998 Moran et al.
`5,717,879 A *
`
`6/1999 Anderson et al.
`345/342
`5,917,488 A *
`5,966,122 A * 10/1999 Itoh ........................... 345/328
`
`5,999,173 A * 12/1999 Ubillos ....................... 345/328
`6,084,582 A *
`7/2000 Qureshi et al.
`.. 345/835
`......
`6,097,389 A *
`8/2000 Morris et al.
`.. 345/346
`6,121,966 A *
`9/2000 Teodosio .......
`.. 345/346
`6,166,735 A * 12/2000 Dom et al.
`........
`345/339
`6,333,752 B1 * 12/2001 Hasegawa et al.
`.......... 345/764
`6,346,951 B1 *
`2/2002 Mastronardi ............. 345/716
`6,362,900 B1 *
`3/2002 Squilla et al.
`..
`.. 358/442
`6,453,078 B2 *
`9/2002 Bubie et al.
`.......
`.. 382/305
`6,624,826 B1 *
`9/2003 Balabanovic ............... 345/727
`
`
`
`
`
`OTHER PUBLICATIONS
`
`Media Blocks: Physical Containers, Transports, and Con-
`trols for Online Media, MIT Media Lab.*
`
`* cited by examiner
`
`Primary Examiner—Heather R. Herndon
`Assistant Examiner—Mylinh Tran
`(74)Att0mey, Agent, or Firm—Blakely, Sokoloff, Taylor &
`Zafman LLP
`
`(57)
`
`ABSTRACT
`
`Adisplay showing an imported story track, an authored story
`track, and a story-in-progress track. Each story contains one
`or more objects. The display also shows a large photo
`corresponding to a selected object in a story. Furthermore,
`the display also provides visual forms showing audio clips
`associated with the large photo displayed. The photo may
`have more than one associated audio clips. Control buttons
`are available to navigate among the three tracks and among
`the objects on each track. Other control buttons allow a user
`to record audio clips and to author new stories.
`
`49 Claims, 7 Drawing Sheets
`
`II-
`FY4522
`
`daja 7:51 PM Wed 08 Sep 99
`WW»
`daja 7:43 PM Wed 08 Sep 99
`
`105
`
`110
`
`115
`
`130
`
`135
`
`125
`
`
`
`102
`
`103
`
`104
`
`Samsung-LG—HTC EX. 1009 p. 1
`
`Samsung-LG-HTC Ex. 1009 p. 1
`
`

`

`US. Patent
`
`Dec. 13, 2005
`
`Sheet 1 0f 7
`
`US 6,976,229 B1
`
`m
`
`9
`
`m
`
`or
`
`mm3mmo8:5E5;man
`
`mmamwmoum>>2n.vafimv
`
`))>>>>>>>\<<<(
`
`m9
`
`0:
`
`m:
`
`of
`
`m2
`
`mme
`
`Samsung-LG—HTC EX. 1009 p. 2
`
`Samsung-LG-HTC Ex. 1009 p. 2
`
`

`

`US. Patent
`
`Dec. 13, 2005
`
`Sheet 2 0f 7
`
`US 6,976,229 B1
`
`
`
`omm
`
`omm
`
`mmm
`
`mmm
`
`ovm
`
`Samsung-LG—HTC EX. 1009 p. 3
`
`Samsung-LG-HTC Ex. 1009 p. 3
`
`

`

`US. Patent
`
`Dec. 13, 2005
`
`Sheet 3 0f 7
`
`US 6,976,229 B1
`
`
`
`Samsung-LG—HTC EX. 1009 p. 4
`
`Samsung-LG-HTC Ex. 1009 p. 4
`
`

`

`US. Patent
`
`4
`
`79,6
`
`1B
`
`2,A_.>Es_m__n2figéesovm.A___Ex.3me:twammiinpxogmwcemvmA.._Ex.msm§otwaomizgmgfigaggagmv
`
`
`
`HAmgmmav
`:53A..ov:mE..nEmesvWAwgmwsv
`0.
`mAmm___E§v
`
`
`2;\ll\Absogngmoig2E?
`
`.Aw2=E£v
`
`
`
`
`
`Am=o._.nco_m._®>_Eme
`
`m.m,vGE
`
`A..8:m_..uu_
`
`A>__Emtv
`
`Samsung-LG—HTC EX. 1009 p. 5
`
`Samsung-LG-HTC Ex. 1009 p. 5
`
`

`

`US. Patent
`
`Dec. 13, 2005
`
`Sheet 5 0f 7
`
`US 6,976,229 B1
`
`
`
`
`
`ovmJ£258-28?A._mm__u$mto._3m.Na_E9mamm\\d=;__nu_22mofismv
`
`Ammmeqv
`
`mm.GE
`
`omm
`
`Aaofiv
`
`
`
`omm\A.__Ex.NSmEBhoam2>ancmfimuflws993v
`
`AmmmEV
`
`Ammmév
`
`£228.2ngAszzaoopmaa3%
`
`mmmKA_._Ex.F£95toamm§§z=u9mv£os223v
`
`mmm/£958-06szA__o=u$wtoEm.F2E:m2¢m\\dz§u222m225v
`
`
`aegogéfizmvA._._.zm_>_aooo__nmab83v
`
`ALEx.vsoeaaamwizc=umfinSmE293v
`
`A\.._Ex.m£m§m..oamw\\n9:...umfiuSmEEsocv
`
`AW5.r.
`
`”:29?_Exov
`
`AEEmv
`
`mom
`
`AmmmEV
`
`
`
`
`
`A\___Ex.Ems::mammizzmmficeme893v
`
`
`
`A\.._Ex.m£oEanom\\nBEmgmuSmEEsouv
`
`A\._>m05__nmqa83v
`
`Ammmezv
`
`
`
`A5:0.THEE?Ex?
`
`AmmmEV
`
`ABEmV
`
`AomeV
`
`A=>m0pm=uoab33v
`
`AmmmE<v
`
`oom
`
`Atom?
`
`Samsung-LG—HTC EX. 1009 p. 6
`
`Samsung-LG-HTC Ex. 1009 p. 6
`
`
`
`
`
`
`

`

`US. Patent
`
`Dec. 13, 2005
`
`Sheet 6 0f 7
`
`US 6,976,229 B1
`
`mow
`
`
`
`
`
`gamma?4=8_.Nae_aaamm;§..nu_mmmgfieav4=8...ae_\N_mamm\§5u2mome_-__ev
`EmNEON;”78-82.05:
`
`
`
`
`Am:0.Flash;2wa
`
`-388m-o‘“one?mnmwwoov.ooummmmyuegooegoobezfivAE”;at”;
`
`=8m=nc:o_m:oE_u-=Ev coacméuémozuevAanz,5?“;:o_m:mE_o.nE:£vA._mmm_.u>>
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`-mammmwm-$23?$33.?325%.momiov.moommom-@553
`
`
`
`
`
`
`
`
`
`
`
`
`
`.Swot?whmmmm-omvvmom-Emwmwm.omvvmom-390R.9&4on
`
`
`
`
`
`
`
`
`
`
`
`-803$-558m;905mm-353%-$232-«$33.$va
`
`
`
`
`
`
`
`
`
`
`
`
`
`-N85?352%.wwmmmfiw.38mm?mEmNmN-EmmBTmvmmvg
`
`
`
`
`
`
`
`\
`.3582-A2937
`
`
`
`
`8883.2882.8882.388$.858$-E55
`
`o.OE
`
`Samsung-LG—HTC EX. 1009 p. 7
`
`
`
`
`
`mamas.game.8822.N285-agentammge
`
`
`
`
`
`
`
`
`
`mmnmmo:tmmmmm.momewmoéoowomfi-588$333:.88me
`
`
`
`
`
`
`
`.SmwNTmmmvmmz‘.Emtma-$50vamommmmmwtmvmmommr.
`
`
`
`
`
`
`
`
`
`-mmvmm0-$385.oomommm-mm~mmmm-0235-Nrmvmmv.womowov
`
`
`
`
`
`
`
`
`
`Emma-8883.$892.533.238:.$282
`
`
`
`
`
`
`
`Samsung-LG-HTC Ex. 1009 p. 7
`
`
`
`

`

`US. Patent
`
`Dec. 13, 2005
`
`Sheet 7 0f 7
`
`US 6,976,229 B1
`
` Pointing Device
`
`1%
`
`Display
`m
`
`Processor
`El
`
`‘
`
`’
`
`Disk Drive
`m
`
`
`
`Input/Output
`Device
`
`M
`
`Read-only Memory
`
`3’:
`
`Random Access Memory
`E
`
` Printer
`
`m
`
`700
`
`FIG. 7
`
`Samsung-LG—HTC EX. 1009 p. 8
`
`Samsung-LG-HTC Ex. 1009 p. 8
`
`

`

`US 6,976,229 B1
`
`1
`METHOD AND APPARATUS FOR
`STORYTELLING WITH DIGITAL
`PHOTOGRAPHS
`
`FIELD OF THE INVENTION
`
`The present invention relates generally to field of image
`retrieval and organization. More specifically,
`the present
`invention is directed to multimedia creation using digital
`objects (e.g., images).
`
`BACKGROUND
`
`Photographs play a central role in many types of informal
`storytelling. One of the most common and enjoyable uses for
`photographs is to share stories about experiences, travels,
`friends and family. Almost everyone has experience with
`this form of storytelling, which ranges from the exchange of
`personal reminiscences to family and cultural histories. The
`World Wide Web can facilitate the sharing of such stories in
`digital form and has inspired a movement towards “digital
`storytelling.” Stories in digital form are referred to herein as
`digital stories. Digital photographs have an advantage over
`print photographs in that users can search for and retrieve
`them both by their content (e.g., features such as color and
`texture) and by their metadata (e.g., user-supplied text
`annotations). Today, most digital stories are created by
`people with computer skills using special-purpose software
`for editing images and authoring Web pages. Furthermore,
`these digital stories are created on systems relying on
`graphical interfaces dependent on windows, or tables, and
`some type of pointing or cursor control device. Such systems
`are typically not portable and are complex to use. These
`features make these systems difficult to share. These systems
`also require significant training and are inadequate for mass
`consumer use.
`
`SUMMARY OF THE INVENTION
`
`A method and system that combines capabilities for
`storing, authoring, and viewing various forms of digital
`media are described. In one embodiment, a visual interface
`having three areas is provided. The first area displays three
`tracks of images. One track displays images that are stored
`on the device. A second track displays images of authored
`stories. The third track displays one or more images asso-
`ciated with a story currently being authored on the device.
`Control buttons are available to navigate among the three
`tracks and among the images on each track.
`The second area of the visual interface displays a larger
`version of an image corresponding to a thumbnail image
`selected in any of the three tracks in the first area.
`The third area of the visual interface displays a represen-
`tation of one or more audio clips and other information
`associated with the image being displayed in the second area
`of the visual interface.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present invention is illustrated by way of example in
`the following drawings in which like references indicate
`similar elements. The following drawings disclose various
`embodiments of the present invention for purposes of illus-
`tration only and are not intended to limit the scope of the
`invention.
`
`FIG. 1 illustrates an exemplary embodiment of the display
`screen of one embodiment of a digital story-telling system.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`FIG. 2 illustrates an exemplary embodiment of control
`buttons of the digital story-telling system.
`FIG. 3 illustrates an exemplary expand/collapse view of a
`story.
`FIG. 4 illustrates an exemplary XML metadata file that the
`system reads upon initialization of the system.
`FIG. 5A illustrates an exemplary XML metadata for a
`story.
`FIG. 5B illustrates another exemplary XML metadata file
`for a story.
`FIG. 6 illustrates an exemplary XML metadata file for a
`single photograph.
`
`DETAILED DESCRIPTION
`
`A multimedia story creation and playback system is
`disclosed. The following detailed description sets forth
`numerous specific details to provide a thorough understand-
`ing of the invention. However, those of ordinary skill in the
`art will appreciate that
`the invention may be practiced
`without these specific details. In other instances, well-known
`methods, procedures, protocols, components, algorithms,
`and circuits have not been described in detail so as not to
`obscure the invention.
`
`Some portions of the detailed descriptions that follow are
`presented in terms of algorithms and symbolic representa-
`tions of operations on data bits within a computer memory.
`These algorithmic descriptions and representations are the
`means used by those skilled in the data processing arts to
`most effectively convey the substance of their work to others
`skilled in the art. An algorithm is here, and generally,
`conceived to be a self-consistent sequence of steps leading
`to a desired result. The steps are those requiring physical
`manipulations of physical quantities. Usually, though not
`necessarily, these quantities take the form of electrical or
`magnetic signals capable of being stored, transferred, com-
`bined, compared, and otherwise manipulated. It has proven
`convenient at
`times, principally for reasons of common
`usage, to refer to these signals as bits, values, elements,
`symbols, characters, terms, numbers, or the like.
`It should be borne in mind, however, that all of these and
`similar terms are to be associated with the appropriate
`physical quantities and are merely convenient labels applied
`to these quantities. Unless specifically stated otherwise as
`apparent from the following discussion, it is appreciated that
`throughout the description, discussions utilizing terms such
`as “processing” or “computing” or “calculating” or “deter-
`mining” or “displaying” or the like, refer to the action and
`processes of a computer system, or similar electronic com-
`puting device, that manipulates and transforms data repre-
`sented as physical (electronic) quantities within the com-
`puter system’s registers and memories into other data
`similarly represented as physical quantities within the com-
`puter system memories or registers or other such informa-
`tion storage, transmission or display devices.
`The present invention also relates to apparatus for per-
`forming the operations herein. This apparatus may be spe-
`cially constructed for the required purposes, or it may
`comprise a general purpose computer selectively activated
`or reconfigured by a computer program stored in the com-
`puter. Such a computer program may be stored in a computer
`readable storage medium, such as, but is not limited to, any
`type of disk including floppy disks, optical disks, CD-
`ROMs, and magnetic-optical disks, read-only memories
`(ROMs),
`random access memories (RAMs), EPROMs,
`
`Samsung-LG—HTC EX. 1009 p. 9
`
`Samsung-LG-HTC Ex. 1009 p. 9
`
`

`

`US 6,976,229 B1
`
`3
`EEPROMs, magnetic or optical cards, or any type of media
`suitable for storing electronic instructions, and each coupled
`to a computer system bus.
`The algorithms and displays presented herein are not
`inherently related to any particular computer or other appa-
`ratus. Various general-purpose systems may be used with
`programs in accordance with the teachings herein, or it may
`prove convenient to construct more specialized apparatus to
`perform the required method steps. The required structure
`for a variety of these systems will appear from the descrip-
`tion below.
`In addition,
`the present
`invention is not
`described with reference to any particular programming
`language. It will be appreciated that a variety of program-
`ming languages may be used to implement the teachings of
`the invention as described herein.
`
`Overview
`
`A multimedia storytelling system that provides a user
`capabilities to share digital photographs and stories. In one
`embodiment,
`the system allows the user to seamlessly
`switch between browsing, viewing, authoring, and playing
`back the photographs and/or stories. In one embodiment, a
`multimedia storytelling system includes three components: a
`storage component, a display component, and an interaction
`component. The storage component stores digital media
`objects which are displayed by the display component. The
`system provides the interaction component (e.g., control
`buttons) that allow a user to navigate the digital media
`objects and create stories, or playlists. For purposes herein,
`a story, or a “playlist”, is an ordered collection of digital
`media objects, such as, for example, images or video clips,
`with one or more narration tracks. Imported stories on the
`top track may be merely a single photograph with or without
`a narration track. Photographs may be imported in batches,
`similar to that of a roll of film. Similarly, a collection may
`comprise the images of multiple pages in a single document.
`Thus, although these may not be a narration track,
`the
`imported collection of photographs or single photograph
`may represent a story for purposes herein.
`In one embodiment, the digital media stored in the system
`includes photographs, images, audio and video clips. The
`digital media may include raw objects, such as, for example
`individual photographs, as well as authored objects that
`combine multiple forms of media into integrated stories. An
`example of an authored object is a sequence of photographs
`with an audio, or narration, clip associated with or accom-
`panying the photographs. In one embodiment, the time of
`capture for an object is known and is stored along with the
`object in the system.
`New objects may be input into the system automatically
`without requiring any action from the user. New objects are
`input into the system using one or more sources, such as, for
`example, on flash memory cards containing digital photo-
`graphs, video capture camera, reading digital photographs or
`video clips from floppy or CD-ROM drives, network (e.g.,
`Web) downloads, etc. In one embodiment, the user inserts a
`flash memory card into a slot in the system and the photo-
`graphs from the flash memory card are automatically copied
`and stored in the system. In another embodiment, the system
`is able to receive email messages with attached photographs
`and stories. The photographs and/or stories are then auto-
`matically accessible. It would be apparent to one skilled in
`the art the various ways multimedia objects may be input
`into the system.
`A narration may take a variety of forms, such as, for
`example, recorded audio, digital ink, or typed notes. In one
`embodiment, a narration track may apply to a single object
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`
`or to a number of objects (e.g., a narration track for a
`sequence of images). There may be multiple narration tracks
`for a single object. That is, the same digital object (photo-
`graph) may have several associated narrations.
`The display component of the system provides multiple
`tracks displaying digital media objects to the user. One track
`displays digital media objects that are stored in the system.
`A second track displays digital media objects that have been
`integrated into authored stories. The third track displays one
`or more digital media objects associated with a story cur-
`rently being authored using the system.
`The display component of the system also allows the user
`to navigate through the objects in storage. This may include
`being able to browse through raw objects (e.g., objects
`without narrations associated or attached thereto), browse
`through stories, and play back previously created stories. In
`one embodiment, playing back a story involves showing the
`user a slideshow of images along with any accompanying
`narrations.
`
`The interaction component of the system allows the user
`to create new stories. In one embodiment,
`the user may
`create new stories using one or more of several approaches.
`For example, a user may author a story by starting with an
`empty or blank story, selecting objects to add to the story,
`and adding narrations to individual objects or a range of
`objects. The selection of objects and the addition of narra-
`tions may be performed in any order. A user may author a
`new story by performing edit or delete operations on an
`existing story.
`In one embodiment, the system has a modeless interface,
`giving the users an interface of a small number of buttons
`and no pointing device or touchscreen. In another embodi-
`ment, the system may be implemented on a typical desktop
`computer or on a standalone appliance. The user controls for
`such a system may utilize standard input devices, such as,
`for example, a mouse, a keyboard, a touchscreen, a touch
`pad, or a physical control environment of buttons and/or
`sliders designed specifically for the interaction described
`herein.
`
`FIG. 1 illustrates one embodiment of a display screen and
`interface of a system. Media objects may be loaded onto the
`system from various sources, such as, for example, digital
`cameras, digital video capture, microphones, scanners, or
`may be created on the system itself. These objects are then
`stored in the system on a storage medium (e.g., random
`access memory, a hard drive, flash memory).
`As illustrated in FIG. 1,
`the system provides a visual
`interface 100 which splits the screen into three general areas.
`In one embodiment, the first area 102 at the top of the screen,
`the second, or central, area 103 at the bottom right side of the
`screen, and the third area 104 at the bottom left side of the
`screen.
`
`The first area 102 provides a graphical representation for
`browsing and navigating through media objects (e.g., pho-
`tographs, etc.). In one embodiment, there are three horizon-
`tal tracks of thumbnail images, tracks 105, 110 and 115, each
`of which can be navigated by scrolling.
`In one embodiment, the top track 105 shows images of
`existing photographs ordered by time of creation (or other
`such features). The time of creation may be indicated by
`time stamps. These may be grouped into “rolls of film” or
`other natural clusters. In the case of scanned prints, imported
`stories may correspond to literal rolls of film. In the case of
`digital photographs, the photographs may correspond to a
`set of photographs downloaded from the camera in one
`session. The top track 105 may also display all or many
`existing photographs currently stored in the system. That is,
`
`Samsung-LG—HTC EX. 1009 p. 10
`
`Samsung-LG-HTC Ex. 1009 p. 10
`
`

`

`US 6,976,229 B1
`
`5
`any image stored on the system may be displayed. The
`photographs displayed in the top track 105 may be stories.
`For example, one or more of the photographs in the top track
`105 may be an imported story.
`In one embodiment, the imported stories may be gener-
`ated automatically as a result of a database query. For
`example, a database query for “all images stored on a device
`containing blue” returns images having the “blue” identifier.
`The resulting images are then imported into the system. Note
`that there is no requirement that the first track 105 include
`any stories at all. That is, the first track 105 may contain zero
`or more stories, each of which having one or more images.
`Within an imported story, the photographs are ordered
`chronologically. Photographs from digital cameras may be
`ordered according to when they were taken, while images
`scanned from print photographs are ordered by scanning
`time. In one embodiment, in order to distinguish separate
`stories, the system uses alternating background colors with
`each color corresponding to different story.
`The middle, or second, track 110 contains authored sto-
`ries, each including a sequence of one or more photographs
`selected by an individual. Each story appears as a sequence
`of thumbnail images. In one embodiment, the imported and
`authored stories are ordered according to their time of
`creation with separate stories being visually distinguished
`using different colored backgrounds. In another embodi-
`ment, stories in a track may be separated by spaces in the
`display. The size of the spaces may depend on the length of
`time between the story creation times. For example, in one
`embodiment, photographs taken on the same day are closer
`together on the second track 110 than those separated by a
`day or a weekend. The space may be a function of a time
`duration, but not necessarily linear. For example,
`there
`might be set increments for “day”, “weekend”, “week” and
`“more than a week”.
`
`The bottom, or third, track 115 represents a story being
`authored. That is, the bottom track 115 includes a working
`set of thumbnail images that have been selected and manipu-
`lated by the user. The set of images may only appear in the
`bottom track 115 for the current session with the system. If
`the user wishes to further author a story after a session has
`ended, the user selects the authored story for inclusion into
`the bottom track 115.
`
`In one embodiment, a thumbnail image appears in the
`bottom track 115 if it has been added to the working set by
`pressing one or more control buttons (e.g., either of the +
`(add) or record buttons, as detailed below). At any time, the
`story being authored has a pending status. When the story is
`completed and saved by the user, it joins the set of authored
`stories in the middle track 110. In one embodiment,
`the
`middle track 110 and the bottom track 115 need not be
`
`visually distinguished, nor for that matter, any of the tracks.
`The display of the three tracks 105, 110, and 115 enables
`an essentially modeless interface where a user can simulta-
`neously view authored stories, view/navigate through pho-
`tographs and view/create new stories. The display also
`provides helpful context for viewing the current image. For
`example, when the user selects a thumbnail and views an
`image from one story, adds it to the current story, then adds
`annotation, the context of that image with the surrounding
`thumbnails in both the original and current story remains
`visible and helpful for annotation.
`image appears
`In one embodiment, each thumbnail
`exactly once in the top track 105. It may appear in any
`number of stories in the middle track 110 and appears in the
`bottom track 115 only if it has been explicitly added to or
`annotated as part of the story currently being authored.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`In a typical interaction, the user comes across a thumbnail
`image and adds it to the working set (bottom track 115). The
`users may also record a related voice narration to accom-
`pany the selected thumbnail image. At the end of the session,
`all the thumbnail images in the bottom track 115 are grouped
`into a single story, and the story is then appended or added
`to the middle track 110.
`
`The central area 103 of the screen display shows a large
`image 120. The large image 120 corresponds to a thumbnail
`image of a story or photograph currently being selected in
`the tracks in the first area 102. This allows the user to view
`
`photographs at a size that is comfortable and enjoyable (as
`opposed to the smaller thumbnail versions in tracks 105, 110
`and 115). In another embodiment, other objects may require
`other display characteristics.
`The third area 104 of the screen display shows the
`available audio narrations of the photograph currently dis-
`played in the central area 103. In another embodiment, other
`information, such as, for example, creation date may also be
`displayed in the third area 104. As shown in FIG. 1, there are
`two narrations 130 and 135 available for the photograph 120
`in the central area 103 indicated by the depiction of the wave
`signal. In one embodiment, each narration has a visual
`display corresponding to the length of time of the narration
`(e.g., the longer the signal wave the longer the narration),
`together with related recording information, such as, for
`example, time and date recorded, and the name of recording
`user.
`
`FIG. 2 illustrates an exemplary embodiment of the control
`buttons and the display of a system for creating and/or
`viewing multimedia stories. In one embodiment, the control
`buttons are integrated into and positioned at the sides of the
`body of the system to enable the user manipulate the controls
`while holding the system in two hands. This further removes
`the need for a keyboard, mouse, or other device that forces
`the user to let go of the system.
`As illustrated in FIG. 2,
`the control buttons in this
`embodiment include navigation buttons such as a scroll left
`button 205 to scroll the images in a selected track to the left,
`a scroll right button 210 to scroll the images in a selected
`track to the right, and a track selection button 215 to select
`the track to scroll. In one embodiment, a jog shuttle is used
`to control scrolling.
`In one embodiment, a colored (e.g., yellow) vertical line
`255 indicates the selected track and the selected thumbnail
`
`image of this track. The vertical line 255 is shown in the first
`thumbnail image of the bottom track. The selected thumb-
`nail image corresponds to the large image displayed in the
`central area 250. The track selection button 215 moves the
`vertical line 255 between the three different tracks.
`
`When either of the scroll buttons 205 or 210 is pressed,
`the new selected thumbnail image on the current track either
`to the left or to the right side of the previously selected
`thumbnail image is selected and displayed in the central area
`250. As a different
`thumbnail
`image moves under the
`vertical line 255, the corresponding image is displayed in the
`central area 250. Note that any type of indicator or indication
`may be used on the screen to select an image. It should also
`be noted that while in one embodiment, the scroll button 205
`or 210 causes the images in the track with the vertical line
`255 to move to the left or right while leaving the vertical line
`255 in place, the system could keep the images in place
`while moving the vertical line 255 to select an image. In one
`embodiment,
`the navigation between the images may be
`controlled by an external joystick communicating with the
`system through a joy stick interface.
`
`Samsung-LG—HTC Ex. 1009 p. 11
`
`Samsung-LG-HTC Ex. 1009 p. 11
`
`

`

`US 6,976,229 B1
`
`7
`In one embodiment, the user may quickly traverse the
`thumbnail images on a track by using variable-speed scroll-
`ing. At standard speed, the display appears as shown in FIG.
`2. In one embodiment, standard speed refers to pressing one
`time on the left scroll button 205 or the right scroll button
`210. This moves the track by exactly one thumbnail image.
`Faster scrolling speeds may be enabled by rendering low
`resolution versions of the thumbnail images that are quickly
`accessible from a separate index in the system and by not
`rendering the image in the central area 250 or the narrations
`in the audio area 260. In one embodiment, such images may
`be cached in memory to increase access speed. To increase
`scrolling speed, the user may need to hold down a scroll
`buttons 205 or 210 for a longer period of time. When this
`occurs,
`the images appear to be scrolling faster by the
`vertical line indicator 255.
`
`In one embodiment, an expand/collapse button 265 con-
`trols the view or the number of photographs displayed for
`each story on one of the three tracks. In the expanded view,
`every thumbnail image in a story is shown. In one embodi-
`ment, the expanded view is the default view. In the collapsed
`view, each story is represented by a predetermined number
`of thumbnail images (e.g., one thumbnail image). In one
`embodiment, a first thumbnail image of the story is used to
`represent the story. The collapsed view allows for faster
`navigation. FIG. 3 illustrates an exemplary expand/collapse
`view of a story. Referring to FIG. 3, the two stories 305 and
`310, with thumbnail images belonging to one story having
`the same colored background, are shown in expanded form.
`The same two stories are illustrated in collapsed form in
`thumbnails 315 and 320, with the first thumbnail image of
`each story 305 and 310 being used as the thumbnail image
`in the corresponding collapsed form.
`Referring back to FIG. 2, the cluster of buttons at the
`bottom left of the system 200 provides the user with addi-
`tional controls for authoring, browsing and playing back of
`stories. A play button 220 is used to enter a play mode to
`allow an individual to view existing stories. During the play
`mode, the system starts playback from the currently selected
`thumbnail image. In one embodiment, although not neces-
`sary, each thumbnail image has an audio clip associated with
`it as part of a story in which it resides. Furthermore, each
`thumbnail image may be part of more than one story, and
`thus may have more than one audio clip associated with it,
`one audio clip for each story. However, an audio clip may or
`may not contain any recorded narration. If a narration exists,
`it is played through a built-in loudspeaker in the system (not
`shown). If there is no recorded narration, there will be a
`pause of a certain length of time, for example one second.
`After the narration is played or after the pause (due to not
`having the narration),
`the selected track automatically
`scrolls forward to a next thumbnail image in the story. In one
`embodiment, the next thumbnail image is determined based
`on a default viewing and storytelling direction of left-to-
`right. If the user navigates to a new thumbnail image while
`the system is playing, for example scrolling left or right,
`changing to a different
`track, playback of the currently
`playing audio clip is stopped. If the user takes no further
`action for a brief length of time (e.g., one second),
`the
`system plays the audio clip associated with the newly
`selected thumbnail image and continues with that story from
`that point on.
`As discussed above, the selected thumbnail image may
`have more than one associated audio clip and corresponding
`narrations. Whenever a thumbnail image is selected, all of
`the audio narrations associated with that thumbnail image
`are displayed in the audio area 260. In one embodiment, if
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`the selected thumbnail image has more than one associated
`audio clip, and when the system is playing an audio clip,
`pressing the play button 220 causes the system to advance to
`a next audio clip.
`Referring to FIG. 1, there are two available narrations for
`the selected thumbnail image, narration 120 and narration
`125. Each narration is marked with the time of recording and
`the name of the recording user. The length of the wavy lines
`is proportional to the duration of the audio. The narration
`associated with the selected story is listed first, as in narra-
`tion 120. Referring to FIG. 2, the narration associated with
`the selected story is played by default when the play button
`220 is pressed. Pressing the play button 220 multiple times
`in quick succession selects one of the alternate audio clips
`and playback “jumps” to the corresponding story, providing
`a method of automatic hyperlinking between stories.
`The system remains in the play mode until the stop button
`225 is pressed or until the end of story is reached. When this
`occurs, the system enters into stop mode. While in the stop
`mode, no audio clip is played by the system.
`In one embodiment, while the system is in the play mode,
`the remove (—) button 235 has no effect on the play mode of
`the system. However, pressing the add (+) button 230 adds
`the currently displayed image to the working set. Thus, the
`system appends the currently displayed image to the work-
`ing set, but it does not stop the play mode. Furthermore, if
`the user presses the save button 245 while the system is in
`the play mode, the system stops playing the audio clip and
`performs a save operation. Furthermore, if the use