(12) United States Patent
`Torres et ai.
`
`111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US006738075Bl
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,738,075 Bl
`*May 18, 2004
`
`(54) METHOD AND APPARATUS FOR CREATING
`AN INTERACTIVE SLIDE SHOW IN A
`DIGITAL IMAGING DEVICE
`
`(75)
`
`Inventors: Dan Torres, San Carlos, CA (US); Eric
`C. Anderson, San Jose, CA (US)
`
`(73) Assignee: FlashPoint Technology, Inc.,
`Peterborough, NH (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.c. 154(b) by 0 days.
`
`This patent is subject to a terminal dis(cid:173)
`claimer.
`
`(21) Appl. No.: 09/223,961
`
`(22) Filed:
`
`Dec. 31, 1998
`
`Int. CI? ................................................ H04N 5/222
`(51)
`(52) U.S. CI. .................. ... ... ........... 345/723; 348/333.05
`(58) Field of Search ................................. 345/723, 732,
`3451734, 788; 348/333.05, 333, 332, 352
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`6,038,545 A * 3/2000 Mandeberg et al.
`
`6,072,480 A * 6/2000 Gorbet et al. ............... 345/302
`6,215,523 B1 * 4/2001 Anderson ........ .. ...... 348/333.5
`6,237,010 B1 * 5/2001 Hui et al.
`................... 707/502
`6,249,316 B1 * 6/2001 Anderson ................ 348/333.5
`6,317,141 B1 * 11/2001 Pavlay et al. ... .. ... ....... 345/732
`
`* cited by examiner
`
`Primary Examiner-Cao (Kevin) Nguyen
`(74) Attorney, Agent, or Firm~awyer Law Group LLP
`
`(57)
`
`ABSTRACT
`
`A method and apparatus for interactively presenting a slide
`show in a digital imaging device. The method aspect of the
`present invention includes first creating a slide show from
`randomly selected ones of the heterogeneous media objects
`stored in the digital imaging device, where each one of the
`heterogeneous media objects includes at least one media
`type such as, a still image, video, and audio. When the slide
`show is subsequently presented, each one of the media
`objects in the slide show is automatically displayed on a
`display the order selected. According to the present
`invention, the user is able to change the order the media
`objects will be played back, and able to control how long
`each one of the media objects will be played back, to thereby
`interactively present the slide show from the digital imaging
`device.
`
`.......... 705/15
`
`14 Claims, 13 Drawing Sheets
`
`356
`
`354
`
`302
`
`D
`
`140
`
`D ]-352
`
`Image #314
`10:26AM
`12/4/96
`
`306
`
`Mark
`308a
`
`Edit
`
`Save
`
`~310
`
`308b
`
`308c
`
`Q20Sa
`
`Q20Sb
`
`Q206C
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 1
`
`

`

`I--"
`~
`(I)
`""-l
`b
`00
`tj
`-..CJ\
`rJ'l
`
`e
`
`'""'" ~
`'""'" o ....,
`~ .....
`'JJ. =(cid:173)~
`
`~
`C
`C
`N
`~CIO
`'""'"
`'-<
`~
`~
`
`~ = .....
`~ .....
`~
`•
`rJl
`d •
`
`FIG. 1
`
`128
`
`Memory
`
`Non-volatile
`
`126
`DRAM
`
`124
`CPU
`
`136
`
`134
`
`Supply I-Manager
`Power
`
`Power
`
`154
`
`I 1152",'
`
`\
`
`122
`
`Device
`Storage
`
`Mass
`
`150
`I/O
`
`USB
`
`'---
`
`-
`
`IRDA
`
`I
`ill
`Codec
`Video
`
`120
`Codec
`Audio
`
`,1.
`
`Out
`Video
`
`138\
`
`~ 128 ~ Controller
`Vic eo
`
`132
`
`,
`,
`
`,
`I
`
`Mixer
`
`148
`
`Status LCD
`
`146
`
`Buttons
`
`k. Audio
`
`142
`
`In
`
`140
`LCD
`
`114
`
`-
`
`.. Processing ...
`
`ill
`DSP
`
`Image
`
`ill
`Device
`Imaging
`
`ill
`
`Com pute r
`
`100
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 2
`
`

`

`u.s. Patent
`
`May 18, 2004
`
`Sheet 2 of 13
`
`US 6,738,075 Bl
`
`DD
`DDD
`DDD
`
`Mark
`
`View
`
`140
`
`110
`
`Edit
`
`Q206a Q206b Q206C
`
`202
`
`212
`
`FIG.2A
`
`204
`
`200
`
`210~
`
`~
`100
`
`I
`
`208
`
`FIG.28
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 3
`
`

`

`u.s. Patent
`
`May 18, 2004
`
`Sheet 3 of 13
`
`US 6,738,075 Bl
`
`Icon
`D
`rJi1
`[[[I]
`
`~
`D::]
`f]
`
`•
`•
`•
`
`Media Types
`
`Single Still Image
`
`Burst or Time Lapse
`
`Panorama
`
`Video
`
`Audio Clip
`
`Text File
`
`•
`•
`•
`
`FI G. 3
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 4
`
`

`

`u.s. Patent
`
`May 18, 2004
`
`Sheet 4 of 13
`
`US 6,738,075 Bl
`
`304
`
`302
`
`140
`
`Mark
`308a
`
`Edit
`
`Save
`308e
`
`308b
`
`Q206a
`
`Q206b Q206C
`
`FIG.4A
`
`302
`
`D
`
`356
`
`354
`
`Image #314
`12/4/96
`10:26AM
`
`306
`
`Save
`M ark
`~--~3~0~8-a--~~3~08~b--~~30~8~e
`
`=:J- 310
`
`Q206a Q206b Q206C
`
`FIG.4B
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 5
`
`

`

`u.s. Patent
`
`May 18, 2004
`
`Sheet 5 of 13
`
`US 6,738,075 Bl
`
`Press MarX'
`Function Key
`
`510
`
`Select Next
`Image
`
`Display Mark
`Number
`
`504
`
`S06
`
`Update Mati; Sankey
`to Unma/i(
`
`514
`
`Select a Function
`to Apply to the Group
`
`524
`Renumber all
`Higher Number
`Marks
`
`FIG. 5
`
`516
`
`520
`
`Remove MarX
`Number from cell
`
`y
`
`526
`
`Updale Unmaf1( Sonkey
`to Mark
`
`1 - - -....
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 6
`
`

`

`u.s. Patent
`
`May 18, 2004
`
`Sheet 6 of 13
`
`US 6,738,075 BI
`
`140
`
`300
`
`\
`
`300 \
`
`Il:::J
`
`Add'S·SS. ~
`
`[] '"'" ~LE
`V-
`u::::J [i 1 0
`D []]Q]
`
`''""'
`
`No I.
`
`B:::]
`
`n:::]
`
`302
`
`Mark
`\..
`308a
`
`Edit
`\..
`
`308b
`
`Save
`\..
`308e
`
`Q20sa Q20Sb Q20SC
`
`FIG.6
`
`140
`
`Add,es se
`s
`
`~ []
`[] E-M a,1
`n:::] [i~
`V - 322
`2 ~ ~ ~
`I(J
`
`Sou nd
`Nol e
`
`D
`
`Mark
`\.
`308a
`
`Edjt
`\..
`
`308b
`
`Save
`\.
`
`308e
`
`Q20sa Q20Sb Q20SC
`
`FIG.7
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 7
`
`

`

`u.s. Patent
`
`May 18, 2004
`
`Sheet 7 of 13
`
`US 6,738,075 BI
`
`140
`
`342 \
`
`3
`
`Nol e
`
`E· Mail
`Ad dresse
`
`0
`~ s
`
`~ [E
`So und ~ []~
`[]~[]
`
`Mark
`\.. 308a
`
`Edit
`\. 30Bb
`
`Save
`\. 308e
`
`Q206a
`
`Q206b
`
`Q206C
`
`FIG.8
`
`';-360'
`
`Directory
`
`IRool
`
`Slide Show 1
`
`Media Object 6
`
`Media Object 8
`
`Media Object 1
`
`';-360
`
`Meta Data File
`
`Pointer to Media Object 6
`Position(1 )
`Duration of Play(5 sec.)
`Pointer to Audio File
`
`Pointer to Media Object 8
`Position (2)
`Duration of Play(Auto )
`Pointer 10 Audio File
`
`Pointer to Med ia Object 1
`Position(3)
`Duration of Play(3 sec. )
`Pointer 10 Audio File
`
`FIG.9A
`
`FIG.98
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 8
`
`

`

`u.s. Patent
`
`May 18, 2004
`
`Sheet 8 of 13
`
`US 6,738,075 BI
`
`---- -------, , , , , , ,
`
`~
`j
`------------t;::::::::=t------'
`
`100
`
`FIG. 10
`
`350
`
`Slide Show
`Edit Screen
`400
`
`404
`
`Pro~erties 408
`------ .... _-
`-----_ .. _--
`
`384
`
`380
`
`140
`
`J- 352
`
`402
`
`Choose
`
`Q 308a
`
`Hel
`
`Q 308b
`
`=r- 310
`Q 308e
`
`206a
`
`206b
`
`206c
`
`FIG. 11
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 9
`
`

`

`u.s. Patent
`
`May 18, 2004
`
`Sheet 9 of 13
`
`US 6,738,075 BI
`
`422
`
`424
`
`140
`
`Image
`Editing
`Screen
`420
`
`404
`
`436
`
`Video
`Editing
`Screen
`430
`
`440
`
`~ Original MOdified~ J- 352
`+ Contrast
`<> Brightness
`<> Color
`<>
`Q 308a
`
`Choose
`
`Q 308b
`
`J- 310
`Q 308e
`
`402
`
`206a
`
`206b
`
`206c
`
`FIG. 12
`
`438
`
`J- 352
`
`402
`
`<> Cue
`1,11-----1 g ~~~~=~~~=~~~ ====::::::
`J- 310
`Q 308a Q 308b
`206b Q 308c
`
`L--_____ .........
`
`Zoom I n
`
`Zoom 0 ut
`
`206a
`
`206c
`
`FIG. 13
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 10
`
`

`

`u.s. Patent
`
`May 18, 2004
`
`Sheet 10 of 13
`
`US 6,738,075 Bl
`
`Video
`Editing
`Screen
`430
`
`404
`
`440
`
`• Cue
`<> ---------
`<> ---------
`Q 308b
`Insert Q 308a
`
`438
`
`J- 352
`
`402
`
`J- 310
`Q 308e
`
`206a
`
`206b
`
`206c
`
`FIG. 14
`
`438
`
`J- 352
`
`402
`
`J- 310
`
`436
`
`404
`
`Video
`Editing
`Screen
`430
`
`440
`
`<> Preview
`<> Cue
`<> ---------
`<> ---------
`Q 308b
`Q 308a
`
`Move
`
`Choose
`
`206a
`
`206b
`
`FIG. 15
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 11
`
`

`

`u.s. Patent
`
`May 18, 2004
`
`Sheet 11 of 13
`
`US 6,738,075 Bl
`
`436
`
`404 --r-__
`
`Video
`Editing
`Screen
`430
`
`Cue
`
`438
`
`J- 352
`
`402
`
`440
`
`436
`
`440
`
`Video
`Editing
`Screen
`430
`
`~~~~ ____ ~~ ____ ~~~ 310
`
`Q 308a
`
`206a
`
`Q 308b Q 308c
`
`206b
`
`206c
`
`FIG. 16
`
`Cue
`
`438
`
`J- 352
`
`402
`
`~.......;;....~~ __ ~;.;..;.,...;...~ __ ~~~ ~ 310
`
`Q 308a
`
`206a
`
`Q 308b Q 308c
`
`206b
`
`206c
`
`FIG. 17
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 12
`
`

`

`u.s. Patent
`
`May 18, 2004
`
`Sheet 12 of 13
`
`US 6,738,075 Bl
`
`436
`
`Audio
`Editing
`Screen
`450
`
`438
`
`J- 352
`
`404~~~--~~~~======~
`
`402
`
`+ Play
`o L...;C~u:...;;.e _____ ----I
`o ~---=---=---=====:
`o ...... --_---_---_-__ ~
`~Z~o~om~ln~~Z~o~o~m~O~u_t~ ____ .-~ ==t- 310
`
`Q 308a Q 308b Q 308c
`
`206a
`
`206b
`
`206c
`
`FIG. 18
`
`Text Editing
`Screen
`460
`
`462
`
`466
`hony shows
`The Seventh Sy
`evidence of a c an e n style
`464 ~ from his earlier ones
`Change\
`abc d[illf 9 h
`j
`klmnopqrst
`uvwxyz
`~~T~e~~ __ ~S~h~ift~~ ____ .-~ ==t- 310
`206a Q 308b Q 308c
`Q 308a
`
`J- 352
`
`402
`
`I
`
`,
`
`/
`
`206b
`
`206c
`
`FIG. 19
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 13
`
`

`

`u.s. Patent
`
`May 18, 2004
`
`Sheet 13 of 13
`
`US 6,738,075 BI
`
`Play List
`
`200a
`
`Next
`
`FIG.20
`
`Previous
`
`200b
`
`482
`
`140
`
`J- 352
`
`Choose
`
`Q 308a
`
`206a
`
`:=l- 310
`
`Q 308b Q 308c
`
`206b
`
`206c
`
`FIG.21
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 14
`
`

`

`US 6,738,075 Bl
`
`1
`METHOD AND APPARATUS FOR CREATING
`AN INTERACTIVE SLIDE SHOW IN A
`DIGITAL IMAGING DEVICE
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`The present invention is related to the following
`co-pending U.S. Patent Applications:: Ser. No. 09/223,962
`entitled "Method And Apparatus For Creating A Multimedia
`Presentation From Heterogeneous Media Objects In A Digi(cid:173)
`tal Imaging Device," and Ser. No. 09/223,960 entitled
`"Method And Apparatus For Editing Heterogeneous Media
`Objects In A Digital Imaging Device", both filed concur(cid:173)
`rently herewith; and Ser. No. 08/716,018 entitled "Method
`And System For Displaying Images And Associated Media
`Types In The Interface Of A Digital Camera," filed Sep. 9,
`1996.
`
`FIELD OF THE INVENTION
`
`The present invention relates generally to a digital imag(cid:173)
`ing device and more particularly to a method and apparatus
`for creating, editing and presenting a multimedia presenta(cid:173)
`tion comprising heterogeneous media objects in the digital
`imaging device.
`
`BACKGROUND OF THE INVENTION
`
`2
`Since digital cameras capture images and sound in digital
`format, their use for creation of multimedia presentations is
`ideal. However, despite their capability to record still
`images, audio, and video, today's digital cameras require the
`5 user to be very technologically proficient in order to create
`multimedia presentations.
`For example, in order to create a multimedia presentation,
`the user first captures desired images and video with the
`camera, and then downloads the images to a personal
`10 computer or notebook computer. There, the user may import
`the images and video directly into a presentation program,
`such as Microsoft PowerPoint™. The user may also edit the
`images and video using anyone of a number of image
`editing software applications. After the Power Point presen-
`15 tation has been created, the user must connect the PC or
`notebook to a projector to display the presentation. Finally,
`the user typically controls the play back of the presentation
`using a remote control.
`Due to the limitations of today's digital cameras in terms
`20 of capabilities and features, the user is forced to learn how
`to operate a computer, image editing software, and a pre(cid:173)
`sentation program in order to effectively create and display
`the multimedia presentation. As the use of digital cameras
`becomes increasingly mainstream, however, the number of
`25 novice computer users will increase. Indeed, many users will
`not even own a computer at all. Therefore, many camera
`owners will be precluded from taking advantage of the
`multimedia capabilities provided by digital cameras.
`What is needed is an improved method for creating,
`editing, and displaying a multimedia presentation using
`images and/or video from a digital. imaging device. The
`present invention addresses such a need.
`
`The use of digital cameras is rapidly proliferating and
`they may one day overtake 35 mm SLR's in terms of
`worldwide sales. There are basically three types of digital 30
`cameras; digital still cameras, digital video cameras, and
`hybrid digital-video cameras.
`Still digital cameras are used primarily for capturing high
`quality static photographs, and offer a less expensive alter(cid:173)
`native to digital video cameras. Still digital cameras are 35
`typically less expensive because they have far less process(cid:173)
`ing power and memory capacity than digital video cameras.
`Digital video cameras differ from digital still cameras in
`a number of respects. Digital video cameras are used to
`capture video at approximately thirty frames per second at
`the expense of image quality. Digital video cameras are
`more expensive than still cameras because of the extra
`hardware needed. The uncompressed digital video signals
`from all the low-resolution images require huge amounts
`memory storage, and high-ratio real-time compression
`schemes, such as MPEG, are essential for providing digital
`video for today's computers. Until recently, most digital
`video recorders used digital magnetic tape as the primary
`storage media, which has the disadvantage of not allowing
`random access to the data.
`Hybrid digital-video cameras, also referred to as multi(cid:173)
`media recorders, are capable of capturing both still JPEG
`images and video clips, with or without sound. One such
`camera, the M2 Multimedia Recorder by Hitachi America,
`Ltd., Brisbane, Calif., stores the images on a PC card hard 55
`disk (PCMCIA Type III), which provides random access to
`the recorded video data.
`All three types of cameras typically include a liquid(cid:173)
`crystal display (LCD) or other type of display screen on the
`back of the camera. Through the use of the LCD, the digital 60
`cameras operate in one of two modes, record and play. In
`record mode, the display is used as a viewfinder in which the
`user may view an object or scene before taking a picture. In
`play mode, the display is used a playback screen for allow(cid:173)
`ing the user to review previously captured images and/or 65
`video. The camera may also be connected to a television for
`displaying the images on a larger screen.
`
`SUMMARY OF THE INVENTION
`
`The present invention provides a method and apparatus
`for interactively presenting a slide show in a digital imaging
`device. The method aspect of the present invention includes
`first creating a slide show from randomly selected ones of
`40 the heterogeneous media objects stored in the digital imag(cid:173)
`ing device, where each one of the heterogeneous media
`objects includes at least one media type such as, a still
`image, video, and audio. When the slide show is subse(cid:173)
`quently presented, each one of the media objects in the slide
`45 show is automatically displayed on a display the order
`selected. According to the present invention, the user is able
`to change the order the media objects will be played back,
`and able to control how long each one of the media objects
`will be played back, to thereby interactively present the slide
`50 show from the digital imaging device.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a block diagram illustrating of one preferred
`embodiment of a digital video camera (DVC) for use in
`accordance with the present invention;
`FIGS. 2A and 2B are diagrams depicting an exemplary
`form factor design. for the DVe.
`FIG. 3 is a table listing example media types that may be
`captured and stored by the DVe.
`FIGS. 4A-4B is a diagram illustrating one preferred
`embodiment of the review mode screen.
`FIG. 5 is a flowchart depicting the process of creating an
`ordered group of heterogeneous media objects in accordance
`with the present invention.
`FIGS. 6-8 are diagrams illustrating examples of marking
`heterogeneous media objects.
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 15
`
`

`

`US 6,738,075 Bl
`
`3
`FIGS. 9A-9B is a diagram illustrating a slide show object
`implemented as a metadata file.
`FIG. 10 is a diagram illustrating the DVC connected to
`external projector, and alternatively to a television.
`FIG. 11 is a diagram illustrating the components of the
`slide-show edit screen in accordance with the present inven(cid:173)
`tion.
`FIG. 12 is a diagram illustrating the image editing screen.
`FIG. 13 is a diagram illustrating the video editing screen.
`FIGS. 14-17 are diagrams illustrating the process of
`editing a video on the DVC by creating and moving a video
`clip.
`FIG. 18 is a diagram illustrating an audio editing screen
`for editing audio media types.
`FIG. 19 is a diagram illustrating a text editing screen for
`editing text media types.
`FIG. 20 is a diagram illustrating the mapping of the
`four-way control during slide show presentation.
`FIG. 21 is a diagram illustrating the properties page of a
`media object.
`
`DETAILED DESCRIPTION OF IRE
`INVENTION
`
`The present invention is a method and apparatus for
`creating and presenting a multimedia presentation compris(cid:173)
`ing heterogeneous media objects stored in a digital imaging
`device. The following description is presented to enable one
`of ordinary skill in the art to make and use the invention and
`is provided in the context of a patent application and its
`requirements. Although the present invention will be
`described in the context of a digital video camera, various
`modifications to the preferred embodiment will be readily
`apparent to those skilled in the art and the generic principles
`herein may be applied to other embodiments. That is, any
`digital imaging device used to store and display and/or
`video, could incorporate the features described hereinbelow
`and that device would be within the spirit and scope of the
`present invention. Thus, the present invention is not
`intended to be limited to the embodiment shown but is to be
`accorded the widest scope consistent with the principles and
`features described herein.
`Referring now to FIG. 1, a block diagram of one preferred
`embodiment of a digital video camera (DVC) is shown for
`use in accordance with the present invention. The DVC 100
`is preferably capable of capturing and displaying various
`types of image data including digital video and high(cid:173)
`resolution still images.
`The DVC 100 comprises an imaging device 110, a com(cid:173)
`puter 112, and a hardware user interface 114. The Imaging
`device 110 includes an image sensor (not shown), such as a
`charged coupled device (CCD) or a CMOS sensor, for
`capturing frames of image data in bayer format. The image
`frames are transferred from the imaging device 110 to the
`computer 112 for processing, storage, and display on the
`hardware user interface 114.
`The computer 112 includes an image processing digital(cid:173)
`signal-processor (DSP) 116, a video codec 132, an audio
`codec 132, a mass storage device 122, a CPU 124, a DRAM
`126, an internal nonvolatile memory, a mixer, and a video
`control 132. The computer 112 also includes a power supply
`134, a power manager 136, and a system bus 138 for
`connecting the main components of the computer 112.
`The hardware interface 114 for interacting with the user
`includes a display screen 140 for displaying the digital video
`
`20
`
`15
`
`4
`and still images, an audio subsystem 142 for playing and
`recording audio, buttons and dials 146 for operating the
`DVC 100, and an optional status display 148.
`The CPU 124 may include a conventional microprocessor
`5 device for controlling the overall operation of camera. In the
`preferred embodiment, The CPU 124 is capable of concur(cid:173)
`rently running multiple software routines to control the
`various processes of camera within a multithreaded envi(cid:173)
`ronment. In a preferred embodiment, The CPU 124 runs an
`10 operating system that includes a menu-driven GUI. An
`example of such software is the Digita™ Operating Envi(cid:173)
`ronment by FlashPoint Technology of San Jose, Calif.
`Although the CPU 124 is preferably a microprocessor, one
`or more DSP 116's (digital signal processor) or ASIC's
`(Application Specific Integrated Circuit) could also be used.
`Non-volatile memory 128, which may typically comprise
`a conventional read-only memory or flash memory, stores a
`set of computer readable program instructions that are
`executed by the CPU 124. Input/Output interface (110) 150
`is an interface device allowing communications to and from
`computer 112. For example, 110 150 permits an external host
`computer (not shown) to connect to and communicate with
`computer 118.
`Dynamic Random-Access-Memory (DRAM) 126 is a
`25 contiguous block of dynamic memory that may be selec(cid:173)
`tively allocated for various storage functions. DRAM 126
`temporarily stores both raw and compressed image data and
`is also used by CPU 124 while executing the software
`routines used within computer 112. The raw image data
`30 received from imaging device 110 is temporarily stored in
`several input buffers (not shown) within DRAM 126. A
`frame buffer (not shown) is used to store still image and
`graphics data via the video control 132 and/or the mixer.
`Power supply 134 supplies operating power to the various
`35 components of camera. Power manager 136 communicates
`via line with power supply 134 and coordinates power
`management operations for camera. In the preferred
`embodiment, power supply 134 provides operating power to
`a main power bus 152 and also to a secondary power bus
`40 154. The main power bus 152 provides power to imaging
`device 110, 110 150, Non-volatile memory 128 and remov(cid:173)
`able memory. The secondary power bus 154 provides power
`to power manager 136, CPU 124 and DRAM 126.
`Power supply 134 is connected to main batteries and also
`45 to backup batteries 360. In the preferred embodiment, a
`camera user may also connect power supply 134 to an
`external power source. During normal operation of power
`supply 134, the main batteries (not shown) provide operat(cid:173)
`ing power to power supply 134 which then provides the
`50 operating power to camera via both main power bus 152 and
`secondary power bus 154. During a power failure mode in
`which the main batteries have failed (when their output
`voltage has fallen below a minimum operational voltage
`level) the backup batteries provide operating power to power
`55 supply 134 which then provides the operating power only to
`the secondary power bus 154 of camera.
`FIGS. 2A and 2B are diagrams depicting an exemplary
`form factor design for the DVC 100, shown here as a
`clam-shell design having a rotatable imaging device 110.
`60 FIG. 2Ais a top view of the DVC 100 in an opened position,
`while FIG. 2B is a top view of the DVC 100 in a closed
`position. FIG. 2A shows the display screen 140, a four-way
`navigation control 200, a mode dial 202, a display button
`204, a set of programmable soft keys 206, a shutter button
`65 208, a menu button 210, and an audio record button 212.
`The mode dial 202 is used to select the operating modes
`for DVC 100, which include a capture mode (C) for record-
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 16
`
`

`

`US 6,738,075 Bl
`
`5
`ing video clips and for capturing images, a review mode (R)
`for quickly viewing the video clips and images on the
`display screen 140, and a play mode (P) for viewing
`full-sized images on the display screen 140.
`When the DVC 100 is placed into capture mode and the 5
`display screen 140 is activated, the camera displays a "live
`view" of the scene viewed through the camera lens on the
`display screen 140 as a successive series of real-time frames.
`If the display screen 140 is not activated, then the user may
`view the scene through a conventional optical viewfinder 10
`(not shown).
`Referring to FIGS. 1 and 2A, during live view, the
`imaging device 110 transfers raw image data to the image
`processing DSP 116 at 30 frames per second (fps), or 60
`fields per second. The DSP 116 performs gamma correction 15
`and color conversion, and extracts exposure, focus, and
`white balance settings from the image data and converts the
`data into CCIR 650 streaming video. (CCIR 650 is an
`international standard for digital video designed to encom(cid:173)
`pass both NTSC and PAL analog signals, providing an 20
`NTSC-equivalent resolution of 720x486 pixels at 30 fps. It
`requires 27 MB per second and uses three signals: one 13.5
`MB/sec luminance (gray scale) and two 6.75 MB/sec
`chrominance (color)).
`After processing, the streaming video from the DSP 116 25
`is transferred to the mixer for the overlay of optional
`graphics and/or images onto the video. The graphics data
`from the DRAM's 126 frame buffer is transferred to the
`mixer in synch with streaming video, where the mixer
`combines the graphic data with the video. After the stream(cid:173)
`ing video and the graphics are combined, the video is
`displayed on the display screen 140 via the video control
`132. A video out port is also provided to display the video
`on an external display device.
`When the user initiates the video capture function to
`record the digital video, the streaming video output from the
`DSP 116 is also transferred to the video codec 132 for
`compression and storage. The video codec 132 performs
`MPEG-2 encoding on the streaming video during recording, 40
`and performs MPEG-2 decoding during playback. The video
`codec 132 may include local memory, such as 32 Mbits of
`SDRAM 126 for example, for MPEG-2 motion estimation
`between frames. Such video codecs 132 are commercially
`available from Sony Electronics (CXD1922QO) and Mat- 45
`sushita Electronics Corp.
`As the video codec 132 compresses the digital video, the
`compressed video stream is transferred to a temporary buffer
`in DRAM 126. Simultaneously, audio is recorded by the
`audio subsystem 142 and transferred to the audio codec 132
`for compression into a compressed audio format, such
`MPEG Audio Layer 3 (MP3), which is common internet
`format. In an alternative embodiment, the audio could be
`compressed into AC-3 format, a well-known Dolby Digital
`audio recording technology that provides six surround(cid:173)
`sound audio channels.
`The CPU 124 mixes the compressed video and audio into
`a specified format, such as MPEG-2, for example. After the
`compressed MPEG-2 data is generated, the CPU 124 trans(cid:173)
`fers the MPEG-2 data to the removable mass-storage device
`122 for storage. In a preferred embodiment, the mass storage
`device 122 comprises a randomly accessible 3-inch record(cid:173)
`able DVD drive from Toshiba/Panasonic, or a one-inch 340
`MB MicroDrive™ from IBM, for example.
`The video architecture inputs the video stream from the
`DSP 116 directly into the mixer, rather than first storing the
`video in memory and then inputting the video to the mixer,
`
`6
`in order to save bus bandwidth. However, if sufficient bus
`bandwidth is provided (e.g., 100 MHz), the video stream
`could be first stored in memory.
`Although the resolution of the display screen 140 may
`vary, the display screen 140 resolution is usually much less
`than the resolution of the image data that's produced by
`imaging device 110 when the user captures a still image at
`full resolution. Typically, the resolution of display screen
`140 is Y4 the video resolution of a full resolution image.
`Since the display screen 140 is capable of only displaying
`images at Y4 resolution, the images generated during the live
`view process are also Y4 resolution.
`As stated above, the DVC 100 is capable of capturing
`high-resolution still images in addition to video. When the
`user initiates the capture function to capture a still or
`sequential image, the image device captures a frame of
`image data at a resolution set by user. The DSP 116 performs
`image processing on the raw CCD data to convert the frame
`of data into YCC color format, typically YCC 2:2:2 format
`(YCC is an abbreviation for Luminance, Chrominance-red
`and Chrominance-blue). Alternatively, the data could be
`converted into RGB format (Red, Green, Blue).
`After the still image has been processed, the image is
`compressed, typically in JPEG format, and stored as an
`image file on the mass storage device 122. A JPEG engine
`(not shown) for compressing and decompressing the still
`images may be provided in the image processing DSP 116,
`the video codec 132, provided as a separate, unit, or per(cid:173)
`formed in software by the CPU 124.
`After the image has been compressed and stored, live
`view resumes to allow the capture of another image. The
`user may continue to either capture still images, capture
`video, or switch to play or review mode to playback and
`view the previously stored video and images on the display
`screen 140. In a preferred embodiment, the DVC 100 is
`capable of capturing several different media types, as shown
`in FIG. 3.
`FIG. 3 is a table listing example media types that may be
`captured and stored by the DVC 100. Also shown are the
`corresponding icons that are used to indicate to the media
`type. The media types include a single still image, a time
`lapse or burst image, a panorama, a video segment, an audio
`clip, and a text file.
`A still image is a high-quality, single image that may have
`a resolution of 1536x1024 pixels, for example. A time-lapse
`image is a series of images automatically captured by the
`DVC 100 at predefined time intervals for a defined duration
`(e.g. capturing a picture every five minutes for an hour). A
`50 burst image is similar to a time-lapse, but instead of cap(cid:173)
`turing images for defined period of time, the DVC 100
`captures as many images as possible in a brief time frame
`(e.g., a couple seconds). A panorama image is an image
`comprising several overlapping images of a larger scene that
`55 have been stitched together. A burst image, a timelapse
`image, and a panorama image are each objects that include
`multiple still images, therefore, they may be referred to as a
`sequential images.
`In addition to capturing different image-based media
`60 types, the DVC 100 can capture other media types, such as
`audio clips and text. The user can record a voice message to
`create a stand-alone audio clip, or the user may record a
`voice message and have it attached to an image to annotate
`the image. Audio clips may also be downloaded from an
`65 external source to add sound tracks to the captured objects.
`A text media type is created by entering letters through the
`buttons on the user interface. The text along with graphics
`
`30
`
`35
`
`RIM v. MMI
`IPR2013-00016
`Exhibit 1011, p. 17
`
`

`

`US 6,738,075 Bl
`
`7
`can be overlaid as watermarks on the images or, the text can
`be saved in a file to create a text-based media type.
`In a preferred embodiment, one or more of the different
`media types can be combined to form a single media object.
`Since various combinations may be formed, such as single 5
`image with sound, or burst image with text, etc, the DVC
`100 can be described at storing heterogeneous media
`objects, each comprising a particular combination of media
`types, such as images, video, sound, and texvgraphics. Some
`types of media objects are formed automatically by the DVC 10
`100, such as a captured image or an annotated image, others
`are formed manually by the user.
`After media objects are created and stored, the user may
`view the media objects by switching the camera to play
`mode or review mode. In play mode, the camera 100 allows
`the user to view screen-sized images in the display screen
`140 in the orientation that the image was captured. Play
`mode also allows the user to hear recorded sound associated
`with a displayed image, and to play back sequential groups
`of images (time lapse, burst, and panorama images) and to
`view movies from the video.
`In review mode, the DVC 100 enables the user to rapidly
`review the contents of the DVe. In addition, the media
`objects may be edited, sorted, printed, and transferred to an
`external source.
`Referring now to FIG. 4A, a diagram illustrating one
`preferred embodiment of the review mode screen is shown.
`Moving the mode dial 202 (FIG. 2) to access the review
`mode enables the user to view all the media objects in the 30
`camera along with the specific media types associated with
`each of the objects.
`The first embodiment of the review mode screen displays
`a series of object cells 300 that represent the media objects
`stored on the DVC 100, and a command bar 310. The 35
`display screen 140 is shown here as displaying nine object
`cells 300, although other numbers are also suitable.
`The user may navigate through a series of displayed
`object cells 300 in the display screen 140 using the four-way
`navigation control 200. The object cell 300 currently 40
`selected by the four-way navigation control 200 is indicated
`by a highlighted area 302, which in this embodiment is
`shown as selection rectangle. Other shapes or indications
`that a object cell 300 is the currently active object cell are
`also suitable.
`Each object cell 300 includes an image area 304 and an
`icon/information area 306. In the case of a still image, the
`image area 304 of a object cell 300 displays a thumb