US006215523B1
`(10) Patent No.
`a2) United States Patent
`US 6,215,523 B1
`Anderson
`(45) Date of Patent:
`*Apr. 10, 2001
`
`
`5,493,335
`(54) METHOD AND SYSTEM FOR
`2/1996 Parulski et al... 348/233
`
`5,521,717
`ACCELERATING A USER INTERFACE OF
`5/1996) Maeda oo... cccccceseseeeecees 358/426
`
`
`
`AN IMAGE CAPTURE UNIT DURING 6/1996 Watanabe 0...eeeecto 348/2315,528,293
`11/1996 Roberts et al.
`eee 348/207
`5,576,757
`
`1/1998 Schelling et al.
`ww 358/296
`REVIEW MODE
`5,706,097
`
`we 395/349
`1/1998 Dwyeret al.
`5,706,457 *
`
`(75)
`Inventor: Eric C. Anderson, San Jose, CA (US)
`.. 395/615
`5,724,579
`3/1998 Suzuki
`.....
`
`
`76658
`(73) Assignee: FlashPoint Technology, Inc., San Jose,
`oito08 aga
`=A
`
`5,796,428 *
`8/1998 Matsumoto.....
`.. 348/231
`CA (US)
`
`5,861,918 *
`6/1999 Anderson et al.
`cu. 348/231
`
`.. 345/328
`5,966,122 * 10/1999 Itoh......
`
`
`5,999,173 * 12/1999 Ubillus oe
`eceeesseeeeseeeeee 345/328
`8/2000 Andersonet al... 348/233
`6,097,431 *
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`US.C. 154(b) by 0 days.
`
`(*) Notice:
`
`This patent is subject to a terminal dis-
`claimer.
`(21) Appl. No.: 08/872,587
`
`cited by examiner
`Primary Examiner—Wendy R. Garber
`Assistant Examiner—Alicia M. Harrington
`(74) Attorney, Agent, or Firm—Sawyer Law Group LLP
`
`Int. C1? one HO4N 5/222; B25G 3/18
`(S51)
`(52) US. C1. ceeeesessscsescseesneeneeenes 348/333.05; 345/328
`(58) Field of Search 0... 348/231, 333,
`348/334, 552, 333.01, 333.02, 333.05, 345/353,
`328
`
`(56)
`
`.
`References Cited
`U.S. PATENT DOCUMENTS
`
`5,016,107
`5/1991 Sasson et al... 358/209
`....
`++ 358/209
`eer : B904 pacha et al.
`
`
`..
`395/139
`ae
`erlin et al.
`we 395/157
`5,345,552
`9/1994 Brown ou...
`
`«. 395/162
`5,414,811
`5/1995 Parulski et al.
`...
`7/1995 Heilveil et al. wc 395/166
`5,434,969
`
`The present invention provides a method and system for
`accelerating a user interface of an image capture unit. The
`method and system comprises providing a plurality of
`thumbnail images on the display, each of the plurality of
`thumbnail images being associated with a captured image.
`The method and system also include selecting one of the
`plurality of thumbnail images to be displayed as a resized
`thumbnail imageonthedisplay, the resized thumbnail image
`being a larger version of the selected one of the plurality of
`thumbnail images. Through the present invention the user
`interface allowsa user to quickly review imagesin an image
`capture unit such as a digital camera orthe like.
`
`26 Claims, 17 Drawing Sheets
`
`
`
`
`
`
`402
`
`702
`
`704
`
`409
`
`410a
`
`410b
`
`706
`
`708
`
`Image #314
`12/4/96
`10:26 AM
`
`1
`
`SAMSUNG 1031
`
`SAMSUNG 1031
`
`1
`
`

`

`U.S. Patent
`
`Apr.10, 2001
`
`Sheet 1 of 17
`
`US 6,215,523 B1
`
`110
`
`ne Imaging
`
`112
`
`4Tame
`
`
`Cc
`
`t
`
`FIG. 1
`
`2
`
`

`

`U.S. Patent
`
`Apr. 10, 2001
`
`Sheet 2 of 17
`
`US 6,215,523 BI
`
`
`
`
`
`Joyiysua]C)
`
`
`
`
`IO\WAUOD[pusIS
`“esis_|
`
`SUIUIL
`
`JOyeIIUIH
`
`¢OW
`
`aseull
`
`1OSUas
`
`qed
`
`d0RJ1OqUespna/vsopeuywegjeandoS|
`
`SIOIO|
`
`VOC
`
`3
`
`
`
`
`

`

`U.S. Patent
`
`Apr. 10, 2001
`
`Sheet 3 of 17
`
`US 6,215,523 BI
`
`SUISPULTT
`
`JOTAIC]
`
`
`;dolsmris
`
`s[eIq2psuonng
`[aesaon||dO’807
`=r
`_—<
`
`souionegdnyoeg
`solianegUr
`daLAdNOO
`
`907
`
`vOV
`
`€ON
`
`JIOMOdg
`
`Joseuryy]
`
`IOMOdg
`
`Ajddns
`
`OSE
`
`4
`
`
`
`
`
`

`

`U.S. Patent
`
`Apr.10, 2001
`
`Sheet 4 of 17
`
`US 6,215,523 B1
`
`346
`
`538 +
`
`WORKING MEMORY
`
`530
`
`Frame Buffers
`
`InputBufferd
`
`InputBufferN
`
`RAM DISK
`
`InputBuffer2-
`
`532 SYSTEM AREA
`
`534
`
`FIG. 4A
`
`5
`
`

`

`U.S. Patent
`
`Apr.10, 2001
`
`Sheet 5 of 17
`
`US 6,215,523 B1
`
`
`
`FIG. 4B
`
`6
`
`

`

`U.S. Patent
`
`Apr. 10, 2001
`
`Sheet 6 of 17
`
`US 6,215,523 BI
`
`VIA
`q9¢cr eOCL
`
`ds
`
`7
`
`

`

`U.S. Patent
`
`Apr.10, 2001
`
`Sheet 7 of 17
`
`US 6,215,523 B1
`
`600
`
`602
`
`604
`
`
`
`Compressed
`Image Data
`
`
`
`
`Thumbnail
`(uncompressed)
`
`
`
`Screennail
`(optionally compressed)
`
`
`
`
`Image Tags
`
`FIG. 6
`
`
`
`606
`
`608
`
`610
`
`8
`
`

`

`U.S. Patent
`
`Apr. 10, 2001
`
`Sheet 8 of 17
`
`US 6,215,523 BI
`
`Jreuquing,|Zz7pODAUOISIOAUOD
`
`|euuse1D§|passardtog|FPUorssoAUOD
`seul]SUIZISAY2°
`
`
`CCCOOA|=surelyZZTOOA|MIAaATT|ademwey|dul|osemymey|eseUy
`909919
`809vI9
`cOv9CS
`
`CCPOOAUOISsaIdulo7
`/PTTaeeee-47Teo“”
`
`
`
`“Wayaseul|}wTIZIuorssaiduios||sIajoodssiajoods®SuIsssd0I1g||!“UOT
`
`uolelouaLHeyedsiojjngvedIAI
`cl9Besvit
`
`LO/H
`
`IIQRAOUWIOY
`
`AJOUINIA
`
`QJ’
`
`sroyjng
`
`-819
`
`oO
`
`9
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Apr.10, 2001
`
`Sheet 9 of 17
`
`US 6,215,523 B1
`
`)
`
`
`
`OC
`
`rQ
`an
`
`
`
`=<
`
`wt &
`wn Se
`“He
`
`©a
`
`&
`Ge x
`<i
`
`s
`
`22
`
`_
`
`409
`
`402
`
`410b
`
`410a
`
`10
`
`

`

`U.S. Patent
`
`Apr.10, 2001
`
`Sheet 10 of 17
`
`US 6,215,523 B1
`
`
`
`Invoke Review Mode
`
`For each new Thumbnail Imageto be Displayed,
`Fetch the Thumbnail Image from the
`Corresponding Image Data File
`
`Crop each Thumbnail Image around Center
`to Generate Small Thumbnails
`and Write into Draw Buffer
`
`720
`
`722
`
`724
`
`726
`
`728
`
`730
`
`For the Currently Selected Image, Resizeits
`Thumbnail Image to Generate a Larger
`Resized Thumbnail and Write into Draw Buffer
`
`Write Additional Image Information
`Corresponding to Selected Image
`into the Draw Buffer
`
`Copy the Draw Buffer into
`Display Buffer and Display
`
`732
`
`9
`
`
`
`Image
`
`Scrolling
`
`FIG. 9
`
`11
`
`11
`
`

`

`U.S. Patent
`
`Apr.10, 2001
`
`Sheet 11 of 17
`
`US 6,215,523 B1
`
`FIG.10
`
`
`410b
`
`410a
`
`:
`
`09
`-
`
`e—S
`
`12
`
`

`

`U.S. Patent
`
`Apr.10, 2001
`
`Sheet 12 of 17
`
`US 6,215,523 B1
`
`Invoke Play Mode
`
`Fetch Screennail Image
`Corresponding to Selected Image
`
`Decompress Current Screennail
`Image and Display
`
`806
`
`
`
`Navigation
`Decompress Next
`Button Depressed
`
`Screennail Image
`
`
`or Held Down
`and Display
`
`?
`
`Fetch Compressed Image Corresponding
`to Selected Image and Decompress
`
`Replace Screennail Image with
`Decompressed Compressed Image
`
`800
`
`802
`
`804
`
`810
`
`812
`
`
`
`
`
`
`
`814
`
`
`
`
`
`
`
`Navigation
`Interrupt Current
`
`Process if not yet
`Button Depressed
`9
`Complete
`
`
`
`
`816
`
`FIG. LIA
`
`13
`
`13
`
`

`

`U.S. Patent
`
`Apr.10, 2001
`
`Sheet 13 of 17
`
`US 6,215,523 B1
`
`402
`
`402
`
`FIG. 11B
`
`402
`
`FIG. 11C
`
`FIG. 11D
`
`14
`
`

`

`U.S. Patent
`
`Apr.10, 2001
`
`Sheet 14 of 17
`
`US 6,215,523 B1
`
`346
`
`WORKING MEMORY
`
`530
`
`532 SYSTEM AREA
`
`~
`Frame Buffers
`536 4
`( Speculation Buffer 1
`Speculation Buffer 2
`Speculation Buffer 3
`
`-a
`
`Speculation Buffer N
`
`RAM DISK
`
`534
`
`15
`
`

`

`U.S. Patent
`
`Apr.10, 2001
`
`Sheet 15 of 17
`
`US 6,215,523 B1
`
`Review or Play Mode Invoked
`
`900
`
`Get Numberof Available Speculation Buffers
`
`902
`
`Assign Buffer Organization around
`Currently Selected Image
`
`Get (N) of Currently Selected Image
`
`904
`
`906
`
`910
`
`Free Unneeded Buffer
`
`
`
`Y_<
`
`
`908 S.N.
`
`Buffer Update
`Required
`
`9
`
`
`
`
`Image Data
`Speculative
`Buffer Update
`Buffer Free
`
`
`
`Required
`
`?
`
`?
`
`
`
`User's
`
`Scrolling Method
`Get Buffer
`Decompress
`’ Changed
`
`
`
`and Resize
`9
`914
`
`
`
`
`719|Compressed Decompress Screennail
`
`
`Image
`Dati
`
`“wee
`for the Image
`
`Needing Update
`
`
`<
`918
`
`eview
`or Play Mode
`Y
`Changed
`
`?
`FIG. 13
`
`
`
`N
`
`¥
`
`
`922
`
`Numberof
`Available Buffers
`
`Changed
`?
`
`916
`
`920
`
`End
`
`N
`
`16
`
`16
`
`

`

`U.S. Patent
`
`Apr.10, 2001
`
`Sheet 16 of 17
`
`US 6,215,523 B1
`
`2
`
`3
`
`v
`
` ]
`
`4
`
`5
`
`M
`
`Ne
`
`FIG. 14A
`
`5
`
`3
`
`1
`
`2
`
`4
`
`\
`
`\
`
`FIG. 14B
`
`Dy
`
`J
`
`17
`
`

`

`U.S. Patent
`
`Apr.10, 2001
`
`Sheet 17 of 17
`
`US 6,215,523 B1
`
`Invoked Play Mode
`
`Fetch Screennail Image Corresponding to Selected Image
`
`Decompress Current Screennail Image and Display
`
`Invoke Speculative Decompression
`
`930
`
`932
`
`934
`
`936
`
`
`
`N
`
`
`
`
`
`
`Decompress
`
`
`Navigation
`next
`Screennail
`Button Depressed
`
`Image and
`or Held Down
`
`
`Display
`?
`
`
`940
`
`
` Decom.
`
`
`High-Res Image
`Available
`?
`
`
`
`
`
`Copy Decom.
`
`
`Decom.
`High-Res Image
`
`
`
`Image
`
`
`Available
`?
` Fetch Compressed Image Corresponding
`to Selected Image and Decompress
`
`Replace Screennail Image with
`Decompressed Compressed Image
`
` 950
`
`
`
`
`
`Interrupt Current
`Navigation
`Process if not
`Button
`
`
`yet Complete
`Depressed
`
`
`
`18
`
`948
`
`FIG. 15
`
`18
`
`

`

`US 6,215,523 B1
`
`1
`METHOD AND SYSTEM FOR
`ACCELERATING A USER INTERFACE OF
`AN IMAGE CAPTURE UNIT DURING
`REVIEW MODE
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`invention is related to the following
`The present
`co-pending U.S. patent applications Ser. No. 08/872,651
`entitled “A Method And System For Generating An
`Enhanced Image File In An Image Capture Unit;” Ser. No.
`08/872,578 entitled “A Method And System For Accelerat-
`ing A User Interface Of An Image Capture Unit During Play
`Mode;” and Ser. No. 08/872,588 entitled “A Method And
`System For Speculative Decompression of Compressed
`Image Data In An Image Capture Unit,” which are assigned
`to the assignee of the present application and filed on the
`same date as the present application.
`
`FIELD OF THE INVENTION
`
`The present invention relates generally to an image cap-
`ture unit and more particularly to a method and system for
`accelerating a user interface in such a unit.
`
`BACKGROUND OF THE INVENTION
`
`Modern digital cameras for taking pictures of scenes and
`the like typically include an imaging device which is con-
`trolled by a computer running a single threaded process.
`Whenan image is captured, the imaging device is exposed
`to light and generates raw image data representing the
`image. The raw image data is typically stored in a single
`image buffer where it is then processed and compressed by
`the processor. Many types of compression schemesare used
`to compress the image data, with the joint photographic
`expert group (JPEG) standard being the most popular. After
`the processor processes and compresses the raw image data
`into JPEG imagefiles, the processor stores the JPEG image
`files into an internal memory or on an external memorycard.
`Some digital cameras are also equipped with a liquid-
`crystal display (LCD)or other type of display screen on the
`back of the camera. Through the use of the LCD,
`the
`processor can cause digital camera to operate in one of two
`modes,record and play, although some cameras only have a
`record mode. In record mode, the LCD is used as a view-
`finder in which the user may view an object or scene before
`taking a picture. In play mode, the LCD is used a playback
`screen for allowing the user to review previously captured
`images either individually or in arrays of four, nine, or
`sixteen images.
`Besides the LCD, digital camera user interfaces also
`include a number of buttons or switches for setting the
`camera into one of the two modes and for navigating
`between images in play mode. For example, most digital
`cameras include two buttons labeled “—” and “+” that enable
`
`a user to navigate or scroll through captured images. For
`example, if the user is reviewing images individually, mean-
`ing that single images are displayed full-sized in the LCD,
`pressing one of navigation buttons causes the currently
`displayed image to be replaced by the next image.
`To display a captured image in play mode, the processor
`mustfirst access the JPEG imagefile corresponding to the
`captured image from memory, decompress the image data
`sixteen horizontal lines at time, and then send the decom-
`pressed data to the LCD. When the user presses the navi-
`gation button to see the next image, the user sees the next
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`image slowly replace the previously displayed image from
`top to bottom. Due to the amountof processing involved,the
`display of the entire image may take several seconds in some
`cameras. Not only is the image decompression time slow,
`but conventional digital cameras also do not allow other
`camera operations while the JPEG image is being
`decompressed, which meansthat the user cannot abort the
`decompression process. Thus, if the user decides halfway
`through reviewing a current image that another image is
`preferred, the camera will not recognize the next action until
`the current image is fully displayed.
`Accordingly, a user interface of this type appears to the
`user as being slow and non-responsive when attempting to
`access multiple images on a digital camera. Speed of access
`to these images has becomeincreasingly important as these
`types of cameras have become more widely available.
`There is a need, therefore, to provide images on a display
`device which allows the user to review multiple captured
`images, while simultaneously providing a display and a
`readout for a particular image in an efficient and straight-
`forward manner. It
`is similarly important
`to be able to
`rapidly identify a recognizable representation of the image.
`Finally, the system and method should be more responsive
`to the user than previously known systems. For example, in
`a plurality of captured images, it would be useful to quickly
`identify two or more particular images quickly, with a
`minimum of effort. It is also important to provide more
`efficient ways to quickly navigate througha series of images.
`The system should be implementable in a simple and cost
`effective fashion and should be easily handled by a user. The
`present invention addresses such a need.
`SUMMARYOF THE INVENTION
`
`The present invention provides a method and system for
`accelerating a user interface of an image capture unit. The
`method and system comprises providing a plurality of
`thumbnail images on the display, each of the plurality of
`thumbnail images being associated with a captured image.
`The method and system also include selecting one of the
`plurality of thumbnail images to be displayed as a resized
`thumbnail imageon the display, the resized thumbnail image
`being a larger version of the selected one of the plurality of
`thumbnail images. Through the present invention the user
`interface allows a user to quickly review imagesin an image
`capture unit such as a digital camera orthe like.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a block diagram of a digital camera that operates
`in accordance with the present invention.
`FIG. 2 is a block diagram of one preferred embodiment
`for the imaging device of FIG. 1.
`FIG. 3 is a block diagram of one preferred embodiment
`for the computer of FIG. 1.
`FIG. 4A is a memory map showingthe preferred embodi-
`ment of the Dynamic Random-Access-Memory (DRAM).
`FIG. 4B is a block diagram illustrating the contents of one
`of the input buffers and the contents of the frame buffer.
`FIGS. 5A and 5B are diagrams depicting the back and top
`view, respectively, of digital camera.
`FIG. 6 is a block diagram illustrating an enhanced format
`of still image file in accordance with the present invention.
`FIG. 7 is a block diagram illustrating the image file
`generation process, which begins when the camera is in
`capture mode and the user presses the shutter button to
`capture an image.
`19
`
`19
`
`

`

`US 6,215,523 B1
`
`3
`FIG. 8 is a diagram illustrating the operation and appear-
`ance of the accelerated user interface during review mode in
`accordance with a preferred embodiment of the present
`invention.
`
`FIG. 9 is a flow chartillustrating the process of acceler-
`ating the user interface of the digital camera when in review
`mode.
`
`FIG. 10 is a diagram illustrating the operation and appear-
`ance of the accelerated user interface during play mode in
`accordance with a preferred embodiment of the present
`invention.
`
`illustrating the process of
`FIG. 11A is a flow chart
`accelerating the user interface of the digital camera when in
`play mode.
`FIGS. 11B, 11C, and 11D illustrate an example of a
`screennail displayed on an LCD screen and then updated
`with a higher-resolution image as the higher-resolution
`image is decompressed in accordance with the present
`invention.
`
`FIG. 12 is a memory map of the DRAM illustrating the
`reallocation of the input buffers as speculation buffers in
`accordance with the present invention.
`FIG. 13 is flow chart depicting the speculative decom-
`pression process in a preferred embodimentof the present
`invention.
`
`FIGS. 14A and 14B are block diagramsillustrating the
`buffer organization that is assigned to images in responseto
`automatic and manual scrolling methods, respectively.
`FIG. 15 is a flow chart illustrating the process of accel-
`erating the user interface of play mode using the speculative
`decompression process.
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`The present invention relates to a method and system for
`accelerating the review and navigation through a series of
`images on an image capture unit. The following description
`is presented to enable one of ordinary skill in the art to make
`and use the invention and is provided in the context of a
`patent application and its requirements. Althoughthe present
`invention will be described in the context of a digital
`camera, various modifications to the preferred embodiment
`will be readily apparent to those skilled in the art and the
`generic principles herein may be applied to other embodi-
`ments. That is, any image capture device which displays
`images,
`icons and/or other items, could incorporate the
`features described hereinbelow and that device would be
`
`within the spirit and scope of the present invention. Thus,the
`present inventionis not intendedto be limited to the embodi-
`ment shownbutis to be accorded the widest scope consistent
`with the principles and features described herein.
`The present invention is a method and system for accel-
`erating a graphical user interface of an image capture unit
`using expanded imagefiles that allow for the rapid display
`of captured images.
`Referring now to FIG. 1, a block diagram of a digital
`camera 110 is shownfor use in accordance with the present
`invention. Camera 110 preferably comprises an imaging
`device 114, a system bus 116 and a computer 118. Imaging
`device 114 is optically coupled to an object 112 and elec-
`trically coupled via system bus 116 to computer 118. Once
`a photographer has focused imaging device 114 on object
`112 and, using a capture button or some other means,
`instructed camera 110 to capture an image of object 112,
`computer 118 commands imaging device 114 via system bus
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`116 to capture raw image data representing object 112. The
`captured raw imagedata is transferred over system bus 116
`to computer 118 which performs various image processing
`functions on the image data before storing it in its internal
`memory. System bus 116 also passes various status and
`control signals between imaging device 114 and computer
`118.
`
`Referring now to FIG. 2, a block diagram of one preferred
`embodiment of imaging device 114 is shown. Imaging
`device 114 typically comprises a lens 220 having an iris, a
`filter 222, an image sensor 224, a timing generator 226, an
`analog signal processor (ASP) 228, an analog-to-digital
`(A/D) converter 230, an interface 232, and one or more
`motors 234.
`
`In operation, imaging device 114 captures an image of
`object 112 via reflected light impacting image sensor 224
`along optical path 236. Image sensor 224, which is typically
`a charged coupled device (CCD), responsively generates a
`set of raw image data in CCD format representing the
`captured image 112. The raw image data is then routed
`through ASP 228, A/D converter 230 and interface 232.
`Interface 232 has outputs for controlling ASP 228, motors
`234 and timing generator 226. From interface 232, the raw
`image data passes over system bus 116 to computer 118.
`Referring now to FIG. 3, a block diagram of one preferred
`embodiment for computer 118 is shown. System bus 116
`provides connection paths between imaging device 114, an
`optional power manager 342, central processing unit (CPU)
`344, dynamic random-access memory (DRAM) 346,input/
`output interface (I/O) 348, non-volatile memory 350, and
`buffers/connector 352. Removable memory 354 connects to
`system bus 116 via buffers/connector 352. Alternately, cam-
`era 110 may be implemented without removable memory
`354 or buffers/connector 352.
`Power manager 342 communicates via line 366 with
`power supply 356 and coordinates power management
`operations for camera 110. CPU 344 typically includes a
`conventional processor device for controlling the operation
`of camera 110. In the preferred embodiment, CPU 344 is
`capable of concurrently running multiple software routines
`to control the various processes of camera 110 within a
`multithreaded environment. DRAM 346 is a contiguous
`block of dynamic memory which may beselectively allo-
`cated to various storage functions. LCD controller 390
`accesses DRAM 346 and transfers processed imagedata to
`LCDscreen 402 for display.
`1/O 348 is an interface device allowing communications
`to and from computer 118. For example, I/O 348 permits an
`external host computer (not shown) to connect
`to and
`communicate with computer 118. I/O 348 also interfaces
`with a plurality of buttons and/or dials 404, and an optional
`status LCD 406, which in addition to the LCD screen 402,
`are the hardware elements of the camera’s user interface
`408.
`
`Non-volatile memory 350, which may typically comprise
`a conventional read-only memory or flash memory, stores a
`set of computer-readable program instructions to control the
`operation of camera 110. Removable memory 354 serves as
`an additional image data storage area and is preferably a
`non-volatile device, readily removable and replaceable by a
`camera 110 user via buffers/connector 352. Thus, a user who
`possesses several removable memories 354 may replace a
`full removable memory 354 with an empty removable
`memory 354to effectively expand the picture-taking capac-
`ity of camera 110. In the preferred embodiment of the
`present
`invention,
`removable memory 354 is typically
`20
`
`20
`
`

`

`US 6,215,523 B1
`
`5
`implemented using a flash disk. Power supply 356 supplies
`operating power to the various components of camera 110.
`In the preferred embodiment, power supply 356 provides
`operating power to a main power bus 362 and also to a
`secondary powerbus 364. The main power bus 362 provides
`powerto imaging device 114, I/O 348, non-volatile memory
`350 and removable memory 354. The secondary power bus
`364 provides power to power manager 342, CPU 344 and
`DRAM 346.
`
`Powersupply 356 is connected to main batteries 358 and
`also to backup batteries 360. In the preferred embodiment,
`a camera 110 user may also connect power supply 356 to an
`external power source. During normal operation of power
`supply 356, the main batteries 358 provide operating power
`to power supply 356 which then provides the operating
`power to camera 110 via both main power bus 362 and
`secondary power bus 364. During a power failure mode in
`which the main batteries 358 have failed (when their output
`voltage has fallen below a minimum operational voltage
`level) the backup batteries 360 provide operating power to
`power supply 356 which then provides the operating power
`only to the secondary power bus 364 of camera 110.
`Referring now to FIG. 4A, a memory map showing one
`embodiment of dynamic random-access-memory (DRAM)
`346 is shown. In the preferred embodiment, DRAM 346
`includes RAM disk 532, a system area 534, and working
`memory 530.
`RAM disk 532 is a memoryarea used for storing raw and
`compressed image data and typically is organized in a
`“sectored” format similar to that of conventional hard disk
`
`drives. In the preferred embodiment, RAM disk 532 uses a
`well-known and standardized file system to permit external
`host computer systems,via I/O 348, to readily recognize and
`access the data stored on RAM disk 532. System area 534
`typically stores data regarding system errors (for example,
`why a system shutdown occurred) for use by CPU 344 upon
`a restart of computer 118.
`Working memory 530 includes various stacks, data struc-
`tures and variables used by CPU 344 while executing the
`software routines used within computer 118. Working
`memory 530 also includes several input buffers 538 for
`temporarily storing sets of raw image data received from
`imaging device 114, and a frame buffer 536 for storing data
`for display on the LCD screen 402.
`In a preferred
`embodiment, each input buffer 538 and the frame buffer 536
`are split into two separate buffers, called ping-pong buffers
`(shownby the dashedlines), to improve the display speed of
`the digital camera and to prevent the tearing of the image in
`the display 402.
`Referring now to FIG. 4B, the contents of one of the input
`buffers 538 and the contents of the frame buffer 536 are
`illustrated. As shown, each input buffer 538 includes an
`input buffer A and an input buffer B, and the frame buffer
`536 includes a frame buffer A and a frame buffer B. The
`input buffers A and B alternate between an input cycle and
`a processing cycle. During the input cycle, the input buffers
`538 are filled with raw image data from the image device
`114, and during the processing cycle, CPU 344 processes the
`raw data and transmits the processed data to the frame
`buffers 536.
`
`Referring now to FIG. 7, in a preferred embodiment, the
`processing is performed by a live view generation process
`612, which is stored in non-volatile memory 350 and
`executed on CPU 344. However, the image processing can
`also be implemented using hardware. During the execution
`of the live view generation process 612, the CPU 344 takes
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`the raw image data from the input buffers 538, typically in
`CCD format, and performs color space conversion on the
`data. The conversion process performs gamma correction
`and converts the raw CCD data into either a RGB or YCC
`color format which is compatible with the LCD screen 402.
`(RGBis an abbreviation for Red, Green, Blue, and YCCis
`an abbreviation for Luminance, Chrominance-red and
`Chrominance-blue). After converting the data to YCC, the
`YCCimagedata is stored in the frame buffer 536. The LCD
`controller 390 then transfers the processed image data from
`the frame buffers to the LCD screen 402 for display.
`The resolution of the LCD screen 402 may vary; however,
`the LCD screen resolution is usually much less than the
`resolution of the image data that’s produced by imaging
`device 114 when the user captures an imageat full resolu-
`tion. Typically,
`the resolution of LCD is % the video
`resolution of a full resolution image. The size of the input
`buffers 538 may also vary, but in a preferred embodiment,
`two of the input buffers 538 are required to contain a full
`resolution image. One input buffer 538 can therefore contain
`one image captured at % resolution. Since the LCD is
`capable of displaying images at % resolution, the images
`generated during the live view process are also % resolution
`and are therefore be stored in one-half, or in one of the
`ping-pong buffers of an input buffer 538.
`Referring again to FIG. 4B, the ping-pong buffers are
`utilized during live view mode as follows. While input
`buffer A is filled with image data, the data from input buffer
`B is processed and transmitted to frame buffer B. At the
`same time, previously processed data in frame buffer A is
`output to the LCD screen 402 for display. While input buffer
`B is filled with image data, the data from input buffer A is
`processed and transmitted to frame buffer A. At the same
`time, previously processed data in frame buffer B is output
`to the LCD screen 402 for display.
`FIGS. 5A and 5B are diagrams depicting the preferred
`hardware components of the camera’s 110 user interface
`408. FIG. 5A is back view of the camera 110 showing the
`LCDscreen 402, a four-way navigation control button 409,
`an overlay button 412, a menu button 414, and a set of
`programmable soft keys 416. FIG. 5B is a top view of the
`camera 110 showing a shutter button 418, and a mode dial
`420. The camera may optionally include status LCD 406,
`status LCD scroll and select buttons 422 and 424, a sound
`record button 426, and zoom-in, zoom-out buttons 426a and
`426b.
`
`Accordingly, the user interface 408 of the digital camera
`can be accelerated when accessing captured images through
`a combination of various features of the present invention.
`The features include multiple operating modes forthe digital
`camera, the use of multiple thumbnail images for supporting
`the rapid display of images by eliminating the need to
`decompress image data in certain operating modes, and
`speculative decompression of compressed image data for
`supporting the rapid display of full-sized images in other
`operating modes. The use of thumbnails and speculative
`decompression enables the images to be reviewed and
`navigated more rapidly via representations which can be
`displayed quickly. Through the present invention the inter-
`action rate and responsiveness to the user is increased
`significantly over conventional navigation systems.
`As stated above, in one aspect of the present invention, the
`digital camera is provided with several different operating
`modes for supporting various camera functions. Although
`the digital camera includes multiple operating mode, the
`modes relevant
`to this description are capture (record),
`21
`
`21
`
`

`

`US 6,215,523 B1
`
`7
`review, and play mode. In capture mode, the camera 100
`supports the actions of preparing to capture an image, and
`capturing an image throughthe use of either the LCD screen
`402 alone or the status LCD 406 with the aid of an optional
`optical viewfinder (not shown). In review mode, the camera
`100 supports the actions of reviewing camera contents,
`editing and sorting images, and printing and transferring
`images. In play mode, the camera 100 allows the user to
`view screen-sized images in the LCD screen 402 in the
`orientation that the image was captured. Play mode also
`allows the user to hear recorded sound associated to a
`displayed image, and to play back sequential groupings of
`images, which may comprise time lapse, slide show, and
`burst image images. The user preferably switches between
`the capture, review, and play modes, using the mode dial
`420. When the camerais placed into a particular mode, that
`mode’s default screen appears in the LCD screen 402 in
`which a set of mode-specific items, such as images, icons,
`and text, are displayed. Through the use of multiple oper-
`ating modes,
`the camera functions and features can be
`categorized, which allowsfor faster access to those features
`and functions than would be possible by nesting all the
`features in one play modeas in conventional digital cameras.
`Anotheraspect of the present invention that allows for the
`acceleration of the user interface 408 is the use of multiple
`thumbnails associated with each captured image, which are
`smaller, reduced-resolution versions of the higher-resolution
`compressed image data. In a preferred embodimentof the
`present invention, two types of thumbnail images are asso-
`ciated with each captured image and are included in the
`image’s datafile.
`Referring now to FIG. 6, a block diagram of an enhanced
`format of still image file in accordance with the present
`invention is shown. The image file 600 includes a header
`602, compressed image data 604, a thumbnail image 606, a
`screennail 608, and an imagetag field 610. The imagefile
`600 mayalso include a soundfile (not shown)if a sound clip
`has been attached to the particular image.
`The header 602 includes information identifying the par-
`ticular image file and the image represented by the image
`data 604. The image data 604 is the actual data comprising
`the full-sized captured image in compressed form, prefer-
`ably in JPEG format. Although the user can typically choose
`the resolution mode in which imagesare captured, once an
`image is processed and compressed, the compressed image
`data 604 is the high-resolution representation of the image
`compared to the thumbnail 606 and screennail 608. If the
`image is captured at a resolution of 640x480 pixels, for
`example, then the compressed image data 604 is typically
`fifty-to-sixty kilobytes in size.
`The thumbnail image 606 is a small, uncompressed low-
`resolution version of the image. In a preferred embodiment,
`the pixel size of thumbnail image 606 is less than the display
`size of the LCD screen 402 (e.g., 80x60 pixels), and has a
`storage size of approximately ten kilobytes.
`The screennail image 608 is a medium-resolution version
`of the image and in a preferred embodiment
`is also
`compressed, although compressing the screennail 608 is
`optional. Unlike the thumbnail image 606, the screennail
`image 608 is display-sized and fills the visible area of the
`LCD screen 402 when displayed.
`In a preferred
`embodiment,
`the pixel size of a compressed screennail
`image 608 is preferably 288x216 and requires approxi-
`mately fifteen kilobytes to store.
`The image tag field 610 includes information, preferably
`in the form of tags, regarding the image represented by the
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`image data 604. Media type tags, for instance, indicate all
`the media types associated with the image, such as whether
`the image is a single image or a panorama image, for
`example. In certain operating modes, the media type tags are
`used to select the type of icon that is displayed in the LCD
`402 along side the thumbnail image 606. Besides media
`tags, the imagetag field 610 may also include other types of
`tags for storing additional information regarding the image
`and/or the camera 110 itself. For example, a tag could be
`used to indicate the settings of the camera 110