Unlted States Patent
`
`[19]
`
`Sarbadhikari et al.
`
`[11] Patent Number:
`
`[45] Date of Patent:
`Dec. 19, 1995
`
`5,477,264
`
`lllliillllllllIllIllll|||||Illlllllll|||l||||l|||||||||||Illlllllllllllllll
`USOOS477264A
`
`[54] ELECTRONIC IMAGING SYSTEM USING A
`REMOVABLE SOFTWARE-ENHANCED
`STORAGE DEVICE
`
`[75]
`
`Inventors: Kama! K. Sarbadhikari, Penfield;
`John R. Fredlund; Kenneth A.
`ParuISki, both 0f Rochester, all of NY,
`
`[73] Assignee: Eastman Kodak Company, Rochester,
`N.Y.
`
`[21] Appl. No.: 219,608
`.
`[22] Flled:
`
`Mar- 29, 1994
`
`Int. Cl.6 ....................................................... H04N 5/76
`[51]
`
`[52] U.S. Cl. ............
`348/231; 348/233, 348/552
`[58] Field of Search ..................................... 348/222, 231,
`343/232, 233, 552; 358/9091, 906; 258/903,
`909; H04N 5/76,’5/907, 5/92
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`4,500,183
`2/1985 Tanikawa .................................. 354/21
`
`43524381
`6/1985 K0195“?
`358/29
`
`4,574,319
`3/1986 K131115111 .......
`.... 358/335
`4,728,978
`3/1988 Inoue et al.
`.....
`. 354/2891
`
`4,853,733
`..
`_______ 354/412
`8/1989 Watanabe et a1.
`4,855,779
`8/1989 Ishikawa et a1.
`354/412
`4,994,844
`2/1991 Azuma et al.
`......
`354/412
`5,016,107
`5/1991 Sasson et a1.
`353/209
`5,018,017
`5/1991 Sasaki et a1.
`358/209
`5,023,637
`6/1991 Lorton et al.
`354/106
`5,027,214
`5/1991 Fujimori
`---------
`-- 358/209
`
`5’032’855
`7/1991 Taniguchi 6‘ 31-
`----- 354/21
`
`””‘35345/‘1951?
`g'ggggfi 1313:; 11:11:13???
`
`' """""
`’
`’
`e
`'
`5,103,250
`4/1992 Arifuku etal.
`354/106
`.
`
`8/1992 Roberts ...............
`5,138,459
`348/552
`...................... 354/106
`5,142,310
`3/1992 Taniguchi et a1.
`
`
`
`INPUT SECTION (A)
`
`5,153,729 10/1992 Saito
`........... 358/209
`5,155,513
`10/1992 Matsumura et al.
`. 354/106
`
`5,226,145
`7/1993 Moronaga ..........
`.. 348/231
`5,262,868
`11/1993 Kaneko ................................... 348/233
`
`OTHER PUBLICATIONS
`
`USSN 988,517 filed Dec. 10, 1992, Entitled “Electronic
`Camera with Memory Card Interface to a Computer”, Ken-
`neth A. Parulski et a1.
`USSN 085,519 filed Jun. 30, 1993, Entitled “Apparatus and
`Method for Adaptively Interpolating a Full Color Image
`Utilizing Chrominance Gradients”, Claude A. Laroche et al.
`Personal Computer Memory Card International Association
`(PCMCIA) “PC Card Standard”, Release 2.0, Sep. 1991.
`
`Primary Examiner—James J_ Groody
`Assistant Examiner—Sherrie Hsia
`Attorney, Agent, or Finn—David M. Woods
`ABSTRACT
`57
`
`1
`[
`An electronic imaging system includes a digital electronic
`camera for capturing and storing images in a removable
`storage device, which is also preloaded with enhancement
`files for effecting the operation of the system. The camera
`includes an optical section for establishing the optical
`parameters of image capture, an image sensing section for
`electrically capturing the image, and a signal processing
`.
`.
`.
`.
`section for operatmg upon the electrrcally captured image
`prior to storage. The several sections of the camera are
`coordinated and controlled by a Programmable Processor,
`which is capable of receiving the enhancement files pre-
`loaded into the storage device. These files may contain
`software for updating the operating code of the camera, for
`modifying the electrically captured image in selected ways,
`for modifying camera in special situations, or for commu-
`nicating non—captured image—like data, such as text and
`image overlays, to the camera.
`,
`
`47 Claims, 11 Drawing Sheets
`
`COMPRESSION AND RECORDING
`SECTION (B)
`
`25
`
`
`r-—————-—-—-/ \————\
`/6 [- ——————
`
`l
`INAGE BUFFER l r/5
`22
`
`I
`.' 1/
`
`EXPOSURE
`°
`SECTION
`
`
`
`
`FLASH
`
`DIGITAL
`I
`SIGNAL
`PROCESSOR
`' —
`
`I
`
`i FHA/.45]I
`
`l
`
`]
`
`|
`
`30,,
`
`IIIIIIIIIIIIIIIlII
`
`28
`
`f]
`
`20
`
`
`l'_ _
`
`’9
`
`INPUTS
`
`FfifigfiE
`
`ALGORITHMS .ROCESSOR
`
`PROCESSING
`
`OPERATION
`DISPLAY
`
`
`
`-N$TRUCT/0NMEMORY
`
`3/
`
`N0 CARD. CARD FULL
`PERFORMANCE
`ENHANCEMENTS
`
`SPEC/AL EFFECTS
`WERLAY
`
`HTC Exhibit 1003
`
`HTC Exhibit 1003
`
`

`

`US. Patent
`
`Dec. 19, 1995
`
`Sheet 1 of 11
`
`5,477,264
`
`M$S§$$$R6tl
`
`tommmuétl
`
`mmfikkmom
`
`mélkk§x3<
`
`~kt<tekwt\6E
`
`HTC Exhibit 1003
`
`HTC Exhibit 1003
`
`
`
`

`

`US. Patent
`
`Dec. 19, 1995
`
`Sheet 2 of 11
`
`5,477,264
`
`358205353
`
`\Eokmx‘
`
`BN.b\1+
`
`
`
`gatoommSQSammESo...3205mmSm:
`
`====fi===n=__e_\mist_=========_u_
`#55A_mm_mmfibmmeg_rI|||.i/\i..||..|\_IIIIIIIl_9a»205mm
`mommmoomm_zotomm-35m.mmbmomxm
`
`
`SMEMQSIzm
`
`MEI
`
`35.605
`
`£158?.IL
`
`ON
`
`.53
`
`on‘«888$
`
`dBqmédmfio23&5Nzotémmo'-
`
`”5&39tmmb0m
`
`ESEEEQ
`
`BEEEEEHI
`
`mtmtmEommmlmm<§$mmm
`StESE:
`
`HTC Exhibit 1003
`
`HTC Exhibit 1003
`
`
`
`
`
`

`

`US. Patent
`
`Dec. 19, 1995
`
`Sheet 3 of 11
`
`5,477,264
`
`
`
`TURN ON
`ELECTRONIC
`
`CAMERA
`
`POI/ER UP CAMERA FUNCTIONS
`SUCH AS FLASH.VIEWFINDER.
`USER DISPLAY.CAPTURE
`ELECTRONICS. ETC.
`
`
`
`
`
`ALGORITHMS
`
`/ PRESENT IN \ N0
`REMOVABLE IMAGE
`DATA STORAGE ?
`
`NOT SUBJECT
`OF THIS
`INVENTION
`
`YES
`
`
`DOWNLOAD ALGORITHM
`
`(MAY BE USER SELECTED
`ONE OF MANY)
`
`
`
`
`
`_
`
`HTC Exhibit 1003
`
`EXPOSURE
`
`
`DES/RED BY
`
`
`
`USER ?
`
`
`
`
`YES
`
`CAPTURE
`
`IMAGE
`
`APPLY
`
`DOWNLOADED
`
`
`
`ALGORITHM
`
`
`
`
`STORE IMAGE
`ON REMOVABLE
`
`STORAGE MEANS
`
`FIG.3
`
`HTC Exhibit 1003
`
`

`

`US. Patent
`
`Dec. 19, 1995
`
`Sheet 4 of 11
`
`5,477,264
`
`mmdhm
`
`20.N9§S
`
`‘mm<m6km
`
`m2<m:
`
`mn®<>9¥Mt
`<k<QMG<§\---
`
`NV
`
`mmmoomm
`
`:tokDb
`
`Qmmdkm
`
`mw<:\
`
`v6E
`
`mmukth
`
`wtbkm<o
`
`mm¥<§
`
`AINI
`
`HTC Exhibit 1003
`
`HTC Exhibit 1003
`
`
`
`
`
`
`
`

`

`US. Patent
`
`Dec. 19, 1995
`
`Sheet 5 of 11
`
`5,477,264
`
`CAPTURE
`
`IMAGE
`
`DIG/TIZE
`
`STORE IMAGE DATA
`
`ON REMOVABLE
`
`STORAGE MEANS
`
`POSSIBLE TIME LAPSE
`(CAMERA NO LONGER
`IN 'READY' MODE)
`
`
`
`
`
`
`
`
`
`
`DOWNLOAD IMAGE DATA
`FROM REMOVABLE IMAGE
`
`DATA STORAGE MEANS
`
`DOWNLOAD PROCESSING
`
`ALGORITHMS FROM
`
`REMOVABLE IMAGE DATA
`
`STORAGE MEANS
`
`
`
`
`
`
`PROCESS
`
`IMAGE
`
`
`
`
`STORE PROCESSED
`IMAGE IN IMAGE
`DATA STORAGE MEANS
`
`FIG.5
`
`HTC Exhibit 1003
`
`HTC Exhibit 1003
`
`

`

`US. Patent
`
`Dec. 19, 1995
`
`Sheet 6 of 11
`
`5,477,264
`
`CAPTURE
`IMAGE
`
`
`
`
`
`BEGIN PROCESSING
`IMAGE ACCORDING TO
`DOWNLOADED
`ALGORITHMS
`
`
`NEW
`CAPTURE
`
`DESIRED ?
`
`
`
`
`
`
`
`
`
`INTERRUPT
`PROCESSING
`
`CONTINUE
`PROCESSING
`
`
`
`STORE PARTIALLY
`
`PROCESSED IMAGE IN
`
`
`PROCESSING
` COMPLETE ?
`FOR THIS IMAGE
`
`
`
`
`YES
`
`MANNER SUCH THAT
`PROCESSING MAY
`RESUME LATER
`
`WITHOUT PROBLEMS
`
`
`
`
`
`
`
`NO
`
`PROCESSING
`COMPLETE FOR
`ALL IMAGES
`7
`'
`
`
`
`CAPTURE
`NEW IMAGE
`
`..
`
`YES
`
`
`'READY' MODE
`
`RETURN
`
`TO
`
`V FIG. 6
`
`HTC Exhibit 1003
`
`HTC Exhibit 1003
`
`

`

`US. Patent
`
`Dec. 19, 1995
`
`Sheet 7 of 11
`
`5,477,264
`
`#:tozmmn
`
`@048
`
`zQBBOtQMt
`
`$8502mma
`
`estmmmootl
`
`thnioo0&3
`
`mm<Eo9
`
`hmustzm
`
`SSS
`
`mm»
`
`£38Qmuzvfzmmm:
`
`zotoaotlmt
`
`m3iht§
`
`45.362HMS
`
`QMMQM
`
`39om:
`
`Sufismobs
`
`NSEOM.
`
`hmostzm
`
`,3va
`
`my»
`
`MQE~6qu<No.05
`
`9th6Q
`
`.mhtékgm.
`
`Emhm
`
`20zméemace0k
`
`69.9...atthG
`
`«DQVGQN
`
`
`
`
`
`QMMlm$330Ichkm
`
`,mev.§EEK
`
`
`
`Weaktmmb
`
`mmmbkoi
`
`m2§w¢t<tmMSWmm:
`
`2938mmmm§<as?
`
`
`
`mStktolIOKQNNSEQQ
`
`MI.»has$53.
`
`mzottoozo:
`
`QQOQMt
`
`K6E
`
`HTC Exhibit 1003
`
`
`
`
`
`Q56”EmmfiEmma
`
`
`
`
`
`Qtfib”Emmitmm:
`
`HTC Exhibit 1003
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`US. Patent
`
`Dec. 19, 1995
`
`Sheet 8 of 11
`
`5,477,264
`
`VACANT PORTION
`
`OF 'PRE-EXPOSED'
`
`IMAGE
`
`d“ A“ Star 4“
`
`
`
`
`
`CAPTURED IMAGE
`
`'PRE-EXPOSED' IMAGE
`DATA FILE
`
`m AM Stw 4“
`
`COMBINED IMAGE
`
`F/G.8
`
`HTC Exhibit 1003
`
`HTC Exhibit 1003
`
`

`

`US. Patent
`
`Dec. 19, 1995
`
`Sheet 9 of 11
`
`5,477,264
`
`PUSH BUTTON TO CHOOSE
`THIS OVERLAY
`
`.
`
`SEL®ECT
`
`NEXT
`
`WILL GO HERE/
`
`/ '
`
`/ Y
`
`OUR IMAGE
`
`0N CAMERA DISPLAY
`AND/0R VIEWF/NDER
`
`PUSH BUTTON FOR NEXT
`OVERLAY POSITION
`
`F/G.9
`
`HTC Exhibit 1003
`
`HTC Exhibit 1003
`
`

`

`US. Patent
`
`D
`
`w
`
`mS
`
`1
`
`7
`
`%
`
`mgktm§\
`
`mm.mm
`_m1Emmgmmfifi
`
`Sign
`
`tbmmmuotm
`
`
`
`GSQEOQMQQ:205mmmn$oo
`
`«my>6:on
`
`3czcfiomm5&3
`
`
`
`
`mE
`
`mgimowé
`1,mémmmoomm
`
`1mofizmommmm
`
`mimfiufimzm
`
`MiiE655oz3&5uzotémmo
`
`M,EOE:
`
`3%5%“
`
`
`
`mkomntmQSQNQMxguikoflm
`
`4A
`
`7,a9wt
`
`\mmSnE
`
`,Ems
`
`mH
`
`\\
`
`HTC Exhibit 1003
`
`HTC Exhibit 1003
`
`
`
`
`
`

`

`US. Patent
`
`Dec. 19, 1995
`
`Sheet 11 of 11
`
`5,477,264
`
`mm
`
`Hgkfitwotl
`
`tommmobtl
`
`Fm.
`
`BESSIE?
`
`
`
`EDEN:mm<§
`
` 3
`
`
`
`mggtiEgg?“
`
`xt0§$<
`
`
`
`
`
`mxstbtoefix
`
`ngkom
`
`Q00
`
`/2
`
`M23
`
`HTC Exhibit 1003
`
`HTC Exhibit 1003
`
`
`
`

`

`1
`ELECTRONIC IMAGING SYSTEM USING A
`REMOVABLE SOFTWARE-ENHANCED
`STORAGE DEVICE
`
`FIELD OF THE INVENTION
`
`This invention pertains to the field of electronic imaging
`and, in particular, to electronic imaging with an electronic
`still camera that utilizes a removable storage device for
`storing images.
`
`BACKGROUND OF THE INVENTION
`
`As understood in the prior art, a digital electronic still
`camera is a device which uses an electronic sensor to capture
`an image; signal processing to modify the captured image
`signal, e.g., as a function of ambient light, and to represent
`it numerically; and some storage device to preserve the
`numerical image data. Ordinarily, the image storage device
`is merely a receptacle for the image data.
`It is further known for a digital electronic still camera to
`use a removable storage device, such as an integrated circuit
`memory card, to store images. For instance, U.S. Pat. No.
`5,016,107 describes an electronic still camera utilizing
`image compression and providing digital storage in a remov-
`able memory card having a static random access memory.
`Software that controls the camera operation is located in
`non—removable programmable read only memory (PROM)
`integrated circuits in the camera. In this camera, the inte-
`grated circuits in the removable memory card store image
`data and a directory locating the data.
`Other electronic camera systems described in the prior art
`record an image header along with the digital image data.
`The header typically describes some characteristics about
`the image or the camera. For instance, in U.S. Pat. No.
`5,018,017, such header information indicates the classifica—
`tion of the information; the imaging system in use; the date
`and time; the compression mode; as well as image-specific
`information, such as flash use, white balance data, exposure
`value, and shutter speed. Furthermore, US. Pat. No. 5,153,
`729 shows memory capacity, a free area pointer, a battery
`expiration date, and a write inhibit flag in the header area of
`a memory card. Typically, each system conveys such char-
`acteristic data from the electronic camera by way of the
`memory card (where the data is temporarily stored) to a
`peripheral device, such as a player. The object is to increase
`the downstream functionality of the system by allowing
`utilization of image-related data in subsequent processing of
`the image data in a downstream peripheral device. The
`camera or imaging device generally utilizes little of the
`stored image characteristic data, one exception being data
`regarding the utilization of the storage device itself, i.e., how
`much memory space is remaining for further pictures. In the
`latter case, the camera accesses such data to display utili-
`zation to the camera user.
`
`Coordination between a camera and a computer is gen-
`erally shown in patent application Ser. No. 988,517, “Elec-
`tronic Camera with Memory Card Interface to a Computer”,
`which was filed on Dec. 10, 1992 and assigned to common
`assignee with the present application. This application
`describes a camera that is specially shaped so as to plug
`directly into the memory card slot of a portable computer;
`the camera is then usable as a unit with the portable
`computer. The camera contains EPROM memory with code
`for in-camera white balance and gamma correction, and also
`stores the compiled code which is used (by the computer) to
`operate the camera and to process the images from the
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`5 ,477 ,264
`
`2
`
`sensor color filter array to obtain a full resolution, color
`corrected image. The image data is captured by the camera
`and directly downloaded to the computer, where it is pro-
`cessed with code obtained from the camera. As with prior
`memory card systems, the code is used downstream (in the
`computer) to process image data.
`In a camera intended for acquisition of image data for use
`in a computer, such as described in Ser. No. 988,517, it is
`well understood that data corresponding to captured images
`may also be stored in a storage device that is capable of
`being removed from the camera. Such a storage device is
`often considered analogous to film in a conventional film
`camera. Just as it is known to encode film with marks that
`are useful in downstream processing of the film, it is also
`known for an electronic image storage device to contain
`non—image data which is useful in modifying the image data.
`Following the film analogy, such additional data is useful in
`downstream processing, such as in a computer, to modify the
`image data. Unlike film, however, the additional non-image
`data may be read by a variety of peripheral devices, includ—
`ing the camera itself. This provides an opportunity to deal
`with a number of imaging problems.
`For instance, in most known electronic imaging systems,
`the software which controls the camera ordinarily cannot be
`updated without purchasing an entirely new camera, con—
`taining a lens, CCD image sensor, etc. (Albeit, a few high
`end film cameras utilize a removable program module from
`which the camera can download exposure algorithms.) It is
`well known that computer programs are updated more
`regularly than computer hardware, since the software can be
`refined to provide higher performance (i.e. better image
`quality) or more features while using the same hardware. An
`electronic camera can use a programmable digital processor
`controlled by software. Such a camera otfers an opportunity
`for software updates, if the system is designed to facilitate
`such updates. What is needed is a method for software
`updates of the camera algorithms for higher performance,
`and for adding new features to a camera. Such a method
`should be quick and convenient for the user, preferably
`without requiring the use of additional storage modules.
`Existing memory devices are capable of limited two-way
`communication with an electronic camera, such as to report
`to the camera on memory space availability. An opportunity
`exists for enhancing such “upstream” communication to
`provide altogether new applications for an electronic carn-
`era.
`
`SUMMARY OF THE INVENTION
`
`The opportunity in the prior art for upstream utilization of
`the memory capability of a removable storage device is
`embodied, according to the invention, by an electronic
`imaging system including an electronic camera for capturing
`and storing images in a removable storage device which is
`also preloaded with software for operating the imaging
`system. The imaging system includes an optical section for
`establishing the optical parameters of image capture, an
`image sensing section for electrically capturing an image
`provided by the optical section, a signal processing section
`for processing the electrically captured image, a program—
`mable processor for controlling the sections of the system,
`and means for loading the programmable processor with the
`software preloaded in the removable storage device for
`afiecting operation of one or more of the sections of the
`system.
`Consequently, the invention provides an electronic still
`photography system which can be controlled by software
`
`HTC Exhibit 1003
`
`HTC Exhibit 1003
`
`

`

`5,477,264
`
`3
`located on a removable memory (e.g., a memory card) used
`with an electronic still camera, Known cameras use the
`removable memory to store images and data related to an
`individual image (date, color temperature of the illuminant,
`etc.) or data related to the camera (type of color filter array,
`sensor defect locations, etc.) which is transferred to an
`image reproducing device. Except for data about memory
`space, the communication is one way. A camera according to
`this invention, however, uses the removable media to trans-
`fer software previously recorded on the memory card (such
`as improved camera firmware to process the image, or a
`special access code) to the camera. The advantageous effect
`is that the operation of the camera or playback device can be
`improved some years after the camera is purchased, by using
`an “enhanced” memory card containing improved software
`for operating the camera or playback device. Moreover, such
`two-way communication opens use of the storage device to
`other enhancement possibilities, e.g., to modify image data
`in selected ways, to modify camera performance for special
`situations, or to communicate non-captured images (special
`overlays) to the camera.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The prior art and the invention will be described in
`relation to the drawings, wherein
`FIG. 1 is a diagram of a known electronic imaging system
`incorporating an electronic camera for inputting images to a
`computer;
`FIG. 2 is a block diagram of an electronic camera con—
`figured according to the invention to receive and process
`enhancement files;
`
`FIG. 3 is a flow diagram of the steps involved in applying
`an enhancement algorithm to the camera of FIG. 2;
`FIG. 4 is a block diagram of the capture chain showing the
`points at which processing according to the invention may
`occur;
`
`10
`
`15
`
`20
`
`25
`
`3O
`
`35
`
`FIG. 5 is a flow diagram of the steps involved in pro-
`cessing an image sometime after storage in the camera of
`FIG. 2;
`
`40
`
`FIG. 6 is a flow diagram of the steps involved in interrupt
`processing in the camera of FIG. 2;
`FIG. 7 is a flow diagram of the steps involved in control—
`ling access to processing in the camera of FIG. 2;
`FIG. 8 is a pictorial illustration of the application of
`pre-existing overlay images to images captured by the
`camera of FIG. 2;
`
`FIG. 9 is a pictorial illustration of the use of an electronic
`viewfinder in the camera of FIG. 2 to frame the captured
`image with respect to an overlay image;
`FIG. 10 shows an additional embodiment of an electronic
`
`camera configured according to the invention; and
`FIG. 11 shows an electronic imaging system incorporat-
`ing the camera of FIG. 10 by means of a cable connection.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIIVIENTS
`
`Because electronic still cameras employing electronic
`sensors are well known,
`the present description will be
`directed in particular to elements forming part of, or coop-
`erating more directly with, apparatus in accordance with the
`present
`invention. Elements not specifically shown or
`described herein may be selected from those known in the
`art.
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`
`FIG. 1 shows a known electronic still photography system
`useful for inputting images to a computer. The system
`includes an electronic camera 1 with an electronic sensing
`section 1a, a digital processing section 1b, a memory card
`slot 2, a removable memory card 3, a computer 4 with a
`built-in memory card reader 5, and a printer 6. Images are
`captured by the sensing section 1a in the camera 1, manipu-
`lated in the processing section 1b, stored on the memory
`card 3, and transferred from the card 3 to the computer 4.
`The software used to control the sensing and processing
`sections 1a and 1b in the camera is entirely located in
`firmware memory 7 in the camera 1 and cannot be updated
`without opening the camera cover, unsoldering the firmware
`memory circuit, and replacing it with a new circuit contain-
`ing updated code. The application software used to control
`the playback operations of the computer is supplied, at least
`in part, by the camera manufacturer. It
`is stored in a
`computer hard drive 8, and downloaded to a computer
`random access memory (RAM) memory 9a, where the
`program is used by the central processing unit (CPU) 9b. To
`update this software, the user must be provided with a new
`program on a floppy disc, which can be transferred to the
`hard drive 8. For example, the floppy disk may contain a
`“plug-in” module for an imaging software application pro-
`vided with the camera. The plug~in contains both code to
`allow the images to be downloaded from the camera to the
`computer, and software code to create color images from the
`digitized camera image data. The software code implements
`algorithms for processing the color filter array sampled
`image data supplied by the camera. These algorithms
`include color interpolation, color balance, and color correc-
`tion. A typical color interpolation algorithm is described in
`greater detail in US. Ser. No. 085,519, “Apparatus and
`method for adaptively interpolating a full color image uti-
`lizing chrominance gradients”, filed Jun. 30, 1993, which is
`assigned to Eastman Kodak Company.
`. The invention is based on the recognition that the storage
`device,
`i.e., the memory card 3 in FIG. 1, need not be
`primarily oriented to storage of data for downstream (i.e.,
`computer) processing. The storage device may additionally
`contain files (data, code, etc.) which are capable of upstream
`modification of image data and camera performance. This
`provides an opportunity for affecting image data during the
`capture and manipulation stages prior to storage. Further-
`more,
`these files may be accessed automatically by the
`camera, or selected by the user by means of appropriate
`intervention through the camera. Therefore, the removable
`image storage device is capable of two-way communication
`with the camera. That is, the card slot interface is not limited
`to “dumping” image data downstream to the removable
`storage device, but also allows communication from the
`removable storage device upstream to the camera as well.
`Inasmuch as the files are software which enhance the
`
`operation of the camera, the digital storage device may be
`viewed as a software-enhanced device, and the files are
`hereinafter referred to as enhancement data files.
`
`A number of advantages arise from such two-way com-
`munication. Enhancement files can be included in the digital
`storage device which are not in themselves images. Image
`processing software, look-up tables, matrices, compression
`tables, dynamic range optimization tables, and other files
`capable of affecting the captured image data can be included
`in the digital storage device. Typical algorithms which can
`be modified, or updated,
`thus include color filter array
`interpolation algorithms, noise reduction algorithms, edge
`sharpening algorithms, color reproduction algorithms, com-
`pression algorithms, or dynamic range optimization algo-
`
`HTC Exhibit 1003
`
`HTC Exhibit 1003
`
`

`

`5,477,264
`
`5
`rithms. In this manner, the camera need only be capable of
`executing the functions and transformations contained in the
`storage device, and need not require the storage capacity to
`contain all information necessary to execute the processing
`of the image data. Such non-image files, which cause
`processing of the captured image data, may process the
`captured image data such that the most “true-to-scene"
`reproduction is achieved, or instead provide special effects
`of the kind which change the image into some desired
`alteration of the original scene.
`Another type of non-image file which can be contained in
`the digital storage device enhances the operation of the
`camera. Enhancement files can be included which control
`
`capture parameters such as exposure time, aperture setting,
`flash range output, and so on. A particular set of control
`parameters can be used to tailor the capture parameters to the
`type of scene or subject being captured. This type of file
`might also allow diagnostic tests of the camera to insure
`proper operation and to detect fault conditions in the camera
`system. A further type of enhancement file contained in the
`storage device comprises pre—existing image data files, i.e.,
`files with images not captured by the camera system. Such
`files are accompanied by code which instructs the camera on
`how to merge the pre-existing image files with those cap-
`tured by the camera. In this manner graphic overlays and
`text can be superimposed on the captured images by the
`camera. Proper execution of such a feature might include a
`user interface for selection of pre-existing image and overlay
`combinations, and also a viewfinder which helps the user to
`compose the captured image so as to complement
`the
`selected overlay. Whether for image processing, enhanced
`performance, or pre-existing images, the algorithms them-
`selves are conventional and not themselves part of this
`invention. However, where they are stored, and how they are
`used, is within the scope of the invention. In that connection,
`another function of the invention is to store replacement, or
`updated, software files in the digital storage device so that
`software updates can be added to the camera after its initial
`purchase by the subsequent purchase of appropriately-en-
`hanced storage devices.
`The invention also concerns the sequence of execution of
`operations made possible by files contained in the image
`storage device. It is anticipated that the capture of images
`may occur in a rapid and/or random fashion. It may be
`undesirable to interrupt image processing to initiate another
`capture. If that is the case, processing above and beyond that
`which is necessary to achieve initial storage would not have
`to occur until the camera is no longer in a “ready” mode.
`That is, no processing takes place until the camera is no
`longer being used to capture images and is placed in an “off”
`or “sleep” mode. Alternatively, the camera could interrupt
`processing operations to carry out capture operations. Once
`the capture is complete, the camera can return to processing
`operations.
`FIG. 2 shows an electronic still camera capable of using
`a “software enhanced” storage device as taught by the
`invention. The electronic still camera is divided generally
`into an input section A and a compression and recording
`section B. The input section A includes an exposure section
`10 for directing image light from a subject (not shown)
`toward an image sensor 12. A flash unit 11 may be used to
`illuminate the subject. Although not shown, the exposure
`section 10 includes conventional optics for directing the
`image light through a diaphragm, which regulates the optical
`aperture, and a shutter, which regulates exposure time. The
`sensor 12, which includes a two-dimensional array of pho-
`tosites corresponding to picture elements of the image, is a
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`
`conventional charge~coupled device (CCD) using, e.g.,
`either well-known interline transfer or frame transfer tech-
`niques. The sensor 12 is exposed to image light so that
`analog image charge information is generated in respective
`photosites. The charge information is applied to an output
`diode 14, which converts the charge information to analog
`image signals corresponding to respective picture elements.
`The analog image signals are applied to an A/D converter
`16, which generates a digital image signal from the analog
`input signals for each picture element.
`The digital signals are applied to an image buffer 18 for
`storing one or more images, albeit shown in FIG. 2 as a
`random access memory (RAM) with storage capacity for a
`plurality of still images. A programmed control processor 20
`generally controls the input section A of the camera by
`initiating and controlling exposure (by operation of the
`diaphragm and shutter (not shown) in the exposure section
`10), by controlling the flash unit 11, by generating the
`horizontal and vertical clocks needed for driving the sensor
`12 and for clocking image information therefrom, and by
`enabling the A/D converter 16 in conjunction with the image
`buffer 18 for each signal segment relating to a picture
`element. (The control processor 20 would ordinarily include
`a microprocessor coupled with a system timing circuit.)
`Exposure control would typically be effected by using
`ambient light information from a photocell 19.
`Once a certain number of digital image signals have been
`accumulated in the image buffer 18, the stored signals are
`applied to a programmed digital signal processor 22, which
`controls the throughput processing rate for the compression
`and recording section B of the camera. The algorithms and
`other operating code used by the processor 22 are stored in
`the algorithm memory 28. The digital signal processor 22
`compresses each still image stored in the image buffer 18
`according to a known image compression algorithm, such as
`the well-known JPEG (Joint Photographic Experts Group)
`discrete cosine transformation-based compression algo-
`rithm. The processor 22 applies a compression algorithm
`from the memory 28 to the digital image signals, and sends
`the compressed signals to a removable storage device via an
`interface 26. While a memory card 24 is shown in this
`system as the storage device, other devices may be used,
`such as a floppy disk magnetic medium, a small hard drive,
`or optical storage (in the latter cases, suitable conventional
`reading/writing apparatus would be provided in the camera,
`e.g., magnetic or optical read/write head, etc.) A represen-
`tative memory card is a card adapted to the PCMCIA card
`interface standard, such as described in the PC Card Stan-
`dard, Release 2.0, published by The Personal Computer
`Memory Card International Association, Sunnyvale, Calif,
`September 1991. Moreover, while compression is shown in
`the camera of FIG. 2, compression is not a necessary feature
`of the invention, and uncompressed data could be stored
`instead.
`
`The memory card 24 contains solid state memory 24a,
`such as Flash EPROM memory, which the card uses to store
`image data files. In addition, the memory card 24 contains
`additional memory to store enhancement data files 24b to
`modify the camera operation, or the images captured by the
`camera, or to provide pre-existing overlay images. These
`software enhancements are normally programmed by the
`card manufacturer before the card is sold to the user, and can
`be stored in the Card Information Structure (CIS) of the
`PCMCIA Format
`(as described in the aforementioned
`Release 2.0). The “software enhancement” code may be
`stored in separate non—erasable memory on the card 24, or in
`a portion of the memory space of the memory 24a used to
`store the image data.
`The input section A operates at a rate commensurate with
`
`HTC Exhibit 1003
`
`HTC Exhibit 1003
`
`

`

`5,477,264
`
`7
`
`normal operation of the camera while compression, which
`may consume more time, can be relatively divorced from the
`input rate. The exposure section 10 exposes the sensor 12 to
`image light for a time period dependent upon exposure
`requirements, for example, a time period between 1/1000
`second and several seconds. The image charge is then swept
`from the photosites in the sensor 12, converted to a digital
`format, and written into the image buffer 18. The repetition
`rate of the driving signals provided by the control processor
`20 to the sensor 12, the A/D converter 16 and the bufi°er 18
`are accordingly generated to achieve such a transfer. The
`processing throughput rate of the compression and recording
`section B may be determined by the character of an image,
`i.e., the amount of detail versus redundant information, and
`the speed of the digital signal processor 22.
`An operation display panel 30 is connected to the control
`processor 20 for displaying information useful in operation
`of the camera. Such information might
`include typical
`photographic data, such as shutter speed, aperture, exposure
`bias, color balance (auto, tungsten, fluorescent, daylight),
`high/low resolution, low battery, low light, exposure modes
`(aperture preferred, shutter preferred), and so on. Moreover,
`other information unique to this type of camera is displayed.
`For instance, the memory card 24 would ordinarily include
`a directory signifying the beginning and ending of each
`stored image. This would show on the display 30 as either
`(or both) the number of images stored or the number of
`image spaces remaining, or estimated to be remaining. In
`certain applications, an electronic viewfinder 29 would be
`included for displaying images either before or after storage.
`The control processor 20 also generates important infor-
`mation about the condition and capability of the memory
`card 24. Specifically, the interface 26 is queried for the
`presence of a card 24 and, if no card is connected, a “no
`card” display is produced on the operation display 30.
`Likewise, if a card is present but it is full of images, a “card
`full” display is produced. While not shown, the display 30
`may show the number of “remaining images.” If enhance-
`ment files 24b are present on the card 24, they can be
`identified on the operation display 30. For instance, perfor-
`mance enhancements, such as special set—up for portraiture,
`can be identified; processing enhancements, such as special
`effects, can be identified; and overlay enhancements, such as
`birthday or holiday picture borders, can be identified. User
`intervention in order to specify the several enhancement
`opportunities, or to set conventional capture parameters, is
`obtained from a user inputs section 21. Such inputs could be
`one or more selector switches operable alone or in conjunc-
`tion with prompts from the operation display 30 or the
`electronic viewfinder 29.
`
`The camera also contains fir