15;
`United States Patent
`Sarbadhikariet al.
`
`ACAARA
`
`US005477264A
`
`5,477,264
`[11] Patent Number:
`
`[45] Date of Patent:
`Dec. 19, 1995
`
`[54]
`
`[75]
`
`[73]
`
`[21]
`
`[22]
`
`[51]
`[52]
`[58]
`
`ELECTRONIC IMAGING SYSTEM USING A
`REMOVABLE SOFTWARE-ENHANCED
`STORAGE DEVICE
`
`Inventors: Kamal K. Sarbadhikari, Penfield;
`John R, Fredlund; Kenneth A.
`Parulski, both of Rochester, all of N.Y.
`
`Assignee: Eastman Kodak Company, Rochester,
`N.Y.
`
`Appl. No.: 219,608
`
`Filed:
`
`Mar.29, 1994
`
`5,153,729 10/1992 Saito cascscsscsssossssssseseesotsennecen 358/209
`
`10/1992 Matsumura etal.
`5,155,513
`354/106
`7/1993 Moronaga...........
`. 348/231
`5,226,145
`
`5,262,868
`11/1993 Kaneko.......
`. 348/233
`OTHER PUBLICATIONS
`
`USSN 988,517 filed Dec. 10, 1992, Entitled “Electronic
`Camera with Memory Card Interface to a Computer”, Ken-
`neth A. Parulski et al.
`USSN 085,519 filed Jun. 30, 1993, Entitled “Apparatus and
`Method for Adaptively Interpolating a Full Color Image
`Utilizing Chrominance Gradients”, Claude A. Laroche etal.
`Personal Computer Memory Card International Association
`(PCMCIA) “PC Card Standard”, Release 2.0, Sep. 1991.
`
`Int. C1.° .
`... HO4N 5/76
`
`US.Cl...........
`348/231; 348/233; 348/552
`
`Field of Search ...0..0.....ccccssssseeceees 348/222, 231,
`348/232, 233, 552; 358/909.1, 906; 258/903,
`909; HO4N 5/76; 5/907, 5/92
`
`Primary Examiner—James J. Groody
`Assistant Examiner—Sherrie Hsia
`Attorney, Agent, or Firm—David M. Woods
`
`[57]
`
`ABSTRACT
`
`[56]
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`An electronic imaging system includes a digital electronic
`camera for capturing and storing images in a removable
`storage device, which is also preloaded with enhancement
`files for effecting the operation of the system. The camera
`4,500,183
`2/1985 Tamikawa .u.......ccscssssssersseeeseees 354/21
`includes an optical section for establishing the optical
`
`4,524,381
`.. 358/29
`6/1985 Konishi
`....
`
`parameters of image capture, an image sensing section for
`4,574,319
`wee 358/335
`3/1986 Konishi
`....
`electrically capturing the image, and a signal processing
`
`4,728,978
`3/1988 Inoueetal.......
`354/289.1
`section for operating upon the electrically captured image
`
`4,853,733
`...........
`-. 354/412
`8/1989 Watanabe et al.
`prior to storage. The several sections of the camera are
`
`4,855,779
`8/1989 Ishikawa et al. oe we 354/412
`coordinated and controlled by a programmable processor,
`4,994,844
`«. 354/412
`2/1991 Azuma etal, ...
`which is capable of receiving the enhancementfiles pre-
`5,016,107
`358/209
`5/1991 Sassonet al.
`5,018,017
`5/1991 Sasaki et al.
`358/209
`loaded into the storage device. These files may contain
`5,023,637
`6/1991 Lorton etal.
`.. 354/106
`software for updating the operating code of the camera, for
`
`5,027,214
`6/1991 Fujimori
`..........
`.. 358/209
`modifying the electrically captured image in selected ways,
`
`
`
`5,032,855 7/1991 Taniguchi et al.oo...sees 354/21
`for modifying camera in special situations, or for commu-
`w+ 354/413
`5,070,355
`12/1991 Inoueetal.......
`
`nicating non-captured image-like data, such as text and
`354/195.1
`5,086,311
`2/1992 Nakaetal.
`..
`
`image overlays, to the camera.
`5,103,250
`4/1992 Arifuku et al.
`354/106
`5,138,459
`8/1992 Roberts .........00«
`wes 348/552
`5,142,310
`8/1992 Taniguchi et al. oc sesesee 354/106
`
`
`
`INPUT SECTION (A)
`
`47 Claims, 11 Drawing Sheets
`
`COMPRESSION AND RECORDING
`SECTION (8)
`
`22
`
`INPUTS
`
`ai
`
`39—|
`
`FIRMWARE
`MEMORY
`
`OPERATION
`
`Tc DISPLAY
`
`me
`U7
`
`Fa
`INSTRUCTION
`MEMORY
`
`NO CARD,CARD FULL
`
`PERFORMANCE
`ENHANCEMENTS
`SPECIAL EFFECTS
`OVERLAY
`
`Bt
`
`PETITIONERS EX1020
`Page 1
`
`6 -o-ooror 7A
`|
`IMAGE BUFFER|
`DIGITAL
`26
`HI
`|
`eV
`
`EXPOSURE||| .
`SIGNAL
`ii
`|
`SECTION
`PROCESSOR eT
`J
`500
`LILO
`|
`4
`|
`
`|
`|
`
`[FRAME 7
`
`28
`
`PROCESSING
`i-— ALGORITHMS
`
`20
`
`
`
`10
`
`N
`
`FLASH
`UNIT
`
`30
`
`PETITIONERS EX1020
`Page 1
`
`

`

`U.S. Patent
`
`Dec. 19, 1995
`
`Sheet 1 of 11
`
`5,477,264
`
`£
`
`|}vava
`
`JIGVNNVYIOUd
`
`YOSSTI0Ud
`
`AWV1dS/Id
`
`———————
`
`
`
`Joni
`'——_ N3FFYIS
`
`vivaJOYA)|
`
`JYVMLIOS
`
`SWHLYODTV
`
`YALNIYd
`
`Old
`
`(LUVYOlYd)
`
`PETITIONERS EX1020
`Page 2
`
`PETITIONERS EX1020
`Page 2
`
`
`
`

`

`U.S. Patent
`
`Dec. 19, 1995
`
`Sheet 2 of 11
`
`5,477,264
`
`TINAGYVI'dHVION
`
`NOILVYFAO
`
`AV1dSIG
`
`SLIFIdFWiIddS
`
`SININFONVHNA
`
`JINVNYOAYTd
`
`AVTETAO
`
`If
`
`
`
`NOILONGLSNI
`
`AYONIN
`
`Wa
`
`(9)NOILITS
`
`TWNIIS
`
`YOSSFIIOUd
`
`%e
`
`INISSFIO
`
`92ds|3gi’|aadING
`STAjLNINFONVHNASWHLIYOIWVa
`
`
`
`
`QNIGYOIZYONYNOISSFYdNOD(V)NOILIFSLNANI
`
`FOVMI|arene,penne9!
`
`
`S.LNINI6/
`
`AYOWIN
`
`JUVMANYof
`
`FIYNSOSXF
`NOILIAS
`iat|¢[e
`
`PETITIONERS EX1020
`Page 3
`
`PETITIONERS EX1020
`Page 3
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Dec. 19, 1995
`
`Sheet 3 of 11
`
`5,477,264
`
`
`
`TURN ON
`
`ELECTRONIC
`CAMERA
`
`
`
`
`
`POWER UP CAMERA FUNCTIONS
`SUCH AS FLASH.VIEWFINDER,
`USER DISPLAY,CAPTURE
`ELECTRONICS, ETC.
`
`ALGORITHMS
`
`YES
`
`
`DOWNLOAD ALGORITHM
`
`(MAY BE USER SELECTED
`ONE OF MANY)
`
`<p PRESENT IN S NO
`REMOVABLE IMAGE
`DATA STORAGE ?
`
`
`
`
`
`
`
`
`
`
`
`EXPOSURE
`DESIRED BY
`USER ?
`
`YES
`
`CAPTURE
`IMAGE
`
`APPLY
`DOWNLOADED
`ALGORITHM
`
`
`
`
`STORE IMAGE
`ON REMOVABLE
`
`STORAGE MEANS
`FIG. 3
`
`NOT SUBJECT
`OF THIS
`INVENTION
`
`PETITIONERS EX1020
`Page 4
`
`PETITIONERS EX1020
`Page 4
`
`

`

`U.S. Patent
`
`Dec. 19, 1995
`
`Sheet 4 of 11
`
`5,477,264
`
`—TNYOLS
`
`SWAN
`
`FYOLS
`
`JTIGVAOWIY>NOJFVNI
`
`VIVGJOVANI
`
`ov
`
`SSFI0"d
`
`WYOAOL
`
`G3FYOLS
`
`JOVANI
`
`ypOA
`
`AZILIIG
`
`JIYNLAVI
`
`JOVI
`
`+2
`
`PETITIONERS EX1020
`Page 5
`
`PETITIONERS EX1020
`Page 5
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Dec. 19, 1995
`
`Sheet 5 of 11
`
`5,477,264
`
`CAPTURE
`IMAGE
`
`DIGITIZE
`
`
`
`
`
`
`STORE IMAGE DATA
`ON REMOVABLE
`STORAGE MEANS
`
`POSSIBLE TIME LAPSE
`(CAMERA NO LONGER
`IN "READY" MODE)
`
`DOWNLOAD IMAGE DATA
`FROM REMOVABLE IMAGE
`DATA STORAGE MEANS
`
`
`
`
`
`
`
`
`
`DOWNLOAD PROCESSING
`ALGORITHMS FROM
`
`REMOVABLE IMAGE DATA
`
`
`STORAGE MEANS
`
`PROCESS
`IMAGE
`
`
`
`STORE PROCESSED
`IMAGE IN IMAGE
`
`DATA STORAGE MEANS
`
`
`FIG.5
`
`PETITIONERS EX1020
`Page 6
`
`PETITIONERS EX1020
`Page 6
`
`

`

`US. Patent
`
`Dec. 19, 1995
`
`Sheet 6 of 11
`
`5,477,264
`
`CAPTURE
`IMAGE
`
`BEGIN PROCESSING
`IMAGE ACCORDING TO
`DOWNLOADED
`ALGORITHMS
`
`
`
`
`
`
`
`
`NEW
`CAPTURE
`
`
`DESIRED ?
`
`
`CONTINUE
`INTERRUPT
`PROCESSING
`
`
`PROCESSING
`
`
`
`
`PROCESSING
`FOR THIS_IMAGE
`
`
`
`
`STORE PARTIALLY
`COMPLETE ?
`PROCESSED IMAGE IN
`MANNER SUCH THAT
`PROCESSING MAY
`RESUME LATER
`
`WITHOUT PROBLEMS
`
`
`
`
`
`
`CAPTURE
`NEW IMAGE
`
`
`
`
`"READY" MODE
`
`TO
`
`—FIG.E
`
`PETITIONERS EX1020
`Page 7
`
`PETITIONERS EX1020
`Page 7
`
`

`

`U.S. Patent
`
`Dec. 19, 1995
`
`Sheet 7 of 11
`
`5,477,264
`
`
`
`YOT0DGFONVHNTASN
`
`NOLLINGOUdFY
`
`SWNHLSOSNV
`
`WAYONFSN
`
`40709
`
`NOILONGOYITY
`
`TWHYONASN
`
`d44dS
`
`(OO!OS)
`
`
`
`
`
`GFddSVYFINWDHOLMS
`
`
`
`NONINI300)OL
`
`‘O00!'0S9{)GYvI
`
`(00F‘002
`
`
`
`OWISLYASNIYISN
`
`
`
`YFINdINODO.LNI
`
`
`
`GavSLYASNIYsaSN
`
`
`
`VesANVDO.LNI
`
`WHYONFSN
`
`INISSTIONd
`
`VYJY¥OLS
`
`
`
`JWIYOI0D
`
`SAYSI
`
`edUHN/V1G.
`
`elFISLG
`
`SIDALNled
`
`SIWLYASN
`
`SFIUNLIId
`
`
`
`ONINIDHVHSF9G7ASN
`
`
`
`NOILONGAYFASIONGNV
`
`SLVYLYOSYOAIZINILIO
`
`
`
`JHLAINOJYOLS
`
`AINOYHIONON
`
`GuY0DdY
`
`LOA
`
`PETITIONERS EX1020
`Page 8
`
`PETITIONERS EX1020
`Page 8
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Dec. 19, 1995
`
`Sheet 8 of 11
`
`5,477,264
`
`VACANT PORTION
`OF "PRE-EXPOSED"
`IMAGE
`
`—— All Star ~——
`
`CAPTURED IMAGE
`
`"PRE-EXPOSED' IMAGE
`DATA FILE
`
`—~ All Star——
`
`
`
`COMBINED IMAGE
`FIG.8
`
`PETITIONERS EX1020
`Page 9
`
`PETITIONERS EX1020
`Page 9
`
`

`

`U.S. Patent
`
`Dec. 19, 1995
`
`Sheet 9 of 11
`
`5,477,264
`
`PUSH BUTTON TO CHOOSE
`THIS OVERLAY
`
`— He He Ho ~——
`
`/
`
`/ Y
`
`“CO
`
`NEXT
`
`OUR IMAGE
`
`WILL GO A
`
`ON CAMERA DISPLAY
`AND/OR VIEWFINDER
`
`PUSH BUTTON FOR NEXT
`OVERLAY POSITION
`
`FIG.G
`
`PETITIONERS EX1020
`Page 10
`
`PETITIONERS EX1020
`Page 10
`
`

`

`U.S. Patent
`
`Dec. 19, 1995
`
`Sheet 10 of 11
`
`5,477,264
`
`
`
`
`
`Tigvsved
`
`
`
`SLIFAATWiddadSATWOLOFT
`
`ON/IGYOIFYCNVNOISSFYdNOD
`
`NOILIFS
`
`
`
`
`
`(V)NOILQTSLAIN!
`
`geZZg/ (G@)
`
`
`TINAGeVI'GHVION
`
`SININFONVHNA
`
`JOINVWYOSed
`
`INISSFIOUd
`
`SWHLUYOTIV
`
`
`
`NOILIVEFdAO
`
`AVIdS/G
`
`WLIIG
`
`WNIIS
`
`YOSSFIO"
`
`je
`
`Yasn
`
`leSLNdINI
`“fa
`
`/
`
`
`
`[YIIANE,FON!
`
`AWTY5/0JUVMN
`
`AYOWAN
`
`9O!'Old
`
`PETITIONERS EX1020
`Page 11
`
`PETITIONERS EX1020
`Page 11
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Dec. 19, 1995
`
`Sheet 11 of 11
`
`5,477,264
`
`&f
`
`TIGVNNWY90Ud
`
`YOSSFIO"
`
`INV
`
`AYONWAN
`
`TLVTIOA-NON
`
`
`
`AYOWAINFOWHI
`
`sS
`£
`
`Sf
`
`
`
`IUMNESFTEVSVYF
`
`AYONAN
`
`ry
`SNIT
`
`(——y
`
`JIYVMLAIOS of
` q}
`le]ow|(=—*
`
`1SSE
`
`
`
`Old
`
`YAFLNANOD
`
`SWHLIYODTV
`
`YIN
`
`PETITIONERS EX1020
`Page 12
`
`PETITIONERS EX1020
`Page 12
`
`
`
`

`

`5,477,264
`
`1
`ELECTRONIC IMAGING SYSTEM USING A
`REMOVABLE SOFTWARE-ENHANCED
`STORAGE DEVICE
`
`FIELD OF THE INVENTION
`
`This invention pertains to the field of electronic imaging
`and, in particular, to electronic imaging with an electronic
`still camera that utilizes a removable storage device for
`storing images.
`
`10
`
`BACKGROUND OF THE INVENTION
`
`As understood in the prior art, a digital electronic still
`camera is a device which usesan electronic sensor to capture
`an image; signal processing to modify the captured image
`signal, e.g., as a function of ambient light, and to represent
`it numerically; and some storage device to preserve the
`numerical image data. Ordinarily, the image storage device
`is merely a receptacle for the image data.
`It is further known for a digital electronic still camera to
`use a removable storage device, such as an integratedcircuit
`memory card, to store images. For instance, U.S. Pat. No.
`5,016,107 describes an electronic still camera utilizing
`image compression and providing digital storage in a remov-
`able memory card having a static random access memory.
`Software that controls the camera operation is located in
`non-removable programmable read only memory (PROM)
`integrated circuits in the camera. In this camera, the inte-
`grated circuits in the removable memory card store image
`data and a directory locating the data.
`Otherelectronic camera systems described in the prior art
`record an image header along with the digital image data.
`The header typically describes some characteristics about
`the image or the camera. For instance, in U.S. Pat. No.
`5,018,017, such header information indicates the classifica-
`tion of the information; the imaging system in use; the date
`and time; the compression mode; as well as image-specific
`information, such as flash use, white balance data, exposure
`value, and shutter speed. Furthermore, U.S. Pat. No. 5,153,
`729 shows memory capacity, a free area pointer, a battery
`expiration date, and a write inhibit flag in the header area of
`a memory card. Typically, each system conveys such char-
`acteristic data from the electronic camera by way of the
`memory card (where the data is temporarily stored) to a
`peripheral device, such as a player. The object is to increase
`the downstream functionality of the system by allowing
`utilization of image-related data in subsequent processing of
`the image data in a downstream peripheral device. The
`camera or imaging device generally utilizes little of the
`stored image characteristic data, one exception being data
`regarding theutilization of the storage deviceitself, i.c., how
`much memory space is remaining for further pictures. In the
`latter case, the camera accesses such data to display utili-
`zation to the camera user.
`
`Coordination between a camera and a computer is gen-
`erally shown in patent application Ser. No. 988,517, “Elec-
`tronic Camera with Memory Card Interface to a Computer”,
`which wasfiled on Dec. 10, 1992 and assigned to common
`assignee with the present application. This application
`describes a camera that is specially shaped so as to plug
`directly into the memory card slot of a portable computer,
`the camera is then usable as a unit with the portable
`computer. The camera contains EPROM memory with code
`for in-camera white balance and gammacorrection, and also
`stores the compiled code which is used (by the computer) to
`operate the camera and to process the images from the
`
`20
`
`25
`
`30
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`sensor color filter array to obtain a full resolution, color
`corrected image. The image data is captured by the camera
`and directly downloaded to the computer, where it is pro-
`cessed with code obtained from the camera. As with prior
`memory card systems, the code is used downstream (in the
`computer) to process image data.
`In a camera intended for acquisition of image data for use
`in a computer, such as described in Ser. No. 988,517,it is
`well understood that data corresponding to captured images
`may also be stored in a storage device that is capable of
`being removed from the camera. Such a storage device is
`often considered analogous to film in a conventional film
`camera. Just as it is known to encode film with marks that
`are useful in downstream processing of the film, it is also
`knownfor an electronic image storage device to contain
`non-image data whichis useful in modifying the image data.
`Following the film analogy, such additional data is useful in
`downstream processing, such as in a computer, to modify the
`image data. Unlike film, however, the additional non-image
`data may be read bya variety of peripheral devices, includ-
`ing the camera itself. This provides an opportunity to deal
`with a number of imaging problems.
`For instance, in most known electronic imaging systems,
`the software which controls the camera ordinarily cannot be
`updated without purchasing an entirely ncw camera, con-
`taining a lens, CCD imagesensor, etc. (Albeit, a few high
`end film camerasutilize a removable program module from
`which the camera can download exposure algorithms.) It is
`well known that computer programs are updated more
`regularly than computer hardware, since the software can be
`refined to provide higher performance(i.e. better image
`quality) or more features while using the same hardware. An
`electronic camera can use a programmable digital processor
`controlled by software. Such a camera offers an opportunity
`for software updates, if the system is designed to facilitate
`such updates. What is needed is a method for software
`updates of the camera algorithms for higher performance,
`and for adding new features to a camera. Such a method
`should be quick and convenient for the user, preferably
`without requiring the use of additional storage modules.
`Existing memory devices are capable of limited two-way
`communication with an electronic camera, such as to report
`to the camera on memory space availability. An opportunity
`exists for enhancing such “upstream” communication to
`provide altogether new applications for an electronic cam-
`era.
`
`SUMMARYOF THE INVENTION
`
`The opportunity in the prior art for upstream utilization of
`the memory capability of a removable storage device is
`embodied, according to the invention, by an electronic
`imaging system including an electronic camera for capturing
`and storing images in a removable storage device which is
`also preloaded with software for operating the imaging
`system. The imaging system includes an optical section for
`establishing the optical parameters of image capture, an
`image sensing scction for electrically capturing an image
`provided by the optical section, a signal processing section
`for processing the electrically captured image, a program-
`mable processorfor controlling the sections of the system,
`and means for loading the programmable processor with the
`software preloaded in the removable storage device for
`affecting operation of one or more of the sections of the
`system.
`Consequently, the invention provides an electronicstill
`photography system which can be controlled by software
`
`PETITIONERS EX1020
`Page 13
`
`PETITIONERS EX1020
`Page 13
`
`

`

`5,477,264
`
`3
`located on a removable memory (e.g., a memory card) used
`with an electronic still camera. Known cameras use the
`removable memory to store images and data related to an
`individual image (date, color temperature of the illuminant,
`etc.) or data related to the camera (type ofcolorfilter array,
`sensor defect locations, etc.) which is transferred to an
`image reproducing device. Except for data about memory
`space, the communication is one way. A camera according to
`this invention, however, uses the removable media to trans-
`fer software previously recorded on the memory card (such
`as improved camera firmware to process the image, or a
`special access code) to the camera. The advantageouseffect
`is that the operation of the camera or playback device can be
`improved someyears after the camera is purchased, by using
`an “enhanced” memory card containing improved software
`for operating the camera or playback device. Moreover, such
`two-way communication opensuse of the storage device to
`other enhancement possibilities, e.g., to modify image data
`in selected ways, to modify camera performancefor special
`situations, or to communicate non-captured images(special
`overlays) to the camera.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The prior art and the invention will be described in
`relation to the drawings, wherein
`FIG. 1 is a diagram of a knownelectronic imaging system
`incorporating an electronic camera for inputting images to a
`computer;
`FIG. 2 is a block diagram of an electronic camera con-
`figured according to the invention to receive and process
`enhancementfiles;
`FIG.3 is a flow diagram of the steps involved in applying
`an enhancementalgorithm to the camera of FIG. 2;
`FIG.4 is a block diagram of the capture chain showing the
`points at which processing according to the invention may
`occur;
`
`FIG. 5 is a flow diagram of the steps involved in pro-
`cessing an image sometime after storage in the camera of
`FIG.2;
`FIG.6 is a flow diagram of the steps involvedin interrupt
`processing in the camera of FIG. 2;
`FIG.7 is a flow diagram of the steps involved in control-
`ling access to processing in the camera of FIG. 2;
`FIG. 8 is a pictorial illustration of the application of
`pre-existing overlay images to images captured by the
`camera of FIG.2;
`FIG.9 is a pictorial illustration of the use of an electronic
`viewfinder in the camera of FIG. 2 to frame the captured
`image with respect to an overlay image;
`FIG. 10 shows an additional embodimentof an electronic
`camera configured according to the invention; and
`FIG. 11 showsan electronic imaging system incorporat-
`ing the camera of FIG. 10 by meansof a cable connection.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENTS
`
`Because electronic still cameras employing electronic
`sensors are well known, the present description will be
`directed in particular to elements forming part of, or coop-
`erating more directly with, apparatus in accordance with the
`present
`invention. Elements not specifically shown or
`described herein may be selected from those knownin the
`art.
`
`10
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`FIG. 1 shows a knownelectronic still photography system
`useful for inputting images to a computer. The system
`includes an electronic camera 1 with an electronic sensing
`section la, a digital processing section 1b, a memory card
`slot 2, a removable memory card 3, a computer 4 with a
`built-in memory card reader 5, and a printer 6. Images are
`captured by the sensing section la in the camera 1, manipu-
`lated in the processing section 1b, stored on the memory
`card 3, and transferred from the card 3 to the computer 4.
`The software used to control the sensing and processing
`sections la and 1b in the camera is entirely located in
`firmware memory 7 in the camera 1 and cannot be updated
`without opening the camera cover, unsoldering the firmware
`memory circuit, and replacing it with a new circuit contain-
`ing updated code. The application software used to control
`the playback operations of the computeris supplied,at least
`in part, by the camera manufacturer. It is stored in a
`computer hard drive 8, and downloaded to a computer
`random access memory (RAM) memory 9a, where the
`program is used bythe central processing unit (CPU) 9b. To
`update this software, the user must be provided with a new
`program on a floppy disc, which can be transferred to the
`hard drive 8. For example, the floppy disk may contain a
`“plug-in” module for an imaging software application pro-
`vided with the camera. The plug-in contains both code to
`allow the images to be downloaded from the camera to the
`computer, and software code to create color images from the
`digitized camera image data. The software cade implements
`algorithms for processing the color filter array sampled
`image data supplied by the camera, These algorithms
`include color interpolation, color balance, and color correc-
`tion. A typical color interpolation algorithm is described in
`greater detail in U.S. Ser. No. 085,519, “Apparatus and
`method for adaptively interpolating a full color image uti-
`lizing chrominance gradients”, filed Jun. 30, 1993, which is
`assigned to Eastman Kodak Company.
`. The invention is based on the recognitionthat the storage
`device,
`i.e.,
`the memory card 3 in FIG. 1, need not be
`primarily oriented to storage of data for downstream (i.e.,
`computer) processing. The storage device may additionally
`contain files (data, code, etc.) which are capable of upstream
`modification of image data and camera performance. This
`provides an opportunity for affecting image data during the
`capture and manipulation stages prior to storage. Further-
`more,
`these files may be accessed automatically by the
`camera, or selected by the user by means of appropriate
`intervention through the camera. Therefore, the removable
`image storage device is capable of two-way communication
`with the camera. Thatis, the card slot interface is not limited
`to “dumping” image data downstream to the removable
`storage device, but also allows communication from the
`removable storage device upstream to the camera as well.
`Inasmuch as the files are software which enhance the
`operation of the camera, the digital storage device may be
`viewed as a software-enhanced device, and the files are
`hereinafter referred to as enhancement data files.
`
`A numberof advantages arise from such two-way com-
`munication. Enhancementfiles can be included in the digital
`storage device which are not in themselves images. Image
`processing software, look-up tables, matrices, compression
`ables, dynamic range optimization tables, and other files
`capable ofaffecting the captured image data can be included
`in the digital storage device. Typical algorithms which can
`be modified, or updated,
`thus include color filter array
`interpolation algorithms, noise reduction algorithms, edge
`sharpening algorithms, color reproduction algorithms, com-
`pression algorithms, or dynamic range optimization algo-
`
`PETITIONERS EX1020
`Page 14
`
`PETITIONERS EX1020
`Page 14
`
`

`

`5,477,264
`
`5
`rithms. In this manner, the camera need only be capable of
`executing the functions and transformations contained in the
`storage device, and need not require the storage capacity to
`contain all information necessary to execute the processing
`of the image data. Such non-image files, which cause
`processing of the captured image data, may process the
`captured image data such that the most “true-to-scene”
`reproduction is achieved, or instead provide special effects
`of the kind which change the image into some desired
`alteration of the original scence.
`Another type of non-imagefile which can be contained in
`the digital storage device enhances the operation of the
`camera. Enhancementfiles can be included which control
`capture parameters such as exposure time, aperture setting,
`flash range output, and so on. A particular set of control
`parameters can be usedto tailor the capture parameters to the
`type of scene or subject being captured. This type offile
`might also allow diagnostic tests of the camera to insure
`proper operation and to detect fault conditions in the camera
`system. A further type of enhancementfile contained in the
`storage device comprises pre-existing image data files, i.e.,
`files with images not captured by the camera system. Such
`files are accompanied by code which instructs the camera on
`how to merge the pre-existing image files with those cap-
`tured by the camera. In this manner graphic overlays and
`text can be superimposed on the captured images by the
`camera. Proper execution of such a feature might include a
`user interface for selection of pre-existing image and overlay
`combinations, and also a viewfinder which helps the user to
`compose the captured image so as to complement
`the
`selected overlay. Whether for image processing, enhanced
`performance,or pre-existing images, the algorithms them-
`selves are conventional and not themselves part of this
`invention. However, where they are stored, and how they are
`used, is within the scope of the invention.In that connection,
`another function of the invention is to store replacement, or
`updated, software files in the digital storage device so that
`software updates can be added to the cameraafterits initial
`purchase by the subsequent purchase of appropriately-en-
`hanced storage devices.
`The invention also concerns the sequence of execution of
`operations made possible by files contained in the image
`storage device. It is anticipated that the capture of images
`may occur in a rapid and/or random fashion. It may be
`undesirable to interrupt image processing to initiate another
`capture.If that is the case, processing above and beyondthat
`which is necessary to achieveinitial storage would not have
`to occur until the camera is no longer in a “ready” mode.
`That is, no processing takes place until the camera is no
`longer being used to capture images andis placed in an “off”
`or “sleep” mode. Alternatively, the camera could interrupt
`processing operations to carry out capture operations. Once
`the capture is complete, the camera can return to processing
`operations.
`FIG. 2 showsan electronic still camera capable of using
`a “software enhanced” storage device as taught by the
`invention. The electronic still camera is divided generally
`into an input section A and a compression and recording
`section B. The input section A includes an exposure section
`10 for directing image light from a subject (not shown)
`toward an image sensor 12. A flash unit 11 may be used to
`illuminate the subject. Although not shown, the exposure
`section 10 includes conventional optics for directing the
`imagelight through a diaphragm, which regulates the optical
`aperture, and a shutter, which regulates exposure time. The
`sensor 12, which includes a two-dimensional array of pho-
`tosites corresponding to picture elements of the image,is a
`
`10
`
`20
`
`25
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`conventional charge-coupled device (CCD) using, e.g.,
`either well-knowninterline transfer or frame transfer tech-
`niques. The sensor 12 is exposed to image light so that
`analog image charge information is generated in respective
`photosites. The charge information is applied to an output
`diode 14, which converts the charge information to analog
`image signals corresponding to respective picture elements.
`The analog image signals are applied to an A/D converter
`16, which generates a digital image signal from the analog
`input signals for each picture element.
`The digital signals are applied to an image buffer 18 for
`storing one or more images, albeit shown in FIG. 2 as a
`random access memory (RAM) with storage capacity for a
`plurality of still images. A programmedcontrol processor 20
`generally controls the input section A of the camera by
`initiating and controlling exposure (by operation of the
`diaphragm and shutter (not shown) in the exposure section
`10), by controlling the flash unit 11, by generating the
`horizontal and vertical clocks needed for driving the sensor
`12 and for clocking image information therefrom, and by
`enabling the A/D converter 16 in conjunction with the image
`buffer 18 for each signal segment relating to a picture
`element. (The control processor 20 would ordinarily include
`a microprocessor coupled with a system timing circuit.)
`Exposure control would typically be effected by using
`ambient light information from a photocell 19.
`Oncea certain number ofdigital image signals have been
`accumulated in the image buffer 18, the stored signals are
`applied to a programmed digital signal processor 22, which
`controls the throughput processing rate for the compression
`and recording scction B of the camera. The algorithms and
`other operating code used by the processor 22 are stored in
`the algorithm memory 28. The digital signal processor 22
`compresses eachstill image stored in the image buffer 18
`according to a known image compression algorithm, such as
`the well-known JPEG (Joint Photographic Experts Group)
`discrete cosine transformation-based compression algo-
`rithm. The processor 22 applies a compression algorithm
`from the memory 28to the digital image signals, and sends
`the compressed signals to a removable storage device via an
`interface 26. While a memory card 24 is shown in this
`system as the storage device, other devices may be used,
`such as a floppy disk magnetic medium,a small hard drive,
`or optical storage (in the latter cases, suitable conventional
`reading/writing apparatus would be provided in the camera,
`€.g., magnetic or optical read/write head, etc.) A represen-
`tative memory card is a card adapted to the PCMCIAcard
`interface standard, such as described in the PC Card Stan-
`dard, Release 2.0, published by The Personal Computer
`Memory Card International Association, Sunnyvale,Calif.,
`September 1991. Moreover, while compression is shown in
`the camera of FIG. 2, compression is not a necessary feature
`of the invention, and uncompressed data could be stored
`instead.
`
`The memory card 24 contains solid state memory 24a,
`such as Flash EPROM memory, which the card uses to store
`image data files. In addition, the memory card 24 contains
`additional memory to store enhancement data files 24b to
`modify the camera operation, or the images captured by the
`camera, or to provide pre-existing overlay images. These
`software enhancements are normally programmed by the
`card manufacturer before the card is sold to the user, and can
`be stored in the Card Information Structure (CIS) of the
`PCMCIA Format
`(as described in the aforementioned
`Release 2.0). The “software enhancement” code may be
`stored in separate non-erasable memory onthe card 24,or in
`a portion of the memory space of the memory 24a used to
`store the image data.
`The input section A operates at a rate commensurate with
`
`PETITIONERS EX1020
`Page 15
`
`PETITIONERS EX1020
`Page 15
`
`

`

`5,477,264
`
`7
`normal operation of the camera while compression, which
`may consume moretime, can be relatively divorced from the
`input rate. The exposure section 10 exposes the sensor 12 to
`image light for a time period dependent upon exposure
`requirements, for example, a time period between “ooo
`second and several seconds. The image charge is then swept
`from the photosites in the sensor 12, convertedto a digital
`format, and written into the image buffer 18. The repetition
`tate of the driving signals provided by the control processor
`20 to the sensor 12, the A/D converter 16 and the buffer 18
`are accordingly generated to achieve such a transfer. The
`processing throughputrate of the compression and recording
`section B may be determined by the character of an image,
`ie., the amountof detail versus redundant information, and
`the speed of the digital signal processor 22.
`An operation display panel 30 is connected to the control
`processor 20 for displaying information useful in operation
`of the camera. Such information might include typical
`photographic data, such as shutter speed, aperture, exposure
`bias, color balance (auto, tungsten, fluorescent, daylight),
`high/low resolution, low battery, low light, exposure modes
`(aperture preferred, shutter preferred), and so on. Moreover,
`other information uniqueto this type of camerais displayed.
`For instance, the memory card 24 would ordinarily include
`a directory signifying the beginning and ending of each
`stored image. This would show on the display 30 as either
`(or both) the number of images stored or the number of
`image spaces remaining, or estimated to be remaining. In
`certain applications, an electronic viewfinder 29 would be
`included for displaying imageseither beforeorafter storage.
`The control processor 20 also generates importantinfor-
`mation about the condition and capability of the memory
`card 24. Specifically, the interface 26 is queried for the
`presence of a card 24 and,if no card is connected, a “no
`card” display is produced on the operation display 30.
`Likewise, if a card is present butit is full of images, a “card
`full” display is produced. While not shown, the display 30
`may show the numberof “remaining images.” If enhance-
`ment files 24b are present on the card 24,
`they can be
`identified on the operation display 30. For instance, perfor-
`mance enhancements, such as special set-up for portraiture,
`can be identified; processing enhancements, such as special
`effects, can be identified; and overlay enhancements, such as
`birthday or holiday picture borders, can be identified. User
`intervention in order to specify the several enhancement
`opportunities, or to set conventional capture parameters,is
`obtained from a user inputs section 21. Such inputs could be
`one or