`US005477264A
`[11] Patent Number:
`5,477,264
`[45] Date of Patent:
`Dec. 19, 1995
`
`United States Patent [19]
`Sarbadhikari et al.
`
`[54] ELECTRONIC IMAGING SYSTEM USING A
`REMOVABLE SOFTWARE-ENHANCED
`STORAGE DEVICE
`
`[75] Inventors: Kama! K. Sarbadhikari, Pen?eld;
`John R. Fredlund; Kenneth A.
`Parulski, both of Rochester, all of NY.
`
`[73]
`
`Assignee: Eastman Kodak Company, Rochester,
`NY.
`
`Appl. No.: 219,608
`
`[21]
`[22] Filedi
`
`Mar- 29, 1994
`
`[51] Int. Cl.6 ..................................................... .. H04N 5/76
`[52] US. Cl. .......... ..
`348/231; 348/233; 348/552
`[58] Field of Search ................................... .. 348/222, 231,
`348/232, 233, 552; 358/9091, 906; 258/903,
`909; H04N 5/76; 5/907, 5/92
`_
`References Cited
`U_S_ PATENT DOCUMENTS
`
`[56]
`
`4,500,183
`4’524’381
`
`2/1935
`6/1985
`
`8/1989
`4’853’733
`3/1989
`4:355:779
`'
`4,994,844 2/1991
`5,016,107
`5/1991 Sasson et a1. ......................... .. 353/209
`5,018,017
`5/1991 Sasaki et a1.
`358/209
`5,023,637
`6/1991 Lorton 6H1.
`354/106
`5,027,214
`6/ 1991 F“li_m°?_ ------- --
`353/209
`---
`1;;
`gamgu‘ithlalet 31-
`. . . .. 354/1951
`2/1992 Naka etal. . . . . . . . . .
`354/106
`4/1992 Arifuku et a1. .
`348/552
`8/1992 Roberts ............. ..
`8/1992 Taniguchi et a1. .................... .. 354/106
`
`one e
`
`.
`
`. . . . . . . . .
`
`. . . . . . ..
`
`,
`
`,
`
`5,086,311
`5,103,250
`5,138,459
`5,142,310
`
`5,153,729 10/1992 Saito ......................... .ZI ......... .. 358/209
`5,155,513 10/1992 Matsumura et al.
`.
`5,226,145
`7/1993 Moronaga ....... ..
`..
`5,262,868 11/1993 Kaneko ................................. .. 348/233
`OTHER PUBLICATIONS
`
`USSN 988,517 ?led Dec. 10, 1992, Entitled “Electronic
`Camera with Memory Card Interface to a Computer”, Ken
`neth A. Parulski et al.
`USSN 085,519 ?led Jun. 30, 1993, Entitled “Apparatus and
`Method for Adaptively Interpolating a Full Color Image
`Utilizing Chrorninance Gradients”, Claude A. Laroche et al.
`Personal Computer Memory Card International Association
`(PCMCIA) “PC Card Standard”, Release 2.0, Sep. 1991.
`
`priman, Examiner_james ]_ Groody
`Assistant Examiner-Shame Hsia
`Attorney, Agent, or Firm—David M. Woods
`
`ABSTRACT
`[57]
`An electronic imaging system includes a digital electronic
`camera for capturing and storing images in a removable
`storage device, which is also preloaded with enhancement
`?les for effecting the operation of the system. The camera
`includes an optical section for establishing the optical
`parameters of image capture, an image sensing section for
`electrically capturing the image, and a signal processing
`section for operating upon the electrically captured image
`prior to storage. The several sections of the camera are
`coordrnated and controlled by a programmable processor,
`which is Capable of receiving the enhancement ?les Pre
`loaded into the storage device. These ?les may contain
`software for updating the operating code of the camera, for
`modifying the electrically captured image in selected ways,
`for modifying camera in special situations, or for commu
`
`-
`
`-
`
`_
`
`_.
`
`-
`
`_ -
`
`?lcginivggg 2:12:12: 33:21]“ data’ Such as text and
`g
`y ’
`'
`
`47 Claims, 11 Drawing Sheets
`
`INPUT SECTION (A)
`
`COMPRESSION AND RECORDING
`SECTION (B)
`/6 I‘ ____ ———1 r--—-—-—-—--/ \—-————\
`
`"
`
`25
`
`IIIIIII IIIIIIIII
`l l
`HHHIHIIIIIH
`
`24b
`
`ENHANCEMENT /
`FILES
`[24
`WAGE \
`DATA
`~24<1
`FILES
`
`NO CARD.CARD FULL
`
`PERFORMANCE
`ENHANCEMENTS
`SPEC/AL EFFECTS
`
`OVERLAY
`
`l4
`/
`?
`,2
`
`EXPOSURE
`SECTION
`/ k
`I0 I]
`/
`FLASH
`UN/T
`
`22
`1 /\/ | IMAGE BUFFER I r/5
`DIGITAL
`|
`.' l/
`A/D ._>|
`' l_" SIGNAL
`23
`I
`I
`| T’ PROCESSOR
`| FRAME ‘ I
`i /300
`i
`I
`
`‘ (28
`PROCESSING
`r___-, ALGORITHMS
`'
`20 q ‘
`p
`1_______________ PROCESSOR
`
`—
`
`30
`f
`
`PC \ USER
`'9
`INPUTS \
`2/
`
`,
`
`,
`
`-* OPERATION
`:
`D/SPLAy
`
`32» FIRMWARE
`MEMORY
`
`:
`,
`29
`
`5w.-
`
`RAM
`INSTRUCTION
`MEMORY
`
`<
`
`Sony, Ex. 1017, p.1
`
`
`
`US. Patent
`
`Dec. 19, 1995
`
`Sheet 1 of 11
`
`5,477,264
`
`\ ?gzgmwqmm
`
`E3 5% +
`
`m. w / \
`
`x (\l /\ \\
`r 3 E
`
`Eg?:
`
`“E335
`
`/ N
`
`
`@8385 ll. $35 Al! Al Q8
`
`$2M: a?
`
`y
`
`55%8
`
`2% 5%
`
`E3 5% mmétsm
`Y 3&0 w DQ/K
`
`am) $358?
`
`A m
`
`m
`
`Emma
`
`Emma VEQ
`
`m
`
`SEQ
`
`Sony, Ex. 1017, p.2
`
`
`
`U.S. Patent
`
`Dec. 19, 1995
`
`I1f02twhS
`
`5,477,264
`
`mémzmoixzm_mo2§mo..Emm
`
`dunqxédméQ<3&5
`
`zotammmo
`I,
`
`-=---—=--—
`
`
`
`Mmntk_..~.2..m\s.mU\<<I\<M.W3.I.Q&O®<<..lI
`
`m3mmm8.E
`
`__
`
`==.==.==.=.”mommmoomm
`
`on.mommmoomm
`
`Eozma
`
`
`mbmfiméammm_mt<>$t\u.NM
`
`
`
`
`
`xmosm:x<.Em>o\<c\k\m«®~mkm2\T
`
`.§§m
`
`zotomm
`
`€55\__mmmi_mmmmzmmess_/lllll/\IIl|.l\_Illlll...._9E.
`
`
`
`uzamoommQ3>§mmmmo$8...3zotommSm:
`
`mmsmoocfi
`
`zotomm
`
`Sony, Ex. 1017, p.3
`
`
`
`US. Patent
`
`Dec. 19, 1995
`
`Sheet 3 of 11
`
`5,477,264
`
`TURN ON
`E LE CTRONIC
`CAMERA
`
`POI/ER UP CAMERA FUNCTIONS
`SUCH AS FLASH.VIEWFINDER.
`USER DISPLAY.CAPTURE
`ELECTRONICS. ETC.
`
`1
`
`ALGORITHMS
`{ PRESENT IN
`REMOVABLE IMAGE
`DATA STORAGE ?
`
`N0
`
`NOT SUBJECT
`OF THIS
`INVENTION
`
`DOWNLOAD ALGORITHM
`(MAY BE 'USER SELECTED
`ONE OF MANY)
`
`EXPOSURE
`DESIRED BY
`USER ?
`
`CAPTURE
`IMAGE
`l
`APPLY
`DOWNLOADED
`ALGORITHM
`@
`STORE IMAGE
`ON REMOVABLE
`STORAGE MEANS
`
`FIG.3
`
`Sony, Ex. 1017, p.4
`
`
`
`US. Patent
`
`Dec. 19, 1995
`
`Sheet 4 0f 11
`
`5,477,264
`
`mmmGQtm
`
`Qmtekm
`
`$6M Gk
`
`M633
`
`Nv/ 1
`
`v GI
`
`MNCGE
`mtbklqd
`
`M553
`
`AINI
`AINI
`
`Sony, Ex. 1017, p.5
`
`
`
`US. Patent
`
`Dec. 19, 1995
`
`Sheet 5 0f 11
`
`5,477,264
`
`CAPTURE
`IMAGE
`
`l
`
`DIG/T IZ E
`
`l
`
`STORE IMAGE DATA
`ON REMOVABLE
`STORAGE MEANS
`
`__\)\_ POSSIBLE TIME LAPSE
`(CAMERA NO LONGER
`IN ‘READY’ MODE)
`
`DOWNLOAD IMAGE DATA
`FROM REMOVABLE IMAGE
`DATA STORAGE MEANS
`
`l
`
`DOWNLOAD PROCESSING
`ALGORITHMS FROM
`REMOVABLE IMAGE DATA
`STORAGE MEANS
`
`l
`
`PROCESS
`IMAGE
`
`l
`
`STORE PROCESSED
`IMAGE lN IMAGE
`DATA STORAGE ' MEANS
`
`FIG.5
`
`Sony, Ex. 1017, p.6
`
`
`
`US. Patent
`
`Dec. 19, 1995
`
`Sheet 6 of 11
`
`5,477,264
`
`CAPTURE
`IMAGE
`
`BEGIN PROCESSING
`IMAGE ACCORDING TO
`DOWNLOADED
`ALGORITHMS
`
`NEW
`CAPTURE
`DESIRED ?
`
`INT ERRUPT
`PROCESSING
`
`l
`
`STORE PARTIALLY
`PROCESSED IMAGE IN
`MANNER SUCH THAT
`PROCESSING MAY
`RESUME LATER
`WITHOUT PROBLEMS
`
`l
`
`CAPTURE
`NEW IMAGE
`I
`
`'
`
`CONTINUE
`PROCESSING
`
`PROCESSING
`F OR THIS IMAGE
`COMPLETE ?
`
`PROCESSING
`COMPLETE FOR
`ALL IMAGES
`
`RETURN
`TO
`‘READY’ MODE
`
`v FIG. 6
`
`Sony, Ex. 1017, p.7
`
`
`
`U.S. Patent
`
`599119;1GED
`
`Sheet 7 of 11
`
`.3Eozmm:
`
`Q38
`
`zotuaacmmmm
`
`m$§Eom.
`
`bmufizzm
`
`mntvd
`
`.356:was
`
`qmmmm
`
`398:
`
`m.$§.E8.
`
`hmoizzm
`
`«Q36
`
`Q56whimsEm:
`
`
`
`mm.sm=80.55
`
`$3EmmmsEm:
`
`<tm3Gobs
`
`
`
`Q38nmufitzmmm:
`
`Etuaotmmm
`
`m$EmoS<
`
`
`
`
`
`Emma<tm§.Gtuhkm
`
`$9.0m.at3%202m\<QMQ8ck
`
`«8v.8N
`
`ssiozmm:
`
`e>..mmm8E
`
`
`
`ES::38<mmoh
`
`mesaM‘
`
`.m$§\ESm.
`
`Emhm
`
`
`
`.mm$§ksttl
`
`mmxfimums
`
`mm§.§m
`
`uszmmfizmmeowmm:
`
`Steam:M902Qz<
`
`mb<EmEmomQMNSCQQ
`
`mlk\§<cmuch
`
`m:8:82o:
`
`Qtoomt
`
`Sony, Ex. 1017, p.8
`
`
`
`US. Patent
`
`Dec. 19, 1995
`
`Sheet 8 of 11
`
`5,477,264
`
`VACANT PORTION
`OF ‘PRE -E X POSE D’
`IMAGE
`
`\ m Shard“
`
`81111 g g e 1r
`
`CAPTURED IMAGE
`
`'PRE-EXPOSED' IMAGE
`
`\ / DATA FILE
`
`x All Shut’ J‘
`
`S1111 g ger
`COMBINED IMAGE
`F/G.8
`
`Sony, Ex. 1017, p.9
`
`
`
`US. Patent
`
`Dec. 19, 1995
`
`Sheet 9 of 11
`
`5,477,264
`
`PUSH BUTTON TO CHOOSE
`THIS OVERLAY
`
`SEL®ECT
`
`.
`
`NEXT
`
`/ '
`/
`YOUR IMAGE
`WILL GO HERE/
`
`ON ‘CAMERA DISPLAY
`AND/OR V/EWF/NDER
`
`PUSH BUTTON FOR NEXT
`OVERLAY POSITION
`
`F/G.9
`
`Sony, Ex. 1017, p.10
`
`
`
`U.S. Patent
`
`Dec. 19, 1995
`
`am01LleehS
`
`11
`
`5,477,264
`
`
`
`
`
`mbmmmm..<Bmmmx.U<uE..Bm<m
`
`.flm<m<mm
`
`\eJmm>0mm§$mE
`
`EDEN:
`
`m§.uSmo2<$<m
`
`moz<:mo..Emm
`
`
`
`
`
`.33Qtvnu.Qt<QO2x<:>m.w\QI\<O\.~<tMlD
`
`..§mmm.8.E____
`mommm8E
`
`mzmtmouéinmi!
`
`
`
`mo<..Em.§.235x..._mi
`
`_:En..@mm,.§._
`
`€23
`
`an
`
`mzBm8mmQ2<zemmmmniou
`
`
`
`
`
`B.zotomm3zotomm.5l><
`
`Sony, Ex. 1017, p.11
`
`
`
`U.S. Patent
`
`59919:1ceD
`
`Sheet 11 of 11
`
`5,477,264
`
`mES§.z§
`
`Eozmzmug
`
`mm
`
`
`
`.flm<§<m8mo.mcsxtk
`
`Eo:m:
`
`mommmuomm
`
`
`
`mt<.$§$km4m<m<mm_
`
`Eoxm:
`
`mm
`
`‘Emma
`
`mm.Sn$803x
`
`méthtQ1.
`
`E5
`
`Sony, Ex. 1017, p.12
`
`
`
`5 ,477 ,264
`
`1
`ELECTRONIC IMAGING SYSTEM USING A
`REMOVABLE SOFTWARE-ENHANCED
`STORAGE DEVICE
`
`FIELD OF THE INVENTION
`
`This invention pertains to the ?eld of electronic imaging
`and, in particular, to electronic imaging with an electronic
`still camera that utilizes a removable storage device for
`storing images.
`
`10
`
`BACKGROUND OF THE INVENTION
`
`2
`sensor color ?lter array to obtain a full resolution, color
`corrected image. The image data is captured by the camera
`and directly downloaded to the computer, where it is pro
`cessed with code obtained from the camera. As with prior
`memory card systems, the code is used downstream (in the
`computer) to process image data.
`In a camera intended for acquisition of image data for use
`in a computer, such as described in Ser. No. 988,517, it is
`well understood that data corresponding to captured images
`may also be stored in a storage device that is capable of
`being removed from the camera. Such a storage device is
`often considered analogous to ?lm in a conventional ?lm
`camera. Just as it is known to encode ?lm with marks that
`are useful in downstream processing of the ?lm, it is also
`known for an electronic image storage device to contain
`non-image data which is useful in modifying the image data.
`Following the ?lm analogy, such additional data is useful in
`downstream processing, such as in a computer, to modify the
`image data. Unlike ?lm, however, the additional non-image
`data may be read by a variety of peripheral devices, includ
`ing the camera itself. This provides an opportunity to deal
`with a number of imaging problems.
`For instance, in most known electronic imaging systems,
`the software which controls the camera ordinarily cannot be
`updated without purchasing an entirely new camera, con
`taining a lens, CCD image sensor, etc. (Albeit, a few high
`end ?lm cameras utilize a removable program module from
`which the camera can download exposure algorithms.) It is
`well known that computer programs are updated more
`regularly than computer hardware, since the software can be
`re?ned to provide higher performance (i.e. better image
`quality) or more features while using the same hardware. An
`electronic camera can use a programmable digital processor
`controlled by software. Such a camera otfers an opportunity
`for software updates, if the system is designed to facilitate
`such updates. What is needed is a method for software
`updates of the camera algorithms for higher performance,
`and for adding new features to a camera. Such a method
`should be quick and convenient for the user, preferably
`without requiring the use of additional storage modules.
`Existing memory devices are capable of limited two-way
`communication with an electronic camera, such as to report
`to the camera on memory space availability. An opportunity
`exists for enhancing such “upstream” communication to
`provide altogether new applications for an electronic carn
`era.
`
`SUMMARY OF THE INVENTION
`
`The opportunity in the prior art for upstream utilization of
`the memory capability of a removable storage device is
`embodied, according to the invention, by an electronic
`imaging system including an electronic camera for capturing
`and storing images in a removable storage device which is
`also preloaded with software for operating the imaging
`system. The imaging system includes an optical section for
`establishing the optical parameters of image capture, an
`image sensing section for electrically capturing an image
`provided by the optical section, a signal processing section
`for processing the electrically captured image, a program
`mable processor for controlling the sections of the system,
`and means for loading the programmable processor with the
`software preloaded in the removable storage device for
`affecting operation of one or more of the sections of the
`system.
`Consequently, the invention provides an electronic still
`photography system which can be controlled by software
`
`25
`
`30
`
`35
`
`As understood in the prior art, a digital electronic still
`camera is a device which uses an electronic sensor to capture
`an image; signal processing to modify the captured image
`signal, e.g., as a function of ambient light, and to represent
`it numerically; and some storage device to preserve the
`numerical image data. Ordinarily, the image storage device
`is merely a receptacle for the image data.
`It is further known for a digital electronic still camera to
`use a removable storage device, such as an integrated circuit
`memory card, to store images. For instance, U.S. Pat. No.
`5,016,107 describes an electronic still camera utilizing
`image compression and providing digital storage in a remov
`able memory card having a static random access memory.
`Software that controls the camera operation is located in
`non-removable programmable read only memory (PROM)
`integrated circuits in the camera. In this camera, the inte
`grated circuits in the removable memory card store image
`data and a directory locating the data.
`Other electronic camera systems described in the prior art
`record an image header along with the digital image data.
`The header typically describes some characteristics about
`the image or the camera. For instance, in U.S. Pat. No.
`5,018,017, such header information indicates the classi?ca
`tion of the information; the imaging system in use; the date
`and time; the compression mode; as well as image-speci?c
`information, such as ?ash use, white balance data, exposure
`value, and shutter speed. Furthermore, US. Pat. No. 5,153,
`729 shows memory capacity, a free area pointer, a battery
`expiration date, and a write inhibit ?ag in the header area of
`a memory card. Typically, each system conveys such char
`acteristic data from the electronic camera by way of the
`memory card (where the data is temporarily stored) to a
`peripheral device, such as a player. The object is to increase
`the downstream functionality of the system by allowing
`utilization of image-related data in subsequent processing of
`the image data in a downstream peripheral device. The
`camera or imaging device generally utilizes little of the
`stored image characteristic data, one exception being data
`regarding the utilization of the storage device itself, i.e., how
`much memory space is remaining for further pictures. In the
`latter case, the camera accesses such data to display utili
`zation to the camera user.
`Coordination between a camera and a computer is gen
`erally shown in patent application Ser. No. 988,517, “Elec
`tronic Camera with Memory Card Interface to a Computer”,
`which was ?led on Dec. 10, 1992 and assigned to common
`assignee with the present application. This application
`describes a camera that is specially shaped so as to plug
`directly into the memory card slot of a portable computer;
`the camera is then usable as a unit with the portable
`computer. The camera contains EPROM memory with code
`for in-camera white balance and gamma correction, and also
`stores the compiled code which is used (by the computer) to
`operate the camera and to process the images from the
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Sony, Ex. 1017, p.13
`
`
`
`5,477,264
`
`3
`located on a removable memory (e.g., a memory card) used
`with an electronic still camera. Known cameras use the
`removable memory to store images and data related to an
`individual image (date, color temperature of the illuminant,
`etc.) or data related to the camera (type of color ?lter array,
`sensor defect locations, etc.) which is transferred to an
`image reproducing device. Except for data about memory
`space, the communication is one way. A camera according to
`this invention, however, uses the removable media to trans
`fer software previously recorded on the memory card (such
`as improved camera ?rmware to process the image, or a
`special access code) to the camera. The advantageous effect
`is that the operation of the camera or playback device can be
`improved some years after the camera is purchased, by using
`an “enhanced” memory card containing improved software
`for operating the camera or playback device. Moreover, such
`two-way communication opens use of the storage device to
`other enhancement possibilities, e.g., to modify image data
`in selected ways, to modify camera performance for special
`situations, or to communicate non-captured images (special
`overlays) to the camera.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The prior art and the invention will be described in
`relation to the drawings, wherein
`FIG. 1 is a diagram of a known electronic imaging system
`incorporating an electronic camera for inputting images to a
`computer;
`FIG. 2 is a block diagram of an electronic camera con
`?gured according to the invention to receive and process
`enhancement ?les;
`FIG. 3 is a ?ow diagram of the steps involved in applying
`an enhancement algorithm to the camera of FIG. 2;
`FIG. 4 is a block diagram of the capture chain showing the
`points at which processing according to the invention may
`occur;
`FIG. 5 is a ?ow diagram of the steps involved in pro
`cessing an image sometime after storage in the camera of
`FIG. 2;
`FIG. 6 is a flow diagram of the steps involved in interrupt
`processing in the camera of FIG. 2;
`FIG. 7 is a flow diagram of the steps involved in control
`ling access to processing in the camera of FIG. 2;
`FIG. 8 is a pictorial illustration of the application of
`pre-existing overlay images to images captured by the
`camera of FIG. 2;
`FIG. 9 is a pictorial illustration of the use of an electronic
`view?nder in the camera of FIG. 2 to frame the captured
`image with respect to an overlay image;
`FIG. 10 shows an additional embodiment of an electronic
`camera con?gured according to the invention; and
`FIG. 11 shows an electronic imaging system incorporat
`ing the camera of FIG. 10 by means of a cable connection.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIIVIENTS
`
`Because electronic still cameras employing electronic
`sensors are well known, the present description will be
`directed in particular to elements forming part of, or coop
`erating more directly with, apparatus in accordance with the
`present invention. Elements not speci?cally shown or
`described herein may be selected from those known in the
`art.
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`FIG. 1 shows a known electronic still photography system
`useful for inputting images to a computer. The system
`includes an electronic camera 1 with an electronic sensing
`section 1a, a digital processing section 1b, a memory card
`slot 2, a removable memory card 3, a computer 4 with a
`built-in memory card reader 5, and a printer 6. Images are
`captured by the sensing section 1a in the camera 1, manipu
`lated in the processing section 1b, stored on the memory
`card 3, and transferred from the card 3 to the computer 4.
`The software used to control the sensing and processing
`sections 1a and 1b in the camera is entirely located in
`?rmware memory 7 in the camera 1 and cannot be updated
`without opening the camera cover, unsoldering the ?rmware
`memory circuit, and replacing it with a new circuit contain
`ing updated code. The application software used to control
`the playback operations of the computer is supplied, at least
`in part, by the camera manufacturer. It is stored in a
`computer hard drive 8, and downloaded to a computer
`random access memory (RAM) memory 9a, where the
`program is used by the central processing unit (CPU) 912. To
`update this software, the user must be provided with a new
`program on a ?oppy disc, which can be transferred to the
`hard drive 8. For example, the ?oppy disk may contain a
`“plug-in” module for an imaging software application pro
`vided with the camera. The plug~in contains both code to
`allow the images to be downloaded from the camera to the
`computer, and software code to create color images from the
`digitized camera image data. The software code implements
`algorithms for processing the color ?lter array sampled
`image data supplied by the camera. These algorithms
`include color interpolation, color balance, and color correc
`tion. A typical color interpolation algorithm is described in
`greater detail in U.S. Ser. No. 085,519, “Apparatus and
`method for adaptively interpolating a full color image uti
`lizing chrominance gradients”, ?led Jun. 30, 1993, which is
`assigned to Eastman Kodak Company.
`. The invention is based on the recognition that the storage
`device, i.e., the memory card 3 in FIG. 1, need not be
`primarily oriented to storage of data for downstream (i.e.,
`computer) processing. The storage device may additionally
`contain ?les (data, code, etc.) which are capable of upstream
`modi?cation of image data and camera performance. This
`provides an opportunity for affecting image data during the
`capture and manipulation stages prior to storage. Further
`more, these ?les may be accessed automatically by the
`camera, or selected by the user by means of appropriate
`intervention through the camera. Therefore, the removable
`image storage device is capable of two-way communication
`with the camera. That is, the card slot interface is not limited
`to “dumping” image data downstream to the removable
`storage device, but also allows communication from the
`removable storage device upstream to the camera as well.
`Inasmuch as the ?les are software which enhance the
`operation of the camera, the digital storage device may be
`viewed as a software-enhanced device, and the ?les are
`hereinafter referred to as enhancement data ?les.
`A number of advantages arise from such two-way com
`munication. Enhancement ?les can be included in the digital
`storage device which are not in themselves images. Image
`processing software, look-up tables, matrices, compression
`tables, dynamic range optimization tables, and other ?les
`capable of affecting the captured image data can be included
`in the digital storage device. Typical algorithms which can
`be modi?ed, or updated, thus include color ?lter array
`interpolation algorithms, noise reduction algorithms, edge
`sharpening algorithms, color reproduction algorithms, com
`pression algorithms, or dynamic range optimization algo
`
`Sony, Ex. 1017, p.14
`
`
`
`5,477,264
`
`5
`rithms. In this manner, the camera need only be capable of
`executing the functions and transformations contained in the
`storage device, and need not require the storage capacity to
`contain all information necessary to execute the processing
`of the image data. Such non-image ?les, which cause
`processing of the captured image data, may process the
`captured image data such that the most “true-to-scene”
`reproduction is achieved, or instead provide special effects
`of the kind which change the image into some desired
`alteration of the original scene.
`Another type of non-image ?le which can be contained in
`the digital storage device enhances the operation of the
`camera. Enhancement ?les can be included which control
`capture parameters such as exposure time, aperture setting,
`?ash range output, and so on. A particular set of control
`parameters can be used to tailor the capture parameters to the
`type of scene or subject being captured. This type of ?le
`might also allow diagnostic tests of the camera to insure
`proper operation and to detect fault conditions in the camera
`system. A further type of enhancement ?le contained in the
`storage device comprises pre-existing image data ?les, i.e.,
`?les with images not captured by the camera system. Such
`?les are accompanied by code which instructs the camera on
`how to merge the pre-existing image ?les with those cap
`tured by the camera. In this manner graphic overlays and
`text can be superimposed on the captured images by the
`camera. Proper execution of such a feature might include a
`user interface for selection of pre-existing image and overlay
`combinations, and also a view?nder which helps the user to
`compose the captured image so as to complement the
`selected overlay. Whether for image processing, enhanced
`performance, or pre-existing images, the algorithms them
`selves are conventional and not themselves part of this
`invention. However, where they are stored, and how they are
`used, is within the scope of the invention. In that connection,
`another function of the invention is to store replacement, or
`updated, software ?les in the digital storage device so that
`software updates can be added to the camera after its initial
`purchase by the subsequent purchase of appropriately-en
`hanced storage devices.
`The invention also concerns the sequence of execution of
`operations made possible by ?les contained in the image
`storage device. It is anticipated that the capture of images
`may occur in a rapid and/or random fashion. It may be
`undesirable to interrupt image processing to initiate another
`capture. If that is the case, processing above and beyond that
`which is necessary to achieve initial storage would not have
`to occur until the camera is no longer in a “ready” mode.
`That is, no processing takes place until the camera is no
`longer being used to capture images and is placed in an “off”
`or “sleep” mode. Alternatively, the camera could interrupt
`processing operations to carry out capture operations. Once
`the capture is complete, the camera can return to processing
`operations.
`FIG. 2 shows an electronic still camera capable of using
`a “software enhanced” storage device as taught by the
`invention. The electronic still camera is divided generally
`into an input section A and a compression and recording
`section B. The input section A includes an exposure section
`10 for directing image light from a subject (not shown)
`toward an image sensor 12. A ?ash unit 11 may be used to
`illuminate the subject. Although not shown, the exposure
`section 10 includes conventional optics for directing the
`image light through a diaphragm, which regulates the optical
`aperture, and a shutter, which regulates exposure time. The
`sensor 12, which includes a two-dimensional array of pho
`tosites corresponding to picture elements of the image, is a
`
`15
`
`25
`
`40
`
`45
`
`55
`
`60
`
`65
`
`6
`conventional charge~coupled device (CCD) using, e.g.,
`either well-known interline transfer or frame transfer tech
`niques. The sensor 12 is exposed to image light so that
`analog image charge information is generated in respective
`photosites. The charge information is applied to an output
`diode 14, which converts the charge information to analog
`image signals corresponding to respective picture elements.
`The analog image signals are applied to an A/D converter
`16, which generates a digital image signal from the analog
`input signals for each picture element.
`The digital signals are applied to an image buffer 18 for
`storing one or more images, albeit shown in FIG. 2 as a
`random access memory (RAM) with storage capacity for a
`plurality of still images. A programmed control processor 20
`generally controls the input section A of the camera by
`initiating and controlling exposure (by operation of the
`diaphragm and shutter (not shown) in the exposure section
`10), by controlling the ?ash unit 11, by generating the
`horizontal and vertical clocks needed for driving the sensor
`12 and for clocking image information therefrom, and by
`enabling the A/D converter 16 in conjunction with the image
`buffer 18 for each signal segment relating to a picture
`element. (The control processor 20 would ordinarily include
`a microprocessor coupled with a system tinting circuit.)
`Exposure control would typically be effected by using
`ambient light information from a photocell 19.
`Once a certain number of digital image signals have been
`accumulated in the image buffer 18, the stored signals are
`applied to a programmed digital signal processor 22, which
`controls the throughput processing rate for the compression
`and recording section B of the camera. The algorithms and
`other operating code used by the processor 22 are stored in
`the algorithm memory 28. The digital signal processor 22
`compresses each still image stored in the image buffer 18
`according to a known image compression algorithm, such as
`the well-known JPEG (Joint Photographic Experts Group)
`discrete cosine transformation-based compression algo
`rithm. The processor 22 applies a compression algorithm
`from the memory 28 to the digital image signals, and sends
`the compressed signals to a removable storage device via an
`interface 26. While a memory card 24 is shown in this
`system as the storage device, other devices may be used,
`such as a floppy disk magnetic medium, a small hard drive,
`or optical storage (in the latter cases, suitable conventional
`reading/writing apparatus would be provided in the camera,
`e.g., magnetic or optical read/write head, etc.) A represen
`tative memory card is a card adapted to the PCMCIA card
`interface standard, such as described in the PC Card Stan
`dard, Release 2.0, published by The Personal Computer
`Memory Card International Association, Sunnyvale, Calif,
`September 1991. Moreover, while compression is shown in
`the camera of FIG. 2, compression is not a necessary feature
`of the invention, and uncompressed data could be stored
`instead.
`The memory card 24 contains solid state memory 24a,
`such as Flash EPROM memory, which the card uses to store
`image data ?les. In addition, the memory card 24 contains
`additional memory to store enhancement data ?les 24b to
`modify the camera operation, or the images captured by the
`camera, or to provide pre-existing overlay images. These
`software enhancements are normally programmed by the
`card manufacturer before the card is sold to the user, and can
`be stored in the Card Information Structure (CIS) of the
`PCMCIA Format (as described in the aforementioned
`Release 2.0). The “software enhancement” code may be
`stored in separate non-erasable memory on the card 24, or in
`a portion of the memory space of the memory 24a used to
`store the image data.
`The input section A operates at a rate commensurate with
`
`Sony, Ex. 1017, p.15
`
`
`
`5,477,264
`
`10
`
`7
`normal operation of the camera while compression, which
`may consume more time, can be relatively divorced from the
`input rate. The exposure section 10 exposes the sensor 12 to
`image light for a time period dependent upon exposure
`requirements, for example, a time period between 1/1000
`second and several seconds. The image charge is then swept
`from the photosites in the sensor 12, converted to a digital
`format, and written into the image buffer 18. The repetition
`rate of the driving signals provided by the control processor
`20 to the sensor 12, the A/D converter 16 and the bu?°er 18
`are accordingly generated to achieve such a transfer. The
`processing throughput rate of the compression and recording
`section B may be determined by the character of an image,
`i.e., the amount of detail versus redundant information, and
`the speed of the digital signal processor 22.
`An operation display panel 30 is connected to the control
`processor 20 for displaying information useful in operation
`of the camera. Such information might include typical
`photographic data, such as shutter speed, aperture, exposure
`bias, color balance (auto, tungsten, ?uorescent, daylight),
`high/low resolution, low battery, low light, exposure modes
`(aperture preferred, shutter preferred), and so on. Moreover,
`other information unique to this type of camera is displayed.
`For instance, the memory card 24 would ordinarily include
`a directory signifying the beginning and ending of each
`stored image. This would show on the display 30 as either
`(or both) the number of images stored or the number of
`image spaces remaining, or estimated to be remaining. In
`certain applications, an electronic view?nder 29 would be
`included for displaying images either before or after storage.
`The control processor 20 also generates important infor—
`mation about the condition and capability of the memory
`card 24. Speci?cally, the interface 26 is queried for the
`presence of a card 24 and, if no card is connected, a “no
`card" display is produced on the operation display 30.
`Likewise, if a card is present but it is full of images, a “card
`full” display is produced. While not shown, the display 30
`may show the number of “remaining images.” If enhance
`ment ?les 24b are present on the card 24, they can be
`identi?ed on the operation display 30. For instance, perfor
`mance enhancements, such as special set-up for portraiture,
`can be identi?ed; processing enhancements, such as special
`effects, can be identi?ed; and overlay enhancements, such as
`birthday or holiday picture borders, can be identi?ed. User
`intervention in order to specify the several enhancement