Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 1 of 19 PageID #: 5779
`Case 5:19-cv-00036—RWS Document 136-9 Filed 11/18/19 Page 1 of 19 PageID #: 5779
`
`EXHIBIT 9
`
`EXHIBIT 9
`
`

`

`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 2 of 19 PageID #: 5780
`
`United States Patent (19)
`Parulski et al.
`
`USOO58284.06A
`Patent Number:
`11
`(45) Date of Patent:
`
`5,828,406
`Oct. 27, 1998
`
`54 ELECTRONIC CAMERA HAVING A
`PROCESSOR FOR MAPPING IMAGE PXEL
`SIGNALS INTO COLOR DISPLAY PIXELS
`
`O 456 369 A2 11/1991 European Pat. Off. ......... HO4N 5/14
`O 533 107 A2 3/1993 European Pat. Off. ......... HO4N 1/21
`WO 89/12939 6/1989 WIPO ............................ HO4N 5/232
`
`75 Inventors: Kenneth A. Parulski, Rochester;
`Timothy J. Tredwell, Fairport, both of
`N.Y.
`
`73 Assignee: Eastman Kodak Company, Rochester,
`N.Y.
`
`OTHER PUBLICATIONS
`
`“A Multimedia Color Camera Providing Multi-Format
`Digital Images” by Takuya Imaide, Toshiro Kinugasa,
`Yoshimichi Kudo, and Naoki Yamamoto. From IEEE Trans
`actions on Consumer Electronics, Aug. 1993, No. 3.
`
`21 Appl. No.: 367,399
`22 Filed:
`Dec. 30, 1994
`(51) Int. Cl. ............................................... H04N 5/225
`52 U.S. Cl. ............................................. 348/220; 348/333
`58 Field of Search ..................................... 348/220, 222,
`348/221,333,273, 391
`
`56)
`
`Primary Examiner Wendy Garber
`Attorney, Agent, or Firm-David M. Woods
`57
`ABSTRACT
`An electronic camera uses a relatively more complex digital
`image processing technique in a still image mode to produce
`high quality Still images, and a relatively more simple image
`processing technique in a motion preview mode to produce
`preview images of acceptable quality prior to initiation of
`References Cited
`the still image mode. The more complex digital technique is
`U.S. PATENT DOCUMENTS
`done in Software in a general purpose processor Section 35,
`while the more Simple digital technique is implemented in a
`4,541,010 9/1985 Alston ..................................... 348/333
`fixed digital circuit in an application Specific integrated
`4,691.253 9/1987 Silver ...
`... 360/33.1
`circuit 27, which also implements timing and control func
`4,714,963 12/1987 Vogel.
`... 348/220
`tions. The motion preview mode uses a shorter image
`4,876,590 10/1989 Parulski ...
`... 348/333
`E. 3.19. Shita - - - - -
`- - - - is: readout period than the still mode and further involves
`5. 440343 8/1995 Pa.rulski et al...
`... 348f220
`mapping image Sensor pixels into a fewer number of color
`5,444,483 8/1995 Maeda ..............
`34s.220
`display pixels on a color LCD display 10. The mapping
`5,943,335 2/1996 Parulski et al.......................... 348,273
`further converts color pixel signals from a mosaic array into
`a different color pattern on the color LCD display 10.
`FOREIGN PATENT DOCUMENTS
`O 405 491 A2 1/1991 European Pat. Off. ......... HO4N 5/91
`
`15 Claims, 10 Drawing Sheets
`
`M
`
`22 24
`
`-----
`
`2O
`
`fo
`
`1 MAGE C O)
`L/GH7
`y
`7O
`
`MAGE
`SENSOR
`
`H2YV2 FDG
`
`ANALOG
`GAN 8
`CDS
`
`34
`
`AAD
`
`12 fa
`
`--m-
`COLOR LCD
`DSPLAY
`
`CONTROLS
`
`18
`
`ZOOM Foc
`US
`44, 46 APERTURE
`It
`48
`LENS MOTOR
`DRIVERS
`
`5O
`
`3O
`2s
`
`SENSOR
`
`SESSF
`CIRCUIT
`
`54
`
`PHOTOSYSTEMS
`INTERFACE
`
`FLASH CONTROL
`27-1 cowroLSECTIOw
`
`53
`
`SENSOR
`BiA" | 29M28MBE, CAF
`15
`- -
`- -
`- -
`LCDPIXEL
`LCR 90.9F1-Mux
`REMAPPER
`ADJUST
`-
`
`64
`
`IMAGE
`STATESTICS
`
`62 63 N5s
`
`STATUS
`8.
`CONTROLS
`
`CONTROL
`NTERFACE
`52
`HIGH SPEED
`INTERFACE
`
`
`
`35
`
`33
`)
`DRAM
`MEMORY
`
`? 42
`
`43
`f
`EPROM
`"FRMWARE"
`
`MEMORY
`
`L PROCESSOR SECrow
`
`36
`(-
`DIGITAL
`acces
`
`Y---
`
`PCMCIA
`iNTERFACE
`REMOVABLE
`PCMC A
`MEMORY CARD
`
`4O
`
`26
`
`

`

`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 3 of 19 PageID #: 5781
`
`U.S. Patent
`
`Oct. 27, 1998
`
`Sheet 1 of 10
`
`5,828,406
`
`
`
`
`
`
`
`
`
`Greg regr
`egre Greg
`Refers FIG B
`Greg Regr
`
`

`

`9SaC
`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 4 of 19 PageID #: 5782
`2m
`w
`
`1o5.U
`
`
`
`a.M_W4,5%:
`
`130282:9523%255mS,_emmW»Dvmm;mm
`1|81|.1I,1,1,.:1I.11!1w&mm-
`:m2.3III-.5.2Eam2.55ooézqmw8._5.8vmwkmomzmm..$63
`.5528MUEEEEdS8.5.25:93...nmgBEES.
`
`
`
`msmzmaoszmm3%:
`
`:08zoommemzmmmemzmm.208mm3man.
`
`
`
`
`
`
`
`42:28Pw111'.1111.lllllll1ll..mm_855:2.mmSENMW$§®EkwmmmaSwim:2:
`0.a111mm353=$33wP99w1:muS.vmmm
`SDDII19:“..Q30528m8ooh.m>N:W3N\o2.
`
`
`onE.38$on
`3352mmj35%msmmmbsmm_aa_
`mum;mo9__%mx225:34mmmmqimmfiwmmwawmflwmoss.m2“...
`
`
`
`
`
`11$01.0000.—..EXED04_ImmmowzwmWVIllmn]!-I..I111I1.Vmm...
`
`
`«:3on$.ngwagon.vm$19.11-1l|$2514mvvwmisSo2.
`
`Eozmz.W585%;.mommmoommzmmqumizE952Nb\h\
`
`
`
`
`
`
`
`

`

`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 5 of 19 PageID #: 5783
`
`U.S. Patent
`
`Oct. 27, 1998
`
`Sheet 3 of 10
`
`5,828,406
`
`
`
`66 68
`
`A/G 3A
`
`
`
`V2
`
`A/G 3B
`
`No. HORIZONTAL REGISTER
`
`76
`
`El-in a horizontal Register
`
`CHANNEL 2
`
`HIA H1B H2
`
`CCD LINES
`
`
`
`
`
`
`
`G B G B G B G
`IEEEEEEE FIG 4
`LINE 4 B G B G B G B G
`GRG RIGRG R
`LINE 6 BG B G B G B G
`G R G R G R G R
`LINE 8 B G B G B G B G
`
`
`
`

`

`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 6 of 19 PageID #: 5784
`
`U.S. Patent
`
`Oct. 27, 1998
`
`Sheet 4 of 10
`
`5,828,406
`
`Q s
`
`C s
`
`C
`
`ar
`
`Q) s
`
`N
`
`S
`
`Q
`
`S
`
`s
`
`s
`
`S
`
`s
`
`

`

`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 7 of 19 PageID #: 5785
`
`U.S. Patent
`
`Oct. 27, 1998
`
`Sheet 5 of 10
`
`5,828,406
`
`Q
`
`s &
`
`'m
`
`Q "a
`
`S
`
`is Cs
`
`N Q. N. e
`N
`SV
`
`Q
`O
`
`N S S
`
`S.
`
`S.
`
`S
`
`s
`
`s\-v- /
`
`

`

`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 8 of 19 PageID #: 5786
`
`U.S. Patent
`
`Oct. 27, 1998
`
`Sheet 6 of 10
`
`5,828,406
`
`5E See
`st at
`
`O
`\
`N
`N S S
`
`J
`
`(s
`S
`
`(s
`
`

`

`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 9 of 19 PageID #: 5787
`
`U.S. Patent
`
`Oct. 27, 1998
`
`Sheet 7 of 10
`
`5,828,406
`
`CCD L NES
`
`ELMINATED WA
`LNE 6 BGBGBGBG "FAST DUMP"
`
`LINE 3
`
`LINE 4
`
`
`
`LINE 15 GRG RIGRG R
`LINE 16 BG BG BG B G
`
`LINE 7
`
`LNE 8
`LINE 9
`
`LNE 20
`
`

`

`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 10 of 19 PageID #: 5788
`
`U.S. Patent
`
`Oct. 27, 1998
`
`Sheet 8 of 10
`
`5,828,406
`
`8 9/-/
`
`
`
`

`

`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 11 of 19 PageID #: 5789
`
`U.S. Patent
`
`Oct. 27, 1998
`
`Sheet 9 of 10
`
`5,828,406
`
`
`
`CCD
`COUNTERS
`
`
`
`AEG, SELECT
`
`3 Y 24
`(16-6 ) B/7
`REGISTERS
`
`16 MSBS
`TO PROCESSOA
`/WTERFACE
`
`6 B17 FULL ADDEAP
`WITH 16 B/7 HALFADDEAP
`
`
`
`
`
`COUMTCL K
`
`A/G 9A
`
`
`
`
`
`
`
`REG, SELECT
`
`6
`(16-6) BIT
`AEGISTERS
`
`16 MSBS TO
`PROCESSOR
`//WTEFACE
`
`6B/T FULL ADDEF
`WITH 16 B/TAALA
`ADDER
`
`
`
`
`
`8 MSBs
`( G ONLY)
`
`DETA/L CLAK
`
`A/G 9B
`
`

`

`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 12 of 19 PageID #: 5790
`
`U.S. Patent
`
`Oct. 27, 1998
`
`Sheet 10 of 10
`
`5,828,406
`
`CCD
`IMAGE
`DATA
`
`F/G fC)
`
`
`
`CFA
`/WTERPOLATE
`
`COLOR
`MATRIX
`
`
`
`
`
`TONE
`coRRECT
`
`DE7A/L
`EYTRACT
`
`IMAGE
`EDGE
`EWHAWCE O COMPRESSION
`
`
`
`APGE9
`/MAGE
`Af STOPAGE
`-—4 —-
`
`zoMN zooMoUT PREVIEW CAPTURE
`I
`I
`I
`I- 96
`SEWSODAF/VERS
`
`
`
`94
`
`SEWSOA. TIM/WG
`C/ACU/7
`
`23
`
`
`
`

`

`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 13 of 19 PageID #: 5791
`
`1
`ELECTRONIC CAMERA HAVING A
`PROCESSOR FOR MAPPING IMAGE PIXEL
`SIGNALS INTO COLOR DISPLAY PIXELS
`
`5,828,406
`
`25
`
`35
`
`40
`
`FIELD OF THE INVENTION
`The invention pertains to an electronic Still camera for
`composing and capturing Still images, and, more
`particularly, to an electronic camera having a “motion”
`mode for previewing a Scene and a “still” mode for capturing
`a particular image in the Scene.
`BACKGROUND OF THE INVENTION
`Consumer camcorders which include the capability of
`recording analog motion and/or Still images on 8 mm or
`15
`VHS videotape have been developed by a number of com
`panies. Motion images are recorded in the Same manner as
`in any Standard camcorder. These cameras include a single
`chip charge coupled device (CCD) Sensor having a color
`filter array that provides a Spatially color-Sampled image. To
`record Still images, the user pushes a “still capture' button
`at the desired instant. The image obtained from the CCD
`Sensor is temporarily Stored in a digital memory. The image
`is then read from the memory and recorded onto the Video
`tape. Some camcorders include color liquid crystal displayS
`(LCD), which are also spatially color-sampled devices.
`Some are relatively large, for example, ranging from
`approximately 2.5" to 4" in diagonal. Such a display is used,
`instead of a normal eyepiece Viewfinder, to allow the user to
`properly frame the Subject and View the imageS as they are
`being recorded. It is also used to View the recorded images
`as the Videotape is played back.
`FIG. 1A shows a typical color LCD display, in which the
`liquid crystal material is trapped between an upper glass
`plate 1 and a lower glass plate 2. The upper plate 1 has a
`common transparent electrode 3 and an array 4 of color
`filters surrounded by a black mask 5. The lower plate 2
`includes an array 6 of transparent pixel electrodes juxta
`posed underneath the array 4 of color filters. Individual pixel
`electrodes are activated via thin film transistors (TFT)7 that
`are controlled from a Video signal on the Source lines 8 and
`a Scanning Signal on the gate line 9. The LCD display
`includes the usual polarizer layers (not shown) on the glass
`plates 1 and 2, Such that activation of Selected transparent
`pixel electrodes allows light to pass through the correspond
`ing color filters and reflect to the viewer, thereby creating a
`color image. A typical LCD display Such as the Epson LB
`2F-BC00, manufactured by Seiko-Epson Company, Japan,
`has about 240 lines of pixels and about 300 pixels per line,
`with an image aspect ratio of 4:3. Such an aspect ratio allows
`the entire area of the image obtained from the 4:3 aspect
`ratio NTSC format CCD sensor to be displayed on the LCD
`Screen, So that the LCD Screen composition will be the same
`as the image that is recorded by the camcorder NTSC format
`recorder, for later display on an NTSC format television
`display. Note that because the LCD has only 240 lines of
`pixels, the interlaced NTSC Signal is displayed using a
`“repeat field” technique, where both the odd and even fields
`from the NTSC format sensor are displayed using the same
`lines of pixels on the LCD. This LCD, like most commer
`cially available LCDS, has “rectangular pixels, rather than
`Square pixels, where the distance between pixels in the
`horizontal direction is for example % the distance in the
`vertical direction. The LCD pixels are overlaid with a
`diagonal RGB stripe pattern as shown in FIG. 1B.
`In camcorders, the processing for both the Still images and
`the motion images is identical. Such processing is normally
`
`65
`
`45
`
`50
`
`55
`
`60
`
`2
`implemented by hardwired analog integrated circuits,
`although camcorders which use digital image processing
`integrated circuits have been Such camcorders convert the
`signal from the CCD sensor into an NTSC composite or
`component format Signal, which is provided to a Video
`recording Subsystem or a video output jack. The color LCD
`display includes circuitry to decode the NTSC composite or
`component Signal back into Spatially Subsampled RGB
`signals to drive the individual RGB pixels on the LCD
`SCSO.
`In a System oriented toward Still photography, and in
`particular a digital Still System, it would be desirable to avoid
`the necessity of generating an NTSC format Signal in order
`to reduce the complexity of the required circuitry. In a totally
`digital System, that is, both the recording and display chan
`nels are digital, it is further desirable to minimize incom
`patibility between the channels. The problem is to achieve
`these objective in an architecture that minimizes cost and
`complexity and maximizes user handling.
`SUMMARY OF THE INVENTION
`This problem is solved according to the invention by a
`number of features. In one aspect, the electronic camera is
`operable in a still image mode according to a relatively more
`complex digital image processing technique to produce high
`quality Still images, and in a motion preview mode accord
`ing to a relatively more Simple digital image processing
`technique to produce a preview image of acceptable quality
`prior to initiation of the Still image mode. Such an archi
`tecture is particularly adapted to mapping an array of color
`image pixels from a Sensor into an array of color display
`pixels on an LCD display comprising discrete LCD display
`pixels fewer in number than image sensor pixels. In that
`case, a relatively simple digital processing technique com
`bines Same-colored image pixel Signals into a fewer number
`of pixels that correspond to the arrangement of the color
`display pixels.
`The advantage of the invention is that the two modes can
`be tailored for a relatively low quality “motion” mode and
`a much higher quality “still” mode. The motion mode
`images from the CCD Sensor are processed by a hardwired
`digital Signal processing circuit that generates low
`resolution, Spatially Subsampled digital image data which
`can directly drive the relatively low resolution LCD display.
`This reduces the complexity and clock frequency of the
`required circuitry, compared to generating an NTSC format
`Signal, as is normally done in the prior art. The Still mode
`image from the CCD Sensor is processed by a general
`purpose processor (CPU) which executes an image process
`ing Software program in order to produce a high quality
`digital still image.
`BRIEF DESCRIPTION OF THE DRAWINGS
`The invention will be described in relation to the
`drawings, whereon
`FIGS. 1A and 1B show the structure and color filter
`pattern of a known color liquid crystal display (LCD);
`FIG. 2 is a block diagram of an electronic camera incor
`porating dual modes for composing and capturing a Still
`image according to the invention;
`FIGS. 3A and 3B are diagrams of progressive Scan image
`sensors useful with the camera of FIG. 2;
`FIG. 4 is a diagram of the Bayer color filter geometry for
`the sensor used with the camera of FIG. 1;
`FIG. 5 shows the line timing for the still mode of
`operation;
`
`

`

`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 14 of 19 PageID #: 5792
`
`5,828,406
`
`3
`FIGS. 6A and 6B show the line timing for the preview
`mode of operation;
`FIG. 7 shows the Special line skipping pattern used in the
`preview mode,
`FIG. 8 shows further detail of the preview mode process
`ing circuit shown in FIG. 2;
`FIGS. 9A and 9B show further detail of the image
`Statistics processor shown in FIG. 2;
`FIG. 10 shows one example of still mode image process
`Ing,
`FIG. 11 shows the effect of pixel mapping from a sensor
`to an LCD display, each having different aspect ratios, and
`FIG. 12 shows an enhancement to the block diagram of
`FIG. 2 in which a different sensor clock frequency is used in
`each of the dual modes.
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENT
`Ablock diagram of a camera incorporating dual modes of
`processing according to the invention is shown in FIG. 2.
`The camera includes an electronic color display, for
`example, a color liquid crystal (LCD) display 10 of the type
`shown in FIG. 1A, and a user control Section 12 having a
`number of user control buttons, including Zoom buttons 14,
`a preview button 15 and a capture button 16. To take a still
`picture, the user turns on the camera (using a power Switch
`(not shown), which may be automatically enabled when the
`user depresses the Zoom buttons 14 or the preview button 15,
`or partially depresses the capture button 16). The user
`composes the picture by depressing the "Zoom in” or "Zoom
`out” buttons 14, and by adjusting the position of the camera,
`while observing the display image. When the user is Satisfied
`with the composition on the color LCD display 10, the user
`depresses the capture button 16. The camera then captures a
`Single Still image, firing a flash 18 if necessary when the
`ambient illumination level is low. The still image is focused
`upon an image Sensor 20 by a motor driven Zoom lens 22.
`The intensity of the image light upon the Sensor 20 is
`regulated by a motor-driven, variable, mechanical aperture
`24, while exposure time is regulated electronically by appro
`priate clocking of the sensor 20. The still image from the
`image Sensor 20 is processed and digitally Stored on a
`removable memory card 26.
`Control of the Sensor is provided by a timing and control
`Section 27, which is an application Specific integrated circuit
`(ASIC) with processing and timing functions, for both
`capture and preview operating modes. For instance, the
`timing and control Section 27 includes a Sensor timing
`circuit 28 for controlling the image Sensor functions. The
`Sensor timing circuit 28 provides the Signals to enable Sensor
`drivers 30, which provide horizontal clocks (H1, H2), ver
`tical clocks (V1,V2), as well as a signal FDG for activating
`a drain Structure on the Sensor 20. The output of the image
`Sensor 20 is amplified and processed in an analog gain and
`Sampling (correlated double sampling (CDS)) circuit 32, and
`converted to digital form in A/D converter stage 34. The A/D
`output signal is provided to a processor Section 35, which
`includes a digital processor 36 for temporarily Storing the
`still images in a DRAM memory 38. The digital processor
`36 then performs image processing on the Still images, and
`finally Stores the processed images on the removable
`memory card 26 via a memory card interface circuit 40,
`which may use the PCMCIA 2.0 standard interface. An
`EPROM memory 42 is used to store the firmware which
`operates the processor 36. The components of the processor
`35 are interconnected through a data bus 43, which also
`connects to the timing and control Section 27 and to the card
`interface 40.
`
`4
`The motor-driven Zoom lens 22 includes a Zoom motor
`44, a focus motor 46, and an aperture motor 48 (all con
`trolled by lens motor drivers 50). The timing and control
`Section 27 further includes a control interface 52 connected
`to the lens motor drivers 50 and to a flash control circuit 53
`via a photosystem interface block 54, which controls the
`operation of the Zoom lens 22 and the flash 18. The lens
`Zoom position is controlled by the photosystem interface
`block 54 based on position input from the Zoom control
`buttons 14 through a user status and control section 55. The
`focusing, exposure control, and white balance is done
`automatically, as is typically the case in consumer camcord
`ers. This is done by computing "image Statistics' in an image
`statistics processor 60 in the real-time ASIC (timing and
`control Section 27) as preview images are continuously read
`out of the image Sensor 20. The computed values are then
`used by a program implemented in the digital processor 36,
`which decides how to adjust the focus motor, aperture,
`analog gain control, and analog white balance controls via
`the control interface 52 and the photosystems interface 54 on
`the ASIC timing and control section 27. Although the digital
`processor 36 and the control interface 52 are shown as being
`within two Separate Sections, in Some implementations the
`Same component could be used to perform both of these
`functions (as well as other of the recited functions). Sensor
`image data is passed to the processor Section 35 through a
`high speed interface 56 in the timing and control section 27.
`The Sensor image data is also directed to the color LCD
`display 10 through a preview mode processing circuit 58.
`The timing and control section (ASIC) 27 is operable in
`two modes, a relatively low quality “motion' mode and a
`much higher quality “Still” mode. In the motion mode,
`images from the Sensor 20 are processed by the preview
`mode processing circuit 58; in the Still mode, images from
`the sensor 20 are processed in the processor 35. The pro
`ceSSor 35 is a Software-driven digital processing System that
`is slower than the ASIC 27. The preview mode processing
`circuit 58 is a hardwired digital Signal processing circuit
`(part of the ASIC 27) that generates low resolution, spatially
`Subsampled digital image data which can directly drive the
`relatively low resolution color LCD display 10. This reduces
`the complexity and clock frequency of the required circuitry,
`compared to generating an NTSC format Signal, as is
`normally done in the prior art. The preview mode processing
`circuit includes a pixel remapper 62 for mapping the greater
`number of image pixels from the sensor 20 into the lesser
`number of display pixels (i.e., corresponding to the array 6
`of transparent pixel electrodes in FIG. 1) in the color LCD
`display 10. The color Saturation of the remapped pixels is
`then adjusted in a color adjustment circuit 63 and its output
`is applied to a multiplexer 64. The multiplexer 64 selects
`image data either from the preview mode processing circuit
`58, producing a preview image, or from the high Speed
`interface 56, which allows for suitably preprocessed viewing
`of Stored images.
`In this camera, the image processing used to create the
`preview mode is done in the timing and control ASIC 27,
`Since the processing must be done rapidly. About 60 images
`per Second are processed in preview mode. However, Since
`the image quality of the displayed image is limited by the
`resolution and color gamut of the LCD screen of the LCD
`color display 10, there is no need for elaborate image
`processing. Therefore, Simple “preview mode” image pro
`cessing is performed in a fixed digital circuit embedded in
`the preview mode processing circuit 58 (which is part of the
`ASIC). The quality requirements for the still mode are much
`greater, Since these images will be downloaded to a
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`

`

`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 15 of 19 PageID #: 5793
`
`5,828,406
`
`15
`
`6
`The Sensor 20 uses a progressive Scan readout method,
`which allows the entire image to be read out in a Single Scan.
`The accumulated or integrated charge for the photodiodes
`comprising the pixels 66 is transported from the photoSites
`to light protected vertical (parallel) registers 68 by applying
`a large positive Voltage to the phase-one vertical clock (V1).
`This reads out every row, or line, into the Vertical registers
`68. The image pixel charge is then transported from the
`vertical registers 68 to the horizontal register 70 by two
`phase clocking of the vertical clocks (V1,V2). Between the
`Vertical and horizontal registers is the fast dump Structure
`72, which is further described in the Performance Specifi
`cation document for the KAI-0400CM sensor. By setting a
`Suitable positive potential on a fast dump gate line FDG,
`charge from the row of pixel values currently adjacent to the
`fast dump structure 72 is transferred from the CCD directly
`into the sensor Substrate 74 rather than to the horizontal
`register 70. This dump, or line clear, is accomplished during
`the vertical-to-horizontal transfer time. When properly con
`trolled by the sensor timing circuit 28, the fast dump
`structure 72 allows lines of charge to be eliminated.
`The timing and control Section 27 operates the electronic
`camera shown in FIG. 2 in the two aforementioned modes,
`including a first “still” mode wherein all rows of image pixel
`charge corresponding to each line are progressively read out
`through the horizontal register 70 during a Single Scan, and
`a Second “motion' mode wherein Some of the rows of image
`pixel charge corresponding to Some lines are eliminated
`through the fast dump Structure 72 prior to readout. AS
`applied to the embodiment of FIG. 2, the first mode corre
`sponds to a high quality Still imaging mode while the Second
`mode corresponds to a special "line Skipping” mode for
`driving the color LCD display 10. In the second mode, the
`timing and control Section 27 controls the fast dump Struc
`ture 72 to eliminate two or more consecutive lines of image
`charge from the image Sensor 20 for every one or more lines
`of image charge that are transferred to the horizontal register
`70 for thereby generating a pattern of lines (shown in FIG.
`7) suitable for driving the LCD display in a “repeat field”
`mode. An appropriate Video Signal which displays the entire
`3:2 aspect ratio Sensor image on the 4:3 aspect ratio LCD,
`without introducing geometric distortion, is generated by
`alternately eliminating two or four consecutive lines of
`image charge for every pair of lines of image charge that are
`transferred to the horizontal register 70.
`The sensor timing circuit 28 is controlled by the control
`interface 52 to provide the clock signals V1,V2, H1, H2, and
`the gate Signal FDG according to the two modes of opera
`tion. The timing signals for the first mode are shown in FIG.
`5; those for the second mode are shown in FIG. 6a and 6b.
`The two-phase cycling of signals V1 and V2 control the
`transfer of lines of image pixel charge from the vertical
`registers 68 to the horizontal register 70. The two-phase
`cycling of Signals H1 and H2 control the transfer of a stream
`of color pixel signals from the horizontal register 70 to
`Subsequent circuits in the camera. The level of the Signal
`FDG determines whether the image charge is dumped to the
`substrate 74 or transferred to the horizontal register 70.
`When the sensor 20 is clocked using the first timing mode
`shown in FIG. 5 for all lines of the sensor, all lines of the
`Sensor are clocked out, one after the other, through the
`horizontal register 70, processed in Subsequent camera
`circuitry, and stored in the removable memory 26. This
`timing mode provides a high quality progressive Scan Still
`image, but may take /30 Second or longer to read out the Still
`image. Such timing, however, is acceptable for Still mode
`usage, and, as mentioned before, does not require unusually
`
`S
`computer, and may be displayed on a high resolution CRT
`display, or printed on a high quality thermal printer.
`Therefore, the digital image processing must be more elabo
`rate. By using the processor 36 to implement Software
`procedures Stored in the firmware memory 42, complex
`procedures can be implemented. These procedures can take
`Several Seconds to complete, Since real-time operation is not
`required. Use of firmware-stored software allows the still
`mode image processing to be upgraded without requiring a
`new ASIC design. In effect, what happens is that a relatively
`leSS complex digital image processing technique is used in
`the motion preview mode, but at a higher data rate, and a
`relatively more complex digital image processing technique
`is used in the Still mode, but at a slower data rate.
`Since the update rate, that is, the number of images that
`need to be supplied per unit time, is different for the still
`mode than for the motion mode, it is beneficial to use
`different clock frequencies for the different modes of opera
`tion. For example, as shown in FIG. 12, a System oscillator
`100 produces a 12 mHz clock frequency for use in the
`motion mode to obtain more updates/Second (e.g., 60 images
`per second), while a divider 102 divides by two to provide
`a 6 mHz clock frequency for the still mode. The lower
`frequency allows more time to accurately position the clamp
`and Sample pulses So as to avoid CCD output Signal tran
`25
`Sitions. This increases noise immunity in the Still mode. A
`multiplexer 104 is enabled by the control interface 52 to
`determine which clock frequency is applied to the Sensor
`timing circuit 28. Though not specifically shown, the
`changed timing is also communicated to the A/D Stage 34
`and other timing and control circuits.
`The Sensor 20 is a progressive Scan interline image Sensor
`having a noninterlaced architecture, as shown in more detail
`in FIG. 3A. The sensor comprises a two-dimensional array
`of photoSites 66, e.g. photodiodes, arranged in rows and
`columns of image pixels, a plurality of Vertical registers 68
`adjacent photoSite columns for transferring rows of image
`pixel charge from the photoSites 66 to a horizontal register
`70 for readout responsive to clock signals from the sensor
`drivers 30, and a charge drain (specifically, a fast dump
`structure 72) interposed between the output of the vertical
`registers 68 and the horizontal register 70 for eliminating
`complete rows of image pixels at a time from the image
`Sensor 20. A preferred image Sensor is the Kodak model
`CCD KAI-0400CM image sensor, which has approximately
`512 active lines with approximately 768 active pixels per
`line and an image aspect ratio of 3:2. This Sensor is
`described in a Performance Specification document avail
`able from Eastman Kodak Company, Rochester, N.Y. Each
`pixel is 9 microns “Square', Since both the vertical and
`horizontal distances between the centers of adjacent pixels
`are 9 microns. The 3:2 image aspect ratio of the CCD sensor,
`although wider than the 4:3 aspect ratio of the display, is
`considered to be a preferred aspect ratio for Still
`photography, in that the Standard 35 mm film format, and
`Standard 4R (4x6) prints also have a 3:2 image aspect ratio.
`The Sensor uses a color filter array pattern known as the
`“Bayer checkerboard'pattern, described in U.S. Pat. No.
`3,971,065, which is shown in FIG. 4. Such a color filter array
`is characterized by a mosaic pattern in which the filter colors
`alternate in both line and column directions. In the normal
`operating mode, all of the image pixels on the Sensor are
`transferred as color image pixels to the horizontal register
`70, which delivers a stream of color pixel signals to the
`analog gain and CDS circuit 32 (see FIG. 2). The color pixel
`Signals are Subsequently converted to digital pixel Signals in
`the A/D converter 34.
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`

`

`Case 5:19-cv-00036-RWS Document 136-9 Filed 11/18/19 Page 16 of 19 PageID #: 5794
`
`5,828,406
`
`8
`and 3:2 aspect ratio) is to have an LCD display with
`512x768 pixels and the same aspect ratio and color filter
`pattern. However, this would be a custom, costly LCD.
`Instead, LCDS have fewer display pixels than image pixels
`on the image Sensor, normally have a 4:3 image aspect ratio,
`and use the diagonally striped RGB pattern shown in FIG.
`1B. In this discussion, an LCD pixel array of about 240
`lineSX312 pixels per line is assumed.
`Therefore, the Sensor pixels are processed in a "pixel
`mapping circuit', Such as the LCD pixel remapper 62. A
`block diagram of this circuit is shown in FIG. 8. Note that
`there are 768/2=384 green or red/blue pixels per line on the
`sensor (see FIG. 4). There are about 300/3=100 green, red,
`or blue pixels per line on the LCD (see FIG. 1B). Thus, there
`are approximately 4 as many LCD pixels per line (per color)
`as there are Sensor pixels per line. Therefore, the basic plan
`is to combine Same-colored image pixels into a fewer
`number of intermediate, combination pixel Signals that are
`then mapped into the color display pixels. For instance, a
`Simple “pixel mapping circuit maps four green Sensor pixel
`Signals into one green LCD pixel for one line by Summing
`two green Sensor values, Spaced apart by 4 CCD pixel
`positions, in the green pixel Summer 76 and dividing by two
`via bit-shift wiring. The necessary delay is provided by the
`registerS 82 clocked at one fourth the pixel rate, further
`delayed by one pixel clock. It also maps four red Sensor
`pixels into one LCD pixel in the same manner (using the
`red/blue summer 78), and also stores this value in a 100 pixel
`FIFO 80. The FIFO 80 compensates for the fact that the
`Sensor has line Sequential red and blue pixels, by Supplying
`blue pixels on the red Sensor lines, and red pixels on the blue
`Sensor lines. Four pixel delays are provided by the registers
`82 clocked at one fourth pixel rate (CLK/4). The mapping
`process is basically, therefore, a proceSS which, in its Sim
`plest form, involves averaging of Signals to produce a
`Smaller number of output color pixels than input color
`pixels. (The CFA interpolation algorithm discussed in ref
`erence to FIG. 10, on the other hand, produces a larger
`number of "output color pixels than input color pixels.)
`Alternate groups of 2 or 4 lines of Sensor values are
`discarded during preview mode by using the fast dump gate
`shown in FIG. 3A, as described in connection with the “line
`skipping” mode. This allows the Sensor readout time to be
`decreased by more than 72 during the preview mode.
`Another feature of the design is that by removing the
`NTSC rate driving