(19) 1 Europaisches Patentamt
`
`European Patent Office
`
`Office europeen des brevets
`
`111111111111111111111111111111111111111111111111111111111111111111111111111
`
`(11)
`
`EP 0 720 388 A2
`
`(12)
`
`EUROPEAN PATENT APPLICATION
`
`(43) Date of publication:
`03.07.1996 Bulletin 1996/27
`
`(21) Application number: 95120480.9
`
`(22) Date of filing: 22.12.1995
`
`(84) Designated Contracting States:
`DE FR GB
`
`(30) Priority: 30.12.1994 US 367399
`
`(71) Applicant: EASTMAN KODAK COMPANY
`Rochester, New York 14650 (US)
`
`(72) Inventors:
`• Parulski, Kenneth A.,
`c/o Eastman Kodak Co.
`Rochester, New York 14650-220 (US)
`
`(51) Int. Cl.6: H04N 9/04
`
`• Tredwell, Timothy J.,
`c/o Eastman Kodak Co.
`Rochester, New York 14650-220 (US)
`
`(74) Representative: Schmidt, Peter, Dipl.-lng. et al
`KODAK Aktiengesellschaft
`Patentabteilung
`70323 Stuttgart (DE)
`
`(54)
`
`An electronic camera having dual modes for previewing and capturing still images
`
`(57)
`An electronic camera uses a relatively more
`complex digital image processing technique in a still
`image mode to produce high quality still images, and a
`relatively more simple image processing technique in a
`motion preview mode to produce preview images of
`acceptable quality prior to initiation of the still image
`mode. The more complex digital technique is done in
`software in a general purpose processor section (35),
`while the more simple digital technique is implemented
`in a fixed digital circuit in an application specific inte-
`
`grated circuit (27), which also implements timing and
`control functions. The motion preview mode uses a
`shorter image readout period than the still mode and fur(cid:173)
`ther involves mapping image sensor pixels into a fewer
`number of color display pixels on a color LCD display
`(1 0). The mapping further converts color pixel signals
`from a mosaic array into a different color pattern on the
`color LCD display (10).
`
`10
`
`C\1 <
`co
`co
`M
`0
`C\1
`I'
`0
`c.
`w
`
`FIG. 2
`
`Rfc'{f~~LE
`MEP.«JRYCARO
`
`26
`
`Printed by Rank Xerox (UK) Business Services
`2.12.4/3.4
`
`HTC Exhibit 1039
`
`

`
`EP 0 720 388 A2
`
`2
`
`Description
`
`FIELD OF THE INVENTION
`
`The invention pertains to an electronic still camera
`for composing and capturing still images, and, more par(cid:173)
`ticularly, to an electronic camera having a "motion" mode
`for previewing a scene and a "still" mode for capturing a
`particular image in the scene.
`
`BACKGROUND OF THE INVENTION
`
`Consumer camcorders which include the capability
`of recording analog motion and/or still images on 8mm
`or VHS videotape have been developed by a number of
`companies. Motion images are recorded in the same
`manner as in any standard camcorder. These cameras
`include a single chip charge coupled device (CCD) sen-
`sor having a color filter array that provides a spatially
`color-sampled image. To record still images, the user
`pushes a "still capture" button at the desired instant. The
`image obtained from the CCD sensor is temporarily
`stored in a digital memory. The image is then read from
`the memory and recorded onto the videotape. Some
`camcorders include color liquid crystal displays (LCD),
`which are also spatially color-sampled devices. Some
`are relatively large, for example, ranging from approxi(cid:173)
`mately 2.5" to 4" in diagonal. Such a display is used,
`instead of a normal eyepiece viewfinder, to allow the user
`to properly frame the subject and view the images as
`they are being recorded. It is also used to view the
`recorded images as the videotape is played back.
`Figure 1 A shows a typical color LCD display, in
`which the liquid crystal material is trapped between an
`upper glass plate 1 and a lower glass plate 2. The upper
`plate 1 has a common transparent electrode 3 and an
`array 4 of color filters surrounded by a black mask 5. The
`lower plate 2 includes an array 6 of transparent pixel elec(cid:173)
`trodes juxtaposed underneath the array 4 of color filters.
`Individual pixel electrodes are activated via thin film tran-
`sistors (TFT) 7 that are controlled from a video signal on
`the source lines 8 and a scanning signal on the gate line
`9. The LCD display includes the usual polarizer layers
`(not shown) on the glass plates 1 and 2, such that acti(cid:173)
`vation of selected transparent pixel electrodes allows
`light to pass through the corresponding color filters and
`reflect to the viewer, thereby creating a color image. A
`typical LCD display such as the Epson LB 2F-BCOO,
`manufactured by Seiko-Epson Company, Japan, has
`about 240 lines of pixels and about 300 pixels per line,
`with an image aspect ratio of 4:3. Such an aspect ratio
`allows the entire area of the image obtained from the 4:3
`aspect ratio NTSC format CCD sensor to be displayed
`on the LCD screen, so that the LCD screen composition
`will be the same as the image that is recorded by the
`camcorder NTSC format recorder, for later display on an
`NTSC format television display. Note that because the
`LCD has only 240 lines of pixels, the interlaced NTSC
`signal is displayed using a "repeat field" technique,
`
`5
`
`10
`
`where both the odd and even fields from the NTSC for(cid:173)
`mat sensor are displayed using the same lines of pixels
`on the LCD. This LCD, like most commercially available
`LCDs, has "rectangular" pixels, rather than square pix-
`els, where the distance between pixels in the horizontal
`direction is for example 2/3 the distance in the vertical
`direction. The LCD pixels are overlaid with a diagonal
`RGB stripe pattern as shown in Figure 1 B.
`In camcorders, the processing for both the still
`images and the motion images is identical. Such
`processing is normally implemented by hardwired ana(cid:173)
`log integrated circuits, although camcorders which use
`digital image processing integrated circuits have been
`produced. Such camcorders convert the signal from the
`15 CCD sensor into an NTSC composite or component for(cid:173)
`mat signal, which is provided to a video recording sub(cid:173)
`system or a video output jack. The color LCD display
`includes circuitry to decode the NTSC composite or com(cid:173)
`ponent signal back into spatially subsampled RGB sig-
`20 nals to drive the individual RGB pixels on the LCD
`sensor.
`In a system oriented toward still photography, and in
`particular a digital still system, it would be desirable to
`avoid the necessity of generating an NTSC format signal
`in order to reduce the complexity of the required circuitry.
`In a totally digital system, that is, both the recording and
`display channels are digital, it is further desirable to min(cid:173)
`imize incompatibility between the channels. The problem
`is to achieve these objective in an architecture that min(cid:173)
`imizes cost and complexity and maximizes user han(cid:173)
`dling.
`
`25
`
`30
`
`SUMMARY OF THE INVENTION
`
`This problem is solved according to the invention by
`a number of features. In one aspect, the electronic cam(cid:173)
`era is operable in a still image mode according to a rel(cid:173)
`atively more complex digital image processing technique
`to produce high quality still images, and in a motion pre(cid:173)
`view mode according to a relatively more simple digital
`image processing technique to produce a preview image
`of acceptable quality prior to initiation of the still image
`mode. Such an architecture is particularly adapted to
`mapping an array of color image pixels from a sensor
`into an array of color display pixels on an LCD display
`comprising discrete LCD display pixels fewer in number
`than image sensor pixels. In that case, a relatively simple
`digital processing technique combines same-colored
`image pixel signals into a fewer number of intermediate
`pixels that correspond to the arrangement of the color
`display pixels.
`The advantage of the invention is that the two modes
`can be tailored for a relatively low quality "motion" mode
`and a much higher quality "still" mode. The motion mode
`images from the CCD sensor are processed by a hard(cid:173)
`wired digital signal processing circuit that generates low
`resolution, spatially subsampled digital image data
`which can directly drive the relatively low resolution LCD
`display. This reduces the complexity and clock frequency
`
`35
`
`40
`
`45
`
`50
`
`55
`
`2
`
`HTC Exhibit 1039
`
`

`
`3
`
`EP 0 720 388 A2
`
`4
`
`5
`
`15
`
`the user depresses the capture button 16. The camera
`then captures a single still image, firing a flash 18 if nec(cid:173)
`essary when the ambient illumination level is low. The
`still image is focused upon an image sensor 20 by a
`motor driven zoom lens 22. The intensity of the image
`light upon the sensor 20 is regulated by a motor-driven,
`variable, mechanical aperture 24, while exposure time is
`regulated electronically by appropriate clocking of the
`sensor 20. The still image from the image sensor 20 is
`10 processed and digitally stored on a removable memory
`card 26.
`Control of the sensor is provided by a timing and
`control section 27, which is an application specific inte(cid:173)
`grated circuit (ASIC) with processing and timing func-
`tions, for both capture and preview operating modes. For
`instance, the timing and control section 27 includes a
`sensor timing circuit 28 for controlling the image sensor
`functions. The sensor timing circuit 28 provides the sig(cid:173)
`nals to enable sensor drivers 30, which provide horizon-
`tal clocks (H1, H2), vertical clocks (V1, V2), as well as a
`signal FDG for activating a drain structure on the sensor
`20. The output of the image sensor 20 is amplified and
`processed in an analog gain and sampling (correlated
`double sampling (CDS)) circuit 32, and converted to dig-
`ital form in AID converter stage 34. The AID output signal
`is provided to a processor section 35, which includes a
`digital processor 36 for temporarily storing the still
`images in a DRAM memory 38. The digital processor 36
`then performs image processing on the still images, and
`finally stores the processed images on the removable
`memory card 26 via a memory card interface circuit 40,
`which may use the PCMCIA 2.0 standard interface. An
`EPROM memory 42 is used to store the firmware which
`operates the processor 36. The components of the proc-
`35 essor 35 are interconnected through a data bus 43,
`which also connects to the timing and control section 27
`and to the card interface 40.
`The motor-driven zoom lens 22 includes a zoom
`motor 44, a focus motor 46, and an aperture motor 48
`(all controlled by lens motor drivers 50). The timing and
`control section 27 further includes a control interface 52
`connected to the I ens motor drivers 50 and to a flash con(cid:173)
`trol circuit 53 via a photosystem interface block 54, which
`controls the operation of the zoom lens 22 and the flash
`45 18. The lens zoom position is controlled by the photosys(cid:173)
`tem interface block 54 based on position input from the
`zoom control buttons 14 through a user status and con(cid:173)
`trol section 55. The focusing, exposure control, and white
`balance is done automatically, as is typically the case in
`50 consumer camcorders. This is done by computing
`"image statistics" in an image statistics processor 60 in
`the real-time ASIC (timing and control section 27) as pre(cid:173)
`view images are continuously read out of the image sen(cid:173)
`sor 20. The computed values are then used by a program
`implemented in the digital processor 36, which decides
`how to adjust the focus motor, aperture, analog gain con(cid:173)
`trol, and analog white balance controls via the control
`interface 52 and the photosystems interface 54 on the
`ASIC timing and control section 27. Although the digital
`
`of the required circuitry, compared to generating an
`NTSC format signal, as is normally done in the prior art.
`The still mode image from the CCD sensor is processed
`by a general purpose processor (CPU) which executes
`an image processing software program in order to pro-
`duce a high quality digital still image.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The invention will be described in relation to the
`drawings, whereon
`
`Figures 1 A and 1 B show the structure and color filter
`pattern of a known color liquid crystal display (LCD);
`Figure 2 is a block diagram of an electronic camera
`incorporating dual modes for composing and cap(cid:173)
`turing a still image according to the invention;
`Figures 3A and 38 are diagrams of progressive scan
`image sensors useful with the camera of Figure 2;
`Figure 4 is a diagram of the Bayer color filter geom(cid:173)
`etry for the sensor used with the camera of Figure 1 ;
`Figure 5 shows the line timing for the still mode of
`operation;
`Figures 6A and 68 show the line timing for the pre(cid:173)
`view mode of operation;
`Figure 7 shows the special line skipping pattern
`used in the preview mode;
`Figure 8 shows further detail of the preview mode
`processing circuit shown in Figure 2;
`Figures 9A and 98 show further detail of the image
`statistics processor shown in Figure 2;
`Figure 1 0 shows one example of still mode image
`processing;
`Figure 11 shows the effect of pixel mapping from a
`sensor to an LCD display, each having different
`aspect ratios; and
`Figure 12 shows an enhancement to the block dia(cid:173)
`gram of Figure 2 in which a different sensor clock
`frequency is used in each of the dual modes.
`
`DETAILED DESCRIPTION OF THE PREFERRED
`EMBODIMENT
`
`A block diagram of a camera incorporating dual
`modes of processing according to the invention is shown
`in Figure 2. The camera includes an electronic color dis(cid:173)
`play, for example, a color liquid crystal (LCD) display 10
`of the type shown in Figure 1 A, and a user control section
`12 having a number of user control buttons, including
`zoom buttons 14, a preview button 15 and a capture but(cid:173)
`ton 16. To take a still picture, the user turns on the camera
`(using a power switch (not shown), which may be auto(cid:173)
`matically enabled when the user depresses the zoom
`buttons 14 or the preview button 15, or partially
`depresses the capture button 16). The user composes
`the picture by depressing the "zoom in" or "zoom out"
`buttons 14, and by adjusting the position of the camera,
`while observing the display image. When the user is sat(cid:173)
`isfied with the composition on the color LCD display 10,
`
`20
`
`25
`
`30
`
`40
`
`55
`
`3
`
`HTC Exhibit 1039
`
`

`
`5
`
`EP 0 720 388 A2
`
`6
`
`processor 36 and the control interface 52 are shown as
`being within two separate sections, in some implemen(cid:173)
`tations the same component could be used to perform
`both of these functions (as well as other of the recited
`functions). Sensor image data is passed to the processor
`section 35 through a high speed interface 56 in the timing
`and control section 27. The sensor image data is also
`directed to the color LCD display 1 0 through a preview
`mode processing circuit 58.
`The timing and control section (ASIC) 27 is operable
`in two modes, a relatively low quality "motion" mode and
`a much higher quality "still" mode. In the motion mode,
`images from the sensor 20 are processed by the preview
`mode processing circuit 58; in the still mode, images
`from the sensor 20 are processed in the processor 35.
`The processor 35 is a software-driven digital processing
`system that is slower than the ASIC 27. The preview
`mode processing circuit 58 is a hardwired digital signal
`processing circuit (part of the ASIC 27) that generates
`low resolution, spatially subsampled digital image data
`which can directly drive the relatively low resolution color
`LCD display 10. This reduces the complexity and clock
`frequency of the required circuitry, compared to gener(cid:173)
`ating an NTSC format signal, as is normally done in the
`prior art. The preview mode processing circuit includes
`a pixel remapper 62 for mapping the greater number of
`image pixels from the sensor 20 into the lesser number
`of display pixels (i.e., corresponding to the array 6 of
`transparent pixel electrodes in Figure 1) in the color LCD
`display 10. The color saturation of the remapped pixels
`is then adjusted in a color adjustment circuit 63 and its
`output is applied to a multiplexer 64. The multiplexer 64
`selects image data either from the preview mode
`processing circuit 58, producing a preview image, or from
`the high speed interface 56, which allows for suitably pre-
`processed viewing of stored images.
`In this camera, the image processing used to create
`the preview mode is done in the timing and control ASIC
`27, since the processing must be done rapidly. About 60
`images per second are processed in preview mode.
`However, since the image quality of the displayed image
`is limited by the resolution and color gamut of the LCD
`screen of the LCD color display 10, there is no need for
`elaborate image processing. Therefore, simple "preview
`mode" image processing is performed in a fixed digital
`circuit embedded in the preview mode processing circuit
`58 (which is part of the ASIC). The quality requirements
`for the still mode are much greater, since these images
`will be downloaded to a computer, and may be displayed
`on a high resolution CRT display, or printed on a high
`quality thermal printer. Therefore, the digital image
`processing must be more elaborate. By using the digital
`processor 36 to implement software procedures stored
`in the firmware memory 42, complex procedures can be
`implemented. These procedures can take several sec-
`onds to complete, since real-time operation is not
`required. Use of firmware-stored software allows the still
`mode image processing to be upgraded without requir-
`ing a new ASIC design. In effect, what happens is that a
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`4
`
`relatively less complex digital image processing tech(cid:173)
`nique is used in the motion preview mode, but at a higher
`data rate, and a relatively more complex digital image
`processing technique is used in the still mode, but at a
`slower data rate.
`Since the update rate, that is, the number of images
`that need to be supplied per unit time, is different for the
`still mode than for the motion mode, it is beneficial to use
`different clock frequencies for the different modes of
`operation. For example, as shown in Figure 12, a system
`oscillator 100 produces a 12 mHz clock frequency for use
`in the motion mode to obtain more updates/second (e.g.,
`60 images per second), while a divider 102 divides by
`two to provide a 6 mHz clock frequency for the still mode.
`The lower frequency allows more time to accurately posi(cid:173)
`tion the clamp and sample pulses so as to avoid CCD
`output signal transitions. This increases noise immunity
`in the still mode. A multiplexer 104 is enabled by the con(cid:173)
`trol interface 52 to determine which clock frequency is
`applied to the sensor timing circuit 28. Though not spe(cid:173)
`cifically shown, the changed timing is also communi(cid:173)
`cated to the AID stage 34 and other timing and control
`circuits.
`The sensor 20 is a progressive scan interline image
`sensor having a noninterlaced architecture, as shown in
`more detail in Figure 3A. The sensor comprises a two(cid:173)
`dimensional array of photosites 66, e.g. photodiodes,
`arranged in rows and columns of image pixels, a plurality
`of vertical registers 68 adjacent photosite columns for
`transferring rows of image pixel charge from the pho(cid:173)
`tosites 66 to a horizontal register 70 for readout respon(cid:173)
`sive to clock signals from the sensor drivers 30, and a
`charge drain (specifically, a fast dump structure 72) inter(cid:173)
`posed between the output of the vertical registers 68 and
`the horizontal register 70 for eliminating complete rows
`of image pixels at a time from the image sensor 20. A
`preferred image sensor is the Kodak model ceo KAI-
`0400CM image sensor, which has approximately 512
`active lines with approximately 768 active pixels per line
`and an image aspect ratio of 3:2. This sensor is
`described in a Performance Specification document
`available from Eastman Kodak Company, Rochester,
`New York. Each pixel is 9 microns "square", since both
`the vertical and horizontal distances between the centers
`of adjacent pixels are 9 microns. The 3:2 image aspect
`ratio of the CCD sensor, although wider than the 4:3
`aspect ratio of the display, is considered to be a preferred
`aspect ratio for still photography, in that the standard
`35mm film format, and standard 4R (4" x 6") prints also
`have a 3:2 image aspect ratio. The sensor uses a color
`filter array pattern known as the "Bayer checkerboard"
`pattern, described in U.S. patent 3,971 ,065, which is
`shown in Figure 4. Such a color filter array is character(cid:173)
`ized by a mosaic pattern in which the filter colors alter(cid:173)
`nate in both line and column directions. In the normal
`operating mode, all of the image pixels on the sensor are
`transferred as color image pixels to the horizontal regis(cid:173)
`ter 70, which delivers a stream of color pixel signals to
`the analog gain and CDS circuit 32 (see Figure 2). The
`
`HTC Exhibit 1039
`
`

`
`7
`
`EP 0 720 388 A2
`
`8
`
`color pixel signals are subsequently converted to digital
`pixel signals in the AID converter 34.
`The sensor 20 uses a progressive scan readout
`method, which allows the entire image to be read out in
`a single scan. The accumulated or integrated charge for
`the photodiodes comprising the pixels 66 is transported
`from the photosites to light protected vertical (parallel)
`registers 68 by applying a large positive voltage to the
`phase-one vertical clock (V1). This reads out every row,
`or line, into the vertical registers 68. The image pixel
`charge is then transported from the vertical registers 68
`to the horizontal register 70 by two-phase clocking of the
`vertical clocks (V1, V2). Between the vertical and hori(cid:173)
`zontal registers is the fast dump structure 72, which is
`further described in the Performance Specification doc-
`ument for the KAI-0400CM sensor. By setting a suitable
`positive potential on a fast dump gate line FDG, charge
`from the row of pixel values currently adjacent to the fast
`dump structure 72 is transferred from the CCD directly
`into the sensor substrate 74 rather than to the horizontal
`register 70. This dump, or line clear, is accomplished dur-
`ing the vertical-to-horizontal transfer time. When prop-
`erly controlled by the sensor timing circuit 28, the fast
`dump structure 72 allows lines of charge to be elimi(cid:173)
`nated.
`The timing and control section 27 operates the elec(cid:173)
`tronic camera shown in Figure 2 in the two aforemen(cid:173)
`tioned modes, including a first "still" mode wherein all
`rows of image pixel charge corresponding to each line
`are progressively read out through the horizontal register
`70 during a single scan, and a second "motion" mode
`wherein some of the rows of image pixel charge corre(cid:173)
`sponding to some lines are eliminated through the fast
`dump structure 72 prior to readout. As applied to the
`embodiment of Figure 2, the first mode corresponds to
`a high quality still imaging mode while the second mode
`corresponds to a special "line skipping" mode for driving
`the color LCD display 10. In the second mode, the timing
`and control section 27 controls the fast dump structure
`72 to eliminate two or more consecutive lines of image
`charge from the image sensor 20 for every one or more
`lines of image charge that are transferred to the horizon-
`tal register 70 for readout, thereby generating a pattern
`of lines (shown in Figure 7) suitable for driving the LCD
`display in a "repeat field" mode. An appropriate video sig-
`nal which displays the entire 3:2 aspect ratio sensor
`image on the 4:3 aspect ratio LCD, without introducing
`geometric distortion, is generated by alternately elimi(cid:173)
`nating two or four consecutive lines of image charge for
`every pair of lines of image charge that are transferred
`to the horizontal register 70.
`The sensor timing circuit 28 is controlled by the con(cid:173)
`trol interface 52 to provide the clock signals V1, V2, H1,
`H2, and the gate signal FDG according to the two modes
`of operation. The timing signals for the first mode are
`shown in Figure 5; those for the second mode are shown
`in Figure 6a and 6b. The two-phase cycling of signals V1
`and V2 control the transfer of lines of image pixel charge
`from the vertical registers 68 to the horizontal register
`
`5
`
`10
`
`15
`
`20
`
`25
`
`70. The two-phase cycling of signals H1 and H2 control
`the transfer of a stream of color pixel signals from the
`horizontal register 70 to subsequent circuits in the cam(cid:173)
`era. The level of the signal FDG determines whether the
`image charge is dumped to the substrate 74 or trans(cid:173)
`ferred to the horizontal register 70. When the sensor 20
`is clocked using the first timing mode shown in Figure 5
`for all lines of the sensor, all lines of the sensor are
`clocked out, one after the other, through the horizontal
`register 70, processed in subsequent camera circuitry,
`and stored in the removable memory 26. This timing
`mode provides a high quality progressive scan still
`image, but may take 1/30 second or longer to read out
`the still image. Such timing, however, is acceptable for
`still mode usage, and, as mentioned before, does not
`require unusually high speed components and, indeed,
`may benefit from a lower speed clock.
`To provide an image to the color LCD display 10, a
`lower resolution image is suitable, but the update rate
`must be sufficient to provide good motion rendition and
`eliminate display flicker. Moreover, the sensor 20
`includes the aforementioned array of color filters
`arranged in a particular mosaic color pattern (e.g., the
`checkerboard Bayer pattern of Fig. 3), and the lines of
`image charge that are transferred to the horizontal reg(cid:173)
`ister 70 should preserve that particular color pattern in
`the pattern of lines that are generated for the line-skip(cid:173)
`ping readout. To provide this kind of image, the sensor
`is read out in the second mode as shown in Figure 7,
`30 using the timing shown in Figures 6A and 6B. As shown
`in Figure 6A, the first two lines (1 and 2) are read out as
`in the normal mode. These provide a green-red and a
`blue-green line. The next two lines (3 and 4) are elimi(cid:173)
`nated by turning on the fast dump structure 72 during the
`time that these lines are transferred past the fast dump
`structure 72. Next, as shown in Figure 6B, lines 5 and 6
`are read out normally, and then lines 7, 8, 9, and 10 are
`eliminated. Next, the Figure 6A timing is used to read out
`lines 11 and 12, while eliminating lines 13 and 14, and
`then the Figure 6B timing is used to read out lines 15 and
`16, while eliminating lines 17-20. This process proceeds
`for the entire image readout period, during which 102
`pairs of lines are read out, and 154 pairs of lines are elim(cid:173)
`inated.
`This special "line skipping" readout mode, as shown
`in Figure 7, allows the sensor 3:2 aspect ratio image to
`be fully displayed on a 4:3 aspect ratio LCD without "geo(cid:173)
`metric distortion", that is, without stretching the image
`vertically, and without cropping off the horizontal edges
`of the image from the image sensor. This allows the LCD
`to properly show the entire 3:2 aspect ratio image cap-
`tured by the sensor, so that an image can be properly
`composed.
`As the 512 lines of the CCD imager are read out
`using the special "line skipping" mode, they are dis(cid:173)
`played using only 204 out of the approximately 240 LCD
`lines of pixels. The remaining approximately 361ines can
`either be masked behind a bezel, so that they are not
`visible, or preferably may be used to display status infor-
`
`35
`
`40
`
`45
`
`50
`
`55
`
`5
`
`HTC Exhibit 1039
`
`

`
`9
`
`EP 0 720 388 A2
`
`10
`
`mation, such as the time-of-day, image number, or a
`"push-button menu" for the user buttons. Figure 11
`shows a useful application Of such conversions. A sen(cid:173)
`sor having a 3:2 aspect ratio is shown mapped into an
`image area 90 of an LCD display screen 92 having a 4:3
`aspect ratio. A proportional remapping leaves a status
`area 94 available for other purposes, specifically to show
`text indicating the function of a set of reconfigurable con-
`trol buttons 96 in the control section 12. The function of
`the buttons is specified by the user status and control
`section 55 (Figure 2). This status information graphics
`data can be supplied by the digital processor to the LCD
`10 via high speed interface 56, when the MUX 64 is con(cid:173)
`trolled so as to use the digital data from interface 56,
`rather than from circuit 58, for supplying data to the final
`approximately 32 lines of the display 10.
`The "line skipping" readout causes some minor ver(cid:173)
`tical sampling artifacts, but these are not noticeable in
`the small LCD displays. The pixels output for the sensor
`20 in line skipping mode continue to have the Bayer-type
`color filter repeating pattern, so that they can be proc(cid:173)
`essed using processing techniques designed for the
`Bayer pattern.
`The processing complexity of the camera of Figure
`2 is considerably simplified by directly mapping the RGB
`sensor pixels 66 to the RGB pixels of the display 10. The
`easiest way to do this for the image sensor 20 (5121ines
`x 768 pixels and 3:2 aspect ratio) is to have an LCD dis(cid:173)
`play with 512 x 768 pixels and the same aspect ratio and
`color filter pattern. However, this would be a custom,
`costly LCD. Instead, LCDs have fewer display pixels than
`image pixels on the image sensor, normally have a 4:3
`image aspect ratio, and use the diagonally striped RGB
`pattern shown in Figure 1 B. In this discussion, an LCD
`pixel array of about 240 lines x 312 pixels per line is
`assumed.
`Therefore, the sensor pixels are processed in a
`"pixel mapping circuit", such as the LCD pixel remapper
`62. A block diagram of this circuit is shown in Figure 8.
`Note that there are 768/2 = 384 green or red/blue pixels
`per line on the sensor (see Figure 4). There are about
`300/3 = 100 green, red, or blue pixels per line on the LCD
`(see Figure 1 B). Thus, there are approximately 1/4 as
`many LCD pixels per line (per color) as there are sensor
`pixels per line. Therefore, the basic plan is to combine
`same-colored image pixels into a fewer number of inter(cid:173)
`mediate, combination pixel signals that are then mapped
`into the color display pixels. For instance, a simple "pixel
`mapping" circuit maps four green sensor pixel signals
`into one green LCD pixel for one line by summing two
`green sensor values, spaced apart by 4 CCD pixel posi(cid:173)
`tions, in the green pixel summer 76 and dividing by two
`via bit-shift wiring. The necessary delay is provided by
`the registers 82 clocked at one fourth the pixel rate, fur-
`ther delayed by one pixel clock. It also maps four red sen-
`sor pixels into one LCD pixel in the same manner (using
`the red/blue summer 78), and also stores this value in a
`100 pixel FIFO 80. The FIFO 80 compensates for the fact
`that the sensor has line sequential red and blue pixels,
`
`5
`
`by supplying blue pixels on the red sensor lines, and red
`pixels on the blue sensor lines. Four pixel delays are pro(cid:173)
`vided by the registers 82' clocked at one fourth pixel rate
`(CLK/4). The mapping process is basically, therefore, a
`process which, in its simplest form, involves averaging
`of signals to produce a smaller number of output color
`pixels than input color pixels. (The CFA interpolation
`algorithm discussed in reference to Figure 10, on the
`other hand, produces a larger number of "output" color
`10 pixels than input color pixels.) Alternate groups of 2 or 4
`lines of sensor values are discarded during preview
`mode by using the fast dump gate shown in Figure 3A,
`as described in connection with the "line skipping" mode.
`This allows the sensor readout time to be decreased by
`15 more than 1/2 during the preview mode.
`Another feature of the design is that by removing the
`NTSC rate driving circuitry from the color LCD display
`1 0, the active matrix LCD can be updated at a slower
`frequency than is normally used. This reduces the cost
`20 and power consumption of the LCD driver circuits (not
`shown). For example, the LCD can be updated at 30 Hz
`(provided the LCD active matrix display is designed so
`as not to exhibit noticeable flicker at this update rate),
`instead of 60 Hz.
`Once the LCD pixel values are calculated, the LCD
`color adjust circuit 63 increases the color saturation of
`the image by forming R-G and 8-G color difference sig(cid:173)
`nals, and adding or subtracting a fraction of these signals
`in