(12) United States Patent
`Yu et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,611,289 B1
`Aug. 26, 2003
`
`USOO6611289B1
`
`(54) DIGITAL CAMERAS USING MULTIPLE
`SENSORS WITH MULTIPLE LENSES
`(76) Inventors: Yanbin Yu, 1965. Una Ct., Fremont, CA
`(US) 94539; Zhongxuan Zhang, 44453
`Cavisson Ct., Fremont, CA (US) 94539
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(*) Notice:
`
`5,726.707 A
`3/1998 Sakurai et al.
`6,021,172 A
`2/2000 Fossum ....................... 377/60
`FOREIGN PATENT DOCUMENTS
`JP
`O6-351O29
`12/1994 ............ HO4N/O/09
`JP
`2OOO-OSO295
`2/2000 .......... HO4N/O/093
`sk -
`cited by examiner
`Primary Examiner Wendy R. Garber
`Assistant Examiner-Catherine Toppin
`(74) Attorney, Agent, or Firm Joe Zheng
`
`Jan. 15, 1999
`(22) Filed:
`(51) Int. Cl." ......................... H04N 9/093; H04N 9/009
`(52) U.S. Cl. ........................ 348,265. 348,272. 348,279
`(58) Field of Search ................................. 348/272, 279
`348/265; HO4N 9/090 o093
`s
`s
`References Cited
`
`(56)
`
`U.S. PATENT DOCUMENTS
`3,971.065 A 7/1976 Bayer
`4506.294. A
`3/1985 Nagumo ...................... 358/50
`5,081525 A
`1/1992 Akiyama ..................... 358/51
`5,414,465 A * 5/1995 Kodama et al. .....
`... 348/264
`5,436,661. A
`7/1995 Yamamoto et al. ......... 348/263
`
`
`
`An improved digital camera that produces digital images of
`high qualities without using expensive image Sensors and
`optics is disclosed. The disclosed digital cameras use mul
`tiple image Sensors with multiple lenses. One of the multiple
`image Sensors is made to be responsive to all intensity
`information in visible color spectrum and a (gray intensity)
`image resulting from the Sensor is used to compensate lost
`information in images from other image Sensors responsive
`to certain colors. A final color image is obtained by a digital
`image processing circuitry that performs pixel registration
`process with reference to the gray intensity image So that a
`true color image with true resolution is obtained therefrom.
`
`31 Claims, 9 Drawing Sheets
`
`326
`\- - - - -
`- - - -
`- -
`
`r
`
`s
`
`^
`
`R
`
`318
`/
`
`302
`310, \
`
`Control
`H
`- Circuit
`- V - 342
`320
`- 304
`G /
`- 312
`Control
`- KH
`Circuit
`-1
`322
`344
`B /
`
`- - -
`
`Control
`- .
`.
`Circuit
`a
`346
`314
`".
`,
`324
`.306
`Control
`K-
`Circuit
`
`Y.
`
`348
`
`316
`
`3O8
`
`328
`--------------y
`:
`; 352
`AD --
`:
`N330:
`:
`: 354.
`:
`|
`
`:
`:
`:
`:
`:
`|
`
`AID
`V
`
`332
`
`300
`
`-360
`MEM
`
`340
`
`338
`/
`Digital
`- Image
`ProCessing
`
`:
`|
`:
`
`|
`
`AID
`V
`
`334
`
`--> AID
`f
`\ .
`-------- 336
`
`|
`
`I
`
`.
`
`356
`
`358
`
`332
`YWY
`
`Post
`Circuitry
`
`N
`350
`
`OU
`
`
`Ex.1005 / Page 1 of 17Ex.1005 / Page 1 of 17
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Aug. 26, 2003
`
`Sheet 1 of 9
`
`US 6,611,289 B1
`
`
`
`
`Ex.1005 / Page 2 of 17Ex.1005 / Page 2 of 17
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Aug. 26, 2003
`
`Sheet 2 of 9
`
`US 6,611,289 B1
`
`3.
`
`S
`
`&
`
`R.
`
`I
`
`R. R.
`
`s &
`
`s
`->
`
`
`Ex.1005 / Page 3 of 17Ex.1005 / Page 3 of 17
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Aug. 26, 2003
`
`Sheet 3 of 9
`
`US 6,611,289 B1
`
`099
`
`£ ‘61-I
`
`
`
`
`
`
`
`
`
`
`Ex.1005 / Page 4 of 17Ex.1005 / Page 4 of 17
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Aug. 26, 2003
`
`Sheet 4 of 9
`
`US 6,611,289 B1
`
`|×]], [[, ][, ]
`
`
`
`
`Ex.1005 / Page 5 of 17Ex.1005 / Page 5 of 17
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Aug. 26, 2003
`
`Sheet 5 of 9
`
`US 6,611,289 B1
`
`
`
`S.
`
`3
`
`S
`
`S
`
`S.
`
`s
`
`3
`
`
`Ex.1005 / Page 6 of 17Ex.1005 / Page 6 of 17
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Aug. 26, 2003
`
`Sheet 6 of 9
`
`US 6,611,289 B1
`
`
`
`N
`
`
`
`N-S
`
`
`
`OO
`O
`CO
`
`
`Ex.1005 / Page 7 of 17Ex.1005 / Page 7 of 17
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Aug. 26, 2003
`
`Sheet 7 of 9
`
`US 6,611,289 B1
`
`89 "61-I
`
`Z ?9
`
`O O O O O O O O O O
`
`O O (O O O O O (O O O
`
`
`Ex.1005 / Page 8 of 17Ex.1005 / Page 8 of 17
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Aug. 26, 2003
`
`Sheet 8 of 9
`
`US 6,611,289 B1
`
`
`
`&
`
`s
`
`
`
`s
`5
`Ol
`s
`
`O
`
`
`Ex.1005 / Page 9 of 17Ex.1005 / Page 9 of 17
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Aug. 26, 2003
`
`Sheet 9 of 9
`
`US 6,611,289 B1
`
`8O2
`
`Providing multiple image
`sensors with multiple lenses
`
`
`
`Providing means for each of image
`sensors responsive to different regions
`of the visible color spectrum
`
`Generating digital images from
`the images Sensors via an A/D apparatuS
`
`804
`
`806
`
`808
`
`Storing the digital images
`
`810
`
`Performing Digital Image processing
`
`812
`
`814
`
`816
`
`818
`
`820
`
`Generate
`Control
`Signals
`
`Increase
`dynamic
`range
`
`Generate
`motion
`VeCtOr
`
`Enhance
`images
`
`
`
`830
`
`Produce a color image
`
`Fig. 8
`
`
`Ex.1005 / Page 10 of 17Ex.1005 / Page 10 of 17
`
`TESLA, INC.TESLA, INC.
`
`

`

`1
`DIGITAL CAMERAS USING MULTIPLE
`SENSORS WITH MULTIPLE LENSES
`
`US 6,611,289 B1
`
`2
`design and manufacturing cost for a higher resolution
`Sensor, however, would be evaluated at many magnitudes of
`the lower resolution Sensors. Therefore there is a great need
`for a generic Solution that makes digital cameras capable of
`producing high resolution images without enormously
`incurring the cost of photoSensitive chips with multimillion
`photocells.
`A Second noticeable quality between digital cameras and
`film-based cameras is the dynamic range. Films have the
`necessary chemical pigments to make colors much more
`Vivid and more adaptive to light conditions than current
`digital cameras can do. This is largely due to the limited
`pixel depth the current digital cameras could produce and
`the limited Sensitivity of the photocells in the image Sensor.
`There is thus a further need for digital cameras that produce
`better colors and details in a greater range.
`There are many other quality factors that limit the popu
`larity of digital cameras although it is well understood that
`the digital cameras are the much preferred image acquisition
`means. Solutions that fundamentally improve the image
`qualities without incurring Substantial cost are always well
`come and being Seriously and continuously Sought.
`SUMMARY OF THE INVENTION
`Recent product introductions, technological
`advancements, and price cuts, along with the emergence of
`email and the World Wide Web, have helped make digital
`cameras the hottest new category of consumer electronics
`products. But the image qualities, noticeably the image
`resolutions and color dynamic ranges, have limited the
`popularity of digital cameras among consumers. Under the
`constraints of improving image qualities without incurring
`Substantial costs to the digital cameras, the present invention
`discloses improved digital cameras that use multiple image
`Sensors with multiple lenses.
`The present invention has been made in consideration of
`the above described problems and needs and has particular
`applications to digital cameras that are demanded to produce
`digital images of high qualities. According to one aspect of
`the present invention, an improved digital camera uses four
`image Sensors, each having its own lens, of which three
`image Sensors are made responsive to the three primary
`colors and the fourth one made responsive to all intensity
`information. Using a set of digital image processes embed
`ded in a digital signal processing chip, images from the three
`color image Sensors are processed with reference to the
`image from the black-and-white image Sensor and Subse
`quently produce high quality and film-like true color digital
`images.
`With the unique configuration, there are many obvious
`benefits and advantages. First, the resolutions of the image
`Sensors are fully used. Second each of the image Sensors is
`only responsible for one color; thereby the expensive pro
`ceSS of coating a mosaic of Selectively transmissive filters
`Superimposed in pixel-based registration on one image Sen
`Sor is eliminated and Subsequently no micro-lenses process
`is needed. Third, the image from the black-and-white image
`Sensor captures all information including details that the
`three color image Sensors may have missed. Further, because
`the resolutions of the image Sensors are fully used, for the
`Same resolution of color images, the image Sensors would
`relatively have smaller number of pixels, which typically
`leads to high yield, higher Sensitivity, less croSS-talking, and
`lower clocking rate. Besides, the size of the image Sensors
`could be Smaller, resulting in Smaller optical lenses.
`According to one embodiment, the present invention is an
`improved digital camera comprising:
`
`
`Ex.1005 / Page 11 of 17Ex.1005 / Page 11 of 17
`
`TESLA, INC.TESLA, INC.
`
`15
`
`25
`
`BACKGROUND OF THE INVENTION
`1. Field of the Invention
`The present invention generally relates to digital cameras
`and more particularly relates to digital color cameras that
`use four Sensors, three for the tricolor Sensing and the fourth
`for full-color Sensing to improve the qualities of digital
`images therefrom.
`2. Description of the Related Art
`Digital photography is one of the most exciting technolo
`gies to emerge during the twentieth century. With the
`appropriate hardware and Software (and a little knowledge),
`anyone can put the principles of digital photography to
`work. Digital cameras are on the cutting edge of digital
`photography. Recent product introductions, technological
`advancements, and price cuts, along with the emergence of
`email and the World Wide Web, have helped make digital
`cameras the hottest new category of consumer electronics
`products.
`Digital cameras, however, do not work in the way as the
`traditional film cameras do. In fact, they are more closely
`related to computer Scanners, copiers, or fax machines. Most
`digital cameras use an image Sensor or photoSensitive
`device, Such as charged-coupled device (CCD) or Comple
`mentary Metal-Oxide Semiconductor (CMOS) to sense a
`Scene. The photosensitive device reacts to light reflected
`from the Scene and can translate the Strength of that reaction
`into a numeric equivalent. By passing light through red,
`green, and blue filters, for example, the reaction can be
`gauged for each Separate color spectrum. When the readings
`are combined and evaluated via Software, the camera can
`determine the Specific color of each Segment of the picture.
`Because the image is actually a collection of numeric data,
`it can easily be downloaded into a computer and manipu
`lated for more artistic effects.
`Nevertheless, there are many cases in which digital cam
`40
`eras Simply could not be used due to the limited resolutions
`from today's digital cameras. Film-based photographs have
`immeasurably higher resolutions than digital cameras. The
`comparison magnitude may be Somewhere millions of pixels
`Versus tens thousands of pixels in the digital cameras.
`Although, it is theoretically possible to design a photosen
`sitive chip with multimillion of pixels, the cost of such chip
`would be a forbidden number and may consequently drag
`the digital cameras out of the consumer market.
`FIG. 1 shows a typical image Sensor or photoSensitive
`chip 100 used in digital cameras. Photosensitive chip 100
`comprises a plurality of photocells arranged in an array. A
`mosaic of Selectively transmissive filters is Superimposed in
`registration with each of the photocells So that a first, Second
`and third Selective group of photocells are made to Sense the
`red, green and blue range of the Visible Spectrum, respec
`tively. The number of the photocells in photosensitive chip
`100 typically determines the resolutions of digital images
`resulting therefrom. The horizontal resolution is by the
`number of the photocells in a row 102 and the vertical
`resolution is by the number of the photocells in a column
`104. Because of the alternating positions of the designated
`photocells, for example, 106 for red photocells and 108 for
`green photocells, the actual resolutions for a color image
`have been Significantly reduced.
`To have color images with higher resolutions, the number
`of photocells in a Sensor must be increased. The actual
`
`35
`
`45
`
`50
`
`55
`
`60
`
`65
`
`

`

`US 6,611,289 B1
`
`3
`four image Sensors, closely positioned with respect to a
`common plane,
`four lenses, each mounted in front of one of the four
`image Sensors,
`first three of the four image Sensors being respectively
`sensitive to three different regions of visible color
`Spectrum; a fourth Sensor of the four image Sensors
`being Sensitive to a full region of the visible color
`Spectrum; the full region including the three different
`regions in the visible color spectrum;
`the four image Sensors producing, independently and
`respectively, four intensity imageS when being exposed
`to an imaging target, first three images of the four
`intensity images respectively from the first three of the
`four image Sensors and a fourth image of the four
`intensity images from the fourth Sensor of the four
`image Sensors,
`an analog-to-digital converting circuitry coupled to the
`four image Sensors and digitizing the four intensity
`images to produce four digital images, first three of the
`four digital images corresponding to the first three
`images and a fourth digital image of the four digital
`images corresponding to the fourth image of the four
`intensity images,
`an image memory, coupled to the analog-to-digital con
`Verting circuitry, for Storing the four digital images, and
`a digital image processing circuitry coupled to the image
`memory and receiving the four digital images, produc
`ing a color image of the imaging target from the four
`digital images.
`According to one embodiment, the present invention is a
`method for producing digital images of high qualities, the
`method comprising:
`obtaining three Scalar images from three image Sensors
`closely positioned in a common plane with reference to
`an image target;
`obtaining a gray intensity image from a fourth image
`Sensor, the fourth image Sensor closely positioned in
`the common plane with the three image Sensors;
`digitizing the three Scalar intensity images and the gray
`intensity image to produce three Scalar digital images
`and a gray digital image;
`buffering the three Scalar digital images and the gray
`digital image in an image memory; and
`producing a color image from the three Scalar digital
`imageS processed in conjunction with the gray digital
`image.
`Objects and benefits, together with the foregoing are
`attained in the exercise of the invention in the following
`description and resulting in the embodiment illustrated in the
`accompanying drawings.
`BRIEF DESCRIPTION OF THE DRAWINGS
`These and other features, aspects, and advantages of the
`present invention will become better understood with regard
`to the following description, appended claims, and accom
`panying drawings where:
`FIG. 1 shows a typical image Sensor used in existing
`digital cameras,
`FIG. 2 illustrates a representation of a color image as a
`vector image comprising three Scalar images, each from a
`distinct colored Sensor,
`FIG. 3 shows a block diagram of an improved digital
`camera employing multiple lenses and Sensors according to
`one embodiment of the present invention;
`
`5
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`FIGS. 4A and 4B illustrate two possible spatial arrange
`ments of four lenses used in the improved digital camera of
`FIG. 3;
`FIG. 5 illustrates a virtual color image formed by com
`bining four intensity images Stored in image memories,
`FIG. 6A illustrates a red image being processed for pixel
`registration with respect to a reference image;
`FIG. 6B illustrates a pixel being adjusted to the reference
`coordinates to minimize the weighted characteristic differ
`ence between two windowed groups of pixels;
`FIG. 7 demonstrates the dynamic ranges of images from
`color image Sensors are expanded with reference to an
`intensity image from a B/W image Sensor; and
`FIG. 8 shows a process flow diagram of the present
`invention according to one embodiment.
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`In the following detailed description of the present
`invention, numerous Specific details are Set forth in order to
`provide a thorough understanding of the present invention.
`However, it will become obvious to those skilled in the art
`that the present invention may be practiced without these
`Specific details. The description and representation herein
`are the common means used by those experienced or skilled
`in the art to most effectively convey the Substance of their
`work to others skilled in the art. In other instances, well
`known methods, procedures, components, and circuitry have
`not been described in detail to avoid unnecessarily obscuring
`aspects of the present invention.
`Referring now to the drawings, in which like numerals
`refer to like parts throughout the several views. FIG. 2
`depicts a representation of a color image 200. From the
`human color vision theory, it is known three primary colors
`are Sufficient enough to represent all colors visible by human
`eyes. Of all possible three primary colors, red (R), green (G)
`and blue (B) are the most popular ones that are used to
`reproduce colors. Televisions, for example, use three com
`ponent signals R, G and B to display most of the visible
`colors. Similarly to represent a color Scene or object, three
`images in red, green, and blue are generally Sufficient to
`reproduce the original colors of the Scene or object. Hence
`color image 200 is represented by a red, green and blue
`image 202, 204 and 206.
`From a mathematical perspective, color image C200 may
`be referred to as a vector image, because each pixel at
`coordinates (i,j), expressed as C(i,j) 208, in the color image
`is a vector 210 that includes three scalar values R(i,j) 212,
`G(i,j) 214 and B(i,j) 216. In order words, obtaining a color
`image is equivalent to obtaining three Scalar images, Such as
`202, 204 and 206. This principle has been used in profes
`Sional Video camera recorders in which a prism is often used
`to split an incoming reflected light from a Scene into three
`distinct lights, each banded by a distinct region in the Visible
`light spectrum. For example, an incoming reflected light is
`Split to red, green and blue lights, namely the red light covers
`the red portion in the light Spectrum, the green light covers
`the green portion in the light spectrum and the blue covers
`the blue portion in the light spectrum.
`FIG. 3 shows a block diagram of an improved digital
`camera 300 employing multiple lens and Sensors according
`to one embodiment of the present invention. Fundamentally
`and distinctly different from existing digital cameras,
`improved digital camera 300 uses four identical image
`sensors 302, 404, 306, and 308. Preferably, the image
`
`
`Ex.1005 / Page 12 of 17Ex.1005 / Page 12 of 17
`
`TESLA, INC.TESLA, INC.
`
`

`

`25
`
`S
`sensors are Complementary Metal-Oxide Semiconductor
`(CMOS) type. It should be noted that the present invention
`can be equally applied to Charged-Coupled Device (CCD)
`type image Sensors as well.
`One of the distinctions of the present inventions from
`prior arts is that the image Sensors used herein are
`monochromatic, namely none of the image Sensors are
`coated with a mosaic of Selectively transmissive filters in
`pixel-based registration. Practically, using monochromatic
`image Sensors rather than a color image Sensor (which is
`coated with the selectively transmissive filters) has a lot of
`advantages. AS described above, one of the obvious ones is
`the full use of the sensor resolutions.
`Each of image sensors 302, 304, 306, and 308 is inte
`grated respectively with a uniform transmissive filter, not
`shown explicitly in the figure, referred to as a color filter
`herein. To be more specific, if output 318 of image sensor
`302 is designated for a red signal, the color filter is basically
`a red filter only transmitting red portion of target 326.
`Similarly the color filters for image sensors 304 and 306 are
`a green filter and a blue filter, respectively. It should be
`pointed out that red, green and blue filters in the present
`example are preferable, but may be integrated into a lens.
`That means that lenses 310, 312 and 314 are colored
`accordingly according to another embodiment. Further other
`choices of three primary colors will work the same as more
`explained below.
`The fourth image sensor 308 is not specifically coated
`with a color filter. According to one embodiment, fourth
`image sensor 308 is integrated with filter 316 that is full
`transparent, allowing all components of Visible light to pass
`through. In other words, there may not need any filter in
`front of image Sensor 208 according to one aspect of the
`present invention. Because Some image Sensors like CCD
`35
`types tend to have high Sensitivity in red portion or beyond
`in the light spectrum, potentially decreasing image quality.
`It is preferable to have a proper light (band) filter that
`obstructs anything beyond the visible light spectrum (430
`nm -680 nm).
`Output signals from image sensors 302,304,306, and 308
`are respectively sent to Analog-to-Digital (A/D) circuitry
`328 that digitizes the output signals respectively. According
`to one embodiment, A/D converter 328 comprises four
`individual (channel) A/D converters 330,332,334 and 336;
`each coupled to one of the four image Sensors respectively.
`AS Such, Sensed images of target 326 from image Sensors
`302,304,306, and 308 can be digitized in parallel, yielding
`high Signal throughput rates. Further, each of A/D converters
`330, 332,334 and 336 may be integrated directly with one
`corresponding image Sensor, So that the output of the image
`Sensor is a digital image.
`Alternatively, output signals from image Sensors 302,
`304, 306, and 308 may respectively sent to Analog-to
`Digital (A/D) circuitry 328 that digitizes the output signals
`independently and sequentially if A/D circuitry 328 is a
`Standalone and Separate A/D converter.
`Nevertheless outputs from A/D circuitry 328 or A/D
`converters 330, 332, 334 and 336 are four intensity images
`352, 354, 356 and 358 that are preferably stored in image
`memory 360. It should be noted, however, these four inten
`sity images 352, 354, 356 and 358 are not the scalar images
`of the color image of target 326 from the same perspective.
`FIG. 4A illustrates an arrangement of four lenses 402, 404,
`406 and 408. Because four image sensors 302,304,306, and
`308 of FIG. 3 are independently exposed to a target, for
`simplicity, four lenses 402, 404, 406 and 408 may be also
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 6,611,289 B1
`
`15
`
`6
`Viewed as four colored image Sensors, namely four mono
`chromatic image Sensors are each coated with a distinct
`color filter. For example, 402 is an image Sensor with a red
`filter thereon, 404 is an image Sensor with a green filter
`thereon, 406 an image sensor with a blue filter thereon and
`408 an image sensor with a band filter thereon. Image
`sensors 402, 404, 406 and 408 results in respectively four
`intensity images 412, 414, 416 and 418 when exposed to
`target 400 that is simply assumed to be a point. Hence, four
`intensity images 412, 414, 416 and 418 each has a pixel or
`a small group of pixels 422, 424, 426 and 428 representing
`target 400. Because of the different spatial positions of
`image sensors 402, 404, 406 and 408, pixels 422, 424, 426
`and 428 do not register to a common point as respectively
`illustrated in four intensity images 412, 414, 416 and 418.
`Therefore four intensity images 412, 414, 416 and 418 can
`not be simply combined to form a color image of target 400.
`FIG. 4B shows another arrangement of four lenses 402,
`404, 406 and 408 in a regular camera lens opening 410 So
`that the exterior appearance of a digital camera with multiple
`Sensors and multiple lenses may look the same as a regular
`film camera. Regardless of other possible arrangements of
`image sensors behind four lenses 402,404, 406 and 408, it
`can be appreciated to those skilled in the art that four images
`412, 414, 416 and 418, resulting respectively from four
`lenses 402, 404, 406 and 408 will have to be registered
`before forming a color image therefrom.
`Returning now to FIG. 3, there is a digital image pro
`cessing circuitry 330 that performs many functions as
`described below.
`One of the functions that digital image processing cir
`cuitry 330 performs is to control the operations of image
`sensors 302,304,306 and 308. Four independent feedback
`signals 332 are generated in circuitry 330 and determined
`from four digital image 352, 354, 356 and 358 resulting
`respectively from image sensors 302, 304, 306 and 308.
`Feedback signals 332 are then used respectively to control
`each of image sensors 302,304,306 and 308. For example,
`if digital image 354 from green image sensor 304 tends to be
`Saturated, technically image Sensor 304 should be leSS
`exposed. In digital image processing circuitry 330, digital
`image 354 is first analyzed, from which a corresponding
`feedback signal can be generated to shorten the exposure
`time of image sensor 304.
`Given an intensity image, there are many ways to deter
`mine if the given image is Saturated due to an overexposure.
`One of the available ways is to determine from a histogram
`of the given image, which is explained in great detail in
`“Digital Image Processing” by Rafael C. Gonzalez from
`Addison-Wesley Publisher. Large population concentrated
`on the high end of the histogram is an indication of Satura
`tion. Thus a control signal to reduce a predefined exposure
`time can be generated. Upon receiving the control Signal, the
`control circuit acts accordingly. To be more specific, control
`circuits 342, 344, 346 and 348 receive respectively the
`control Signals 332, each generated with respect to a corre
`sponding histogram in digital image processing circuitry 338
`and independently and respectively control image Sensors
`302,304,306 and 308. It is understood to those skilled in the
`art that there are many other causes that may need to control
`image sensors 302,304,306 and 308 independently, such as
`gain and offset controls. All necessary controls signals may
`be obtained from digital image processing circuitry 338 that
`operates on four digital image 352, 354, 356 and 358
`respectively and independently resulting from image Sensors
`302,304, 306 and 308.
`Hence, one of the features in the present invention using
`four image Sensors with four color filters is the independent
`
`
`Ex.1005 / Page 13 of 17Ex.1005 / Page 13 of 17
`
`TESLA, INC.TESLA, INC.
`
`

`

`US 6,611,289 B1
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`7
`control of each of the image Sensors. This is not possible or
`could be complicated procedure in a Single Sensor coated
`with a mosaic of selective transmissive filters. As will be
`appreciated below, there are many other features in the
`present invention including high Sensitivities, high dynamic
`ranges, achievement of true colors and increased SNR
`(signal-to-noise ratio).
`For completeness, along with digital image processing
`circuitry 338, post circuitry 350 includes all necessary
`circuits to further process vector or color image 340. It is
`understood to those skilled in the art that Some of the
`functions performed may include image compression in a
`conventional format such as JPEG and necessary I/O func
`tions So that color images generated by digital camera 300
`may be downloaded to other computing devices for further
`artistic editing/processing and Subsequently for printing on
`glossy paper or publication on the Internet or World Wide
`Web.
`In the above description of FIG. 3, it is inherently implied
`that image sensors 302,304,306 and 308 are identical. It is
`true when the primary colors are red, green and blue.
`However, those skilled in the art will understand that image
`sensors 302, 304, 306 and 308 being identical is not the
`requirement to practice the present invention. For example,
`image sensors 302, 304 and 306 are integrated with filters
`that may cause the image Sensors to produce images signals
`similar to YIQ signals used in NTSC television system. In
`other words, if one of the three images from image Sensors
`302, 304 and 306 produces a luminance signal representing
`the light intensity of a color target 326 and the two images
`are the chrominance images, the resolutions of the chromi
`nance images can be only one half of the luminance image,
`hence two of image sensors 302,304 and 306 need to have
`one half of the resolutions of the third one. This is taking the
`advantage of the color Sensitivity in human color visions.
`Further it is also understood to those skilled in the art that
`the unique configuration of multiple Sensors and multilenses
`disclosed herein may be applied to black-and-white digital
`cameras in which there is only one monochromatic image
`Sensor Sensing only the intensity of an imaging target. Using
`an additional image Sensor, Such as image Sensor 308 in FIG.
`3 can help to modify image qualities of the original image
`from the monochromatic image Sensor. The following
`description is based on the embodiment illustrated in FIG.3,
`those skilled in the art can appreciate that the description is
`equally applied to the black-and-white digital cameras.
`Referring to FIG. 5, there is illustrated a virtual color
`image 500 formed by combining four intensity images 502,
`504,506 and 508. From virtual color image 500, a color or
`vector image 520 can be derived. Four intensity images 502,
`504, 506 and 508 correspond respectively to four intensity
`images from four image sensors 302, 304, 306 and 308 of
`FIG. 3. As the name indicates, virtual color image 500 is not
`actually formed to occupy an image memory but is for
`illustrative purpose. Since four intensity images 502, 504,
`506 and 508 are typically kept in a memory such as 360 of
`FIG.3 after being digitized, color image 520 can be derived
`from virtual color image 500 that utilizes all data in intensity
`images 502,504,506 and 508.
`Given reference coordinates, intensity images 502, 504
`and 506 are respectively processed to be registered with
`intensity image 508. In other words, pixels 503',505' and
`507 will be mapped to coordinates 509', all pixel values in
`scalar images 510, 512 and 514 are derived from other pixel
`values in the image.
`To be more Specific, pixels in four intensity images 502,
`504, 506 and 508 from image sensors, even very closed
`
`8
`positioned, are not in registration. That means that four
`respective pixels at the same coordinates in four intensity
`images 502,504,506 and 508 do not correspond to the same
`Spatial point in a Scene. Therefore, color image 510 must be
`derived from four intensity images 502,504,506 and 508.
`According to one embodiment, intensity image 508,
`referred to as B/W image herein, is used as a reference
`image, which means vector pixels in color image 500 are
`registered with scalar pixels in B/W image 508 as shown in
`virtual color image 500. FIG. 6 shows a registration process
`according to one embodiment of the present invention.
`A.sliding block or window 600 is Superimposed respectively
`over a reference image 602 and a red (R) image 604. It
`should be noted that reference image 602 corresponds to
`B/W image 508 and R image 604 corresponds to red image
`502 of FIG. 5 provided the registration for red image 502 is
`proceeded first. The size of window 600 is preferably a
`square, for example; 3 by 3, 5 by 5 or 7 by 7. Because
`window 600 is used to determine a pair of corresponding
`pixels in both reference image 602 and R image 604, the
`center coefficient 608 is normally weighted heavier than the
`rest of the surrounding coefficients 606.
`For all pixels surrounded by window 600, a set of statistic
`characteristics for the Surrounded pixels are computed, Such
`as a mean value and first or Second-order derivations. To be
`more specific, the Set of Statistic characteristics for the pixels
`surrounded by window 600 in reference image 602 is first
`determined. Then a corresponding Set of Statistic character
`istics for the pixels surrounded by window 600 in R image
`604 is calculated. AS described before, image Sensors are
`closely positioned, the Spatial position offset is normally
`small. Therefore window 600 does not have to be large and
`the corresponding set of statistic characteristics for the
`pixels surrounded by window 600 in R image 604 shall be
`close to that for the pixels surrounded by window 600 in
`reference image 602.
`Given image 602 being the reference, an adjustment of the
`center coordinates by window 600 can be derived by match
`ing the Statistic ch