(12) United States Patent
`Tao
`
`USOO6809765B1
`(10) Patent No.:
`US 6,809,765 B1
`(45) Date of Patent:
`Oct. 26, 2004
`
`(54) DEMOSAICING FOR DIGITAL IMAGING
`DEVICE USING PERCEPTUALLY UNIFORM
`COLOR SPACE
`
`(*) Notice:
`
`(75) Inventor: Bo Tao, Sunnyvale, CA (US)
`(73) Assignees: Sony Corporation, Tokyo (JP); Sony
`Electronics Inc, Park Ridge, NJ (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`(21) Appl. No.: 09/412,187
`(22) Filed:
`Oct. 5, 1999
`(51) Int. Cl. .......................... H04N 5/335; H04N 1/46;
`G06K 9/36; G06K 9/32; GO3F 3/08
`(52) U.S. Cl. ....................... 348.273. 382.276,382,300.
`358/515; 358/518; 358/525
`(58) Field of Search ................................. 348,273,277,
`348/278, 280, 222.1; 382/275 276. 277
`s
`s
`300. 358f515 51s 525
`s
`s
`s
`References Cited
`
`(56)
`
`5,778,106 A * 7/1998 Juenger et al. ............. 382/275
`6.249,797 B1 * 6/2001 Kovacevic et al. ......... 708/300
`6.295,087 B1 * 9/2001 Nohda ........................ 348/273
`6,392,699 B1 * 5/2002 Acharya ........
`... 348/273
`6,404.918 B1 * 6/2002 Hel-or et al. ............... 382/167
`OTHER PUBLICATIONS
`Ron Kimmel, Sep. 1999, IEEE, vol. 8, No. 9, pp.
`1221-1228.
`Ping Wah Pong et al., 1997, IEEE, 1063-6390/97, pp.
`280-285.
`* cited by examiner
`Primary Examiner Andrew Christensen
`ASSistant Examiner Nhan Tran
`(74) Attorney, Agent, or Firm Valley Oak Law
`(57)
`ABSTRACT
`A digital camera is provided having a lens for focusing light
`from an image through a RGB mosaiced filter on to a Solid
`State photosensor array containing a matrix of CCDs. The
`CCDS provide single-channel mosaiced image output. A
`demosaicing proceSS includes a first color transformation
`after Separation for determining luminance and a luminance
`interpolation for interpolation of missing luminance com
`ponents. The demosaicing process further includes a Second
`color transformation for obtaining chrominance compo
`U.S. PATENT DOCUMENTS
`nents. A chrominance interpolation for interpolation of miss
`34.8/660
`4,743,960 A * 5/1988 Duvic et all
`5,053,861. A 10/1991 Tsai et al. .375f240.01 ing chrominance components and an inverse color transfor
`5172227 A 12/1992 Tsai et al. ................ 375/240.2
`mation transforms the components into the RGB space for
`5,185,661. A * 2/1993 Ng ............................. 358/505
`combination into an image output.
`5,528.292 A * 6/1996
`24 Claims, 6 Drawing Sheets
`5,596,367 A
`1/1997
`
`:
`
`
`
`
`
`
`
`
`
`
`
`MAGE OUTPUT
`
`FROM 2
`
`f
`
`SEPARATION
`
`EIRST COLORTRANSFORMATION
`
`LMINANCENTERPOATON
`
`Tois
`
`202
`
`284
`
`286
`
`28
`
`30
`
`22
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1031, p. 1
`
`

`

`U.S. Patent
`
`Oct. 26, 2004
`
`Sheet 1 of 6
`
`US 6,809,765 B1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`3
`
`4
`
`5
`
`6
`
`B G B G B G
`GRG RIGR
`B G B G B G
`
`B G B G B G
`FIG. 2 (PRIOR ART)
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1031, p. 2
`
`

`

`U.S. Patent
`
`Oct. 26, 2004
`
`Sheet 2 of 6
`
`US 6,809,765 B1
`
`
`
`FIG. 3 (PRIOR ART)
`
`FIG. 4
`
`70
`LIGHT 1N
`
`PHOTOSENSOR CONVERSION
`
`A/D CONVERSION
`
`DEMOSAICING
`
`IMAGE PROCESSING
`
`MAGE OUTPUT
`
`71
`
`72
`
`74
`
`76
`
`78
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1031, p. 3
`
`

`

`U.S. Patent
`
`Oct. 26, 2004
`
`Sheet 3 of 6
`
`US 6,809,765 B1
`
`FROM 72
`
`74
`1N/
`
`SEPARATION
`
`FIRST COLORTRANSFORMATION
`
`LUMINANCE INTERPOLATION
`
`SECOND COLORTRANSFORMATION
`
`CHROMINANCE INTERPOLATION
`
`NVERSE COLORTRANSFORMATION
`
`COMBINATION
`
`TO 76
`
`FIG. 5
`
`200
`
`202
`
`204
`
`206
`
`208
`
`210
`
`212
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1031, p. 4
`
`

`

`U.S. Patent
`
`Oct. 26, 2004
`
`Sheet 4 of 6
`
`US 6,809,765 B1
`
`104
`
`100
`
`102
`
`
`
`
`
`GOG0IGO
`0 G0IGOG
`GOGOGO
`O GOGOG
`GOGOGO
`0 G0IGOG
`
`OTROROR
`000000
`OROROR
`0 0 0000
`OROROR
`000000
`
`
`
`
`
`
`
`
`
`
`
`F.G. 6
`
`10
`
`
`
`112
`
`114
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`GGGGGG RRRRRR
`GGGGGG RRRRRR
`GGGGGG RRRRRR
`GGGGGG RRRRRR
`GGGGGG RRRRRR
`GGGGGG RRRRRR
`FIG. 7
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1031, p. 5
`
`

`

`U.S. Patent
`
`Oct. 26, 2004
`
`Sheet 5 of 6
`
`US 6,809,765 B1
`
`6
`
`
`
`g
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`F.G. 8
`
`FIG. 9
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1031, p. 6
`
`

`

`U.S. Patent
`
`Oct. 26, 2004
`
`Sheet 6 of 6
`
`US 6,809,765 B1
`
`
`
`
`
`
`
`
`
`FIG. 10
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1031, p. 7
`
`

`

`US 6,809,765 B1
`
`1
`DEMOSAICING FOR DIGITAL MAGING
`DEVICE USING PERCEPTUALLY UNIFORM
`COLOR SPACE
`
`CROSS-REFERENCE TO RELATED
`APPLICATION
`The present application contains Subject matter related to
`a concurrently filed U.S. Patent Application by Bo Tao
`entitled “DEMOSAICING USINGWAVELETFILTERING
`FOR DIGITAL IMAGING DEVICE". The related applica
`tion is identified by application Ser. No. 09/412,351, is
`assigned to the same assignees, and is hereby incorporated
`by reference.
`
`TECHNICAL FIELD
`The present invention relates generally to digital Video
`and Still cameras, and more Specifically, to Such digital
`cameras using a Single-channel, Solid-State imaging Sensor
`array.
`
`15
`
`BACKGROUND OF THE INVENTION
`Solid-State digital video and Still cameras are becoming
`more and more popular. Most of the current digital cameras
`use a single channel, Solid-State photosensor array. For
`example, most of the digital Still cameras on the market now
`are so-called 1-CCD (single channel charge coupled device)
`cameras. The Solid-State photosensors are CCDS are
`arranged in an array in a matrix fashion with each CCD
`defining an element in the matrix called a “pixel’. By
`placing a mosaiced color filter over the photoSensor array, a
`Single channel image is created where each pixel has a Single
`color component. For example for a red-green-blue (RGB)
`System, a pixel has a single red (R), green (G), or blue (B)
`color component of a full color image. This forms a single
`channel mosaiced color image. The Single channel mosaiced
`color image has to be demosaiced Subsequently So that a full
`RGB (three-channel) image is created where each pixel
`contains the three RGB color components.
`Each Single-channel digital camera has its own unique
`demosaicing method. A number of patents have been filed on
`demosaicing methods such as U.S. Pat. No. 5,552,827
`granted to Maenaka et al. titled “Color Video Camera with
`a Solid State image Sensing Device” and U.S. Pat. No.
`4,716,455 granted to Ozawa et al. titled “Chrominance
`Signal Interpolation Device for a Color Camera'. However,
`these demosaicing methods did not take into account the
`filtering and Sampling process performed by the optical lens
`and the CCD. Thus, they often generate images that are not
`Sharp and/or have aliasing effects.
`Many programs have been published and/or patented on
`demosaicing. However, currently, demosaicing is done in a
`perceptually nonuniform color Space, Such as RGB space.
`Thus, false color artifacts can appear and the picture Sharp
`neSS is also limited. These have been long Standing problems
`in the industry.
`SUMMARY OF THE INVENTION
`The present invention provides a demosaicing proceSS
`which includes a first color transformation into a perceptu
`ally uniform color Space for determining luminance and a
`luminance interpolation for interpolation of the missing
`luminance component values. The demosaicing process fur
`ther includes a Second color transformation for determining
`chrominance where the color components are known using
`the luminance values. A chrominance interpolation is per
`formed to interpolate missing chrominance components and
`an inverse color transformation transforms the chrominance
`components into the RGB colors.
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`The present invention provides a digital camera having a
`lens for focusing light from an image through a RGB
`mosaiced filter on to a Solid State photoSensor array con
`taining a matrix of CCDs. The CCDs provide a single
`channel mosaiced image output. A demosaicing process
`includes a first color transformation for determining lumi
`nance and a luminance interpolation for interpolation of the
`missing luminance component values. The demosaicing
`process further includes a Second color transformation for
`transforming the other known color components using the
`luminance. A chrominance interpolation is performed to
`interpolate missing chrominance components and an inverse
`color transformation transforms the components into the
`RGB colors. The present invention further provides Super
`resolution imaging for any application involving color
`image data interpolation.
`The present invention further provides for generating
`high-resolution image/video from a low-resolution one. The
`present invention further provides a method to interpolate an
`NTSC or CCIR-601 signal into a HDTV signal. The above
`and additional advantages of the present invention will
`become apparent to those skilled in the art from a reading of
`the following detailed description when taken in conjunction
`with the accompanying drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 is a Solid State digital camera incorporating the
`present invention;
`FIG. 2 is a mosaiced image in a Bayer pattern;
`FIG. 3 is a mosaiced image in a Strip pattern;
`FIG. 4 is a flow chart of the process used in the Solid state
`digital camera of FIG. 1;
`FIG. 5 is a flow chart of the process used in the demo
`Saicing of the present invention;
`FIG. 6 is a Series of Separated mosaiced image planes
`which are outputs of the Separation process of the present
`invention;
`FIG. 7 is a Series of Separated demosaiced image planes
`which are inputs to the combination process of the present
`invention;
`FIG. 8 is the output of the second color transformation of
`the present invention;
`FIG. 9 is the output of the chrominance interpolation of
`the present invention; and
`FIG. 10 is a full RGB image output after the demosaicing
`of the present invention.
`DETAILED DESCRIPTION OF THE
`INVENTION
`Referring now to FIG. 1, therein is shown a solid state
`digital camera 10 having a camera body 12 and an attached
`lens system 14 with a lens filter 15. A single lens reflex
`camera is shown as an example where the present invention
`is implemented, but it would be evident that the present
`invention would work for all Still and motion cameras as
`well.
`The camera 10 has a mirror 16 which pivots on a pivot 18
`to initially direct light both from an image as well as ambient
`light through the lens filter 15 and the lens system 14 up to
`a prism 20 and out through an eyepiece 22.
`When a picture is being taken, the mirror 16 is pivoted up
`So as to allow light to Strike a recording medium 24. In a
`digital still camera (DSC), the digital recording medium 24
`is a matrix of photosensors, Such as analog output, Single
`channel, charge-coupled devices (CCDS) 26. Each of the
`CCDS 26 represents one picture element, or pixel, of the
`picture and the full array is called the “mosaic' pattern.
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1031, p. 8
`
`

`

`3
`Although cyan, magenta, yellow, and green filters can be
`used, each of the CCDs 26 herein has a green (G), red (R),
`or blue (B) filter over it, generally designated as the mosaic
`filter 28. The image recorded initially by these CCDs 26 is
`then electronically or magnetically recorded digitally for
`later playback. A processing unit 30 incorporating the
`present invention performs the processing of the Signals
`from the CCDs 26 to the output of the image.
`Referring now to FIG. 2, therein is shown a portion of a
`mosaiced image 50 in a pattern having columns 52, labeled
`“i” with only columns 1 through 6 shown, and rows 54,
`labeled “” with only rows 1 through 6 shown. One example
`of a mosaiced image 50 is a pattern disclosed in U.S. Pat.
`No. 3,971,065 granted to Bryce Bayer titled “Color Imaging
`Array', which is known as a Bayer pattern. Row 1 starts
`with G and alternates with R. Row 2 starts with B and
`15
`alternates with G. The subsequent rows alternate with GRG
`and then BGB.
`Referring, now to FIG. 3, therein is shown a portion of
`another mosaiced image 55 in a pattern having columns 57,
`labeled “i” with only columns 1 through 6 shown, and rows
`59, labeled “” with only rows 1 through 6 shown. One
`example of a mosaiced image 55 is known as a “strip'
`pattern. Column 1 is all G; column 2 is all R; and column 3
`is all B. The columns then repeatedly repeat G, R, and then
`B.
`Referring now to FIG. 4, therein is shown a flow chart 70
`of the Solid State digital camera 10 with light from an image
`passing through the mosaic filter 28 on to the CCDs 26. The
`CCDs 26 photo-electrically convert the light falling on them
`into proportional analog electrical Signals in photoSensor
`conversion 71. The analog signals are converted to digital
`Signals by an analog to digital (A/D) conversion 72. The
`digital signals are then output for demosaicing 74. After
`demosaicing 74, the output is Subject to image processing
`76. In the image processing 76, the output of the demosa
`icing 74 will go through the Series of conventional image
`processing Steps, including white balancing and
`compression, and finally be output to the image output 78 for
`user devices Such as a recording unit or display device.
`Generally, as would be evident to those skilled in the art, the
`mechanisms or units for performing the Steps of the method
`would be a matter of design based on the descriptions
`provided herein.
`Referring now to FIG. 5, therein is shown the demosaic
`ing 74 program using color transformations into perceptu
`ally uniform color spaces. The term “luminance (L) is used
`herein in a broad sense to refer to the color vector which is
`the major contributor of luminance information. The term
`"chrominance” refers to those color vectors a and b* other
`than the luminance color vectors which provide a basis for
`defining an image. The digital signals from the A/D con
`version 72 are separated into image planes in a separation
`200 process.
`Referring now to FIG. 6, therein is shown a series of three
`mosaiced image planes 100,102, and 104 which are outputs
`of the separation 200.
`Referring back to FIG. 5, the outputs of the separation 200
`undergo a first color transformation 202, which transforms
`the known color components into a perceptually uniform
`color Space where L can be determined. L is then used to
`interpolate missing L components by luminance interpola
`tion 204. The output of the luminance interpolation 204 then
`undergoes a Second color transformation 206 which trans
`forms the known color components into chrominance Space.
`The output of the second color transformation 206 is input
`ted into chrominance interpolation 208. After interpolation
`of the missing chrominance components, an inverse color
`transformation 210 is applied which is combined in combi
`nation 212 process. The combination 212 outputs to the
`image output 78.
`
`65
`
`45
`
`50
`
`55
`
`60
`
`US 6,809,765 B1
`
`25
`
`35
`
`40
`
`4
`Referring now to FIG. 7, therein is shown a series of three
`demosaiced image planes 110, 112, and 114 which are the
`inputs to the combination 212.
`Referring now to FIG. 8, therein is shown the output of
`the second color transformation is 206 which is an array 250
`Starting with a column with alternating Land Lb alternating
`with columns with alternating La and L.
`Referring now to FIG. 9, therein is shown the output of
`the chrominance interpolation 208 which is an array 260
`filled with Lab.
`Referring now to FIG. 10, therein is shown a full demo
`saiced RGB image 270 that is an array filled with RGB.
`In use, light from the image would enter the camera 10
`through the lens filter 15 and lens system 14 to be reflected
`by the mirror 16 which would be in the down position. The
`light would be reflected upwards into the prism 20 and be
`reflected through the prism 20 to exit out the eyepiece 22
`where the user could see the exact image Seen through the
`lens filter 15 and the lens system 14.
`When a picture is being taken, the mirror 16 pivots up and
`out of the way So light passes through the mosaic filter 28
`and strikes the single-channel solid-state CCDs 26. After a
`Single-channel image is created from the mosaiced Sensor
`array, each pixel has one color component, R, G, or B and
`a mosaiced image 50 in the Bayer pattern of FIG. 2 or a
`mosaiced image 55 in the strip pattern of FIG. 3 is formed.
`The mosaiced image 50 or 55 will be digitized by the A/D
`conversion 72.
`Demosaicing is Subsequently applied to generate the
`missing color components. For example, for the pixel at the
`upper-left corner (bold-faced and italicized) in FIG. 2
`(PRIOR ART), only the G component is created from the
`CCDs 26. The missing R and B components have to be
`generated by demosaicing 74. As yet another example, for
`the pixel in the middle R (bold-faced and italicized), the
`missing G and B components will be generated through
`demosaicing. The output is Subsequently Subject to demo
`Saicing 74 where the mosaiced image 50is. demosaiced and
`the full RGB image 270 is generated. The full RGB image
`270 after demosaicing is shown in FIG. 8.
`The demosaicing 74 plays an important role in the digital
`camera 10. It has direct impact on the image quality. The
`filter used in demosaicing needs to have good frequency
`response, be interpolating, and be Smooth. To meet these
`requirements interpolators, or filters, Such as “wavelet fil
`ters” are used in the present invention. Wavelet filters have
`been used for the analysis and the Synthesis of Signals,
`especially when Such signals correspond to Sounds or
`imageS. Previously, they have been used for imageS pre
`dominantly in image compression. A “wavelet transform'
`allows the representation of any arbitrary signal as a Super
`position of wavelets. The wavelets are functions generated
`from a single function by dilations and translations and they
`allow decomposition of a signal into different levels (each of
`which is further decomposed with a resolution adapted to the
`particular level). A description of the wavelet filters is
`provided in concurrently filed U.S. Patent Application Ser.
`No. 09/412,351 entitled DEMOSAICING USING WAVE
`LET FILTERING FOR DIGITAL IMAGING DEVICE, by
`Bo Tao, Supra, which is incorporated by reference. The
`demosaicing 74 of the present invention is shown in detail
`in FIG. 5.
`The mosaiced output from A/D conversion 72, as shown
`in FIG. 2, is provided for demosaicing 74, as shown in FIG.
`4. The mosaiced output is first Separated by the data sepa
`ration 200 into the three Separated mosaiced image planes
`100, 102, and 104, as shown in FIG. 6. The mosaiced image
`planes 100, 102 and 104 are the inputs for first color
`transformation 202 Such that:
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1031, p. 9
`
`

`

`US 6,809,765 B1
`
`L=f(G),
`R'=f(R), and
`B'=f(B),
`where: f()=log() or f(t)=()' where C. can be chosen to be
`/3 or a reasonable number that approximates the retinal
`response of the human vision System.
`In the alternative, f can be in other forms that approxi
`mates the retinal response of the human vision System.
`The L component of the output of the first color trans
`formation 202 is in a perceptually uniform color Space by
`Virtue of the above approximation and provides the lumi
`nance which can be Subsequently used.
`A number of methods can be used for estimating the
`missing L component values, including linear, bilinear, and
`bicubic, among others. The output image of the luminance
`interpolation has a known L component at each pixel posi
`tion.
`The luminance interpolation 204 uses a quincunx wavelet
`filter in the preferred embodiment. The filter is two
`dimensional. The coefficients of one Such filter are given in
`Table I below:
`
`6
`The luminance interpolation 204 performs the following
`Summation:
`
`4.
`
`with k given by:
`
`4
`4
`k = X. X. F (m, n).
`
`15
`
`Other thresholds can be used also. The output of the
`luminance interpolation 204 is provided for the second color
`transformation 206.
`With L known at all pixel positions now, R' and B' are
`subject to a second color transformation 206 where the R
`and B components are known to determine the chrominance
`components:
`a=L-R'; and
`b*=L-B'
`
`TABLE I
`
`O
`O
`O
`O
`O
`O
`-1
`O
`O
`O
`O
`O
`O
`
`O
`O
`O
`O
`O
`-6
`O
`-6
`O
`O
`O
`O
`O
`
`O
`O
`O
`O
`-15
`O
`60
`O
`-15
`O
`O
`O
`O
`
`O
`O
`O
`-20
`O
`294
`-128
`294
`O
`-20
`O
`O
`O
`
`O
`O
`-15
`O
`456
`-384
`-993
`-384
`456
`O
`-15
`O
`O
`
`O
`-6
`O
`294
`-384
`-2604
`4992
`-2604
`-384
`294
`O
`-6
`O
`
`-1
`O
`60
`-128
`-993
`4992
`26608
`4992
`-993
`-128
`60
`O
`-1
`
`O
`
`O
`294
`-384
`-2604
`4992
`-2604
`-384
`294
`O
`
`O
`
`O
`O
`-15
`O
`456
`-384
`-993
`-384
`456
`O
`-15
`O
`O
`
`O
`O
`O
`-20
`O
`294
`-128
`294
`O
`-20
`O
`O
`O
`
`O
`O
`O
`O
`-15
`O
`60
`O
`-15
`O
`O
`O
`O
`
`O
`O
`O
`O
`O
`-6
`O
`-6
`O
`O
`O
`O
`O
`
`However, other wavelet filters can be used as well. Where
`F represents this filter and X, represents the L output, at
`pixel position (i,j), the interpolated data is given by the
`following equation:
`
`40
`
`6
`
`Yi (i, j) = X
`
`6
`
`where Y, is the interpolated L output as shown in FIG. 8.
`
`An alternative embodiment of this invention can be
`realized by thresholding the filter F, i.e. Setting those
`coefficients of F. Smaller than a threshold to zero. This can
`reduce the computational cost. One thresholded filter, F, is
`shown in Table II below with the threshold set at 15:
`
`TABLE II
`
`O
`0 -15
`O
`294
`O -2O
`O
`-384
`-15
`O
`456
`O
`294 -3.84 -2604
`60 -128 -993
`4992
`O
`294 -3.84 -2604
`-15
`O
`456
`-384
`O -2O
`O
`294
`O
`0 -15
`O
`
`O
`O
`0 -15
`60
`O
`294
`O -2O
`-128
`-384.
`456
`0 -15
`-993
`4992 -2604 -3.84
`294
`O
`26608
`4992 -993 -128
`60
`4992 -2604 -3.84
`294
`O
`-993
`-384.
`456
`0 -15
`-128
`294
`O -2O
`O
`60
`0 -15
`O
`O
`
`45
`
`50
`
`55
`
`60
`
`65
`
`After the second color transformation 206, the image
`output is in the form of the array 250 shown in FIG. 8.
`The missing chrominance Signals a and b are then
`subject to chrominance interpolation 208. A number of
`algorithms can be used in the chrominance interpolation
`208, Such as linear, bilinear, bicubic, and so forth. One Such
`interpolation filter is given by:
`F=-0.045636 -0.028772 0.29563.6 0.557543 0.295636
`-0.028772 -0.045636)
`However, other filters can be employed too. Two filtering
`processes need to be performed, once in the horizontal
`direction and the other one in the vertical direction. The
`order in which to perform the two processes is arbitrary. In
`the following, the program does the Vertical filtering first.
`When the filtering is applied in the vertical direction, at pixel
`position (i,j), the output is given by:
`
`where: X. represents the at output of the Second color
`transformation 206.
`After Z has been computed, each row is flipped. Then:
`Z(i,j)=Z(i.N-1-)
`where: Z, represent this new signal with pixel indeX
`i=0,..., N-1, j=0,..., N-1.
`Horizontal filtering is performed on Z and the at output
`of the chrominance interpolation 208 is given by:
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1031, p. 10
`
`

`

`US 6,809,765 B1
`
`7
`
`8
`
`Each row of Y. is flipped to get the output Y. Then:
`
`The b interpolation uses the same filter F as the a*
`interpolation. The only difference between these two inter
`polations is that in the b interpolation, the two flipping
`processes are performed during vertical filtering, while no
`flipping is needed for horizontal filtering.
`The output of the chrominance interpolation is in the form
`of the array 260 of FIG. 9 with Lab values known at all
`pixel positions.
`An inverse color transformation 210 brings the image 260
`of FIG. 9 back into RGB space by doing the following
`computation:
`
`1O
`
`15
`
`After combination 212, the full RGB image 260 output is
`then provided to the image processing 76. In the image
`processing 76, the RGB image will go through the Series of
`conventional image processing Steps, including white bal
`ancing and compression, and finally be output as the image
`output 78.
`AS described above, the present invention provides Super
`resolution imaging for any application involving data inter
`polation and more particularly for generating high
`resolution image/video from a low-resolution image/video.
`One application is in the interpolation of an NTSC or
`CCIR-601 signal into a HDTV signal.
`While the invention has been described in conjunction
`with a specific best mode, it is to be understood that many
`alternatives, modifications, and variations will be apparent
`to those skilled in the art in light of the aforegoing descrip
`tion. Accordingly, it is intended to embrace all Such
`alternatives, modifications, and variations which fall within
`the Spirit and Scope of the included claims. All matterS Set
`forth herein or shown in the accompanying drawings are to
`be interpreted in an illustrative and non-limiting Sense.
`What is claimed is:
`1. A demosaicing method comprising the Steps of:
`receiving patterned mosaiced images in a plurality of
`colors, and
`demosaicing icing including the Steps of:
`Separating the patterned mosaiced images into a plu
`rality of Single color planes,
`a first color transformation on the plurality of Single
`color planes to determine luminance;
`a luminance interpolation on the output of Said first
`color transformation;
`a Second color transformation on the output or Said
`luminance interpolation to determine chrominance;
`a chrominance interpolation on the output of Said
`Second color transformation;
`an inverse color transformation on the output of Said
`chrominance interpolation to provide Single color
`images, and
`combining the output of Said inverse color transforma
`tion into a demosaiced color images,
`wherein the luminance interpolation makes an L interpo
`lation that:
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`where: Y, is a mosaiced output of the L interpolation, F.
`represents a filter, and X, represents a luminance output.
`2. The demosaicing method as claimed in claim 1 wherein
`Said Step of chrominance interpolation includes a step using
`a wavelet filter for filtration of at least one of the plurality of
`Single color images.
`3. The demosaicing method as claimed in claim 1 wherein
`Said Step of chrominance interpolation includes a step using
`a single-dimensional quincunx wavelet filter for filtration of
`at least one of the plurality of Single color images.
`4. The demosaicing method as claimed in claim 1 wherein
`Said Step of: luminance interpolation includes a step using a
`Single-dimensional quincunx wavelet filter.
`5. A demosaicing method in a digital camera comprising
`the Steps of:
`focusing light from an image through a lens,
`filtering the light into mosaiced red (R), green (G), and
`blue (B) lights;
`converting the mosaiced R, G, and Blights into patterned
`mosaiced images, and
`demosaicing including the Steps of:
`Separating the patterned mosaiced images into Single R,
`G, and B color planes,
`a first color transformation on the plurality of Single of
`R, G, and B color planes to determine luminance L,
`a luminance interpolation on the output of Said first
`color transformation;
`a Second color transformation on the output of Said
`luminance interpolation to determine chrominance,
`a* and b*;
`a chrominance interpolation on the output of Said
`Second color transformation;
`an inverse color transformation on the output of Said
`chrominance interpolation to provide Single color
`images, and
`combining the output of Said inverse color transforma
`tion into a demosaiced RBG image output,
`wherein the first color transformation makes a determi
`nation that:
`L=f(G), R'=f(R), and B'-f(B),
`where: L is the luminance and f()=log(); and
`luminance interpolation makes an L interpolation that:
`
`6
`
`Yi (i, j) = X
`
`6
`
`where: Y, is a mosaiced output of the L interpolation, F.
`represents a filter, and X, represents a luminance
`output.
`6. The demosaicing method in a digital camera as claimed
`in claim 5 wherein said steps of:
`Second color transformation makes the transformation
`that:
`a=L-R" and b*=L-B'
`where: a* and b* are chrominance components,
`chrominance interpolation makes the interpolation of
`missing a and b; and
`inverse color transformation makes the inverse color
`transformation that:
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1031, p. 11
`
`

`

`US 6,809,765 B1
`
`9
`7. The demosaicing method in a digital camera as claimed
`in claim 6 wherein Said Step of chrominance interpolation
`includes a step of filtration using a one-dimensional wavelet
`filter.
`8. The demosaicing method in a digital camera as claimed
`in claim 7 wherein Said Step of filtration using Said one
`dimensional wavelet filter is performable in a horizontal
`direction and in a vertical direction.
`9. The demosaicing method in a digital camera as claimed
`in claim 8 including a step of: flipping the output of the
`filtration twice after the vertical direction filtration.
`10. The demosaicing method in a digital camera as
`claimed in claim 5 wherein Said Step of: luminance inter
`polation includes a Step of filtration using a quincunx
`wavelet filter.
`11. The demosaicing method in a digital camera as
`claimed in claim 10 wherein said step of filtration uses a
`threshold quincunx wavelet filter whereby the filter coeffi
`cients of Said quincunx wavelet filter below a predetermined
`value are Set to Zero.
`12. A digital camera comprising:
`a lens for focusing light from an image;
`a mosaiced filter for converting the light into mosaiced
`colors,
`a photoSensor array associated with Said mosaiced filter
`and having a plurality of elements for receiving each of
`the plurality of mosaiced colors and providing a plu
`rality of outputs as a patterned mosaic image;
`a demosaicing unit connected to Said photosensor array
`and including:
`a first color transformation unit operatively connected
`to Said photosensor array,
`a luminance interpolation unit operatively connected to
`Said first color transformation unit;
`a Second color transformation unit operatively con
`nected to Said luminance interpolation unit;
`a chrominance interpolation unit operatively connected
`to Said Second color transformation unit;
`an inverse color transformation unit operatively con
`nected to Said chrominance interpolation unit; and
`an image output device operatively connected to Said
`demosaicing unit to output a demosaiced image,
`wherein luminance interpolation makes an the L interpo
`lation that:
`
`6
`
`Yi (i, j) = X
`
`6
`
`where: Y, is a mosaiced output of the L interpolation, F.
`represents a filter, and X, represents a luminance
`output.
`13. The digital camera System as claimed in claim 12
`wherein: Said demosaicing unit includes a separation unit for
`Separating the plurality of colors into Separate color images,
`and Said demosaicing unit includes a combination unit for
`combining Said Separate color images.
`14. The digital camera System as claimed in claim 13
`wherein: one of Said interpolation units includes a one
`dimensional wavelet filter.
`15. The digital camera system as claimed in claim 14
`wherein: Said one-dimensional wavelet filter is capable of
`performing a filtration in a horizontal direction and a filtra
`tion in a vertical direction.
`16. The digital camera system as claimed in claim 15
`wherein: Said chrominance interpolation unit is capable of
`manipulating the output of a filtration a plurality of times a
`filtration.
`
`10
`17. The digital camera System as claimed in claim 12
`wherein: one to Said interpolation units includes a quincunx
`wavelet filter.
`18. The digital camera system as claimed in claim 17
`wherein: said quincunx wavelet filter is thresholded whereby
`the filter coefficients of said quincunx wavelet filter below a
`predetermined value are set to Zero.
`19. A digital camera comprising:
`a lens for focusing light from an image;
`a mosaiced filter for converting the light into mosaiced R,
`G, and B lights;
`a photoSensor array associated with Said mosaiced filter
`and having a plurality of elements for receiving each of
`the plurality of mosaiced R, G, and B lights and
`providing a plurality of outputs as a patterned mosaic
`image,
`a demosaicing unit connected to Said photoSensor array
`and including:
`a first color transformation unit operatively connected
`to Said photosensor arra