(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2010/0067789 A1
`Cai et al.
`(43) Pub. Date:
`Mar. 18, 2010
`
`US 20100067789A1
`
`(54) RECONSTRUCTION OF IMAGE INA BAYER
`PATTERN
`
`(75) Inventors: NE ES R CA
`ity sung Hsu, Uuperuno,
`
`Correspondence Address:
`MCROSOFT CORPORATION
`ONE MCROSOFT WAY
`REDMOND, WA 98052 (US)
`
`(73) Assignee:
`
`MICROSOFT CORPORATION,
`Redmond, WA (US)
`
`(21) Appl. No.:
`
`12/212,792
`
`(22) Filed:
`
`Sep. 18, 2008
`
`Publication Classification
`
`(51) Int. Cl.
`(2006.01)
`G06K 9/36
`(52) U.S. Cl. ........................................................ 382/166
`
`(57)
`ABSTRACT
`Architecture for decoding (demosaicing) a Source image and
`performing reconstruction directly from the Bayer pattern to
`reduce memory size and improve communication bandwidth.
`The architecture can be easily implemented in hardware such
`as in field programmable gate arrays (FPGAs) and applica
`tion-specific integrated circuits (ASICs).
`
`- 100
`
`
`
`
`
`
`
`104
`
`SOURCE
`IMAGE
`MOSAIC
`
`
`
`
`
`
`
`
`
`DECODING
`COMPONENT
`
`
`
`RECONSTRUCTION
`COMPONENT
`
`TARGET
`MAGE
`MOSAIC
`
`Align Ex. 1011
`U.S. Patent No. 9,962,244
`
`0001
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 1 of 10
`
`US 2010/0067789 A1
`
`104
`
`
`
`
`
`
`
`DECODING
`COMPONENT
`
`RECONSTRUCTION
`COMPONENT
`
`TARGET
`MAGE
`MOSAIC
`
`FIG. I.
`
`0002
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 2 of 10
`
`US 2010/0067789 A1
`
`104
`
`
`
`SOURCE
`IMAGE
`MOSAIC
`
`
`
`
`
`
`
`DECODING
`COMPONENT
`
`
`
`
`
`
`
`
`
`RECONSTRUCTION
`COMPONENT
`
`TARGET
`IMAGE
`MOSAIC
`
`INTERPOLATION
`ALGORITHMS
`
`FIG. 2
`
`0003
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 3 of 10
`
`US 2010/0067789 A1
`
`AT 300
`
`
`
`
`
`
`
`
`
`
`
`RX,Y
`RX -- 2,
`RX, -- 2
`lrox+2.Y +2).
`
`RX-8, Y.
`ROX-1.Y-2
`RX -- i.Y-- 3
`
`RY, Y
`Rox + 2. Y-1
`fX, Y - :
`RX -- 3. Y + 1
`
`308
`
`
`
`a
`
`----
`
`2.
`
`s:
`3
`:
`& y & X 2 3
`+++
`a
`3.
`s
`
`RX-3, Y-1}
`R(X+3.Y-1
`RX-1, Y+1)
`RX + i. Y+ 1
`
`FIG. 3
`
`0004
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 4 of 10
`
`US 2010/0067789 A1
`
`a 400
`
`402 N
`
`SCENARIO S1
`
`404 N
`
`SCENARIO S2
`
`
`
`0005
`
`

`

`
`
`0006
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 6 of 10
`
`US 2010/0067789 A1
`
`
`
`0007
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 7 of 10
`
`US 2010/0067789 A1
`
`
`
`DECODE SOURCE MOSAIC OF
`PXELS OF SOURCE IMAGE INTO
`BASE COLOR PATTERNS
`
`RECONSTRUCT BASE COLOR
`PATTERNS CONCURRENTLY
`WITH DECODING TO GENERATE
`TARGET PIXELS IN TARGET
`MOSAIC OF TARGET IMAGE
`
`700
`
`702
`
`FIG. 7
`
`0008
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 8 of 10
`
`US 2010/0067789 A1
`
`START
`
`
`
`RECEIVE DECODED BASE COLOR
`PATTERNS
`
`800
`
`CREATE FULL COLOR PATTERNS
`FOREACH OF THE BASE COLOR
`PATTERNS USING BILINEAR
`INTERPOLATION
`
`CALCULATE LOCATION OF
`SOURCE PXEL BASED ON
`CORRESPONDING TARGET PIXEL
`
`APPLY PIXEL DISTRIBUTION
`ALGORITHM BASED ON
`LOCATION OF SOURCE PIXEL
`
`802
`
`804
`
`806
`
`FIG. 8
`
`0009
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 9 of 10
`
`US 2010/0067789 A1
`
`
`
`SELECTION PIXEL DISTRIBUTION
`ALGORITHM
`
`900
`
`CALCULATE COLOR VALUE FOR
`RED AND BLUE TARGET PIXEL
`BASED ON LOCATION OF
`SOURCE PIXELUSING THREE
`ADJACENT VIRTUAL PIXELS
`
`CALCULATE COLOR VALUE FOR
`GREEN TARGET PIXEL BASED ON
`LOCATION OF SOURCE PIXEL
`USING TWO ADJACENT VIRTUAL
`PXELS AND SIX REAL PIXELS
`
`902
`
`904
`
`FIG, 9
`
`0010
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 10 of 10
`
`US 2010/0067789 A1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`1002
`
`REMOVABLE
`MEMORY
`SUBSYSTEM
`
`PROCESSING
`UNIT(S)
`
`MEMORY
`SUBSYSTEM
`1 OO
`
`1012
`
`
`
`rs4.
`MODULES
`
`STORAGE
`INTERFACE(S)
`
`STORAGE
`SUBSYSTEM(S)
`
`WRE/WIRELESS
`COMMUNICATIONS
`SUBSYSTEM
`
`
`
`GRAPHICS
`INTERFACE(S)
`
`DISPLAY
`
`
`
`
`
`
`
`ON BOARD
`USER INPUT
`DEVICES
`
`/O DEVICE
`INTERFACE(S)
`
`FIG. I.0
`
`NETWORKS,
`COMPUTERS,
`WIRELESS
`PERIPHERALS,
`WIRELESS
`INPUT
`DEVICES,...
`
`EXTERNAL
`DISPLAY (S)
`
`EXTERNAL
`USER INPUT
`DEVICES
`1034
`OUTPUT
`PERIPHERALS
`
`0011
`
`

`

`US 2010/0067789 A1
`
`Mar. 18, 2010
`
`RECONSTRUCTION OF IMAGE IN ABAYER
`PATTERN
`
`BACKGROUND
`0001 Single and multi-sensor cameras are predominant in
`image or video capture systems. The cameras typically
`employ CCD or CMOS sensors that utilize a Bayer pattern
`color filter array (where each pixel only contains one color
`value). When using a Bayer pattern color filter array, the
`captured image is a mosaic of red, green, and blue (RGB)
`colors where each pixel is missing two of the three color
`values. Accordingly, these missing color values need to be
`interpolated to obtain an approximation of the full RGB val
`ues for each pixel to provide a richer output image.
`0002 Vendors can provide different methods to decode
`the source image to the full RGB patterns in the output image,
`including bilinear reconstruction, nearest neighbor, cubic,
`cubic spline, etc., all which have different computational
`speed and accuracy tradeoffs. As handheld devices become
`Smaller, the amount of available memory and computing
`power can become limiting factors while consumer demand
`remains high for better image and video quality.
`0003. However, many of the reconstruction processes for
`images are based on the full RGB pattern; therefore, memory
`size and communication bandwidth remain big issues and are
`not easy to implement in hardware.
`
`SUMMARY
`0004. The following presents a simplified summary in
`order to provide a basic understanding of some novel embodi
`ments described herein. This Summary is not an extensive
`overview, and it is not intended to identify key/critical ele
`ments or to delineate the scope thereof. Its sole purpose is to
`present some concepts in a simplified form as a prelude to the
`more detailed description that is presented later.
`0005. The disclosed architecture decodes (demosaics) a
`Source image and performs reconstruction directly from the
`Source mosaic pattern (e.g., Bayer) to the target mosaic pat
`tern (e.g., Bayer) to reduce memory size and improve com
`munication bandwidth. Moreover, the architecture can be
`easily implemented in hardware such as in field program
`mable gate arrays (FPGAs) and application-specific inte
`grated circuits (ASICs).
`0006. The architecture employs multiple different algo
`rithms for calculating the target pixel based on real and virtual
`source pixels based on the location of the source pixel. The
`algorithms apply to full red, blue and green base color pat
`terns.
`0007 To the accomplishment of the foregoing and related
`ends, certain illustrative aspects are described herein in con
`nection with the following description and the annexed draw
`ings. These aspects are indicative of the various ways in
`which the principles disclosed herein can be practiced, all
`aspects and equivalents of which are intended to be within the
`Scope of the claimed Subject matter. Other advantages and
`novel features will become apparent from the following
`detailed description when considered in conjunction with the
`drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0008 FIG. 1 illustrates a computer-implemented image
`processing system in accordance with the disclosed architec
`ture.
`
`0009 FIG. 2 illustrates a more detailed image processing
`system that employs interpolation algorithms for specific
`interpolation based on the position of the real pixels in the
`Source mosaic.
`0010 FIG. 3 illustrates the resulting equations of the
`above scenarios for red pixels.
`(0011
`FIG. 4 illustrates the scenarios of FIG. 3 on pixel
`SetS.
`0012 FIG. 5 illustrates the scenarios for the green pixel
`reconstruction in a Bayer pattern.
`0013 FIG. 6 illustrates a demosaicing process that
`decodes a mosaic and the equations for generating interpo
`lated pixels from the real pixels.
`0014 FIG. 7 illustrates a method of processing an image.
`0015 FIG. 8 illustrates a method of reconstructing a
`mosaic pattern.
`0016 FIG. 9 illustrates a method of applying pixel distri
`bution algorithms.
`0017 FIG. 10 illustrates a block diagram of a computing
`system operable to execute mosaic reconstruction in accor
`dance with the disclosed architecture.
`
`DETAILED DESCRIPTION
`
`0018. In the bilinear reconstruction of full RGB patterns,
`each pixel has three color values related to red (R), green (G)
`and blue (B). In image/video frame reconstruction, the target
`pixel is reconstructed from real pixels of the source image/
`frame. Denoting Tas the transform matrix and (X, Y) the
`target pixel coordinates, the relevant pixel in the Source
`image/frame is at the position (x, y). Since X and y are not
`likely to be integers, the Source pixel value can be calculated
`by interpolating from four adjacentreal pixels around the new
`position in Source image/frame.
`0019. The following Equation Set 1 is a bilinear recon
`struction method for full RGB patterns.
`
`x, y, 1 = T. X, Y, 1'
`
`1 - Ay-Ay-AyAy
`AX - AvAy
`Ay-AvAy
`AXAy
`1 - Ay-Ay + AvAy'
`AX - AvAy
`Ay-AyAy
`AXAy
`1 - Ay-Ay + AvAy I
`AX - AvAy
`Ay-AvAy
`AXAy
`
`R(X, Y)
`R(X + 1, Y)
`R(X, Y + 1)
`R(X + 1, Y + 1)
`G(X, Y)
`G(X + 1, Y)
`G(X, Y + 1)
`G(X + 1, Y + 1)
`B(X, Y)
`B(X + 1, Y)
`B(X, Y + 1)
`B(X + 1, Y + 1)
`
`R
`“”
`
`G
`“”
`
`B
`“”
`
`where X=x), Y=y; x means the largest integer that is
`smaller than X; AXX-X and Ayy-Y.
`0020 Many imagers (e.g., CMOS and CCD image/video
`sensors) use a Bayer pattern as the color filter array. Because
`of bandwidth or memory size limitation, it is sometimes
`desired to have the reconstruction outputs in the Bayer pattern
`the same as the Bayer pattern of the Source image/frame. The
`relevant pixels in the Source image/frame also use the four
`adjacent real pixels of the same color to interpolate the target
`
`0012
`
`

`

`US 2010/0067789 A1
`
`Mar. 18, 2010
`
`pixel value. The interpolation method is similar to that
`described above for full RGB reconstruction.
`0021
`Demosaicing (also referred to as mosaic decoding)
`interpolates a mosaic image in a Bayer pattern to an image in
`full RGB pattern. Bilinear interpolation can also be employed
`to do this. In bilinear interpolation, real values of real pixels
`are left untouched, and the interpolated color value is
`obtained from adjacent real pixel values of the same color.
`(The following set of equations (here, applied to the red (R)
`pixels) will be referred to generally hereinas Equation Set 3.)
`For example, where ared pixel R8 in the target image maps to
`a center position of four Surrounding real red pixels, the target
`red pixel R8 is interpolated by the simple average of the color
`values of the Surrounding real pixels in the source image Such
`that interpolated pixel R8=(R2+R4+R12+R14)/4. Similarly,
`for a green pattern target pixel that maps back to a position
`centered in four Surrounding real green pixels, the target
`green pixel G8 is interpolated by the simple average of the
`values of the Surrounding real green pixels in the Source
`image such that interpolated green pixel G8=(G3+G7+G9--
`G13)/4.
`0022. Where the target pixel maps directly to the source
`pixel, interpolation is such that the value of the target pixel is
`equivalent to the value of the real source pixel; thus, for
`example, an interpolated blue pixel B8 in the target image is
`the same as the real pixel value such that B8-B8. Where the
`target pixel maps to a position between two adjacent real
`pixels, the target pixel R13 in the red pattern, for example, is
`interpolated as the simple average of the adjacent real red
`pixel values, such that R13=(R12+R14)/2.
`0023 The disclosed architecture facilitates reconstruction
`directly from the Bayer pattern, which improves the perfor
`mance greatly while keeping the simplicity of implementa
`tion in hardware. Unlike reconstruction in full RGB patterns,
`reconstruction directly from the Bayer pattern facilitates a
`reduction in memory size and communication bandwidth.
`0024. Reference is now made to the drawings, wherein
`like reference numerals are used to refer to like elements
`throughout. In the following description, for purposes of
`explanation, numerous specific details are set forth in order to
`provide a thorough understanding thereof. It may be evident,
`however, that the novel embodiments can be practiced with
`out these specific details. In other instances, well known
`structures and devices are shown in block diagram form in
`order to facilitate a description thereof. The intention is to
`cover all modifications, equivalents, and alternatives falling
`within the spirit and scope of the claimed subject matter.
`0025 FIG. 1 illustrates a computer-implemented image
`processing system 100 in accordance with the disclosed
`architecture. The system 100 includes a decoding component
`102 for decoding a source image mosaic 104 of pixels of a
`Source image into source base color patterns, and a recon
`struction component 106 for reconstructing target pixels in a
`target mosaic 108 of a target image directly from the Source
`base color patterns.
`0026. As will be described herein, the mosaic of the source
`image and the mosaic of the target image can be Bayer pat
`terns. The reconstruction component 106 selects an interpo
`lation algorithm that calculates a color value for a target pixel
`based on a source pixel distribution that includes real and
`interpolated source pixels. The pixel distribution includes
`three interpolated source pixels and a real pixel, and applies to
`red and blue base color patterns. The pixel distribution can
`
`also include two interpolated source pixels and six real pixels,
`which applies to a green base color pattern.
`0027. In other words, the reconstruction occurs directly
`from the source mosaic to the target mosaic as a concurrent
`operation. The mosaics can be Bayer patterns such that the
`two processes of demosaicing and bilinear reconstruction are
`combined using equations that facilitate the direct reconstruc
`tion to provide better performance with nearly equal simplic
`ity.
`0028 FIG. 2 illustrates a more detailed image processing
`system 200 that employs interpolation algorithms 202 for
`specific interpolation based on the position of the real pixels
`in the source mosaic 104. The system 200 includes the decod
`ing component 102 for decoding a source Bayer pattern of
`pixels of a source image into Source base color patterns, and
`the reconstruction component 106 for reconstructing target
`pixels in a target mosaic of a target image directly from the
`Source base color patterns to generate target pixels in a target
`Bayer pattern of a target image. The reconstruction compo
`nent 106 selects one of the interpolation algorithms 202 that
`calculates an interpolated color value for a target pixel based
`on a real and interpolated pixel distribution in the base color
`patterns.
`0029. As is described herein below, the pixel distribution
`can include three interpolated Source pixels and a real pixel
`for red and blue base color patterns. The pixel distribution can
`include two interpolated Source pixels and six real pixels for
`a green base color pattern.
`0030 The description for the interpolation algorithms 202
`is based on red, blue and green pixels; however, it is to be
`understood that other color spaces can be employed. Such as
`for luma (Y) and chroma (Cr and Cb) components, for
`example.
`0031. The equations below are derived from the Equations
`Set 1 and Equation Set 3 above. The derivation for the con
`current computation of decoding and reconstruction begins
`with an input image in full RGB pattern, and according to the
`Equation Set 1, a target pixel value can be obtained in the
`output (or target) image using:
`
`R(X, Y) = R(x, y) =
`
`1 - Ay-Ay + AvAy
`AX-AXAy
`Ay-Ay-Ay
`AXAy
`
`R(X, Y)
`R(X + 1, Y)
`R(X, Y + 1)
`R(X + 1, Y + 1)
`
`0032. However, in the disclosed implementation, the
`Source image is a mosaic, as well as the target image. Since
`the images are mosaics (e.g., Bayer) patterns, this means that
`the source image in not a full RGB pattern. Demosaicing is
`used to obtain an interpolated (or “virtual) full RGB pattern.
`Thus, for the above equation, there can be four scenarios,
`described in general form as the following.
`0033 Scenario S1 is where R(X,Y) is a real pixel, while
`RCX--1,Y), R(X,Y+1), and RCX--1,Y+1) are not real pixels in
`the Bayer pattern of the source image. These non-real pixel
`values are then calculated from other real pixels, such that,
`
`0013
`
`

`

`US 2010/0067789 A1
`
`Mar. 18, 2010
`
`-continued
`
`-continued
`
`2
`R(X, Y +1): R(X, Y) + R(X, Y +2)
`
`R(X, Y)+R(X +2, Y)+
`R(X, Y+2)+R(X+2, Y+2)
`R(X+1,Y+1)=+
`
`Scenario $2 is where ROGl ,Y) is areal pixel, while
`[0034]
`R(X, Y), R(X, Y+l), and ROG l, Y+ l) are non-real pixels in
`the Bayer pattern, and are calculated from other real pixels.
`
`R(X, Y)R(X, Y) +
`R(X + 2, Y)
`l—Ax—Ay+AxAy ’ f
`_
`Ax-AXAV
`R(X, Y)+R(X, Y+2)
`Ay —AxAy
`' f _
`AxAy
`R(X, Y) + R(X +2, Y) +
`R(X, Y+2)+R(X+2, Y+2)
`f
`
`R(X, Y) =
`
`R(X—1, Y)+R(X+1, Y)
`2
`R(X—1,Y)+R(X+1,Y)+
`R(X—1,Y+2)+R(X+1,Y+2)
`4
`R(X+1,Y)+R(X+1,Y+2)
`2
`
`R(X,Y+1)=
`
`R(X+1,Y+1)=
`
`Scenario S3 is where R(X,Y+l) is a real pixel, while
`[0035]
`R(X, Y), ROGl , Y), and ROG l, Y+ l) are non-real pixels in
`the Bayer pattern, and are calculated from other real pixels.
`
`
`_ fl _ E + 4"“ ’
`4
`2
`2
`
`M Amy
`7 ‘ T
`Ay
`AxAy
`2
`4
`
`AxAy
`4
`
`R(X, Y)
`R(X + 2, Y)
`R(X, Y+2)
`R(X + 2, Y + 2)
`
`[0038]
`
`In scenario S2,
`
`l—Ax—Ay+AxAy ’
`Ax—AxAy
`Ay—AxAy
`AxAy
`
`-
`
`R(X,Y)
`R(X+1, Y)
`R(X,Y+l)
`R(X+1, Y+1)
`
`=
`
`R(X,Y—1)+R(X,Y+1)
`
`R(X“ Y1) = R(x, y) =
`
`R(X, Y) =
`
`f R
`
`(X,Y—1)+R(X+2,Y—1)+
`R(X,Y+1)+R(X+2,Y+1)
`4
`
`R(X+1,Y)=
`
`R(X+1,Y+1)=
`
`R(X, Y+1)+R(X+2, Y+1)
`2
`
`Scenario S4 is where, ROG l, Y+l) is a real pixel,
`[0036]
`while ROi, Y), ROG l , Y), and ROi, Y+l) are non-real pixels
`in the Bayer pattern, and are calculated from other real pixels.
`
`R(X—1, Y)+R(X+1, Y)
`2
`
`l-AX-Ay+AxAy’
`
`R(X+1,Y)R(X—1,Y)+
`
`Ax-AXAV
`Ay-AxAy
`AxAy
`
`_ R(X+1,Y)+R(X—1,Y+2)+
`R(X+1, Y+2)
`4
`R(X+1, Y)+R(X+1, Y+2)
`2
`
`2
`
`2
`
`4
`
`
`4
`
`AxAy
`Ay
`Ax
`1
`2+7_T_T _
`Ay
`AxAy
`4
`4
`
`AxAy
`Ay
`T +
`4
`
`R(X— 1, Y)
`R<X+1.Y)
`R(X — l, Y+2)
`R(X+1, Y+2)
`
`R(X—1,Y—1)+R(X+1,Y—1)+
`R(X—1,Y+1)+R(X+1,Y+1)
`
`f R
`
`(X+1,Y—1)+R(X+1,Y+1)
`
`f R
`
`(X—1,Y+1)+R(X+1,Y+1)
`2
`
`R(X, Y)=
`
`R(X+1, Y)=
`
`R(X, Y+1)=
`
`[0037] Making the substitutions in the Equation Set 1 for
`the red (R) pixels, the scenarios are the following. In scenario
`s1,
`
`[0039]
`
`In scenario S3,
`
`R(X“ Y1) = R(X, y) =
`
`l—Ax—Ay+AxAy ’
`Ax — AxAy
`Ay—AxAy
`AxAy
`
`'
`
`R(X,Y)
`R(X + 1, Y)
`R(X,Y+l)
`R(X+l, Y+l)
`
`=
`
`R(X,, Yr) =R(x, y)=
`
`l—Ax—Ay+AxAy ’
`Ax—AxAy
`Ay—AxAy
`MA};
`
`-
`
`R(X,Y)
`R(X +1, Y)
`R(X, Y+1)
`R(X+1, Y+1)
`
`=
`
`0014
`
`0014
`
`

`

`US 2010/0067789 A1
`
`Mar. 18, 2010
`
`-continued
`
`R(X, Y - 1) + R(X, Y + 1)
`2
`R(X, Y - 1) + R(X+2, Y - 1) +
`1 - Ax-Ay + AvAyl
`R(X, Y + 1) + R(X + 2, Y + 1)
`AX - AvAy
`Ay-Ay-Ay — —-
`AXAy
`R(X, Y + 1)R(X, Y + 1) +
`R(X + 2, Y + 1)
`2
`
`AX Ay AXAy
`1
`5- - - + -
`AX AXAy
`- - - -
`1. As AyAvAy
`2
`4
`AX AXAy
`- + -
`
`4
`
`2
`
`R(X, Y - 1)
`R(X + 2, Y - 1)
`R(X, Y + 1)
`
`R(X + 2, Y + 1)
`
`0040. In scenario S4,
`
`R(X, Y,) = R(x, y) =
`
`1 - Ay-Ay + AvAy I
`AX - AXAy
`Ay-Ay-Ay
`AXAy
`
`R(X, Y)
`R(X + 1, Y)
`R(X, Y + 1)
`R(X + 1, Y + 1)
`
`1 - Ay-Ay + AvAy'
`AX - AvAy
`Ay-Ay-Ay
`AXAy
`
`R(X - 1, Y - 1) + R(X + 1, Y - 1) +
`R(X - 1, Y + 1) + R(X + 1, Y + 1)
`4
`| R(X + 1, Y-1)+ R(X + 1, Y + 1) |
`2
`R(X - 1, Y + 1) + R(X + 1, Y + 1)
`2
`R(X + 1, Y + 1)
`
`1
`
`AX Ay AXAy I'
`- - - + -
`1
`AX Ay AXAy
`4 * 4
`4
`4
`1. As Ay AvAy
`4 4
`4
`4
`1
`AX Ay AXAy
`+ 1 + i + -
`
`R(X - 1, Y - 1)
`R(x + 1, Y - 1)
`R(X-1, Y + 1)
`R(X + 1, Y + 1)
`
`FIG. 3 illustrates the resulting equations of the
`0041
`above scenarios 300 for red pixels. Although derived for red
`pixels, the above scenarios also apply to the blue (B) pixels.
`Equation 302 (also referred to as scenario S1) calculates the
`color values for interpolated pixels when the real pixel is in a
`top-left position of a 4-pixel group. Equation 304 (also
`referred to as scenario S2) calculates the color values for
`
`interpolated pixels when the real pixel is in a top-right posi
`tion of a 4-pixel group. Equation 306 (also referred to as
`scenario S3) calculates the color values for interpolated pixels
`when the real pixel is in a bottom-left position of a 4-pixel
`group. Equation 308 (also referred to as scenario S4) calcu
`lates the color values for interpolated pixels when the real
`pixel is in a bottom-right position of a 4-pixel group.
`0042 FIG. 4 illustrates the scenarios of FIG. 3 on pixel
`sets 400. On a first pixel set 402, the first scenario S1 selects
`equation 302, since the real pixel P11 is in the top-left corner
`of three adjacent interpolated pixels P12, P21, and P22. On a
`second pixel set 404, the second scenario S2 selects equation
`304, since the real pixel P13 is in the top-right corner of three
`adjacent interpolated pixels P12, P22, and P23. On a third
`pixel set 406, the third scenario S3 selects equation 306, since
`the real pixel P31 is in the bottom-left corner of three adjacent
`interpolated pixels P21, P22, and P32. On a fourth pixel set
`408, the fourth scenario S4 selects equation308, since the real
`pixel P33 is in the bottom-right corner of three adjacent
`interpolated pixels P22, P23, and P32.
`0043. From above, it is known that in full RGB pattern
`reconstruction, the bilinear method uses the four adjacentreal
`pixels of the same color to interpolate the target pixel value. In
`FIG. 4, it is also desirable to real source pixels are used to
`interpolate the target pixel value; however, three of the four
`adjacent pixels are not real pixels captured from sensor, but
`virtual pixels interpolated from other real pixels. Thus, there
`can be four different pixel distributions for performing inter
`polation for reconstruction.
`0044 FIG. 5 illustrates the scenarios 500 for the green
`pixel reconstruction in a Bayer pattern. The scenarios 500
`include a first scenario 502 and a second scenario 504. The
`first scenario S1 involves real pixels in the top-left and bot
`tom-right positions, and the second scenario S2 involves real
`pixels in the top-right and bottom-left positions.
`0045 Although not derived here, the derivation method
`for the green pixel reconstruction is similar to the derivation
`applied for the red and blue pixels. The number of green
`pixels in the Bayer pattern is twice the number of the red or
`blue pixels. For green pixels, the relevant pixel in the source
`image may locate in the evaluation box of scenario S1 or
`scenario S2. Combining the equations for bilinear reconstruc
`tion and demosaicing, the target green pixel values can be
`calculated as in Equation 3 below.
`0046 For pixels located in the evaluation box of scenario
`S1:
`
`G(x, y) =
`(x, y)
`
`p
`
`AX AXAy
`- - - -
`3Ax 3Ay AvAy
`- - - - - - + - - || Pis
`AX AXAy
`P2
`- - - -
`P24
`Ay AXAy
`Ps
`4
`4
`P33
`A A AXA
`s -- s + - y
`P42
`Ay AXAy
`- - - -
`
`0015
`
`

`

`US 2010/0067789 A1
`
`Mar. 18, 2010
`
`0047 For pixels located in the evaluation box of scenario
`S2:
`
`1. As Ay AvAy
`4 4
`4
`4
`||P.
`- -, +
`13Ax-Ay-AvAy||
`4
`4
`4
`2
`P24
`“” A 3Ay AvAylp
`y
`y
`33
`4 - 4 * 4 - 2 ||
`AXAy
`P4
`4
`AXAy
`4
`
`where x,y,1'=T*X,Y, 1'; X=xandY=y): x means the
`largest integer that is Smaller than X, and AXX-X and Ayy
`Y.
`0048. In other words, scenario S1 calculates the interpo
`lated pixels P23 and P32 based on real pixels P13, P22, P24,
`P31, P33, and P42. The second scenario S2 calculates the
`interpolated pixels P23 and P34 based on real pixels P13, P22.
`P24, P33, P35, and P44.
`0049 FIG. 6 illustrates a demosaicing process 600 that
`decodes a mosaic 602 and the equations for generating inter
`polated pixels from the real pixels. The mosaic 602 includes
`red pixels (denoted as octagons), blue pixels (denoted as
`squares) and green pixels (denoted as diamonds). The mosaic
`602 is decoded into the three base (or constituent) color
`patterns: agreen (G) pixel pattern 604, a blue (B) pixel pattern
`606, and a red (R) pixel pattern 608. In the green color pattern
`604, the real pixels are, for example, G1, G3, G5, etc. The
`Equation Set 3 describes the general methods for computing
`the interpolated pixels G2, G4, G6, etc. For example, the
`interpolated pixel color value for G2 is calculated as (G1--G3
`)/2; the interpolated pixel color value for G8 is calculated as
`(G3+G7+G9+G13)/4, and so on, for all interpolated pixels.
`The same Equation Set 3 applies for calculating the interpo
`lated color values for the blue pixel pattern 606 and the red
`pixel pattern 608.
`0050. The following example applies one of the equation
`scenarios above for red pixel reconstruction for Bayer pat
`tern. The pixels represented by circles with cross-hatching are
`the virtual target pixels interpolated from adjacent real Source
`pixels by the reconstruction method. The pixels represented
`as octagons are the real (original) red pixels of the Source
`image captured by the image sensor. In other words, P11 is the
`real pixel that captured from sensor, and P12, P21, P22 are the
`virtual target pixels that are interpolated from other real
`source pixels. For example, P11=P11, P12=(P11+P13)/2:
`P21=(P11+P31)/2; and, P22=(P11+P13+P31+P33)/4.
`0051. The transform matrix T can be the following, for
`example.
`
`Accordingly, if X=1 and Y, , then x=1.5 and y=1.5 (using
`Equation Set 1) such that the source real pixel becomes P11.
`
`Thus, the first scenario S1 applies where the real pixel is in the
`top-left corner. In another example, if X=2 and Y-2, then
`x=2.5 and y=2.5 (using Equation Set 1), and real pixel P22 is
`in bottom-right corner Such that the equation of scenario S4
`applies. This process is the same for the blue pixels.
`0.052
`For pixels located in the evaluation box of scenario
`S1:
`
`R(x, y) =
`(x, y)
`
`AX Ay AXAy I'
`1 - - - + -
`AX AXAy
`- - - -
`Ay AXAy
`2
`4
`AXAy
`4
`
`P
`P3
`Ps
`P33
`
`0053 For pixels located in the evaluation box of scenario
`S2:
`
`R(x, y) =
`
`AX Ay AXAy
`1
`5- - - + -
`P
`1. As Ay AvAy
`2 t 2
`4
`4 || Pi,
`Ay AvAy
`Ps
`4
`4
`P33
`Ay AXAy
`-- --
`4
`4
`
`0054 For pixels located in the evaluation box of scenario
`S3:
`
`R(x, y) =
`
`AX Ay AXAy I'
`1
`5- - - + -
`Ay AXAy
`P
`- - - -
`P3
`1
`AX -- Ay AXAy
`Ps
`2 - 4
`2 - 4 ||
`Ay AXAy
`- + -
`
`0055 For pixels located in the evaluation box of scenario
`S4:
`
`1. As Ay AvAy
`- - - - - -
`1
`Av Ay AvAy
`- - - - - - - - - -
`4 4
`4
`4
`R(x, y) = 1
`AX Ay AXAy
`- - - - - - - - - -
`4 4
`4
`4
`1
`AX Ay AXAy
`+ 1 + i + -
`
`I
`P3
`Ps
`P33
`
`where x, y, 1'=T*DX, Y, 1'; X=x), Y=y): x means the
`largest integer that is Smaller than X, and AXX-X, Ayy-Y.
`0056 Included herein is a set of flow charts representative
`of exemplary methodologies for performing novel aspects of
`the disclosed architecture. While, for purposes of simplicity
`of explanation, the one or more methodologies shown herein,
`for example, in the form of a flow chart or flow diagram, are
`
`0016
`
`

`

`US 2010/0067789 A1
`
`Mar. 18, 2010
`
`shown and described as a series of acts, it is to be understood
`and appreciated that the methodologies are not limited by the
`order of acts, as some acts may, in accordance therewith,
`occur in a different order and/or concurrently with other acts
`from that shown and described herein. For example, those
`skilled in the art will understand and appreciate that a meth
`odology could alternatively be represented as a series of inter
`related States or events, such as in a state diagram. Moreover,
`not all acts illustrated in a methodology may be required for a
`novel implementation.
`0057 FIG. 7 illustrates a method of processing an image.
`At 700, a source mosaic of pixels of a source image is decoded
`into base color patterns. At 702, the base color patterns are
`reconstructed concurrently with the decoding to generate tar
`get pixels in a target mosaic of a target image. In other words,
`the target mosaic is reconstructed directly from the input
`mosaic.
`0058 FIG. 8 illustrates a method of reconstructing a
`mosaic pattern. At 800, decoded base color patterns are
`received. At 802, a full color pattern is created for each of the
`base color patterns using bilinear interpolation. At 804, the
`location of the source pixel is calculated based on the corre
`sponding target pixel. At 806, a pixel distribution algorithm is
`applied based on the location of the source (real) pixel.
`0059 FIG. 9 illustrates a method of applying pixel distri
`bution algorithms. At 900, a pixel distribution algorithm is
`selected. At 902, the color value for the red and blue target
`pixels is calculated based on the location of the Source pixel
`using three adjacent virtual pixels. At 904, the color value for
`the green target pixels is calculated based on the location of
`the six source pixels and using two adjacent virtual pixels.
`0060. As used in this application, the terms “component'
`and “system are intended to refer to a computer-related
`entity, either hardware, a combination of hardware and soft
`ware, Software, or Software in execution. For example, a
`component can be, but is not limited to being, a process
`running on a processor, a processor, a hard disk drive, mul
`tiple storage drives (of optical and/or magnetic storage
`medium), an object, an executable, a thread of execution, a
`program, and/or a computer. By way of illustration, both an
`application running on a server and the server can be a com
`ponent. One or more components can reside within a process
`and/or thread of execution, and a component can be localized
`on one computer and/or distributed between two or more
`computers. The word “exemplary' may be used herein to
`mean serving as an example, instance, or illustration. Any
`aspect or design described herein as “exemplary' is not nec
`essarily to be construed as preferred or advantageous over
`other aspects or designs.
`0061 Referring now to FIG. 10, there is illustrated a block
`diagram of a computing system 1000 operable to execute
`mosaic reconstruction inaccordance with the disclosed archi
`tecture. In order to provide additional context for various
`aspects thereof, FIG. 10 and the following discussion are
`intended to provide a brief, general description of the suitable
`computing system 1000 in which the various aspects can be
`implemented. While the description above is in the general
`context of computer-executable instructions that can run on
`one or more computers, those skilled in the art will recognize
`that a novel embodiment also can be implemented in combi
`nation with other program modules and/or as a combination
`of hardware and software.
`0062. The computing system 1000 for implementing vari
`ous aspects includes the computer 1002 having processing
`
`unit(s) 1004, a system memory 1006, and a system bus 1008.
`The processing unit(s) 1004 can be any of various commer
`cially available processors such as single-processor, multi
`processor, single-core units and multi-core units. Moreover,
`those skilled in the art will appreciate that the novel methods
`can be practiced with other computer system configurations,
`including minicomputers, mainframe computers, as well as
`personal computers (e.g., desktop, laptop, etc.), hand-held
`computing devices, microprocessor-based or programmable
`consumer electronics, and the like, each of which can be
`operatively coupled to one or more associated devices.
`0063. The system memory 1006 can include volatile
`(VOL) memory 1010 (e.g., random access memory (RAM))
`and non-volatile memory (NON-VOL) 1012 (e.g., ROM,
`EPROM, EEPROM, etc.). A basic input/output system
`(BIOS) can be stored in the non-volatile memory 1012, and
`includes the basic routines that facilitate the communication
`of data and signals between components within the computer
`1002, such as during startup. The volatile memory 1010 can
`also include a high-speed RAM such as static RAM for cach
`ing data.
`0064. The system bus 1008 provides an interface for sys
`tem components including, but not limited to, the memory
`subsystem 1006 to the processing unit(s) 1004. The system
`bus 1008 can be any of several types of bus structure that can
`further interconnect to a memory bus (with or without a
`memory controller), and a peripheral bus (e.g., PCI, PCIe,
`AGP, LPC, etc.), using any of a variety of commercially
`available bus architectures.
`0065. The computer 1002 further includes storage sub
`system(s) 1014 and storage interface(s) 1016 for interfacing
`the storage subsystem(s) 1014 to the system bus 1008 and
`other desired computer components. The storage Subsystem
`(s) 1014 can include one or more of a hard disk drive (HDD),
`a magnetic floppy disk drive (FDD), and/or optical disk stor
`age drive (e.g., a CD-ROM drive DVD drive), for example.
`The storage interface(s) 1016 can include inter