APPL-1001 / Page 1 of 16
`Apple Inc. v. Corephotonics
`
`

`

`US 9,538,152 B2
` Page 2
`
`(51)
`
`Int. Cl.
`H04N 9/73
`H04N 5/225
`H04N 9/04
`H04N 9/097
`(52) US. Cl.
`CPC ......... H04N 5/23296 (2013.01); H04N 9/045
`(2013.01); H04N 9/097 (2013.01); H04N
`2209/045 (2013.01)
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`7,676,146 B2
`8,094,208 B2 *
`
`8,134,115 B2
`8,149,327 B2
`8,179,457 B2 *
`
`3/2010 Border et 31.
`1/2012 Myhrvold ............ H04N 5/2254
`348/222.1
`
`3/2012 Koskinen et al.
`4/2012 Lin et al.
`5/2012 Koskinen .......... H01L 27/14621
`250/226
`
`8,439,265 B2
`8,542,287 B2
`8,553,106 B2
`8,660,420 B2
`2008/0030592 A1
`2010/0277619 A1
`2011/0064327 A1*
`
`2011/0121421 A1
`2011/0285730 A1
`2011/0292258 A1
`
`* cited by examiner
`
`5/2013 Ferren et al.
`9/2013 Griifith et 31.
`10/2013 Scarff
`2/2014 Chang
`2/2008 Border et al.
`11/2010 Scarff
`3/2011 Dagher
`
`5/2011 Charbon et al.
`11/2011 Lai et al.
`12/2011 Adler et al.
`
`................... G06T 5/004
`382/263
`
`APPL-1001 / Page 2 of 16
`
`APPL-1001 / Page 2 of 16
`
`

`

`U.S. Patent
`
`Jan. 3, 2017
`
`Sheet 1 of 7
`
`US 9,538,152 B2
`
`Dual Aperture Camera
`
`
`
`Wide Subset
`
`Tele Subset
`
`Processor
`
`108
`
`4'— Zoom factor
`
`
`
`
`
`¢
`Image
`
`FIG. 1A
`
`110
`
`112
`
`Wide Sensor image
`‘.
`
`Tele Sensor image
`
`FIG. 1B
`
`APPL-1001 / Page 3 of 16
`
`APPL-1001 / Page 3 of 16
`
`

`

`U.S. Patent
`
`Jan.3,2017
`
`Sheet2 0f7
`
`US 9,538,152 B2
`
`V’fiifi'h 3
`
`
`
`
`
`
`
`.1».
`(f
`, W;
`55m -
`
`B
`
`R
`
`B R
`
`B
`
`R B
`
`B R
`
`8 R
`R
`B
`
`‘GWRWGR
`
`
`
`R
`
`............._...
`EMR.
`
`
`
`La
`
`
`
`FIG. 3
`
`APPL-1001 / Page 4 of 16
`
`APPL-1001 / Page 4 of 16
`
`

`

`U.S. Patent
`
`Jan. 3, 2017
`
`Sheet 3 of 7
`
`US 9,538,152 B2
`
`
`
`404
`
`
`
`
`
`8EG E-E---===. G
` RT,
`
`FIG. 4
`
`M
`
`f??? IEég:";+,§§ét:::.s.GR.IG'fl6
`
`- u- u
`- u "u" "-"u u u -
`-
`-
`u- u
`:
`E6
`E;
`E..-
`
`FIG. 5
`
`APPL-1001 / Page 5 of 16
`
`APPL-1001 / Page 5 of 16
`
`

`

`U.S. Patent
`
`Jan.3,2017
`
`Sheet4 0f7
`
`US 9,538,152 B2
`
`Width 1
`
`GRGRGR
`GRGRGRGRGRGRGRGRGR
`
`BGBGBG
`BGBGBGBGBGBGBGBGBG
`
`GRGR
`GRGRGR
`
`
`GRGRGRBBGRRGBBGRRG
`BGBGBGRGRBGBRGRBGB
`
`
`
`
`
`GRGRGRGBRGRBGBRGRB
`GB
`RG
`RG
`
`BGBGBGRRGgBBGRRGBB
`RGRBGBRGR
`GRGRGRBGB
`GBRGRBGBR
`BGBGBGGRB
`RRGBBGRRG
`GRGRGRBBG
`BGBRGRBGB
`G B
`BGBGBGRGR
`
`GRBGBRGRB
`GRGRGRGBR
`BBGRRGBBG
`BGBGBGRRG
`
`RGRBGBRGR
`GRGRGRBGB
`BGBGBGGRB
`GBRGRBGBR
`
`RGRGR
`G
`G
`
`119Q.on
`3'N
`
`RGRG
`
`GBGB
`
`RGRG
`
`B RGRGR
`
`G R
`
`GRGR
`GBGBG
`
`604
`
`Imfiyfi
`
`206
`
` B BGBGBG GRGRGRGRGR Width 2
`
`
`
`
`
`
`
`.nuga”G........
`
`
`
`
`
`
`
`
`RGRGRGRGRGRGRGR
`
`GRGRGRGRG
`
`FIG. 6
`
`700
`
`704
`
`
`
`
`
`
`ppJppJpp.....».*”mafia
`
`....
`
`
`
`
`
`APPL-1001 / Page 6 of 16
`
`APPL-1001 / Page 6 of 16
`
`
`

`

`U.S. Patent
`
`Jan.3,2017
`
`Sheet5 0f7
`
`US 9,538,152 B2
`
`
`
`800
`
`
`
`
`
`ERR
`RRE 88R83
`
`
`
`
`R E
`
`
`
`
` BR
`
` 83R
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 8
`
`GR
`{SERGE
`GB8
`863
`G,
`
`802
`
`904
`
`902
`
`FIG. 9
`
`APPL-1001 / Page 7 of 16
`
`APPL-1001 / Page 7 of 16
`
`

`

`U.S. Patent
`
`Jan. 3, 2017
`
`Sheet 6 of 7
`
`US 9,538,152 B2
`
`Wide and Teie
`
`pixel data
`
`Perform de-mosaicing on the Wide subset image according
`to the pixel array
`
`1000
`
`1004
`
`Map the Teie image acquired with the Tele subset camera
`into the Wide image
`
`1002
`
`Transfer data from Wide and Tele images to form a high
`quality zoom image
`
`FIG. 10
`
`APPL-1001 / Page 8 of 16
`
`APPL-1001 / Page 8 of 16
`
`

`

`U.S. Patent
`
`Jan. 3, 2017
`
`Sheet 7 of 7
`
`US 9,538,152 B2
`
`1100
`
`Wide image
`
`Wide-Tele image
`
`Te1e image
`
`
`
`FIG. 11A
`
`Wide sensor (X1)
`
`Wide—Tele sensor (X15)
`
`Tele sensor (X2)
`
`Standard pattern
`
`Wide/Wide
`
`—
`
`Teie
`area CPA
`
`Overiap
`
`or standard CFA
`
`Clear pattern or standard
`CFA
`
`Clear pattern
`
`Wide~
`Tale/Tale
`
`Overlap area
`CFA
`
`1102
`
`1104
`
`1106
`
`FIG. 1113
`
`APPL-1001 / Page 9 of 16
`
`APPL-1001 / Page 9 of 16
`
`

`

`US 9,538,152 B2
`
`1
`HIGH RESOLUTION THIN
`MULTI-APERTURE IMAGING SYSTEMS
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a National Phase application from PCT
`patent application PCT/IB2013/060356 which claims prior-
`ity from US. Provisional Patent Application No. 61/730,570
`having the same title and filed Nov. 28, 2013, which is
`incorporated herein by reference in its entirety.
`
`FIELD
`
`Embodiments disclosed herein relate in general to multi-
`aperture imaging (“MAI”) systems (where “multi” refers to
`two or more apertures) and more specifically to thin MAI
`systems with high color resolution and/or optical zoom.
`
`BACKGROUND
`
`integrated into mobile (cell)
`Small digital cameras
`phones, personal digital assistants and music players are
`becoming ubiquitous. Each year, mobile phone manufactur-
`ers add more imaging features to their handsets, causing
`these mobile imaging devices to converge towards feature
`sets and image quality that customers expect from stand-
`alone digital still cameras. Concurrently, the size of these
`handsets is shrinking, making it necessary to reduce the total
`size of the camera accordingly while adding more imaging
`features. Optical Zoom is a primary feature of many digital
`still cameras but one that mobile phone cameras usually
`lack, mainly due to camera height constraints in mobile
`imaging devices, cost and mechanical reliability.
`Mechanical zoom solutions are common in digital still
`cameras but are typically too thick for most camera phones.
`Furthermore, the F/# (“F number) in such systems typically
`increases with the zoom factor (ZF) resulting in poor light
`sensitivity and higher noise (especially in low-light sce-
`narios). In mobile cameras, this also results in resolution
`compromise, due to the small pixel size of their image
`sensors and the diffraction limit optics associated with the
`F/#.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`One way of implementing zoom in mobile cameras is by
`over-sampling the image and cropping and interpolating it in
`accordance with the desired ZF. While this method is
`
`45
`
`it results in thick optics and in an
`mechanically reliable,
`expensive image sensor due to the large number of pixels
`associated therewith. As an example, if one is interested in
`implementing a 12 Megapixel camera with X3 ZF, one
`needs a sensor of 108 Megapixels.
`Another way of implementing zoom, as well as increasing
`the output resolution, is by using a dual-aperture imaging
`(“DAI”) system. In its basic form, a DAI system includes
`two optical apertures which may be formed by one or two
`optical modules, and one or two image sensors (e. g., CMOS
`or CCD) that grab the optical image or images and convert
`the data into the electronic domain, where the image can be
`processed and stored.
`The design of a thin MAI system with improved resolu-
`tion requires a careful choice of parameters coupled with
`advanced signal processing algorithms to support the output
`of a high quality image. Known MAI systems, in particular
`ones with short optical paths, often trade-off functionalities
`and properties, for example zoom and color resolution, or
`image resolution and quality for camera module height.
`Therefore, there is a need for, and it would be advantageous
`
`50
`
`55
`
`60
`
`65
`
`2
`
`to have thin MAI systems that produce an image with high
`resolution (and specifically high color resolution) together
`with zoom functionality.
`Moreover, known signal processing algorithms used
`together with existing MAI systems often further degrade
`the output
`image quality by introducing artifacts when
`combining information from different apertures. A primary
`source of these artifacts is the image registration process,
`which has to find correspondences between the different
`images that are often captured by different sensors with
`different color filter arrays (CFAs). There is therefore a need
`for, and it would be advantageous to have an image regis-
`tration algorithm that is more robust to the type of CFA used
`by the cameras and which can produce better correspon-
`dence between images captured by a multi-aperture system.
`
`SUMMARY
`
`Embodiments disclosed herein teach the use of multi-
`
`aperture imaging systems to implement thin cameras (with
`short optical paths of less than about 9 mm) and/or to realize
`optical zoom systems in such thin cameras. Embodiments
`disclosed herein further teach new color filter arrays that
`optimize the color information which may be achieved in a
`multi-aperture imaging system with or without zoom. In
`various embodiments, a MAI
`system disclosed herein
`includes at least two sensors or a single sensor divided into
`at least two areas. Hereinafter, the description refers to “two
`sensors”, with the understanding that they may represent
`sections of a single physical sensor (imager chip). Exem-
`plarily, in a dual-aperture imaging system, a left sensor (or
`left side of a single sensor) captures an image coming from
`a first aperture while a right sensor (or right side of a single
`sensor) captures an image coming from a second aperture. In
`various embodiments disclosed herein, one sensor is a
`“Wide” sensor while another sensor is a “Tele” sensor, see
`e.g. FIG. 1A. The Wide sensor includes either a single
`standard CFA or two different CFAs: a non-standard CFA
`
`with higher color sampling rate positioned in an “overlap
`area” of the sensor (see below description of FIG. 1B) and
`a standard CFA with a lower color sampling rate surrounding
`the overlap area. When including a single standard CFA, the
`CFA may cover the entire Wide sensor area. A “standard
`CF ” may include a RGB (Bayer) pattern or a non-Bayer
`pattern such as RGBE, CYYM, CYGM, RGBW#1,
`RGBW#2 or RGBW#3. Thus, reference may be made to
`“standard Bayer” or “standard non-Bayer” patterns or filters.
`As used herein, “non-standard CFA” refers to a CFA that is
`different in its pattern that CFAs listed above as “standard”.
`Exemplary non-standard CFA patterns may include repeti-
`tions of a 2x2 micro-cell in which the color filter order is
`RR-BB, RB-BR or YC-CY where Y:Yellow:Green+Red,
`C:Cyan:Green+Blue; repetitions of a 3x3 micro-cell
`in
`which the color filter order is GBR-RGB-BRG; and repeti-
`tions of a 6x6 micro-cell in which the color filter order is
`RBBRRB-RWRBWB-BBRBRR-RRBRBB-BWBRWR-
`BRRBBR, or
`BBGRRG-RGRBGB-GBRGRB-RRGBBG-BGBRGR-GR-
`BGBR, or
`RBBRRB-RGRBGB-BBRBRR-RRBRBB-BGBRGR-
`BRRBBR, or,
`RBRBRB-BGBRGR-RBRBRB-BRBRBR-RGRBGB-BR-
`BRBR.
`
`The Tele sensor may be a Clear sensor (i.e. a sensor
`without color filters) or a standard CFA sensor. This arrange-
`ment of the two (or more than two) sensors and of two (or
`more than two) Wide and Tele “subset cameras” (or simply
`
`APPL-1001 /Page 10 of 16
`
`APPL-1001 / Page 10 of 16
`
`

`

`US 9,538,152 B2
`
`3
`“subsets”) related to the two Wide and Tele subsets. Each
`sensor provides a separate image (referred to respectively as
`a Wide image and a Tele image), except for the case of a
`single sensor, where two images are captured (grabbed) by
`the single sensor (example above). In some embodiments,
`zoom is achieved by fusing the two images, resulting in
`higher color resolution that approaches that of a high quality
`dual-aperture zoom camera. Some thin MAI systems dis-
`closed herein therefore provide zoom, super-resolution, high
`dynamic range and enhanced user experience.
`In some embodiments,
`in order to reach optical zoom
`capabilities, a different magnification image of the same
`scene is grabbed by each subset, resulting in field of view
`(FOV) overlap between the two subsets. In some embodi-
`ments, the two subsets have the same zoom (i.e. same FOV).
`In some embodiments, the Tele subset is the higher zoom
`subset and the Wide subset is the lower zoom subset. Post
`
`processing is applied on the two images grabbed by the MAI
`system to fuse and output one fused (combined) output
`zoom image processed according to a user ZF input request.
`In some embodiments, the resolution of the fused image may
`be higher than the resolution of the Wide/Tele sensors. As
`part of the fusion procedure, up-sampling may be applied on
`the Wide image to scale it to the Tele image.
`In an embodiment
`there is provided a multi-aperture
`imaging system comprising a first camera subset that pro-
`vides a first image, the first camera subset having a first
`sensor with a first plurality of sensor pixels covered at least
`in part with a non-standard CFA, the non-standard CFA used
`to increase a specific color sampling rate relative to a same
`color sampling rate in a standard CFA; a second camera
`subset that provides a second image, the second camera
`subset having a second sensor with a second plurality of
`sensor pixels either Clear or covered with a standard CFA;
`and a processor configured to process the first and second
`images into a combined output image.
`In some embodiments, the first and the second camera
`subsets have identical FOVs and the non-standard CFA may
`cover an overlap area that includes all the pixels of first
`sensor,
`thereby providing increased color resolution.
`In
`some such embodiments, the processor is further configured
`to, during the processing of the first and second images into
`a combined output image, register respective first and sec-
`ond Luma images obtained from the first and second images,
`the registered first and second Luma images used together
`with color information to form the combined output image.
`In an embodiment, the registration includes finding a cor-
`responding pixel in the second Luma image for each pixel in
`the first Luma image, whereby the output image is formed
`by transferring information from the second image to the
`first image. In another embodiment, the registration includes
`finding a corresponding pixel in the first Luma image for
`each pixel in the second Luma image, whereby the output
`image is formed by transferring information from the first
`image to the second image.
`In some embodiments, the first camera subset has a first
`FOV, the second camera subset has a second, smaller FOV
`than the first FOV, and the non-standard CFA covers an
`overlap area on the first sensor that captures the second FOV,
`thereby providing both optical zoom and increased color
`resolution. In some such embodiments,
`the processor is
`further configured to, during the processing of the first and
`second images into a combined output image and based on
`a ZF input, register respective first and second Luma images
`obtained from the first and second images, the registered first
`and second Luma images used together with color informa-
`tion to form the combined output image. For a ZF input that
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`
`defines an FOV greater than the second FOV, the registration
`includes finding a corresponding pixel in the second Luma
`image for each pixel
`in the first Luma image and the
`processing includes forming the output image by transfer-
`ring information from the second image to the first image.
`For a ZF input that defines an FOV smaller than or equal to
`the second FOV, the registration includes finding a corre-
`sponding pixel in the first Luma image for each pixel in the
`second Luma image, and the processing includes forming
`the output image by transferring information from the first
`image to the second image.
`In an embodiment
`there is provided a multi-aperture
`imaging system comprising a first camera subset that pro-
`vides a first image, the first camera subset having a first
`sensor with a first plurality of sensor pixels covered at least
`in part with a standard CFA; a second camera subset that
`provides a second image, the second camera subset having
`a second sensor with a second plurality of sensor pixels
`either Clear or covered with a standard CFA; and a processor
`configured to register
`first and second Luma images
`obtained respectively from the first and second images and
`to process the registered first and second Luma images
`together with color information into a combined output
`image.
`In some embodiments, the first and the second camera
`subsets have identical first and second FOVs. In some such
`
`embodiments, the registration includes finding a correspond-
`ing pixel in the second Luma image for each pixel in the first
`Luma image and the processing includes forming the output
`image by transferring information from the second image to
`the first image. In other such embodiments, the registration
`includes finding a corresponding pixel in the first Luma
`image for each pixel in the second Luma image and the
`processing includes forming the output image by transfer-
`ring information from the first image to the second image.
`In some embodiments, the first camera subset has a first
`FOV, the second camera subset has a second, smaller FOV
`than the first FOV, and the processor is further configured to
`register the first and second Luma images based on a ZF
`input. For a ZF input that defines an FOV greater than the
`second FOV, the registration includes finding a correspond-
`ing pixel in the second Luma image for each pixel in the first
`Luma image and the processing includes forming the output
`image by transferring information from the second image to
`the first image. For a ZF input that defines an FOV smaller
`than or equal to the second FOV, the registration includes
`finding a corresponding pixel in the first Luma image for
`each pixel in the second Luma image, and the processing
`includes forming the output image by transferring informa-
`tion from the first image to the second image.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Non-limiting examples of embodiments disclosed herein
`are described below with reference to figures attached hereto
`that are listed following this paragraph. The drawings and
`descriptions are meant to illuminate and clarify embodi-
`ments disclosed herein, and should not be considered lim-
`iting in any way.
`FIG. 1A shows schematically a block diagram illustrating
`a dual-aperture zoom imaging system disclosed herein;
`FIG. 1B shows an example of an image captured by the
`Wide sensor and the Tele sensor while illustrating the
`overlap area on the Wide sensor;
`FIG. 2 shows schematically an embodiment of a Wide
`sensor that may be implemented in a dual-aperture zoom
`imaging system disclosed herein;
`
`APPL-1001 /Page 11 of 16
`
`APPL-1001 / Page 11 of 16
`
`

`

`US 9,538,152 B2
`
`5
`FIG. 3 shows schematically another embodiment of a
`Wide camera sensor that may be implemented in a dual-
`aperture zoom imaging system disclosed herein;
`FIG. 4 shows schematically yet another embodiment of a
`Wide camera sensor that may be implemented in a dual-
`aperture zoom imaging system disclosed herein;
`FIG. 5 shows schematically yet another embodiment of a
`Wide camera sensor that may be implemented in a dual-
`aperture zoom imaging system disclosed herein;
`FIG. 6 shows schematically yet another embodiment of a
`Wide camera sensor that may be implemented in a dual-
`aperture zoom imaging system disclosed herein;
`FIG. 7 shows schematically yet another embodiment of a
`Wide camera sensor that may be implemented in a dual-
`aperture zoom imaging system disclosed herein;
`FIG. 8 shows schematically yet another embodiment of a
`Wide camera sensor that may be implemented in a dual-
`aperture zoom imaging system disclosed herein;
`FIG. 9 shows schematically yet another embodiment of a
`Wide camera sensor that may be implemented in a dual-
`aperture zoom imaging system disclosed herein;
`FIG. 10 shows a schematically in a flow chart an embodi-
`ment of a method disclosed herein for acquiring and out-
`putting a zoom image;
`FIG. 11A shows exemplary images captured by a triple
`aperture zoom imaging system disclosed herein;
`FIG. 11B illustrates schematically the three sensors of the
`triple aperture imaging system of FIG. 11A.
`
`DETAILED DESCRIPTION
`
`Embodiments disclosed herein relate to multi-aperture
`imaging systems that include at least one Wide sensor with
`a single CFA or with two different CFAs and at least one Tele
`sensor. The description continues with particular reference
`to dual-aperture imaging systems that include two (Wide and
`Tele) subsets with respective sensors. Athree-aperture imag-
`ing system is described later with reference to FIGS. 11A-
`11B.
`
`The Wide sensor includes an overlap area (see description
`of FIG. 1B) that captures the Tele FOV. The overlap area
`may cover the entire Wide sensor or only part of the sensor.
`The overlap area may include a standard CFA or a non-
`standard CFA. Since the Tele image is optically magnified
`compared to the Wide image, the effective sampling rate of
`the Tele image is higher than that of the Wide image. Thus,
`the effective color sampling rate in the Wide sensor is much
`lower than the Clear sampling rate in the Tele sensor. In
`addition, the Tele and Wide images fusion procedure (see
`below) requires up-scaling of the color data from the Wide
`sensor. Up-scaling will not improve color resolution. In
`some applications,
`it
`is therefore advantageous to use a
`non-standard CFA in the Wide overlap area that increases
`color resolution for cases in which the Tele sensor includes
`
`only Clear pixels. In some embodiments in which the Tele
`sensor includes a Bayer CFA, the Wide sensor may have a
`Bayer CFA in the overlap area. In such embodiments, color
`resolution improvement depends on using color information
`from the Tele sensor in the fused output image.
`FIG. 1A shows schematically a block diagram illustrating
`a dual-aperture zoom imaging (“DAZI”) system 100 dis-
`closed herein. System 100 includes a dual-aperture camera
`102 with a Wide subset 104 and a Tele subset 106 (each
`subset having a respective sensor), and a processor 108 that
`fuses two images, a Wide image obtained with the Wide
`subset and a Tele image obtained with the Tele subset, into
`a single fused output image according to a user-defined
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`“applied” ZF input or request. The ZF is input to processor
`108. The Wide sensor may include a non-standard CFA in an
`overlap area illustrated by 110 in FIG. 1B. Overlap area 110
`is surrounded by a non-overlap area 112 with a standard
`CFA (for example a Bayer pattern). FIG. 1B also shows an
`example of an image captured by both Wide and Tele
`sensors. Note that “overlap” and “non-overlap” areas refer to
`parts of the Wide image as well as to the CFA arrangements
`of the Wide sensor. The overlap area may cover different
`portions of a Wide sensor, for example half the sensor area,
`a third of the sensor area, a quarter of the sensor area, etc.
`A number of such Wide sensor CFA arrangements are
`described in more detail with reference to FIGS. 2-9. The
`
`non-standard CFA pattern increases the color resolution of
`the DAZI system.
`The Tele sensor may be Clear (providing a Tele Clear
`image scaled relative to the Wide image) or may include a
`standard (Bayer or non-Bayer) CFA. It in the latter case, it
`is desirable to define primary and auxiliary sensors based on
`the applied ZF. If the ZF is such that the output FOV is larger
`than the Tele FOV, the primary sensor is the Wide sensor and
`the auxiliary sensor is the Tele sensor. If the ZF is such that
`the output FOV is equal to, or smaller than the Tele FOV, the
`primary sensor is the Tele sensor and the auxiliary sensor is
`the Wide sensor. The point of view defined by the output
`image is that of the primary sensor.
`FIG. 2 shows schematically an embodiment of a Wide
`sensor 200 that may be implemented in a DAZI system such
`as system 100. Sensor 200 has a non-overlap area 202 with
`a Bayer CFA and an overlap area 204 covered by a non-
`standard CFA with a repetition of a 4x4 micro-cell in which
`the color filter order is BBRR-RBBR-RRBB-BRRB. In this
`
`figure, as well as in FIGS. 3-9, “Width 1” and “Height 1”
`refer to the full Wide sensor dimension. “Width 2” and
`
`“Height 2” refer to the dimensions of the Wide sensor
`overlap area. Note that in FIG. 2 (as in following FIGS. 3-5
`and 7, 8) the empty row and column to the left and top of the
`overlap area are for clarity purposes only, and that the sensor
`pixels follow there the pattern of the non-overlap area (as
`shown in FIG. 6). In overlap area 204, R and B are sampled
`at 1/20'5 Nyquist frequency in the diagonal (left to right)
`direction with 2 pixel
`intervals instead of at 1/2 Nyquist
`frequency in a standard Bayer pattern.
`FIG. 3 shows schematically an embodiment of a Wide
`sensor 300 that may be implemented in a DAZI system such
`as system 100. Sensor 300 has a non-overlap area 302 with
`a Bayer CFA and an overlap area 304 covered by a non-
`standard CFA with a repetition of a 2x2 micro-cell in which
`the color filter order is BR-RB. In the overlap area, R and B
`are sampled at 1/20'5 Nyquist frequency in both diagonal
`directions.
`
`FIG. 4 shows schematically an embodiment of a Wide
`sensor 400 that may be implemented in a DAZI system such
`as system 100. Sensor 400 has a non-overlap area 402 with
`a Bayer CFA and an overlap area 404 covered by a non-
`standard CFA with a repetition of a 2x2 micro-cell in which
`the color filter order is YC-CY, where Y:Yellow:Green+
`Red, C:Cyan:Green+Blue. As a result, in the overlap area,
`R and B are sampled at 1/20'5 Nyquist frequency in a diagonal
`direction. The non-standard CFA includes green information
`for registration purposes. This allows for example registra-
`tion between the two images where the object is green, since
`there is green information in both sensor images.
`FIG. 5 shows schematically an embodiment of a Wide
`sensor 500 that may be implemented in a DAZI system such
`as system 100. Sensor 500 has a non-overlap area 502 with
`a Bayer CFA and an overlap area 504 covered by a non-
`
`APPL-1001 /Page 12 of 16
`
`APPL-1001 / Page 12 of 16
`
`

`

`US 9,538,152 B2
`
`7
`standard CFA with a repetition of a 6x6 micro-cell in which
`the color filter order is RBBRRB-RWRBWB-BBRBRR-
`
`RRBRBB-BWBRWR-BRRBBR, where “W” represents
`White or Clear pixels. In the overlap area, R and B are
`sampled at a higher frequency than in a standard CFA. For
`example, in a Bayer pixel order, the Red average sampling
`rate (“RS”) is 0.25 (sampled once for every 4 pixels). In the
`overlap area pattern, R5 is 0.44.
`FIG. 6 shows schematically an embodiment of a Wide
`sensor 600 that may be implemented in a DAZI system such
`as system 100. Sensor 600 has a non-overlap area 602 with
`a Bayer CFA and an overlap area 604 covered by a non-
`standard CFA with a repetition of a 6x6 micro-cell in which
`the color filter order is BBGRRG-RGRBGB-GBRGRB-
`
`RRGBBG-BGBRGR-GRBGBR. In the overlap area, R and
`B are sampled at a higher frequency than in a standard CFA.
`For example, in the overlap area pattern, R5 is 0.33 vs. 0.25
`in a Bayer pixel order.
`FIG. 7 shows schematically an embodiment of a Wide
`sensor 700 that may be implemented in a DAZI system such
`as system 100. Sensor 700 has a non-overlap area 702 with
`a Bayer CFA and an overlap area 704 covered by a non-
`standard CFA with a repetition of a 3x3 micro-cell in which
`the color filter order is GBR-RGB-BRG. In the overlap area,
`R and B are sampled at a higher frequency than in a standard
`CFA. For example, in the overlap area pattern, R5 is 0.33 vs.
`0.25 in a Bayer pixel order.
`FIG. 8 shows schematically an embodiment of a Wide
`sensor 800 that may be implemented in a DAZI system such
`as system 100. Sensor 800 has a non-overlap area 802 with
`a Bayer CFA and an overlap area 804 covered by a non-
`standard CFA with a repetition of a 6x6 micro-cell in which
`the color filter order is RBBRRB-RGRBGB-BBRBRR-
`
`RRBRBB-BGBRGR-BRRBBR. In the overlap area, R and
`B are sampled at a higher frequency than in a standard CFA.
`For example, in the overlap area pattern, R5 is 0.44 vs. 0.25
`in a Bayer pixel order.
`FIG. 9 shows schematically an embodiment of a Wide
`sensor 900 that may be implemented in a DAZI system such
`as system 100. Sensor 900 has a non-overlap area 902 with
`a Bayer CFA and an overlap area 904 covered by a non-
`standard CFA with a repetition of a 6x6 micro-cell in which
`the color filter order is RBRBRB-BGBRGR-RBRBRB-
`
`BRBRBR-RGRBGB-BRBRBR. In the overlap area, R and
`B are sampled at a higher frequency than in a standard CFA.
`For example, in the overlap area pattern, R5 is 0.44 vs. 0.25
`in a Bayer pixel order.
`Processing Flow
`In use, an image is acquired with imaging system 100 and
`is processed according to steps illustrated in a flowchart
`shown in FIG. 10. In step 1000, demosaicing is performed
`on the Wide overlap area pixels (which refer to the Tele
`image FOV) according to the specific CFA pattern. If the
`CFA in the Wide overlap area is a standard CFA, a standard
`demosaicing process may be applied to it. If the CFA in the
`Wide overlap area is non-standard CFA, the overlap and
`non-overlap subsets of pixels may need different demosaic-
`ing processes. That is, the Wide overlap area may need a
`non-standard demosaicing process and the Wide non-over-
`lap area may need a standard demosaicing process. Exem-
`plary and non-limiting non-standard demosaicing interpola-
`tions for the overlap area of each of the Wide sensors shown
`in FIGS. 2-9 are given in detail below. The aim of the
`demosaicing is to reconstruct missing colors in each pixel.
`Demosaicing is applied also to the Tele sensor pixels if the
`Tele sensor is not a Clear only sensor. This will result in a
`Wide subset color image where the colors (in the overlap
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`area) hold higher resolution than those of a standard CFA
`pattern. In step 1002, the Tele image is registered (mapped)
`into the Wide image. The mapping includes finding corre-
`spondences between pixels in the two images. In step 1002,
`actual registration is performed on luminance Tele and Wide
`images (respectively LumaTeZe and LumaMde) calculated
`from the pixel information of the Tele and Wide cameras.
`These luminance images are estimates for the scene lumi-
`nance as captured by each camera and do not include any
`color information. If the Wide or Tele sensors have CFAs,
`the calculation of the luminance images is performed on the
`respective demosaiced images. The calculation of the Wide
`luminance image varies according to the type of non-
`standard CFA used in the Wide overlap area. If the CFA
`permits calculation of a full RGB demosaiced image, the
`luminance image calculation is straightforward. If the CFA
`is such that it does not permit calculation of a full RGB
`demosaiced image, the luminance image is estimated from
`the available color channels. If the Tele sensor is a Clear
`
`sensor, the Tele luminance image is just the pixel informa-
`tion. Performing the registration on luminance images has
`the advantage of enabling registration between images cap-
`tured by sensors with different CFAs or between images
`captured by a standard CFA or non-standard CFA sensor and
`a standard CFA or Clear sensor and avoiding color artifacts
`that may arise from erroneous registration.
`In step 1004, the data from the Wide and Tele images is
`processed together with the registration information from
`step 1002 to form a high quality output zoom image. In cases
`where the Tele sensor is a Clear only sensor,
`the high
`resolution luminance component
`is taken from the Tele
`sensor and color resolution is taken from the Wide sensor. In
`cases where the Tele sensor includes a CFA, both color and
`luminance data are taken from the Tele subset to form the
`
`high quality zoom image. In addition, color and luminance
`data is taken from the Wide subset.
`
`Exemplary Process for Fusing a Zoom Image
`1. Special Demosaicing
`In this step, the Wide image is interpolated to reconstruct
`the missing pixel values. Standard demosaicing is applied in
`the non-overlap area. If the overlap area includes a standard
`CFA, standard demosaicing is applied there as well. If the
`overlap area includes a non-standard CFA, a special demo-
`saicing algorithm is applied, depending on the CFA pattern
`used. In addition, in case the Tele sensor has a CFA, standard
`demosaicing is applied to reconstruct
`the missing pixel
`values in each pixel location and to generate a full RGB
`color image.
`2. Registration Preparation
`Tele image: a luminance image LumaTeZe is calculated
`from the Tele sensor pixels. If the Tele subset has a
`Clear sensor, Lumame is simply the sensor pixels data.
`If the Tele subset has a standard CFA, LumaTeZe is
`calculated from the demosaiced Tele image.
`Wide image: as a first step, in case the Wide overlap CFA
`permits estimating the luminance component of the
`image, the luminance component is calculated from the
`demosaiced Wide image, Lumande. If the CFA is one
`of those depicted in FIGS. 4-9, a luminance image is
`calculated first. If the CFA is one of the CFAs depicted
`in FIG. 2 or FIG. 3, a luminance image is not calcu-
`lated. Instead, the following registration step is per-
`formed between a weighted average of the demosaiced
`channels of the Wide image and LumaTeZe. For conve-
`nience, this weighted average image is also denoted
`Lumande. For example, if the Wide sensor CFA in the
`overlap region is as shown in FIG. 2, the demosaiced
`
`APPL-1001 /Page 13 of 16
`
`APPL-1001 / Page 13 of 16
`
`

`

`US 9,538,152 B2
`
`9
`channels RWME and BWlde are averaged to create
`Lumande
`according
`to
`Lumande:(f1 *Rm019+
`fl*BMde)/(f1+fl), where f1 may be f1:1 and f2 may be
`f2:1.
`
`Low-pass filtering is applied on the Tele luminance image
`in order to match its spa