(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2016/0050374. A1
`Shabtay et al.
`(43) Pub. Date:
`Feb. 18, 2016
`
`US 20160050374A1
`
`(54) DUAL APERTURE ZOOM DIGITAL CAMERA
`(71) Applicant: Corephotonics Ltd., Tel-Aviv (IL)
`(72) Inventors: Gal Shabtay, Tel-Aviv (IL); Ephraim
`Goldenberg, Ashdod (IL); Oded
`Gigushinski, Tel-Aviv (IL); Noy Cohen,
`Tel-Aviv (IL)
`(21) Appl. No.: 14/880,251
`(22) Filed:
`Oct. 11, 2015
`Related U.S. Application Data
`(63) Continuation of application No. 14/365,711, filed on
`Jun. 16, 2014, now Pat. No. 9,185.291.
`s
`s
`s u. Y- a-s
`Publication Classification
`
`(51) Int. Cl.
`H04N 5/232
`GO2B I3/00
`H04N 5/225
`
`(2006.01)
`(2006.01)
`(2006.01)
`
`(52) U.S. Cl.
`CPC ......... H04N 5/23296 (2013.01); H04N5/2258
`(2013.01); H04N 5/23245 (2013.01); G02B
`13/009 (2013.01)
`
`(57)
`
`ABSTRACT
`
`A dual-aperture Zoom digital camera comprising a Wide
`imaging section and a Tele imaging section, each section
`including a lens with a respective FOV, image sensor and
`image signal processor, and a camera controller coupled to
`the Wide and Tele imaging sections and operative to process
`Wide and Tele sensor data to obtain a fused image of the
`object in a capture or stills mode, and to process without
`fusion Wide and Tele sensor data to obtain, in a video mode,
`images with an output resolution and a smooth transition
`when switching between a lower Zoom factor (ZF) value and
`a higher Zoom factor value or vice versa, wherein at the lower
`ZF the output resolution is determined by the Wide sensor and
`wherein at the higher ZF value the output resolution is deter
`mined by the Tele sensor.
`
`Wide lens 102
`
`Tele lens 108
`
`Wide SensOr 104
`
`Tele SensOr 110
`
`Wide ISP 106
`
`Tele ISP 112
`
`
`
`Camera Controller 114
`
`Sensor control 116
`
`User control 118
`
`Video processing
`126
`
`Capture processing
`128
`
`Ex. 3001
`IPR2020-00488
`
`

`

`Patent Application Publication
`
`Feb. 18, 2016 Sheet 1 of 9
`
`US 2016/0050374, A1
`
`100
`
`
`
`FIG. 1
`
`

`

`Patent Application Publication
`
`Feb. 18, 2016 Sheet 2 of 9
`
`US 2016/0050374, A1
`
`
`
`FIG. 2
`
`2O2
`
`204
`
`

`

`Patent Application Publication
`
`Feb. 18, 2016 Sheet 3 of 9
`
`US 2016/0050374, A1
`
`Wide
`Rolling
`i- Shutter"
`1/frame as
`Krvicare
`
`:
`
`Time
`
`O
`
`Blank
`
`Wide sensor
`Line Number
`
`
`
`
`
`
`
`Overlapping
`FOW
`
`
`
`
`
`xwRw :
`
`o& T e
`
`o1/frame rateok
`Tele
`Rolling
`Shutter
`Time :
`
`Tele
`Vertical
`Blank
`
`Tele
`8rExposure
`Time
`
`Time
`
`Tele sensor
`Line Number
`
`
`
`FIG. 3
`
`

`

`Patent Application Publication
`
`Feb. 18, 2016 Sheet 4 of 9
`
`US 2016/0050374, A1
`
`Aa
`
`Horizontal Blank
`
`wiGioia.Sö.
`
`
`
`as a
`
`O
`
`
`
`citizeisei
`
`FIG. 4
`
`

`

`Patent Application Publication
`
`Feb. 18, 2016 Sheet 5 of 9
`
`US 2016/0050374, A1
`
`
`
`ISP: Perform image signal
`processing on data received form
`each sensor to obtain processed
`Wide and Tele images
`SO2
`
`Rectification: Align processed
`Wide and Tele images to be on an
`epipolar line to obtain aligned
`(rectified) images
`504
`
`Registration: Map the aligned
`Wide and Tele images to obtain a
`registration map
`506
`
`Resampling: Process registration
`map and processed Tele image to
`obtain a re-sampled Tele image
`508
`
`Decision: Use re-sampled Tele
`image and Wide image to detect
`errors in the registration and to
`provide a decision output
`510
`
`Fusion: Fuse the decision Output,
`re-sampled Tele image and Wide
`image into a fused ZOOm image
`S12
`
`FIG. 5
`
`

`

`Patent Application Publication
`
`Feb. 18, 2016 Sheet 6 of 9
`
`US 2016/0050374, A1
`
`Choose sensor(s) to be operational
`602
`
`
`
`Optionally, calculate color balance if
`two (Wide and Tele) images are
`provided by the two sensors.
`604
`
`Optionally, apply calculated color
`balance in one of the images
`606
`
`registration
`perform
`Optionally,
`between the Wide and Tele images to
`Output a transformation coefficient
`608
`
`Set an AF position using the
`transformation coefficient
`610
`
`Process an output of any of steps
`602-608 to obtain a processed image
`612
`
`Resample the processed image
`according to the transformation
`coefficient, requested ZF and output
`Video resolution
`612
`
`FIG. 6
`
`

`

`Patent Application Publication
`
`Feb. 18, 2016 Sheet 7 of 9
`
`US 2016/0050374, A1
`
`Effective
`Resolution
`
`
`
`FIG 7
`
`AZOOmdown
`<-G>
`AZOOmup
`
`User ZOOm factor
`
`

`

`Patent Application Publication
`
`Feb. 18, 2016 Sheet 8 of 9
`
`US 2016/0050374, A1
`
`806 808 810
`802 804 N
`(77
`A /
`
`
`
`L
`
`- 6.12
`f
`F# - 3.4
`TTL - 4.7mm
`Semi field: 20dcg.
`
`FIG. 8
`
`

`

`Patent Application Publication
`
`Feb. 18, 2016 Sheet 9 of 9
`
`US 2016/0050374, A1
`
`so 908 t 910
`
`N.
`
`3i E, as ot
`
`
`
`- 7.14
`f
`I # - 3.5
`TTL - 5.8mm
`Semi field: 22.7 deg.
`
`FIG. 9
`
`

`

`US 2016/0050374, A1
`
`Feb. 18, 2016
`
`DUAL APERTURE ZOOMIDIGITAL CAMERA
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`0001. This application is a Continuation application of
`U.S. patent application Ser. No. 14/365,711 filed Jun. 16,
`2014 (now allowed), and is related to and claims priority from
`U.S. Provisional Patent Application No. 61/834,486 having
`the same title and filed Jun. 13, 2013, which is incorporated
`herein by reference in its entirety.
`
`FIELD
`0002 Embodiments disclosed herein relate in general to
`digital cameras and in particular to thin Zoom digital cameras
`with both still image and video capabilities
`
`BACKGROUND
`0003 Digital camera modules are currently being incor
`porated into a variety of host devices. Such host devices
`include cellular telephones, personal data assistants (PDAs),
`computers, and so forth. Consumer demand for digital cam
`era modules in host devices continues to grow.
`0004 Host device manufacturers prefer digital camera
`modules to be Small, so that they can be incorporated into the
`host device without increasing its overall size. Further, there
`is an increasing demand for cameras in host devices to have
`higher-performance characteristics. One such characteristic
`that many higher-performance cameras (e.g., Standalone digi
`tal still cameras) have is the ability to vary the focal length of
`the camera to increase and decrease the magnification of the
`image, typically accomplished with a Zoom lens, now known
`as optical Zooming Optical Zooming is typically accom
`plished by mechanically moving lens elements relative to
`each other. Such Zoom lens are typically more expensive,
`larger, and less reliable than fixed focal length lenses. An
`alternative approach for approximating the Zoom effect is
`achieved with what is known as digital Zooming With digital
`Zooming, instead of varying the focal length of the lens, a
`processor in the camera crops the image and interpolates
`between the pixels of the captured image to create a “magni
`fied' but lower-resolution image.
`0005 Attempts to use two different lenses to approximate
`the effect of a Zoom lens are known, see e.g. US Patent
`Publications No. 2008/0030592 and 2010/0277619. In US
`2008/0030592, the two sensors are operated simultaneously
`to capture an image. The respective lenses have different focal
`lengths, so even though each lens/sensor combination is
`aligned to look in the same direction, each will capture an
`image of the same subject but with two different fields of view
`(FOVs). One sensor is commonly called “Wide' and the other
`“Tele'. Each sensor provides a separate image, referred to
`respectively as “Wide” (or “W) and “Tele” (or “T”) images.
`The W-image reflects a wider FOV and has lower resolution
`than the T-image. The images are then Stitched (fused)
`together to form a composite image. In the composite image,
`the central portion is formed by the relatively higher-resolu
`tion image taken by the lens/sensor combination with the
`longer focal length, and the peripheral portion is formed by a
`peripheral portion of the relatively lower-resolution image
`taken by the lens/sensor combination with the shorter focal
`length. The user selects a desired amount of Zoom and the
`composite image is used to interpolate values from the chosen
`amount of Zoom to provide a respective Zoom image. The
`
`solution offered by US 2008/0030592 requires, in video
`mode, very large processing resources in addition to high
`framerate requirements, high power consumption (since both
`cameras are fully operational). It does not provide reference
`to disparity range and does not provide reference of how to
`avoid registration errors.
`0006 US 2010/0277619 teaches a camera with a pair of
`lens/sensor combinations, the two lenses having different
`focal lengths, so that the image from one of the combinations
`has a field of view approximately 2-3 times greater than the
`image from the other combination. As a user of the camera
`requests a given amount of Zoom, the Zoomed image is pro
`vided from the lens/sensor combination having the field of
`view that is next larger than the requested field of view. Thus,
`if the requested field of view is less than the smaller field of
`view combination, the Zoomed image will be created from the
`image captured by that combination, using cropping and
`interpolation if necessary. Similarly, if the requested field of
`view is greater than the smaller field of view combination, the
`Zoomed image will be created from the image captured by the
`other combination, using cropping and interpolation if nec
`essary. The solution offered by US 2010/0277619 leads to
`parallax artifacts when moving to the Tele camera in video
`mode. It does not provide reference to disparity range and its
`heavy computational requirements and does not provide ref
`erence of how to avoid registration errors.
`0007. In both US 2008/0030592 and US 2010/0277619,
`different focal length systems cause Tele and Wide matching
`FOVs to be exposed at different times using CMOS sensors,
`which degrades the overall image quality. Different optical F
`numbers (“Fi”) cause image intensity differences. Working
`with such a dual sensor system requires double bandwidth
`Support, i.e. additional wires from the sensors to the following
`HW component. Neither US 2008/0030592 nor US 2010/
`0277619 deal with false registration.
`0008. Therefore there is a need for, and it would be advan
`tageous to have thin digital cameras with optical Zoom oper
`ating in both video and still mode that do not suffer from the
`commonly encountered problems and disadvantages, some
`of which are listed above.
`
`SUMMARY
`
`0009 Embodiments disclosed herein teach the use of
`dual-aperture (also referred to as dual-lens or two-sensor)
`optical Zoom digital cameras. The cameras include two cam
`era “subsets’, a Wide subset and a Tele subset, each subset
`including a lens, an image sensor and an image signal pro
`cessor (ISP). The Tele subset is the higher Zoom subset and
`the Wide subset is the lower Zoom subset. In some embodi
`ments, the lenses are thin lenses with short optical paths of
`less than about 9 mm The thickness/effective focal length
`(EFL) ratio of the Tele lens is smaller than 0.9. Such a camera
`includes two sensors or a single sensor divided into at least
`two areas. Hereinafter, the description refers to “two sen
`sors, with the understanding that they may be sections of a
`single physical sensor (imager chip). The camera may be
`operated in both still and video modes. In still mode, Zoom is
`achieved by fusing W and T images, with the resulting
`“fused image including always information from both
`images. In video mode, full and Smooth optical Zoom is
`achieved by switching between the W and T images. To avoid
`discontinuities in video mode, the Switching includes apply
`ing additional processing blocks.
`
`

`

`US 2016/0050374, A1
`
`Feb. 18, 2016
`
`0010. In order to reach optical Zoom capabilities, a differ
`ent magnification image of the same scene is captured by each
`camera subset, resulting in FOV overlap between the two
`Subsets processing is applied on the two images grabbed by
`the multi-aperture imaging system to fuse and output one
`image processed according to a user Zoom factor request. As
`part of the fusion procedure, up-sampling may be applied on
`one or both of the images to Scale it to the image grabbed by
`the Tele subset or to a scale defined by the user. The fusion or
`up-sampling may be applied to only some of the pixels of a
`sensor. Down-sampling can be performed as well if the output
`resolution is Smaller than the sensor resolution.
`0011. The cameras and associated methods disclosed
`herein address and correct many of the problems and disad
`Vantages of known dual aperture optical Zoom digital cam
`eras. They provide an overall Zoom solution that refers to all
`aspects: optics, algorithmic processing and system hardware
`(HW). The proposed solution distinguishes between video
`and capture mode in the processing flow, and specifies the
`optical requirements and HW requirements. In addition, it
`provides an innovative optical design which enables low TTL
`focal length ratio using a specific lens curvature order.
`0012. In addition to enabling optical Zoom, a dual-aper
`ture Zoom system described herein can also enable a unique
`photography feature which is typically not available in com
`pact camera modules currently available in cell-phones. This
`feature is very shallow depth of focus (DOF), particularly
`suited for portrait photos. The DOF of a lens decreases as the
`focal length increases.
`0013 When portrait photos are captured with large cam
`eras (such as DSLRs), usually a 50-80 mm lens is used,
`coupled with a wide-open aperture setting to create a shallow
`DOF effect. Due to the large focal length, objects that are in
`front or behind the plane of focus appear very blurry, and a
`nice foreground-to-background contrast is achieved. How
`ever, it is difficult to create such a blur using a compact camera
`with a relatively short focal length and Small aperture size,
`Such as a cell-phone camera. In some embodiments, a dual
`aperture Zoom system disclosed herein can be used to capture
`a shallow-DOF photo by taking advantage of the longer focal
`length of the Tele lens. The reduced DOF effect provided by
`this relatively long focal length can be further enhanced in the
`final image by fusing data from an image captured with the
`Wide-lens at the same time. Depending on the object dis
`tance, while the Tele lens is focused on the subject of the
`photo, the Wide lens can be focused to a very near distance, so
`that objects behind the subject appear very blurry. Once the
`two images are captured, information from the out-of-focus
`blurred background in the Wide image is fused with the
`original Tele image background information, to result in a
`blurrier background and even shallower DOF.
`0014. In an embodiment there is provided a dual-aperture
`Zoom digital camera comprising a Wide imaging section
`operative to provide a Wide image of an object, the Wide
`imaging section including a lens with a wide FOV, a Wide
`sensor and a Wide image signal processor
`00.15
`(ISP), a Tele imaging section operative to provide a
`Tele image of the object, the Tele imaging section including a
`lens with a narrow FOV, a Tele sensor and a Tele ISP, and a
`camera controller coupled to the Wide and Tele imaging
`sections and operative to process Wide and Tele sensor data to
`obtain a fused image of the object in a capture or stills mode,
`and to process without fusion Wide and Tele sensor data to
`obtain, in a video mode, images with an output resolution and
`
`a smooth transition when Switching between a lower Zoom
`factor (ZF) value and a higher ZF value or vice versa, wherein
`at the lower ZF the output resolution is determined by the
`Wide sensor and wherein at the higher ZF value the output
`resolution is determined by the Tele sensor, whereby the
`processing provides a continuous Zoom.
`0016. In an embodiment there is provided a method for
`providing continuous optical Zoom in a digital camera that
`includes a Wide imaging section including a lens with a wide
`FOV, a Wide sensor and a Wide ISP and a Tele imaging
`section including a lens with a narrow FOV, a Tele sensor and
`a Tele ISP, the method comprising the steps of imaging an
`object to obtain
`0017 Wide sensor data and Tele sensor data, processing
`with fusion the Wide and Tele sensor data to obtain a fused
`image of the object in a capture or stills mode, and processing
`without fusion the Wide and Tele sensor data to obtain, in a
`Video mode, images with an output resolution and a Smooth
`transition when switching between a lower Zoom factor (ZF)
`value and a higher ZF value or vice versa, wherein at the lower
`ZF the output resolution is determined by the Wide sensor and
`wherein at the higher ZF value the output resolution is deter
`mined by the Tele sensor.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0018 Non-limiting examples of embodiments disclosed
`herein are described below with reference to figures attached
`hereto that are listed following this paragraph. Identical struc
`tures, elements or parts that appear in more than one figure are
`generally labeled with a same numeral in all the figures in
`which they appear. The drawings and descriptions are meant
`to illuminate and clarify embodiments disclosed herein, and
`should not be considered limiting in any way.
`0019 FIG. 1 shows schematically a block diagram illus
`trating a dual-aperture Zoom imaging system disclosed
`herein;
`0020 FIG. 2 shows an example of Wide sensor, Tele sen
`sor and their respective FOVs;
`0021
`FIG. 3 shows a schematically embodiment of
`CMOS sensor image grabbing vs. time;
`0022 FIG. 4 shows schematically a sensor time configu
`ration which enables sharing one sensor interface using dual
`sensor Zoom system;
`0023 FIG. 5 shows an embodiment of a method disclosed
`herein for acquiring a Zoom image in capture mode;
`0024 FIG. 6 shows an embodiment of a method disclosed
`herein for acquiring a Zoom image in video/preview mode;
`0025 FIG. 7 shows a graph illustrating an effective reso
`lution Zoom factor,
`0026 FIG. 8 shows one embodiment of a lens block in a
`thin camera disclosed herein;
`0027 FIG.9 shows another embodiment of a lens block in
`a thin camera disclosed herein.
`
`DETAILED DESCRIPTION
`0028 FIG. 1 shows schematically a block diagram illus
`trating an embodiment of a dual-aperture Zoom imaging sys
`tem (also referred to simply as “camera') disclosed herein
`and numbered 100. Camera 100 comprises a Wide imaging
`section (“subset') that includes a Wide lens block 102, a Wide
`image sensor 104 and a Wide image processor 106. Camera
`100 further comprises a Tele imaging section (“subset) that
`includes a Tele lens block 108, a Tele image sensor 110 and a
`
`

`

`US 2016/0050374, A1
`
`Feb. 18, 2016
`
`Tele image processor 112. The image sensors may be physi
`cally separate or may be part of a single larger image sensor.
`The Wide sensor pixel size can be equal to or different from
`the Tele sensor pixel size. Camera 100 further comprises a
`camera fusion processing core (also referred to as “control
`ler) 114 that includes a sensor control module 116, a user
`control module 118, a video processing module 126 and a
`capture processing module 128, all operationally coupled to
`sensor control block 110. User control module 118 comprises
`an operational mode function 120, a region of interest (ROI)
`function 122 and a Zoom factor (ZF) function 124.
`0029 Sensor control module 116 is connected to the two
`sub-cameras and to the user control module 118 and used to
`choose, according to the Zoom factor, which of the sensors is
`operational and control the exposure mechanism and the sen
`Sor readout. Since Zoom is achieved by sensor oversampling,
`for most Zoom factors, only one sensor is operational in video
`mode. This is also true for the Auto Focus (AF) mechanism.
`Mode choice function 120 is used for choosing capture/video
`modes. The capture mode may include a “burst mode”. ROI
`function 122 is used to choose region of interest. Zoom factor
`function 124 is used to choose a Zoom factor. Video process
`ing module 126 is connected to the mode choice function 120
`and used for high frame rate video processing. Capture pro
`cessing module 128 is connected to the mode choice function
`120 and used for high image quality still mode images.
`0030 The video processing module is applied when the
`user desires to shoot in video mode. The capture processing
`module is applied when the user wishes to shoot still pictures.
`Following is a detailed description and examples of different
`methods of use of camera 100.
`Design for continuous Zoom in Video Mode
`0031. In order to reach high quality optical continuous
`Zooming in video mode while reaching real optical Zoom,
`optical system design is taken into account as follows:
`(1)
`Tan(FOVutte). Tan (FOVIEE)-PL wide PL,
`“Tan” refers to "tangent', while FOV, and FOV, refer
`respectively to Wide and Tele lens fields of view (in degrees).
`The FOV is measured from the center axis to the corner of the
`sensor. PL, and PL
`referrespectively to the “in-line”
`(i.e. in a line) number of sensor pixels and in-line number of
`output video format pixels. For example, in order to get full
`optical Zoom continuous experience with a 12 Mp sensor
`(sensor dimensions: 4000x3000) and a required 1080p (di
`mension: 1920x1080) video format, the FOV ratio should be
`4000/1920–2,083. Moreover, if the Wide lens FOV is given as
`FOV, 37.5°, the required Tele lens FOV is 20.2°. The
`Zoom Switching point is set according to the ratio between
`sensor pixels in-line and the number of pixels in-line in the
`video format and defined as:
`(2)
`Zovitch PL WIDE/FL video
`Maximum optical Zoom is reached according to the following
`formula:
`(3)
`Z-Tan (FOViride). Tan (FOVIEE)*PL title/PL,
`For example: for the configuration defined above and assum
`ing PL-4000 and PL-1920, Zmax=4.35.
`0032. In an embodiment, the sensor control module has a
`setting that depends on the Wide and Tele FOVs and on a
`sensor oversampling ratio, the setting used in the configura
`tion of each sensor. In an embodiment, the Wide and Tele
`FOVs and the oversampling ratio satisfy the condition
`0.8PL/PL-Tan (FOV)/Tan (FOV)<1.
`ideo
`
`2PL/PL. “Oversampling ratio” is defined as the
`ratio between the number of pixels in the sensor vs the num
`ber of pixels in the output image. For example, when using a
`4000x3000 sensor and when outputting a 1920x1080 image,
`the oversampling ratio is 4000/1920–2.0833
`
`Still Camera Mode Operation/Function
`0033. In still camera mode, the obtained image is fused
`from information obtained by both sub-cameras at all Zoom
`levels, see FIG. 2, which shows a first (“Wide') sensor 202
`and a second (“Tele) sensor 204 and their respective FOVs.
`Exemplarily, as shown, the Tele sensor FOV is half the Wide
`sensor FOV. The still camera mode processing includes two
`stages: (1) setting HW settings and configuration, where a
`first objective is to control the sensors in such a way that
`matching FOVs in both images (Tele and Wide) are scanned
`at the same time. A second objective is to control the relative
`exposures according to the lens properties. A third objective is
`to minimize the required bandwidth from both sensors for the
`ISPs; and (2) image processing that fuses the Wide and the
`Tele images to achieve optical Zoom, improves SNR and
`provides wide dynamic range.
`0034 FIG. 3 shows image line numbers vs. time for an
`image section captured by CMOS sensors. A fused image is
`obtained by line (row) scans of each image. To prevent match
`ing FOVs in both sensors to be scanned at different time, a
`special configuration is applied on both image sensors while
`keeping the same frame rate. The difference in FOV between
`the sensors determines the relationship between the rolling
`shutter time and the vertical blanking time for each sensor.
`The Scanning is synchronized such that the same points of the
`object in each view are obtained simultaneously.
`0035) Specifically with reference to FIG.3 and according
`to an embodiment of a method disclosed herein, the fused
`image acquisition process includes setting the Tele sensor
`vertical blanking timeVB to equal the Wide sensor vertical
`blanking time VB plus half the Wide sensor rolling shutter
`time RST
`setting the Tele and Wide sensor exposure
`times ET and ET to be equal or different, setting the
`Tele sensor rolling shutter time RST to be half the Wide
`sensor rolling shutter time RST and setting the frame rates
`of the two sensors to be equal. This procedure results in
`identical image pixels in the Tele and Wide sensor images
`being exposed at the same time.
`0036. The exposure times applied to the two sensors could
`be different, for example, in order to reach same image inten
`sity using different F if and different pixel size for the Tele and
`Wide systems. In this case, the relative exposure time should
`be configured according to the formula below:
`ETri-ETui (FHz/FH, i.)*(Pixel sizer/Pixel
`(4)
`sizewia)
`Other exposure time ratios may be applied to achieve wide
`dynamic range and improved
`0037 SNR. Fusing two images with different intensities
`will result in wide dynamic range image.
`0038. In more detail with reference to FIG. 3, in the first
`stage, after the user chooses a required Zoom factor ZF, the
`sensor control module configures each sensor as follows:
`0039 1) Cropping index Wide sensor:
`
`

`

`US 2016/0050374, A1
`
`Feb. 18, 2016
`
`where PC is the number of pixels in a column, and Y is the row
`number
`0040 2) Cropping index Tele sensor:
`If ZF-Tan (FOV)/Tan (FOV), then
`Y=/2PC(1-(1/ZF)-Tan(FOV), Tan (FOV
`wide.)))
`
`Y/2 PC(1+(1/ZF)-TancFOV), Tan (FOV
`wide))
`If ZF<Tan (FOV)/Tan (FOV), then
`
`Y1-PCTele
`This will result in start exposing the Tele sensor with a delay
`of:
`
`(5)
`(1-ZF/((Tan (FOV)/Tan (FOV))): 1/(2-PFS)
`where FPS is the sensors frame per second configuration. In
`cases where ZF-Tan (FOV)/Tan (FOV), no delay will
`be introduced between Tele and Wide exposure starting point.
`For example, for a case where Tan (FOV)/Tan (FOV)
`=2 and ZF=1, the Tele image first pixel is exposed 4 (1/PFS)
`sec after the Wide image first pixel was exposed.
`0041 After applying the cropping according to the
`required Zoom factor, the sensor rolling shutter time and the
`vertical blank should be configured in order to satisfy the
`equation to keep the same frame rate
`(6)
`VB+RST VB1+RST
`0042 FIG. 3 exemplifies Eq. (6), One way to satisfy Eq.
`(6) is to increase the RST-Controlling the RST may be
`done by changing the horizontal blanking (HB) of the Wide
`sensor. This will cause a delay between the data coming out
`from each row of the Wide sensor.
`0043 Generally, working with a dual-sensor system
`requires multiplying the bandwidth to the following block,
`for example the ISP. For example, using 12 Mp working at 30
`fps, 10 bit per pixel requires working at 3.6 Gbit/sec. In this
`example, Supporting this bandwidth requires 4 lanes from
`each sensor to the respective following ISP in the processing
`chain. Therefore, working with two sensors requires double
`bandwidth (7.2 Gbit/sec) and 8 lanes connected to the respec
`tive following blocks. The bandwidth can be reduced by
`configuring and synchronizing the two sensors. Conse
`quently, the number of lanes can behalf that of a conventional
`configuration (3.6 Gbit/sec).
`0044 FIG. 4 shows schematically a sensor time configu
`ration that enables sharing one sensor interface using a dual
`sensor Zoom system. For simplicity, assuming the Tele sensor
`image is magnified by a factor of 2 compared with the Wide
`sensor image, the Wide sensor horizontal blanking time
`HB is set to twice the Wide sensor line readout time. This
`causes a delay between output Wide lines. This delay time
`matches exactly the time needed to output two lines from the
`Tele sensor. After outputting two lines from the Tele sensor,
`the Tele sensor horizontal blanking time HB is set to be
`one Wideline readout time, so, while the Wide sensor outputs
`a row from the sensor, no data is being output from the Tele
`sensor. For this example, every 3" line in the Tele sensor is
`delayed by an additional HB. In this delay time, one line
`from the Wide sensor is output from the dual-sensor system.
`After the sensor configuration stage, the data is sent in parallel
`or by using multiplexing into the processing section.
`
`FIG. 5 shows an embodiment of a method disclosed
`004.5
`herein for acquiring a Zoom image in capture mode. In ISP
`step 502, the data of each sensor is transferred to the respec
`tive ISP component, which performs on the data various
`processes Such as denoising, demosaicing, sharpening, Scal
`ing, etc., as known in the art. After the processing in step 502,
`all following actions are performed in capture processing core
`128: in rectification step 504, both Wide and Tele images are
`aligned to be on the epipolar line; in registration step 506,
`mapping between the Wide and the Tele aligned images is
`performed to produce a registration map; in resampling step
`508, the Tele image is resampled according to the registration
`map, resulting in a re-sampled Tele image; in decision step
`510, the re-sampled Tele image and the Wide image are
`processed to detect errors in the registration and to provide a
`decision output; in fusion step 512, the decision output, re
`sampled Tele image and the Wide image are fused into a
`single Zoom image.
`0046. To reduce processing time and power, steps 506,
`508, 510,512 could be bypassed by not fusing the images in
`non-focused areas. In this case, all steps specified above
`should be applied on focused areas only. Since the Tele opti
`cal system will introduce shallower depth of field than the
`Wide optical system, defocused areas will suffer from lower
`contrast in the Tele system.
`Video Mode Operation/Function
`0047. In this mode, sensor oversampling is used to enable
`continuous Zoom experience in video mode. Processing is
`applied to eliminate the changes in the image during cross
`over from one Sub-camera to the other. Zoom from 1 to Z,
`is performed using the Wide sensor only. From Z, and on,
`it is performed mainly by the Tele sensor. To prevent jumps”
`(roughness in the image), Switching to the Tele image is done
`using a Zoom factor which is a bit higher (Z+AZoom)
`than ZAZoom is determined according to the systems
`properties and is different for cases where Zoom-in is applied
`and cases where Zoom-out is applied (AZoomiz AZoom).
`This is done to prevent residual jumps artifacts to be visible at
`a certain Zoom factor.The Switching between sensors, for an
`increasing Zoom and for decreasing Zoom, is done on a dif
`ferent Zoom factor.
`0048. The Zoom video mode operation includes two
`stages: (1) sensor control and configuration, and (2) image
`processing. In the range from 1 to Z, only the Wide
`sensor is operational, hence, power can be supplied only to
`this sensor. Similar conditions hold for a
`0049 Wide AF mechanism. From Z+AZoom to Z,
`only the Tele sensor is operational, hence, power is Supplied
`only to this sensor. Similarly, only the Tele sensor is opera
`tional and power is supplied only to it for a Tele AF mecha
`nism. Another option is that the Tele sensor is operational and
`the Wide sensor is working in low frame rate. From Z
`to
`Z+AZoom, both sensors are operational.
`0050 FIG. 6 shows an embodiment of a method disclosed
`herein for acquiring a Zoom image in video/preview mode for
`3 different Zoom factor (ZF) ranges: (a) ZF range=1:Z
`Sviticia
`(b) ZF range-Z, Z+AZoom, and (c) Zoom factor
`range–Z,+AZoom, Z. The description is with refer
`ence to a graph of effective resolution vs. Zoom value (FIG.
`7). In step 602, sensor control module 116 chooses (directs)
`the sensor (Wide, Tele or both) to be operational. Specifically,
`if the ZF range=1:Z. module 116 directs the Wide sensor
`to be operational and the Tele sensor to be non-operational. If
`
`

`

`US 2016/0050374, A1
`
`Feb. 18, 2016
`
`the ZF range is Z.Z., +AZoom, module 116 directs
`both sensors to be operational and the Zoom image is gener
`ated from the Wide sensor. If the ZF range is Z+
`AZoom: Z, module 116 directs the Wide sensor to be
`non-operational and the Tele sensor to be operational. After
`the sensor choice in step 602, all following actions are per
`formed in video processing core 126. Optionally, in step 604,
`color balance is calculated if two images are provided by the
`two sensors. Optionally yet, in step 606, the calculated color
`balance is applied in one of the images (depending on the
`Zoom factor). Further optionally, in step 608, registration is
`performed between the Wide and Tele images to output a
`transformation coefficient. The transformation