US009 185291B1
`
`(12) United States Patent
`Shabtay et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9,185,291 B1
`Nov. 10, 2015
`
`(54) DUAL APERTURE ZOOM DIGITAL CAMERA
`
`(56)
`
`References Cited
`
`(71) Applicant: Corephotonics Ltd., Tel-Aviv (IL)
`
`(72) Inventors: Gal Shabtay, Tel-Aviv (IL); Ephraim
`Goldenberg, Ashdod (IL); Oded
`Gigushinski, Tel-Aviv (IL); Noy Cohen,
`Tel-Aviv (IL)
`
`(73) Assignee: Corephotonics Ltd., Tel-Aviv (IL)
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.:
`
`14/365,711
`
`(22) PCT Filed:
`
`Jun. 12, 2014
`
`(86). PCT No.:
`S371 (c)(1),
`(2) Date:
`
`Jun. 16, 2014
`
`(87) PCT Pub. No.: WO2014/199338
`PCT Pub. Date: Dec. 18, 2014
`
`(51)
`
`(52)
`
`(58)
`
`(2006.01)
`(2006.01)
`
`Int. C.
`H04N 5/232
`H04N 5/225
`U.S. C.
`CPC ........... H04N 5/23245 (2013.01); H04N5/225
`(2013.01); H04N5/2259 (2013.01); H04N
`5/23212 (2013.01); H04N 5/23296 (2013.01)
`Field of Classification Search
`CPC .................................................. HO4N 5/23296
`USPC ..................................................... 348/240.99
`See application file for complete search history.
`
`U.S. PATENT DOCUMENTS
`
`5,172,235 A * 12/1992 Wilm ................... HO4N 5,2254
`348, 149
`5,436,660 A * 7/1995 Sakamoto ................ HO4N 5/33
`348,229.1
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`JP
`WO
`
`2006238325 A * 9, 2006 ............. HO4N 5,232
`201310512 A2
`T 2013
`(Continued)
`OTHER PUBLICATIONS
`JP 2006238325 A; Camera system for clear display of video in
`display apparatus, displays simultaneously or records image of wide
`camera and image of tele camera by Switching to tele video and wide
`video during movement of tele camera; CANON; Sep. 2006; English
`Abstract.
`(Continued)
`Primary Examiner — Roberto Velez
`Assistant Examiner — Cynthia Segura
`(74) Attorney, Agent, or Firm — Nathan & Associates Patent
`Agents Ltd.; Menachem Nathan
`(57)
`ABSTRACT
`A dual-aperture Zoom digital camera operable in both still
`and video modes. The camera includes Wide and Tele imag
`ing sections with respective lens/sensor combinations and
`image signal processors and a camera controller operatively
`coupled to the Wide and Tele imaging sections. The Wide and
`Tele imaging sections provide respective image data. The
`controller is configured to combine in still mode at least some
`of the Wide and Tele image data to provide a fused output
`image from a particular point of view, and to provide without
`fusion continuous Zoom video mode output images, each
`output image having a given output resolution, wherein the
`Video mode output images are provided with a smooth tran
`sition when switching between a lower Zoom factor (ZF)
`value and a higher ZF value or vice versa, and wherein at the
`lower ZF the output resolution is determined by the Wide
`sensor while at the higher ZF value the output resolution is
`determined by the Tele sensor.
`22 Claims, 8 Drawing Sheets
`
`Teleier8 (8.
`
`Wide sensor 1848
`
`Tee sens 1:0
`
`Wide ISP 6
`
`Tee SP 12
`
`
`
`sniera Controller 14
`
`Sensor controil i8
`
`Uses controi i8
`
`2)
`
`2
`
`32s.
`
`widex processixg
`26
`
`Still processing
`28
`
`Apple v. Corephotonics
`Exhibit 2009
`IPR2018-01133
`
`Exhibit 2009 Page 1 of 18
`
`

`

`US 9,185,291 B1
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2007, 0146503 A1* 6, 2007 Shiraki ................ HO4N 3, 1593
`348,231.3
`2/2008 Border ................... HNS
`
`2008/0030592 A1
`
`6,977,676 B1
`
`12/2005 Sato ................. G08B 13, 19628
`348, 14.08
`
`7.305,180 B2 12/2007 Labaziewicz et al.
`ck
`7.388,182 B2 * 6/2008 Schofield ............. B609:
`
`7,561,191 B2
`7,676,146 B2,
`7,724,300 B2 *
`
`7/2009 May et al.
`3, 2010 Border et al.
`5/2010 Misawa ............... HO4N 5,2253
`348,33305
`7,843,499 B2 * 1 1/2010 Watanabe ........ G08B 13, 19643
`348/143
`
`4/2012 Lin et al.
`8, 149,327 B2
`5, 2013 Ferren et al.
`8.439,265 B2
`9, 2013 Griffith et al.
`8,542,287 B2
`8,553,106 B2 10, 2013 Scarff
`8,660,420 B2
`2/2014 Chang
`8,731.390 B2
`5/2014 Goldenberg et al.
`8,860,849 B2 * 10/2014 Misawa ............... HO4N 5.2258
`348,240.2
`2003/0227556 A1* 12/2003 Doyle ............... G06F 3.0481
`348,239
`2006/0275025 A1 12/2006 Labaziewicz .......... HO4N 5,225
`396/72
`2007, 0109399 A1* 5, 2007 Sekimoto ............. HO4N 5,2253
`348/36
`
`2008/0218613 A1* 9/2008 Janson ................... GO3B 1500
`2008/02196.54 A1* 9, 2008 Bord
`Hoax:
`OCC . . . . . . . . . . . . . . . . .
`396,89
`2009/0273688 A1* 11/2009 Nonaka ................ HO4N 5,2259
`348.222.1
`
`2010/0277619 A1 11, 2010 Scarff
`2011/0064327 A1
`3/2011 Dagher et al.
`2012/0026366 A1
`2/2012 Golan .................... HO4N 5,232
`348,240.2
`
`1/2015 Dror et al.
`2015.0029601 A1
`2015/0085174 A1* 3/2015 Shabtay .......... HO4N 5/23296
`348,336
`2015/0244942 A1* 8/2015 Shabtay ............. HO4N 5/23296
`348,240.3
`
`FOREIGN PATENT DOCUMENTS
`
`WO
`WO
`
`1, 2015
`2O15001519 A2
`2, 2015
`2015O15383 A2
`OTHER PUBLICATIONS
`
`PCT/IB2014/062180 Search Report of the international search
`authority, Mar. 2015.
`
`* cited by examiner
`
`Apple v. Corephotonics
`Exhibit 2009
`IPR2018-01133
`
`Exhibit 2009 Page 2 of 18
`
`

`

`U.S. Patent
`
`Nov. 10, 2015
`
`Sheet 1 of 8
`
`US 9,185,291 B1
`
`O
`
`Wide sensor 104
`
`Tele Sensor 10
`
`Wide SP ()6
`
`Tee SP 12
`
`
`
`Sensor control 16
`
`ser control 8
`
`Camera Controker 14
`
`Tele camera
`
`FG. A
`
`F.G. 18
`
`Apple v. Corephotonics
`Exhibit 2009
`IPR2018-01133
`
`Exhibit 2009 Page 3 of 18
`
`

`

`U.S. Patent
`
`Nov. 10, 2015
`
`Sheet 2 of 8
`
`US 9,185,291 B1
`
`
`
`Jele
`
`2O2
`
`20.
`
`FG 2
`
`Apple v. Corephotonics
`Exhibit 2009
`IPR2018-01133
`
`Exhibit 2009 Page 4 of 18
`
`

`

`U.S. Patent
`
`Nov. 10, 2015
`
`Sheet 3 of 8
`
`US 9,185,291 B1
`
`Wide sensor
`Line Number
`
`if frame rate
`
`
`
`Wide
`Rolling
`Shutter
`Time
`
`:
`
`:
`
`:
`
`Overlapping
`FOW
`
`8
`8
`
`8
`w
`
`:
`
`&
`s.
`:
`W
`:
`
`s
`
`Exposure
`re
`
`
`
`:
`
`:
`:
`:
`
`8
`8
`8
`
`or 1/fra me raters:
`tele
`Roising
`Shutter
`Time :
`
`ete
`Vertical
`Bank
`
`ete
`rExposure
`Tire
`
`fee sensor
`Line Number \ s
`
`s
`
`
`
`
`
`ine
`
`ine
`
`Apple v. Corephotonics
`Exhibit 2009
`IPR2018-01133
`
`Exhibit 2009 Page 5 of 18
`
`

`

`U.S. Patent
`
`Nov. 10, 2015
`
`Sheet 4 of 8
`
`US 9,185,291 B1
`
`
`
`Horizontal Blank
`
`Ei.
`ES56 Six
`TE533i Eik T.
`
`XER.
`ESSE
`ESSEE
`EOK.
`Hisis
`
`F.G. 4
`
`Apple v. Corephotonics
`Exhibit 2009
`IPR2018-01133
`
`Exhibit 2009 Page 6 of 18
`
`

`

`U.S. Patent
`
`Nov. 10, 2015
`
`Sheet 5 of 8
`
`US 9,185,291 B1
`
`
`
`ISP. perform image signal
`processing on data received from
`each Sensor to obtain processed
`Wide and Tele images
`S)2
`
`Rectification: align processed
`Wide and Tele images to be on an
`epipolar line to obtain aligned
`(rectified) inhages
`S04
`
`Registration: map the aligned
`Wide and Tele images to obtain a
`registration map
`506
`
`ReSampling: proceSS registration
`map and processed Tele image to
`obtain a re-sampled Tele image
`508
`
`Decision: use re-sampled Tele
`image and Wide image to detect
`errors in the registration and to
`provide a decision output
`S()
`
`Fusion: fuse the decision output,
`re-sampled Tele image and Wide
`image into a fused Zoom image
`S2
`
`FIG.S
`
`Apple v. Corephotonics
`Exhibit 2009
`IPR2018-01133
`
`Exhibit 2009 Page 7 of 18
`
`

`

`U.S. Patent
`
`Nov. 10, 2015
`
`Sheet 6 of 8
`
`US 9,185,291 B1
`
`Choose sensor(s) to be operational
`602
`
`Optionally, calculate color balance if
`two (Wide and Tele) images are
`provided by the two sensors.
`604
`
`Optionally, apply calculated color
`balance in one of the images
`606
`
`registration
`perform
`Optionally,
`between the Wide and Tele images to
`output a transformation coefficient
`608
`
`
`
`Set an AF position using the
`transformation coefficient
`6)
`
`Process an output of any of Steps
`602-608 to obtain a processed image
`62
`
`Resample the processed image
`according to the transformation
`coefficient, requested ZF and output
`video resolution
`64
`
`FIG. 6
`
`Apple v. Corephotonics
`Exhibit 2009
`IPR2018-01133
`
`Exhibit 2009 Page 8 of 18
`
`

`

`U.S. Patent
`
`Nov. 10, 2015
`
`Sheet 7 of 8
`
`US 9,185,291 B1
`
`Effective
`Resolution
`
`
`
`F.G. 7
`
`User Zoom factor
`
`AZOOmdown
`K-Ge.
`AZoom,
`
`Apple v. Corephotonics
`Exhibit 2009
`IPR2018-01133
`
`Exhibit 2009 Page 9 of 18
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov. 10, 2015
`Nov. 10, 2015
`
`Sheet 8 of 8
`Sheet 8 of 8
`
`US 9,185,291 B1
`US 9,185,291 Bl
`
`808
`
`804.
`
`
`
`802
`
`806
`806
`
`80
`810
`
`
`
`FG. 8
`FIG. 8
`
`
`
`
`
`
`
` {
`
`/
`i
`é;
`
`|
`f
`é
`
`é
`
`a
`
`904
`
`908
`
`
`
`
`
`902
`902
`
`906
`
`9 O
`
`FIG. 9
`FIG. 9
`
`Apple v. Corephotonics
`Exhibit 2009
`IPR2018-01133
`
`Exhibit 2009 Page 10 of 18
`
`Apple v. Corephotonics
`Exhibit 2009
`IPR2018-01133
`
`Exhibit 2009 Page 10 of 18
`
`

`

`US 9, 185,291 B1
`
`1.
`DUAL APERTURE ZOOMIDIGITAL CAMERA
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`This application is related to and claims priority from US
`Provisional Patent Application No. 61/834,486 having the
`same title and filed Jun. 13, 2013, which is incorporated
`herein by reference in its entirety.
`
`10
`
`FIELD
`
`Embodiments disclosed herein relate in general to digital
`cameras and in particular to thin Zoom digital cameras with
`both still image and video capabilities
`
`15
`
`BACKGROUND
`
`2
`Multi-aperture imaging systems and associated methods
`are described for example in US Patent Publications No.
`2008/0030592, 2010/0277619 and 2011/0064327. In US
`2008/0030592, two sensors are operated simultaneously to
`capture an image imaged through an associated lens. A sensor
`and its associated lens form a lens/sensor combination. The
`two lenses have different focal lengths. Thus, even though
`each lens/sensor combination is aligned to look in the same
`direction, each captures an image of the same Subject but with
`two different fields of view (FOVs). One sensor is commonly
`called “Wide” and the other “Tele'. Each sensor provides a
`separate image, referred to respectively as “Wide (or “W)
`and “Tele” (or “T”) images. AW-image reflects a wider FOV
`and has lower resolution than the T-image. The images are
`then stitched (fused) together to form a composite (“fused’)
`image. In the composite image, the central portion is formed
`by the relatively higher-resolution image taken by the lens/
`sensor combination with the longer focal length, and the
`peripheral portion is formed by a peripheral portion of the
`relatively lower-resolution image taken by the lens/sensor
`combination with the shorter focal length. The user selects a
`desired amount of Zoom and the composite image is used to
`interpolate values from the chosen amount of Zoom to provide
`a respective Zoom image. The solution offered by US 2008/
`0030592 requires, in video mode, very large processing
`resources in addition to high frame rate requirements and
`high power consumption (since both cameras are fully opera
`tional).
`US 2010/0277619 teaches a camera with two lens/sensor
`combinations, the two lenses having different focal lengths,
`so that the image from one of the combinations has a FOV
`approximately 2-3 times greater than the image from the
`other combination. As a user of the camera requests a given
`amount of Zoom, the Zoomed image is provided from the
`lens/sensor combination having a FOV that is next larger than
`the requested FOV. Thus, if the requested FOV is less than the
`smaller FOV combination, the Zoomed image is created from
`the image captured by that combination, using cropping and
`interpolation if necessary. Similarly, if the requested FOV is
`greater than the smaller FOV combination, the Zoomed image
`is created from the image captured by the other combination,
`using cropping and interpolation if necessary. The Solution
`offered by US 2010/0277619 leads to parallax artifacts when
`moving to the Tele camera in video mode.
`In both US 2008/0030592 and US 2010/0277619, different
`focal length systems cause Tele and Wide matching FOVs to
`be exposed at different times using CMOS sensors. This
`degrades the overall image quality. Different optical F num
`bers (“Fi”) cause image intensity differences. Working with
`Sucha dual sensor System requires double bandwidth Support,
`i.e. additional wires from the sensors to the following HW
`component. Neither US 2008/0030592 nor US 2010/
`0277619 deal with registration errors. Neither US 2008/
`000592 nor US 2010/0277619 refer to partial fusion, i.e.
`fusion of less than all the pixels of both Wide and Tele images
`in still mode.
`US 2011/006.4327 discloses multi-aperture imaging sys
`tems and methods for image data fusion that include provid
`ing first and second sets of image data corresponding to an
`imaged first and second scene respectively. The scenes over
`lap at least partially in an overlap region, defining a first
`collection of overlap image data as part of the first set of
`image data, and a second collection of overlap image data as
`part of the second set of image data. The second collection of
`overlap image data is represented as a plurality of image data
`Sub-cameras Such that each of the Sub-cameras is based on at
`least one characteristic of the second collection, and each
`
`Digital camera modules are currently being incorporated
`into a variety of host devices. Such host devices include
`cellular telephones, personal data assistants (PDAs), comput
`ers, and so forth. Consumer demand for digital camera mod
`ules in host devices continues to grow.
`Host device manufacturers prefer digital camera modules
`to be small, so that they can be incorporated into the host
`device without increasing its overall size. Further, there is an
`increasing demand for Such cameras to have higher-perfor
`mance characteristics. One such characteristic possessed by
`many higher-performance cameras (e.g., standalone digital
`still cameras) is the ability to vary the focal length of the
`camera to increase and decrease the magnification of the
`image. This ability, typically accomplished with a Zoom lens,
`is known as optical Zooming. "Zoom’ is commonly under
`stood as a capability to provide different magnifications of the
`same scene and/or object by changing the focal length of an
`optical system, with a higher level of Zoom associated with
`greater magnification and a lower level of Zoom associated
`with lower magnification. Optical Zooming is typically
`accomplished by mechanically moving lens elements relative
`to each other. Such Zoom lenses are typically more expensive,
`larger and less reliable than fixed focal length lenses. An
`alternative approach for approximating the Zoom effect is
`achieved with what is known as digital Zooming. With digital
`Zooming, instead of varying the focal length of the lens, a
`processor in the camera crops the image and interpolates
`between the pixels of the captured image to create a magnified
`but lower-resolution image.
`Attempts to use multi-aperture imaging systems to
`approximate the effect of a Zoom lens are known. A multi
`aperture imaging system (implemented for example in a digi
`tal camera) includes a plurality of optical Sub-systems (also
`referred to as “sub-cameras'). Each sub-camera includes one
`or more lenses and/or other optical elements which define an
`aperture Such that received electro-magnetic radiation is
`imaged by the optical Sub-system and a resulting image is
`directed towards a two-dimensional (2D) pixelated image
`sensor region. The image sensor (or simply “sensor”) region
`is configured to receive the image and to generate a set of
`image databased on the image. The digital camera may be
`aligned to receive electromagnetic radiation associated with
`scenery having a given set of one or more objects. The set of
`image data may be represented as digital image data, as well
`known in the art. Hereinafter in this description, “image'
`“image data' and “digital image data” may be used inter
`changeably. Also, “object' and "scene' may be used inter
`changeably.
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Apple v. Corephotonics
`Exhibit 2009
`IPR2018-01133
`
`Exhibit 2009 Page 11 of 18
`
`

`

`US 9, 185,291 B1
`
`3
`Sub-camera spans the overlap region. A fused set of image
`data is produced by an image processor, by modifying the first
`collection of overlap image databased on at least a selected
`one of, but less than all of the image data Sub-cameras. The
`systems and methods disclosed in this application deal Solely
`with fused still images.
`None of the known art references provide a thin (e.g. fitting
`in a cell-phone) dual-aperture Zoom digital camera with fixed
`focal length lenses, the camera configured to operate in both
`still mode and video mode to provide still and video images,
`wherein the camera configuration uses partial or full fusion to
`provide a fused image in still mode and does not use any
`fusion to provide a continuous, Smooth Zoom in video mode.
`Therefore there is a need for, and it would be advantageous
`to have thin digital cameras with optical Zoom operating in
`both video and still mode that do not suffer from commonly
`encountered problems and disadvantages, some of which are
`listed above.
`
`10
`
`15
`
`4
`ments and HW requirements. In addition, it provides an inno
`vative optical design that enables a low TTL/EFL ratio using
`a specific lens curvature order.
`Due to the large focal length, objects that are in front or
`behind the plane of focus appear very blurry, and a nice
`foreground-to-background contrast is achieved. However, it
`is difficult to create such a blur using a compact camera with
`a relatively short focal length and Small aperture size, such as
`a cell-phone camera. In some embodiments, a dual-aperture
`Zoom system disclosed herein can be used to capture a shal
`low DOF photo (shallow compared with a DOF of a Wide
`camera alone), by taking advantage of the longer focal length
`of the Tele lens. The reduced DOF effect provided by the
`longer Tele focal length can be further enhanced in the final
`image by fusing data from an image captured simultaneously
`with the Wide lens. Depending on the distance to the object,
`with the Tele lens focused on a subject of the photo, the Wide
`lens can be focused to a closer distance than the Subject so that
`objects behind the subject appear very blurry. Once the two
`images are captured, information from the out-of-focus
`blurred background in the Wide image is fused with the
`original Tele image background information, providing a
`blurrier background and even shallower DOF.
`In an embodiment there is provided a Zoom digital camera
`comprising a Wide imaging section that includes a fixed focal
`length Wide lens with a Wide FOV, a Wide sensor and a Wide
`image signal processor (ISP), the Wide imaging section
`operative to provide Wide image data of an object or scene; a
`Tele imaging section that includes a fixed focal length Tele
`lens with a Tele FOV that is narrower than the Wide FOV, a
`Tele sensor and a Tele ISP, the Tele imaging section operative
`to provide Tele image data of the object or scene; and a
`camera controller operatively coupled to the Wide and Tele
`imaging sections, the camera controller configured to com
`bine in still mode at least some of the Wide and Tele image
`data to provide a fused output image of the object or scene
`from a particular point of view (POV), and to provide without
`fusion continuous Zoom video mode output images of the
`object or scene, a camera controller operatively coupled to the
`Wide and Tele imaging sections, the camera controller con
`figured to combine in still mode at least some of the Wide and
`Tele image data to provide a fused output image of the object
`or scene from a particular point of view and to provide with
`out fusion continuous Zoom video mode output images of the
`object or scene, each output image having a respective output
`resolution, wherein the video output images are provided
`with a smooth transition when switching between a lower
`Zoom factor (ZF) value and a higher ZF value or vice versa,
`wherein at the lower ZF value the output resolution is deter
`mined by the Wide sensor, and wherein at the higher ZF value
`the output resolution is determined by the Tele sensor.
`In an embodiment, the camera controller configuration to
`provide video output images with a Smooth transition when
`switching between a lower ZF value and a higher ZF value or
`Vice versa includes a configuration that uses at high ZF sec
`ondary information from the Wide camera and uses at low ZF
`secondary information from the Tele camera. As used herein,
`'secondary information” refers to white balance gain, expo
`Sure time, analog gain and color correction matrix.
`In a dual-aperture camera image plane, as seen by each
`Sub-camera (and respective image sensor), a given object will
`be shifted and have different perspective (shape). This is
`referred to as point-of-view (POV). The system output image
`can have the shape and position of either sub-camera image or
`the shape or position of a combination thereof. If the output
`image retains the Wide image shape then it has the Wide
`perspective POV. If it retains the Wide camera position then it
`
`SUMMARY
`
`Embodiments disclosed herein teach the use of dual-aper
`ture (also referred to as dual-lens or two-sensor) optical Zoom
`digital cameras. The cameras include two Sub-cameras, a
`Wide Sub-camera and a Tele Sub-camera, each Sub-camera
`including a fixed focal length lens, an image sensor and an
`image signal processor (ISP). The Tele sub-camera is the
`higher Zoom sub-camera and the Wide sub-camera is the
`lower Zoom Sub-camera. In some embodiments, the lenses are
`thin lenses with short optical paths of less than about 9 mm. In
`some embodiments, the thickness/effective focal length
`(EFL) ratio of the Telelens is smaller than about 1. The image
`sensor may include two separate 2D pixelated sensors or a
`single pixelated sensor divided into at least two areas. The
`digital camera can be operated in both still and video modes.
`In still mode, Zoom is achieved “with fusion’ (full or partial),
`by fusing W and T images, with the resulting fused image
`including always information from both W and T images.
`Partial fusion may be achieved by not using fusion in image
`areas where the Tele image is not focused. This advanta
`geously reduces computational requirements (e.g. time).
`In video mode, optical Zoom is achieved “without fusion',
`by switching between the W and T images to shorten com
`putational time requirements, thus enabling high video rate.
`To avoid discontinuities in Video mode, the Switching
`includes applying additional processing blocks, which
`include image scaling and shifting.
`In order to reach optical Zoom capabilities, a different
`magnification image of the same scene is captured (grabbed)
`by each camera Sub-camera, resulting in FOV overlap
`between the two Sub-cameras. Processing is applied on the
`two images to fuse and output one fused image in still mode.
`The fused image is processed according to a user Zoom factor
`request. As part of the fusion procedure, up-sampling may be
`applied on one or both of the grabbed images to Scale it to the
`image grabbed by the Tele sub-camera or to a scale defined by
`the user. The fusion or up-sampling may be applied to only
`Some of the pixels of a sensor. Down-sampling can be per
`formed as well if the output resolution is smaller than the
`sensor resolution.
`The cameras and associated methods disclosed herein
`address and correct many of the problems and disadvantages
`of known dual-aperture optical Zoom digital cameras. They
`provide an overall Zoom solution that refers to all aspects:
`optics, algorithmic processing and system hardware (HW).
`The proposed solution distinguishes between video and still
`mode in the processing flow and specifies the optical require
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Apple v. Corephotonics
`Exhibit 2009
`IPR2018-01133
`
`Exhibit 2009 Page 12 of 18
`
`

`

`5
`has the Wide position POV. The same applies for Tele images
`position and perspective. As used in this description, the
`perspective POV may be of the Wide or Tele sub-cameras,
`while the position POV may shift continuously between the
`Wide and Tele sub-cameras. In fused images, it is possible to
`register Tele image pixels to a matching pixel set within the
`Wide image pixels, in which case the output image will retain
`the Wide POV (“Wide fusion”). Alternatively, it is possible to
`register Wide image pixels to a matching pixel set within the
`Tele image pixels, in which case the output image will retain
`the Tele POV (“Telefusion'). It is also possible to perform the
`registration after either Sub-camera image is shifted, in which
`case the output image will retain the respective Wide or Tele
`perspective POV.
`In an embodiment there is provided a method for obtaining
`Zoom images of an object or scene in both still and video
`modes using a digital camera, the method comprising the
`steps of providing in the digital camera a Wide imaging
`section having a Wide lens with a Wide FOV, a Wide sensor
`and a Wide image signal processor (ISP), a Tele imaging
`section having a Tele lens with a Tele FOV that is narrower
`than the Wide FOV, a Tele sensor and a Tele ISP, and a camera
`controller operatively coupled to the Wide and Tele imaging
`sections; and configuring the camera controller to combine in
`still mode at least some of the Wide and Tele image data to
`provide a fused output image of the object or scene from a
`particular point of view, and to provide without fusion con
`tinuous Zoom video mode output images of the object or
`scene, each output image having a respective output resolu
`tion, wherein the video mode output images are provided with
`a smooth transition when switching between a lower ZF value
`and a higher ZF value or vice versa, and wherein at the lower
`ZF value the output resolution is determined by the Wide
`sensor while at the higher ZF value the output resolution is
`determined by the Tele sensor.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Non-limiting examples of embodiments disclosed herein
`are described below with reference to figures attached hereto
`that are listed following this paragraph. The drawings and
`descriptions are meant to illuminate and clarify embodiments
`disclosed herein, and should not be considered limiting in any
`way.
`FIG. 1A shows schematically a block diagram illustrating
`a dual-aperture Zoom imaging system disclosed herein;
`FIG. 1B is a schematic mechanical diagram of the dual
`aperture Zoom imaging system of FIG. 1A:
`FIG. 2 shows an example of Wide sensor, Tele sensor and
`their respective FOVs;
`FIG.3 shows a schematically embodiment of CMOS sen
`sor image grabbing vs. time;
`FIG. 4 shows Schematically a sensor time configuration
`which enables sharing one sensor interface using dual sensor
`Zoom system;
`FIG.5 shows an embodiment of a method disclosed herein
`for acquiring a Zoom image in capture mode;
`FIG. 6 shows an embodiment of a method disclosed herein
`for acquiring a Zoom image in Video/preview mode;
`FIG. 7 shows a graph illustrating an effective resolution
`Zoom factor,
`FIG. 8 shows one embodiment of a lens block in a thin
`camera disclosed herein;
`FIG.9 shows another embodiment of a lens block in a thin
`camera disclosed herein.
`
`DETAILED DESCRIPTION
`
`FIG. 1A shows schematically a block diagram illustrating
`an embodiment of a dual-aperture Zoom imaging system (also
`
`US 9, 185,291 B1
`
`6
`referred to simply as “digital camera' or “camera') disclosed
`herein and numbered 100. Camera 100 comprises a Wide
`imaging section ("sub-camera') that includes a Wide lens
`block 102, a Wide image sensor 104 and a Wide image pro
`cessor 106. Camera 100 further comprises a Tele imaging
`section (“sub-camera') that includes a Tele lens block 108, a
`Tele image sensor 110 and a Tele image processor 112. The
`image sensors may be physically separate or may be part of a
`single larger image sensor. The Wide sensor pixel size can be
`equal to or different from the Tele sensor pixel size. Camera
`100 further comprises a camera fusion processing core (also
`referred to as “controller) 114 that includes a sensor control
`module 116, a user control module 118, a video processing
`module 126 and a capture processing module 128, all opera
`tionally coupled to sensor control block 110. User control
`module 118 comprises an operational mode function 120, a
`region of interest (ROI) function 122 and a Zoom factor (ZF)
`function 124.
`Sensor control module 116 is connected to the two sub
`cameras and to the user control module 118 and used to
`choose, according to the Zoom factor, which of the sensors is
`operational and to control the exposure mechanism and the
`sensor readout. Mode choice function 120 is used for choos
`ing capture/video modes. ROI function 122 is used to choose
`a region of interest. As used herein, “ROI is a user defined as
`a Sub-region of the image that may be exemplarily 4% or less
`of the image area. The ROI is the region on which both
`Sub-cameras are focused on. Zoom factor function 124 is used
`to choose a Zoom factor. Video processing module 126 is
`connected to mode choice function 120 and used for video
`processing. Still processing module 128 is connected to the
`mode choice function 120 and used for high image quality
`still mode images. The video processing module is applied
`when the user desires to shoot in video mode. The capture
`processing module is applied when the user wishes to shoot
`still pictures.
`FIG. 1B is a schematic mechanical diagram of the dual
`aperture Zoom imaging system of FIG. 1A. Exemplary
`dimensions: Wide lens TTL=4.2 mm and EFL–3.5 mm; Tele
`lens TTL=6 mm and EFL=7 mm: both Wide and Tele sensors
`/3 inch. External dimensions of Wide and Tele cameras:
`width (w) and length (1)–8.5 mm and height (h)=6.8 mm.
`Distance “d between camera centers=10 mm.
`Following is a detailed description and examples of differ
`ent methods of use of camera 100.
`Design for Continuous and Smooth Zoom in Video Mode
`In an embodiment, in order to reach high quality continu
`ous and Smooth optical Zooming in video camera mode while
`reaching real optical Zoom using fixed focal length Sub-cam
`eras, the system is designed according to the following rules
`(Equations 1-3):
`(1)
`Tan(FOWu). Tan(FOVI)-PLui/PL,
`where Tan refers to "tangent, while FOV, and F refer
`respectively to the Wide and Tele lens fields of view (in
`degrees). As used herein, the FOV is measured from the
`center axis to the corner of the sensor (i.e. half the angle of the
`normal definition). PL
`and PL
`refer respectively to
`the “in-line' (i.e. in a line) number of Wide sensor pixels and
`in-line number of output video format pixels. The ratio
`PL/PL is called an “oversampling ratio'. For
`example, in order to get full and continuous optical Zoom
`experience with a 12 Mp sensor (sensor dimensions 4000x
`3000) and a required 1080p (dimension 1920x1080) video
`format, the FOV ratio should be 4000/1920–2,083. More
`over, if the Wide lens FOV is given as FOV=37.5°, the
`required Tele lens FOV is 20.2 The Zoom switching point is
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Apple v. Corephotonics
`Exhibit 2009
`IPR2018-01133
`
`Exhibit 2009 Page 13 of 18
`
`

`

`US 9, 185,291 B1
`
`7
`set according to the ratio between sensor pixels in-line and the
`number of pixels in-line in the video format and defined as:
`
`(2)
`Zovitch PL wide/PLvideo
`Maximum optical Zoom is reached according to the following
`formula:
`
`(3)
`Z-Tan(FOWu). Tan(FOVI)*PL,
`For example: for the configuration defined above and assum
`ing PL-4000 and PL-1920, Z=4.35.
`In an embodiment, the sensor control module has a setting
`that depends on the Wide and Tele FOVs and on a sensor
`oversampling ratio, the setting used in the configuration of
`each sensor. For example, when using a 4000x3000 sensor
`and when outputting a 1920x1080 image, the oversampling
`ratio is 4000/1920=2.0833.
`In an embodiment, the Wide and Tele FOVs and the over
`sampling ratio Satisfy the condition
`
`10
`
`15
`
`8
`Wide systems. In this case, the relative exposure time may be
`configured according to the formula below:
`ETri-ETria. (Fitti Finia.)-(Pixel sizewia Pixel
`sizete..)
`(5)
`Other exposure time ratios may be applied to achieve wide
`dynamic range and improved SNR. Fusing two images with
`different intensities will result in wide dynamic range image.
`In more detail with reference to FIG. 3, in the first stage,
`after the user chooses a required Zoom factor ZF, the sensor
`control module configures each sensor as follows:
`1) Cropping index Wide sensor:
`Yvie
`/2 PC (1-1/ZF)
`
`where PC is the number of pixels in a column, and Y is the row
`number
`2) Cropping index Tele sensor:
`If