I 1111111111111111 1111111111 111111111111111 111111111111111 IIIIII IIII IIII IIII
`USO 102254 79B2
`
`c12) United States Patent
`Shabtay et al.
`
`(IO) Patent No.: US 10,225,479 B2
`Mar.5,2019
`(45) Date of Patent:
`
`(54) DUAL APERTURE ZOOM DIGITAL
`CAMERA
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`(71) Applicant: Corephotonics Ltd., Tel Aviv (IL)
`
`(72)
`
`Inventors: Gal Shabtay, Tel Aviv (IL); Ephraim
`Goldenberg, Ashdod (IL); Oded
`Gigushinski, Herzlia (IL); Noy Cohen,
`Tel Aviv (IL)
`
`(73) Assignee: Corephotonics Ltd., Tel Aviv (IL)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days.
`
`(21) Appl. No.: 16/048,242
`
`(22) Filed:
`
`Jul. 28, 2018
`
`(65)
`
`Prior Publication Data
`
`US 2018/0359423 Al
`
`Dec. 13, 2018
`
`Related U.S. Application Data
`
`(63) Continuation of application No. 15/865,869, filed on
`Jan. 9, 2018, which is a continuation of application
`(Continued)
`
`(51)
`
`Int. Cl.
`H04N 51232
`H04N 51225
`
`(2006.01)
`(2006.01)
`(Continued)
`
`(52) U.S. Cl.
`CPC ....... H04N 5123296 (2013.01); G02B 13/009
`(2013.01); G02B 13/0015 (2013.01);
`(Continued)
`( 58) Field of Classification Search
`CPC ........................ G02B 13/0015; G02B 27/0075
`See application file for complete search history.
`
`2,354,503 A
`2,378,170 A
`
`7/1944 Cox
`6/1945 Aldin
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`CN
`CN
`
`10/2008
`101276415 A
`10/2012
`102739949 A
`(Continued)
`
`OTHER PUBLICATIONS
`
`Statistical Modeling and Performance Characterization of a Real(cid:173)
`Time Dual Camera Surveillance System, Greienhagen et al., Pub(cid:173)
`lisher: IEEE, 2000, 8 pages.
`(Continued)
`Primary Examiner - Cynthia Segura
`(74) Attorney, Agent, or Firm - Nathan and Associates;
`Menachem Nathan
`ABSTRACT
`(57)
`A dual-aperture zoom digital camera operable in both still
`and video modes. The camera includes Wide and Tele
`imaging sections with respective lens/sensor combinations
`and image signal processors and a camera controller opera(cid:173)
`tively coupled to the Wide and Tele imaging sections. The
`Wide and Tele imaging sections provide respective image
`data. The controller is configured to combine in still mode at
`least some of the Wide and Tele image data to provide a
`fused output image from a particular point of view, and to
`provide without fusion continuous zoom video mode output
`images, each output image having a given output resolution,
`wherein the video mode output images are provided with a
`smooth transition when switching between a lower zoom
`factor (ZF) value and a higher ZF value or vice versa, and
`wherein at the lower ZF the output resolution is determined
`by the Wide sensor while at the higher ZF value the output
`resolution is determined by the Tele sensor.
`40 Claims, 8 Drawing Sheets
`
`Choo•eS<."LNl\lSJlobeop~rntrnnal
`602
`
`APPL-1001 / Page 1 of 21
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`US 10,225,479 B2
`Page 2
`
`Related U.S. Application Data
`
`No. 15/424,853, filed on Feb. 5, 2017, now Pat. No.
`1 O 015 408 which is a continuation of application
`N~. 14/880'.251, filed on Oct. 11, 2015, now Pat. No.
`9 661 233 which is a continuation of application No.
`14/365,711, filed as application No. PCT/IB2014/
`062180 on Jun. 12, 2014, now Pat. No. 9,185,291.
`
`(60) Provisional application No. 61/834,486, filed on Jun.
`13, 2013.
`
`(51)
`
`(2006.01)
`(2006.01)
`
`Int. Cl.
`G02B 13/00
`G02B 27100
`(52) U.S. Cl.
`G02B 2710075 (2013.01); H04N 51225
`CPC .........
`(2013.01); H04N 512258 (2013.01); H04N
`512259 (2013.01); H04N 5123212 (2013.01);
`H04N 5123232 (2013.01); H04N 5123245
`(2013.01)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2,441,093 A
`3,388,956 A
`3,524,700 A
`3,864,027 A
`3,942,876 A
`4,134,645 A
`4,199,785 A
`4,338,001 A
`4,465,345 A
`5,000,551 A
`5,005,083 A
`5,032,917 A
`5,051,830 A
`5,248,971 A
`5,287,093 A
`5,436,660 A
`5,444,478 A
`5,459,520 A
`5,657,402 A
`5,682,198 A
`5,768,443 A
`5,926,190 A
`5,940,641 A
`5,982,951 A
`6,101,334 A
`6,128,416 A
`6,148,120 A
`6,208,765 Bl
`6,268,611 Bl
`6,549,215 B2
`6,611,289 Bl
`6,643,416 Bl
`6,650,368 Bl
`6,654,180 B2
`6,680,748 Bl
`6,714,665 Bl
`6,724,421 Bl
`6,738,073 B2
`6,741,250 Bl
`6,750,903 Bl
`6,778,207 Bl
`7,002,583 B2
`7,015,954 Bl
`7,038,716 B2
`7,187,504 B2
`7,199,348 B2
`7,206,136 B2
`7,248,294 B2
`7,256,944 B2
`7,305,180 B2
`
`5/1948 Aklin
`6/1968 Eggert et al.
`8/1970 Eggert et al.
`2/1975 Harada
`3/1976 Betensky
`1/1979 Sugiyama et al.
`4/1980 McCullough et al.
`7/1982 Matsui
`8/1984 Yazawa
`3/1991 Shibayama
`4/1991 Grage et al.
`7/1991 Aschwanden
`9/1991 von Hoessle
`9/1993 Mandi
`2/1994 Amano et al.
`7/1995 Sakamoto
`8/1995 Lelong et al.
`10/1995 Sasaki
`8/1997 Bender et al.
`10/1997 Katayama et al.
`6/1998 Michael et al.
`7/1999 Turkowski et al.
`8/1999 McIntyre et al.
`11/1999 Katayama et al.
`8/2000 Pantone
`10/2000 Oura
`11/2000 Sussman
`3/2001 Bergen
`7/2001 Pettersson et al.
`4/2003 Jouppi
`8/2003 Yu et al.
`11/2003 Daniels et al.
`11/2003 Doron
`11/2003 Ori
`1/2004 Monti
`3/2004 Hanna et al.
`4/2004 Glatt
`5/2004 Park et al.
`5/2004 Furlan et al.
`6/2004 Miyatake et al.
`8/2004 Lee et al.
`2/2006 Rabb, III
`3/2006 Foote et al.
`5/2006 Klein et al.
`3/2007 Horiuchi
`4/2007 Olsen et al.
`4/2007 Labaziewicz et al.
`7/2007 Slatter
`8/2007 Labaziewicz et al.
`12/2007 Labaziewicz et al.
`
`3/2008 Fortier
`7,339,621 B2
`3/2008 Gold, Jr.
`7,346,217 Bl
`4/2008 Cheatle et al.
`7,365,793 B2
`8/2008 Doyle
`7,411,610 B2
`9/2008 Baudisch et al.
`7,424,218 B2
`3/2009 Hosono
`7,509,041 B2
`4/2009 Chen et al.
`7,515,351 B2
`5/2009 Barkan et al.
`7,533,819 B2
`7/2009 Tang
`7,564,635 Bl
`11/2009 Davis
`7,619,683 B2
`1/2010 Tsai
`7,643,225 Bl
`2/2010 Tang
`7,660,049 B2
`3/2010 Tang
`7,684,128 B2
`3/2010 Sano
`7,688,523 B2
`4/2010 Tang et al.
`7,692,877 B2
`4/2010 Iyama
`7,697,220 B2
`6/2010 Toyofuku
`7,738,016 B2
`6/2010 Chen et al.
`7,738,186 B2
`8/2010 Huntsberger et al.
`7,773,121 Bl
`7,777,972 Bl
`8/2010 Chen et al.
`10/2010 Lin
`7,813,057 B2
`10/2010 Tang et al.
`7,821,724 B2
`11/2010 Tang et al.
`7,826,149 B2
`11/2010 Tsai
`7,826,151 B2
`1/2011 Chen et al.
`7,869,142 B2
`2/2011 LeGall et al.
`7,880,776 B2
`3/2011 Tang
`7,898,747 B2
`3/2011 Chen et al.
`7,916,401 B2
`4/2011 Li et al.
`7,918,398 B2
`6/2011 Tang
`7,957,075 B2
`6/2011 Tang
`7,957,076 B2
`6/2011 Tang
`7,957,079 B2
`6/2011 Tang et al.
`7,961,406 B2
`6/2011 Olsen et al.
`7,964,835 B2
`7/2011 Deever et al.
`7,978,239 B2
`8,000,031 Bl
`8/2011 Tsai
`8/2011 Yoshihito Souma
`8,004,777 B2
`12/2011 Tang
`8,077,400 B2
`2/2012 Culbert et al.
`8,115,825 B2
`4/2012 Lin et al.
`8,149,327 B2
`4/2012 Ozaki
`8,149,523 B2
`4/2012 Jo et al.
`8,154,610 B2
`7/2012 Tang
`8,218,253 B2
`7/2012 Tang
`8,228,622 B2
`7/2012 Chen
`8,233,224 B2
`8,238,695 Bl
`8/2012 Davey et al.
`8/2012 Lin
`8,253,843 B2
`9/2012 Dahi et al.
`8,274,552 B2
`10/2012 Sato
`8,279,537 B2
`1/2013 Tang et al.
`8,363,337 B2
`3/2013 Long et al.
`8,390,729 B2
`3/2013 Cho et al.
`8,391,697 B2
`3/2013 Tang et al.
`8,395,851 B2
`3/2013 Georgiev et al.
`8,400,555 Bl
`3/2013 Chen et al.
`8,400,717 B2
`5/2013 Ferren et al.
`8,439,265 B2
`5/2013 Muukki et al.
`8,446,484 B2
`5/2013 Yamanaka et al.
`8,451,549 B2
`7/2013 Ueda et al.
`8,483,452 B2
`8/2013 Chen et al.
`8,503,107 B2
`8,508,649 B2 * 8/2013 Reshidko .
`
`8,514,491 B2
`8,514,502 B2
`8,547,389 B2
`8,553,106 B2
`8,587,691 B2
`8,619,148 Bl
`8,780,465 B2
`8,803,990 B2
`8,810,923 B2
`8,854,745 Bl
`8,896,655 B2
`8,958,164 B2
`8,976,255 B2
`9,019,387 B2
`9,025,073 B2
`9,025,077 B2
`9,041,835 B2
`
`8/2013 Duparre
`8/2013 Chen
`10/2013 Hoppe et al.
`10/2013 Scarff
`11/2013 Takane
`12/2013 Watts et al.
`7/2014 Chae
`8/2014 Smith
`8/2014 Shinohara
`10/2014 Chen
`11/2014 Mauchly et al.
`2/2015 Kwon et al.
`3/2015 Matsuoto et al.
`4/2015 Nakano
`5/2015 Attar et al.
`5/2015 Attar et al.
`5/2015 Honda
`
`G02B 13/0045
`348/345
`
`APPL-1001 / Page 2 of 21
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`US 10,225,479 B2
`Page 3
`
`(56)
`
`References Cited
`
`2008/0030592 Al*
`
`2/2008 Border .
`
`2008/0030611 Al
`2008/0084484 Al
`2008/0117316 Al
`2008/0218611 Al
`2008/0218612 Al
`2008/0218613 Al*
`
`2/2008 Jenkins
`4/2008 Ochi et al.
`5/2008 Orimoto
`9/2008 Parulski et al.
`9/2008 Border et al.
`9/2008 Janson
`
`2008/0219654 Al *
`
`9/2008 Border .
`
`U.S. PATENT DOCUMENTS
`
`9,137,447 B2
`9,185,291 Bl
`9,215,377 B2
`9,215,385 B2
`9,229,194 B2
`9,235,036 B2
`9,270,875 B2
`9,279,957 B2
`9,286,680 Bl
`9,344,626 B2
`9,360,671 Bl
`9,369,621 B2
`9,402,032 B2 *
`9,413,930 B2
`9,413,984 B2
`9,420,180 B2
`9,438,792 B2
`9,485,432 Bl
`9,488,802 B2
`9,568,712 B2
`9,578,257 B2
`9,618,748 B2
`9,678,310 B2
`9,681,057 B2
`9,723,220 B2
`9,736,365 B2
`9,736,391 B2
`9,762,812 B2 *
`9,768,310 B2
`9,800,798 B2
`9,817,213 B2
`9,851,803 B2
`9,894,287 B2
`9,900,522 B2
`9,927,600 B2
`2002/0005902 Al
`2002/0063711 Al
`2002/007 5258 Al
`2002/0122113 Al
`2003/0030729 Al
`2003/0093805 Al
`2003/0160886 Al
`2003/0202113 Al
`2004/0008773 Al
`2004/0017386 Al
`2004/0027367 Al
`2004/0061788 Al
`2004/0240052 Al
`2005/0013509 Al
`2005/0046740 Al
`2005/0141103 Al
`2005/0157184 Al
`2005/0168840 Al
`2005/0200718 Al
`2006/0054782 Al
`2006/0056056 Al
`2006/0125937 Al
`2006/0170793 Al
`2006/017 5549 Al
`2006/0187310 Al
`2006/0187322 Al
`2006/0187338 Al
`2007 /0024737 Al
`2007/0146503 Al*
`
`9/2015 Shibuno
`11/2015 Shabtay et al.
`12/2015 Sokeila et al.
`12/2015 Luo
`1/2016 Yoneyama et al.
`1/2016 Kato et al.
`2/2016 Brisedoux et al.
`3/2016 Kanda et al.
`3/2016 Jiang et al.
`5/2016 Silverstein et al.
`6/2016 Zhou
`6/2016 Malone et al.
`7/2016 Dror
`8/2016 Geerds
`8/2016 Attar et al.
`8/2016 Jin
`9/2016 Nakada et al.
`11/2016 Medasani et al.
`11/2016 Chen et al.
`2/2017 Dror et al.
`2/2017 Attar et al.
`4/2017 Munger et al.
`6/2017 Iwasaki et al.
`6/2017 Attar et al.
`8/2017 Sugie
`8/2017 Laroia
`8/2017 Du et al.
`9/2017 Belote .
`9/2017 Ahn et al.
`10/2017 Ravirala et al.
`11/2017 Mercado
`12/2017 Fisher et al.
`2/2018 Qian et al.
`2/2018 Lu
`3/2018 Goldenberg et al.
`1/2002 Yuen
`5/2002 Park et al.
`6/2002 Park et al.
`9/2002 Foote
`2/2003 Prentice et al.
`5/2003 Gin
`8/2003 Misawa et al.
`10/2003 Yoshikawa
`1/2004 Itokawa
`1/2004 Liu et al.
`2/2004 Pilu
`4/2004 Bateman
`12/2004 Minefuji et al.
`1/2005 Samadani
`3/2005 Davis
`6/2005 Nishina
`7/2005 Nakanishi et al.
`8/2005 Kobayashi et al.
`9/2005 Lee
`3/2006 Olsen et al.
`3/2006 Ahiska et al.
`6/2006 LeGall et al.
`8/2006 Pasquarette et al.
`8/2006 Miller et al.
`8/2006 Janson et al.
`8/2006 Janson et al.
`8/2006 May et al.
`2/2007 Nakamura et al.
`6/2007 Shiraki
`
`2007/0177025 Al
`2007/0188653 Al
`2007/0189386 Al
`2007 /0229983 Al
`2007 /0257184 Al
`2007/0285550 Al
`2008/0017557 Al
`2008/0024614 Al
`2008/0025634 Al
`
`8/2007 Kopet et al.
`8/2007 Pollock et al.
`8/2007 Imagawa et al.
`10/2007 Saori
`11/2007 Olsen et al.
`12/2007 Son
`1/2008 Witdouck
`1/2008 Li et al.
`1/2008 Border et al.
`
`G02B 9/60
`
`G02B 5/005
`
`H04N 3/1593
`348/231.3
`
`H04N 5/232
`348/218.1
`
`G03B 15/00
`348/262
`H04N 5/23212
`396/89
`
`2008/0304161 Al
`2009/0086074 Al
`2009/0122195 Al
`2009/0122406 Al
`2009/0122423 Al
`2009/0128644 Al
`2009/0219547 Al
`2009/0252484 Al
`2009/0295949 Al
`2010/0013906 Al
`2010/0020221 Al
`2010/0060746 A9
`2010/0103194 Al
`2010/0238327 Al
`2010/0283842 Al
`2011/0001838 Al
`2011/0064327 Al*
`
`12/2008 Souma
`4/2009 Li et al.
`5/2009 Van Baar et al.
`5/2009 Rouvinen et al.
`5/2009 Park et al.
`5/2009 Camp et al.
`9/2009 Kauhanen et al.
`10/2009 Hasuda et al.
`12/2009 Ojala
`1/2010 Border et al.
`1/2010 Tupman et al.
`3/2010 Olsen et al.
`4/2010 Chen et al.
`9/2010 Griffith et al.
`11/2010 Guissin et al.
`1/2011 Lee
`3/2011 Dagher ................... G06T 5/004
`382/263
`
`H04N 5/232
`348/240.2
`
`2011/0080487 Al
`2011/0115965 Al
`2011/0128288 Al
`2011/0164172 Al
`2011/0229054 Al
`2011/0234853 Al
`2011/0234881 Al
`2011/0242286 Al
`2011/0242355 Al
`2012/0026366 Al*
`
`4/2011 Venkataraman et al.
`5/2011 Engelhardt et al.
`6/2011 Petrou et al.
`7/2011 Shintani et al.
`9/2011 Weston et al.
`9/2011 Hayashi et al.
`9/2011 Wakabayashi et al.
`10/2011 Pace et al.
`10/2011 Gorna et al.
`2/2012 Golan
`
`2012/0062780 Al
`2012/0069235 Al
`2012/0075489 Al
`2012/0092777 Al
`2012/0105579 Al
`2012/0105708 Al
`2012/0154929 Al
`2012/0196648 Al
`2012/0229663 Al
`2012/0249815 Al
`2012/0287315 Al
`2012/0320467 Al
`2013/0002928 Al
`2013/0093842 Al
`2013/0135445 Al
`2013/0182150 Al
`2013/0201360 Al
`2013/0202273 Al
`2013/0208178 Al
`2013/0235224 Al
`2013/0250150 Al
`2013/0258044 Al
`2013/0286488 Al
`2013/0321668 Al
`2014/0022436 Al
`2014/0049615 Al
`2014/0118584 Al
`2014/0192238 Al
`2014/0192253 Al
`2014/0204480 Al
`2014/0285907 Al
`2014/0293453 Al
`2014/0313316 Al
`2014/0362242 Al
`2014/0362274 Al
`2015/0002683 Al
`2015/0042870 Al
`2015/0092066 Al
`2015/0116569 Al
`
`3/2012 Morihisa
`3/2012 Imai
`3/2012 Nishihara
`4/2012 Tochigi et al.
`5/2012 Jeon et al.
`5/2012 Hagiwara
`6/2012 Tsai et al.
`8/2012 Havens et al.
`9/2012 Nelson et al.
`10/2012 Bohn et al.
`11/2012 Huang et al.
`12/2012 Baik et al.
`1/2013 Imai
`4/2013 Yahata
`5/2013 Dahi et al.
`7/2013 Asakura
`8/2013 Song
`8/2013 Ouedraogo et al.
`8/2013 Park
`9/2013 Park et al.
`9/2013 Malone et al.
`10/2013 Betts-LaCroix
`10/2013 Chae
`12/2013 Karnath
`1/2014 Kim et al.
`2/2014 Uwagawa
`5/2014 Lee et al.
`7/2014 Attar et al.
`7/2014 Laroia
`7/2014 Jo et al.
`9/2014 Tang et al.
`10/2014 Ogino et al.
`10/2014 Olsson et al.
`12/2014 Takizawa
`12/2014 Christie et al.
`1/2015 Hu et al.
`2/2015 Chan et al.
`4/2015 Geiss et al.
`4/2015 Mercado
`
`APPL-1001 / Page 3 of 21
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`US 10,225,479 B2
`Page 4
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2015/0154776 Al
`2015/0162048 Al
`2015/0195458 Al
`2015/0215516 Al
`2015/0237280 Al
`2015/0242994 Al
`2015/0253647 Al
`2015/0271471 Al
`2015/0316744 Al
`2015/0334309 Al
`2016/0044250 Al
`2016/0070088 Al
`2016/0085089 Al
`2016/0154202 Al
`2016/0154204 Al
`2016/0187631 Al
`2016/0191819 Al*
`
`6/2015
`6/2015
`7/2015
`7/2015
`8/2015
`8/2015
`9/2015
`9/2015
`11/2015
`11/2015
`2/2016
`3/2016
`3/2016
`6/2016
`6/2016
`6/2016
`6/2016
`
`Zhang et al.
`Hirata et al.
`Nakayama et al.
`Dolgin
`Choi et al.
`Shen
`Mercado
`Hsieh et al.
`Chen
`Peng et al.
`Shabtay et al.
`Koguchi
`Mercado
`Wippermann et al.
`Lim et al.
`Choi et al.
`Sakai
`
`2016/0202453 Al*
`
`7/2016
`
`Iwasawa .
`
`Shikata
`2016/0212358 Al
`7/2016
`Du et al.
`2016/0301840 Al
`10/2016
`Mercado
`2016/0313537 Al
`10/2016
`2016/0353012 Al
`Kao et al.
`12/2016
`Zhu et al.
`2017/0019616 Al
`1/2017
`2017 /0054911 Al *
`Lee
`2/2017
`2017/0102522 Al
`Jo
`4/2017
`2017/0115471 Al
`Shinohara
`4/2017
`2017/0214846 Al
`Du et al.
`7/2017
`2017/0214866 Al
`Zhu et al.
`7/2017
`Song et al.
`2017/0289458 Al
`10/2017
`2018/0096487 Al *
`Nash.
`4/2018
`Avivi et al.
`2018/0120674 Al
`5/2018
`Tang et al.
`2018/0150973 Al
`5/2018
`2018/0152640 Al*
`Shabtay
`5/2018
`Lee et al.
`2018/0224630 Al
`8/2018
`Baldwin et al.
`2018/0241922 Al
`8/2018
`2018/0295292 Al
`Lee et al.
`10/2018
`2018/0359423 Al* 12/2018 Shabtay
`
`H04N 5/23245
`348/333.12
`G02B 13/18
`359/684
`
`H04N 5/23293
`
`H04N 5/217
`
`H04N 5/2258
`
`H04N 5/2258
`
`FOREIGN PATENT DOCUMENTS
`
`CN
`CN
`EP
`EP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`
`103024272 A
`104297906 A
`1536633 Al
`2523450 Al
`S54157620 A
`S59121015 A
`04211230 A
`H07318864 A
`08271976 A
`2003298920 A
`2004133054 A
`2005099265 A
`2006238325 A
`2007228006 A
`2007306282 A
`
`4/2013
`1/2015
`6/2005
`11/2012
`12/1979
`7 /1984
`8/1992
`12/1995
`10/1996
`10/2003
`4/2004
`4/2005
`9/2006
`9/2007
`11/2007
`
`JP
`JP
`JP
`KR
`KR
`KR
`KR
`WO
`WO
`
`2008076485 A
`2012203234 A
`2013106289 A
`20100008936 A
`20140014787 A
`20140023552 A
`101477178 Bl
`2013058111 Al
`2013063097 Al
`
`4/2008
`10/2012
`5/2013
`1/2010
`2/2014
`2/2014
`12/2014
`4/2013
`5/2013
`
`OTHER PUBLICATIONS
`
`A 3MPixel Multi-Aperture Image Sensor with 0.7µm Pixels in
`0.llµm CMOS, Fife et al., Stanford University, 2008, 3 pages.
`Dual camera intelligent sensor for high definition 360 degrees
`surveillance, Scotti et al., Publisher: IET, May 9, 2000, 8 pages.
`Dual-sensor foveated imaging system, Hua et al., Publisher: Optical
`Society of America, Jan. 14, 2008, 11 pages.
`Defocus Video Matting, McGuire et al., Publisher: ACM SIG(cid:173)
`GRAPH, Jul. 31, 2005, 11 pages.
`Compact multi-aperture imaging with high angular resolution,
`Santacana et al., Publisher: Optical Society of America, 2015, 10
`pages.
`Multi-Aperture Photography, Green et al., Publisher: Mitsubishi
`Electric Research Laboratories, Inc., Jul. 2007, 10 pages.
`Multispectral Bilateral Video Fusion, Bennett et al., Publisher:
`IEEE, May 2007, 10 pages.
`Super-resolution imaging using a camera array, Santacana et al.,
`Publisher: Optical Society of America, 2014, 6 pages.
`Optical Splitting Trees for High-Precision Monocular Imaging,
`McGuire et al., Publisher: IEEE, 2007, 11 pages.
`High Performance Imaging Using Large Camera Arrays, Wilburn et
`al., Publisher: Association for Computing Machinery, Inc., 2005, 12
`pages.
`Real-time Edge-Aware Image Processing with the Bilateral Grid,
`Chen et al., Publisher: ACM SIGGRAPH, 9 pages.
`Superimposed multi-resolution imaging, Caries et al., Publisher:
`Optical Society of America, 2017, 13 pages.
`Viewfinder Alignment, Adams et al., Publisher: EUROGRAPHICS,
`2008, 10 pages.
`Dual-Camera System for Multi-Level Activity Recognition, Bodor
`et al., Publisher: IEEE, Oct. 2014, 6 pages.
`Engineered to the task: Why camera-phone cameras are different,
`Giles Humpston, Publisher: Solid State Technology, Jun. 2009, 3
`pages.
`A compact and cost effective design for cell phone zoom lens,
`Chang et al., Sep. 2007, 8 pages.
`Consumer Electronic Optics: How small a lens can be? The case of
`panomorph lenses, Thibault et al., Sep. 2014, 7 pages.
`Optical design of camera optics for mobile phones, Steinich et al.,
`2012, pp. 51-58 (8 pages).
`The Optics of Miniature Digital Camera Modules, Bareau et al.,
`2006, 11 pages.
`Modeling and measuring liquid crystal tunable lenses, Peter P.
`Clark, 2014, 7 pages.
`Mobile Platform Optical Design, Peter P. Clark, 2014, 7 pages.
`
`* cited by examiner
`
`APPL-1001 / Page 4 of 21
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Mar.5,2019
`
`Sheet 1 of 8
`
`US 10,225,479 B2
`
`,----------------------------------------------------------------------------------~ I
`
`l- -
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`- -I
`
`1- -
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`- -,
`
`I
`
`Wide lens 102
`
`Tele Jens I 08
`
`I
`I
`
`- - - - - - - -
`
`I
`
`Wide sensor I 04
`
`Tele sensor 110
`
`Wide ISP 106
`
`, l~--T-.e-l_e _1s_P_u_2 __ ~
`
`I ~----------------------------1
`Camera Controller 114
`
`Sensor control 116
`
`User control 118
`
`Video processing
`126
`
`Still processing
`128
`
`~ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - I
`
`FlG. lA
`
`Tele camera
`
`FIG. 1B
`
`APPL-1001 / Page 5 of 21
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Mar.5,2019
`
`Sheet 2 of 8
`
`US 10,225,479 B2
`
`Tele
`
`204
`
`202
`
`FIG. 2
`
`APPL-1001 / Page 6 of 21
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Mar.5,2019
`
`Sheet 3 of 8
`
`US 10,225,479 B2
`
`Wide sensor
`Line Number
`
`Wide
`I
`Rolling
`I
`~1/frame rate~~ Sh
`t ~
`Wide
`I
`ut er
`I
`Time
`l.A____
`-..1
`1
`1--..Vertical~
`I
`I
`Blank
`-------'•1~---,
`.------'·,..___,
`I
`I
`
`Wide
`
`r i
`TL_
`
`Overlapping 1
`
`I
`I
`I
`I
`1/frame rate~1
`:
`I
`j
`, Tele
`I
`: Rolling;
`1
`! Shutter!
`I
`: Time :
`I
`:(cid:141)
`~:~
`:
`
`Tele
`Vertical
`Blank
`
`'
`
`!
`}+!
`-----~
`
`\
`
`Time
`
`Tele
`Exposure
`Time
`
`Tele sensor\
`Line Number
`
`______________ \ __________ ...,. Time
`
`FIG. 3
`
`APPL-1001 / Page 7 of 21
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Mar.5,2019
`
`Sheet 4 of 8
`
`US 10,225,479 B2
`
`Horizontal Blank
`
`..................... Jfl?!_!~Pf!,JJ)~~~\.s .................... ):
`
`• . . . . . . . . . . . . . . . . . . . H~p.z,~l!J);/ W!'9~. . . . . . . . . . . . . . . . . . . . J
`•
`n
`
`.................. ··············;··~;~x-··
`
`FIG. 4
`
`APPL-1001 / Page 8 of 21
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Mar.5,2019
`
`Sheet 5 of 8
`
`US 10,225,479 B2
`
`ISP: Perform image signal
`processing on data received fom1
`each sensor to obtain processed
`Wide and Tele images
`502
`!
`Rectification: Align processed
`Wide and Tele images to be on an
`epipolar line to obtain aligned
`(rectified) images
`504
`
`~
`Registration: Map the aligned
`Wide and Tele images to obtain a
`registration map
`506
`i
`
`Resampling: Process registration
`map and processed Tele image to
`obtain a re-sampled Tele image
`508
`
`1
`
`Decision: Use re-sampled Tele
`image and Wide image to detect
`errors in the registration and to
`provide a decision output
`510
`
`l
`
`Fusion: Fuse the decision output,
`re-sampled Tele image and Wide
`image into a fused zoom image
`512
`
`FIG. 5
`
`APPL-1001 / Page 9 of 21
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Mar.5,2019
`
`Sheet 6 of 8
`
`US 10,225,479 B2
`
`,.__ Choose sensor(s) to be operational
`602
`
`l
`
`,.__
`
`,.__
`
`Optionally, calculate color balance if
`two (Wide and Tele) images are
`provided by the two sensors.
`604
`
`l
`
`Optionally, apply calculated color
`balance in one of the images
`606
`
`l
`Optionally,
`perform
`registration
`,.__ between the Wide and Tele images to
`output a transfonnation coefficient
`608
`
`l
`
`Set an AF position using the
`transfonnation coefficient
`610
`
`Process an output of any of steps
`
`612
`
`--- 602-608 to obtain a processed image
`l
`
`Resample the processed image
`according to the transformation
`coefficient, requested ZF and output
`video resolution
`614
`
`FIG. 6
`
`APPL-1001 / Page 10 of 21
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Mar.5,2019
`
`Sheet 7 of 8
`
`US 10,225,479 B2
`
`Effective
`Resolution
`
`FIG. 7
`
`~Zoomdown
`<
`)
`~Zoomup
`
`User Zoom factor
`
`APPL-1001 / Page 11 of 21
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Mar.5,2019
`
`Sheet 8 of 8
`
`US 10,225,479 B2
`
`804
`
`.. -------( [~((7
`
`\ \ \
`
`\
`
`..._ ____ _
`,
`
`\
`..._____J
`
`\..,_FFFF~
`
`802
`
`806
`
`810
`
`FIG. 8
`
`904
`
`908
`
`902
`
`906
`
`910
`
`FIG. 9
`
`APPL-1001 / Page 12 of 21
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`FIELD
`
`Embodiments disclosed herein relate in general to digital
`cameras and in particular to thin zoom digital cameras with 25
`both still image and video capabilities
`
`BACKGROUND
`
`US 10,225,479 B2
`
`1
`DUAL APERTURE ZOOM DIGITAL
`CAMERA
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a Continuation application of U.S.
`patent application Ser. No. 15/865,869, filed Jan. 9, 2018,
`which was a Continuation application of U.S. patent appli(cid:173)
`cation Ser. No. 15/424,853 filed Feb. 5, 2017, which was a
`Continuation application of U.S. patent application Ser. No.
`14/880,251 filed Oct. 11, 2015 (issued as U.S. Pat. No.
`9,661,233), which was a Continuation application of U.S.
`patent application Ser. No. 14/365,711 filed Jun. 16, 2014
`(issued as U.S. Pat. No. 9,185,291), which was a 371
`application from international patent application PCT/
`IB2014/062180 filed Jun. 12, 2014, and is related to and
`claims priority from U.S. Provisional Patent Application No.
`61/834,486 having the same title and filed Jun. 13, 2013,
`which is incorporated herein by reference in its entirety.
`
`2
`image sensor region. The image sensor ( or simply "sensor")
`region is configured to receive the image and to generate a
`set of image data based on the image. The digital camera
`may be aligned to receive electromagnetic radiation associ-
`5 ated with scenery having a given set of one or more objects.
`The set of image data may be represented as digital image
`data, as well known in the art. Hereinafter in this description,
`"image" "image data" and "digital image data" may be used
`interchangeably. Also, "object" and "scene" may be used
`10 interchangeably.
`Multi-aperture imaging systems and associated methods
`are described for example in US Patent Publications No.
`2008/0030592, 2010/0277619 and 2011/0064327. In US
`2008/0030592, two sensors are operated simultaneously to
`15 capture an image imaged through an associated lens. A
`sensor and its associated lens form a lens/sensor combina(cid:173)
`tion. The two lenses have different focal lengths. Thus, even
`though each lens/sensor combination is aligned to look in
`the same direction, each captures an image of the same
`20 subject but with two different fields of view (FOYs). One
`sensor is commonly called "Wide" and the other "Tele".
`Each sensor provides a separate image, referred to respec(cid:173)
`tively as "Wide" (or "W") and "Tele" (or "T") images. A
`W-image reflects a wider FOY and has lower resolution than
`the T-image. The images are then stitched (fused) together to
`form a composite ("fused") image. In the composite image,
`the central portion is formed by the relatively higher(cid:173)
`resolution image taken by the lens/sensor combination with
`the longer focal length, and the peripheral portion is formed
`30 by a peripheral portion of the relatively lower-resolution
`Digital camera modules are currently being incorporated
`into a variety of host devices. Such host devices include
`image taken by the lens/sensor combination with the shorter
`focal length. The user selects a desired amount of zoom and
`cellular telephones, personal data assistants (PDAs), com(cid:173)
`the composite image is used to interpolate values from the
`puters, and so forth. Consumer demand for digital camera
`chosen amount of zoom to provide a respective zoom image.
`modules in host devices continues to grow.
`Host device manufacturers prefer digital camera modules 35 The solution offered by US 2008/0030592 requires, in video
`mode, very large processing resources in addition to high
`to be small, so that they can be incorporated into the host
`device without increasing its overall size. Further, there is an
`frame rate requirements and high power consumption (since
`both cameras are fully operational).
`increasing demand for such cameras to have higher-perfor(cid:173)
`US 2010/0277619 teaches a camera with two lens/sensor
`mance characteristics. One such characteristic possessed by
`many higher-performance cameras ( e.g., standalone digital 40
`combinations, the two lenses having different focal lengths,
`still cameras) is the ability to vary the focal length of the
`so that the image from one of the combinations has a FOY
`camera to increase and decrease the magnification of the
`approximately 2-3 times greater than the image from the
`image. This ability, typically accomplished with a zoom
`other combination. As a user of the camera requests a given
`lens, is known as optical zooming. "Zoom" is commonly
`amount of zoom, the zoomed image is provided from the
`lens/sensor combination having a FOY that is next larger
`understood as a capability to provide different magnifica- 45
`than the requested FOY. Thus, if the requested FOY is less
`tions of the same scene and/or object by changing the focal
`length of an optical system, with a higher level of zoom
`than the smaller FOY combination, the zoomed image is
`associated with greater magnification and a lower level of
`created from the image captured by that combination, using
`cropping and interpolation if necessary. Similarly, if the
`zoom associated with lower magnification. Optical zooming
`is typically accomplished by mechanically moving lens 50
`requested FOY is greater than the smaller FOY combination,
`elements relative to each other. Such zoom lenses are
`the zoomed image is created from the image captured by the
`typically more expensive, larger and less reliable than fixed
`other combination, using cropping and interpolation if nec(cid:173)
`focal length lenses. An alternative approach for approximat(cid:173)
`essary. The solution offered by US 2010/0277619 leads to
`ing the zoom effect is achieved with what is known as digital
`parallax artifacts when moving to the Tele camera in video
`zooming. With digital zooming, instead of varying the focal 55
`mode.
`In both US 2008/0030592 and US 2010/0277619, differ(cid:173)
`length of the lens, a processor in the camera crops the image
`and interpolates between the pixels of the captured image to
`ent focal length systems cause Tele and Wide matching
`FOYs to be exposed at different times using CMOS sensors.
`create a magnified but lower-resolution image.
`This degrades the overall image quality. Different optical F
`Attempts to use multi-aperture imaging systems to
`approximate the effect of a zoom lens are known. A multi- 60
`numbers ("F#'') cause image intensity differences. Working
`aperture imaging system (implemented for example in a
`with such a dual sensor system requires double bandwidth
`digital camera) includes a plurality of optical sub-systems
`support, i.e. additional wires from the sensors to the follow(cid:173)
`(also referred to as "sub-cameras"). Each sub-camera
`ing HW component. Neither US 2008/0030592 nor US
`includes one or more lenses and/or other optical elements
`2010/0277619 deal with registration errors. Neither
`US2008/000592 nor US 2010/0277619 refer to partial
`which define an aperture such that received electro-magnetic 65
`radiation is imaged by the optical sub-system and a resulting
`fusion, i.e. fusion ofless than all the pixels of both Wide and
`image is directed towards a two-dimensional (2D) pixelated
`Tele images in still mode.
`
`APPL-1001 / Page 13 of 21
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`3
`US 2011/0064327 discloses multi-aperture imaging sys(cid:173)
`tems and methods for image data fusion that include pro(cid:173)
`viding first and second sets of image data corresponding to
`an imaged first and second scene respectively. The scenes
`overlap at least partially in an overlap region, defining a first
`collection of overlap image data as part of the first set of
`image data, and a second collection of overlap image data as
`part of the second set of image data. The second collection
`of overlap image data is represented as a plurality of image
`data sub-cameras such that each of the sub-cameras is based 10
`on at least one characteristic of the second collection, and
`each sub-camera spans the overlap region. A fused set of
`image data is produced by an image processor, by modifying
`the first collection of overlap image data based on at least a
`selected one of, but less than all of, the image data sub(cid:173)
`cameras. The systems and methods disclosed in this appli(cid:173)
`cation deal solely with fused still images.
`None of the known art references provide a thin (e.g.
`fitting in a cell-phone) dual-aperture zoom digital camera
`with fixed focal length lenses, the camera configured to
`operate in both still mode and video mode to provide still
`and video images, wherein the camera configuration uses
`partial or full fusion to provide a fused image in still mode
`and does not use any fusion to provide a continuous, smooth
`zoom in video mode.
`Therefore there is a need for, and it would be advanta(cid:173)
`geous to have thin digital cameras with optical zoom oper(cid:173)
`ating in both video and still mode that do not suffer from
`commonly encountered problems and disadvantages, some
`of which are listed above.
`
`SUMMARY
`
`Embodiments disclosed herein teach the use of dual(cid:173)
`aperture (also referred to as dual-lens or two-sensor) optical
`zoom digital cameras. The cameras include two sub-cam(cid:173)
`eras, a Wide sub-camera and a Tele sub-camera, each
`sub-camera including a fixed focal length lens, an image
`sensor and an image signal processor (ISP). The Tele sub(cid:173)
`camera is the higher zoom sub-camera and the Wide sub- 40
`camera is the lower zoom sub-camera. In some embodi(cid:173)
`ments, the lenses are thin lenses with short optical paths of
`less than about 9 mm. In some embodiments, the thickness/
`effective focal length (EFL) ratio of the Tele lens is smaller
`than about 1. The image sensor may include two separate 2D 45
`pixelated sensors or a single pixelated sensor divided into at
`least two areas. The digital camera can be operated in both
`still and video modes. In still mode, zoom is achieved "with
`fusion" (full or partial), by fusing Wand T images, with the
`resulting fused image including always information from 50
`both Wand T images. Partial fusion may be achieved by not
`using fusion in image areas where the Tele image is not
`focused. This advantageously
`reduces computational
`requirements (e.g. time).
`In video mode, optical zoom is achieved "without fusion", 55
`by switching between the W and T images to shorten
`computational time requirements, thus enabling h