(12) United States Patent
`Olsen et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9.232,158 B2
`Jan. 5, 2016
`
`US009232158B2
`
`LARGE DYNAMIC RANGE CAMERAS
`
`Inventors:
`
`Applicant: PROTARIUS FILO AG, L.L.C., Dover,
`DE (US)
`Richard Ian Olsen, Truckee, CA (US);
`Darryl L. Sato, Irvine, CA (US);
`Feng-Qing Sun, Austin, TX (US);
`James Gates, Carlsbad, CA (US)
`Callahan Cellular L.L.C., Wilmington,
`DE (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 176 days.
`Appl. No.: 14/063,236
`Filed:
`Oct. 25, 2013
`
`Assignee:
`
`Notice:
`
`(54)
`(71)
`
`(72)
`
`(73)
`
`(*)
`
`(21)
`(22)
`(65)
`
`(63)
`
`(58) Field of Classification Search
`CPC ................... H01L 27/14618: H01L 27/14621;
`H01L 27/14625; H01L 27/14634; H01L
`27/14645; H04N 5/2253; H04N 5/2254;
`H04N 5/2353; H04N 5/332: H04N 5/335
`USPC ........................ 250/208.1; 348/273, 302, 308
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`3,609,367 A
`3,971,065. A
`
`9, 1971 Barron
`7/1976 Bayer
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`JP
`
`O 599 470
`1 032 045
`62-011264
`
`6, 1994
`8, 2000
`1, 1987
`
`OTHER PUBLICATIONS
`
`Communication on EP Application 0.57939274, mailed Jul. 8, 2015.
`(Continued)
`
`Primary Examiner — Seung C Sohn
`
`ABSTRACT
`(57)
`A digital camera includes a plurality of channels and a pro
`cessing component operatively coupled to the plurality of
`channels. Each channel of the plurality of channels includes
`an optics component and a sensor that includes an array of
`photo-detectors. The processing component is configured to
`separately control an integration time of each channel, where
`a first integration time of a first channel is less than a second
`integration time of a second channel. The processing compo
`nent is also configured to combine data from the plurality of
`channels to generate an image.
`
`20 Claims, 24 Drawing Sheets
`
`Prior Publication Data
`US 2014/0049660 A1
`Feb. 20, 2014
`Related U.S. Application Data
`Continuation of application No. 13/681,603, filed on
`Nov. 20, 2012, now Pat. No. 8,598,504, which is a
`continuation of application No. 13/465.229, filed on
`May 7, 2012, now Pat. No. 8.334,494, which is a
`(Continued)
`
`(51)
`
`Int. C.
`HOIL 27/00
`H04N 5/335
`
`(2006.01)
`(2011.01)
`(Continued)
`
`(52)
`
`U.S. C.
`CPC ............ H04N 5/3355 (2013.01); G02B3/0062
`(2013.01); G02B3/0075 (2013.01); G02B 9/12
`(2013.01); HOIL 27/14618 (2013.01); HOIL
`27/14621 (2013.01); HOIL 27/14625 (2013.01);
`HOIL 27/14645 (2013.01); HOIL 31/0232
`(2013.01); H04N5/2253 (2013.01); H04N
`5/2254 (2013.01); H04N 5/2353 (2013.01);
`(Continued)
`
`
`
`26OD “, 26OE
`Light
`
`260F 30
`
`
`Ex.1001 / Page 1 of 40Ex.1001 / Page 1 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`US 9.232,158 B2
`Page 2
`
`Related U.S. Application Data
`continuation of application No. 12/496,854, filed on
`Jul. 2, 2009, now Pat. No. 8,198,574, which is a con
`tinuation of application No. 11/788,122, filed on Apr.
`19, 2007, now Pat. No. 7,564,019, and a continuation
`in-part of application No. 1 1/212.803, filed on Aug.
`25, 2005, now abandoned.
`Provisional application No. 60/795,882, filed on Apr.
`28, 2006, provisional application No. 60/604.854,
`filed on Aug. 25, 2004, provisional application No.
`60/695,946, filed on Jul. 1, 2005.
`
`(2006.01)
`(2006.01)
`(2014.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2011.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`
`Int. C.
`GO2B 9/12
`HOIL 27/46
`HOIL 3L/0232
`H04N 5/225
`H04N 5/235
`H04N 5/33
`H04N 5/353
`H04N 9/04
`H04N 9/09
`HO)4N 9/097
`GO2B3/OO
`U.S. C.
`CPC ............... H04N 5/332 (2013.01); H04N 5/335
`(2013.01); H04N5/3532 (2013.01); H04N
`9/045 (2013.01); H04N 9/09 (2013.01); H04N
`9/097 (2013.01); G02B3/0031 (2013.01);
`G02B3/0043 (2013.01); HOIL 27/14634
`(2013.01); HOIL 2924/0002 (2013.01); H04N
`2209/049 (2013.01)
`
`(60)
`
`(51)
`
`(52)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4, 1982 Abell et al.
`4.323,925 A
`5, 1983. Howe
`4.385,373 A
`8/1987 Wagner
`4,688,080 A
`1/1990 Tanaka
`4,894,672 A
`4, 1991 Grage et al.
`5,005,083. A
`9, 1991 Von Hoessle
`5,051,830 A
`7, 1995 Sakamoto
`5,436,660 A
`5,604,534 A * 2/1997 Hedges ................ GO1C 11,025
`348, 106
`
`8, 1997 Yamazaki
`5,654,752 A
`5,691,765 A 11/1997 Schieltz et al.
`5,694,165 A 12/1997 Yamazaki et al.
`5,742,659 A
`4, 1998 Atac et al.
`5,760,832 A
`6, 1998 Yamanaka et al.
`5,766.980 A
`6/1998 Ohtagaki et al.
`5,850,479 A 12/1998 Terry et al.
`6,137,535 A 10/2000 Meyers
`6,375,075 B1
`4/2002 Ackley et al.
`6,381,072 B1
`4/2002 Burger
`6,429,898 B1
`8, 2002 Shoda et al.
`6,437,335 B1
`8, 2002 Bohn
`6,570,613 B1
`5, 2003 Howell
`6,611,289 B1
`8, 2003 Yu et al.
`6,617,565 B2
`9, 2003 Wu
`6,714,239 B2
`3, 2004 Guidash
`6,727,521 B2
`4/2004 Merrill
`6,765,617 B1
`7/2004 Tangen et al.
`6,833,873 B1 12/2004 Suda
`6,834,161 Bl
`12/2004 Stiehler
`6,841,816 B2
`1/2005 Merrill et al.
`6,859,299 B1
`2/2005 Chiao
`6,882,368 B1
`4/2005 Suda
`6,885,398 B1
`4/2005 Sladen
`6,885.404 B1
`4/2005 Suda
`
`6,885,508
`6,903,770
`6,946,647
`6,952,228
`6,999,130
`7,095, 159
`7,115,853
`7,123,298
`7,170,665
`7,199,348
`7,206,136
`7,214,926
`7,223,954
`7,236,306
`7,239,345
`7,256.944
`7,280,290
`7,305,180
`7,358.483
`7,362,357
`7,379,104
`7,417,674
`7,436,038
`
`7.460,160
`2002fOO2O845
`2002fOO24606
`2002/0051071
`2002fOO67416
`2002fOO75481
`2002fOO89596
`2002fO113888
`2002/O122124
`2002fO142798
`2003/0020814
`2003, OO86013
`2003/OO957 11
`2003/O151685
`2003.0160886
`2003/0209651
`2003/0234907
`2004/0012688
`2004/0012689
`2004/OO17620
`2004/OO27687
`2004/0080638
`2004/OO95495
`2004/O183918
`2005/0O24731
`2005/O128335
`2005/O128509
`2005/0134712
`2005, 01601 12
`2005/02486.67
`2005/0285955
`2006/0087572
`2006/O108505
`2006/O125936
`2006/0187322
`2006/01873.38
`2006/0222220
`2007,0002159
`
`4/2005 Yamaguchi et al.
`B2
`6/2005 Kobayashi et al.
`B1
`9, 2005 O'Neill et al.
`B1
`B2 10, 2005 Yoneda et al.
`B2
`2/2006 Tanigawa
`B2
`8, 2006 Machida
`B2 10/2006 Jiang et al.
`B2 10, 2006 Schroeder et al.
`B2
`1/2007 Kaneko et al.
`B2
`4/2007 Olsen et al.
`B2
`4/2007 Labaziewicz et al.
`B2
`5/2007 Gruhlke et al.
`B2
`5/2007 McNulty
`B2
`6/2007 Janson, Jr. et al.
`B1
`7/2007 Rogina
`B2
`8/2007 Labaziewicz et al.
`B2 10, 2007 Araki et al.
`B2 12/2007 LabaZiewicz et al.
`B2
`4/2008 Mitsugi et al.
`B2
`4/2008 Brown et al.
`B2
`5/2008 Hattori et al.
`B2
`8, 2008 Gruhlke et al.
`B2 * 10/2008 Engelmann ............... GO1J 5/02
`257/233
`B2 12/2008 Hershey et al.
`A1
`2/2002 Ogura et al.
`A1
`2/2002 Yuki et al.
`A1
`5, 2002 Itano et al.
`A1
`6/2002 Yoneda et al.
`A1
`6/2002 Roustaei
`A1
`7, 2002 Suda
`A1
`8, 2002 Sonoda et al.
`A1
`9, 2002 Suda
`A1 10/2002 Miyake
`A1
`1/2003 Ono
`A1
`5/2003 Aratani
`A1
`5/2003 McGuinness et al.
`A1
`8, 2003 La Grone
`A1
`8, 2003 Misawa et al.
`A1 11/2003 Iwasaki
`A1 12/2003 Kawai
`A1
`1/2004 Tinnerino et al.
`A1
`1/2004 Tinnerino et al.
`A1
`1/2004 Kaneko et al.
`A1
`2/2004 Bittner et al.
`A1
`4/2004 Lee
`A1
`5/2004 Inokuma et al.
`A1
`9/2004 Squilla et al.
`A1
`2/2005 Mitchell et al.
`A1
`6/2005 Kolehmainen et al.
`A1
`6/2005 Tokkonen et al.
`A1
`6/2005 Gruhlke et al.
`A1
`7/2005 Makela et al.
`Al 11/2005 Schweng et al.
`A1 12/2005 Utz et al.
`A1
`4/2006 Schroeder
`A1
`5, 2006 Gruhlke et al.
`A1
`6/2006 Gruhlke et al.
`A1
`8/2006 Janson, Jr. et al.
`A1
`8/2006 May et al.
`A1 10, 2006 Yamano et al.
`A1
`1/2007 Olsen et al.
`
`OTHER PUBLICATIONS
`
`Communication from the European Patent Office on European Patent
`Application 0.57939274, mailed Feb. 10, 2012.
`Duparre et al., “Artificial apposition compound eye fabricated by
`micro-optics technology. Applied Optics, vol. 43, No. 22, Aug.
`2004, pp. 4303-4310.
`Duparre et al., “Artificial compound eyes—different concepts and
`their application to ultra flat image acquisition sensors.” Proceedings
`of SPIE, vol. 5346 (SPIE, Bellingham, WA, 2004), pp. 89-100.
`Duparre et al., “Microoptical telescope compound eyet.' Optics
`Express, vol. 13, No. 3, Feb. 2005, pp. 889-903.
`Duparre et al., “Theoretical analysis of an artificial Superposition
`compound eye for application in ultra flat digital image acquisition
`devices.” Proceedings of SPIE, vol. 5249, 2004, pp. 408-418.
`
`Ex.1001 / Page 2 of 40Ex.1001 / Page 2 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`US 9.232,158 B2
`Page 3
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Duparre et al., “Ultra-Thin Camera Based on Artificial Apposition
`Compound Eyes.” Proc. 10th Microoptics Conference MOC '04,
`Jena, 2004, Paper E-2 (2 pages).
`Final Office Action on U.S. Appl. No. 1 1/212.803, mailed Sep. 20.
`2007.
`First Office Action for Chinese Application 200580.032374.0, notifi
`cation date Feb. 5, 2010.
`International Preliminary Report on Patentability for PCT/US2005/
`030256 issued Mar. 17, 2009.
`International Preliminary Report on Patentability for PCT/US2006/
`025781 issued Mar. 10, 2009.
`International Search Report and Written Opinion for PCT/US05/
`30256 mailed Jul. 7, 2008.
`International Search Report and Written Opinion for PCT/US06/
`25781 mailed Jul 22, 2008.
`Kitamura et al., “Reconstruction of a high-resolution image on a
`compound-eye image-capturing system. Applied Optics, vol. 43.
`No. 8, Mar. 2004, pp. 1719-1727.
`Miki et al., “A Study of Multi-Stack Silicon-Direct Wafer Bonding
`for MEMS Manufacturing.” 2002, IEEE, pp. 407-410.
`Miyatake et al., “Thin observation module by bound optics
`(TOMBO0 with color filters.” SPEI and IS&T, vol. 5301, 2004, pp.
`7-12.
`Non-final Office Action on U.S. Appl. No. 1 1/212.803, mailed Feb. 7,
`2007.
`Non-Final Office Action on U.S. Appl. No. 1 1/265,669, mailed Feb.
`28, 2006.
`Non-Final Office Action on U.S. Appl. No. 1 1/322,959, mailed Aug.
`8, 2007.
`Non-Final Office Action on U.S. Appl. No. 1 1/788,122, mailed Jun.
`27, 2008.
`Non-Final Office Action on U.S. Appl. No. 1 1/888,546, mailed May
`1, 2008.
`Non-Final Office Action on U.S. Appl. No. 1 1/888,570, mailed May
`28, 2008.
`Norvell, Robin, “Shellcase Debuts Ultra-Thin Miniaturization for
`Optics.” Jul. 8, 2005, 1 page.
`Notice of Allowance for U.S. Appl. No. 1 1/478,242, mailed Dec. 30,
`2009.
`Notice of Allowance for U.S. Appl. No. 1 1/888,546, mailed Dec. 14.
`2009.
`Notice of Allowance of U.S. Appl. No. 1 1/825,382, mailed May 5,
`2010.
`Notice of Allowance on U.S. Appl. No. 1 1/729,132, mailed Oct. 11,
`2011.
`Notice of Allowance on U.S. Appl. No. 1 1/265,669, mailed Oct. 27.
`2006.
`Notice of Allowance on U.S. Appl. No. 11/788, 122, mailed Mar, 9.
`2009.
`Notice of Allowance on U.S. Appl. No. 1 1/788,279, mailed Oct. 28,
`2010.
`Notice of Allowance on U.S. Appl. No. 1 1/810,623, mailed Feb. 2,
`2011.
`Notice of Allowance on U.S. Appl. No. 1 1/888,546, mailed Dec. 2,
`2008.
`Notice of Allowance on U.S. Appl. No. 1 1/888,546, mailed Jun. 3,
`2009.
`
`Notice of Allowance on U.S. Appl. No. 1 1/888,570, mailed Mar. 23,
`2009.
`Notice of Allowance on U.S. Appl. No. 1 1/888,582, mailed Aug. 6,
`2010.
`Notice of Allowance on U.S. Appl. No. 12/496,854, mailed Feb. 16,
`2012.
`Notice of Allowance on U.S. Appl. No. 13/006,351, mailed May 17,
`2011.
`Notice of Allowance on U.S. Appl. No. 13/006,351, mailed Dec. 21.
`2012.
`Notice of Allowance on U.S. Appl. No. 13/100,725, mailed Jul 23,
`2012.
`Notice of Allowance on U.S. Appl. No. 13/345,007, mailed Jan. 14.
`2013.
`Notice of Allowance on U.S. Appl. No. 13/465.229, mailed Aug. 20,
`2012.
`Notice of Allowance on U.S. Appl. No. 13/647,708, mailed Sep. 13,
`2013.
`Notice of Allowance on U.S. Appl. No. 13/681,603, mailed Jul 22,
`2013.
`Notice of Allowance on U.S. Appl. No. 13/786,803, mailed Oct. 10,
`2013.
`Office Action for U.S. Appl. No. 1 1/825,382, mailed Oct. 29, 2009.
`Office Action on U.S. Appl. No. 1 1/788,279, mailed Jan. 21, 2010.
`Office Action on U.S. Appl. No. 1 1/810,623, mailed Aug. 18, 2010.
`Office Action on U.S. Appl. No. 1 1/478,242, mailed Sep. 16, 2009.
`Office Action on U.S. Appl. No. 1 1/729, 132, mailed Feb. 3, 2011.
`Office Action on U.S. Appl. No. 1 1/788,120, mailed Apr. 16, 2010.
`Office Action on U.S. Appl. No. 1 1/788, 120, mailed May 19, 2009.
`Office Action on U.S. Appl. No. 1 1/788, 120, mailed Jul. 30, 2010.
`Office Action on U.S. Appl. No. 1 1/788, 120, mailed Sep. 18, 2009.
`Office Action on U.S. Appl. No. 1 1/788,279, mailed Aug. 4, 2010.
`Office Action on U.S. Appl. No. 1 1/810,623 mailed Feb. 4, 2010.
`Search Report for EP Application 0.57939274, dated Feb. 26, 2010.
`Second Office Action on Chinese Application 200580.032374.0,
`issued Sep. 9, 2010.
`Shogenji et al., “Bimodal fingerprint capturing system based on
`compound-eye imaging module. Applied Optics, vol.43, No. 6, Feb.
`2004, pp. 1355-1359.
`Shogenji et al., “Multispectral imaging using compact compound
`optics.” Optics Express, vol. 12, No. 8, Apr. 2004, pp. 1643-1655.
`Stager et al., “Replicated Micro-Optics for Automotive Applica
`tions.” SPIE European Workshop on Photonics in the Automobile,
`Geneva, 2004. (8 pages).
`Tanida et al., “Compact image capturing system based on compound
`imaging and digital reconstruction.” Proceedings of SPIE, vol. 4455.
`2001, pp. 34-41.
`Tanida, “Color imaging with an integrated compound imaging sys
`tem.” Optics Express, vol. 11, No. 18, Sep. 2003, pp. 2109-2117.
`Third Office Action issued on Chinese Application 200580.032374.0,
`mailed May 24, 2011 (with English translation).
`Volkelet al., “Miniaturization of Imaging Systems.” mstnews, Feb.
`2003, pp. 36-38.
`Volkelet al., “Miniaturized imaging systems.” Elsevier Science B.V.
`Microelectronic Engineering 67-68 (2003), pp. 461-472.
`Wood et al., “Resolution Improvement for Compound Eye Images
`Through Lens Diversity.” IEEE, Signal Processing Society, DSP/SPE
`Workshop, Aug. 2, 2004 (5 pages).
`
`* cited by examiner
`
`
`Ex.1001 / Page 3 of 40Ex.1001 / Page 3 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 1 of 24
`
`US 9.232,158 B2
`
`
`
`
`Ex.1001 / Page 4 of 40Ex.1001 / Page 4 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 2 of 24
`
`US 9.232,158 B2
`
`
`
`
`Ex.1001 / Page 5 of 40Ex.1001 / Page 5 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 3 of 24
`
`US 9.232,158 B2
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
`
`r------------+--------------
`
`JOSS000IJ.
`
`-- - - - - - - - - -• • • • • - - - - - - - - - - - -}
`
`Joss0001)
`23euI
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - A - - - - - - - - - - - - - - - - - - - - - - - -
`
`
`
`
`
`
`
`
`
`
`
`
`
`seese essessee
`- - - - - - - - - - - - - - - - - - -a
`
`- - - - - - - - - -
`
`n----------
`
`<
`es
`We
`N
`
`
`Ex.1001 / Page 6 of 40Ex.1001 / Page 6 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 4 of 24
`
`US 9.232,158 B2
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
`
`i
`
`JOSS000IJ.
`
`IOS$300J?
`238Uup
`
`07%
`
`e
`Wed
`er
`
`r---~--~~~- - - --------------••• • ?
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -/- - - - - - - - - - - - - - - - - - - - - - - - - -
`
`ran or
`
`a- - - - - in a war -
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Ex.1001 / Page 7 of 40Ex.1001 / Page 7 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`U.S. Patent
`
`L
`
`
`
`''i
`
`.=1:So''NAN!twi<'at='
`weetSLTeTSaSLeeeaaaoggennaEES3!mo
`
`vonelso0]
`
`omy
`
`Jo{[oU0d
`
` itt!4“Nn’:Swii3i!=a'!i
`
`
`
`
`
`
`US 9.232,158 B2
`
`N—pocesserca
`
`
`
` er)aoJOU09I98/)
`
`‘5&co2ODonawa~~
`—-SB=»aa
`
`
`Ex.1001 / Page 8 of 40Ex.1001 / Page 8 of 40
`SU
`
`TESLA, INC.TESLA, INC.
`
`
`

`

`U.S. Patent
`
`US 9.232,158 B2
`
`
`
`
`
`was w w - - - w w a saw - - - - - ww.s aws s - - - - - - - - - -
`
`
`Ex.1001 / Page 9 of 40Ex.1001 / Page 9 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 7 of 24
`
`US 9.232,158 B2
`
`Select integration time for each image sensor
`camera channel from range of integration times,
`702
`
`Integrate photosignals from each camera
`channel to form single frame of data,
`704
`
`Correct pixel response of each pixel of frame for
`non-linearity, gain and/or offset as appropriate.
`T06
`
`Combine data of multiple camera channels to form continuous
`monotonic representation of combined pixel response,
`T08
`
`Determine integration time control settings that provide
`optimum dynamic range on subsequent frames of data
`using information of resultant image.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Ex.1001 / Page 10 of 40Ex.1001 / Page 10 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 8 of 24
`
`US 9.232,158 B2
`
`
`
`
`Ex.1001 / Page 11 of 40Ex.1001 / Page 11 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 9 of 24
`
`US 9.232,158 B2
`
`A S
`
`oy
`
`
`
`
`
`s
`yers
`en
`
`C-9
`N
`tw.
`con
`
`
`Ex.1001 / Page 12 of 40Ex.1001 / Page 12 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 10 of 24
`
`US 9.232,158 B2
`
`
`
`
`Ex.1001 / Page 13 of 40Ex.1001 / Page 13 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 11 of 24
`
`US 9.232,158 B2
`
`
`
`WZII I
`
`
`Ex.1001 / Page 14 of 40Ex.1001 / Page 14 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 12 of 24
`
`US 9.232,158 B2
`
`ZI’OIH
`
`
`
`
`Ex.1001 / Page 15 of 40Ex.1001 / Page 15 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan.5, 2016
`
`Sheet 13 of 24
`
`US 9.232,158 B2
`
`
`
`5
`
`i
`
`5.
`
`-
`
`
`Ex.1001 / Page 16 of 40Ex.1001 / Page 16 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 14 of 24
`
`US 9.232,158 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Ex.1001 / Page 17 of 40Ex.1001 / Page 17 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 15 of 24
`
`US 9.232,158 B2
`
`JOSS2001)
`
`W80?I
`
`
`
`?I'0IH
`
`
`Ex.1001 / Page 18 of 40Ex.1001 / Page 18 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 16 of 24
`
`US 9.232,158 B2
`US 9,232,158 B2
`
`
`
`“—VZISI“—VZISI
`
`q91DidVOTDIA
`
`\—VZISI
`
`“—VZISI
`
`91DIdO9TDI
`
`
`
`
`Ex.1001 / Page 19 of 40Ex.1001 / Page 19 of 40
`Ex.1001 / Page 19 of 40
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`
`
`

`

`U.S. Patent
`
`US 9.232,158 B2
`
`OLI'OIH
`
`"ORLI
`
`
`
`
`Ex.1001 / Page 20 of 40Ex.1001 / Page 20 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 18 of 24
`
`US 9.232,158 B2
`
`
`
`
`Ex.1001 / Page 21 of 40Ex.1001 / Page 21 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 19 of 24
`
`US 9.232,158 B2
`
`
`
`Cld
`en
`ww.
`cN
`y
`
`
`Ex.1001 / Page 22 of 40Ex.1001 / Page 22 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 20 of 24
`
`US 9.232,158 B2
`
`- - - - w - or w w - w - r - - - - - - - - - - - - - - - -- as a -s a se - - - - - -
`
`- - - - - - - - - - - - e s we - - - -
`
`
`
`JOSS000IJ.
`
`
`
`---- — • • - ~- - - - - - - - - - - - - - - - - - - - - - - - - - - - - ~ ~ ~ ~ ~ – – – – – – – – – – – –/-º, VOZ"OIH
`
`IOS$300.j?
`
`JOSS000IJ.
`
`JOSS000I?
`
`
`
`
`
`---
`
`000Z
`
`
`Ex.1001 / Page 23 of 40Ex.1001 / Page 23 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 21 of 24
`
`US 9.232,158 B2
`US 9,232,158 B2
`
` d0¢Old
`
`
`Ex.1001 / Page 24 of 40Ex.1001 / Page 24 of 40
`Ex.1001 / Page 24 of 40
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`”141ti11tI‘vt‘tit'!''‘‘I141Vi1i1l'!‘I451i‘ii
`
`)tt1t13tta4fita
`
`[oe eree poe rere ee
`t‘
`''
`
`
`
`Jossed0lgjouuey.)
`
`jouu0, amsodxq
`
`[ONWOD Joao] org
`
`oo] jeusig Sojeny
`
`1807] UUM]OD
`
`we eaeoe fe ee ee rm em
`
`!t’1'1'i1'l'!!i114i
`
`)i1ti
`
`)'
`
`111'i1'’!441tt1't' ti11tt:tu
`
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 22 of 24
`
`US 9.232,158 B2
`US 9,232,158 B2
`
`
`
`Image
`
`Pipeline
`
`suOKoUN,] elamwes pre
`SurUg [aX1q “SULAOpTL
`‘Mo007 ‘snooy OWN
`
`Joysnfpy ode]
`
`FIG.20C
`
`
`Ex.1001 / Page 25 of 40Ex.1001 / Page 25 of 40
`Ex.1001 / Page 25 of 40
`
`TESLA, INC.TESLA, INC.
`TESLA,INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 23 of 24
`
`US 9.232,158 B2
`
`r an - were r - - - - - was as were - w - - w w as a w w w w w - or - - - - - - - - - - - -
`
`
`
`(S? "39)
`90eyrau Indino
`
`i
`
`.
`
`.
`
`. .
`
`.
`
`. - - as we - a - - - we
`
`as w w w
`
`were as se is a ways as we as war se vs see tie way
`
`
`Ex.1001 / Page 26 of 40Ex.1001 / Page 26 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`U.S. Patent
`
`Jan. 5, 2016
`
`Sheet 24 of 24
`
`US 9.232,158 B2
`
`- - - - - - - - - - - - - - - - - - - - - - - - n w w
`
`JOS$2001)
`
`1503 3320||
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Ex.1001 / Page 27 of 40Ex.1001 / Page 27 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`1.
`LARGE DYNAMIC RANGE CAMERAS
`
`US 9,232,158 B2
`
`CROSS-REFERENCE TO RELATED PATENT
`APPLICATIONS
`
`This application is a Continuation of U.S. patent applica
`tion Ser. No. 13/681,603, filed Nov. 20, 2012, which is a
`Continuation of U.S. patent application Ser. No. 13/465.229,
`filed May 7, 2012, (now U.S. Pat. No. 8,334,494), which is a
`Continuation of U.S. patent application Ser. No. 12/496,854,
`filed Jul. 2, 2009 (now U.S. Pat. No. 8,198,574), which is a
`Continuation of U.S. patent application Ser. No. 11/788,122,
`filed Apr. 19, 2007 (now U.S. Pat. No. 7,564,019), which is a
`Continuation-In-Part of U.S. patent application Ser. No.
`11/212.803, filed Aug. 25, 2005, which claims priority to U.S.
`Provisional Patent Application No. 60/695,946 filed on Jul. 1,
`2005 and to U.S. Provisional Patent Application No. 60/604,
`854 filed on Aug. 25, 2004; U.S. patent application Ser. No.
`11/788,122 also claims priority to U.S. Provisional Patent
`App. No. 60/795,882 filed Apr. 28, 2006. Each of the above
`referenced patent applications is incorporated herein by ref
`erence in its entirety.
`
`TECHNICAL FIELD
`
`The following disclosure relates generally to optical
`devices and more particularly to expanding the dynamic
`exposure range in digital cameras.
`
`BACKGROUND
`
`Image sensors can be realized with semiconductors based
`on their capability to convert locally impinging light energy
`into a proportional amount of electronic charge. This elec
`tronic charge "Q is often referred to as photo-charge, and can
`be integrated within the pixel on a storage device Such as a
`reverse biased diode or as a pre-charged metal-oxide-semi
`conductor capacitance. The finite charge storage capacitance
`within each pixel limits the amount of integrated photo
`charge. Dynamic range is measured as the ratio of the maxi
`mum photo-charge that can be meaningfully integrated in a
`pixel of the imager to the pixel noise level.
`Intrascene dynamic range refers to the range of incident
`light that can be accommodated by an image sensorina single
`frame of pixel data. Examples of high dynamic scenes range
`scenes include an indoor room with a window view of the
`outdoors, an outdoor scene with mixed shadows and bright
`Sunshine, and evening or night scenes combining artificial
`lighting and shadows. In a typical charge coupled device
`(CCD) or CMOS active pixel sensor (APS), the available
`dynamic range is in a range of 1,000:1 to about 4,000:1.
`Unfortunately, many outdoor and indoor scenes with highly
`varying illumination have a dynamic range significantly
`greater than 4,000:1. Image sensors with intrascene dynamic
`range significantly greater than 4,000:1 are required to meet
`many imaging requirements.
`The dynamic range of an image sensor can be increased by
`using multiple exposure times and/or integration times. For
`example, U.S. Pat. No. 4,647,975 describes a method based
`on the acquisition of two or more images, each having an
`exposure time. Once numerous images have been taken at
`different exposure times, the images have to be fused or
`merged to form one single piece of pixel information having
`a wide dynamic range. U.S. Pat. Nos. 4,647.975, 5,168,532,
`and 5,671,013 disclose the use of a selection rule to combine
`information from the most Suitable of the multiple images.
`The merged pixel information or value is then multiplied by a
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`suitable factor that corrects for the respective exposure times.
`This method however exhibits undesirable temporal aliasing
`if the scene or camera is moving because the two or more
`images having different exposure times are captured using the
`same image sensor and thus are not captured concurrently.
`Despite improvements in Solid-state image sensor and
`digital camera technology, the light signal or brightness range
`of scenes often exceeds the dynamic range of the sensor. For
`this reason, numerous methods have been described in the art
`of image sensors to extend the dynamic or signal range. A
`summary of some methods is presented in: “Wide dynamic
`range sensors'. Optical Engineering, Vol. 38, No. 10, pp.
`1650-1660, October 1999. Methods to provide wide dynamic
`range imaging capability with a single image sensor include:
`(a) logarithmic or compressed response photo-detection; (b)
`multiple integration and charge storage capability within
`each pixel; (c) frequency based sensors, where the sensor
`output is converted to pulse frequency; (d) local integration
`time control, where different areas within the sensor can have
`different exposure times; (e) signal charge versus integration
`time rate (signal slope) measurement, (f) analog to digital
`conversion per pixel; and (g) autonomous pixel control.
`These methods require complex pixel circuitry and are diffi
`cult to implement in Small pixel areas without taking up area
`required for the photo-detection mechanism (such as a pho
`todiode).
`Typical scenes imaged by digital cameras have light levels
`that span a range including low light (1-100 lux), moderate
`light (100-1000 lux), and bright light (1000-1,000,000 lux)
`under outdoor conditions. To accommodate lighting changes
`from scene to scene (the interscene dynamic range) an elec
`tronic shutter is used to change the integration time of all
`pixels in an array from frame to frame. To cover a single scene
`that might involve indoor lighting (100 lux) and outdoor
`lighting (100,000 lux), the required intrascene dynamic range
`is approximately 10,000:1, corresponding to 80 dB (14-bits).
`This exceeds the dynamic range of a single image sensor
`using a single integration time (typically 3,100:1 correspond
`ing to 70 dB (12-bits)). Therefore, there is a need for a digital
`camera in which the effective single-frame dynamic exposure
`range is expanded.
`
`INCORPORATION BY REFERENCE
`
`Each patent, patent application, and/or publication men
`tioned in this specification is hereinincorporated by reference
`in its entirety to the same extent as if each individual patent,
`patent application, and/or publication was specifically and
`individually indicated to be incorporated by reference. dia
`gram of a digital camera that includes multiple channels,
`under an embodiment.
`FIG. 1 is a block diagram of a conventional digital camera.
`FIG. 2 is a block diagram of a digital camera that includes
`multiple channels, under an embodiment.
`FIG. 3 is a block diagram of a digital camera Subsystem,
`under an embodiment.
`FIG. 4 is a digital camera Subsystem, under an alternative
`embodiment.
`FIG. 5 is a digital camera subsystem, under another alter
`native embodiment.
`FIG. 6 is a digital camera subsystem, under still another
`alternative embodiment.
`FIG. 7 is a block diagram of a method for forming images
`having a large dynamic range, under an embodiment.
`FIG. 8 is a block diagram of a digital camera, under an
`embodiment.
`
`
`Ex.1001 / Page 28 of 40Ex.1001 / Page 28 of 40
`
`TESLA, INC.TESLA, INC.
`
`

`

`US 9,232,158 B2
`
`10
`
`15
`
`25
`
`30
`
`35
`
`3
`FIG. 9 is an exploded view of a digital camera subsystem,
`under an embodiment.
`FIG. 10 is a block diagram of a digital camera having a
`three array/lens configuration, under an embodiment.
`FIG. 11 is a block diagram of a digital camera Subsystem
`that employs separate arrays on one image sensor, under an
`embodiment.
`FIG. 12 is a block diagram of arrays, each of which receives
`a respective color as passed by a respective lens, under an
`embodiment.
`FIG. 13 is a block diagram of processing circuitry of a
`digital camera Subsystem, under an embodiment.
`FIG. 14 is a block diagram of signal processing circuitry,
`under an embodiment.
`FIG. 15 is an exploded perspective view of a digital cam
`era, under an embodiment.
`FIGS. 16A-16D are schematic exploded representations of
`one embodiment of an optics portion, under an embodiment.
`FIGS. 17A-17C are schematic representations of a sensor
`array, under an embodiment.
`FIG. 18 is a schematic cross-sectional view of a digital
`camera apparatus, under an embodiment.
`FIG. 19 is a schematic perspective view of a digital camera
`apparatus having one or more optics portions with the capa
`bility to provide color separation, under an embodiment.
`FIG. 20A is a block diagram of a processor of a digital
`camera Subsystem, under an embodiment.
`FIG. 20B is a block diagram of a channel processor of a
`digital camera Subsystem, under an embodiment.
`FIG. 20O is a block diagram of an image pipeline of a
`digital camera Subsystem, under an embodiment.
`FIG. 20D is a block diagram of an image post processor of
`a digital camera Subsystem, under an embodiment.
`FIG. 21 is a block diagram of digital camera system,
`including system control components, under an embodiment.
`
`DETAILED DESCRIPTION
`
`Digital camera systems and methods are described below
`that provide an expanded effective single-frame dynamic
`exposure range. The digital camera systems and methods,
`collectively referred to herein as digital camera systems, gen
`erally include two or more camera channels. Each channel
`includes an optics component and an image sensor having an
`active area including multiple picture elements (pixels). Each
`sensor is controlled under or with an independent signal inte
`gration time as described in detail below. Image capture by
`the camera is performed using the multiple camera channels,
`and each camera channel is controlled during a frame under
`an independent integration time. The integration time for
`each channel can be automatically controlled and/or con
`trolled in response to a user input. A wide range of integration
`times are available for use in an embodiment. The images
`obtained in each channel under the different integration times
`are obtained simultaneously or nearly simultaneously, so
`undesirable temporal aliasing from moving scenes or camera
`motion is minimized.
`The digital camera systems, through the use of multiple
`integration times, enable formation of an image having a wide
`dynamic range by combining the data from all camera chan
`nels to form a composite single frame. The wide dynamic
`range of an embodiment is generally greater that the dynamic
`range obtained by a conventional digital camera with a single
`image sensor channel and one integration time.
`Conventional digital cameras may have a maximum photo
`signal storage capacity that limits the dynamic range of the
`particular system. The photo-signal charge is stored on a
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`capacitor within the pixel area. The charge handling capacity
`is limited by the maximum Voltage Swing in the integrated
`circuitry and the storage capacitance within the pixel. The
`amount of integrated photo-charge is directly related to the
`time the image sensor collects and integrates signal from the
`scene. This is known as integration time. A long integration
`time is desired for weak signals since more photo-charge is
`integrated within the pixel and the signal-to-noise of the
`digital camera is improved. Once a maximum charge capacity
`is reached, the sensor no longer senses image brightness
`resulting in data loss. Therefore, the use of a single integration
`time for the entire field of view creates an imaging dilemma in
`conventional cameras. The digital camera integration time
`can be set to image low light levels and Saturate bright signals
`or image highlight levels and not detect low light levels (since
`the integrated photo-charge from low light levels is below the
`signal-to-noise of the sensor).
`The digital cameras described herein overcome this
`dynamic range limitation through the use of multiple camera
`channels, including multiple optics and image sensors on a
`single integrated circuit (IC) or semiconductor Substrate. The
`multiple camera channels are configured to image the same
`field of view simultaneously, and each operates indepen
`dently under a different integration time. The digital camera
`can include, for example, a 3 by 3 assembly of image sensors,
`perhaps three sensor of each color (e.g., red (R), green (G),
`and blue (B)) and the integration time of the sensors associ
`ated with each color can be varied, for example, each color
`can have t