APPL-1007/Page 1 of 49
`Apple Inc. v. Corephotonics
`
`

`

`US 7,859,588 B2
`
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`FOREIGN PATENT DOCUMENTS
`
`2006/0193509 A1
`2008/0013941 A1 *
`2008/0218612 A1 *
`
`2008/0218613 A1 *
`
`8/2006 Criminisi et a1.
`1/2008 Daley ......................... 396/121
`9/2008 Border et a1.
`..... 348/262
`
`............... 348/262
`
`9/2008 Janson et a1.
`
`JP
`$8
`
`2005045511
`2002}£$3§§
`
`2/2005
`$88;
`
`* cited by examiner
`
`APPL-1007 / Page 2 of 49
`
`APPL-1007 / Page 2 of 49
`
`

`

`U.S. Patent
`
`c
`
`US 7,859,588 B2
`
`E8No
`
`mvmmmHmofimmkz.
`
`PwOIFmOI
`
`
`
`8,E052om<o
`
`
`
`7”mgm<>02mm>KO_>_m_2
`
`
`
`mom_>_<mmDK<Om0<mmwlr2_
`
`9Ta3%?fi8Hmmae.mmmH
`
`
` _Emacs._fimmwwwwmwfiE53451misjmo_IIIIIlllllllllllllL
`
`
`omowe
`
`mowmmOONE
`
`m0<_>=mm
`
`
`
`SEEDDZOOMm_>_OON
`
`0200mm
`
`
`
`mKOFOEmDOOu
`
`DZ<_>_OONmm
`
`
`
`
`
`Emu—mamm0<_>=mZm._
`
`F.OE
`
`004$0400
`
`m0<_>=ONmum:NVawv
`
`
`
`>m0_>_m=>_mOmzmm
`
`Jog—.200
`
`
`
`DZ<m0www00mm
`
`
`
`mOHémszOZ=>=._.
`
`
`
`mewOme0._.D<
`
`wKOHOmhmD
`
`
`
`wmw>_mn_mMOHOE@300".
`
`x0040Qz<EOON.
`
`pm
`
`0v
`
`APPL-1007 / Page 3 of 49
`
`APPL-1007 / Page 3 of 49
`
`
`
`
`
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 2 of 29
`
`US 7,859,588 B2
`
`42a
`
`10A
`
`423
`
`10A
`
`FIG.23
`
`FIG.2A
`
`APPL-1007 / Page 4 of 49
`
`APPL-1007 / Page 4 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 3 of 29
`
`US 7,859,588 B2
`
`ZOOM POSITION SET TO
`DEFAULT POSITION WHEN
`CAMERA IS POWERED ON
`
`USER INPUT
`ZOOM POSITION
`
`100
`
`1 01
`
`
` ZOOM
`
`
`POSITION
`
`>X ?
`
`
`
`
`SET FIRST IMAGE CAPTURE
`SET SECOND IMAGE CAPTURE
`STAGE TO CAPTURE IMAGES
`STAGE TO CAPTURE IMAGES
`
`
`
`& SECOND IMAGE CAPTURE
`& FIRST IMAGE CAPTURE
`
`
`STAGE FOR AUTO FOCUS
`STAGE FOR AUTO FOCUS
`
`
`
`
`CAPTURE AUTOFOCUS
`IMAGES WITH FIRST
`IMAGE CAPTURE STAGE
`
`CAPTURE AUTOFOCUS
`IMAGES WITH SECOND
`IMAGE CAPTURE STAGE
`
`106
`
`1 26
`
`
`
`
`
`
`
`
`FOCUS FIRST IMAGE
`CAPTURE STAGE
`
`108
`
`128
`
`FOCUS SECOND IMAGE
`CAPTURE STAGE
`
`-
`
`1 10
`
`130
`
`
`
`CAPTURE PREVIEW IMAGES
`CAPTURE PREVIEW IMAGES
`WITH FIRST IMAGE CAPTURE
`WITH SECOND IMAGE CAPTURE
`
`
`STAGE AND DISPLAY
`STAGE AND DISPLAY
`
`
`
`
`
`ZOOM
`ZOOM
`
`BUTTON
`BUTTON
`
`PRESSED
`PRESSED
`
`
`'7
`?
`
`NO
`
`114
`
`
`
`CAPTURE
`BUTTON
`
`
`
`
`?
`
`YES
`
` PRESSED
`
`
`134
`
`CAPTURE
`BUTTON
`PRESSED
`?
`
`
`
`YES
`
`CAPTURE DIGITAL STILL
`IMAGE WITH FIRST
`IMAGE CAPTURE STAGE
`
`1 1 6
`
`1 36
`
`V
`
`CAPTURE DIGITAL STILL
`IMAGE WITH SECOND
`IMAGE CAPTURE STAGE
`
`APPL-1007 / Page 5 of 49
`
`APPL-1007 / Page 5 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 4 of 29
`
`US 7,859,588 B2
`
`wow
`
`wow
`
`wow
`
`wow
`
`CNN
`
`
`
`
`
`O>>._.m=.:.zmw>>._.m_m.EquO._mx_n_m5.>uEsz
`
`
`
`
`
`
`
`mo<s=m=.:.EOEmega.m300u05<mI._.
`
`
`
`
`
`swan—010w_20:60;EOONmm>>o._mi2_mo<._.m
`
`
`
`Oz_ozon_mm_mmoo._.<I._.OwQuiz/”fin:oz<
`
`
`
`mmo<s=wDoOuOS<OEm=.:.z_mumszmu
`
`
`
`wixEu_Ommmsfizm__>_<mmi2.4%
`
`
`
`m300u0.5<ow._n=>_<wn5Qz<omnEOmomE.
`
`
`
`m300m9b<m=.:.Okogjmmmoow_m0<_>__
`
`
`
`O._.mO/fiwmo<s=mmIHOm1...20mmmo<s=
`
`
`
`
`
`
`
`
`
`ZOEmOQEOONmkomdwmmw:
`
`
`
`Io_I>>mszEmEgzo_._._won__>_OON
`
`
`
`
`
`
`
`mmPE<omo;owm:m_mo<._.mmos):
`
`
`
`
`mIHohHmmmod._.z_on_mIHo._.QMEOON
`
`
`
`m_mNSE<omo;0mm:._.02mofibmo<s=mE.
`
`
`
`
`
`zoEmon_EOONomhomjmmmmm:
`
`
`
`
`
`ozEzOmwmmmoomo;mmo<_>=maoomofiz
`
`"*0szPmoa
`
`
`
`
`
`mmo<._.wm0<s=I._.Om2.$szmDOOu
`
`20m”.zo._..5mmmDE<omemammum:
`
`wDQOmOFD<m5.mm: mmo<s=m_.:.
`._.zm_>_m>o_>_mIHszmmEoO._.FwwnEo._wx_n_
`
`
`om_...__.rzmo_m=.:.oz<m>m30zo_._.<mm_._<o
`
`Hmmmo...omommzmzmgmango;mI._.mo
`
`
`mmozimmuzé
`
`
`
`mo<._.mmm:._.n_<omo<s=m=.:.mDQOm
`
`
`moszOmwm300u05<m._.<:._z_O._.5O._.om
`
`mzo_._._mon_._<oon_mm_n_>IEur:O._.Emmm<
`
`
`
`zo._.._.:mmmDE<om_.:.mmImDQmum:mE.szS
`
`
`
`
`
`
`
`
`
`ommspmfiuw.mos):4<z_u_MT;.NmOHFmEOE
`
`V.GE
`
`
`
`
`
`9.60;5%mo;zOEwOasz_§mm._.mo
`
`
`
`m5.9.9302w_mZm:maoou.m_.:.
`
`mwofibmos):EbmI._._>>
`
`
`
`
`
`ommDE<omm<wmo<s=m300m05<
`
`
`
`
`
`
`
`omm
`
`NmN
`
`va
`
`mmm
`
`mmm
`
`oom
`
`APPL-1007 / Page 6 of 49
`
`APPL-1007 / Page 6 of 49
`
`
`
`
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 5 of 29
`
`US 7,859,588 B2
`
`
`
`
`
`m=,:.stEmEoo._.._.zms_._.wD_.0<wzmgwDOOu
`
`
`
`
`
`z_95mewzm._mace“.Ext.".0._.zm_s_w>o_2
`
`
`
`
`
`mDoOm5mmmoaoommOHmo<._.mmmDE/‘om5.
`
`
`
`m@<._.mmm3E<oHt.2_wzmgwDQOmwIHm>o_>_owN
`
`
`
`
`
`
`
`
`
`zoEwon_mDOOm5mm>>mzm5.9.
`
`m.OE
`
`
`
`
`
`
`
`mmDE<0mo;0mm:02mmmo<._.wmm3E<0
`
`m@<._.mmsz<omIHIt>>
`
`mag):mE.It;900;xomIo>._m:o:z_._.zoowNN
`
`
`
`Omn=>wmaE/«O>._mDO:z_._.zoomNN
`
`
`
`EOON zDEmOm
`
`
`
`mo<§_Io=._>>mstEmEDzDEmOm
`
`
`mo<._.mmmnE<omI._.w_mo<._.mmmam<o
`
`EOONmpoflmmmum:
`
`
`
`
`
`mmahnfiamoi):mIHzOwDQOmHmmmmoaoommOH
`
`omommzHzmékwsaxxme:wsoommTz.>u=._.zmo_NwN
`
`
`
`
`
`
`EDIE:m_>=._o.21..m=.:.>mwmeb
`
`
`$5.566mo<s=I._.Om94095.22
`
`
`
`
`
`mmDE<0mom0mm:ozmm._.Ozmo<Hm
`
`
`
`omifizmgmE.oz<m>m20zo_._.<mm_._<o
`
`
`
`50152$50._.__I..m300u05<mE.mm:va
`
`mmo...520E
`
`
`
`
`
`zo._..5mmmDE<om=.:.mmeDmmum:
`
`
`
`_.QOI._.m_>_mzzo._.__I._m5.025:
`
`
`
`m_mm3E<omo”.omm:H02wo<._.mm0<s=mIH
`
`
`
`
`
`MIR9.HmmmodHz_on_mI._.O._.omEOON
`
`
`
`
`
`zo_._._mon_EOONomhomdwmmw:
`
`
`
`
`
`20m“.zO._..Sm_.m_m:._.n_<owMImDmmmwa
`
`
`
`
`
`moszmemaoon55<m._.<_.:z_o._.5O._.ow
`
`omN
`
`NmN
`
`va
`
`wmm
`
`NNN
`
`wnm
`
`APPL-1007 / Page 7 of 49
`
`APPL-1007 / Page 7 of 49
`
`
`
`
`
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 6 of 29
`
`US 7,859,588 B2
`
`
`
`
`
`
`
`CAPTURE A SERIES OF IMAGE SETS WITH OBJECTS
`AT KNOWN DISTANCES WITH THE SHORTER
`FOCAL LENGTH FIRST IMAGE CAPTURE STAGE AND
`THE LONGER FOCAL LENGTH SECOND IMAGE CAPTURE
`STAGE AND A SERIES OF FOCUS LENS POSITIONS
`
`
`
`
`
`
`
`300
`
`302
`
`304
`
`
`
`THE AUTOFOCUS IMAGE FROM THE LOWER
`FOCAL LENGTH FIRST IMAGE STAGE IS CROPPED AND
`UPSAMPLED SO THAT CORRESPONDING FEATURES
`IN THE TWO AUTOFOCUS IMAGES SPAN
`
` THE SAME NUMBER OF PIXELS
`
`
`
`
`
`
`
`CORRELATE THE IMAGES FROM THE SECOND IMAGE
`CAPTURE STAGE TO CORRESPONDING PORTIONS OF
`
`
`THE IMAGES FROM THE CROPPED AND UPSAMPLED
`
`
`IMAGE FROM THE FIRST IMAGE CAPTURE STAGE TO
`
`
`DETERMINE THE PIXEL OFFSET BETWEEN
`THE IMAGES IN EACH IMAGE SET
`
`
`
`
`
`STORE THE DATA OF PIXEL OFFSET BETWEEN
`IMAGES IN EACH IMAGE SET VERSUS KNOWN
`DISTANCE TO OBJECTS AS AN AUTOFOCUS
`
`RANGEFINDER CALIBRATION CURVE
`
`
`
`
`306
`
`FIG. 6
`
`APPL-1007 / Page 8 of 49
`
`APPL-1007 / Page 8 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 7 of 29
`
`US 7,859,588 B2
`
`
`
`
`
`
`
`
`
`CAPTURE A SERIES OF IMAGE SETS WITH OBJECTS
`AT KNOWN DISTANCES WITH THE FIRST IMAGE
`CAPTURE STAGE AND THE SECOND IMAGE CAPTURE
`
`STAGE WHEREIN AUTOFOCUS IS DONE BY THE
`
`"HILL CLIMB METHOD" FOR EACH IMAGE
`
`
` COMPARE THE FOCUS LENS POSITIONS FOR THE TWO
`
`
`
`
`IMAGE CAPTURE STAGES VERSUS THE DISTANCE TO
`THE FOCUSED OBJECTS IN THE IMAGE SETS
`
`
`
`
`
`STORE THE DATA OF FOCUS LENS POSITIONS OF
`
`
`THE FIRST IMAGE CAPTURE STAGE VERSUS THE FOCUS
`LENS POSITIONS OF THE SECOND IMAGE CAPTURE
`
`
`STAGE FOR THE SAME DISTANCE TO FOCUSED
`OBJECTS IN THE IMAGES AS AN AUTOFOCUS
`
`
`"HILL CLIMB METHOD" CALIBRATION CURVE
`
`
`
`400
`
`402
`
`404
`
`FIG. 7
`
`APPL-1007 / Page 9 of 49
`
`APPL-1007 / Page 9 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 8 of 29
`
`US 7,859,588 B2
`
`ZOOM POSITION SET TO DEFAULT
`POSITION WHEN CAMERA IS POWERED ON
`
`100
`
`USER INPUT ZOOM POSITION
`
`101
`
`ZOOM
`
`POSITION
`
`
`>X ?
`
`
`
`
`SET FIRST IMAGE CAPTURE
`
`
`SET SECOND IMAGE CAPTURE
`MODULE TO CAPTURE IMAGES
`MODULE TO CAPTURE IMAGES
`
`
`
`& SECOND IMAGE CAPTURE
`& FIRST IMAGE CAPTURE
`
`
`
`MODULE FOR AUTO FOCUS
`MODULE FOR AUTO FOCUS
`
`
`
`
`CAPTURE AUTOFOCUS IMAGES
`
`CAPTURE AUTOFOCUS IMAGES
`
`
`
`
`
`
`
`AUTOFOCUS FIRST IMAGE
`CAPTURE MODULE
`
`AUTOFOCUS SECOND IMAGE
`CAPTURE MODULE
`
`
`CAPTURE PREVIEW IMAGES
`
`
`CAPTURE PREVIEW IMAGES
`WITH FIRST IMAGE CAPTURE
`WITH SECOND IMAGE CAPTURE
`
`
`MODULE AND DISPLAY
`MODULE AND DISPLAY
`
`
`
`
`
`I
`
`
`ZOOM BUTTON
`
`PRESSED ?
`
`
`ZOOM BUTTON El
`
`PRESSED ?
`
`
`
`CAPTURE BUTTO
`CAPTURE BUTTON
`
`
`PRESSED ?
`PRESSED ?
`
`
`YES
`
`CAPTURE VIDEO IMAGE WITH
`FIRST IMAGE CAPTURE MODULE
`
`CAPTURE VIDEO IMAGES WITH I
`SECOND IMAGE CAPTURE MODULE
`
`
`
`
`
`
`
`
`CHECK IMAGE FOR
`FOCUS QUALITY
`
`NEED TO
`REFOCUS.7
`
`CHECK IMAGE FOR
`FOCUS QUALITY
`
`NEED TO
`REFOCUS ?
`
`FIG. 8
`
`APPL—1007 / Page 10 of 49
`
`APPL-1007 / Page 10 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 9 of 29
`
`US 7,859,588 B2
`
`
`
`
`
`wo/fimmmDE<omo<_>__1.525.200“.mmozog
`
`
`
`
`
`vvv
`
`mg
`
`
`
`mo<s=mDOOmOHD<ozoomwmI._.mbjmmmoo
`
`
`
`
`
`
`
`
`
`OHmo<_>__egg—>223oz<omnEomomI._.I._._>>
`
`
`
`mE.zmm>>._.mmHmquOEven.mI._.mz=2mm:.mo
`
`
`
`
`
`
`
`mmo<_>__mI._.nowzo_._.mon_HmemuEEmomwmo<s=
`
`
`
`omomwzzO_._.ommmoomDQOumI._.szmEmo
`
`
`
`
`
`m300u05<mI._.oz<EmnEothEOE
`
`
`
`
`
`m>m30zo:.<mmj<ommozimozg
`
`m
`
`GE
`
`APPL—1007 / Page 11 of 49
`
`
`
`m._mx_n_”.0mmmsSzms_<m
`
`
`
`N:m0<._.wmos):mE.EOEmo<_>__mDoOmOS<mI._.
`
`
`
`
`
`oz<ommmomo91.525.200...mm.>>o._m1»2_
`
`
`
`wwmaquuwzazoammmmooEst.Om05%?me
`
`
`
`mI._.z<n_mmmo<§mnoon55<O>>._.wE.z_
`
`
`
`
`
`d)
`
`v<
`
`1-
`
`MIHI._._>>mo<_>__w300m0.5<0200mm<mask—<0
`
`
`
`
`
`
`
`ovvmI._.I._._>>mo<_>__w300m0.5<kmmE<mmDE<o
`
`
`
`
`
`
`
`mo<._.mmmEnZomo<s=1.525.200”.mm>>o._
`
`
`
`
`
`APPL-1007 / Page 11 of 49
`
`
`
`
`
`

`

`U.S. Patent
`
`Dec. 28
`
`9
`
`2010
`
`Sheet 10 of 29
`
`US 7,859
`
`9
`
`588 B2
`
`
`
`
`
`thmo»mzoEozoo960;m5.mm<n=200
`
`
`
`
`
`mo<s=murkoz<ME.".0mmOIHOHmo<s=0200mm
`
`m.
`
`mDQOm
`
`mzoEQzOo
`
`owoz<Io
`
`2..6E
`
`
`
`
`
`
`
`
`mow0200mmm5.I._._>>mo<_>__ozoomm<mmDE/‘o00vmo<s=HmmEmI._.I._._>>m0<_>__51E4.mmDE<o
` 0200mmmE.
`
`
`
`
`
`
`mDoomOS<mI._.z_mmmI>>m0<._.mmmDE<Omo<s=m_m300m05<mI._.z_mmmI>>mo<._.wmmDE<o
`
`
`
`I._._>>mo<s=mmEbz<mmDE/xoNovHmmEm5.EOEm0<2_mI._.mo_>omn_
`
`
`
`.boIEEmsfio.21..m5.>mmzoom_50:52m_>=._o.21..m:.:.Emon
`
`
`
`
`
`goon—0.22mIHz_mmm__._>>m0<._.wmmDE<omos):
`
`
`
`
`
`...QOI._.m_>_m_>__.._o.21..m1._.Eszow_
`
`><._n_m_omI._.zom0<s=>>m_>m_m_m<m<mo<._.m
`
`mm3E<omos):
`
`
`
`
`
`5mmmzhmommzoEozoo9.60;m:...moz<Io
`
`
`
`moz<IomE.zoomm<mmofibmmDE<omos):
`
`
`
`
`
`0200mm.mIHmo;mzoEozoowDQOum5.m0
`
`
`
`
`
`maoOmOH3<mI._.oz<m@<._.mmmDE<omo<s=
`
`
`
`m>m30zo_.r<mm_._<o__oOI._.m_>_92:0.21..
`
`
`
`
`
`
`
`APPL—1007 / Page 12 of 49
`
`APPL-1007 / Page 12 of 49
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 11 0f 29
`
`US 7
`
`9
`
`859,588 B2
`
`wvv
`
`m5.It;m0<s=m300m05<0200mm4.mmDE<o
`
`
`
`
`03».Ht.I._._>>mo<s=m300¢05<5%;<mmDE<o
`
`
`
`
`
`m0<._.mmmBnZomo<s=Ihozmj.200;mmIQI
`
`
`
`
`
`
`
`
`
`mo<._.mmmDE<omegs1.525.208mm>>o._
`
`
`
`
`
`owv
`
`va
`
`vwv
`
`
`
`mE.zmmEmmEquo.EXEm1...szmmEo
`
`
`
`
`
`
`
`
`
`mo<2_m300m0.5<0200mmmIHmbjmmmoo
`
`
`
`
`
`O._.moi):om._n_s_<mn_3oz<omniomomE.I._._>>
`
`
`
`
`
`N:mo<._.mmo<_>__mi205wo<s=m300u05<m_.:.
`
`
`
`
`
`oz<omamomom_zo_._._mon_EOONmm>>o._m52_
`
`
`
`mwm3._.<m”_ozEzommmmmooZIPomom._n_§<wn5
`
`
`
`z<mmwmo<s=wDOOmoPD<93.Ht.2_
`
`
`
`wde“.0«$9232m_>_<wm=.:.
`
`
`
`
`
`mmo<s=m1._.“.0mzo_._.mon_Fzmmmniamo.._89%:
`
`
`
`
`
`$025.909.mhwwfio.EXEmTE..Em>zoo
`
`
`
`m5.026:mo_>mommDE<omo<s=mI._.EOE
`
`
`
`
`
`
`
`m>maozo:.<mm_._<ommozEmOz/Ew300m05<
`
`
`
`
`
`
`
`$025.20w_.:.oz_>>o_.._wn22<monoomm
`
`
`
`wmo<s=mI._.mowzQEOmHzmmmurzo9.
`
`:..OE
`
`APPL—1007 / Page 13 of 49
`
`APPL-1007 / Page 13 of 49
`
`
`
`
`
`
`

`

`U.S. Patent
`
`2
`
`S
`
`2
`
`US 7,859,588 B2
`
`wewe
`
`#0
`
`om<oW.53551@0sz
`m92055%?a$0.52
`
`vmmm
`
`50:50:
`
`
`
`on.moEmEZ
`
`0E052m82%
`
`mOmWMOOMQ
`
`Mlem>m02m2nfimmmm<>>2mimmmmm<n__>_oo
`9“m5:de.r.Mfipdmw.L.
`
`Naom~N.......K.-:::L“Emacs._.mOmmmoomn:
`
`Kmmam
`
`N“.0E
`
`><._n_m_n_
`
`0041040
`m0<§_
`02
`
`mJOmHZOO
`
`mmeNV
`
`E2
`
`SEED
`
`mwnEDm
`
`>mO_>_m_>_
`
`0200mm
`
`m0<_>__
`
`mszmw
`
`AOWFZOO
`
`
`
`DZ<mOmwMOON—a
`
`
`
`mOFémszOZ=>__._.
`
`
`
`DZ<_>_OON
`
`
`
`map—.02mDOOm
`
`
`
`mmeOaXmO._.D<
`
`mmOFomHmo
`
`APPL—1007 / Page 14 of 49
`
`EVN
`
`$052
`
`mace”.om
`
`mNr
`
`NNNFEm5
`
`
`
`:2“flawmum”;_meE
`
`meE
`
`mZm:
`
`APPL-1007 / Page 14 of 49
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`iDec.28,2010
`
`Sheet130f29
`
`US 7,859,588 B2
`
`ugm<>02mm
`
`>m02w§
`
`@Dm<O
`
`mm
`
`.5
`
`mm
`
`mm
`
`vm
`
`>m02m2
`
`Dm<0
`
`m0<mEMHZ_
`
`§<m
`
`>m02m2
`
`maO
`
`wm<m200
`
`mm<>>_>_m_n_
`
`>m02m2
`
`we
`
`we
`
`No
`
`FmOI
`
`Om
`
`FwOI
`
`m0<mKMFZ_
`
`vm
`
`Nm
`
`om
`
`"m
`
`EMDOE_
`
`
`
`Sojmofilvfl$33ij_
`
`llllllllllL
`
`.\iowmmoomm.
`
`om
`
`m0<§_mXDEmOmzwwDZwzm;
`
`
`ll...__g.028%208
`
`mm
`
`IL.El
`
`Eu"
`l.—
`I...
`
`mm?
`
`
`
`m0<§_
`
`mszmw
`
`Emmi
`
`DwXE
`
`mZMJ
`
`wDOOu
`
`
`
`mmemDmOhOE
`
`
`
`x0040mDOOmmm
`
`w:
`
`0200mm
`
`mOwwwOOmm
`
`2<mo
`
`
`
`mmemDmmOFOEwDOOu
`
`
`
`x0040DZ<_>_OONcm
`
`meE.VN
`
`\Eozms.mam<
`mmmquDm¥fi
`Emug.“
`
`Dmfk
`
`EOON
`
`x030oz<200Mon
`
`mos):
`
`
`
`
`
`NV08«.300mmm>_mo$652goo;
`
`
`
`><4QQDAOEFZOO0v
`
`mmw:oz<mOmmmooE
`
`
`
`
`
`
`
`90sz00mobémzmo02:2:mmwwwknfikmwg
`
`
`
`Mr0~n~1min.
`
`wv
`
`APPL—1007 / Page 15 of 49
`
`APPL-1007 / Page 15 of 49
`
`
`
`
`
`
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 14 of 29
`
`US 7,859,588 B2
`
`ZOOM POSITION SET TO
`DEFAULT POSITION WHEN
`CAMERA IS POWERED ON
`
`500
`
`
`ZOOM
`POSITION
`
`
`>X ?
`
`
`
`
`
`SET FIRST IMAGE CAPTURE
`SET SECOND IMAGE CAPTURE
`MODULE TO CAPTURE IMAGES
`MODULE TO CAPTURE IMAGES
`
`
`
`& SECOND IMAGE CAPTURE
`& FIRST IMAGE CAPTURE
`
`
`
`MODULE FOR AUTO FOCUS
`MODULE FOR AUTO-FOCUS
`
`
`
`
`
`
`
`CAPTURE AND DISPLAY
`CAPTURE AND DISPLAY
`PREVIEW IMAGES WHILE
`PREVIEW IMAGES WHILE
`
`
`
`PERFORMING CONTINUOUS
`PERFORMING CONTINUOUS
`
`
`
`AUTO-FOCUS USING SECOND
`AUTOFOCUS USING FIRST
`
`
`
`IMAGE CAPTURE MODULE
`IMAGE CAPTURE MODULE
`
`
`
`
`
`
`
`ZOOM
`ZOOM
`BUTTON
`BUTTON
`
`
`PRESSED
`PRESSED
`
`
`9
`7
`
`
`
`NO
`
`NO
`
`
`WHEN SHUTTER BUTTON IS
`
`PRESSED, CAPTURE A PRIMARY
`STILL IMAGE USING FIRST
`IMAGE CAPTURE MODULE SET
`TO PRIMARY FOCUS POSITION
`
`
`
`
`
`
`
`510
`
`530
`
`
`WHEN SHUTTER BUTTON IS
`
`PRESSED, CAPTURE A PRIMARY
`STILL IMAGE USING SECOND
`
`IMAGE CAPTURE MODULE SET
`TO PRIMARY FOCUS POSITION
`
`
`
`
`
`
`
`
`
`
`
`
`
`CAPTURE A SECONDARY
`STILL IMAGE USING SECOND
`IMAGE CAPTURE MODULE
`SET TO SECONDARY
`FOCUS POSITION
`
`512
`
`532
`
`CAPTURE A SECONDARY
`
`
`STILL IMAGE USING FIRST
`
`
`IMAGE CAPTURE MODULE
`
`
`
`
`FOCUS POSITION
`
`
`SET TO SECONDARY
`PRIMARY IMAGE
`
`USE SECONDARY STILL
`
` 514
`IMAGE TO ENHANCE THE
`
`
`DEPTH OF FIELD OF THE
`PRIMARY IMAGE
`
`
`534
`
`USE SECONDARY STILL
`IMAGE TO ENHANCE THE
`DEPTH OF FIELD OF THE
`
`FIG. 14
`
`APPL—1007 / Page 16 of 49
`
`APPL-1007 / Page 16 of 49
`
`

`

`U.S. Patent
`
`01028,2mD
`
`92f051
`
`mm958:7SU
`
`2
`
`m"memmmo"zmooz"-omm
`
`"misjmo"Full:unlf
`
`com
`
`«.53.:an
`
`memmmooma
`
`BI8mmh@F‘
`
`«6“.Qt
`
`#00
`
`wow
`
`coo
`
`moo
`
`APPL—1007 / Page 17 of 49
`
`APPL-1007 / Page 17 of 49
`
`
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 16 of 29
`
`US 7,859,588 B2
`
`618622 610628
`620
`
`FIG.16A
`
`616
`
`626
`
`622614
`
`
`
`FIG.1GB
`
`626
`
`624
`
`620614
`
`622618
`
`610
`
`APPL—1007 / Page 18 of 49
`
`APPL-1007 / Page 18 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 17 of 29
`
`US 7,859,588 B2
`
`
`
`APPL—1007 / Page 19 of 49
`
`APPL-1007 / Page 19 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 18 of 29
`
`US 7,859,588 B2
`
`GPS LOCATION OF THE CAMERA
`PROVIDED BY THE GPS IN THE CAMERA
`
`CAMERA POINTING DIRECTION PROVIDED BY
`THE ELECTRONIC COMPASS IN THE CAMERA
`
`750
`
`752
`
`DISTANCE OFFSETS FROM THE CAMERA TO PORTIONS
`OF THE SCENE PROVIDED BY THE RANGEMAP
`
`754
`
`ANGULAR OFFSET FROM THE CAMERA PROVIDED
`FROM THE LOCATION IN THE FIELD OF VIEW
`
`GPS LOCATIONS FOR PORTIONS OF THE SCENE ARE
`DETERMINED BY ADDING DISTANCE OFFSETS AND THE
`
`POINTING DIRECTION OF THE CAMERA
`
`ANGULAR OFFSETS TO THE GPS LOCATION AND
`
`756
`
`758
`
`A GPS LOCATIONS FOR THE OBJECTS IN THE SCENE
`ARE STORED IN METADATA OR DISPLAYED AS LABELS
`
`760
`
`IN A GPS LOCATION MAP
`
`FIG. 18
`
`APPL—1007 / Page 20 of 49
`
`APPL-1007 / Page 20 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 19 of 29
`
`US 7,859,588 B2
`
`START UP
`
`CAMERA POWER ON
`INITIALIZATION
`
`1 100
`
`SET FIRST AND SECOND CAPTURE
`UNITS TO DEFAULT ZOOM POSITIONS
`
`1 102
`
`CAPTURE AND DISPLAY FIRST AND
`SECOND PREVIEW IMAGES USING FIRST
`AND SECOND CAPTURE UNITS
`
`1 104
`
`
`1106
`
`
`SET SELECTED
`SELECT
`FIRST OR SECOND
`CAPTURE UNIT
`
`PREVIEW
`AS PRIMARY
`
`IMAGE
`CAPTURE UNIT
`
`1 124
`
` 1108
`
`YES
`
`
`IS S1
`SHUTTER BUTTON
`
`
`PRESSED
`
`
`
`NO
`
`
`
`SET DEFAULT CAPTURE UNIT
`AS PRIMARY CAPTURE UNIT
`
`SET NON-PRIMARY CAPTURE UNIT
`AS SCENE ANALYSIS CAPTURE UNIT
`
`1 1 14
`
`PREVIEW MODE
`
`FIG. 19
`
`APPL—1007 / Page 21 of 49
`
`APPL-1007 / Page 21 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 20 of 29
`
`US 7,859,588 B2
`
`
`
`PREVIEW MODE
`
`
`
`
`
`1200
`
`SCENE ANALYSIS CAPTURE UNIT
`
`ANALYZES SCENE
`
`
`
`1202
`
`
`
`
`
`
`1206 SCENE
`
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERS UTILIZING SCENE
`ANALYSIS CAPTURE UNIT DATA
`
`
`
`
`
`1204
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`SCENE ANALYSIS CAPTURE UNIT
`ANALYZES SCENE
`
`1208
`
`CONDITIONS
`
`CHANGED
`
`FIG. 20
`
`APPL—1007 / Page 22 of 49
`
`APPL-1007 / Page 22 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 21 of 29
`
`US 7,859,588 B2
`
`PREVIEW MODE
`WITH THRESHOLD
`
`CAPTURE IMAGE FROM
`
`SCENE ANALYSIS CAPTURE UNIT
`
`
`
`1300
`
`
`
`
`1306
`
`
`1302
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERS UTILIZING SCENE
`
`ANALYSIS CAPTURE UNIT DATA
`
`1304
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`1308
`
`SCENE
`CONDITION CHANGE>X
`
`OR STABLE
`
`FIG. 21
`
`APPL—1007 / Page 23 of 49
`
`APPL-1007 / Page 23 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 22 of 29
`
`US 7,859,588 B2
`
`ENHANCED
`PREVIEW MODE
`
`1400
`
`SET SCENE ANALYSIS CAPTURE UNIT
`ZOOM POSITION RELATIVE TO PRIMARY
`CAPTURE UNIT ZOOM POSITION
`
`1402
`
`1404
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERS UTILIZING SCENE
`ANALYSIS CAPTURE UNIT DATA
`
`1406
`
`CAPTURE PREVIEW IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`1408
`
`ANALYZE SCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS DATA
`
`1410
`
`1412
`
`1414
`
`1416
`
`
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERS UTILIZING RESULTS
`
`OF THE SCENE ANALYSIS
`
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`
`
`CAPTURE IMAGE FROM
`
`SCENE ANALYSIS CAPTURE UNIT
`
`ANALYZE SCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS DATA
`
`
`
`1418
`
`
`SCENE
`
`CONDITION CHANGE>X
`OR STABLE
`
`
`
`FIG. 22
`
`APPL—1007 / Page 24 of 49
`
`APPL-1007 / Page 24 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 23 of 29
`
`US 7,859,588 B2
`
`
`
`NO
`
` 1 500 ZOOM
`
`BUTTON
`
`PRESSED
`
`?
`
`
`
`RETURN TO PREVIEW
`
`ZOOM
`
`
`
`1 502
`
`POSITION ‘
`
`
`PRIMARY CAPTURE
`
`
`UNIT ZOOM
`
`RANGE
`
`SET CURRENT SCENE ANALYSIS
`AND PRIMARY CAPTURE UNITS AS
`PRIMARY AND SCENE ANALYSIS
`CAPTURE UNITS RESPECTIVELY
`
`
`
`1504
`
`1506
`
`
`
`
`
`SET PRIMARY CAPTURE UNIT ZOOM
`POSITION TO SELECTED ZOOM POSITION
`
`
`
`RETURN TO PREVIEW
`
`
`
`FIG. 23
`
`APPL—1007 / Page 25 of 49
`
`APPL-1007 / Page 25 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 24 of 29
`
`US 7,859,588 B2
`
`CAPTURE
`
`1600
`
`
`
`IS S1
`
`
`SHUTTER BUTTON
`PRESSED
`
`
`
`NO
`
`YES
`
`RETURN TO PREVIEW
`
`CAPTURE PREVIEW IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`ANALYZE SCENE UTILIZING
`CAPTURED PREV|EW IMAGE
`
`
`
`
`1602
`
`1604
`
`1606
`
` ANALYSIS
`
`
`COMPLETE
`
`
`1608
`
`SET PRIMARY CAPTURE UNIT PARAMETERS
`UTILIZING RESULTS OF THE ANALYSIS
`
`1610
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`
`1612
`
`IS 82
`
`
` IS FOCUS/
`SHUTTER BUTTON
`
`
`EXPOSURE LOCK
`
`PRESSED
`
`SET
`
`YES
`
`1614
`
`CAPTURE PRIMARY IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`RETURN TO PREV|EW
`
`FIG. 24
`
`APPL—1007 / Page 26 of 49
`
`APPL-1007 / Page 26 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 25 of 29
`
`US 7,859,588 B2
`
`CAPTURE
`
`IS S1
`
`
`
`
`
`1700
`
`
`SHUTTER BUTTON
`
`PRESSED
`
`
`
`YES
`
`NO
`
`RETURN TO PREVIEW
`
`1702
`
`1704
`
`CAPTURE PREVIEW IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`1706
`
`ANALYZE SCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS IMAGES
`
`
`
`1708
`
`1710
`
`
`SET PRIMARY CAPTURE UNIT
`
`PARAMETERS UTILIZING RESULTS
`
`OF THE SCENE ANALYSIS
`
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`
`
`1712
`
`
` IS 32
`
`IS FOCUS/
`
`
`SHUTTER BUTTON
`
`EXPOSURE LOCK
`
`PRESSED
`
`SET
`
`
`YES
`
`1 714
`
`CAPTURE PRIMARY IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`RETURN TO PREVIEW
`
`FIG. 25
`
`APPL—1007 / Page 27 of 49
`
`APPL-1007 / Page 27 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 26 of 29
`
`US 7,859,588 B2
`
`RETURN TO PREVIEW
`
`CAPTURE
`
`
`1 800
`IS 31
`
`
`
`SHUTTER BUTTON
`PRESSED
`
`
`
`N0
`
`YES
`
`CAPTURE PREVIEW IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`ANALYZE SCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS IMAGES
`
`1802
`
`1804
`
`1806
`
`1808
`
`SET PRIMARY CAPTURE UNIT PARAMETERS
`UTILIZING RESULTS OF THE SCENE ANALYSIS
`
`1810
`
`SET SCENE ANALYSIS CAPTURE UNIT
`ASA SECONDARY CAPTURE UNIT
`
`1812
`
`SET SECONDARY CAPTURE UNIT PARAMETERS
`UTILIZING RESULTS OF THE SCENE ANALYSIS
`
`1814
`
`
`
`IS 82
`
`
`SHUTTER BUTTON
`PRESSED
`
`
`
`YES
`
`NO
`
`CAPTURE PRIMARY IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`CAPTURE AN AUGMENTATION IMAGE
`FROM SCENE ANALYSIS CAPTURE UNIT
`
`PRODUCE AN ENHANCED IMAGE FROM
`THE PRIMARY AND AUGMENTATION IMAGES
`
`1 816
`
`1 81 8
`
`1820
`
`RETURN TO PREVIEW
`
`FIG. 26
`
`APPL—1007 / Page 28 of 49
`
`APPL-1007 / Page 28 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 27 of 29
`
`US 7,859,588 B2
`
` I
`
`1\
`
`11
`
`\
`
`,
`
`1\\l
`
`155-4,
`
`“‘3/154
`
`d
`
`153
`
`152
`
`‘K
`‘r
`
`. ‘L
`‘y
`
`f
`
`158
`
`157 156
`
`|ll|ll||ll|l|
`
`l||lll||l|lll
`
`181
`
`180
`
`170
`
`171
`
`X2
`
`b
`
`X1
`
`FIG. 27
`(PRIOR ART)
`
`APPL—1007 / Page 29 of 49
`
`APPL-1007 / Page 29 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 28 of 29
`
`US 7,859,588 B2
`
`START
`
`197
`
`SET LENS TO MID FOCUS AND
`LOAD LOW PASS FILTER
`
`CALCULATE FOCUS VALUE (FV)
`AS IMAGE IS READ OUT
`
`ADJUST STEPPER MOTOR TO
`NEXT NEAR FOCUS POSITION
`
`198
`
`198
`
`199
`
`GET FV OF NEXT IMAGE CAPTURED
`
`STEP IN OPPOSITE
`
`WITH NEWFOCUSSETTING E DIRECTION
`
`COMPARE TO PREVIOUS
`WEIGHTED FOCUS VALUES
`
`FV INCREASES
`
`STEP IN SAME DIRECTION
`UNTIL FV DECREASES
`
`LOAD HIGH PASS FILTER
`AND MEASURE FV USING
`SAME FOCUS POSITION
`
`STEP IN SAME DIRECTION
`UNTIL FV DECREASE
`
`BACK UP AND HOLD AT
`HIGHEST FV FOCUS POSITION
`
`FOCUS OK
`
`FIG. 28
`(PRIOR ART:
`
`APPL—1007 / Page 30 of 49
`
`APPL-1007 / Page 30 of 49
`
`

`

`US. Patent
`
`Dec. 28, 2010
`
`Sheet 29 of 29
`
`US 7,859,588 B2
`
`FOCUS
`/"ADJUSTMENT
`STEPS
`
`A
`
`:
`
`:
`
`B
`
`FOCUS
`VALUE /
`
`
`
`pé‘éfi‘fSN
`
`p2:
`
`FIG. 29
`(PRIOR ART)
`
`APPL—1007 / Page 31 of 49
`
`APPL-1007 / Page 31 of 49
`
`

`

`US 7,859,588 B2
`
`1
`METHOD AND APPARATUS FOR
`OPERATING A DUAL LENS CAMERA TO
`AUGMENT AN IMAGE
`
`FIELD OF THE INVENTION
`
`The present invention relates to a digital camera that pro-
`duces digital image files and, more particularly, to a digital
`camera that uses multiple lenses and image sensors to provide
`an improved imaging capability.
`
`BACKGROUND OF THE INVENTION
`
`Currently, most digital cameras use a zoom taking lens and
`a single color image sensor to capture still and motion images.
`The captured images are then processed to produce digital
`image files, which are stored in a digital memory in the
`camera. The digital image files can then be transferred to a
`computer, displayed, printed, and shared via the Internet.
`In order to capture sharp images of moving subjects, a
`digital camera needs to provide a precise automatic lens
`focusing system (i.e., an autofocus system). The autofocus
`system must be capable of quickly obtaining the correct focus
`in order to minimize the “shutter delay” between the time the
`shutter button is pressed and the still image is captured. The
`autofocus system must also work in a continuous image cap-
`ture mode wherein video images are captured. For instance, in
`a video mode the focus should be adjusted in real-time while
`video images are being continuously captured.
`Many digital cameras and scanners capture images using
`an image sensor and a taking lens system with an adjustable
`focus. Typically, the focus distance of such an adjustable
`focus taking lens system can automatically be set to one of a
`plurality of different settings by sensing, control, and drive
`systems, which are adapted to provide optimal focus of what
`is determined to be a subject area in a scene. Lens systems that
`provide automatically adjustable focus settings based on a
`focus measurement and an adjustable focus lens are referred
`to herein as autofocus systems. Digital cameras typically use
`one of two types of autofocus systems: a rangefinder system
`and a “through-the-lens” focus system.
`A rangefinder system uses rangefinding sensors such as a
`sonic rangefinder or a dual lens rangefinder to determine the
`distance from a camera to one or more portions of a scene
`within a field of view of the rangefinder system. A sonic
`rangefinder measures the phase offset between a projected
`sonic signal and a reflected sonic signal to infer the distance to
`objects in the scene. Dual lens rangefinders contain two
`lenses that are separated by a distance along with two match-
`ing sensor areas that capture matched pairs of images. Dual
`lens rangefinders are commonly used on digital cameras in
`the form of dual lens rangefinder modules which contain two
`lenses separated by a distance along with two matching sen-
`sor areas that capture matched pairs of low resolution images.
`Common dual lens rangefinder-based autofocus systems
`include active and passive systems. Active systems actively
`project light onto the scene, while passive systems work with
`the available light from the scene. Dual lens rangefinder mod-
`ules can be purchased from Fuji Electric in several models
`such as the FM6260W. A dual lens rangefinder module for
`optical apparatus such as a camera is described in U.S. Pat.
`No. 4,606,630, which was issued to Haruki et al. on Aug. 19,
`1986 (and assigned to Fuji Electric). According to the
`description ofthe prior art in this patent, matched pairs of low
`resolution images are analyzed for correlation between the
`two images to determine the offset between the two images
`caused by the separation between the two lenses.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`
`A diagram illustrative of the principle of the operation of a
`conventional rangefinder is shown herein in FIG. 27. In that
`diagram, light from an object 151 is incident on two small
`lenses 152 and 153 which have a sufficiently short focal
`length f that light rays received from the object through dif-
`ferent spaced paths 154 and 155 produce corresponding
`spaced images 157 and 158 in a focal plane 156 which is
`common to the lenses 152 and 153. When the object 151 is at
`an infinite distance, the centers of the images 157 and 158 are
`located at reference positions 170 and 180 in FIG. 27, but
`when the object 151 is located at a closer distance, the centers
`ofthe images are shifted apart to positions 171 and 181. Ifthe
`distance by which the images 157 and 158 are shifted from the
`reference positions 170 and 180 are designated x1 and x2,
`respectively, then the total shift x may be expressed as fol-
`lows:
`
`x:x1+x2:bf/d
`
`Thus, the distance d to the object 151 can be measured by
`d:b~f/x. In this case, b is the distance between the optical axes
`of the small lenses, that is, the base length. To obtain the
`shifted amounts x1 and x2, or the sum x of both, two optical
`sensor arrays 190 and 191 are provided in the focal plane 156
`as shown in FIG. 27. These optical sensor arrays each com-
`prise a plurality of optical sensors, for instance CCD devices,
`and an analog photoelectric signal is generated by each opti-
`cal sensor corresponding to the light intensity at the portion of
`the image which is incident on the sensor. Haruki et al. shows
`a conventional circuit, as well as a higher speed rangefinding
`circuit according to the patent, for obtaining the sum x of the
`shifted distances by comparing two image signal trains com-
`prising the digital image signals from the left and right optical
`sensor arrays.
`
`Basically, the offset information x is used along with the
`lens separation distance b and the focal length f to calculate
`the distanced to the scene by triangulation. The calculated
`distance d to the scene is used to guide the positioning of an
`adjustable focus lens to produce the best image quality. As
`known in the prior art, this adjustment may be based on a
`calibration curve established between the distance to the
`
`scene as measured by the dual lens rangefinder module and a
`series of best focused images as produced by a “through the
`lens” autofocus system. The calibration curve is stored as an
`equation or a look-up table in a microprocessor in the camera.
`Rangefinder-based autofocus systems have the advantage
`ofbeing very fast, some having a response time that can be in
`the range of 0.01-0.05 second. However, the focus quality
`produced by some rangefinder-based autofocus systems can
`vary when they are used in different operating conditions. For
`example, sonic autofocus systems cannot focus through a
`glass window as the glass stops the projected sonic signal,
`thereby causing the autofocus system to focus onto the glass.
`In the case of a dual lens rangefinder autofocus system, the
`accuracy of dual lens rangefinders are typically influenced by
`changes in environmental conditions such as temperature
`and/or humidity. The problem with dual lens rangefinder
`modules is that
`the calibration between the dual
`lens
`
`rangefinder module and the adjustable focus lens position is
`not stable within the normal operating environment for digital
`cameras. Environmental conditions such as changes in tem-
`perature and humidity can cause the distance to the portion of
`the scene as measured by the dual lens rangefinder module to
`change by over 10%. In addition, the measured position ofthe
`adjustable focus taking lens in the adjustable focus taking
`lens system is prone to environmentally induced changes as
`well so that inaccuracies are produced in the control system
`APPL—1007 / Page 32 of 49
`
`APPL-1007 / Page 32 of 49
`
`

`

`US 7,859,588 B2
`
`3
`lens
`for the adjustable focus lens. Consequently, dual
`rangefinder modules are not typically used independently for
`autofocus in digital cameras but instead are used as a rough
`focus adjustment that is supplemented by a “through the lens”
`autofocus system.
`Alternatively,
`the “through-the-lens” autofocus system
`determines a focus state through an analysis of a series of
`autofocus images captured with the adjustable focus lens
`system positioned at a plurality of different focus distances.
`For example, in a typical “through-the-lens” autofocus sys-
`tem a plurality of autofocus images (e.g., 5-20) are captured
`with the adjustable focus lens in a series of different positions
`in a so-called “hill climb” method. This type of autofocus is
`known as “hill climbing” autofocus because it generates a
`sequence ofvalues that increase in level until they pass over a
`peak, i.e., a “hill”. In other words, the lens focus position is
`adjusted automatically until the contrast of the edge detail in
`the image, or a particular area ofthe image, is maximized. For
`instance, the contrast present in each of the autofocus images
`is compared and the autofocus image with the greatest con-
`trast is deemed to have been captured with the best focus
`conditions (often the best focus lens position is further refined
`by interpolating the contrast values between images).
`In order to decrease focusing response time without sacri-
`ficing focusing precision, it is common to use filters to sepa-
`rate not only the higher frequency component of the video
`signal, but also the lower frequency component. For example,
`a lens may be quickly driven in coarse adjustment steps in a
`low frequency range furthest from the maximum focus, and
`then driven in finer adjustment steps in a high frequency range
`nearer to the maximum focus. A flow diagram of a conven-
`tional “hill climbing” contrast autofocus algorithm is shown
`in FIG. 28. This algorithm uses the “hill climbing” contrast
`autofocus method discussed above and shown in the diagram
`of FIG. 29, which illustrates the relationship between the
`focus value obtained from the filters and the lens position. In
`FIG. 29, the abscissa indicates the focusing position of a lens
`along a distance axis, the ordinate indicates the focusing
`evaluation value, and the curves A and B indicate the focusing
`evaluation values for high and low frequency components,
`respectively, relative to a particular in-focus position P.
`Referring to the flow diagram of FIG. 28, the best starting
`point for the algorithm depends on the hyperfocal distance of
`the current lens setting, which is a function ofthe focal length
`setting and the f-number. A distance of about 2 meters is
`typically a good starting point. Then a low frequency band-
`pass filter is loaded (stage 197) and the focus values are read
`out. The algorithm employs a comparison stage 198 to set the
`direction of lens adjustment toward increasing focus values,
`and to determine when the lens is stepped over the “hill”. The
`depth of field, which depends on the present focal length and
`f-number, sets the number of steps, i.e., the next near focus
`position, which should be taken before capturing the next
`frame when using the low frequency bandpass filter. Once the
`peak of the hill is passed (curve B in FIG. 29), a high fre-
`quency bandpass filter is loaded (stage 199), and the lens is
`moved in the opposite direction until the peak of the higher
`“hill” is found (curveA in FIG. 29). The peak focus value may
`use either the weighted average or peak value from numerous
`pixels.
`“Through-the-lens” autofocus systems are very accurate
`since they measure focus quality directly from autofocus
`images captured with the high quality taking lens. Unfortu-
`nately, “through-the-lens” autofocus systems can be rela-
`tively slow in determining a focus setting due to the large
`number of autofocus images that must be captured and com-
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`
`pared. For example, “through-the-lens” autofocus systems
`can take as long as 0.5-2.0 seconds to determine focus con-
`ditions.
`
`Accordingly, in some digital cameras, the two types of
`autofocus systems are used together in a hybrid system in
`which the rangefinder based autofocus system is used to
`provide a fast estimation of the adjustable focus lens location
`that is then followed by the use of the “through-the-lens”
`autofocus system to refine the focus setting. For example,
`U.S. Pat. No. 6,864,474, ent