APPL-1007/Page 1 of 49
`Apple Inc. v. Corephotonics
`
`

`

`US 7,859,588 B2
`
` Page 2
`
`U.S. PATENT DOCUMENTS
`
`FOREIGN PATENT DOCUMENTS
`
`2006/0193509 Al
`2008/0013941 AL*
`2008/0218612 A1*
`
`2008/0218613 Al*
`
`8/2006 Criminisi et al.
`1/2008 Daley weecesseseeeee 396/121
`9/2008 Border etal.
`ue... 348/262
`
`9/2008 Jansonetal. we... 348/262
`
`JP
`wo
`
`2005045511
`s00sonore
`
`2/2005
`boos
`
`* cited by examiner
`
`APPL-1007 / Page 2 of 49
`
`APPL-1007 / Page 2 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 1 of 29
`
`US 7,859,588 B2
`
`6SSSVdWOO
`
`
`
`QuvVOJOVAYSLNI
`
`gsWdavBe
`
`
`
`AYOWANYOSSAOONd
`
`Z606G91Y¥O109||
`
`ASVWIOZ
`
`AV1dSIq
`
`QNVYOSS300Nd
`
`aTSVAOWSYAMONalQNVWOOzes
`
`
`
`
`|WACOW1'4OSSa00ud!JOULNOOSUNSOEXAos,OLOW)
`
`
`AYOWANQuvoSYOLOWSNOO4A
`SNOOA(¢gg—tONVWOOZ+}agSHVMWHIS
`
`
`999CCcl€|
`rmvintao;1YWINTISO|pwAYOWSWSYOLOW
`ve2TTzxXnwteSN31
`deoo]Lsuld
`
`AYOWAWHOSNaS
`
`yaaangVWI
`
`™~
`Vol
`
`WOOZ
`
`APPL-1007 / Page 3 of 49
`
`APPL-1007 / Page 3 of 49
`
`
`
`
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 2 of 29
`
`42a
`
`10A
`
`10A
`
`FIG.2B
`42a
`
`US 7,859,588 B2
`
`APPL-1007/ Page 4 of 49
`
`FIG.2A
`
`APPL-1007 / Page 4 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 3 of 29
`
`US 7,859,588 B2
`
`ZOOM POSITION SET TO
`DEFAULT POSITION WHEN
`CAMERA IS POWERED ON
`
`USER INPUT
`ZOOM POSITION
`
`100
`
`101
`
`POSITION
`
`>X?
`
`
` ZOOM
`
`
`
`
`SET FIRST IMAGE CAPTURE
`SET SECOND IMAGE CAPTURE
`
`STAGE TO CAPTURE IMAGES
`STAGE TO CAPTURE IMAGES
`
`
`
`& SECOND IMAGE CAPTURE
`& FIRST IMAGE CAPTURE
`
`
`STAGE FOR AUTO FOCUS
`STAGE FOR AUTO FOCUS
`
`
`
`
`
`CAPTURE AUTOFOCUS
`IMAGES WITH FIRST
`IMAGE CAPTURE STAGE
`
`
`
`FOCUS FIRST IMAGE
`108
`128
`FOCUS SECOND IMAGE
`CAPTURE STAGE
`CAPTURE STAGE
`
`
`
`
` CAPTURE PREVIEW IMAGES
`
`110
`130
`CAPTURE PREVIEW IMAGES
`WITH SECOND IMAGE CAPTURE
`WITH FIRST IMAGE CAPTURE
`
`
`
`
`STAGE AND DISPLAY
`STAGE AND DISPLAY
`
`ZOOM
`BUTTON
`
`
`PRESSED
`?
`
`
`
`CAPTURE AUTOFOCUS
`IMAGES WITH SECOND
`IMAGE CAPTURE STAGE
`
`106
`
`126
`
`
`
`
`
`134
`
`
`
`
`CAPTURE
`BUTTON
`PRESSED
`?
`
`YES
`
`CAPTUREDIGITAL STILL
`IMAGE WITH FIRST
`IMAGE CAPTURE STAGE
`
`116
`
`136
`
`CAPTUREDIGITAL STILL
`IMAGE WITH SECOND
`IMAGE CAPTURE STAGE
`
`FIG. 3
`
`APPL-1007 / Page 5 of 49
`
`APPL-1007 / Page 5 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 4 of 29
`
`US 7,859,588 B2
`
`Ole
`
`
`
`
`
`NOLLAGSYNLdVOSHLSAHSNdYASNSHLNSHM
`
`GSaYuNLdVOSISOVWNIdAHL‘ZSOLLSWOU
`
`bvSIA
`
`
`
`
`
`
`
`SNOOJLS3dYO4NOILISOdGANIWYSAL3d
`
`COC
`
`
`
`
`
`JOVWISHLWOUSSOVSNOOSOLNVSHL
`
`
`
`
`
`NOILISOdWOOZSLOATASY3aSN
`
`
`
`GAddOUdSINOILISOdWOOZY4SMO1SHLNIS9DVLS
`
`
`
`ONIGNOdSAYNOOLVHLOSGaIdWVSdNGNV
`
`SAOVWNISNOOJOLNVOMLAHLNiSAYNLVAS
`
`
`
`HOIHMSANINYSLACNOILISOdWOOZ
`
`
`
`STAXldJOYSEWNNAWVSAHLNVdS
`
`
`
`
`
`JAuYNLdVOYOIGSNS|JOVLSJ9VWI
`
`y9~
`
`SNOOIJOLNVAATdWVSdNGNVGaddOwdAHL
`
`SNOOJOLNVSHLO1-GSLVISHHOOSISOV!
`
`
`
`SISUNLdVOHOSGASNLONSADVLSAOVWISHL
`
`SHLOLLSASOTOLNIOdSHLOLGAW00Z
`
`
`
`
`
`OMLFHLN3AMLS8L3S440TaXldSHLASILN3dIOlFOVLS
`
`JOVIYSHLOSHLWOUSOV!
`
`
`NOILISOdWOOZG3LO31ASY3SNn
`
`99¢
`
`SNOOFOLNAVAHLASN
`LNAN3SAOWSHLANINYSLAGOLL4S434013Xid
`
`QalsILN3d]SHLONVSAYNDNOILVYSIIVO
`
`
`$34OlG3034NSN31SNOOJSHLJO
`
`
`YACNISSONVY
`
`
`JAOVLSFUNLdVOSDVWIAHLSNDOS
`
`
`AJONSANDASSNOOSOLNVALVILINIOL1SOL0S
`
`
`
`WOdsNOLINSYNLdVDSSHSNdY4SN
`
`
`auNLdVOJUVSSADVWISNOOAOLAV
`
`
`
`
`
`SNOILISOdTWOOSYSdAHYISHLOLLASYY
`
`ONIGNOdSSAYNOOHOSSAVSNDOSOLNV
`
`
`
`SSOVWNISHL40SNOILYOd
`
`
`
`
`
`SS9VLSJOVIHLONISASN371SND04
`
`89C
`
`
`
`
`
`JHLOLGSAOWSISN31SNDOsSHL
`
`
`
`
`
`SAOVLSSOVAIHLOGHLIM
`
`OSC
`
`cGc
`
`ATA
`
`9SC
`
`8S¢
`
`09¢
`
`APPL-1007 / Page 6 of 49
`
`APPL-1007 / Page 6 of 49
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 5 of 29
`
`US 7,859,588 B2
`
`
`
`vGOHLAWSWI19THK.SHLONISN
`
`
`
`
`
`
`
`
`
`AYNALdVOYOsG3SNONIZ8AOVLSSYNLdVO
`
`JOVIAHLHLIMSNO04MOFHOATSNONNILNOO9/2
`
`
`
`OAQIAFYNLdVDAISNONNILNOD9/2
`
`
`
`
`JOVLSAYNLdVOAHLHLIM
`NOILISOdWOOZ NOILISOdWOOZ
`
`
`JOVAIHOIHMSANINYSL3AG
`
`
`
`JOVLSFYNLdVOAHLS|SOVLSFYALdVI
`
`SLOAISSYASN
`
`
`AHLOL1SASO19
`
`
`
`LNIOdSHLOLGANO0ZS|SHNLdVOYO4JASNLONJOVLSSOVWISHL
`
`
`
`
`
`
`
`NOILISOdWOOZGaLOaTsSYssn
`
`
`
`wGOHLAWEWITOTHH,SAOOSOLNVSHLISN79Z
`
`
`
`
`
`
`
`
`
`QAlsILNAdISHLONYSA8NONOILVYEITVO
`
`
`
`
`
`JHLANINSLA0OLLNSAWLSNPOVSN31SNDO4
`
`SNOO41$483ONGONdOLJOVLSANNLdVOAHLNIQ4034N
`
`
`
`SN31SNOO43SHLAOLNAW3AOW
`
`
`
`AOVLISFYNLdVOSHLNISNSTSNO004SHLSAOW992
`
`
`
`
`
`
`
`
`
`NOILISOdSNDOJLS38MANSHLOL
`
`S‘Old
`
`cSOllSWOUd
`
`JAYNLdVOYO4GASNONIAdLONADVLS
`
`AYNLdVOADVWISHLNOSNOO4LSA9FONGOYdOL
`
`
`
`
`Q4d04aNLNAWLSAPGVSN31SNOO4SHLASILNAGIZ9z
`
`
`
`wGOHLAWBAITTIIH,AHLAdSADVLS
`
`
`
`JYNLdVOJOVIHLOGSNOOJOLNV
`
`
`
`
`
`JONANDASSNOOAOLAVALVILINIOLFSOL0S
`
`
`
`NOLINFYNALdVOSHLSAHSNdYASN
`
`
`
`
`
`
`
`
`
`WOddNOLLNGFYNLdVOSAHSNdY3aSN
`
`
`
`
`
`0SZc
`
`CGC
`
`vSZc
`
`8S
`
`CLZ
`
`vlc
`
`APPL-1007 / Page 7 of 49
`
`APPL-1007 / Page 7 of 49
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 6 of 29
`
`US 7,859,588 B2
`
`
`
`
`
`
`CAPTURE A SERIES OF IMAGE SETS WITH OBJECTS
`AT KNOWN DISTANCES WITH THE SHORTER
`FOCAL LENGTH FIRST IMAGE CAPTURE STAGE AND
`THE LONGER FOCAL LENGTH SECOND IMAGE CAPTURE
`STAGE AND A SERIES OF FOCUS LENS POSITIONS
`
`
`
`
`
`
`
`
`THE AUTOFOCUS IMAGE FROM THE LOWER
`FOCAL LENGTH FIRST IMAGE STAGE IS CROPPED AND
`UPSAMPLED SO THAT CORRESPONDING FEATURES
`IN THE TWO AUTOFOCUSIMAGES SPAN
`THE SAME NUMBEROF PIXELS
`
`
`
`
`
`
`
`
`CORRELATE THE IMAGES FROM THE SECOND IMAGE
`CAPTURE STAGE TO CORRESPONDING PORTIONS OF
`
`
`THE IMAGES FROM THE CROPPED AND UPSAMPLED
`
`
`IMAGE FROM THE FIRST IMAGE CAPTURE STAGE TO
`
`
`DETERMINE THE PIXEL OFFSET BETWEEN
`THE IMAGES IN EACH IMAGE SET
`
`
`300
`
`302
`
`304
`
`
`
`STORE THE DATA OF PIXEL OFFSET BETWEEN
`IMAGESIN EACH IMAGE SET VERSUS KNOWN
`DISTANCE TO OBJECTS AS AN AUTOFOCUS
`
`RANGEFINDER CALIBRATION CURVE
`
`
`
`
`306
`
`FIG. 6
`
`APPL-1007/ Page 8 of 49
`
`APPL-1007 / Page 8 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 7 of 29
`
`US 7,859,588 B2
`
`
`
`
`
`CAPTURE A SERIES OF IMAGE SETS WITH OBJECTS
`AT KNOWN DISTANCES WITH THE FIRST IMAGE
`CAPTURE STAGE AND THE SECOND IMAGE CAPTURE
`
`
`STAGE WHEREIN AUTOFOCUSIS DONE BY THE
`
`
`"HILL CLIMB METHOD" FOR EACH IMAGE
`
`
`
`
`
` COMPARETHE FOCUS LENS POSITIONS FOR THE TWO.
`IMAGE CAPTURE STAGES VERSUS THE DISTANCE TO
`THE FOCUSED OBJECTS IN THE IMAGE SETS
`
`
`
`
`
`STORE THE DATA OF FOCUS LENS POSITIONS OF
`
`
`THE FIRST IMAGE CAPTURE STAGE VERSUS THE FOCUS
`
`
`LENS POSITIONS OF THE SECOND IMAGE CAPTURE
`STAGE FOR THE SAME DISTANCE TO FOCUSED
`OBJECTS IN THE IMAGES AS AN AUTOFOCUS
`
`
`"HILL CLIMB METHOD" CALIBRATION CURVE
`
`
`
`400
`
`+—-402
`
`404
`
`FIG. 7
`
`APPL-1007 / Page 9 of 49
`
`APPL-1007 / Page 9 of 49
`
`

`

`
`
`
`SET FIRST IMAGE CAPTURE
`SET SECOND IMAGE CAPTURE
`MODULE TO CAPTURE IMAGES
`MODULE TO CAPTURE IMAGES
`
`
`
`& SECOND IMAGE CAPTURE
`& FIRST IMAGE CAPTURE
`
`
`
`MODULE FOR AUTO FOCUS
`MODULE FOR AUTO FOCUS
`
`
`
`
` 126
`
` 128
`
`CAPTURE AUTOFOCUS IMAGES
`
`AUTOFOCUS SECOND IMAGE
`CAPTURE MODULE
`
`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 8 of 29
`
`US 7,859,588 B2
`
`ZOOM POSITION SET TO DEFAULT
`POSITION WHEN CAMERA |S POWERED ON
`
`100
`
`USER INPUT ZOOM POSITION
`
`47101
`
`
`ZOOM
`POSITION
`
`
`>X?
`
`
`
`
`
`
`
`
`
` 130
`CAPTURE PREVIEW IMAGES
`CAPTURE PREVIEW IMAGES
`WITH FIRST IMAGE CAPTURE
`WITH SECOND IMAGE CAPTURE
`
`
`
`MODULE AND DISPLAY
`
`
`ZOOMBUTTON —ves||
`
`MODULE AND DISPLAY
`
`PRESSED?
`
`
`
`CAPTURE BUTTO
`CAPTURE BUTTON
`PRESSED?
`PRESSED ?
`
`
`
`YES
`
`
`
`CAPTURE VIDEO IMAGE WITH
`CAPTURE VIDEO IMAGES WITH
`FIRST IMAGE CAPTURE MODULE
`SECOND IMAGE CAPTURE MODULE
`
`
`
`
`
`CHECK IMAGE FOR
`CHECK IMAGE FOR
`
`FOCUS QUALITY
`FOCUS QUALITY
`
`
`
`
`
`
`NEED TO
`REFOCUS ?
`
`120
`
`NEED TO
`REFOCUS ?
`
`FIG. 8
`
`APPL-1007 / Page 10 of 49
`
`APPL-1007 / Page 10 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 9 of 29
`
`US 7,859,588 B2
`
`8vP
`
`JHLHLIMADVISNDOSOLNVYGNOOASVSYNALdVO
`
`
`
`OrrAHLHLIMSDVSNOOSOLNVLSUlsVAYNLdVO
`
`JOVLSAYNLdVOADVAHLONSA1WOOdYA9NO1
`
`
`
`
`
`
`
`JOVLSAYNLdVOSOVHLONSWOOdY¥3M01
`
`
`
`
`
`JAYNDNOILVYSITV9YSAGNISSONVY
`
`vor
`
`
`
`
`
`AOVAISNOOAOLNVGNOOSSSHLALV1SYYOO
`
`9bL
`
`
`
`
`
`Q3043NNOILOAYYODSNIOSSHLANINYALAG
`
`
`
`SNOOSOLNVSHLGNVLAS44OAHLWOus
`
`
`AHLNS3SM13813$440
`
`
`TaXxldSHLANINYALSGOlJOVIGATdWVSdNGNVGaddO¥dAHLHLIM
`
`
`
`
`
`SSOVWIAHLJOSNOILYOdLN3Y34S10YOsSAOVUI
`
`
`
`STaXIdJOYSWNNSWVS
`
`
`
`ONVGaddO0u9SIHLONSATWOOYSMO7SHLLNI
`
`
`
`
`
`SIYNLVSSONIGNOdS3YwOOLVHLOSGaldWVSdN
`
`
`
`
`
`JHLNVdSSSDVWISNOOJOLNVOMLSHNI
`
`6
`
`Vlad
`
`APPL-1007 / Page 11 of 49
`
`
`
`
`
`Cry-TAVISJOVISHLWOYsJOVWISNOOSOLNVSHL
`
`
`
`
`
`APPL-1007 / Page 11 of 49
`
`
`
`
`
`

`

`Dec. 28
`
`’
`
`2010
`
`Sheet 10 of 29
`
`US 7,859
`
`2
`
`588 B2
`
`
`
`
`
`AHLYO4SNOILIGNODSNDOJSHLFYVdWOD
`
`
`
`
`
`ADVANIYAHLONVSHLSOASOHLOLSDVWIGNOOAS
`
`snoog
`
`SNOILIGNOD
`
`éQ3DNVHO
`
`OL‘S/d
`
`U.S. Patent
`
`
`
`99rQNOO3SSHLHLIMSOVWIGNOOSSVSYNdVO09rJOVILSYldSHLHLIMSDVLSaisVFYALdVO
` QNOO3SSHLHLIMJOVI
`
`
`
`
`
`
`
`SNOOJOLAVFHLNIFYSHMSAOVLSAYNLAVOJOVISI]SNOOJOLNVSHLNIZYSHMAOVLSSYNLdVO
`
`
`YSHLONVAYNLdVOcorLSdl4SHLWOUSJOVIAHLACIAOUd
`
`
`wGOHLAW8WI1DTHH.SHLAGSNOOSIsCGOHLAWSWITH.SHLAGSNOG
`
`
`
`
`
`,GOHLSWSWI0TH,3HLA@3NOGSI
`
`SNOOSOLNAVSHLNIZYSHMSOVLSAYNLAVOJOVI
`
`
`
`
`AW1dSI0SHLNOADVIMalAaddVSVADVLS
`
`
`
`FYNLdVOSOV
`
`
`
`
`
`LSul4FHLYOsSNOILIGNODSNOOJAHLADNVHO
`
`
`
`
`
`
`
`AONVHOSHLNOGASVEADVISFYNLdVOJOVI
`
`
`
`
`
`QNOOASAHLYOSSNOILIGNOOSNDO4SHL40
`
`
`
`
`
`SNIOSOLNVAHLONVJDVLSAYNLdVdSOVW!
`
`
`
`JAANNOILVYSIIVO.GOHLAWEWI19THH,
`
`
`
`
`
`
`
`APPL-1007 / Page 12 of 49
`
`APPL-1007 / Page 12 of 49
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 11 of 29
`
`US 7
`
`859,588 B2
`
`87V
`
`SHLHLIMS9VWISNOOJOLNVGNOOASVAYNLdVO
`
`
`
`
`JHLHLIMSDVWISNOOJOLAVLSulsVAYALdVO
`
`Ory
`
`FJOVLSFYNLdVOADVIHLONST1WOO3YSHOIH
`
`
`
`
`
`
`JAOVLSSYNLdVOADVHLONATWOOdYSMO1
`
`
`
`
`
`AHLONISNSOIASGAYNLdVOJOVANISHLWOYS
`
`
`
`
`
`O87
`
`
`
`JOVISNOOJOLNVGNOOASSHLALVISYYOO
`
`
`
`
`
`c8V
`
`
`
`SAONVLSIGOLSLASI40TAXldSHLLYAANOD
`
`
`
`
`
`OLJOVIGAIdWVSd/NGNVG3ddOu9SHLHLIM
`
`
`
`
`
`AHLNFAML39LASS4O13XldSHLANIWYSLSC
`
`
`
`SAOVWISHL30SNOILYOdLN3Y33310YOsSS9VWI
`
`
`
`
`
`
`
`
`
`GNVdaddO¥ddS$!NOLLISOdWOOZYSMO1SHLNI
`
`
`
`
`
`SaYNLVAdONIGNOdSAYYODLVHLOSGa1dWVSdN
`
`
`
`
`
`NVdSSA9DVWISNOOSOLNVOMLSHNI
`
`
`
`ST3Xid30YSEWNNAWVSSHL
`
`
`
`JANNNOILVYSITVOYSCNISSONVYSNOOSOLAY
`
`
`
`
`
`v8V
`
`
`
`SAONVLSICSHLONIMOHSdVVSONdOUd
`
`
`
`
`
`SAOVAISHL4OSNOLLYOdLN3Y33410OL
`
`LLOld
`
`APPL-1007/ Page 13 of 49
`
`Cv
`
`
`
`
`
`
`
`
`
`JOVLSJOVWIAHLWOYSSOVWISNOOSOLNVSHL
`
`APPL-1007 / Page 13 of 49
`
`
`
`
`
`
`
`

`

`
`|WaCOW|'OSSAOONd|SunSOdx4aOLNV
`
`
`
`SIgVAOWSYANOWSWyD0710Snood
`
`
`ANOWAZINauvoSUaAINdSHOLOW
`99AS)ccLf€L
`
` mavintao;4}VINTISO|psAYOWSWEoSYOLOWSNDOJS(¢SYVMWMIAEoGNYWoo?z
`1SOHISOH1Suldqsxid
`
`Od3OV4YSNISOYNIsndo4
`
`9SWdvlVv@qHVvO
`
`1SSd|ZdSv|GNOogS
`v99€8d
`JOVI5OLYasnZy[HSv1a|8p
`
`YOSNSAScdSvJOVI
`
`pScS
`
`JOVsAYSLNI
`
`
`
`AXOWSWNYoOsssa00dd
`
`YOSNESSNF]
`
`isuid
`
`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 12 of 29
`
`US 7,859,588 B2
`
`Z606QO1YO10
`
`TOYULNOOD
`
`ONV¥OSS300¥d
`
`YOLVYANASONIAIL
`
`SYOLOALAG
`
`ckOld
`
`APPL-1007 / Page 14 of 49
`
`APPL-1007 / Page 14 of 49
`
`
`
`
`
`

`

`(gg—TSYVMNUIS
`
`
`
`3aVvAOWAaYAYOWSWSNO04ec
`rscs0sYOSNSSN31
`
`
`
`mvinnao,YVINTTSO|ANOWSINOMIHLWOOz€dSV
`
`|WSCOW'!___'YOSS300ud|GnSoSN31
`
`Z606MO010CNVWOOZ5||G/j(anneeeeI~oeeenne-ftYOSNSS
`
`ANOWSWauvoYOLOW
`99c9cLULLCOEL
`1SOH1SOHa3xi4
`9dFOVIYSLNISOYSndo4
`
`Tossprsss sssssassY344na
`
`
`
`
`is-t__SdJoviYOSNSS
`v6YYAYOWSWNOLoywy
`v987)
`AMOWSWGQNOO3SGNOOaS
`
`
`
`
` 39VvNIHyqdYO109syaalud|{suOLOWSNd04
`
`AV1d0SIOJOYLNOD
`Eb‘Olaema)
`
`
`
`YOLVYANADSONIANILSYOLOILAG
`
`STOYLNOD
`
`4asn
`
`ONV¥OSS300dd
`
`
`
`AYNSOdxXsOLNV.
`
`APPL-1007 / Page 15 of 49
`
`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 13 of 29
`
`US 7,859,588 B2
`
`
`
`auvoJOVAYSLNI
`
`9SWVY
`
`6SSSVdWOO
`
`NYOSSS00Nd
`
`MIOTO
`
`QNVWOOZ
`
`SYSAIeNG
`
`SYOLOWSNIDO4
`
`qs
`
`JOVIWao
`
`Isuld
`
`APPL-1007 / Page 15 of 49
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 14 of 29
`
`US 7,859,588 B2
`
`ZOOM POSITION SET TO
`DEFAULT POSITION WHEN
`CAMERA IS POWERED ON
`
`500
`
`
`ZOOM
`POSITION
`
`
`
`>X ?
`
`
`
`
`SET FIRST IMAGE CAPTURE
`4-504
`524
`SET SECOND IMAGE CAPTURE
`
`MODULE TO CAPTURE IMAGES
`MODULE TO CAPTUREIMAGES
`
`
`& SECOND IMAGE CAPTURE
`& FIRST IMAGE CAPTURE
`
`
`
`MODULE FOR AUTO FOCUS
`MODULE FOR AUTO-FOCUS
`
`
`
`
`
`
`
`
`CAPTURE AND DISPLAY
`506
`526
`CAPTURE AND DISPLAY
`PREVIEW IMAGES WHILE
`PREVIEW IMAGES WHILE
`
`
`
`PERFORMING CONTINUOUS
`PERFORMING CONTINUOUS
`
`
`
`AUTOFOCUS USING FIRST
`AUTO-FOCUS USING SECOND
`
`
`
`IMAGE CAPTURE MODULE
`IMAGE CAPTURE MODULE
`
`
`
`
`
`
`
`ZOOM
`ZOOM
`
`BUTTON
`BUTTON
`
`
`PRESSED
`PRESSED
`
`
`?
`?
`
`NO
`
`NO
`
`
`
`
`
`WHEN SHUTTER BUTTON IS
`PRESSED, CAPTURE A PRIMARY
`STILL IMAGE USING FIRST
`
`IMAGE CAPTURE MODULE SET
`TO PRIMARY FOCUS POSITION
`
`
`
`
`
`
`+-510
`
`530
`
`
`WHEN SHUTTER BUTTONIS
`
`PRESSED, CAPTURE A PRIMARY
`STILL IMAGE USING SECOND
`IMAGE CAPTURE MODULE SET
`TO PRIMARY FOCUS POSITION
`
`
`
`
`SET TO SECONDARY FOCUS POSITION
`
`
` 514
`
`USE SECONDARYSTILL
`IMAGE TO ENHANCE THE
`DEPTH OF FIELD OF THE
`PRIMARY IMAGE
`
`
`
`
`
`534
`
`USE SECONDARYSTILL
`IMAGE TO ENHANCE THE
`DEPTH OF FIELD OF THE
`PRIMARY IMAGE
`
`FIG. 14
`
`APPL-1007 / Page 16 of 49
`
`
`
`
`
`
`
`
`
`CAPTURE A SECONDARY
`STILL IMAGE USING FIRST
`
`
`IMAGE CAPTURE MODULE
`
`
`
`
`CAPTURE A SECONDARY
`STILL IMAGE USING SECOND
`IMAGE CAPTURE MODULE
`SET TO SECONDARY
`
`512
`
`532
`
`FOCUS POSITION
`
`APPL-1007 / Page 16 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 15 of 29
`
`US 7,859,588 B2
`
`YyV1NT14S9
`
`YOSSs00Nd
`
`t|WHGOW|PAWINTISO||eyii
`
`~=d6
`
`009
`
`GSb‘SldVSL‘Sid
`
`vyo9
`
`809
`
`009
`
`909
`
`APPL-1007 / Page 17 of 49
`
`APPL-1007 / Page 17 of 49
`
`
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 16 of 29
`
`US 7,859,588 B2
`
`618 610628
`620
`
`626
`
`622614
`
`622
`
`626
`
`624
`
`620614
`
`622618
`
`610
`
`FIG.16A
`
`616
`
`FIG.16B
`
`APPL-1007/ Page 18 of 49
`
`APPL-1007 / Page 18 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 17 of 29
`
`US 7,859,588 B2
`
`APPL-1007 / Page 19 of 49
`
`APPL-1007 / Page 19 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 18 of 29
`
`US 7,859,588 B2
`
`GPS LOCATION OF THE CAMERA
`PROVIDED BY THE GPS IN THE CAMERA
`
`|
`
`CAMERA POINTING DIRECTION PROVIDED BY
`THE ELECTRONIC COMPASSIN THE CAMERA
`
`DISTANCE OFFSETS FROM THE CAMERA TO PORTIONS
`OF THE SCENE PROVIDED BY THE RANGEMAP
`
`ANGULAR OFFSET FROM THE CAMERA PROVIDED
`FROM THE LOCATIONIN THE FIELD OF VIEW
`
`GPS LOCATIONS FOR PORTIONS OF THE SCENE ARE
`DETERMINED BY ADDING DISTANCE OFFSETS AND THE
`ANGULAR OFFSETS TO THE GPS LOCATION AND
`
`POINTING DIRECTION OF THE CAMERA
`
`A GPS LOCATIONS FOR THE OBJECTS IN THE SCENE
`ARE STOREDIN METADATA OR DISPLAYED AS LABELS
`IN A GPS LOCATION MAP
`
`750
`
`752
`
`154
`
`756
`
`158
`
`760
`
`FIG. 18
`
`APPL-1007/ Page 20 of 49
`
`APPL-1007 / Page 20 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 19 of 29
`
`US 7,859,588 B2
`
`START UP
`
`CAMERA POWER ON
`INITIALIZATION
`
`1100
`
`SET FIRST AND SECOND CAPTURE
`UNITS TO DEFAULT ZOOM POSITIONS
`
`1102
`
`CAPTURE AND DISPLAY FIRST AND
`SECOND PREVIEW IMAGES USING FIRST
`AND SECOND CAPTURE UNITS
`
`1104
`
`+1124
`
`NO
`
`
`1106
`
`
`SET SELECTED
`SELECT
`CAPTURE UNIT
`FIRST OR SECOND
`
`PREVIEW
`AS PRIMARY
`
`IMAGE
`CAPTURE UNIT
`
` 1108
`
` IS S1
`YES
`
`SHUTTER BUTTON
`PRESSED
`
`SET DEFAULT CAPTURE UNIT
`AS PRIMARY CAPTURE UNIT
`
`SET NON-PRIMARY CAPTURE UNIT
`AS SCENE ANALYSIS CAPTURE UNIT
`
`1114
`
`PREVIEW MODE
`
`FIG. 19
`
`APPL-1007 / Page 21 of 49
`
`APPL-1007 / Page 21 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 20 of 29
`
`US 7,859,588 B2
`
`
`
`PREVIEW MODE
`
`
`
`SCENE ANALYSIS CAPTURE UNIT
`ANALYZES SCENE
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERSUTILIZING SCENE
`ANALYSIS CAPTURE UNIT DATA
`
`
`
`
`
` CONDITIONS
`CHANGED
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`SCENE ANALYSIS CAPTURE UNIT
`ANALYZES SCENE
`
`FIG. 20
`
`APPL-1007/ Page 22 of 49
`
`APPL-1007 / Page 22 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 21 of 29
`
`US 7,859,588 B2
`
`
`
`PREVIEW MODE
`WITH THRESHOLD
`
`
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERSUTILIZING SCENE
`ANALYSIS CAPTURE UNIT DATA
`
`
`
`
`
`
`
`
`
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`
`
`SCENE
`CONDITION CHANGE>X
`OR STABLE
`
`FIG. 21
`
`APPL-1007/ Page 23 of 49
`
`APPL-1007 / Page 23 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 22 of 29
`
`US 7,859,588 B2
`
`ENHANCED
`PREVIEW MODE
`
`SET SCENE ANALYSIS CAPTURE UNIT
`ZOOM POSITION RELATIVE TO PRIMARY
`CAPTURE UNIT ZOOM POSITION
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`SET PRIMARY CAPTUREUNIT
`PARAMETERSUTILIZING SCENE
`ANALYSIS CAPTURE UNIT DATA
`
`1400
`
`1402
`
`1404
`
`1406
`
`CAPTURE PREVIEW IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`1408~
`
`ANALYZE SCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS DATA
`
`1410
`
`1412
`
`1414
`
`1416~.
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERSUTILIZING RESULTS
`OF THE SCENE ANALYSIS
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`
`
`
`
`
`
`ANALYZE SCENEUTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS DATA
`
`1418
`
`SCENE
`CONDITION CHANGE>X
`OR STABLE
`
`FIG. 22
`
`APPL-1007/ Page 24 of 49
`
`APPL-1007 / Page 24 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 23 of 29
`
`US 7,859,588 B2
`
`RETURN TO PREVIEW
`
`
` BUTTON
`PRESSED
`
`?
`
`
`
`ZOOM
`
`
`POSITION '
`
`PRIMARY CAPTURE
`UNIT ZOOM
`
`
`
`RANGE
`
`SET CURRENT SCENE ANALYSIS
`AND PRIMARY CAPTURE UNITS AS
`PRIMARY AND SCENE ANALYSIS
`CAPTURE UNITS RESPECTIVELY
`
`
`
`SET PRIMARY CAPTURE UNIT ZOOM
`POSITION TO SELECTED ZOOM POSITION
`
`
`
`RETURN TO PREVIEW
`
`FIG. 23
`
`APPL-1007/ Page 25 of 49
`
`APPL-1007 / Page 25 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 24 of 29
`
`US 7,859,588 B2
`
`CAPTURE
`
`IS $1
`SHUTTER BUTTON
`
`1600
`PRESSED NO
`
`
`YES
`
`RETURN TO PREVIEW
`
`1602
`
`1604
`
`1608~.
`
`1610
`
`CAPTURE PREVIEW IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`ANALYZE SCENEUTILIZING
`CAPTURED PREVIEW IMAGE
`
`
`
`
`1606
`
` ANALYSIS
`
`COMPLETE
`
`SET PRIMARY CAPTURE UNIT PARAMETERS
`UTILIZING RESULTS OF THE ANALYSIS
`
`
`
`
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`
`IS $2
`1612
`
`
`
`IS FOCUS/
`
`SHUTTER BUTTON
`EXPOSURE LOCK
`
`
`PRESSED
`SET
`
`
`
`YES
`
`1614
`
`CAPTURE PRIMARY IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`RETURN TO PREVIEW
`
`FIG. 24
`
`APPL-1007/ Page 26 of 49
`
`APPL-1007 / Page 26 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 25 of 29
`
`US 7,859,588 B2
`
`CAPTURE
`
`1702
`
`1704
`
`1706
`
`1708
`
`1710
`
`RETURN TO PREVIEW
`
`IS S1
`
`NO
`
`
`1700
`
`
`
`
`SHUTTER BUTTON
`
`PRESSED
`
`
`
`YES
`
`
`CAPTURE PREVIEW IMAGE
`
`FROM PRIMARY CAPTURE UNIT
`
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`ANALYZE SCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS IMAGES
`
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERSUTILIZING RESULTS
`
`OF THE SCENE ANALYSIS
`
`
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`
`1712 IS $2
`
`IS FOCUS/
`
`
`SHUTTER BUTTON
`EXPOSURE LOCK
`PRESSED
`SET
`
`
`
`YES
`
`
`
`
`1714
`
`CAPTURE PRIMARY IMAGE
`FROM PRIMARY CAPTUREUNIT
`
`RETURN TO PREVIEW
`
`FIG. 25
`
`APPL-1007/ Page 27 of 49
`
`APPL-1007 / Page 27 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 26 of 29
`
`US 7,859,588 B2
`
`RETURN TO PREVIEW
`
`CAPTURE
`
`PRESSED
`
`YES
`
`CAPTURE PREVIEW IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`ANALYZE SCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS IMAGES
`
`SET PRIMARY CAPTURE UNIT PARAMETERS
`UTILIZING RESULTS OF THE SCENE ANALYSIS
`
`SET SCENE ANALYSIS CAPTURE UNIT
`AS A SECONDARY CAPTURE UNIT
`
`1802
`
`1804
`
`1806
`
`1808
`
`1810
`
`1812~.
`
`SET SECONDARY CAPTURE UNIT PARAMETERS
`UTILIZING RESULTS OF THE SCENE ANALYSIS
`
`
`1814
`$$
`
`SHUTTER BUTTON
`PRESSED
`
`
`YES
`
`NO
`
`CAPTURE PRIMARY IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`CAPTURE AN AUGMENTATION IMAGE
`FROM SCENE ANALYSIS CAPTURE UNIT
`
`PRODUCE AN ENHANCED IMAGE FROM
`THE PRIMARY AND AUGMENTATION IMAGES
`
`1816
`
`1818
`
`1820
`
`FIG. 26
`
`APPL-1007/ Page 28 of 49
`
`APPL-1007 / Page 28 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 27 of 29
`
`US 7,859,588 B2
`
`
`
`
`153|/ \| 152
`
`ata
`| zak
`E “Nf
`j
`\
`
`157 156
`
`,
`
`158
`
`emHUTTEFt
`
`
`
`181 170)|171180
`
`
`
`X9
`
`b
`
`Xa
`
`FIG. 27
`(PRIOR ART)
`
`APPL-1007/ Page 29 of 49
`
`APPL-1007 / Page 29 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 28 of 29
`
`US 7,859,588 B2
`
`START
`
`197
`
`SET LENS TO MID FOCUS AND
`LOAD LOW PASSFILTER
`
`CALCULATE FOCUS VALUE(FV)
`AS IMAGE IS READ OUT
`
`ADJUST STEPPER MOTOR TO
`NEXT NEAR FOCUS POSITION
`
`198
`
`198
`
`199
`
`GET FV OF NEXT IMAGE CAPTURED
`
`STEP IN OPPOSITE
`
`WITHNEWFOCUSSETTING — DIRECTION
`
`COMPARE TO PREVIOUS
`WEIGHTED FOCUS VALUES
`
`FV INCREASES
`
`STEP IN SAME DIRECTION
`UNTIL FV DECREASES
`
`LOAD HIGH PASS FILTER
`AND MEASUREFV USING
`SAME FOCUS POSITION
`
`STEP IN SAME DIRECTION
`UNTIL FV DECREASE
`
`BACK UP AND HOLD AT
`HIGHEST FV FOCUSPOSITION
`
`FOCUS OK
`
`FIG. 28
`(PRIOR ART)
`
`APPL-1007/ Page 30 of 49
`
`APPL-1007 / Page 30 of 49
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 29 of 29
`
`US 7,859,588 B2
`
`FOCUS
`~~ ADJUSTMENT
`STEPS
`
`1
`
`A
`
`FOCUS
`VALUE _
`
`3
`
`
`
`POSITION
`
`p—
`
`FIG. 29
`(PRIOR ART)
`
`APPL-1007 / Page 31 of 49
`
`APPL-1007 / Page 31 of 49
`
`

`

`US 7,859,588 B2
`
`1
`METHOD AND APPARATUS FOR
`OPERATING A DUAL LENS CAMERA TO
`AUGMENTAN IMAGE
`
`FIELD OF THE INVENTION
`
`The present invention relates to a digital camera that pro-
`duces digital imagefiles and, more particularly, to a digital
`camerathat uses multiple lenses and image sensorsto provide
`an improved imaging capability.
`
`BACKGROUND OF THE INVENTION
`
`Currently, most digital cameras use a zoom taking lens and
`asingle color image sensorto capturestill and motion images.
`The captured images are then processed to produce digital
`image files, which are stored in a digital memory in the
`camera. The digital image files can then be transferred to a
`computer, displayed, printed, and shared via the Internet.
`In order to capture sharp images of moving subjects, a
`digital camera needs to provide a precise automatic lens
`focusing system (i.e., an autofocus system). The autofocus
`system must be capable of quickly obtaining the correct focus
`in order to minimize the “shutter delay” between the time the
`shutter button is pressed andthe still image is captured. The
`autofocus system must also work in a continuous image cap-
`ture mode wherein video imagesare captured.Forinstance, in
`a video modethe focus should be adjusted in real-time while
`video images are being continuously captured.
`Manydigital cameras and scanners capture images using
`an image sensor and a taking lens system with an adjustable
`focus. Typically, the focus distance of such an adjustable
`focus taking lens system can automatically be set to one of a
`plurality of different settings by sensing, control, and drive
`systems, which are adapted to provide optimal focus of what
`is determined to bea subject area in a scene. Lens systemsthat
`provide automatically adjustable focus settings based on a
`focus measurement and an adjustable focus lens are referred
`to herein as autofocus systems. Digital cameras typically use
`one of two types of autofocus systems: a rangefinder system
`and a “through-the-lens” focus system.
`A rangefinder system uses rangefinding sensors such as a
`sonic rangefinder or a dual lens rangefinder to determine the
`distance from a camera to one or more portions of a scene
`within a field of view of the rangefinder system. A sonic
`rangefinder measures the phase offset between a projected
`sonic signal anda reflected sonic signalto infer the distance to
`objects in the scene. Dual lens rangefinders contain two
`lensesthat are separated by a distance along with two match-
`ing sensor areas that capture matched pairs of images. Dual
`lens rangefinders are commonly used on digital cameras in
`the form of dual lens rangefinder modules which contain two
`lenses separated by a distance along with two matching sen-
`sor areas that capture matchedpairs of low resolution images.
`Common dual lens rangefinder-based autofocus systems
`include active and passive systems. Active systemsactively
`project light onto the scene, while passive systems work with
`the available light from the scene. Duallens rangefinder mod-
`ules can be purchased from Fuji Electric in several models
`such as the FM6260W. A dual lens rangefinder module for
`optical apparatus such as a camera is described in U.S. Pat.
`No. 4,606,630, which was issued to Harukiet al. on Aug. 19,
`1986 (and assigned to Fuji Electric). According to the
`descriptionofthepriorart in this patent, matchedpairs of low
`resolution images are analyzed for correlation between the
`two images to determine the offset between the two images
`caused by the separation between the two lenses.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`A diagram illustrative of the principle of the operation of a
`conventional rangefinder is shown herein in FIG. 27. In that
`diagram, light from an object 151 is incident on two small
`lenses 152 and 153 which have a sufficiently short focal
`length f that light rays received from the object through dif-
`ferent spaced paths 154 and 155 produce corresponding
`spaced images 157 and 158 in a focal plane 156 which is
`commonto the lenses 152 and 153. When the object 151 is at
`an infinite distance, the centers of the images 157 and 158 are
`located at reference positions 170 and 180 in FIG. 27, but
`whenthe object 151 is located at a closer distance, the centers
`ofthe imagesare shifted apart to positions 171 and 181. Ifthe
`distance by which the images 157 and 158are shifted from the
`reference positions 170 and 180 are designated x, and x,,
`respectively, then the total shift x may be expressedas fol-
`lows:
`
`X=X4x7=b-fld
`
`Thus, the distance d to the object 151 can be measured by
`d=b-f/x.In this case, b is the distance between the optical axes
`of the small lenses, that is, the base length. To obtain the
`shifted amounts x, and x,, or the sum x of both, two optical
`sensor arrays 190 and 191 are providedin the focal plane 156
`as shown in FIG. 27. These optical sensor arrays each com-
`prise a plurality of optical sensors, for instance CCD devices,
`and an analog photoelectric signal is generated by each opti-
`cal sensor correspondingtothelight intensity at the portion of
`the image whichis incident on the sensor. Haruki et al. shows
`a conventionalcircuit, as well as a higher speed rangefinding
`circuit accordingto the patent, for obtaining the sum x of the
`shifted distances by comparing two imagesignal trains com-
`prising the digital imagesignals from theleft andrightoptical
`sensorarrays.
`
`Basically, the offset information x is used along with the
`lens separation distance b and the focal length f to calculate
`the distanced to the scene by triangulation. The calculated
`distance d to the scene is used to guide the positioning of an
`adjustable focus lens to produce the best image quality. As
`known in the prior art, this adjustment may be based on a
`calibration curve established between the distance to the
`
`scene as measuredbythe dual lens rangefinder module and a
`series of best focused images as produced bya “through the
`lens” autofocus system. The calibration curve is stored as an
`equation or a look-up table in a microprocessorin the camera.
`Rangefinder-based autofocus systems have the advantage
`ofbeing very fast, some having a responsetime that can be in
`the range of 0.01-0.05 second. However, the focus quality
`produced by some rangefinder-based autofocus systems can
`vary whentheyare usedin different operating conditions. For
`example, sonic autofocus systems cannot focus through a
`glass window as the glass stops the projected sonic signal,
`thereby causing the autofocus system to focus onto the glass.
`In the case of a dual lens rangefinder autofocus system, the
`accuracy of dual lens rangefinders are typically influenced by
`changes in environmental conditions such as temperature
`and/or humidity. The problem with dual lens rangefinder
`modules is that
`the calibration between the dual
`lens
`rangefinder module and the adjustable focus lens position is
`not stable within the normal operating environmentfordigital
`cameras. Environmental conditions such as changes in tem-
`perature and humidity can cause the distanceto the portion of
`the scene as measured by the dual lens rangefinder module to
`change by over 10%. In addition, the measured position ofthe
`adjustable focus taking lens in the adjustable focus taking
`lens system is prone to environmentally induced changes as
`well so that inaccuracies are producedin the control system
`APPL-1007/ Page 32 of 49
`
`APPL-1007 / Page 32 of 49
`
`

`

`US 7,859,588 B2
`
`3
`lens
`for the adjustable focus lens. Consequently, dual
`rangefinder modulesare not typically used independently for
`autofocus in digital cameras but instead are used as a rough
`focus adjustmentthat is supplementedbya “through the lens”
`autofocus system.
`Alternatively,
`the “through-the-lens” autofocus system
`determines a focus state through an analysis of a series of
`autofocus images captured with the adjustable focus lens
`system positioned at a plurality of different focus distances.
`For example, in a typical “through-the-lens” autofocus sys-
`tem a plurality of autofocus images(e.g., 5-20) are captured
`with the adjustable focus lens ina series of different positions
`in a so-called “hill climb” method. This type of autofocusis
`known as “hill climbing” autofocus because it generates a
`sequenceofvalues that increase in level until they pass over a
`peak, i.e., a “hill”. In other words, the lens focus position is
`adjusted automatically until the contrast of the edge detail in
`the image,ora particular area ofthe image, is maximized. For
`instance, the contrast present in each ofthe autofocus images
`is compared and the autofocus image with the greatest con-
`trast is deemed to have been captured with the best focus
`conditions (often the best focuslens position is further refined
`by interpolating the contrast values between images).
`In order to decrease focusing response time withoutsacri-
`ficing focusing precision, it is commonto usefilters to sepa-
`rate not only the higher frequency componentof the video
`signal, but also the lower frequency component. For example,
`a lens may be quickly driven in coarse adjustmentsteps in a
`low frequency range furthest from the maximum focus, and
`then driven in finer adjustmentsteps ina high frequency range
`nearer to the maximum focus. A flow diagram of a conven-
`tional “hill climbing”contrast autofocus algorithm is shown
`in FIG. 28. This algorithm uses the “hill climbing”contrast
`autofocus method discussed above and shownin the diagram
`of FIG. 29, which illustrates the relationship between the
`focus value obtained from thefilters and the lens position. In
`FIG.29, the abscissa indicates the focusing position of a lens
`along a distance axis, the ordinate indicates the focusing
`evaluation value, and the curves A and B indicate the focusing
`evaluation values for high and low frequency components,
`respectively, relative to a particular in-focus position P.
`Referring to the flow diagram of FIG. 28, the best starting
`point for the algorithm depends on the hyperfocal distance of
`the currentlens setting, which is a function ofthe focal length
`setting and the f-number. A distance of about 2 meters is
`typically a goodstarting point. Then a low frequency band-
`pass filter is loaded (stage 197) and the focus values are read
`out. The algorithm employs a comparisonstage 198to set the
`direction of lens adjustment toward increasing focus values,
`and to determine whenthe lensis stepped over the “hill”. The
`depth of field, which dependson the present focal length and
`f-number, sets the numberofsteps, i.e., the next near focus
`position, which should be taken before capturing the next
`frame when using the low frequency bandpassfilter. Once the
`peak of the hill is passed (curve B in FIG. 29), a high fre-
`quency bandpass filter is loaded (stage 199), and the lens is
`movedin the opposite direction until the peak of the higher
`“hill” is found (curve A in FIG. 29). The peak focus value may
`use either the weighted average or peak value from numerous
`pixels.
`“Through-the-lens” autofocus systems are very accurate
`since they measure focus quality directly from autofocus
`images captured with the high quality taking lens. Unfortu-
`nately, “through-the-lens” autofocus systems can be rela-
`tively slow in determining a focus setting due to the large
`numberof autofocus images that must be captured and com-
`
`30
`
`45
`
`4
`pared. For example, “through-the-lens” autofocus systems
`can take as long as 0.5-2.0 seconds to determine focus con-
`ditions.
`
`Accordingly, in some digital cameras, the two types of
`autofocus systems are used together in a hybrid system in
`which the rangefinder based autofocus system is used to
`providea fast estimation of the adjustable focus lens location
`that is then followed by the use of the “through-the-lens”
`autofocus system to refine the focus setting. For example,
`USS. Pat. No. 6,864,474, entitled “Focusing Apparatus for
`Adjusting Focus of an Optical Instrument” and which issued
`Mar. 8, 2005 in the name of Misawa, describes the coordi-
`nated use of a rangefinder-based autofocus system with a
`“through-the-lens” autofocus system. In Misawa, the focus
`position of the adjustable focus taking lens is determined by
`both the rangefinder-based autofocus
`system and the
`“through-the-lens”
`autofocus
`system. The
`difference
`between the adjustable focus taking lens position determined
`by the rangefinder-based autofocus system and the adjustable
`focus taking lens position determined by the “through-the-
`lens” autofocus system is stored for future reference. In sub-
`sequent image capture episodes, the stored difference infor-
`mation is used to refine the number of autofocus images
`captured and analyzed by the “through-the-lens” autofocus
`system in the “hill climb” method to determine the adjustable
`focus lens position for best focus, thereby reducing the num-
`ber of autofocus images captured and processed in cases
`where the rangefinder system is accurate and increasing the
`numberof autofocus images captured and processed in cases
`where the rangefinder is inaccurate. However, the m