(12) United States Patent
`Parulski et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,859,588 B2
`*Dec. 28, 2010
`
`US007859588B2
`
`(54) METHOD AND APPARATUS FOR
`OPERATING ADUAL LENS CAMERATO
`AUGMENT ANIMAGE
`(75) Inventors: Kenneth A. Parulski, Rochester, NY
`(US); Wilbert E. Janson, Jr.,
`Shortsville, NY (US); John N. Border,
`Walworth, NY (US); John R. Fredlund,
`Rochester, NY (US); Joseph A. Manico,
`Rochester, NY (US); Robert J. Parada,
`Jr., Rochester, NY (US)
`(73) Assignee: Eastman Kodak Company, Rochester,
`NY (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 882 days.
`This patent is Subject to a terminal dis
`claimer.
`(21) Appl. No.: 11/684,025
`y x- - -
`9
`Mar. 9, 2007
`
`(*) Notice:
`
`(22) Filed:
`
`(65)
`
`Prior Publication Data
`US 2008/0218611 A1
`Sep. 11, 2008
`(51) Int. Cl
`(2006.01)
`H04N 5/232
`(2006.01)
`GO3B I3/00
`(2006.01)
`GO3B 7/OO
`(52) U.S. Cl. ......................................... 348/349, 396/79
`(58) Field of Classification Search ................. 348/349
`396/72, 79, 100
`See application file for complete search history.
`References Cited
`U.S. PATENT DOCUMENTS
`
`(56)
`
`4,606,630 A
`5,668,597 A
`5,874,994 A
`6,441,855 B1
`
`8, 1986 Haruki et al.
`9, 1997 Parulski et al.
`2f1999 Xie et al.
`8, 2002 Omata et al.
`
`8/2003 Yu et al.
`6,611,289 B1
`3/2005 Misawa
`6,864,474 B2
`7,676,146 B2 * 3/2010 Border et al. ................. 396,80
`2003/0020814 A1
`1/2003 Ono
`2003.0160886 A1
`8, 2003 Misawa et al.
`2005, 0046738 A1
`3, 2005 Sato
`2005/0275747 A1 12/2005 Nayar et al.
`2006/01873 12 A1
`8/2006 LabaZiewicz et al.
`
`(Continued)
`FOREIGN PATENT DOCUMENTS
`
`EP
`
`O 858 208
`
`8, 1998
`
`(Continued)
`Primary Examiner Kelly L Jerabek
`(74) Attorney, Agent, or Firm Thomas J. Strouse; Peyton C.
`Watkins
`
`(57)
`
`ABSTRACT
`
`An electronic camera for producing an output image of a
`scene from a captured image signal includes a first imaging
`stage comprising a first image sensor for generating a first
`sensor output and a first lens for forming a first image of the
`scene on the first image sensor, and a second imaging stage
`comprising a second image sensor for generating a second
`sensor output and a second lens for forming a second image of
`the scene on the second image sensor. The sensor output from
`the first imaging stage is used as a primary output image for
`forming the captured image signal and the sensor output from
`the second imaging stage is used as a secondary output image
`for modifying the primary output image, thereby generating
`an enhanced, captured image signal.
`
`22 Claims, 29 Drawing Sheets
`
`zoospositioSETTo-500
`DEFAUTPOSTONWHEN
`CAMERA SPOWERED ON
`
`
`
`02>2ooNyes
`
`POSTION
`xx
`
`SETFIRSTMAGECAFTURE
`McDUTOCAPTURIMAGES
`&SECONBIMAGECAFTURE
`NODULE FORAUTOFOCUS
`
`CAPTUREANDDSPAY
`FREEWMASES HEE
`PERFORMENGONNU0US
`AUO-F00USUSNGSECOND
`AGECARTURODULE
`
`-504 524s. STSECONDIASCAPTURE
`MODULEO CAFTUREIMAGES
`8FIRSTMASECAPTRE
`BODULEFORAUTOFOCUS
`-506 526-capTUREADISPLAY
`PREVIEWMAGES WHILE
`PERFORMINCONTINUS
`AUTOF00SUSINGFIRST
`AGECAPTURM)
`
`528-1zoo
`B6 NYES
`PRESSED
`
`ZOOY-508
`YES1 B6
`PESSED
`NO
`NO
`ENSTERButtos
`HESHERUros -510 530s.
`PRESSED, CAPTUREAPRIMARY
`PRESSED, CAPTUREAPRIMARY
`STILLMAGEUSINSSECOND
`STRAGEUSINGFIRST
`MAGECATUREMOBLESS
`MAGE CAPREMODULEST
`TPRiMARYFOCUSPOSITIN
`TOPRIMARY FOCUSPOSTION
`
`CATUREASECONDARY
`STILLIMASEUSING SECON)
`IMAGECAPTURESOBULE
`SET TOSEC0SDARY
`FOCUSPOSITION
`
`USESECONARYSTILL
`iMAGETCENHANCETH
`Er?hot of THE
`PRIMARY IMAGE
`
`-512 532-
`
`-514 534-
`
`CAPTUREASEONDARY
`STLE.FMAGEUSNG FIRST
`IMAGECAPTUREMODULE
`SETOSECONARY
`FOCUSPOSION
`
`USESONARY STILL
`IMAGETOENHANCETHE
`DEPTH OFFIDOF THE
`PRIMARYMASE
`
`APPL-1005 / Page 1 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`US 7,859,588 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`FOREIGN PATENT DOCUMENTS
`
`2006/0193509 A1
`8/2006 Criminisi et al.
`JP
`2008/0013941 A1* 1/2008 Daley ......................... 396,121 W
`2008/02186 12 A1* 9, 2008 Border et al. ............... 348,262
`2008/02186 13 A1* 9, 2008 Janson et al. ............... 348,262
`
`* cited by examiner
`
`2005045511
`o'E'
`
`2, 2005
`58.
`
`APPL-1005 / Page 2 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`APPL-1005 / Page 3 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 2 of 29
`
`US 7,859,588 B2
`
`
`
`S.
`
`S
`vine
`
`S
`V
`
`S.
`
`GN
`g
`L
`
`S
`g
`U
`
`APPL-1005 / Page 4 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 3 of 29
`
`US 7,859,588 B2
`
`ZOOMPOSITION SET TO
`DEFAULTPOSITION WHEN
`CAMERA SPOWERED ON
`
`USER INPUT
`ZOOMPOSITION
`
`100
`
`101
`
`ZOOM
`POSITION
`>X?
`
`SET FIRST MAGE CAPTURE
`STAGE TO CAPTURE IMAGES
`& SECOND IMAGE CAPTURE
`STAGE FOR AUTOFOCUS
`
`SET SECONDMAGE CAPTURE
`STAGE TO CAPTURE IMAGES
`8 FIRST IMAGE CAPTURE
`STAGE FOR AUTOFOCUS
`
`CAPTURE AUTOFOCUS
`IMAGES WITH SECOND
`IMAGE CAPTURE STAGE
`
`106 126
`
`CAPTURE AUTOFOCUS
`IMAGES WITH FRST
`IMAGE CAPTURE STAGE
`
`FOCUS FIRST MAGE
`CAPTURE STAGE
`
`108 128
`
`FOCUS SECOND IMAGE
`CAPTURE STAGE
`
`CAPTURE PREVIEW IMAGES
`WITHFIRST IMAGE CAPTURE
`STAGE AND DISPLAY
`
`110 130
`
`CAPTURE PREVIEW IMAGES
`WITH SECOND IMAGE CAPTURE
`STAGE AND DISPLAY
`
`
`
`ZOOM
`BUTTON
`PRESSED
`2
`NO
`
`
`
`ZOOM
`BUTTON
`PRESSED
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`6APTURN(114
`BUTTON
`PRESSED
`?
`YES
`CAPTURE DIGITAL STILL
`IMAGE WITH FIRST
`IMAGE CAPTURE STAGE
`
`
`
`134
`
`
`
`CAPTURE
`BUTTON
`PRESSED
`?
`YES
`CAPTURE DIGITAL STILL
`IMAGE WITH SECOND
`IMAGE CAPTURE STAGE
`
`116 136
`
`FIG. 3
`
`APPL-1005 / Page 5 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`APPL-1005 / Page 6 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`APPL-1005 / Page 7 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 6 of 29
`
`US 7,859,588 B2
`
`CAPTURE ASERIES OF IMAGE SETS WITH OBJECTS
`A KNOWN DISTANCES WITH THE SHORTER
`FOCAL LENGTH FIRST MAGE CAPTURE STAGE AND
`THE LONGER FOCAL LENGTH SECOND IMAGE CAPTURE
`STAGE ANDA SERIES OF FOCUS LENSPOSITIONS
`
`THE AUTOFOCUS IMAGE FROM THE LOWER
`FOCAL LENGTH FIRST IMAGE STAGE IS CROPPED AND
`UPSAMPLED SO THAT CORRESPONDING FEATURES
`IN THE TWOAUTOFOCUS MAGESSPAN
`THE SAMENUMBER OF PXELS
`
`CORRELATE THE IMAGES FROM THE SECOND IMAGE
`CAPTURE STAGE TO CORRESPONDINGPORTIONS OF
`THE IMAGES FROM THE CROPPED AND UPSAMPLED
`IMAGE FROM THE FIRS MAGE CAPTURE STAGETO
`DEERMINE THE PXEL OFFSET BE WEEN
`THE IMAGES IN EACH IMAGE SET
`
`STORE THE DATA OF PIXEL OFFSET BETWEEN
`IMAGES IN EACH IMAGE SET VERSUSKNOWN
`DISTANCETO OBJECTSASAN AUTOFOCUS
`RANGEFINDER CALIBRATION CURVE
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`300
`
`3O2
`
`304
`
`306
`
`FIG. 6
`
`APPL-1005 / Page 8 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 7 Of 29
`
`US 7,859,588 B2
`
`CAPTURE A SERIES OF IMAGE SETS WITH OBJECTS
`A KNOWN DISTANCES WITH THE FIRST MAGE
`CAPTURE STAGE AND THE SECOND IMAGE CAPTURE
`STAGE WHEREN AUTOFOCUS IS DONEBY THE
`"HILL CLIMBMETHOD"FOREACH IMAGE
`
`400
`
`COMPARE THE FOCUSLENSPOSITIONS FOR THE TWO -402
`IMAGE CAPTURESTAGES VERSUS THE DISTANCETO
`THE FOCUSED OBJECTS IN THE IMAGE SETS
`
`STORE THE DATA OF FOCUS LENS POSITIONS OF
`THE FIRST IMAGE CAPTURE STAGE VERSUS THE FOCUS
`LENS POSITIONS OF THE SECOND IMAGE CAPTURE
`STAGE FOR THE SAME DISTANCETO FOCUSED
`OBJECTS N THE IMAGES ASAN AUTOFOCUS
`"HILL CLMBMETHOD" CALIBRATION CURVE
`
`
`
`404
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 7
`
`APPL-1005 / Page 9 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 8 of 29
`
`US 7,859,588 B2
`
`ZOOMPOSITION SET TO DEFAULT
`POSITION WHEN CAMERA SPOWERED ON
`USERINPUT ZOOMPOSITION -101
`
`100
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ZOOM
`POSITION
`>X?
`
`SET FIRST IMAGE CAPTURE
`MODULE TO CAPTURE IMAGES
`& SECOND IMAGE CAPTURE
`MODULE FOR AUTO FOCUS
`
`SET SECOND IMAGE CAPTURE
`MODULE TO CAPTURE IMAGES
`& FIRST IMAGE CAPTURE
`MODULE FOR AUTOFOCUS
`
`CAPTURE AUTOFOCUS IMAGES
`
`CAPTURE AUTOFOCUS IMAGES
`
`AUTOFOCUS FIRST MAGE
`CAPTURE MODULE
`
`CAPTURE PREVIEW IMAGES
`WITH FRST MAGE CAPTURE
`MODULE AND DISPLAY
`
`ZOOM BUTTON
`PRESSED
`
`CAPTURE BUTTO
`PRESSED 2
`
`YES
`
`
`
`
`
`
`
`
`
`AUTOFOCUS SECOND IMAGE
`CAPTURE MODULE
`
`CAPTURE PREVIEW IMAGES
`WITH SECOND IMAGE CAPTURE
`MODULE AND DISPLAY
`
`
`
`ZOOM BUTTON
`PRESSED 2
`
`
`
`CAPTUREBUTTON
`PRESSED 2
`
`CAPTUREVIDEO IMAGE WITH
`FIRST IMAGE CAPTURE MODULE
`
`CAPTUREVIDEO IMAGES WITH
`SECOND IMAGE CAPTURE MODULE
`
`CHECK IMAGE FOR
`FOCUS QUALTY
`120
`
`
`
`NEED TO
`REFOCUS 2
`
`
`
`FIG. 8
`
`CHECK IMAGE FOR
`FOCUS QUALITY
`
`
`
`
`
`NEED TO
`REFOCUS
`
`APPL-1005 / Page 10 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 9 Of 29
`
`US 7,859,588 B2
`
`OO
`S.
`V
`
`6
`
`
`
`
`
`
`
`
`
`
`
`
`
`APPL-1005 / Page 11 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28
`
`9
`
`2010
`
`Sheet 10 of 29
`
`US 7,859
`588 B2
`
`9
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`APPL-1005 / Page 12 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 11 of 29
`
`859,588 B2
`US 7
`9
`
`087
`
`Z87
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`APPL-1005 / Page 13 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`APPL-1005 / Page 14 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 13 Of 29
`
`US 7,859,588 B2
`
`99
`
`79
`Z9
`
`r
`O
`
`ZG
`
`09
`
`99
`
`/9
`
`69
`
`89
`
`ÅRHOWEWN
`
`06 |
`
`99
`
`|×nw Ei – III
`) ||
`
`C|NOOESWOOZ
`
`GNO OES
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`APPL-1005 / Page 15 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 14 of 29
`
`US 7,859,588 B2
`
`ZOOMPOSITION SET TO
`DEFAULT POSITION WHEN
`CAMERA SPOWERED ON
`
`500
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ZOOM
`POSITION
`>X?
`
`SET FIRST IMAGE CAPTURE
`MODULE TO CAPTURE IMAGES
`& SECOND IMAGE CAPTURE
`MODULE FOR AUTOFOCUS
`
`CAPTURE AND DISPLAY
`PREVIEW IMAGES WHILE
`PERFORMING CONTINUOUS
`AUTO-FOCUS USING SECOND
`IMAGE CAPTURE MODULE
`
`
`
`ZOOM
`BUTTON
`PRESSED
`
`
`
`NO
`WHEN SHUTTER BUTTONIS
`PRESSED, CAPTURE APRIMARY
`STILLMAGE USING FIRST
`IMAGE CAPTURE MODULE SET
`TO PRIMARY FOCUS POSITION
`
`510 530
`
`CAPTURE ASECONDARY
`STILLIMAGE USING SECOND
`IMAGE CAPTURE MODULE
`SET TO SECONDARY
`FOCUS POSITION
`
`USE SECONDARY STILL
`IMAGE TO ENHANCE THE
`DEPTH OF FIELD OF THE
`PRIMARY IMAGE
`
`512 532
`
`514 534
`
`SET SECOND IMAGE CAPTURE
`MODULE TO CAPTURE IMAGES
`8 FIRST MAGE CAPTURE
`MODULE FOR AUTO-FOCUS
`
`CAPTURE AND DISPLAY
`PREVIEW IMAGES WHILE
`PERFORMING CONTINUOUS
`AUTOFOCUS USING FIRST
`IMAGE CAPTURE MODULE
`
`
`
`ZOOM
`BUTTON
`PRESSED
`p
`
`
`
`
`
`
`
`
`
`
`
`
`
`WHEN SHUTTER BUTTONIS
`PRESSED, CAPTURE APRIMARY
`STILLIMAGE USING SECOND
`IMAGE CAPTURE MODULE SET
`TO PRIMARY FOCUS POSITION
`
`CAPTUREASECONDARY
`STILL MAGE USING FIRST
`IMAGE CAPTURE MODULE
`SET TO SECONDARY
`FOCUSPOSITION
`
`USE SECONDARY STILL
`IMAGE TO ENHANCE THE
`DEPTH OF FIELD OF THE
`PRIMARY MAGE
`
`F.G. 14
`
`APPL-1005 / Page 16 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 15 Of 29
`
`US 7,859,588 B2
`
`
`
`| HOSSEO
`-OR-Id
`
`009909
`
`709
`
`809
`
`009
`
`APPL-1005 / Page 17 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 16 of 29
`
`US 7,859,588 B2
`
`
`
`CO
`CN
`CO
`
`
`
`879), "50|-||
`
`CN
`o
`
`APPL-1005 / Page 18 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 17 Of 29
`
`US 7,859,588 B2
`
`
`
`&
`
`APPL-1005 / Page 19 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 18 Of 29
`
`US 7,859,588 B2
`
`GPS LOCATION OF THE CAMERA
`PROVIDED BY THE GPS IN THE CAMERA
`
`CAMERAPOINTINGDIRECTION PROVIDED BY
`THE ELECTRONIC COMPASS IN THE CAMERA
`
`750
`
`752
`
`DISTANCE OFFSETS FROM THE CAMERATO PORTIONS
`OF THE SCENE PROVIDED BY THE RANGEMAP
`
`754
`
`ANGUAR OFFSET FROM THE CAMERA PROVIDED
`FROM THE LOCATION IN THE FIELD OF VIEW
`
`
`
`GPSOCATIONS FORPORTIONS OF THE SCENE ARE
`DETERMINED BY ADDING DSTANCE OFFSETS AND THE
`ANGULAR OFFSETS TO THE GPS LOCATIONAND
`PONTING DIRECTION OF THE CAMERA
`
`756
`
`758
`
`A GPSLOCAONS FOR THE OBJECTS IN THE SCENE
`ARESTORED IN METADATA OR DISPLAYED AS LABELS
`INA GPS LOCATION MAP
`
`760
`
`FIG. 18
`
`APPL-1005 / Page 20 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 19 Of 29
`
`US 7,859,588 B2
`
`STARTUP
`
`CAMERA POWER ON
`NTIALIZATION
`
`1100
`
`SET FIRSTAND SECOND CAPTURE
`UNITS TO DEFAULTZOOMPOSITIONS
`
`1102
`
`CAPTURE AND DISPLAY FIRST AND
`SECOND PREVIEW IMAGES USING FIRST
`AND SECOND CAPTURE UNITS
`
`1104
`
`
`
`
`
`
`
`1106
`SELECT
`FIRST OR SECOND
`PREVIEW
`MAGE
`
`SET SELECTED -1124
`CAPTURE UNIT
`AS PRIMARY
`CAPTURE UNIT
`
`YES
`
`
`
`
`
`
`
`SS1
`SHUTTER BUTTON
`PRESSED
`
`1108
`NO
`
`SET DEFAUL CAPTURE UNIT
`AS PRIMARY CAPTURE UNIT
`
`SET NON-PRIMARY CAPTURE UNIT
`AS SCENE ANALYSS CAPTURE UNIT
`
`1114
`
`PREVIEWMODE
`
`FIG. 19
`
`APPL-1005 / Page 21 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 20 Of 29
`
`US 7,859,588 B2
`
`
`
`1200
`
`12O2
`
`
`
`
`
`PREVIEWMODE
`
`SCENE ANALYSIS CAPTURE UNIT
`ANALYZES SCENE
`
`
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERS UTILIZING SCENE
`ANALYSIS CAPTURE UNIT DATA
`
`1204
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`
`
`
`
`
`
`12O6
`
`SCENE ANALYSS CAPTURE UNIT
`ANALYZES SCENE
`
`1208
`
`
`
`SCENE
`CONDITIONS
`CHANGED
`
`FIG. 20
`
`APPL-1005 / Page 22 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 21 of 29
`
`US 7,859,588 B2
`
`
`
`PREVIEWMODE
`WITH THRESHOLD
`
`
`
`
`
`1300
`
`1302
`
`
`
`1304
`
`
`
`1306
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERS UTILIZING SCENE
`ANALYSIS CAPTURE UNIT DATA
`
`
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROMPRIMARY CAPTURE UNIT
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`1308
`
`
`
`
`
`SCENE
`CONDITION CHANGEX
`OR STABLE
`
`FIG. 21
`
`APPL-1005 / Page 23 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 22 of 29
`
`US 7,859,588 B2
`
`ENHANCED
`PREVIEWMODE
`
`1400
`
`SET SCENE ANALYSIS CAPTURE UNIT
`ZOOMPOSITION RELATIVE TO PRIMARY
`CAPTURE UNIT ZOOMPOSITION
`
`1402
`
`1404
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERS UTILIZING SCENE
`ANALYSIS CAPTURE UNIT DATA
`
`1406
`
`CAPTURE PREVIEW IMAGE
`FROMPRIMARY CAPTURE UNIT
`
`1408-N- ANALYZESCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSS DATA
`
`1410
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERS UTILIZING RESULTS
`OF THE SCENEANALYSIS
`
`1412
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROMPRIMARY CAPTURE UNIT
`
`
`
`1414
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`1416-N ANALYZESCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS DATA
`
`
`
`1418
`SCENE
`CONDITION CHANGE2X
`OR STABLE
`
`FIG. 22
`
`APPL-1005 / Page 24 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 23 Of 29
`
`US 7,859,588 B2
`
`1500
`
`
`
`
`
`
`
`ZOOM
`BUTTON
`PRESSED
`
`NO
`
`RETURN TO PREVIEW
`
`1502
`
`
`
`
`
`
`
`ZOOM
`POSITION
`PRIMARY CAPTURE
`UNT ZOOM
`RANGE
`
`1504
`
`
`
`1506
`
`SET CURRENT SCENE ANALYSS
`AND PRIMARY CAPTURE UNITSAS
`PRIMARY AND SCENE ANALYSIS
`CAPTURE UNITS RESPECTIVELY
`
`SET PRIMARY CAPTURE UNIT ZOOM
`POSITION TO SELECTED ZOOMPOSTION
`
`RETURN TO PREVIEW
`
`FIG. 23
`
`APPL-1005 / Page 25 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 24 of 29
`
`US 7,859,588 B2
`
`CAPTURE
`
`1600
`ISS1
`SHUTTER BUTTON
`PRESSED
`
`
`
`NO
`
`YES
`
`RETURN TO PREVIEW
`
`1602
`
`1604
`
`CAPTURE PREVIEW IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`ANALYZE SCENE UTILIZING
`CAPTURED PREVIEW IMAGE
`
`1606
`
`ANALYSIS
`COMPLETE
`
`1608-N-SET PRIMARY CAPTURE UNIT PARAMETERS
`UTILIZING RESULTS OF THEANALYSS
`
`1610
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROMPRIMARY CAPTURE UNIT
`
`
`
`ISS2
`1612
`SHUTTER BUTTON
`PRESSED
`
`
`
`
`
`YES
`
`IS FOCUSI
`EXPOSURE LOCK
`SET
`
`1614
`
`CAPTURE PRIMARY IMAGE
`FROMPRIMARY CAPTURE UNIT
`
`RETURN TO PREVIEW
`
`FIG. 24
`
`APPL-1005 / Page 26 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 25 Of 29
`
`US 7,859,588 B2
`
`RETURN TO PREVIEW
`
`CAPTURE
`
`17
`ISS1
`OO
`SHUTTER BUTTON
`PRESSED
`
`
`
`
`
`NO
`
`YES
`
`1702
`
`1704
`
`CAPTURE PREVIEW IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`1706
`
`ANALYZESCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS IMAGES
`
`1708
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERS UTILIZING RESULTS
`OF THE SCENE ANALYSIS
`
`
`
`
`
`
`
`1710
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`
`
`1712
`SS2
`SHUTTER BUTTON
`PRESSED
`
`
`
`
`
`IS FOCUSI
`EXPOSURE LOCK
`SET
`
`YES
`
`1714
`
`CAPTURE PRIMARY IMAGE
`FROMPRIMARY CAPTURE UNIT
`
`RETURN TO PREVIEW
`
`FIG. 25
`
`APPL-1005 / Page 27 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 26 of 29
`
`US 7,859,588 B2
`
`RETURN TO PREVIEW
`
`CAPTURE
`
`SS1
`1800
`SHUTTER BUTTON
`PRESSED
`
`
`
`NO
`
`YES
`
`CAPTURE PREVIEW IMAGE
`FROMPRIMARY CAPTURE UNIT
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`ANALYZESCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS IMAGES
`
`1802
`
`1804
`
`1806
`
`1808
`
`SET PRIMARY CAPTURE UNIT PARAMETERS
`UTILIZING RESULTS OF THE SCENE ANALYSIS
`
`1810
`
`SET SCENE ANALYSIS CAPTURE UNIT
`ASA SECONDARY CAPTURE UNIT
`
`1812-N- SET SECONDARY CAPTURE UNIT PARAMETERS
`UTILIZING RESULTS OF THE SCENE ANALYSIS
`
`
`
`1814
`ISS2
`SHUTTER BUTTON
`PRESSED
`
`
`
`NO
`
`1816
`
`1818
`
`YES
`CAPTURE PRIMARY IMAGE
`FROMPRIMARY CAPTURE UNIT
`
`CAPTURE ANAUGMENTATION IMAGE
`FROMSCENE ANALYSIS CAPTURE UNIT
`
`1820
`
`PRODUCE AN ENHANCED IMAGE FROM
`THE PRIMARY ANDAUGMENTATION MAGES
`
`RETURN TO PREVIEW
`
`FIG. 26
`
`APPL-1005 / Page 28 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 27 Of 29
`
`US 7,859,588 B2
`
`
`
`d
`
`W
`
`155-,
`
`-154
`
`153
`y-S
`NA-1
`%
`
`158
`
`152
`...
`-N
`N-1
`.
`
`191-I
`
`181
`
`18O
`
`170
`
`171
`
`X2
`
`b
`
`X1
`
`157 156
`
`-190
`
`FIG. 27
`(PRIOR ART)
`
`APPL-1005 / Page 29 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 28 of 29
`
`US 7,859,588 B2
`
`START
`
`197
`
`SET LENS TOMID FOCUSAND
`LOAD LOWPASS FILTER
`
`CALCULATE FOCUS VALUE (FW)
`AS IMAGE IS READ OUT
`
`ADJUST STEPPER MOTOR TO
`NEXT NEAR FOCUS POSITION
`
`GETFW OF NEXT IMAGE CAPTURED
`STEP NOPPOSITE
`WITH NEW FOCUS SETTING e DIRECTION
`
`198
`
`198
`
`199
`
`COMPARE TO PREVIOUS
`WEIGHTED FOCUS VALUES
`FV INCREASES
`
`STEP N SAME DIRECTION
`UNTIL FV DECREASES
`
`LOAD HIGHPASS FILTER
`AND MEASURE FW USING
`SAME FOCUS POSITION
`
`STEPIN SAME DIRECTION
`UNTIL FV DECREASE
`
`BACKUP AND HOLDAT
`HIGHEST FVFOCUS POSITION
`
`FOCUS OK
`
`FIG. 28
`(PRIORART)
`
`APPL-1005 / Page 30 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 29 of 29
`
`US 7,859,588 B2
`
`FOCUS
`1 ADJUSTMENT
`STEPS
`
`A
`
`FOCUS
`VALUE
`
`-1
`
`B
`
`FOCUS
`POSITION
`
`P
`
`(PRIORART)
`
`APPL-1005 / Page 31 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`US 7,859,588 B2
`
`1.
`METHOD AND APPARATUS FOR
`OPERATING ADUAL LENS CAMERATO
`AUGMENT AN MAGE
`
`FIELD OF THE INVENTION
`
`The present invention relates to a digital camera that pro
`duces digital image files and, more particularly, to a digital
`camera that uses multiple lenses and image sensors to provide
`an improved imaging capability.
`
`10
`
`BACKGROUND OF THE INVENTION
`
`2
`A diagram illustrative of the principle of the operation of a
`conventional rangefinder is shown herein in FIG. 27. In that
`diagram, light from an object 151 is incident on two small
`lenses 152 and 153 which have a sufficiently short focal
`length f that light rays received from the object through dif
`ferent spaced paths 154 and 155 produce corresponding
`spaced images 157 and 158 in a focal plane 156 which is
`common to the lenses 152 and 153. When the object 151 is at
`an infinite distance, the centers of the images 157 and 158 are
`located at reference positions 170 and 180 in FIG. 27, but
`when the object 151 is located at a closer distance, the centers
`of the images are shifted apart to positions 171 and 181. If the
`distance by which the images 157 and 158 are shifted from the
`reference positions 170 and 180 are designated X and X,
`respectively, then the total shift X may be expressed as fol
`lows:
`
`Thus, the distanced to the object 151 can be measured by
`d=bf/x. In this case, b is the distance between the optical axes
`of the small lenses, that is, the base length. To obtain the
`shifted amounts X and X, or the Sum X of both, two optical
`sensor arrays 190 and 191 are provided in the focal plane 156
`as shown in FIG. 27. These optical sensor arrays each com
`prise a plurality of optical sensors, for instance CCD devices,
`and an analog photoelectric signal is generated by each opti
`cal sensor corresponding to the light intensity at the portion of
`the image which is incident on the sensor. Haruki et al. shows
`a conventional circuit, as well as a higher speed rangefinding
`circuit according to the patent, for obtaining the sum X of the
`shifted distances by comparing two image signal trains com
`prising the digital image signals from the left and right optical
`sensor arrays.
`Basically, the offset information X is used along with the
`lens separation distance band the focal length f to calculate
`the distanced to the scene by triangulation. The calculated
`distanced to the scene is used to guide the positioning of an
`adjustable focus lens to produce the best image quality. As
`known in the prior art, this adjustment may be based on a
`calibration curve established between the distance to the
`scene as measured by the dual lens rangefinder module and a
`series of best focused images as produced by a “through the
`lens' autofocus system. The calibration curve is stored as an
`equation or a look-up table in a microprocessor in the camera.
`Rangefinder-based autofocus systems have the advantage
`of being very fast, Some having a response time that can be in
`the range of 0.01-0.05 second. However, the focus quality
`produced by some rangefinder-based autofocus systems can
`vary when they are used in different operating conditions. For
`example, Sonic autofocus systems cannot focus through a
`glass window as the glass stops the projected Sonic signal,
`thereby causing the autofocus system to focus onto the glass.
`In the case of a dual lens rangefinder autofocus system, the
`accuracy of dual lens rangefinders are typically influenced by
`changes in environmental conditions such as temperature
`and/or humidity. The problem with dual lens rangefinder
`modules is that the calibration between the dual lens
`rangefinder module and the adjustable focus lens position is
`not stable within the normal operating environment for digital
`cameras. Environmental conditions such as changes in tem
`perature and humidity can cause the distance to the portion of
`the scene as measured by the dual lens rangefinder module to
`change by over 10%. In addition, the measured position of the
`adjustable focus taking lens in the adjustable focus taking
`lens system is prone to environmentally induced changes as
`well so that inaccuracies are produced in the control system
`
`Currently, most digital cameras use a Zoom taking lens and
`a single color image sensor to capture still and motion images.
`The captured images are then processed to produce digital
`image files, which are stored in a digital memory in the
`camera. The digital image files can then be transferred to a
`computer, displayed, printed, and shared via the Internet.
`In order to capture sharp images of moving Subjects, a
`digital camera needs to provide a precise automatic lens
`focusing system (i.e., an autofocus system). The autofocus
`system must be capable of quickly obtaining the correct focus
`in order to minimize the “shutter delay' between the time the
`shutter button is pressed and the still image is captured. The
`autofocus system must also work in a continuous image cap
`ture mode wherein video images are captured. For instance, in
`a video mode the focus should be adjusted in real-time while
`Video images are being continuously captured.
`Many digital cameras and Scanners capture images using
`an image sensor and a taking lens system with an adjustable
`focus. Typically, the focus distance of such an adjustable
`focus taking lens system can automatically be set to one of a
`plurality of different settings by sensing, control, and drive
`systems, which are adapted to provide optimal focus of what
`is determined to be a Subject area in a scene. Lens systems that
`provide automatically adjustable focus settings based on a
`focus measurement and an adjustable focus lens are referred
`to herein as autofocus systems. Digital cameras typically use
`one of two types of autofocus systems: a rangefinder system
`and a “through-the-lens' focus system.
`A rangefinder system uses rangefinding sensors such as a
`Sonic rangefinder or a dual lens rangefinder to determine the
`distance from a camera to one or more portions of a scene
`within a field of view of the rangefinder system. A sonic
`rangefinder measures the phase offset between a projected
`Sonic signal and a reflected Sonic signal to infer the distance to
`objects in the scene. Dual lens rangefinders contain two
`lenses that are separated by a distance along with two match
`ing sensor areas that capture matched pairs of images. Dual
`lens rangefinders are commonly used on digital cameras in
`the form of dual lens rangefinder modules which contain two
`lenses separated by a distance along with two matching sen
`sor areas that capture matched pairs of low resolution images.
`Common dual lens rangefinder-based autofocus systems
`include active and passive systems. Active systems actively
`project light onto the scene, while passive systems work with
`the available light from the scene. Dual lens rangefinder mod
`ules can be purchased from Fuji Electric in several models
`such as the FM6260W. A dual lens rangefinder module for
`optical apparatus Such as a camera is described in U.S. Pat.
`No. 4,606,630, which was issued to Haruki et al. on Aug. 19,
`1986 (and assigned to Fuji Electric). According to the
`description of the prior art in this patent, matched pairs of low
`resolution images are analyzed for correlation between the
`two images to determine the offset between the two images
`caused by the separation between the two lenses.
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`APPL-1005 / Page 32 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`US 7,859,588 B2
`
`10
`
`15
`
`3
`for the adjustable focus lens. Consequently, dual lens
`rangefinder modules are not typically used independently for
`autofocus in digital cameras but instead are used as a rough
`focus adjustment that is supplemented by a “through the lens'
`autofocus system.
`Alternatively, the “through-the-lens' autofocus system
`determines a focus state through an analysis of a series of
`autofocus images captured with the adjustable focus lens
`system positioned at a plurality of different focus distances.
`For example, in a typical “through-the-lens' autofocus sys
`tem a plurality of autofocus images (e.g., 5-20) are captured
`with the adjustable focus lens in a series of different positions
`in a so-called “hill climb' method. This type of autofocus is
`known as "hill climbing autofocus because it generates a
`sequence of values that increase in level until they pass over a
`peak, i.e., a “hill'. In other words, the lens focus position is
`adjusted automatically until the contrast of the edge detail in
`the image, or a particular area of the image, is maximized. For
`instance, the contrast present in each of the autofocus images
`is compared and the autofocus image with the greatest con
`trast is deemed to have been captured with the best focus
`conditions (often the best focus lens position is further refined
`by interpolating the contrast values between images).
`In order to decrease focusing response time without sacri
`ficing focusing precision, it is common to use filters to sepa
`rate not only the higher frequency component of the video
`signal, but also the lower frequency component. For example,
`a lens may be quickly driven in coarse adjustment steps in a
`low frequency range furthest from the maximum focus, and
`then driven infiner adjustment steps in a high frequency range
`nearer to the maximum focus. A flow diagram of a conven
`tional “hill climbing contrast autofocus algorithm is shown
`in FIG. 28. This algorithm uses the “hill climbing contrast
`autofocus method discussed above and shown in the diagram
`of FIG. 29, which illustrates the relationship between the
`focus value obtained from the filters and the lens position. In
`FIG. 29, the abscissa indicates the focusing position of a lens
`along a distance axis, the ordinate indicates the focusing
`evaluation value, and the curves A and Bindicate the focusing
`evaluation values for high and low frequency components,
`respectively, relative to a particular in-focus position P.
`Referring to the flow diagram of FIG. 28, the best starting
`point for the algorithm depends on the hyperfocal distance of
`the current lens setting, which is a function of the focallength
`45
`setting and the f-number. A distance of about 2 meters is
`typically a good starting point. Then a low frequency band
`pass filter is loaded (stage 197) and the focus values are read
`out. The algorithm employs a comparison stage 198 to set the
`direction of lens adjustment toward increasing focus values,
`and to determine when the lens is stepped over the “hill'. The
`depth of field, which depends on the present focal length and
`f-number, sets the number of steps, i.e., the next near focus
`position, which should be taken before capturing the next
`frame when using the low frequency bandpass filter. Once the
`peak of the hill is passed (curve B in FIG. 29), a high fre
`quency bandpass filter is loaded (stage 199), and the lens is
`moved in the opposite direction until the peak of the higher
`“hill' is found(curve A in FIG. 29). The peak focus value may
`use either the weighted average or peak value from numerous
`pixels.
`“Through-the-lens' autofocus systems are very accurate
`since they measure focus quality directly from autofocus
`images captured with the high quality taking lens. Unfortu
`nately, “through-the-lens' autofocus systems can be rela
`tively slow in determining a focus setting due to the large
`number of autofocus images that must be captured and com
`
`55
`
`4
`pared. For example, “through-the-lens' autofocus systems
`can take as long as 0.5-2.0 seconds to determine focus con
`ditions.
`Accordingly, in some digital cameras, the two types of
`autofocus systems are used together in a hybrid system in
`which the rangefinder based autofocus system is used to
`provide a fast estimation of the adjustable focus lens location
`that is then followed by the use of the “through-the-lens'
`autofocus system to refine the focus setting. For example,
`U.S. Pat. No. 6,864,474, entitled "Focusing Apparatus for
`Adjusting Focus of an Optical Instrument” and which issued
`Mar. 8, 2005 in the name of Misawa, describes the coordi
`nated use of a rangefinder-based autofocus system with a
`“through-the-lens' autofocus system. In Misawa, the focus
`position of the adjustable focus taking lens is determined by
`both the rangefinder-based autofocus system and the
`“through-the-lens' autofocus system. The difference
`between the adjustable focus taking lens position determined
`by the rangefinder-based autofocus system and the adjustable
`focus taking lens position determined by the “through-the
`lens' autofocus system is stored for future reference. In sub
`sequent image capture episodes, the stored difference infor
`mation is used to refine the number of autofocus images
`captured and analyzed by the “through-the-lens' autofocus
`system in the “hill climb' method to determine the adjustable
`focus lens position for best focus, thereby reducing the num
`ber of autofocus images captured and processed in cases
`where the rangefinder system is accurate and increasing the
`number of autofocus images captured and processed in cases
`where the rangefinder is inaccurate. However, the method
`described by Misawa assumes that the performance of the
`rangefinder, adjustable focus taking lens system and control
`system are consistent over time, do not fluctuate with varia
`tions in environmental conditions, and do not otherwise
`change or drift over time.
`Once an image is in focus, the “hill climb' method typi
`cally operates over incremental distances near the Subject
`presently focused upon. Then, in refocusing an image, the
`“hill clim