I 1111111111111111 11111 1111111111 111111111111111 IIIII IIIII IIIIII IIII IIII IIII
`US007859588B2
`
`c12) United States Patent
`Parulski et al.
`
`(IO) Patent No.:
`(45) Date of Patent:
`
`US 7,859,588 B2
`*Dec. 28, 2010
`
`(54) METHOD AND APPARATUS FOR
`OPERATING A DUAL LENS CAMERA TO
`AUGMENT AN IMAGE
`
`(75)
`
`Inventors: Kenneth A. Parulski, Rochester, NY
`(US); Wilbert F. Janson, Jr.,
`Shortsville, NY (US); John N. Border,
`Walworth, NY (US); John R. Fredlund,
`Rochester, NY (US); Joseph A. Manico,
`Rochester, NY (US); Robert J. Parada,
`Jr., Rochester, NY (US)
`
`(73) Assignee: Eastman Kodak Company, Rochester,
`NY(US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 882 days.
`
`8/2003 Yu et al.
`6,611,289 Bl
`3/2005 Misawa
`6,864,474 B2
`7,676,146 B2 * 3/2010 Border et al.
`2003/0020814 Al
`1/2003 Ono
`2003/0160886 Al
`8/2003 Misawaetal.
`2005/0046738 Al
`3/2005 Sato
`2005/0275747 Al
`12/2005 Nayar et al.
`2006/0187312 Al
`8/2006 Labaziewicz et al.
`
`................. 396/80
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`
`0 858 208
`
`8/1998
`
`This patent is subject to a terminal dis(cid:173)
`claimer.
`
`(Continued)
`
`(21)
`
`Appl. No.: 11/684,025
`
`(22)
`
`Filed:
`
`Mar. 9, 2007
`
`Primary Examiner-Kelly L Jerabek
`(74)Attorney, Agent, or Firm-Thomas J. Strouse; Peyton C.
`Watkins
`
`(65)
`
`(51)
`
`(52)
`(58)
`
`(56)
`
`Prior Publication Data
`
`(57)
`
`ABSTRACT
`
`US 2008/0218611 Al
`
`Sep. 11, 2008
`
`Int. Cl.
`H04N 51232
`(2006.01)
`G03B 13/00
`(2006.01)
`G03B 17/00
`(2006.01)
`U.S. Cl. ......................................... 348/349; 396/79
`Field of Classification Search ................. 348/349;
`396/72, 79, 100
`See application file for complete search history.
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,606,630 A
`5,668,597 A
`5,874,994 A
`6,441,855 Bl
`
`8/ 1986 Haruki et al.
`9/ 1997 Parulski et al.
`2/ 1999 Xie et al.
`8/2002 Omata et al.
`
`An electronic camera for producing an output image of a
`scene from a captured image signal includes a first imaging
`stage comprising a first image sensor for generating a first
`sensor output and a first lens for forming a first image of the
`scene on the first image sensor, and a second imaging stage
`comprising a second image sensor for generating a second
`sensor output and a second lens for forming a second image of
`the scene on the second image sensor. The sensor output from
`the first imaging stage is used as a primary output image for
`forming the captured image signal and the sensor output from
`the second imaging stage is used as a secondary output image
`for modifying the primary output image, thereby generating
`an enhanced, captured image signal.
`
`22 Claims, 29 Drawing Sheets
`
`APPL-1014 / Page 1 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`US 7,859,588 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`FOREIGN PATENT DOCUMENTS
`
`8/2006 Criminisi et al.
`2006/0193509 Al
`1/2008 Daley ......................... 396/121
`2008/0013941 Al*
`2008/0218612 Al * 9/2008 Border et al.
`............... 348/262
`2008/0218613 Al * 9/2008 Janson et al.
`............... 348/262
`
`JP
`WO
`WO
`
`2005045511
`01/06449
`2005/057278
`
`2/2005
`1/2001
`6/2005
`
`* cited by examiner
`
`APPL-1014 / Page 2 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`00 = N
`
`00
`tit
`\0
`UI
`00
`-....l
`d r.,;_
`
`0 ....
`....
`.....
`rJJ =(cid:173)
`
`('D
`('D
`
`1,0
`N
`
`N
`~CIO
`N
`~
`
`0 ....
`
`0
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`57
`
`56
`
`REMOVABLE
`
`MEMORY
`
`CARD
`
`c ('D
`
`54
`
`I
`66
`
`64
`
`I
`62
`
`INTERFACE
`
`MEMORY
`
`CARD
`
`52
`
`INTERFACE
`
`HOST
`
`MUX1
`
`50
`
`34
`
`I
`
`I A;lfo 1 J r,:::i MUX2 I
`36
`
`22
`
`12
`
`3
`
`1
`
`10A"-.
`
`90
`92
`I
`I
`I
`I
`'~-------'
`_._ ________ ..
`: PROCESSOR I
`I MODEM
`I
`1 CELLULAR :_:CELLULAR:
`--------
`r---------~
`I
`94
`
`MEMORY
`
`58 d~
`
`COMPASS r59
`
`GPS
`
`MEMORY
`
`RAM
`
`PROCESSOR
`
`IMAGE
`
`38
`
`MEMORY
`BUFFER
`DRAM
`
`40
`
`24
`
`FIG. 1
`
`COLOR LCD
`
`DiSPLAY
`IMAGE
`
`70
`
`CONTROLS
`
`USER
`
`42
`
`FLASH--~
`
`48
`
`TIMING GENERATOR
`
`PROCESSOR AND
`
`CONTROL
`
`AUTO EXPOSURE ..__ __ 1
`
`DETECTORS
`
`FOCUS MOTORS
`
`ZOOMAND
`
`46
`
`Sb
`
`15
`
`14e
`
`SENSOR
`IMAGE
`SECOND
`
`LENS
`ZOOM
`SECOND
`
`4 14
`
`2
`
`DRIVERS
`CLOCK
`
`FOCUS MOTORS
`
`WOM~D
`
`Sa
`
`13
`/
`
`12e
`
`l~~~l 11
`
`SENSOR
`
`I
`
`LENS
`ZOOM
`FIRST
`
`APPL-1014 / Page 3 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 2 of 29
`
`US 7,859,588 B2
`
`._Gf8s
`
`(.)
`C'\I
`-q-
`
`~~ -q-
`
`M .. -q-
`
`.
`I
`0
`I'-
`
`tQ
`C\I
`1111 (!)
`it
`
`rn
`C'\I
`-q-
`
`<(
`0
`
`~
`
`<(
`0
`
`~
`
`ro
`C'\I
`-.::t"
`
`APPL-1014 / Page 4 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 3 of 29
`
`US 7,859,588 B2
`
`ZOOM POSITION SET TO
`DEFAULT POSITION WHEN
`CAMERA IS POWERED ON
`
`USER INPUT
`ZOOM POSITION
`
`100
`
`101
`
`YES
`
`SET FIRST IMAGE CAPTURE
`STAGE TO CAPTURE IMAGES
`& SECOND IMAGE CAPTURE
`STAGE FOR AUTO FOCUS
`
`CAPTURE AUTOFOCUS
`IMAGES WITH SECOND
`IMAGE CAPTURE STAGE
`
`FOCUS FIRST IMAGE
`CAPTURE STAGE
`
`CAPTURE PREVIEW IMAGES
`WITH FIRST IMAGE CAPTURE
`STAGE AND DISPLAY
`
`104 124
`
`106 126
`
`108 128
`
`110 130
`
`SET SECOND IMAGE CAPTURE
`STAGE TO CAPTURE IMAGES
`& FIRST IMAGE CAPTURE
`STAGE FOR AUTO FOCUS
`
`CAPTURE AUTOFOCUS
`IMAGES WITH FIRST
`IMAGE CAPTURE STAGE
`
`FOCUS SECOND IMAGE
`CAPTURE STAGE
`
`CAPTURE PREVIEW IMAGES
`WITH SECOND IMAGE CAPTURE
`STAGE AND DISPLAY
`
`YES
`
`NO
`
`YES
`
`NO
`
`CAPTURE DIGITAL STILL
`IMAGE WITH FIRST
`IMAGE CAPTURE STAGE
`
`116 136
`
`CAPTURE DIGITAL STILL
`IMAGE WITH SECOND
`IMAGE CAPTURE STAGE
`
`FIG. 3
`
`APPL-1014 / Page 5 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`00 = N
`
`00
`tit
`\0
`UI
`00
`-....l
`d r.,;_
`
`270
`
`0 ....
`rJJ =- ('D a
`
`.i;...
`
`1,0
`N
`
`268
`
`.
`
`266
`
`-
`
`0 ....
`
`N
`
`0
`
`~
`
`QO
`N
`~
`('D
`c
`
`264
`
`-
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`262
`
`.
`
`FIG. 4
`
`FROM S1 TO S2, THE FINAL IMAGE IS CAPTURED
`WHEN THE USER PUSHES THE CAPTURE BUTTON •
`
`l
`
`DETERMINED POSITION FOR BEST FOCUS
`
`THE FOCUS LENS IS MOVED TO THE
`
`•
`
`STAGE IN THE LOWER ZOOM POSITION IS CROPPED
`
`THE AUTOFOCUS IMAGE FROM THE IMAGE
`
`FEATURES IN THE TWO AUTOFOCUS IMAGES
`AND UPSAMPLED SO THAT CORRESPONDING
`
`SPAN THE SAME NUMBER OF PIXELS
`
`,____
`
`•
`
`IMAGE STAGE IS USED FOR CAPTURE
`ZOOM POSITION DETERMINES WHICH
`
`•
`
`USER SELECTS ZOOM POSITION
`
`IDENTIFY THE PIXEL OFFSET BETWEEN THE TWO
`
`IMAGE FROM THE OTHER IMAGE STAGE TO
`IMAGE IS CORRELATED·TO THE AUTOFOCUS
`THE CROPPED AND UPSAMPLED AUTOFOCUS
`
`AUTOFOCUS IMAGES FOR CORRESPONDING
`
`PORTIONS OF THE IMAGES
`
`l
`
`USE THE AUTOFOCUS RANGEFINDER
`
`CALIBRATION CURVE AND THE IDENTIFIED
`
`PIXEL OFFSET TO DETERMINE THE MOVEMENT
`
`FOCUS THE IMAGE CAPTURE STAGE
`OF THE FOCUS LENS NEEDED TO BEST
`
`!
`
`I
`
`AUTOFOCUS IMAGES ARE CAPTURED
`
`WITH BOTH IMAGE STAGES
`
`•
`
`SO TO S1 TO INITIATE AUTOFOCUS SEQUENCE
`
`USER PUSHES CAPTURE BUTTON FROM
`
`l
`
`ARE SET TO THEIR HYPERFOCAL POSITIONS
`
`FOCUS LENSES IN BOTH IMAGE STAGES
`
`•
`
`'" THE IMAGE STAGE NOT USED FOR CAPTURE IS
`
`ZOOMED TO THE POINT CLOSEST TO THE
`
`USER SELECTED ZOOM POSITION
`
`...
`
`••
`
`260
`
`258
`
`.
`
`256
`
`254
`
`...
`t
`
`252
`
`250
`
`APPL-1014 / Page 6 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`....
`
`0
`0
`N
`
`~
`
`QO
`N
`~
`('D
`c
`
`NO
`
`YES
`
`A
`
`---
`
`=
`
`~
`~
`~
`~
`•
`00
`~
`
`~
`
`CONTINUOUSLY CHECK FOCUS WITH THE IMAGE
`
`CAPTURE STAGE BEING USED FOR CAPTURE
`
`USING THE "HILL CLIMB METHOD"
`
`CONTINUOUSLY CAPTURE VIDEO
`
`WITH THE CAPTURE STAGE
`
`I
`
`278
`
`276
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`STAGES BY THE "HILL CLIMB METHOD"
`AUTOFOCUS BOTH IMAGE CAPTURE
`
`SO TO S1 TO INITIATE AUTOFOCUS SEQUENCE
`
`USER PUSHES CAPTURE BUTTON FROM
`
`THE IMAGE STAGE NOT USED FOR CAPTURE IS
`
`ZOOMED TO THE POINT CLOSEST TO THE
`
`USER SELECTED ZOOM POSITION
`
`I
`
`CAPTURE STAGE IS THE CAPTURE STAGE
`ZOOM POSITION DETERMINES WHICH IMAGE
`
`USER SELECTS ZOOM POSITION
`
`274
`
`272
`
`258
`
`254-4-
`
`252
`
`250-f
`
`00 = N
`
`00
`tit
`\0
`UI
`00
`-....l
`r.,;_
`d
`
`....
`0
`Ul
`.....
`=- ('D
`
`('D
`
`rJJ
`
`1,0
`N
`
`THE CAPTURE STAGE TO PRODUCE BEST FOCUS
`
`MOVEMENT OF THE FOCUS LENS NEEDED IN
`FOCUS LENS ADJUSTMENT TO DETERMINE THE
`
`CALIBRATION CURVE AND THE IDENTIFIED
`USE THE AUTOFOCUS "HILL CLIMB METHOD"
`
`I
`
`MOVE THE FOCUS LENS IN THE CAPTURE STAGE
`
`TO THE NEW BEST FOCUS POSITION
`
`FIG. 5
`
`_;
`
`286
`
`284
`
`I
`
`USER PUSHES THE CAPTURE BUTTON
`
`FROM S1 TO S2
`
`TO PRODUCE BEST FOCUS ON THE IMAGE CAPTURE
`
`2821 IDENTIFY THE FOCUS LENS ADJUSTMENT NEEDED
`
`STAGE NOT BEING USED FOR CAPTURE
`
`APPL-1014 / Page 7 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 6 of 29
`
`US 7,859,588 B2
`
`CAPTURE A SERIES OF IMAGE SETS WITH OBJECTS
`AT KNOWN DISTANCES WITH THE SHORTER
`FOCAL LENGTH FIRST IMAGE CAPTURE STAGE AND
`THE LONGER FOCAL LENGTH SECOND IMAGE CAPTURE
`STAGE AND A SERIES OF FOCUS LENS POSITIONS
`
`'
`
`THE AUTOFOCUS IMAGE FROM THE LOWER
`FOCAL LENGTH FIRST IMAGE STAGE IS CROPPED AND
`UPSAMPLED SO THAT CORRESPONDING FEATURES
`IN THE TWO AUTOFOCUS IMAGES SPAN
`THE SAME NUMBER OF PIXELS
`I
`CORRELATE THE IMAGES FROM THE SECOND IMAGE
`CAPTURE STAGE TO CORRESPONDING PORTIONS OF
`THE IMAGES FROM THE CROPPED AND UPSAMPLED
`IMAGE FROM THE FIRST IMAGE CAPTURE STAGE TO
`DETERMINE THE PIXEL OFFSET BETWEEN
`THE IMAGES IN EACH IMAGE SET
`!
`STORE THE DATA OF PIXEL OFFSET BETWEEN
`IMAGES IN EACH IMAGE SET VERSUS KNOWN
`DISTANCE TO OBJECTS AS AN AUTOFOCUS
`RANGE FINDER CALIBRATION CURVE
`
`.i...--300
`
`.L--302
`
`.i.--304
`
`.i---306
`
`FIG. 6
`
`APPL-1014 / Page 8 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 7 of 29
`
`US 7,859,588 B2
`
`---400
`
`CAPTURE A SERIES OF IMAGE SETS WITH OBJECTS
`AT KNOWN DISTANCES WITH THE FIRST IMAGE
`CAPTURE STAGE AND THE SECOND IMAGE CAPTURE
`STAGE WHEREIN AUTOFOCUS IS DONE BY THE
`"HILL CLIMB METHOD11 FOR EACH IMAGE
`!
`COMPARE THE FOCUS LENS POSITIONS FOR THE TWO ----402
`IMAGE CAPTURE STAGES VERSUS THE DISTANCE TO
`THE FOCUSED OBJECTS IN THE IMAGE SETS
`l
`STORE THE DATA OF FOCUS LENS POSITIONS OF
`THE FIRST IMAGE CAPTURE STAGE VERSUS THE FOCUS
`LENS POSITIONS OF THE SECOND IMAGE CAPTURE
`STAGE FOR THE SAME DISTANCE TO FOCUSED
`OBJECTS IN THE IMAGES AS AN AUTOFOCUS
`"HILL CLIMB METHOD" CALIBRATION CURVE
`
`-~ 404
`
`FIG. 7
`
`APPL-1014 / Page 9 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 8 of 29
`
`US 7,859,588 B2
`
`ZOOM POSITION SET TO DEFAULT
`POSITION WHEN CAMERA IS POWERED ON
`
`100
`
`USER INPUT ZOOM POSITION
`
`101
`
`YES
`
`SET FIRST IMAGE CAPTURE
`MODULE TO CAPTURE IMAGES
`& SECOND IMAGE CAPTURE
`MODULE FOR AUTO FOCUS
`
`CAPTURE AUTOFOCUS IMAGES
`
`AUTOFOCUS FIRST IMAGE
`CAPTURE MODULE
`
`CAPTURE PREVIEW IMAGES
`WITH FIRST IMAGE CAPTURE
`MODULE AND DISPLAY
`
`104
`
`124
`
`106
`
`108
`
`126
`
`128
`
`110
`
`130
`
`SET SECOND IMAGE CAPTURE
`MODULE TO CAPTURE IMAGES
`& FIRST IMAGE CAPTURE
`MODULE FOR AUTO FOCUS
`
`CAPTURE AUTOFOCUS IMAGES
`
`AUTOFOCUS SECOND IMAGE
`CAPTURE MODULE
`
`CAPTURE PREVIEW IMAGES
`WITH SECOND IMAGE CAPTURE
`MODULE AND DISPLAY
`
`YES
`
`NO
`
`112
`
`132
`
`114
`
`134
`
`YES
`
`NO
`
`CAPTURE VIDEO IMAGE WITH
`FIRST IMAGE CAPTURE MODULE
`
`CHECK IMAGE FOR
`FOCUS QUALITY
`
`138
`
`118
`
`119
`
`139
`
`CAPTURE VIDEO IMAGES WITH
`SECOND IMAGE CAPTURE MODULE
`
`CHECK IMAGE FOR
`FOCUS QUALITY
`
`NO
`
`YES
`
`FIG. 8
`
`APPL-1014 / Page 10 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`00 = N
`
`00
`tit
`\0
`UI
`00
`-....l
`d r.,;_
`
`FIG. 9
`
`0 ....
`.....
`rJJ =- ('D
`
`1,0
`
`('D
`
`1,0
`N
`
`0 ....
`
`N
`
`0
`
`~
`
`QO
`N
`~
`
`c ('D
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`-1-446
`
`DETERMINE THE FOCUS CORRECTION NEEDED
`
`FROM THE OFFSET AND THE AUTOFOCUS
`
`RANGEFINDER CALIBRATION CURVE
`
`i.-444
`
`IMAGES FOR DIFFERENT PORTIONS OF THE IMAGES
`
`DETERMINE THE PIXEL OFFSET BETWEEN THE
`WITH THE CROPPED AND UPSAMPLED IMAGE TO
`CORRELATE THE SECOND AUTOFOCUS IMAGE
`
`IN THE LOWER FOCAL LENGTH IS CROPPED AND
`THE AUTOFOCUS IMAGE FROM THE IMAGE STAGE -1-442
`
`UPSAMPLED SO THAT CORRESPONDING FEATURES
`
`IN THE TWO AUTOFOCUS IMAGES SPAN THE
`
`SAME NUMBER OF PIXELS
`
`LONGER FOCAL LENGTH IMAGE CAPTURE STAGE
`CAPTURE A SECOND AUTOFOCUS IMAGE WITH THE -1-448
`
`LOWER FOCAL LENGTH IMAGE CAPTURE STAGE
`CAPTURE A FIRST AUTOFOCUS IMAGE WITH THE -1-440
`
`APPL-1014 / Page 11 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`00 = N
`
`00
`tit
`\0
`UI
`00
`-....l
`d r.,;_
`
`1,0
`N
`0 .....
`0
`....
`.....
`rJJ =(cid:173)
`
`('D
`('D
`
`NO
`
`472
`
`CHANGED
`CONDITIONS
`
`FOCUS
`
`?
`
`FIG. 10
`
`YES
`
`IMAGE CAPTURE STAGE AND THE AUTOFOCUS
`OF THE FOCUS CONDITIONS FOR THE SECOND
`IMAGE CAPTURE STAGE BASED ON THE CHANGE
`CHANGE THE FOCUS CONDITIONS FOR THE FIRST -~464
`
`"HILL CLIMB METHOD" CALIBRATION CURVE
`
`N
`~CIO
`N
`~
`
`0 ....
`
`0
`
`c ('D
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`470
`
`SECOND IMAGE TO THOSE OF THE ANOTHER IMAGE
`
`COMPARE THE FOCUS CONDITIONS FOR THE
`
`--468
`
`CAPTURE ANOTHER IMAGE WITH THE SECOND
`
`IMAGE CAPTURE STAGE WHEREIN THE AUTOFOCUS
`
`IS DONE BY THE "HILL CLIMB METHOD"
`
`• -466
`
`CAPTURE A SECOND IMAGE WITH THE SECOND
`
`IMAGE CAPTURE STAGE WHEREIN THE AUTOFOCUS
`
`IS DONE BY THE "HILL CLIMB METHOD"
`
`-~462
`
`PROVIDE THE IMAGE FROM THE FIRST
`
`PREVIEW IMAGE ON THE DISPLAY
`
`IMAGE CAPTURE STAGE AS A
`
`CAPTURE STAGE WHEREIN THE AUTOFOCUS IS
`CAPTURE A FIRST IMAGE WITH THE FIRST IMAGE • ~460
`
`DONE BY THE "HILL CLIMB METHOD"
`
`APPL-1014 / Page 12 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`00 = N
`
`00
`tit
`\0
`UI
`00
`-....l
`d r.,;_
`
`FIG. 11
`
`c ('D
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`448
`
`HIGHER FOCAL LENGTH IMAGE CAPTURE STAGE
`CAPTURE A SECOND AUTOFOCUS IMAGE WITH THE ·
`
`DETERMINE THE PIXEL OFFSET BETWEEN THE
`WITH THE CROPPED AND UPSAMPLED IMAGE TO
`
`THE SAME NUMBER OF PIXELS ' CORRELATE THE SECOND AUTOFOCUS IMAGE
`
`LOWER FOCAL LENGTH IMAGE CAPTURE STAGE
`CAPTURE A FIRST AUTOFOCUS IMAGE WITH THE • ~440
`
`!
`
`UPSAMPLED SO THAT CORRESPONDING FEATURES
`IN THE LOWER ZOOM POSITION IS CROPPED AND
`THE AUTOFOCUS IMAGE FROM THE IMAGE STAGE -~442
`
`IN THE TWO AUTOFOCUS IMAGES SPAN
`
`0 ....
`....
`....
`.....
`rJJ =(cid:173)
`
`('D
`('D
`
`1,0
`N
`
`N
`~CIO
`N
`~
`
`0 ....
`
`0
`
`.
`
`.
`
`-~480
`
`IMAGES FOR DIFFERENT PORTIONS OF THE IMAGES
`
`TO DIFFERENT PORTIONS OF THE IMAGES
`PRODUCE A MAP SHOWING THE DISTANCES
`
`AUTOFOCUS RANGEFINDER CALIBRATION CURVE
`FROM THE IMAGE CAPTURE DEVICE USING THE
`CONVERT THE PIXEL OFFSETS TO DISTANCES
`
`'
`'
`
`APPL-1014 / Page 13 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`00 = N
`
`00
`tit
`\0
`UI
`00
`-....l
`d r.,;_
`
`0 ....
`N
`....
`.....
`rJJ =(cid:173)
`
`('D
`('D
`
`1,0
`N
`
`N
`~CIO
`N
`~
`
`0 ....
`
`0
`
`c ('D
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`90
`92
`I
`I
`I
`I
`: PROCESSOR I
`I MODEM
`_.,_ ________ .,
`1..1 _______ •
`I
`1 CELLULAR :_;CELLULAR:
`--------
`r---------~
`MEMORY I
`94
`
`58 d~
`
`COMPASS r59
`
`57
`
`56
`
`REMOVABLE
`
`MEMORY
`
`CARD
`
`GPS
`
`INTERFACE
`
`MEMORY
`
`CARD
`
`54
`
`52
`
`!HOST
`
`• PC
`
`/
`64
`
`I
`
`INTERFACE
`
`HOST
`
`66
`
`62
`
`MEMORY
`
`RAM
`
`PROCESSOR
`
`IMAGE
`
`38
`
`MEMORY
`BUFFER
`DRAM
`
`40
`
`24
`
`MUX1
`
`50
`
`34
`
`1 1 •, 1 MUX2
`
`36
`
`FIG. 12
`
`COLOR LCD
`
`DISPLAY
`IMAGE
`
`70
`
`CONTROLS
`
`USER
`
`42
`
`FLASH i----
`
`I
`
`48-.r
`
`TIMING GENERATOR
`
`PROCESSOR AND
`
`CONTROL
`
`AUTO EXPOSURE ,___ __ I
`
`DETECTORS
`
`FOCUS MOTORS
`
`ZOOM AND
`
`46
`
`5b
`
`15
`
`14e
`
`SENSOR
`IMAGE
`SECOND
`
`LENS
`ZOOM
`SECOND
`
`4 14
`
`2
`
`DRIVERS
`CLOCK
`
`MOTORS
`5c-t FOCUS
`
`13
`
`12e
`
`SENSOR
`IMAGE
`FIRST
`
`LENS
`FOCUS
`FIXED
`FIRST
`
`22
`
`12
`
`71
`
`73
`
`APPL-1014 / Page 14 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`00 = N
`
`00
`tit
`\0
`UI
`00
`-....l
`d r.,;_
`
`1,0
`N
`
`~
`
`0 ....
`....
`.....
`rJJ =(cid:173)
`
`('D
`('D
`
`90
`92
`f
`f
`I
`I
`-.L.--------~
`I.J. _______ J
`: PROCESSOR I
`I MODEM
`I
`1 CELLULAR :_;CELLULAR:
`--------
`r---------~
`MEMORY I
`94
`
`58 d~
`
`COMPASS r59
`
`COLOR LCD
`
`DISPLAY
`IMAGE
`
`70
`
`N
`~CIO
`N
`~
`
`0 ....
`
`0
`
`c ('D
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`57
`
`56
`
`GPS
`
`MEMORY
`
`RAM
`
`PROCESSOR
`
`IMAGE
`
`REMOVABLE
`
`MEMORY
`
`CARD
`
`INTERFACE
`
`MEMORY
`
`CARD
`
`54
`
`IHOST
`
`PC
`
`'
`
`66
`
`52
`I /
`64
`
`62
`
`INTERFACE
`
`HOST
`
`50
`
`FIG. 13
`
`FLASH
`48_,_.--~---,
`
`CONTROLS
`
`USER
`
`42
`
`MEMORY
`BUFFER
`DRAM
`
`38
`
`37
`
`MUX1
`
`34
`
`36
`
`• TIMING GENERATOR
`
`PROCESSOR AND
`
`CONTROL
`
`I
`
`40
`
`26
`
`l AUTO EXPOSURE I
`
`DETECTORS
`
`FOCUS MOTORS
`
`ZOOM AND
`
`46
`
`Sc
`
`24
`
`DRIVERS
`CLOCK
`
`15
`
`14e
`
`DRIVERS
`CLOCK
`
`11
`
`SENSOR
`IMAGE
` THIRD
`
`1,
`
`LENS
`1
`ZOOM
`THIRD
`
`74 75 16
`
`FOCUS MOTORS
`
`ZOOM AND
`
`5b
`
`rm
`
`SENSOR
`IMAGE
`SECOND
`
`LENS
`ZOOM
`SECOND
`
`2 4
`MOTOR
`5a-L FOCUS
`
`22
`
`13
`
`12e
`
`f • I
`•
`
`12
`
`I
`
`71
`
`-;t_
`73
`
`SENSOR
`IMAGE
`FIRST
`
`LENS
`FOCUS
`FIXED
`FIRST1
`
`APPL-1014 / Page 15 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 14 of 29
`
`US 7,859,588 B2
`
`ZOOM POSITION SET TO
`DEFAULT POSITION WHEN
`CAMERA IS POWERED ON
`
`500
`
`YES
`
`NO
`
`SET FIRST IMAGE CAPTURE
`MODULE TO CAPTURE IMAGES
`& SECOND IMAGE CAPTURE
`MODULE FOR AUTO FOCUS
`
`CAPTURE AND DISPLAY
`PREVIEW IMAGES WHILE
`PERFORMING CONTINUOUS
`AUTO-FOCUS USING SECOND
`IMAGE CAPTURE MODULE
`
`504 524
`
`506 526
`
`SET SECOND IMAGE CAPTURE
`MODULE TO CAPTURE IMAGES
`& FIRST IMAGE CAPTURE
`MODULE FOR AUTO-FOCUS
`
`CAPTURE AND DISPLAY
`PREVIEW IMAGES WHILE
`PERFORMING CONTINUOUS
`AUTOFOCUS USING FIRST
`IMAGE CAPTURE MODULE
`
`YES
`
`YES
`
`WHEN SHUTTER BUTTON IS
`PRESSED, CAPTURE A PRIMARY
`STILL IMAGE USING FIRST
`IMAGE CAPTURE MODULE SET
`TO PRIMARY FOCUS POSITION
`
`CAPTURE A SECONDARY
`STILL IMAGE USING SECOND
`IMAGE CAPTURE MODULE
`SET TO SECONDARY
`FOCUS POSITION
`
`USE SECONDARY STILL
`IMAGE TO ENHANCE THE
`DEPTH OF FIELD OF THE
`PRIMARY IMAGE
`
`510 530
`
`512 532
`
`514 534
`
`WHEN SHUTTER BUTTON IS
`PRESSED, CAPTURE A PRIMARY
`STILL IMAGE USING SECOND
`IMAGE CAPTURE MODULE SET
`TO PRIMARY FOCUS POSITION
`
`CAPTURE A SECONDARY
`STILL IMAGE USING FIRST
`IMAGE CAPTURE MODULE
`SET TO SECONDARY
`FOCUS POSITION
`
`USE SECONDARY STILL
`IMAGE TO ENHANCE THE
`DEPTH OF FIELD OF THE
`PRIMARY IMAGE
`
`FIG. 14
`
`APPL-1014 / Page 16 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 15 of 29
`
`US 7,859,588 B2
`
`0
`LO
`I
`
`\
`
`0
`
`0) ' \
`
`,-. - - -1
`\
`Cl'.: I
`I
`,-t- - - - - -1
`1W •Q'
`1C!>Ow'
`I
`I 4'.'. Cl'.: Cl) -----1
`i~a..w'
`I -
`(.) I
`. - - - - - - " I
`I
`
`N
`T""" co
`
`co
`T""" co
`
`l _ -
`
`-
`
`- _I
`
`I
`
`-----,
`: Cl'.:
`I :5~ I
`::)W I
`I
`__J O
`,___ _ _ __,
`I
`1-----------' __J Q
`I W~ I
`I
`I
`l ______ _
`I
`(.)
`I
`l t- - - _I
`I
`I
`N
`0)
`
`O
`0
`co
`
`"-I"
`0 co
`
`co
`0 co
`
`0
`0 co
`
`co
`0 co
`
`APPL-1014 / Page 17 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 16 of 29
`
`US 7,859,588 B2
`
`111111111111111 I I I
`
`D D D
`D D
`
`co N
`co co
`
`T"""
`
`T"""
`
`co
`N
`co
`
`"""'
`N
`co
`
`"""' T""" co
`
`0
`N
`co
`
`co
`T""" co
`
`N
`N co
`
`0
`T""
`co
`
`~,
`
`ca
`
`co
`N
`co
`
`"""'
`N
`co
`
`0
`('I)
`co
`
`~
`"t--
`•
`C)
`u:
`
`N
`T"""
`co
`
`co
`N
`co
`
`co
`T""" co
`
`co
`N co
`
`0
`T""
`co
`
`~~ N
`
`T"""
`
`"""'
`co
`N CQ
`N
`co
`(C)
`"t--
`•
`C)
`
`N u:
`
`N
`co
`
`co
`co
`
`T"""
`
`0
`N
`co
`
`APPL-1014 / Page 18 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 17 of 29
`
`US 7,859,588 B2
`
`Fl
`
`1
`
`Fl 17
`
`APPL-1014 / Page 19 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 18 of 29
`
`US 7,859,588 B2
`
`GPS LOCATION OF THE CAMERA
`PROVIDED BY THE GPS IN THE CAMERA
`
`CAMERA POINTING DIRECTION PROVIDED BY
`THE ELECTRONIC COMPASS IN THE CAMERA
`
`. ...--750
`
`_,,,,,,,.. 752
`
`DISTANCE OFFSETS FROM THE CAMERA TO PORTIONS -,,,,.-754
`OF THE SCENE PROVIDED BY THE RANGEMAP
`
`ANGULAR OFFSET FROM THE CAMERA PROVIDED
`FROM THE LOCATION IN THE FIELD OF VIEW
`
`GPS LOCATIONS FOR PORTIONS OF THE SCENE ARE
`DETERMINED BY ADDING DISTANCE OFFSETS AND THE
`ANGULAR OFFSETS TO THE GPS LOCATION AND
`POINTING DIRECTION OF THE CAMERA
`
`A GPS LOCATIONS FOR THE OBJECTS IN THE SCENE
`ARE STORED IN METADATA OR DISPLA YEO AS LABELS
`IN A GPS LOCATION MAP
`
`.. ,,,,,,,.. 756
`
`_,.,,,,_. 758
`
`.. ,,,,.-760
`
`FIG. 18
`
`APPL-1014 / Page 20 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 19 of 29
`
`US 7,859,588 B2
`
`STARTUP
`
`CAMERA POWER ON
`INITIALIZATION
`
`SET FIRST AND SECOND CAPTURE
`UNITS TO DEFAULT ZOOM POSITIONS
`
`CAPTURE AND DISPLAY FIRST AND
`SECOND PREVIEW IMAGES USING FIRST
`AND SECOND CAPTURE UNITS
`
`11 00
`
`11 02
`
`1104
`
`YES
`
`SET SELECTED
`CAPTURE UNIT
`AS PRIMARY
`CAPTURE UNIT
`
`1124
`
`NO
`
`YES
`
`SET DEFAULT CAPTURE UNIT
`AS PRIMARY CAPTURE UNIT
`
`1112
`
`SET NON-PRIMARY CAPTURE UNIT
`AS SCENE ANALYSIS CAPTURE UNIT
`
`1114
`
`PREVIEW MODE
`
`FIG. 19
`
`APPL-1014 / Page 21 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 20 of 29
`
`US 7,859,588 B2
`
`PREVIEW MODE
`
`1200
`
`1202
`
`SCENE ANALYSIS CAPTURE UNIT
`ANALYZES SCENE
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERS UTILIZING SCENE
`ANALYSIS CAPTURE UNIT DATA
`
`1204
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`1206
`
`SCENE ANALYSIS CAPTURE UNIT
`ANALYZES SCENE
`
`SCENE
`CONDITIONS
`CHANGED
`
`NO
`
`YES
`
`FIG. 20
`
`APPL-1014 / Page 22 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 21 of 29
`
`US 7,859,588 B2
`
`PREVIEW MODE
`WITH THRESHOLD
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERS UTILIZING SCENE
`ANALYSIS CAPTURE UNIT DATA
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`1300
`
`1302
`
`1304
`
`1306
`
`NO
`
`YES
`
`FIG. 21
`
`APPL-1014 / Page 23 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 22 of 29
`
`US 7,859,588 B2
`
`1400
`
`1402
`
`1404
`
`1406
`
`1408
`
`1410
`
`1412
`
`1414
`
`1416
`
`ENHANCED
`PREVIEW MODE
`
`SET SCENE ANALYSIS CAPTURE UNIT
`ZOOM POSITION RELATIVE TO PRIMARY
`CAPTURE UNIT ZOOM POSITION
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERS UTILIZING SCENE
`ANALYSIS CAPTURE UNIT DATA
`
`CAPTURE PREVIEW IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`ANALYZE SCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS DATA
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERS UTILIZING RESULTS
`OF THE SCENE ANALYSIS
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`ANALYZE SCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS DATA
`
`NO
`
`FIG. 22
`
`APPL-1014 / Page 24 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 23 of 29
`
`US 7,859,588 B2
`
`ZOOM
`
`YES
`
`YES
`
`>-N_O_( RETURN TO PREVIEW )
`
`NO
`
`1504
`
`SET CURRENT SCENE ANALYSIS
`AND PRIMARY CAPTURE UNITS AS
`PRIMARY AND SCENE ANALYSIS
`CAPTURE UNITS RESPECTIVELY
`
`1506
`
`SET PRIMARY CAPTURE UNIT ZOOM
`POSITION TO SELECTED ZOOM POSITION
`
`RETURN TO PREVIEW
`
`FIG. 23
`
`APPL-1014 / Page 25 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 24 of 29
`
`US 7,859,588 B2
`
`CAPTURE
`
`YES
`
`NO C RETURN TO PREVIEW )
`
`1602
`
`1604
`
`CAPTURE PREVIEW IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`ANALYZE SCENE UTILIZING
`CAPTURED PREVIEW IMAGE
`
`NO
`
`1608
`
`SET PRIMARY CAPTURE UNIT PARAMETERS
`UTILIZING RESULTS OF THE ANALYSIS
`
`1610
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`1614
`
`CAPTURE PRIMARY IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`RETURN TO PREVIEW
`
`FIG. 24
`
`YES
`
`NO
`
`APPL-1014 / Page 26 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 25 of 29
`
`US 7,859,588 B2
`
`CAPTURE
`
`YES
`
`>-N_O ___ ( RETURN TO PREVIEW )
`
`1702
`
`1704
`
`1706
`
`1708
`
`CAPTURE PREVIEW IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`ANALYZE SCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS IMAGES
`
`SET PRIMARY CAPTURE UNIT
`PARAMETERS UTILIZING RESULTS
`OF THE SCENE ANALYSIS
`
`1710
`
`CAPTURE AND DISPLAY PREVIEW
`IMAGE FROM PRIMARY CAPTURE UNIT
`
`1714
`
`CAPTURE PRIMARY IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`RETURN TO PREVIEW
`
`FIG. 25
`
`NO
`
`APPL-1014 / Page 27 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 26 of 29
`
`US 7,859,588 B2
`
`CAPTURE
`
`YES
`
`NO C RETURN TO PREVIEW )
`
`1802
`
`1804
`
`1806
`
`1808
`
`1810
`
`1812
`
`1816
`
`1818
`
`1820
`
`CAPTURE PREVIEW IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`CAPTURE IMAGE FROM
`SCENE ANALYSIS CAPTURE UNIT
`
`ANALYZE SCENE UTILIZING CAPTURED
`PREVIEW AND SCENE ANALYSIS IMAGES
`
`SET PRIMARY CAPTURE UNIT PARAMETERS
`UTILIZING RESULTS OF THE SCENE ANALYSIS
`
`SET SCENE ANALYSIS CAPTURE UNIT
`AS A SECONDARY CAPTURE UNIT
`
`SET SECONDARY CAPTURE UNIT PARAMETERS
`UTILIZING RESULTS OF THE SCENE ANALYSIS
`
`NO
`
`CAPTURE PRIMARY IMAGE
`FROM PRIMARY CAPTURE UNIT
`
`CAPTURE AN AUGMENTATION IMAGE
`FROM SCENE ANALYSIS CAPTURE UNIT
`
`PRODUCE AN ENHANCED IMAGE FROM
`THE PRIMARY AND AUGMENTATION IMAGES
`
`RETURN TO PREVIEW
`
`FIG. 26
`
`APPL-1014 / Page 28 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 27 of 29
`
`US 7,859,588 B2
`
`d
`
`f
`
`191
`
`I ,
`'
`'
`'
`155__,·
`' ' ' ' '
`I '
`' I
`' I
`' I
`' I
`
`I
`
`I
`I
`
`' ' ' ' ' ,
`
`,,
`
`I
`
`'
`
`I
`
`151
`.
`' \ \--154
`' ' .
`' I .
`' ' ' I
`
`I
`\
`I
`I
`I
`I
`I
`I
`
`' I
`
`I
`I
`I
`\
`I
`
`152
`
`153
`
`190
`
`181
`
`180
`
`170
`
`171
`
`b
`
`FIG. 27
`(PRIOR ART)
`
`APPL-1014 / Page 29 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 28 of 29
`
`US 7,859,588 B2
`
`197 -..
`
`198 --
`198 -..
`199 --
`
`C
`
`START
`
`SET LENS TO MID FOCUS AND
`LOAD LOW PASS FILTER
`
`CALCULATE FOCUS VALUE (FV)
`AS IMAGE IS READ OUT
`
`ADJUST STEPPER MOTOR TO
`NEXT NEAR FOCUS POSITION
`
`GET FV OF NEXT IMAGE CAPTURED
`WITH NEW FOCUS SETTING
`
`STEP IN OPPOSITE
`DIRECTION
`
`FV DECREASES
`
`COMPARE TO PREVIOUS
`WEIGHTED FOCUS VALUES
`
`FV INCREASES
`
`STEP IN SAME DIRECTION
`UNTIL FV DECREASES
`
`LOAD HIGH PASS FILTER
`AND MEASURE FV USING
`SAME FOCUS POSITION
`
`STEP IN SAME DIRECTION
`UNTIL FV DECREASE
`
`BACK UP AND HOLD AT
`HIGHEST FV FOCUS POSITION
`
`FOCUS OK
`
`FIG. 28
`(PRIOR ART)
`
`APPL-1014 / Page 30 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 28, 2010
`
`Sheet 29 of 29
`
`US 7,859,588 B2
`
`FOCUS
`,.,,,-- ADJUSTMENT
`STEPS
`
`I
`I
`I
`I
`
`,,--::'~
`/i
`
`I
`
`FOCUS
`VALUE
`
`l
`
`FOCUS
`POSITION
`
`I
`
`P---1
`
`FIG. 29
`(PRIOR ART)
`
`APPL-1014 / Page 31 of 49
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`US 7,859,588 B2
`
`1
`METHOD AND APPARATUS FOR
`OPERATING A DUAL LENS CAMERA TO
`AUGMENT AN IMAGE
`
`FIELD OF THE INVENTION
`
`The present invention relates to a digital camera that pro(cid:173)
`duces digital image files and, more particularly, to a digital
`camera that uses multiple lenses and image sensors to provide
`an improved imaging capability.
`
`BACKGROUND OF THE INVENTION
`
`5
`
`2
`A diagram illustrative of the principle of the operation of a
`conventional rangefinder is shown herein in FIG. 27. In that
`diagram, light from an object 151 is incident on two small
`lenses 152 and 153 which have a sufficiently short focal
`length fthat light rays received from the object through dif(cid:173)
`ferent spaced paths 154 and 155 produce corresponding
`spaced images 157 and 158 in a focal plane 156 which is
`common to the lenses 152 and 153. When the object 151 is at
`an infinite distance, the centers of the images 157 and 158 are
`10 located at reference positions 170 and 180 in FIG. 27, but
`when the object 151 is located at a closer distance, the centers
`of the images are shifted apart to positions 171 and 181. If the
`distance by which the images 157 and 158 are shifted from the
`reference positions 170 and 180 are designated x 1 and x2 ,
`15 respectively, then the total shift x may be expressed as fol(cid:173)
`lows:
`
`Currently, most digital cameras use a zoom taking lens and
`a single color image sensor to capture still and motion images.
`The captured images are then processed to produce digital
`image files, which are stored in a digital memory in the
`camera. The digital image files can then be transferred to a
`computer, displayed, printed, and shared via the Internet.
`In order to capture sharp images of moving subjects, a 20
`digital camera needs to provide a precise automatic lens
`focusing system (i.e., an autofocus system). The autofocus
`system must be capable of quickly obtaining the correct focus
`in order to minimize the "shutter delay" between the time the
`shutter button is pressed and the still image is captured. The 25
`autofocus system must also work in a continuous image cap(cid:173)
`ture mode wherein video images are captured. For instance, in
`a video mode the focus should be adjusted in real-time while
`video images are being continuously captured.
`Many digital cameras and scanners capture images using 30
`an image sensor and a taking lens system with an adjustable
`focus. Typically, the focus distance of such an adjustable
`focus taking lens system can automatically be set to one of a
`plurality of different settings by sensing, control, and drive
`systems, which are adapted to provide optimal focus of what 35
`is determined to be a subject area in a scene. Lens systems that
`provide automatically adjustable focus settings based on a
`focus measurement and an adjustable focus lens are referred
`to herein as autofocus systems. Digital cameras typically use
`one of two types of autofocus systems: a rangefinder system 40
`and a "through-the-lens" focus system.
`A rangefinder system uses rangefinding sensors such as a
`sonic rangefinder or a dual lens rangefinder to determine the
`distance from a camera to one or more portions of a scene
`within a field of view of the rangefinder system. A sonic 45
`rangefinder measures the phase offset between a projected
`sonic signal and a reflected sonic signal to infer the distance to
`objects in the scene. Dual lens rangefinders contain two
`lenses that are separated by a distance along with two match(cid:173)
`ing sensor areas that capture matched pairs of images. Dual 50
`lens rangefinders are commonly used on digital cameras in
`the form of dual lens rangefinder modules which contain two
`lenses separated by a distance along with two matching sen(cid:173)
`sor areas that capture matched pairs oflow resolution images.
`Common dual lens rangefinder-based autofocus systems 55
`include active and passive systems. Active systems actively
`project light onto the scene, while passive systems work with
`the available light from the scene. Dual lens rangefindermod(cid:173)
`ules can be purchased from Fuji Electric in several models
`such as the FM6260W. A dual lens rangefinder module for 60
`optical apparatus such as a camera is described in U.S. Pat.
`No. 4,606,630, which was issued to Haruki et al. on Aug. 19,
`1986 (and assigned to Fuji Electric). According to the
`description of the prior art in this patent, matched pairs oflow
`resolution images are analyzed for correlation between the 65
`two images to determine the offset between the two images
`caused by the separation between the two lenses.
`
`Thus, the distance d to the object 151 can be measured by
`d=b·f/x. In this case, bis the distance between the optical axes
`of the small lenses, that is, the base length. To obtain the
`shifted amounts x 1 and x2 , or the sum x of both, two optical
`sensor arrays 190 and 191 are provided in the focal plane 156
`as shown in FIG. 27. These optical sensor arrays each com(cid:173)
`prise a plurality of optical sensors, for instance CCD devices,
`and an analog photoelectric signal is generated by each opti(cid:173)
`cal sensor corresponding to the light intensity at the portion of
`the image which is incident on the sensor. Haruki et al. shows
`a conventional circuit, as well as a higher speed rangefinding
`circuit according to the patent, for obtaining the sum x of the
`shifted distances by comparing two image signal trains com(cid:173)
`prising the digital image signals from the left and right optical
`sensor arrays.
`Basically, the offset information x is used along with the
`lens separation distance band the focal length fto calculate
`the distanced to the scene by triangulation. The calculated
`distance d to the scene is used to guide the positioning of an
`adjustable focus lens to produce the best image quality. As
`known in the prior art, this adjustment may be based on a
`calibration curve established between the distance to the
`scene as measured by the dual lens rangefinder module and a
`series of best focused images as produced by a "through the
`lens" autofocus system. The calibration curve is stored as an
`equation or a look-up table in a microprocessor in the camera.
`Rangefinder-based autofocus systems have the advantage
`of being very fast, some having a response time that can be in
`the range of 0.01-0.05 second. However, the focus quality
`produced by some rangefinder-based autofocus systems can
`vary when they are used in different operating conditions. For
`example, sonic autofocus systems cannot focus through a
`glass window as the glass stops the projected sonic signal,
`thereby causing th