`
`
`(19) United States
`
`
`
`
`
`
`
`
`(12) Patent Application Publication (10) Pub. No.: US 2008/0030592 A1
`
`
`
`
`
`
`
`
`Border et al. Feb. 7, 2008 (43) Pub. Date:
`
`
`US 20080030592A1
`
`
`
`
`(54) PRODUCING DIGITAL IMAGE WITH
`
`
`
`
`
`DIFFERENT RESOLUTION PORTIONS
`
`
`
`
`(75)
`
`
`
`Inventors:
`
`
`
`
`
`
`John N. Border, Walworth, NY
`fiiUssgotginC3112:Eifilrpons
`
`Rochester, NY (US)
`a
`
`
`
`
`EggfigggfilfiifisfigS(S:OMPANY
`
`
`
`
`
`PATENT LEGAL STAFF
`
`
`
`343 STATE STREET
`
`ROCHESTER NY 14650-2201
`
`’
`
`
`Eastman Kodak Company
`
`
`
`
`
`
`
`
`11/461,574
`
`
`
`
`
`(73) Assignee:
`
`
`(21) Appl. No.:
`
`
`(22)
`
`Filed:
`
`
`
`
`Aug. 1, 2006
`
`
`
`Publication Classification
`
`
`
`
`
`
`(51)
`
`Int. Cl.
`
`
`
`
`
`G06T 5/50
`(200601)
`(52) us. Cl.
`............................. 348/218.1; 348/E05.028
`
`
`
`
`
`
`
`
`ABSTRACT
`(57)
`
`
`image with improved
`A method of producing a digital
`
`
`
`
`
`
`
`
`
`
`
`
`
`resolution during digital zooming, including simultaneously
`
`
`
`
`
`
`
`
`capturing a first low resolution digital image of a scene and
`
`
`
`
`
`
`
`
`a second higher resolution digital image of a portion of
`
`
`
`
`
`
`
`substantially the same scene. A composite image is then
`
`
`
`
`
`
`
`formed by combining the first low-resolution digital image
`
`
`
`
`
`
`
`
`and a corresponding portion of the high resolution digital
`
`
`
`
`
`
`image. Digital zooming of the composite image produces a
`
`
`
`
`
`
`
`
`zoomed image with high resolution throughout the zoom
`
`
`
`
`
`range and improved image quality.
`
`
`
`24
`
`
`F/XEf)
`FOCADL
`‘
`
`LENGIH J
`
`
`LENS
`
`5
`
`
`
`
`
`
`52W EDA—r—
`34
`72
`
`
`
`
`
`
`
`
`
`
`
`I e
`HOSE
`HOST
`S
`5
`50
`H
`
`
`
`
`
`
`
`
`I_NIEREAOE&5
`PC /
`Ist A/D +
`g E
`Ist IMAGE g ASP 7
`
`
`
`
`
`
`52
`L— OONVERIER ‘
`SENSOR
`
`
`
`
`
`
`
`OTC/F
`I )
`‘ MEMORY
`REMOVABL‘
`36
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`«H CARD
`N 2nd A/D .,
`
`'L/NIERFACE
`comm? I‘
`DRIVERS
`3w 7;;
`’3
`CARD
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`I46
`I
`
`
`
`
`
`55
`1
`ZOOM
`l
`IMAGE
`2ND IMAG
`
`
`5
`LENS I
`;
`SENSORE ‘fijAsp 2)
`_
`I
`IPROCESSOR
`
`
`
`
`
`
`
`
`
`
`RAM
`f DIGITAL
`/
`
`
`
`
`MEMORY,
`37
`,5
`MUX
`
`
`CLOCK
`58
`58
`ZOOM AND « DRIVERS
`_
`
`
`I
`FOCUS MOTORS
`}
`I
`I
`HRMWAREI
`
`
`
`
`
`
`L CONIROL
`I
`<5 MEMORY
`
`
`
`
`
`40"— PROCESSOR AND
`
`AUTOFOCUS AND ~— OEIEIIIIOR
`
`
`
`
`
`
`
`46" AUTO EXPOSURE _
`COLOR LCD
`
`f
`
`
`
`
`
`
`70"“ IMACE
`DEIECTORS'
`
`
`DISPLA Y
`
`
`
`
`
`
`‘
`
`I
`
`
`
`
`
`DRAM :
`I
`
`
`—-
`5955?;
`
`
`
`48"» FLASH «m
`
`
`
`
`USER N n
`CONTROLS“
`4‘
`
`
`
`
`
`
`
`.
`
`
`70A
`
`
`
`
`
`MEMORY
`
`
`I
`
`
`
`I
`
`
`"EEZLDLZLR _ 92’
`
`90,\_1
`
`
`IPEOCESSOR
`
`
`
`94%
`
`‘ T ' I 7
`I4—CELLULAR I
`
`
`
`HMODEM
`
`
`
`
`APPL-1009/ Page 1 of 17
`Apple v. Corephotonics
`
`APPL-1009 / Page 1 of 17
`Apple v. Corephotonics
`
`
`
`_*BEER/éE
`TEEJ
`I
`/i$333A.ér{{{{{Mi
`Qmmwomh
`ES:
`
`EE:§aExt#.‘EGET.momzmm
`Fir!s:«888%msass
`Q3to39
`
`«638%NMossE
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ES:
`
`age:a}E7ESESE3“me:We*5mmEN
`w23aEE28”Sam$00)qu
`
`
`
`
`Patent Application Publication
`
`
`
`
`
`
`
`Feb. 7, 2008 Sheet 1 0f 8
`
`
`
`US 2008/0030592 A1
`
`wn
`
`§\\as:
`E7E
`
`.flm§03m,,
`
`xkeém:
`
`EWe
`
`
`
`amass85%:«SEEWEEm933EN639
`
`illES:MEin3E8{a$82$8
`mmm§§9%:8m
`
`*9qu
`
`we$5
`
`x3&3
`
`3%
`
`023:
`
`«E"$me
`
`a)?$9883
`
`@3802.“E?
`
`.32BBQ
`
`x:.0;
`
`Q.3me28
`
`E3
`
`E2
`
`APPL-1009/ Page 2 of 17
`
`APPL-1009 / Page 2 of 17
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Patent Application Publication
`
`
`
`
`
`
`Feb. 7, 2008 Sheet 2 0f 8
`
`
`
`US 2008/0030592 A1
`
`
`
`
`
`3m§e§:95:
`
`
`
`«82mmE23x65%:%a}ENa?§§3gmm2\\as:onmNNmmxmma
`Na.Mg”:EEEEQ
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`:05:956SE§d
`93gag:E§§WEE2w
`
`«8385mugszExt8mgiEN
`
`RESmEE$08:398
`«as2ES::3a”?
`$25%«82mmSE
`
`
`
`
`
`1IIIIIJnlllllJElms
`
`_E8:_émmam}(amugEQEEQ
`
`EsewmmkléEGEEaas«38«2E59#:8qu82Qa2%a:$9823
`is“a:”$ng3$.58
`
`
`$3.3m
`
`:05:ES:mmt§§<t
`
`
`
`m:.UE
`
`Ema
`
`N“.38:28aimIlia.
`
`APPL-1009/ Page 3 of 17
`
`APPL-1009 / Page 3 of 17
`
`
`
`
`
`
`
`Patent Application Publication
`
`
`
`
`
`
`
`Feb. 7, 2008 Sheet 3 0f 8
`
`
`
`
`
`
`
`US 2008/0030592 A1
`
`
`
`
`
`
`
`
`
`
`FIG. 2B
`
`APPL-1009/ Page 4 of 17
`
`APPL-1009 / Page 4 of 17
`
`
`
`
`
`Patent Application Publication
`
`
`
`
`
`
`Feb. 7, 2008 Sheet 4 0f 8
`
`
`
`US 2008/0030592 A1
`
`
`
`i
`
`'PR
`
` 5
`f'c—Eilijiifl
`i MODEM gt.....PM“;
`_________I
`
`
`:
`
`.90
`
`APPL'1009/ Page 5 of 17
`
`APPL-1009 / Page 5 of 17
`
`
`
`
`
`Patent Application Publication
`
`
`
`b.eF
`
`
`000027,
`
`
`
`S
`
`te
`
`0
`
`00
`
`2SU
`
`
`1A295030mw
`
`
`
`
`
`
`Em
`
`
`
`
`
`
`
`mmm
`r,
`511:!
`__.___..__—__—_-__-__J
`
`
`mm--
`
`
`TIllllllllllllllllrl
`
`V-
`
`0Ne
`
`APPL-1009/ Page 6 of 17
`
`APPL-1009 / Page 6 of 17
`
`
`
`
`Patent Application Publication
`
`
`
`
`
`
`
`Feb. 7, 2008 Sheet 6 0f 8
`
`
`
`
`
`
`
`
`
`MK\MQDECO
`
`
`
`mmYVS
`
`
`
`
`
`QMqQE«@wa
`
`m6<3:
`
`
`
`
`
`NON
`
`
`
`
`
`20R«mtMGME
`
`QMZREMRMud
`
`MG<3:
`
`OKOIQQHNN
`
`m6{3:
`
`wQN
`
`mow
`
`
`
`SN
`
`
`
`
`
`US 2008/0030592 A1
`
`m.65%
`
`
`APPL-1009/ Page 7 of 17
`
`APPL-1009 / Page 7 of 17
`
`
`
`
`
`
`
`
`
`
`Patent Application Publication
`
`
`
`
`
`
`Feb. 7, 2008 Sheet 7 0f 8
`
`
`
`US 2008/0030592 A1
`
`-
`
`
`
`220
`
`/\
`
`
`
`204
`
`
`206
`
`
`208
`
`FIG. 6
`
`APPL-1009/ Page 8 of 17
`
`APPL-1009 / Page 8 of 17
`
`
`
`
`
`
`
`
`
`Patent Application Publication
`
`
`
`
`
`
`Feb. 7, 2008 Sheet 8 0f 8
`
`
`
`US 2008/0030592 A1
`
`VNN
`
`MR$00309
`
`
`
`
`
`DMQS
`
`NON
`
`
`
`20R3E{£6wa
`
`km2§<¢mnm5
`
`mm<3:
`
`mmm«\VSOMQS
`
`QRQIQMHHR
`
`OWN
`
`
`
`WNW
`
`
`
`k65%
`
`
`
`
`
`QMQQESWINE
`
`mm<3:
`
`.N>303iEOON
`
`
`EN
`
`
`
`APPL-1009/ Page 9 of 17
`
`APPL-1009 / Page 9 of 17
`
`
`
`
`
`
`
`
`
`US 2008/0030592 A1
`
`
`Feb. 7, 2008
`
`
`
`
`
`PRODUCING DIGITAL IMAGE WITH
`
`
`
`
`DIFFERENT RESOLUTION PORTIONS
`
`
`
`CROSS REFERENCE TO RELATED
`
`
`APPLICATIONS
`
`
`
`
`
`
`
`[0001] Reference is made to commonly assigned US.
`
`
`
`
`
`
`
`
`Patent Application Serial No. 2002/0075258, filed Nov. 23,
`
`
`
`
`
`
`
`
`2001, entitled “Camera System with High Resolution Image
`
`
`
`
`
`
`
`
`
`Inside a Wide Angle View” by Park et al. and US. patent
`
`
`
`
`
`
`
`
`
`application Ser. No. 11/062,174, filed Feb. 18, 2005, entitled
`
`
`
`
`
`
`
`
`“Digital Camera Using Multiple Lenses And Image Sensors
`
`
`
`
`
`
`
`
`
`To Provide An Extended Zoom Range” by Peter Laba-
`
`
`
`
`
`
`
`
`ziewicz, et al., the disclosures of which are incorporated
`herein.
`
`
`FIELD OF THE INVENTION
`
`
`
`
`
`
`
`
`
`
`
`[0002] The present invention relates to a digital camera
`
`
`
`
`
`
`
`
`that uses multiple lenses and image sensors to provide an
`
`
`
`
`
`
`
`
`extended zoom range and the method used to produce a
`
`
`
`
`
`
`
`
`digital image that combines the multiple images produced
`
`
`
`
`by the digital camera.
`BACKGROUND OF THE INVENTION
`
`
`
`
`
`
`
`
`
`
`
`[0003] Currently, most digital cameras use a zoom lens
`
`
`
`
`
`
`
`
`
`and a single color image sensor to capture still and motion
`
`
`
`
`
`
`
`
`images. The captured images are then digitally processed to
`
`
`
`
`
`
`
`
`produce digital image files, which are stored in a digital
`
`
`
`
`
`
`
`
`
`memory in the camera. The digital image files can then be
`
`
`
`
`
`
`
`
`transferred to a computer, displayed, and shared via the
`Internet. The digital camera can be included as part of a
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`mobile telephone, to form a so-called “camera phone.” The
`
`
`
`
`
`
`
`
`
`camera phone can transmit the digital image files to another
`
`
`
`
`
`
`
`camera phone, or to service providers, via a mobile tele-
`
`
`phone network.
`[0004]
`Small camera size and a large zoom range are two
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`very important features of digital cameras. Users prefer to
`
`
`
`
`
`
`
`
`
`
`have a large zoom range (e.g. 5:1 or greater) rather than a
`
`
`
`
`
`
`
`
`
`
`limited zoom range (e.g. 3:1 or smaller). The zoom range is
`
`
`
`
`
`
`
`typically composed of both optical zoom which is provided
`
`
`
`
`
`
`
`
`
`by variable focal length lenses and digital zoom which is
`
`
`
`
`
`
`
`
`provided by a magnification of the digital
`image after
`
`
`
`
`
`
`
`
`
`capture. Variable focal length lenses for large zoom range
`
`
`
`
`
`
`
`
`
`
`are expensive and they increase the size of the digital
`camera. Thus, there are trade-offs between small camera
`
`
`
`
`
`
`
`
`size, large zoom range, and low camera cost which must be
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`made when designing a digital camera. With higher cost
`
`
`
`
`
`
`
`
`cameras, such as single lens reflex cameras, these problems
`
`
`
`
`
`
`are sometimes addressed by using multiple interchangeable
`zoom lenses, such as two 3:1 zoom lenses, e.g., a 28-70 mm
`
`
`
`
`
`
`
`
`zoom and a 70-210 zoom. This arrangement has user incon-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`venience problems and is presently not available for low
`
`
`
`cost digital cameras.
`[0005] A different solution that has been offered by Kodak
`
`
`
`
`
`
`
`
`in the V570 and the V610 cameras is to include two different
`
`
`
`
`
`
`
`
`lens assemblies in the camera with two different focal
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`lengths and two separate image sensors. In this case, each of
`the lens assemblies can be either a fixed focal length lens or
`
`
`
`
`
`
`
`
`
`can have a moderate optical zoom range to reduce the size
`
`
`
`
`
`
`
`
`
`and cost of each of the lens assemblies. Together, the two
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`lens assemblies provide a wide zoom range and a small
`overall size at a lower cost. However, a problem arises when
`
`
`
`
`
`
`
`
`the focal length of the first lens does not match the focal
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`length of the second lens so that the optical zoom is not
`
`
`
`
`
`
`
`
`
`
`continuous over the entire zoom range. In this case, digital
`zoom must be used for zoom between the maximum zoom
`
`
`
`
`
`
`
`
`
`of the first lens and the minimum zoom of the second lens.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`[0006] Digital zoom based on increased magnification of
`
`
`
`
`
`
`the image with a corresponding decrease in resolution is
`
`
`
`
`
`
`
`
`
`
`well known in the art. Although digital zoom is very fast and
`
`
`
`
`
`
`
`simple, the decrease in resolution can produce a perceived
`
`
`
`decrease in image quality.
`[0007]
`In US. Pat. No. 5,657,402, a method is described
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`in which a plurality of digital images are combined to form
`
`
`
`
`
`
`
`
`
`
`an image. US. Pat. No. 5,657,402 addresses the use of
`
`
`
`
`
`
`
`multiple images captured at different times wherein “the
`
`
`
`
`
`
`
`plurality of images of various focal lengths, such as a zoom
`
`
`
`
`
`
`
`
`
`video sequence” (col. 1, lines, 21-22) are captured from the
`same lens. US. Pat. No. 5,657,402 does not address two lens
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`assemblies simultaneously capturing images of the same
`scene.
`
`
`
`
`
`
`
`In US Publication No. 2002/0075258, a panoramic
`[0008]
`
`
`
`
`
`
`camera system is described in which a moveable telephoto
`
`
`
`
`
`camera is additionally used to capture a high-resolution
`
`
`
`
`
`
`
`
`
`
`portion of the scene which is then overlaid onto the pan-
`
`
`
`
`
`
`oramic image. US Publication No. 2002/0075258 describes
`
`
`
`
`
`
`
`the use of a moveable telephoto camera to enable a higher
`
`
`
`
`
`
`resolution of a portion of the image wherein the moveable
`
`
`
`
`
`
`
`
`telephoto camera can be moved to the region of the pan-
`
`
`
`
`
`
`
`
`oramic image where the higher resolution is desired. US
`Publication No. 2002/0075258 does not address the case
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`wherein a wide-angle camera and a telephoto camera are
`
`
`
`
`
`
`
`affixed together for simultaneous capture of the same scene.
`In addition, US Publication No. 2002/0075258 does not
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`disclose the use of a composite image for improved image
`
`
`
`
`quality in a digital zoom system.
`SUMMARY OF THE INVENTION
`
`
`
`
`
`
`
`
`
`
`[0009] The present invention provides a sufficiently com-
`
`
`
`
`
`
`
`
`
`
`pact, low cost, optical system with a large zoom range for a
`
`
`
`
`
`
`
`small, lightweight and relatively inexpensive consumer digi-
`tal camera.
`
`
`[0010] What is therefore needed is a digital camera that
`
`
`
`
`
`
`
`
`
`
`
`provides a rapidly-operating extended zoom range without
`
`
`
`
`
`
`
`unduly increasing the size or cost of the digital camera while
`
`
`
`
`
`
`providing good perceived image quality throughout
`the
`zoom range.
`
`
`
`
`
`
`
`
`
`[0011] An object of the invention is to provide a method
`
`
`
`
`
`
`
`of producing a digital image having portions with different
`
`
`resolutions comprising:
`
`
`
`
`
`
`
`
`a. simultaneously capturing first and second digital
`[0012]
`
`
`
`
`
`
`
`
`images of the same scene wherein the first digital image is
`
`
`
`
`
`
`
`
`
`of a larger portion of the scene than the second digital image
`
`
`
`
`
`
`
`
`wherein the second digital image has a higher resolution
`
`
`
`
`
`
`
`
`than the resolution in the first digital image corresponding to
`
`
`
`
`
`the second digital image; and
`
`
`
`
`
`
`[0013]
`b. combining at least a portion of the second digital
`
`
`
`
`
`
`
`
`image into the corresponding portion of the first digital
`
`
`
`
`
`
`image to thereby provide a digital image having portions
`with different resolutions.
`
`
`
`
`
`
`
`
`
`
`[0014] The present invention is directed to overcoming the
`
`
`
`
`
`
`
`problems set forth above. Briefly summarized, the invention
`
`
`
`
`
`
`includes an electronic camera for producing an image of a
`scene, wherein the camera includes a first image sensor for
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`generating a first sensor output, a first lens with a first focal
`
`
`
`
`
`
`
`
`
`length for forming a first image of the scene on the first
`
`
`
`
`
`
`
`
`image sensor, a second image sensor for generating a second
`
`
`
`
`
`
`
`
`
`sensor output, and a second lens with a second focal length
`APPL—1009 / Page 10 of 17
`
`
`
`
`
`
`
`
`
`
`
`
`
`APPL-1009 / Page 10 of 17
`
`
`
`
`
`US 2008/0030592 A1
`
`
`Feb. 7, 2008
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`length of the first lens for
`that is longer than the focal
`
`
`
`
`
`
`
`
`forming a second image of the same scene on the second
`
`
`
`
`
`
`
`
`
`
`image sensor. The first lens or the second lens can be either
`
`
`
`
`
`
`
`
`fixed focal length lenses or multiple focal length lenses as in
`a zoom lens wherein, the first and second lenses are directed
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`at substantially the same scene and image sets are captured
`
`
`
`
`
`
`
`substantially simultaneously by the first image sensor and
`
`
`
`
`
`
`
`the second image sensor. Portions of the image set captured
`
`
`
`
`
`
`
`
`
`
`
`by the first image sensor and the second image sensor are
`
`
`
`
`
`
`
`then combined to produce a composite image with a higher
`
`
`
`
`
`
`
`
`
`resolution in the portion of the composite image that is
`
`
`
`
`
`
`
`
`provided by the second image sensor due to the longer focal
`
`
`
`
`
`
`
`
`length of the second lens. Subsequent
`images produced
`
`
`
`
`
`
`
`during a digital zooming process are composed largely of the
`
`
`
`
`
`
`
`lower resolution image captured by the first image sensor at
`
`
`
`
`
`
`
`
`low digital zoom values and largely of the higher resolution
`
`
`
`
`
`
`
`
`image as captured by the second image sensor at high digital
`zoom values.
`
`
`
`
`
`
`
`
`
`[0015] By forming a composite image with portions of the
`
`
`
`
`
`
`
`
`
`image from the short focal length lens and portions of the
`
`
`
`
`
`
`
`
`
`image from the longer focal length lens, perceived image
`
`
`
`
`
`
`
`
`quality is improved throughout the zoom range while lens
`
`
`
`
`
`
`
`complexity is reduced, since a continuous zoom ratio can be
`
`
`
`
`
`
`
`
`produced with unmatched lens focal lengths. By capturing
`
`
`
`
`
`
`
`
`images from the two image sensors substantially simulta-
`
`
`
`
`
`
`
`neously, complexities in the image processing are reduced
`
`
`
`
`
`
`
`
`since differences between the two images due to motion of
`the camera or motion within the scene are avoided. It is an
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`additional advantage, that the present invention can avoid
`
`
`
`
`
`
`
`
`the slow response that is typical of an optical zoom system
`
`
`
`
`
`when traversing a large zoom range.
`
`
`
`
`
`
`
`
`features and
`[0016] These and other aspects, objects,
`
`
`
`
`
`
`
`
`
`advantages of the present invention will be more clearly
`
`
`
`
`
`
`understood and appreciated from a review of the following
`
`
`
`
`
`
`
`detailed description of the preferred embodiments and
`
`
`
`
`
`
`
`appended claims, and by reference to the accompanying
`
`drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIGS. 1A and 1B depict a block diagram of a
`[0017]
`
`
`
`
`
`
`
`
`digital camera using a fixed focal length wide-angle lens
`with a first image sensor and a zoom lens, or a longer second
`
`
`
`
`
`
`
`
`
`fixed focal length lens, with a second image sensor accord-
`
`
`
`
`
`
`
`
`
`
`
`
`
`ing to the present invention;
`[0018]
`FIGS. 2A and 2B show front and rear perspective
`
`
`
`
`
`
`
`
`
`
`views of the digital camera;
`[0019]
`FIGS. 3A and 3B are perspective views of the front
`
`
`
`
`
`
`
`
`
`
`
`
`
`and back of a cell phone including a camera with multiple
`
`
`
`
`
`
`
`
`lenses and multiple sensors according to the present inven-
`tion;
`
`FIGS. 4A and 4B show two views of the capture
`[0020]
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`assembly used in the cell phone shown in FIGS. 3A and 3B
`
`
`
`
`
`
`FIG. 5 is a block diagram of the stitching process
`[0021]
`
`
`
`
`
`to create the composite image;
`
`
`
`
`
`
`
`[0022]
`FIG. 6 depicts a wide angle image as captured, a
`
`
`
`
`
`
`telephoto image as captured, and a composite image as
`created by the invention; and
`
`
`
`
`
`
`
`
`
`
`FIG. 7 is a block diagram of the stitching process
`[0023]
`
`
`
`
`
`
`with video images to create a composite video.
`DETAILED DESCRIPTION OF THE
`
`
`INVENTION
`
`
`
`
`
`
`
`
`
`
`
`
`cameras
`employing imaging
`[0024] Because digital
`
`
`
`
`
`
`
`
`devices and related circuitry for signal capture, correction,
`
`
`
`
`
`
`
`
`and exposure control are well known, the present description
`
`
`
`
`
`
`
`will be directed in particular to elements forming part of, or
`
`
`
`
`
`
`
`cooperating more directly with, method and apparatus in
`
`
`
`
`
`
`
`
`accordance with the present invention. Elements not spe-
`
`
`
`
`
`
`
`
`cifically shown or described herein are selected from those
`
`
`
`
`
`
`known in the art. Certain aspects of the embodiments to be
`
`
`
`
`
`
`
`described are provided in software. Given the system as
`
`
`
`
`
`
`
`
`shown and described according to the invention in the
`
`
`
`
`
`
`following materials,
`software not
`specifically shown,
`
`
`
`
`
`
`
`described or suggested herein that is useful for implemen-
`tation of the invention is conventional and within the ordi-
`
`
`
`
`
`
`
`
`
`
`
`
`nary skill in such arts.
`
`
`
`
`
`
`
`
`
`
`[0025]
`In the image capture device that is the subject of the
`
`
`
`
`
`
`
`
`invention, two or more lens systems are associated with a
`
`
`
`
`
`
`
`
`respective number of image sensors. The lenses have dif-
`
`
`
`
`
`
`
`
`
`
`ferent focal lengths and different fields of view within the
`
`
`
`
`
`
`
`
`
`
`
`same scene wherein the field of view of the longer focal
`
`
`
`
`
`
`
`length lenses contains at least a portion of the field of view
`
`
`
`
`
`
`
`
`
`
`of the shorter focal
`length lens. In addition,
`the image
`
`
`
`
`
`
`
`
`
`captured by the image sensor associated with the longer
`
`
`
`
`
`
`
`
`
`focal length lens has a higher resolution than the image
`
`
`
`
`
`
`
`
`
`captured by the image sensor associated with the lens with
`
`
`
`
`the shorter focal length.
`the image
`[0026]
`In the embodiment of the invention,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`capture done by the two or more image sensors is done
`
`
`
`
`
`
`
`substantially simultaneously so that motion artifacts from
`motion of the camera or motion within the scene, do not
`
`
`
`
`
`
`
`
`
`cause differences in the two or more images that are cap-
`
`
`
`
`
`
`
`
`
`tured. The invention discloses the use of the two or more
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`images to form a composite image that includes portions of
`
`
`
`
`
`
`
`
`each of the two or more images for the purpose of providing
`
`
`
`
`
`
`
`
`a digitally zoomed image with uniformly high resolution.
`[0027] Each of the several embodiments of the present
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`invention include an image capture assembly having mul-
`
`
`
`
`
`
`
`
`tiple lenses and multiple image sensors mounted within a
`
`
`
`
`
`
`
`
`digital camera wherein the multiple lenses have different
`
`
`
`
`
`
`
`
`focal lengths and portions of the fields of view are substan-
`
`
`
`
`
`
`
`
`
`
`tially the same and the multiple image sensors can capture
`
`
`
`
`
`
`images simultaneously. The invention describes an arrange-
`
`
`
`
`
`
`
`ment for producing an image that is formed by combining
`
`
`
`
`
`
`
`
`
`the images from the multiple image sensors in a way that
`
`
`
`
`
`
`provides increased resolution in a digitally zoomed image.
`
`
`
`
`
`
`
`
`[0028]
`In each embodiment, the camera captures images
`
`
`
`
`
`
`
`
`from the multiple image sensors simultaneously. Each mul-
`
`
`
`
`
`
`
`
`
`
`tiple lens system contains at least one fixed focal length lens
`
`
`
`
`
`
`
`
`
`
`or variable focal length lens as in an optical zoom lens.
`Moreover, each embodiment includes some type of user
`
`
`
`
`
`
`
`
`control that allows a user to select a zoom amount, which
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`controls both the digital zoom and the optical zoom lens if
`
`
`
`
`
`
`
`
`present. In some embodiments, a single “zoom lens” user
`control is used. e.g., where the “wide” setting selects a wide
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`angle fixed focal length lens and the “tele” setting(s) select
`
`
`
`
`
`
`
`
`
`various positions of a zoom lens.
`In any case, digital
`
`
`
`
`
`
`
`
`
`zooming is used along with any optical zoom that is present
`
`
`
`
`
`
`
`
`
`to provide a continuous zoom “up” from the image obtained
`with the short focal length lens to the maximum focal length
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`of the multiple lenses. All this, of course, can be transparent
`APPL—1009 / Page 11 of 17
`
`APPL-1009 / Page 11 of 17
`
`
`
`
`
`US 2008/0030592 A1
`
`
`Feb. 7, 2008
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`to the user, who simply manipulates the “zoom” user control
`
`
`
`
`
`
`between the “wide” and “tele” settings.
`
`
`
`
`
`
`
`
`[0029] The composite image can be formed during image
`
`
`
`
`
`
`
`
`processing on the camera or later during post processing
`
`
`
`
`
`
`
`
`when the images have been offloaded from the camera. In
`
`
`
`
`
`
`
`
`
`either case, the two images must be matched to locate the
`
`
`
`
`
`
`high-resolution image accurately into the low-resolution
`
`
`
`
`
`
`
`
`
`
`image and then stitched into place so the edge between the
`
`
`
`
`
`
`
`two images in the composite image is not discernible. To
`
`
`
`
`
`
`
`
`enable the composite image to be formed during post
`
`
`
`
`
`
`
`processing, both images in the image set must be stored at
`
`
`
`
`
`
`
`
`
`the time of image capture. In the case of video, by storing the
`low-resolution Video and the high resolution Video,
`the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`zoom ratio can be selected after image capture and adjusted
`as desired at that time.
`
`
`
`
`
`
`
`
`
`
`
`
`[0030] Turning now to FIG. 1A, a digital camera 10A is
`
`
`
`
`
`
`
`described which includes an image capture assembly,
`
`
`
`
`
`
`
`
`including a fixed focal length lens 2 that focuses an image
`
`
`
`
`
`
`
`
`
`
`of a scene (not shown) onto a first image sensor 12, and a
`
`
`
`
`
`
`
`
`zoom lens 3 which focuses an image of the scene onto a
`
`
`
`
`
`
`
`
`second image sensor 14. The image capture assembly 1
`
`
`
`
`
`
`
`
`
`
`provides a first image output signal 126 from the first image
`
`
`
`
`
`
`
`
`
`sensor 12 and a second image output signal 146 from the
`
`
`
`
`second image sensor 14.
`
`
`
`
`
`
`
`
`[0031] The focal length of the fixed focal length lens 2
`
`
`
`
`
`
`
`
`
`generates a wide-angle field of view and has a fixed focus set
`
`
`
`
`
`
`
`
`to a distance near the lens hyperfocal distance of 8 feet so
`
`
`
`
`
`
`
`
`
`that objects from 4 feet to infinity are in focus. Therefore, the
`
`
`
`
`
`
`
`
`
`fixed focal length lens 2 does not need to include a focus
`
`
`
`
`
`
`
`
`adjustment. The fixed focal length lens 2 includes an adjust-
`
`
`
`
`
`
`
`
`able aperture and shutter assembly to control the exposure of
`the first image sensor 12. The zoom lens 3 includes an
`
`
`
`
`
`
`
`
`
`
`optical zoom and autofocus controlled by zoom and focus
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`motors 5 and an adjustable aperture and shutter assembly to
`
`
`
`
`
`
`control the exposure of the image sensor.
`
`
`
`
`
`
`
`
`[0032]
`In a preferred embodiment, the image sensors 12
`
`
`
`
`
`
`
`
`and 14 are single-chip color Megapixel CCD sensors, using
`
`
`
`
`
`
`
`
`the well-known Bayer color filter pattern to capture color
`
`
`
`
`
`
`
`
`
`images. The image sensors 12 and 14 can have, for example,
`
`
`
`
`
`
`
`
`
`
`a 4:3 image aspect ratio and a total of 3.1 effective mega-
`
`
`
`
`
`
`
`
`pixels (million pixels), with 2048 active columns of pixels><
`1536 active rows of pixels. The image sensors 12 and 14 can
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`use a 1/2" type optical format, so that each pixel is approxi-
`mately 3.1 microns tall by 3.1 microns wide. A control
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`processor and timing generator 40 controls the first image
`
`
`
`
`
`
`
`
`
`
`sensor 12 by supplying signals to clock drivers 13, and
`
`
`
`
`
`
`
`controls the second image sensor 14 by supplying signals to
`clock drivers 15.
`
`
`
`
`
`
`
`
`
`
`
`[0033] The control processor and timing generator 40 also
`controls the zoom and focus motors 5 for zoom lens 3, and
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`a flash 48 for emitting light to illuminate the scene. The
`
`
`
`
`
`
`
`
`control processor and timing generator 40 also receives
`
`
`
`
`
`
`
`
`signals from automatic focus and automatic exposure detec-
`tors 46. In an alternative embodiment, instead of using the
`
`
`
`
`
`
`
`automatic focus and automatic exposure detectors 46, the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`image sensor 14 could be used to provide exposure detection
`
`
`
`
`
`
`
`and “through-the-lens” autofocus, as described in com-
`
`
`
`
`
`
`
`monly-assigned U.S. Pat. No. 5,668.597 entitled “Electronic
`
`
`
`
`
`
`
`Camera with Rapid Automatic Focus of an Image upon a
`
`
`
`
`
`
`
`
`
`Progressive Scan Image Sensor” which issued Sep. 26, 1997
`in the names of Kenneth A. Parulski, Masaki Izumi, Seiichi
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Mizukoshi and Nobuyuki Mori,
`incorporated herein by
`reference. User controls 42 are used to control the operation
`
`
`
`
`
`
`
`
`
`
`
`
`
`of the digital camera 10A.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`[0034] The first image output signal 126 from the first
`
`
`
`
`
`
`image sensor 12 is amplified by a first analog signal pro-
`
`
`
`
`
`cessor (ASP 1) 22 and provided to a first analog-to-digital
`
`
`
`
`
`
`
`
`(A/D) converter 34. The second image output signal 146
`
`
`
`
`
`
`from the second image sensor 14 is amplified by a second
`
`
`
`
`
`
`analog signal processor (ASP 2) 24 and provided to a second
`A/D converter 36.
`
`
`
`
`
`
`
`
`
`
`
`
`[0035] The digital data from the A/D converters 34 and 36
`
`
`
`
`
`
`
`
`is provided to digital multiplexer 37. The digital multiplexer
`
`
`
`
`
`
`
`
`37 is used to select which one of the outputs of the two A/D
`converters 34 and 36 is connected to the DRAM buffer
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`memory 38. The digital data is stored in DRAM buffer
`
`
`
`
`
`
`memory 38 and subsequently processed by an image pro-
`
`
`
`
`
`
`
`cessor 50. The processing performed by the image processor
`
`
`
`
`
`
`50 is controlled by firmware stored in fin-ware memory 58,
`
`
`
`
`
`
`
`
`which can be flash EPROM memory. The image processor
`
`
`
`
`
`
`
`
`
`50 processes the input digital image file, which is buffered
`
`
`
`
`
`
`
`
`in a RAM memory 56 during the processing stage. The
`
`
`
`
`
`
`
`
`
`image processor 50 combines the digital data from the A/D
`
`
`
`
`
`
`
`converters 34 and 36 to form a composite image with areas
`
`
`
`
`
`
`
`
`
`of high resolution and areas of lower resolution using a
`method, which constitutes the invention.
`
`
`
`
`
`
`
`
`
`
`
`
`[0036] As shown in FIG. 5, the image processor 50 of
`
`
`
`
`
`
`
`FIGS. 1A and 1B contains an image compositor 202 that
`
`
`
`
`
`
`
`
`
`
`
`receives both the wide image 204 from the fixed focal length
`
`
`
`
`
`
`
`
`
`
`lens 2 and the telephoto image 206 from the zoom lens 3.
`
`
`
`
`
`
`
`The telephoto image 206 is of a smaller portion of the scene
`
`
`
`
`
`
`
`
`
`
`than the wide image 204, but captures this smaller portion
`with greater resolution than the resolution of the wide image
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`204. The image compositor 202 generates a composite
`
`
`
`
`
`
`
`
`
`
`
`image 208 using image data from both the wide image 204
`
`
`
`
`
`
`
`
`
`and the telephoto image 206. Also, the image compositor
`202 receives a zoom amount 210 that can be adjusted by the
`
`
`
`
`
`
`
`
`camera user as will be described below.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`[0037]
`It is desirable for the image compositor 202 to
`
`
`
`
`
`
`
`
`
`generate a composite image 208 that has the highest possible
`
`
`
`
`
`
`
`
`
`
`quality. For illustration, assume that the wide image 204 and
`
`
`
`
`
`
`
`
`
`the telephoto image 206 have the same number of rows R
`
`
`
`
`
`
`
`
`and columns C of pixels, for example, R:1000 and C:1500
`
`
`
`
`
`
`
`and that the relative magnification ratio M of the telephoto
`
`
`
`
`
`
`
`image 206 to the wide image 204 is M:3.
`
`
`
`
`
`
`
`[0038] The image registration determiner 212 determines
`
`
`
`
`
`
`
`
`
`
`the registration between the wide image 204 and the tele-
`
`
`
`
`
`
`
`photo image 206. The coordinate transformation is simply a
`translation and a scale because the image sensors that
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`capture the wide image 204 and the telephoto image 206 are
`
`
`
`
`
`
`coplanar. A convenient way to represent the registration
`
`
`
`
`
`
`
`
`between the images is to find the mapping of the four comer
`
`
`
`
`
`
`
`
`
`pixels of the telephoto image 206 onto the wide image 204.
`
`
`For example,
`
`
`
`Telephoto Image Coordinates
`
`
`(0, 0)
`
`(999, 0)
`
`(0, 1499)
`
`(999, 1499)
`
`
`
`
`
`
`
`Wide Image Coordinates
`
`
`(333, 499.7)
`
`(666, 499.7)
`
`(333, 999.3)
`(666, 999.3)
`
`
`
`
`
`
`
`The registration can also be stored in the form of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`homography HTW that transforms the coordinates of the
`
`
`
`
`
`
`
`telephoto image 206 to the wide image 204.
`
`
`
`
`APPL—1009 / Page 12 of 17
`
`APPL-1009 / Page 12 of 17
`
`
`
`
`
`US 2008/0030592 A1
`
`
`Feb. 7, 2008
`
`
`
`[0039]
`
`
`xw
`
`xc
`
`XT
`xw
`
`
`yw = 11m yr
`
`
`
`1
`1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Where coordinates of the telephoto image 206 are in (row,
`
`
`
`
`
`
`column) notation (yz, xT) and coordinates of the wide image
`
`
`
`
`
`
`204 are (yW, xW). For example,
`[0040]
`
`
`499.7
`
`HTW =
`
`
`
`
`
`l / M
`0
`
`
`
`O
`O
`
`
`
`0
`
`l/M 333 I
`
`l
`
`
`
`
`
`
`
`
`
`
`
`
`
`[0041] The correspondences between the coordinate sys-
`
`
`
`
`
`
`
`
`
`tems represent the registration between the wide image 204
`
`
`
`
`
`
`
`
`and the telephoto image 206. The correspondences are
`
`
`
`
`
`
`preferably determined at the time of manufacture by shoot-
`
`
`
`
`
`
`
`
`
`
`ing test targets, as is well known in the art. If one or both of
`the lenses were a zoom lens rather than a fixed lens., the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`registration correspondences could still be determined at the
`
`
`
`
`
`time of manufacture as a function of the zoom position of the
`
`
`
`
`
`
`
`
`lenses. It should be further noted that while the example
`shows a pure translate and scale transformation, it may be
`
`
`
`
`
`
`
`necessary to correct for a difference in tilt between the two
`
`
`
`
`
`
`
`
`
`
`imaging systems.
`
`
`
`
`
`
`
`[0042] Alternatively, the registration between images can
`be determined using the image information contained in the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`wide image 204 and telephoto image 204. This is well
`
`
`
`
`
`
`
`
`known in the art of