`Apple Inc. v. Corephotonics
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 1 0f 8
`
`US 2008/0030592 A1
`
`
`
`mm
`
`:95:
`
`RmmE33,Vb_J,8ms5%z1%§\\.as:as:%
`
`mmmeE33mg5%gm$95:$5
`
`35
`
`
`
`
`
`
`
`
`Eggfim,7fiqfiz
`
`main
`AI!ES:
`«958%meEgm
`
`
`_7figfi
`
`fIl1ITI\_lIIIII
`
`133%ii
`
`Emmmwwgmfl
`”33$7%
`
`EOE:
`
`{FEj}!9%mammafinsaga
`
`mmw«aE33%$9822
`
`02%
`
`$58qu2%S
`
`$63qu
`
`APPL-1006 / Page 2 of 17
`
`
`
`“mitt$93$8
`
`Q3:03
`
`APPL-1006 / Page 2 of 17
`
`
`
`
`
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 2 0f 8
`
`US 2008/0030592 A1
`
`
`
`
`
`3m§e§:95:
`
`
`
`«82mmE23x65%:%a}ENa?§§3gmm2\\as:onmNNmmxmma
`Na.Mg”:EEEEQ
`
`:05:956SE§d
`93gag:E§§WEE2w
`
`«8385mugszExt8mgiEN
`
`RESmEE$08:398
`«as2ES::3a”?
`$25%«82mmSE
`
`
`
`
`
`1IIIIIJnlllllJElms
`
`_E8:_émmam}(amugEQEEQ
`
`EsewmmkléEGEEaas«38«2E59#:8qu82Qa2%a:$9823
`is“a:”$ng3$.58
`
`
`$3.3m
`
`:05:ES:mmt§§<t
`
`
`
`m:.UE
`
`Ema
`
`N“.38:28aimIlia.
`
`APPL-1006 / Page 3 of 17
`
`APPL-1006 / Page 3 of 17
`
`
`
`
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 3 0f 8
`
`US 2008/0030592 A1
`
`
`
`FIG. 2B
`
`APPL-1006 / Page 4 of 17
`
`APPL-1006 / Page 4 of 17
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 4 0f 8
`
`US 2008/0030592 A1
`
`i
`
`'PR
`
` 5
`f'c—Eilijiifl
`i MODEM gt.....PM“;
`_________I
`
`
`:
`
`.90
`
`APPL'1006/ Page 5 of 17
`
`APPL-1006 / Page 5 of 17
`
`
`
`Patent Application Publication
`
`b.eF
`
`Wm7.,
`
`a
`
`.m
`
`SU
`
`M%5mm8
`
`m$NSNE
`8,
`5fil/:
`__.___..__—__—_-__-__J
`
`
`TIllllllllllllllllrl
`
`V-
`
`Em
`
`
`
`VNQ..........
`
`mNNmIn
`
`APPL-1006 / Page 6 of 17
`
`APPL-1006 / Page 6 of 17
`
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 6 0f 8
`
`US 2008/0030592 A1
`
`MK\MQDECO
`
`mmYVS
`
`
`
`QMqQE«@wa
`
`m6<3:
`
`m.65%
`
`NON
`
`
`
`20R«mtMGME
`
`QMZREMRMud
`
`MG<3:
`
`OKOIQQHNN
`
`m6{3:
`
`wQN
`
`mow
`
`SN
`
`APPL-1006 / Page 7 of 17
`
`APPL-1006 / Page 7 of 17
`
`
`
`
`
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 7 0f 8
`
`US 2008/0030592 A1
`
`-
`
`204
`
`206
`
`220
`
`/\
`
`208
`
`FIG. 6
`
`APPL-1006 / Page 8 of 17
`
`APPL-1006 / Page 8 of 17
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 8 0f 8
`
`US 2008/0030592 A1
`
`VNN
`
`MR$00309
`
`DMQS
`
`NON
`
`20R3EWGMQ
`
`WMEERGRm5
`
`MO«St
`
`mmm«\VSOMQS
`
`QRQIQMHHR
`
`
`
`QMQQESWINE
`
`mm<3:
`
`.N>303iEOON
`
`k65%
`
`OWN
`
`WNW
`
`EN
`
`APPL-1006 / Page 9 of 17
`
`APPL-1006 / Page 9 of 17
`
`
`
`
`
`
`US 2008/0030592 A1
`
`Feb. 7, 2008
`
`PRODUCING DIGITAL IMAGE WITH
`DIFFERENT RESOLUTION PORTIONS
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`[0001] Reference is made to commonly assigned US.
`Patent Application Serial No. 2002/0075258, filed Nov. 23,
`2001, entitled “Camera System with High Resolution Image
`Inside a Wide Angle View” by Park et al. and US. patent
`application Ser. No. 11/062,174, filed Feb. 18, 2005, entitled
`“Digital Camera Using Multiple Lenses And Image Sensors
`To Provide An Extended Zoom Range” by Peter Laba-
`ziewicz, et al., the disclosures of which are incorporated
`herein.
`
`FIELD OF THE INVENTION
`
`[0002] The present invention relates to a digital camera
`that uses multiple lenses and image sensors to provide an
`extended zoom range and the method used to produce a
`digital image that combines the multiple images produced
`by the digital camera.
`
`BACKGROUND OF THE INVENTION
`
`[0003] Currently, most digital cameras use a zoom lens
`and a single color image sensor to capture still and motion
`images. The captured images are then digitally processed to
`produce digital image files, which are stored in a digital
`memory in the camera. The digital image files can then be
`transferred to a computer, displayed, and shared via the
`Internet. The digital camera can be included as part of a
`mobile telephone, to form a so-called “camera phone.” The
`camera phone can transmit the digital image files to another
`camera phone, or to service providers, via a mobile tele-
`phone network.
`[0004]
`Small camera size and a large zoom range are two
`very important features of digital cameras. Users prefer to
`have a large zoom range (e.g. 5:1 or greater) rather than a
`limited zoom range (e.g. 3:1 or smaller). The zoom range is
`typically composed of both optical zoom which is provided
`by variable focal length lenses and digital zoom which is
`provided by a magnification of the digital
`image after
`capture. Variable focal length lenses for large zoom range
`are expensive and they increase the size of the digital
`camera. Thus, there are trade-offs between small camera
`size, large zoom range, and low camera cost which must be
`made when designing a digital camera. With higher cost
`cameras, such as single lens reflex cameras, these problems
`are sometimes addressed by using multiple interchangeable
`zoom lenses, such as two 3:1 zoom lenses, e.g., a 28-70 mm
`zoom and a 70-210 zoom. This arrangement has user incon-
`venience problems and is presently not available for low
`cost digital cameras.
`[0005] A different solution that has been offered by Kodak
`in the V570 and the V610 cameras is to include two different
`lens assemblies in the camera with two different focal
`
`lengths and two separate image sensors. In this case, each of
`the lens assemblies can be either a fixed focal length lens or
`can have a moderate optical zoom range to reduce the size
`and cost of each of the lens assemblies. Together, the two
`lens assemblies provide a wide zoom range and a small
`overall size at a lower cost. However, a problem arises when
`the focal length of the first lens does not match the focal
`length of the second lens so that the optical zoom is not
`
`continuous over the entire zoom range. In this case, digital
`zoom must be used for zoom between the maximum zoom
`of the first lens and the minimum zoom of the second lens.
`
`[0006] Digital zoom based on increased magnification of
`the image with a corresponding decrease in resolution is
`well known in the art. Although digital zoom is very fast and
`simple, the decrease in resolution can produce a perceived
`decrease in image quality.
`[0007]
`In US. Pat. No. 5,657,402, a method is described
`in which a plurality of digital images are combined to form
`an image. US. Pat. No. 5,657,402 addresses the use of
`multiple images captured at different times wherein “the
`plurality of images of various focal lengths, such as a zoom
`video sequence” (col. 1, lines, 21-22) are captured from the
`same lens. US. Pat. No. 5,657,402 does not address two lens
`assemblies simultaneously capturing images of the same
`scene.
`
`In US Publication No. 2002/0075258, a panoramic
`[0008]
`camera system is described in which a moveable telephoto
`camera is additionally used to capture a high-resolution
`portion of the scene which is then overlaid onto the pan-
`oramic image. US Publication No. 2002/0075258 describes
`the use of a moveable telephoto camera to enable a higher
`resolution of a portion of the image wherein the moveable
`telephoto camera can be moved to the region of the pan-
`oramic image where the higher resolution is desired. US
`Publication No. 2002/0075258 does not address the case
`
`wherein a wide-angle camera and a telephoto camera are
`affixed together for simultaneous capture of the same scene.
`In addition, US Publication No. 2002/0075258 does not
`disclose the use of a composite image for improved image
`quality in a digital zoom system.
`
`SUMMARY OF THE INVENTION
`
`[0009] The present invention provides a sufficiently com-
`pact, low cost, optical system with a large zoom range for a
`small, lightweight and relatively inexpensive consumer digi-
`tal camera.
`
`[0010] What is therefore needed is a digital camera that
`provides a rapidly-operating extended zoom range without
`unduly increasing the size or cost of the digital camera while
`providing good perceived image quality throughout
`the
`zoom range.
`[0011] An object of the invention is to provide a method
`of producing a digital image having portions with different
`resolutions comprising:
`[0012]
`a. simultaneously capturing first and second digital
`images of the same scene wherein the first digital image is
`of a larger portion of the scene than the second digital image
`wherein the second digital image has a higher resolution
`than the resolution in the first digital image corresponding to
`the second digital image; and
`[0013]
`b. combining at least a portion of the second digital
`image into the corresponding portion of the first digital
`image to thereby provide a digital image having portions
`with different resolutions.
`
`[0014] The present invention is directed to overcoming the
`problems set forth above. Briefly summarized, the invention
`includes an electronic camera for producing an image of a
`scene, wherein the camera includes a first image sensor for
`generating a first sensor output, a first lens with a first focal
`length for forming a first image of the scene on the first
`image sensor, a second image sensor for generating a second
`sensor output, and a second lens with a second focal length
`APPL—1006 / Page 10 of 17
`
`APPL-1006 / Page 10 of 17
`
`
`
`US 2008/0030592 A1
`
`Feb. 7, 2008
`
`length of the first lens for
`that is longer than the focal
`forming a second image of the same scene on the second
`image sensor. The first lens or the second lens can be either
`fixed focal length lenses or multiple focal length lenses as in
`a zoom lens wherein, the first and second lenses are directed
`at substantially the same scene and image sets are captured
`substantially simultaneously by the first image sensor and
`the second image sensor. Portions of the image set captured
`by the first image sensor and the second image sensor are
`then combined to produce a composite image with a higher
`resolution in the portion of the composite image that is
`provided by the second image sensor due to the longer focal
`length of the second lens. Subsequent
`images produced
`during a digital zooming process are composed largely of the
`lower resolution image captured by the first image sensor at
`low digital zoom values and largely of the higher resolution
`image as captured by the second image sensor at high digital
`zoom values.
`
`[0015] By forming a composite image with portions of the
`image from the short focal length lens and portions of the
`image from the longer focal length lens, perceived image
`quality is improved throughout the zoom range while lens
`complexity is reduced, since a continuous zoom ratio can be
`produced with unmatched lens focal lengths. By capturing
`images from the two image sensors substantially simulta-
`neously, complexities in the image processing are reduced
`since differences between the two images due to motion of
`the camera or motion within the scene are avoided. It is an
`
`additional advantage, that the present invention can avoid
`the slow response that is typical of an optical zoom system
`when traversing a large zoom range.
`features and
`[0016] These and other aspects, objects,
`advantages of the present invention will be more clearly
`understood and appreciated from a review of the following
`detailed description of the preferred embodiments and
`appended claims, and by reference to the accompanying
`drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIGS. 1A and 1B depict a block diagram of a
`[0017]
`digital camera using a fixed focal length wide-angle lens
`with a first image sensor and a zoom lens, or a longer second
`fixed focal length lens, with a second image sensor accord-
`ing to the present invention;
`[0018]
`FIGS. 2A and 2B show front and rear perspective
`views of the digital camera;
`[0019]
`FIGS. 3A and 3B are perspective views of the front
`and back of a cell phone including a camera with multiple
`lenses and multiple sensors according to the present inven-
`tion;
`FIGS. 4A and 4B show two views of the capture
`[0020]
`assembly used in the cell phone shown in FIGS. 3A and 3B
`[0021]
`FIG. 5 is a block diagram of the stitching process
`to create the composite image;
`[0022]
`FIG. 6 depicts a wide angle image as captured, a
`telephoto image as captured, and a composite image as
`created by the invention; and
`
`FIG. 7 is a block diagram of the stitching process
`[0023]
`with video images to create a composite video.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`employing imaging
`cameras
`[0024] Because digital
`devices and related circuitry for signal capture, correction,
`and exposure control are well known, the present description
`will be directed in particular to elements forming part of, or
`cooperating more directly with, method and apparatus in
`accordance with the present invention. Elements not spe-
`cifically shown or described herein are selected from those
`known in the art. Certain aspects of the embodiments to be
`described are provided in software. Given the system as
`shown and described according to the invention in the
`following materials,
`software not
`specifically shown,
`described or suggested herein that is useful for implemen-
`tation of the invention is conventional and within the ordi-
`
`nary skill in such arts.
`[0025]
`In the image capture device that is the subject of the
`invention, two or more lens systems are associated with a
`respective number of image sensors. The lenses have dif-
`ferent focal lengths and different fields of view within the
`same scene wherein the field of view of the longer focal
`length lenses contains at least a portion of the field of view
`of the shorter focal
`length lens. In addition,
`the image
`captured by the image sensor associated with the longer
`focal length lens has a higher resolution than the image
`captured by the image sensor associated with the lens with
`the shorter focal length.
`the image
`[0026]
`In the embodiment of the invention,
`capture done by the two or more image sensors is done
`substantially simultaneously so that motion artifacts from
`motion of the camera or motion within the scene, do not
`cause differences in the two or more images that are cap-
`tured. The invention discloses the use of the two or more
`
`images to form a composite image that includes portions of
`each of the two or more images for the purpose of providing
`a digitally zoomed image with uniformly high resolution.
`[0027] Each of the several embodiments of the present
`invention include an image capture assembly having mul-
`tiple lenses and multiple image sensors mounted within a
`digital camera wherein the multiple lenses have different
`focal lengths and portions of the fields of view are substan-
`tially the same and the multiple image sensors can capture
`images simultaneously. The invention describes an arrange-
`ment for producing an image that is formed by combining
`the images from the multiple image sensors in a way that
`provides increased resolution in a digitally zoomed image.
`[0028]
`In each embodiment, the camera captures images
`from the multiple image sensors simultaneously. Each mul-
`tiple lens system contains at least one fixed focal length lens
`or variable focal length lens as in an optical zoom lens.
`Moreover, each embodiment includes some type of user
`control that allows a user to select a zoom amount, which
`controls both the digital zoom and the optical zoom lens if
`present. In some embodiments, a single “zoom lens” user
`control is used. e.g., where the “wide” setting selects a wide
`angle fixed focal length lens and the “tele” setting(s) select
`various positions of a zoom lens.
`In any case, digital
`zooming is used along with any optical zoom that is present
`to provide a continuous zoom “up” from the image obtained
`with the short focal length lens to the maximum focal length
`of the multiple lenses. All this, of course, can be transparent
`APPL—1006 / Page 11 of 17
`
`APPL-1006 / Page 11 of 17
`
`
`
`US 2008/0030592 A1
`
`Feb. 7, 2008
`
`to the user, who simply manipulates the “zoom” user control
`between the “wide” and “tele” settings.
`[0029] The composite image can be formed during image
`processing on the camera or later during post processing
`when the images have been offloaded from the camera. In
`either case, the two images must be matched to locate the
`high-resolution image accurately into the low-resolution
`image and then stitched into place so the edge between the
`two images in the composite image is not discernible. To
`enable the composite image to be formed during post
`processing, both images in the image set must be stored at
`the time of image capture. In the case of video, by storing the
`low-resolution Video and the high resolution Video,
`the
`zoom ratio can be selected after image capture and adjusted
`as desired at that time.
`
`[0030] Turning now to FIG. 1A, a digital camera 10A is
`described which includes an image capture assembly,
`including a fixed focal length lens 2 that focuses an image
`of a scene (not shown) onto a first image sensor 12, and a
`zoom lens 3 which focuses an image of the scene onto a
`second image sensor 14. The image capture assembly 1
`provides a first image output signal 126 from the first image
`sensor 12 and a second image output signal 146 from the
`second image sensor 14.
`[0031] The focal length of the fixed focal length lens 2
`generates a wide-angle field of view and has a fixed focus set
`to a distance near the lens hyperfocal distance of 8 feet so
`that objects from 4 feet to infinity are in focus. Therefore, the
`fixed focal length lens 2 does not need to include a focus
`adjustment. The fixed focal length lens 2 includes an adjust-
`able aperture and shutter assembly to control the exposure of
`the first image sensor 12. The zoom lens 3 includes an
`optical zoom and autofocus controlled by zoom and focus
`motors 5 and an adjustable aperture and shutter assembly to
`control the exposure of the image sensor.
`[0032]
`In a preferred embodiment, the image sensors 12
`and 14 are single-chip color Megapixel CCD sensors, using
`the well-known Bayer color filter pattern to capture color
`images. The image sensors 12 and 14 can have, for example,
`a 4:3 image aspect ratio and a total of 3.1 effective mega-
`pixels (million pixels), with 2048 active columns of pixels><
`1536 active rows of pixels. The image sensors 12 and 14 can
`use a 1/2" type optical format, so that each pixel is approxi-
`mately 3.1 microns tall by 3.1 microns wide. A control
`processor and timing generator 40 controls the first image
`sensor 12 by supplying signals to clock drivers 13, and
`controls the second image sensor 14 by supplying signals to
`clock drivers 15.
`
`[0033] The control processor and timing generator 40 also
`controls the zoom and focus motors 5 for zoom lens 3, and
`a flash 48 for emitting light to illuminate the scene. The
`control processor and timing generator 40 also receives
`signals from automatic focus and automatic exposure detec-
`tors 46. In an alternative embodiment, instead of using the
`automatic focus and automatic exposure detectors 46, the
`image sensor 14 could be used to provide exposure detection
`and “through-the-lens” autofocus, as described in com-
`monly-assigned U.S. Pat. No. 5,668.597 entitled “Electronic
`Camera with Rapid Automatic Focus of an Image upon a
`Progressive Scan Image Sensor” which issued Sep. 26, 1997
`in the names of Kenneth A. Parulski, Masaki Izumi, Seiichi
`Mizukoshi and Nobuyuki Mori,
`incorporated herein by
`reference. User controls 42 are used to control the operation
`of the digital camera 10A.
`
`[0034] The first image output signal 126 from the first
`image sensor 12 is amplified by a first analog signal pro-
`cessor (ASP 1) 22 and provided to a first analog-to-digital
`(A/D) converter 34. The second image output signal 146
`from the second image sensor 14 is amplified by a second
`analog signal processor (ASP 2) 24 and provided to a second
`A/D converter 36.
`
`[0035] The digital data from the A/D converters 34 and 36
`is provided to digital multiplexer 37. The digital multiplexer
`37 is used to select which one of the outputs of the two A/D
`converters 34 and 36 is connected to the DRAM buffer
`
`memory 38. The digital data is stored in DRAM buffer
`memory 38 and subsequently processed by an image pro-
`cessor 50. The processing performed by the image processor
`50 is controlled by firmware stored in fin-ware memory 58,
`which can be flash EPROM memory. The image processor
`50 processes the input digital image file, which is buffered
`in a RAM memory 56 during the processing stage. The
`image processor 50 combines the digital data from the A/D
`converters 34 and 36 to form a composite image with areas
`of high resolution and areas of lower resolution using a
`method, which constitutes the invention.
`[0036] As shown in FIG. 5, the image processor 50 of
`FIGS. 1A and 1B contains an image compositor 202 that
`receives both the wide image 204 from the fixed focal length
`lens 2 and the telephoto image 206 from the zoom lens 3.
`The telephoto image 206 is of a smaller portion of the scene
`than the wide image 204, but captures this smaller portion
`with greater resolution than the resolution of the wide image
`204. The image compositor 202 generates a composite
`image 208 using image data from both the wide image 204
`and the telephoto image 206. Also, the image compositor
`202 receives a zoom amount 210 that can be adjusted by the
`camera user as will be described below.
`
`It is desirable for the image compositor 202 to
`[0037]
`generate a composite image 208 that has the highest possible
`quality. For illustration, assume that the wide image 204 and
`the telephoto image 206 have the same number of rows R
`and columns C of pixels, for example, R:1000 and C:1500
`and that the relative magnification ratio M of the telephoto
`image 206 to the wide image 204 is M:3.
`[0038] The image registration determiner 212 determines
`the registration between the wide image 204 and the tele-
`photo image 206. The coordinate transformation is simply a
`translation and a scale because the image sensors that
`capture the wide image 204 and the telephoto image 206 are
`coplanar. A convenient way to represent the registration
`between the images is to find the mapping of the four comer
`pixels of the telephoto image 206 onto the wide image 204.
`For example,
`
`Telephoto Image Coordinates
`
`Wide Image Coordinates
`
`(0, 0)
`(999, 0)
`(0, 1499)
`(999, 1499)
`
`(333, 499.7)
`(666, 499.7)
`(333, 999.3)
`(666, 999.3)
`
`The registration can also be stored in the form of the
`homography HTW that transforms the coordinates of the
`telephoto image 206 to the wide image 204.
`APPL—1006 / Page 12 of 17
`
`APPL-1006 / Page 12 of 17
`
`
`
`US 2008/0030592 A1
`
`Feb. 7, 2008
`
`[0039]
`
`xw
`
`xc
`
`XT
`xw
`yw = 11m yr
`1
`1
`
`
`
`
`
`Where coordinates of the telephoto image 206 are in (row,
`column) notation (yz, xT) and coordinates of the wide image
`204 are (yW, xW). For example,
`
`[0040]
`
`l / M
`
`0
`
`499.7
`
`HTW =
`
`
`
`0
`
`O
`
`l/M 333 I
`
`O
`
`l
`
`[0041] The correspondences between the coordinate sys-
`tems represent the registration between the wide image 204
`and the telephoto image 206. The correspondences are
`preferably determined at the time of manufacture by shoot-
`ing test targets, as is well known in the art. If one or both of
`the lenses were a zoom lens rather than a fixed lens., the
`registration correspondences could still be determined at the
`time of manufacture as a function of the zoom position of the
`lenses. It should be further noted that while the example
`shows a pure translate and scale transformation, it may be
`necessary to correct for a difference in tilt between the two
`imaging systems.
`[0042] Alternatively, the registration between images can
`be determined using the image information contained in the
`wide image 204 and telephoto image 204. This is well
`known in the art of image processing (for example, image
`registration is described in US. Pat. No. 6,078,701) and
`generally includes the steps of finding interest points in each
`image, making guesses at corresponding points (i.e. a scene
`feature that appears in both images), determining an initial
`guess at the registration, using that initial guess to refine the
`correspondence point guess, and so on based on comparing
`pixel values or contrast in the two images.
`[0043] The image resampler 214 uses the registration
`information and the zoom amount 210 to produce the
`composite image 208. Preferably, the composite image has
`the same number of rows and columns of pixels as the wide
`image 204 and the telephoto image 206. However, it is well
`known to those skilled in the art that modifying the number
`of rows and columns of pixels (interpolating the image) can
`easily be done so that the image contains the desired number
`of pixels.
`[0044] The zoom amount 210 Z specifies the desired
`relative zoom amount of the produced composite image 208.
`Preferably, when the value of Z:l, then the composite image
`is the wide image 204. On the other hand, when Z-M, then
`the composite image 208 is the telephoto image 206. When
`the zoom amount is between 1 and M, data from both the
`wide image 204 and the telephoto image 206 are used by the
`image resampler 214 to produce the composite image 208.
`[0045] The image resampler 214 applies the zoom amount
`Z as follows: Each pixel position (yc, xc) of the composite
`image 208 is mapped to the coordinates of wide image 204
`according to:
`
`
`
`
`
`l
`
` ‘yw =ch yc]
`
`l
`
`where
`
`l/Z O
`
`(l—M)(C—l)
`
`
`
`ch= O
`
`O
`
`l/Z (l—M)(C—l)]
`
`O
`
`l
`
`In a similar manner, the position (yc, xc) of the composite
`image 208 is mapped to the coordinates of the telephoto
`image 206 using the equation:
`
`XT
`yT
`1
`
`
`
`= (HTWVIHCW
`
`XC
`
`
`
`yc]l
`
`Then, the pixel value of the composite image at position (yc,
`x) is found by interpolation. If the mapped position of (yc,
`xc) in the telephoto image 206 lands within the limits of the
`existing pixels (i.e. 0<flT<:C-l), the pixel value of the
`composite image 208 at position (yc, xc) is found by inter-
`polating pixel values of the telephoto image 206. Otherwise,
`the pixel value of the composite image 208 at position (yc,
`xc) is found by interpolating pixel values of the wide image
`204.
`
`[0046] Those skilled in the art will recognize that the
`above description of producing the values of the composite
`image 208 using pixel values of the wide image 204 and the
`telephoto image 206 can be accomplished in many ways. For
`example,
`it
`is easy to pre-calculate the region of pixel
`locations of the composite image 208 for which the pixel
`values will be produced by interpolating the telephoto image
`206 and the region for which the pixel values will be
`produced by interpolating the wide image 204. This saves
`computational cost but produces the same image data.
`[0047]
`FIG. 6 shows an example set of images. The wide
`image 204 covers a wide portion of the scene and the
`telephoto image 206 covers a smaller portion of the scene,
`but with greater resolution. The produced composite image
`208 uses pixel data from the telephoto image 206 for those
`portions (i.e. the region within the dashed line 220) that are
`in the view of the telephoto image 206 and uses pixel data
`from the wide image 204 otherwise (i.e. the region outside
`the dashed line 220). The dashed line 220 shows where the
`transition is. Thus,
`the composite image 208 has higher
`resolution in the interior and lower resolution on the edges.
`Since the subject of a photograph, especially in consumer
`photography, is likely to be near the center of the scene, the
`subject of the composite image 208 is likely to have the
`highest resolution. It has also been experimentally deter-
`mined that the transition within the composite image 208
`between pixels derived by interpolating the wide image 204
`versus the telephoto image 206 does not product visually
`objectionable artifacts.
`[0048]
`Since lenses 2 and 3 are separated by some dis-
`tance, it is possible that objects very close to the camera will
`appear to have a discontinuity at the transition. In this case,
`it is possible to use standard image processing techniques to
`
`APPL—1006 / Page 13 of 17
`
`APPL-1006 / Page 13 of 17
`
`
`
`US 2008/0030592 A1
`
`Feb. 7, 2008
`
`find objects that are close to the camera and to process these
`regions in a fashion that does not produce a discontinuity
`artifact. For example,
`the pixel values of the composite
`image 208, for objects that are close to the camera and span
`the transition region, can be determined by interpolating the
`wide image 204. A true depth map can also be created and
`used by the image resampler 214 to sample the appropriate
`locations within the telephoto image 206 and the wide image
`204. In this case, the registration model is no longer a simple
`scale translation model.
`
`[0049] A further feature of the present invention is that the
`composite image 208 can be stored on the camera without
`digital zooming. Therefore, digital zooming of the compos-
`ite image 208 can be (lone later, during post processing, to
`create an image for use by the operator for printing or
`sharing. The composite image 208 can be formed during
`image processing on the camera or later during post pro-
`cessing when the images have been offloaded from the
`camera. To enable the composite image to be formed during
`post processing, the wide image 204 and the telephoto image
`206 must both be stored at the time of image capture.
`[0050] The invention can also be applied to a series of
`sequential images as in a video. Referring to FIG. 7, in the
`case of video, two sets of video images, wide video images
`220 and telephoto video images 222 are captured substan-
`tially simultaneously from the two lenses 2 and 3 or 2 and
`4 and the two image sensors 12 and 14 providing video
`images from a short focal length lens 2 and a zoom lens 3
`or a longer second focal length lens 4. The composite video
`224 is formed by combining the two sets of video images
`220 and 222. The composite video 224 can be formed during
`image processing on the camera and stored on the camera or
`the composite video 224 can be formed later during post
`processing when the images have been ofiloaded from the
`camera. To enable the composite video 224 to be formed
`during post processing, the wide video images 220 from the
`short focal length lens 2 and the telephoto video images 222
`from the zoom lens 3 or the longer focal length lens 4 must
`both be stored at the time of image capture. Digital zoom of
`the video images can be accomplished on the camera during
`capture, or on the camera after capture, or during post
`processing after the composite video 224 has been ofiloaded
`from the camera or during post processing when the com-
`posite video 224 is being formed.
`[0051] The processed digital image file is provided to a
`memory card interface 52, which stores the digital image file
`on the removable memory card 54. Removable memory
`cards 54 are one type of removable digital image storage
`medium, and are available in several different physical
`formats. For example, the removable memory card 54 can
`include (without limitation) memory cards adapted to well-
`known formats, such as the Compact Flash, SmartMedia.
`MemoryStick, MMC, SD, or XD memory card formats.
`Other types of removable digital image storage media, such
`as magnetic hard drives, magnetic tape, or optical disks, can
`alternatively be used to store the still and motion digital
`images. Alternatively, the digital camera 10A can use inter-
`nal non-volatile memory (not shown), such as internal flash
`EPROM memory to store the processed digital image files.
`In such an embodiment, the memory card interface 52 and
`the removable memory card 54 are not needed.
`[0052] The image processor 50 performs various image
`processing functions, including color interpolation followed
`by color and tone correction, in order to produce rendered
`
`sRGB image data. The rendered sRGB image data is then
`JPEG compressed and stored as a JPEG image file on the
`removable memory card 54. The rendered sRGB image data
`can also be provided to a host PC 66 via a host interface 62
`communicating over a suitable interconnection, such as a
`SCSI connection, a USB connection or a Firewire connec-
`tion. The JPEG file uses the so-called “Exif” image format
`defined in “Digital Still Camera Image File Format (Exit)”
`version 2.1, July 1998 by the Japan Electronics Industries
`Development Association (JEIDA), Tokyo, Japan. This for-
`mat includes an Exif application segment that stores par-
`ticular image metadata, including the date or time the image
`was captured, as well as the lens f/number and other camera
`settings.
`It should be noted that the image processor 50,
`[0053]
`although typically a programmable image processor, can
`alternatively be a hard-wired custom integrated circuit (IC)
`processor, a general purpose microprocessor, or a combina-
`tion of hard-wired custom IC and programmable processors.
`[0054] The image processor 50 also creates a low-resolu-
`tion “thumbnail” size image, which can be created as
`described in commonly-assigned US. Pat. No. 5,164,831,
`entitled “Electronic Still Camera Providing Multi-Format
`Storage Of Full And Reduced Resolution Images” issued in
`the name of Kuchta, et al., the disclosure of which is herein
`incorporated by reference. After images are captured, they
`can be quickly reviewed on a color LCD image display 70
`by using the thumbnail
`image data. The graphical user
`interface displayed on the color LCD image display 70