APPL-1006 / Page 1 of 17
`Apple Inc. v. Corephotonics
`
`

`

`Patent Application Publication
`
`Feb. 7, 2008 Sheet 1 of 8
`
`US 2008/0030592 Al
`
`
`
`JIBVAOWIYAYONIN
`
`
`
`96
`
`AMONG
`
`
`
`
`
`C5X/yol”SOW99_1SOH[7OS521zZ)
`
`
`
`SuOLDAIG!|avInTiz9c-7gavnTi39~~—|a7yoT09OLNY~9F[ISMAQNYWOVEENGSFUNSOAKI
`AVIdSIO|WIGON|"4OSSI00Ud|<yJONLay
`
`er—GNY40SS300¥d;TOMINOD
`SNO0s0LNY
`»SNI76ggYOSNISong]:-.1¥D04°dJOVANIgSJOY481
`auvaJOVIAAINIdALdIANODfASOWIM
`
`
`au¥)Vvpug
`JAWSrRd
`
`AYONINXW|pveWYTWLIDIGVvYOSNIS
`YOSSIOONd|éJOVIONC96OWI
`
`
`
`
`
`nvag(Oe
`
`7~—_STOULNOD
`
`aan
`
`APPL-1006 / Page 2 of 17
`
`APPL-1006 / Page 2 of 17
`
`
`
`
`
`
`

`

`Patent Application Publication
`
`Feb. 7, 2008 Sheet 2 of 8
`
`US 2008/0030592 Al
`
`#5és
`
`JTIGVAONFY
`
`AYOWIN
`
`ddd
`
`95
`
`AYOWIN
`
`adv
`
`JOVIGAUNI
`
`JOVANI
`
`
`
`aorLSOH
`0SazZzgieezl
`
`aaxiUALYIANODYOSNISyour§/viIJOWM!3S]
`
`
`
`
`
`S~—F6
`
`8
`
`Wd
`
`AYONIN
`
`JUVANGLS
`
`AYOWNIN
`
`AVIdSid
`
` yeon|___faosssooud!
`amea|aYINTED
`
`9740109FINSOdKIOLN~—9¢ONONYSND0IOLNY
`JOMOLSHOLDALIO
`
`dq]Old
`
`STOMLNOOorMSN
`
`APPL-1006 / Page 3 of 17
`
`JOVMANZ
`
`
`
`SNIT
`
`
`
`4OSSI00UdJOYA!ONZGIXld
`
`
`
`
`
`WLIOYOSNIS1¥904
`
`¥9070
`
`
`
`WHGISNIT
`
`9
`
`YUYIANODsung)¥yUvpug
`yom
`
`GNYsOSSII0d+g
`TOHLNOD
`
`cudSHOLOWSNDOI
`
`HSV|~—8r
`HIONTT
`
`APPL-1006 / Page 3 of 17
`
`
`
`
`
`
`
`
`
`

`

`Patent Application Publication
`
`Feb. 7, 2008 Sheet 3 of 8
`
`US 2008/0030592 Al
`
`
`
`
`FIG. 2A
`
`FIG. 2B
`
`APPL-1006 / Page 4 of 17
`
`APPL-1006 / Page 4 of 17
`
`

`

`Patent Application Publication
`
`Feb. 7, 2008 Sheet 4 of 8
`
`US 2008/0030592 Al
`
`{
`1
`1
`!
`i
`i
`
`f
`\
`
`\
`4
`}
`
`f
`t
`4
`
`:
`1
`pot
`jot
`'{
`See”
`
` 1
`t|e
`
`
`
`|_MODEM tar
`
`
`
`
`
`n nee
`| ae
`(CELLULAR iPROCESSOR!
`
`[CELLULAR|
`PROCESSOR |
`
`APPL-1006 / Page 5 of 17
`
`APPL-1006 / Page 5 of 17
`
`

`

`Patent Application Publication
`
`Feb. 7, 2008 Sheet 5 of 8
`
`US 2008/0030592 Al
`
`abe
`
`h,
`
`JOU UO000
`
`019 ee
`
`APPL-1006 / Page 6 of 17
`
`APPL-1006 / Page 6 of 17
`
`

`

`Patent Application Publication
`
`Feb. 7, 2008 Sheet 6 of 8
`
`US 2008/0030592 Al
`
`807
`
`ALISODWOD
`
`JFOVWI
`
`COC
`
`YANINATLIGNOMVALSIOFY
`
`JOVWI
`
`OLOHdITAL
`
`JOVI
`
`YITIAWVSFo
`
`JOVI
`
`LNNOWY
`
`NOOZ
`
`90E
`
`OLE
`
`JOVI
`
`VOC
`
`30M
`
`GOld
`
`APPL-1006 / Page 7 of 17
`
`APPL-1006 / Page 7 of 17
`
`
`
`
`
`

`

`Patent Application Publication
`
`Feb. 7, 2008 Sheet 7 of 8
`
`US 2008/0030592 Al
`
`204
`
`206
`
`220
`
`ZN
`
`208
`
`FIG. 6
`
`APPL-1006 / Page 8 of 17
`
`APPL-1006 / Page 8 of 17
`
`

`

`Patent Application Publication
`
`Feb. 7, 2008 Sheet 8 of 8
`
`US 2008/0030592 Al
`
`voo
`
`ALISOdWOD
`
`OFCIA
`
`COC
`
`NOUVoLSIOFY
`
`AdININGALIG
`
`JOVW!
`
`
`
`SADVWIOFGIA
`
`OLOHd7TAL
`
`ddIdGWVSFY
`
`JOVI
`
`INNOWV
`
`WOOZ
`
`Coo
`
`Ole
`
`ZLOd
`
`APPL-1006 / Page 9 of 17
`
`APPL-1006 / Page 9 of 17
`
`
`
`

`

`US 2008/0030592 Al
`
`Feb. 7, 2008
`
`PRODUCING DIGITAL IMAGE WITH
`DIFFERENT RESOLUTION PORTIONS
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`[0001] Reference is made to commonly assigned U.S.
`Patent Application Serial No. 2002/0075258, filed Nov. 23,
`2001, entitled “Camera System with High Resolution Image
`Inside a Wide Angle View” by Park et al. and US. patent
`application Ser. No. 11/062,174, filed Feb. 18, 2005, entitled
`“Digital Camera Using Multiple Lenses And Image Sensors
`To Provide An Extended Zoom Range” by Peter Laba-
`ziewicz, et al., the disclosures of which are incorporated
`herein.
`
`FIELD OF THE INVENTION
`
`[0002] The present invention relates to a digital camera
`that uses multiple lenses and image sensors to provide an
`extended zoom range and the method used to produce a
`digital image that combines the multiple images produced
`by the digital camera.
`
`BACKGROUND OF THE INVENTION
`
`[0003] Currently, most digital cameras use a zoom lens
`and a single color image sensor to capture still and motion
`images. The captured images are then digitally processed to
`produce digital image files, which are stored in a digital
`memory in the camera. The digital imagefiles can then be
`transferred to a computer, displayed, and shared via the
`Internet. The digital camera can be included as part of a
`mobile telephone, to form a so-called “camera phone.” The
`camera phonecan transmit the digital imagefiles to another
`camera phone, or to service providers, via a mobile tele-
`phone network.
`[0004]
`Small camera size and a large zoom range are two
`very important features of digital cameras. Users prefer to
`have a large zoom range (e.g. 5:1 or greater) rather than a
`limited zoom range(e.g. 3:1 or smaller). The zoom range is
`typically composed of both optical zoom which is provided
`by variable focal length lenses and digital zoom which is
`provided by a magnification of the digital
`image after
`capture. Variable focal length lenses for large zoom range
`are expensive and they increase the size of the digital
`camera. Thus, there are trade-offs between small camera
`size, large zoom range, and low camera cost which must be
`made when designing a digital camera. With higher cost
`cameras, such as single lens reflex cameras, these problems
`are sometimes addressed by using multiple interchangeable
`zoom lenses, such as two 3:1 zoom lenses, e.g., a 28-70 mm
`zoom and a 70-210 zoom. This arrangementhas user incon-
`venience problems and is presently not available for low
`cost digital cameras.
`[0005] A different solution that has been offered by Kodak
`in the V570 and the V610 camerasis to include two different
`lens assemblies in the camera with two different focal
`
`lengths and two separate image sensors. In this case, each of
`the lens assemblies can be either a fixed focal length lens or
`can have a moderate optical zoom range to reduce the size
`and cost of each of the lens assemblies. Together, the two
`lens assemblies provide a wide zoom range and a small
`overall size at a lower cost. However, a problem arises when
`the focal length of the first lens does not match the focal
`length of the second lens so that the optical zoom is not
`
`continuous over the entire zoom range. In this case, digital
`zoom must be used for zoom between the maximum zoom
`of the first lens and the minimum zoom ofthe secondlens.
`[0006] Digital zoom based on increased magnification of
`the image with a corresponding decrease in resolution is
`well knowninthe art. Although digital zoom is very fast and
`simple, the decrease in resolution can produce a perceived
`decrease in image quality.
`[0007]
`In U.S. Pat. No. 5,657,402, a method is described
`in which a plurality of digital images are combined to form
`an image. U.S. Pat. No. 5,657,402 addresses the use of
`multiple images captured at different times wherein “the
`plurality of imagesof various focal lengths, such as a zoom
`video sequence”(col. 1, lines, 21-22) are captured from the
`same lens. U.S. Pat. No. 5,657,402 does not address two lens
`assemblies simultaneously capturing images of the same
`scene.
`
`In US Publication No. 2002/0075258, a panoramic
`[0008]
`camera system is described in which a moveable telephoto
`camera is additionally used to capture a high-resolution
`portion of the scene which is then overlaid onto the pan-
`oramic image. US Publication No. 2002/0075258 describes
`the use of a moveable telephoto camera to enable a higher
`resolution of a portion of the image wherein the moveable
`telephoto camera can be moved to the region of the pan-
`oramic image where the higher resolution is desired. US
`Publication No. 2002/0075258 does not address the case
`wherein a wide-angle camera and a telephoto camera are
`affixed together for simultaneous capture of the same scene.
`In addition, US Publication No. 2002/0075258 does not
`disclose the use of a composite image for improved image
`quality in a digital zoom system.
`
`SUMMARY OF THE INVENTION
`
`[0009] The present invention provides a sufficiently com-
`pact, low cost, optical system with a large zoom range for a
`small, lightweight and relatively inexpensive consumerdigi-
`tal camera.
`[0010] What is therefore needed is a digital camera that
`provides a rapidly-operating extended zoom range without
`unduly increasing the size or cost of the digital camera while
`providing good perceived image quality throughout
`the
`zoom range.
`[0011] An object of the invention is to provide a method
`of producing a digital image having portions with different
`resolutions comprising:
`[0012]
`a. simultaneously capturing first and seconddigital
`images of the same scene wherein thefirst digital image is
`of a larger portion of the scene than the seconddigital image
`wherein the second digital image has a higher resolution
`than the resolution in the first digital image corresponding to
`the second digital image; and
`[0013]
`b. combiningatleast a portion of the seconddigital
`image into the corresponding portion of the first digital
`image to thereby provide a digital image having portions
`with different resolutions.
`
`[0014] The present inventionis directed to overcoming the
`problemsset forth above. Briefly summarized, the invention
`includes an electronic camera for producing an imageof a
`scene, wherein the camera includes a first image sensor for
`generating a first sensor output, a first lens with a first focal
`length for forming a first image of the scene on thefirst
`imagesensor, a second image sensorfor generating a second
`sensor output, and a second lens with a second focal length
`APPL-1006 / Page 10 of 17
`
`APPL-1006 / Page 10 of 17
`
`

`

`US 2008/0030592 Al
`
`Feb. 7, 2008
`
`length of the first lens for
`that is longer than the focal
`forming a second image of the same scene on the second
`image sensor. Thefirst lens or the second lens can be either
`fixed focal length lenses or multiple focal length lenses as in
`a zoom lens wherein, the first and secondlensesare directed
`at substantially the same scene and imagesets are captured
`substantially simultaneously by the first image sensor and
`the second imagesensor. Portions of the image set captured
`by the first image sensor and the second image sensor are
`then combined to produce a composite image with a higher
`resolution in the portion of the composite image that is
`provided by the second imagesensordueto the longer focal
`length of the second lens. Subsequent
`images produced
`during a digital zooming process are composedlargely of the
`lowerresolution image captured bythe first image sensorat
`low digital zoom values andlargely of the higher resolution
`imageas captured by the second imagesensorat high digital
`zoom values.
`
`[0015] By forming a composite image with portions of the
`image from the short focal length lens and portions of the
`image from the longer focal length lens, perceived image
`quality is improved throughout the zoom range while lens
`complexity is reduced, since a continuous zoom ratio can be
`produced with unmatched lens focal lengths. By capturing
`images from the two image sensors substantially simulta-
`neously, complexities in the image processing are reduced
`since differences between the two images due to motion of
`the camera or motion within the scene are avoided. It is an
`
`additional advantage, that the present invention can avoid
`the slow response that is typical of an optical zoom system
`whentraversing a large zoom range.
`features and
`[0016] These and other aspects, objects,
`advantages of the present invention will be more clearly
`understood and appreciated from a review of the following
`detailed description of the preferred embodiments and
`appended claims, and by reference to the accompanying
`drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIGS. 1A and 1B depict a block diagram of a
`[0017]
`digital camera using a fixed focal length wide-angle lens
`with a first image sensor and a zoom lens, or a longer second
`fixed focal length lens, with a second image sensor accord-
`ing to the present invention;
`[0018]
`FIGS. 2A and 2B show front and rear perspective
`views of the digital camera;
`[0019]
`FIGS. 3A and 3B are perspective viewsof the front
`and back of a cell phone including a camera with multiple
`lenses and multiple sensors according to the present inven-
`tion;
`FIGS. 4A and 4B show two views of the capture
`[0020]
`assembly used in the cell phone shown in FIGS. 3A and 3B
`[0021]
`FIG. 5 is a block diagram of the stitching process
`to create the composite image;
`[0022]
`FIG. 6 depicts a wide angle image as captured, a
`telephoto image as captured, and a composite image as
`created by the invention; and
`
`FIG. 7 is a block diagram ofthe stitching process
`[0023]
`with video images to create a composite video.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`employing imaging
`cameras
`[0024] Because digital
`devices and related circuitry for signal capture, correction,
`and exposure control are well known,the present description
`will be directed in particular to elements formingpart of, or
`cooperating more directly with, method and apparatus in
`accordance with the present invention. Elements not spe-
`cifically shown or described herein are selected from those
`knownin the art. Certain aspects of the embodiments to be
`described are provided in software. Given the system as
`shown and described according to the invention in the
`following materials,
`software not
`specifically shown,
`described or suggested herein that is useful for implemen-
`tation of the invention is conventional and within the ordi-
`
`nary skill in such arts.
`[0025]
`Inthe image capture devicethat is the subject of the
`invention, two or more lens systems are associated with a
`respective number of image sensors. The lenses have dif-
`ferent focal lengths and different fields of view within the
`same scene wherein the field of view of the longer focal
`length lenses contains at least a portion ofthe field of view
`of the shorter focal
`length lens. In addition,
`the image
`captured by the image sensor associated with the longer
`focal length lens has a higher resolution than the image
`captured by the image sensor associated with the lens with
`the shorter focal length.
`the image
`[0026]
`In the embodiment of the invention,
`capture done by the two or more image sensors is done
`substantially simultaneously so that motion artifacts from
`motion of the camera or motion within the scene, do not
`cause differences in the two or more images that are cap-
`tured. The invention discloses the use of the two or more
`
`images to form a composite image that includes portions of
`each of the two or more imagesfor the purpose of providing
`a digitally zoomed image with uniformly high resolution.
`[0027] Each of the several embodiments of the present
`invention include an image capture assembly having mul-
`tiple lenses and multiple image sensors mounted within a
`digital camera wherein the multiple lenses have different
`focal lengths and portions of the fields of view are substan-
`tially the same and the multiple image sensors can capture
`images simultaneously. The invention describes an arrange-
`ment for producing an image that is formed by combining
`the images from the multiple image sensors in a way that
`provides increased resolution in a digitally zoomed image.
`[0028]
`In each embodiment, the camera captures images
`from the multiple image sensors simultaneously. Each mul-
`tiple lens system containsat least one fixed focal length lens
`or variable focal length lens as in an optical zoom lens.
`Moreover, each embodiment includes some type of user
`control that allows a user to select a zoom amount, which
`controls both the digital zoom and the optical zoom lensif
`present. In some embodiments, a single “zoom lens” user
`control is used. e.g., where the “wide”setting selects a wide
`angle fixed focal length lens and the “tele” setting(s) select
`various positions of a zoom lens.
`In any case, digital
`zooming is used along with any optical zoom that is present
`to provide a continuous zoom “up” from the image obtained
`with the short focal length lens to the maximum focal length
`of the multiple lenses. All this, of course, can be transparent
`APPL-1006 / Page 11 of 17
`
`APPL-1006 / Page 11 of 17
`
`

`

`US 2008/0030592 Al
`
`Feb. 7, 2008
`
`to the user, who simply manipulates the “zoom”user control
`between the “wide” and “tele” settings.
`[0029] The composite image can be formed during image
`processing on the camera or later during post processing
`when the images have been offloaded from the camera. In
`either case, the two images must be matched to locate the
`high-resolution image accurately into the low-resolution
`image and thenstitched into place so the edge between the
`two images in the composite image is not discernible. To
`enable the composite image to be formed during post
`processing, both images in the image set must be stored at
`the time of image capture. In the case of video, by storing the
`low-resolution video and the high resolution video,
`the
`zoom ratio can be selected after image capture and adjusted
`as desired at that time.
`
`[0030] Turning now to FIG. 1A, a digital camera 10A is
`described which includes an image capture assembly,
`including a fixed focal length lens 2 that focuses an image
`of a scene (not shown) onto a first image sensor 12, and a
`zoom lens 3 which focuses an image of the scene onto a
`second image sensor 14. The image capture assembly 1
`provides a first image output signal 12e from thefirst image
`sensor 12 and a second image output signal 14e from the
`second image sensor 14.
`[0031] The focal length of the fixed focal length lens 2
`generates a wide-angle field of view and hasa fixed focusset
`to a distance near the lens hyperfocal distance of 8 feet so
`that objects from 4 feet to infinity are in focus. Therefore, the
`fixed focal length lens 2 does not need to include a focus
`adjustment. The fixed focal length lens 2 includes an adjust-
`able aperture and shutter assembly to control the exposure of
`the first image sensor 12. The zoom lens 3 includes an
`optical zoom and autofocus controlled by zoom and focus
`motors 5 and an adjustable aperture and shutter assembly to
`control the exposure of the image sensor.
`[0032]
`In a preferred embodiment, the image sensors 12
`and 14 are single-chip color Megapixel CCD sensors, using
`the well-known Bayercolorfilter pattern to capture color
`images. The image sensors 12 and 14 can have, for example,
`a 4:3 image aspect ratio and a total of 3.1 effective mega-
`pixels (million pixels), with 2048 active columnsofpixelsx
`1536 active rowsof pixels. The image sensors 12 and 14 can
`use a ¥4" type optical format, so that each pixel is approxi-
`mately 3.1 microns tall by 3.1 microns wide. A control
`processor and timing generator 40 controls the first image
`sensor 12 by supplying signals to clock drivers 13, and
`controls the second image sensor 14 by supplying signals to
`clock drivers 15.
`
`[0033] The control processor and timing generator 40 also
`controls the zoom and focus motors 5 for zoom lens 3, and
`a flash 48 for emitting light to illuminate the scene. The
`control processor and timing generator 40 also receives
`signals from automatic focus and automatic exposure detec-
`tors 46. In an alternative embodiment, instead of using the
`automatic focus and automatic exposure detectors 46, the
`image sensor 14 could be used to provide exposure detection
`and “through-the-lens” autofocus, as described in com-
`monly-assigned U.S. Pat. No. 5,668.597 entitled “Electronic
`Camera with Rapid Automatic Focus of an Image upon a
`Progressive Scan Image Sensor” which issued Sep. 26, 1997
`in the names of Kenneth A. Parulski, Masaki Izumi, Seiichi
`Mizukoshi and Nobuyuki Mori,
`incorporated herein by
`reference. User controls 42 are used to control the operation
`of the digital camera 10A.
`
`[0034] The first image output signal 12e from the first
`image sensor 12 is amplified by a first analog signal pro-
`cessor (ASP 1) 22 and providedto a first analog-to-digital
`(A/D) converter 34. The second image output signal 14e
`from the second image sensor 14 is amplified by a second
`analog signal processor (ASP 2) 24 and provided to a second
`A/D converter 36.
`
`Thedigital data from the A/D converters 34 and 36
`[0035]
`is providedto digital multiplexer 37. The digital multiplexer
`37 is used to select which one of the outputs of the two A/D
`converters 34 and 36 is connected to the DRAM buffer
`
`memory 38. The digital data is stored in DRAM buffer
`memory 38 and subsequently processed by an image pro-
`cessor 50. The processing performed by the image processor
`50 is controlled by firmware stored in fin-ware memory 58,
`which can be flash EPROM memory. The image processor
`50 processes the input digital image file, which is buffered
`in a RAM memory 56 during the processing stage. The
`image processor 50 combinesthe digital data from the A/D
`converters 34 and 36 to form a composite image with areas
`of high resolution and areas of lower resolution using a
`method, which constitutes the invention.
`[0036] As shown in FIG. 5, the image processor 50 of
`FIGS. 1A and 1B contains an image compositor 202 that
`receives both the wide image 204 from the fixed focal length
`lens 2 and the telephoto image 206 from the zoom lens 3.
`The telephoto image 206 is of a smaller portion of the scene
`than the wide image 204, but captures this smaller portion
`with greater resolution than the resolution of the wide image
`204. The image compositor 202 generates a composite
`image 208 using image data from both the wide image 204
`and the telephoto image 206. Also, the image compositor
`202 receives a zoom amount 210 that can be adjusted by the
`camera user as will be described below.
`
`It is desirable for the image compositor 202 to
`[0037]
`generate a composite image 208 that has the highest possible
`quality. For illustration, assumethat the wide image 204 and
`the telephoto image 206 have the same number of rows R
`and columnsC ofpixels, for example, R=1000 and C=1500
`and that the relative magnification ratio M of the telephoto
`image 206 to the wide image 204 is M=3.
`[0038] The image registration determiner 212 determines
`the registration between the wide image 204 and thetele-
`photo image 206. The coordinate transformation is simply a
`translation and a scale because the image sensors that
`capture the wide image 204 andthe telephoto image 206 are
`coplanar. A convenient way to represent the registration
`between the imagesis to find the mapping of the four corner
`pixels of the telephoto image 206 onto the wide image 204.
`For example,
`
`Telephoto Image Coordinates
`
`Wide Image Coordinates
`
`(0, 0)
`(999, 0)
`(0, 1499)
`(999, 1499)
`
`(333, 499.7)
`(666, 499.7)
`(333, 999.3)
`(666, 999.3)
`
`The registration can also be stored in the form of the
`homography H,,, that transforms the coordinates of the
`telephoto image 206 to the wide image 204.
`APPL-1006 / Page 12 of 17
`
`APPL-1006 / Page 12 of 17
`
`

`

`US 2008/0030592 Al
`
`Feb. 7, 2008
`
`[0039]
`
`Aw
`Yw
`1
`
`XT
`|= Arw| yr
`1
`
`
`
`
`
`Where coordinates of the telephoto image 206 are in (row,
`column) notation (y;, X;) and coordinates of the wide image
`204 are (yj, Xz). For example,
`
`[0040]
`
`xw
`yw |= Hew
`1
`
`
`
` XC
`
`ye
`1
`
`where
`
`cw =
`
`(L-MyYC-1)
`1/Z 0
`0
`1/Z (-MyYC-1)
`0
`0
`1
`
`
`
`In a similar manner, the position (y,, x.) of the composite
`image 208 is mapped to the coordinates of the telephoto
`image 206 using the equation:
`
`Xr
`IT
`1
`
`
`
`= (Hrw)Hew
`
`xc|1
`
`
`
`Then,the pixel value of the composite imageat position(y,,
`X,) is found by interpolation. If the mapped position of(y,,
`X,) in the telephoto image 206 lands within the limits of the
`existing pixels (i.e. 0<=x,;<=C-1), the pixel value of the
`composite image 208 at position (y,., X,) is found by inter-
`polating pixel values of the telephoto image 206. Otherwise,
`the pixel value of the composite image 208 at position (y,,
`X,) is found by interpolating pixel values of the wide image
`204.
`
`[0046] Those skilled in the art will recognize that the
`above description of producing the values of the composite
`image 208 using pixel values of the wide image 204 and the
`telephoto image 206 can be accomplished in many ways. For
`example,
`it
`is easy to pre-calculate the region of pixel
`locations of the composite image 208 for which the pixel
`values will be producedby interpolating the telephoto image
`206 and the region for which the pixel values will be
`produced by interpolating the wide image 204. This saves
`computational cost but produces the same image data.
`[0047]
`FIG. 6 shows an example set of images. The wide
`image 204 covers a wide portion of the scene and the
`telephoto image 206 covers a smaller portion of the scene,
`but with greater resolution. The produced composite image
`208 uses pixel data from the telephoto image 206 for those
`portions(i.e. the region within the dashed line 220) that are
`in the view of the telephoto image 206 and uses pixel data
`from the wide image 204 otherwise (i.e. the region outside
`the dashed line 220). The dashed line 220 shows where the
`transition is. Thus,
`the composite image 208 has higher
`resolution in the interior and lower resolution on the edges.
`Since the subject of a photograph, especially in consumer
`photography, is likely to be near the center of the scene, the
`subject of the composite image 208 is likely to have the
`highest resolution. It has also been experimentally deter-
`mined that the transition within the composite image 208
`between pixels derived by interpolating the wide image 204
`versus the telephoto image 206 does not product visually
`objectionable artifacts.
`[0048]
`Since lenses 2 and 3 are separated by some dis-
`tance, it is possible that objects very close to the camerawill
`appear to have a discontinuity at the transition. In this case,
`it is possible to use standard image processing techniques to
`
`APPL-1006 / Page 13 of 17
`
`1/M 0
`
`499.7
`
`Hr =|
`
`
`
`0
`
`0
`
`1/M 333 |
`
`0
`
`1
`
`[0041] The correspondences between the coordinate sys-
`tems represent the registration between the wide image 204
`and the telephoto image 206. The correspondences are
`preferably determined at the time of manufacture by shoot-
`ing test targets, as is well known intheart. If one or both of
`the lenses were a zoom lens rather than a fixed lens., the
`registration correspondencescouldstill be determined at the
`time of manufacture as a function of the zoom position of the
`lenses. It should be further noted that while the example
`showsa pure translate and scale transformation, it may be
`necessary to correct for a difference in tilt between the two
`imaging systems.
`[0042] Alternatively, the registration between images can
`be determined using the image information contained in the
`wide image 204 and telephoto image 204. This is well
`knownin the art of image processing (for example, image
`registration is described in U.S. Pat. No. 6,078,701) and
`generally includes the steps of finding interest points in each
`image, making guesses at corresponding points(i.e. a scene
`feature that appears in both images), determining an initial
`guess at the registration, using that initial guess to refine the
`correspondence point guess, and so on based on comparing
`pixel values or contrast in the two images.
`[0043] The image resampler 214 uses the registration
`information and the zoom amount 210 to produce the
`composite image 208. Preferably, the composite image has
`the same numberof rows and columnsofpixels as the wide
`image 204 andthe telephoto image 206. However,it is well
`knownto those skilled in the art that modifying the number
`of rows and columnsofpixels (interpolating the image) can
`easily be done sothat the image contains the desired number
`of pixels.
`[0044] The zoom amount 210 Z specifies the desired
`relative zoom amountof the produced composite image 208.
`Preferably, when the value of Z=1, then the composite image
`is the wide image 204. On the other hand, when Z-M,then
`the composite image 208 is the telephoto image 206. When
`the zoom amount is between 1 and M,data from both the
`wide image 204 andthe telephoto image 206 are used by the
`image resampler 214 to produce the composite image 208.
`[0045] The image resampler 214 applies the zoom amount
`Z as follows: Each pixel position (y,, x,) of the composite
`image 208 is mappedto the coordinates of wide image 204
`according to:
`
`APPL-1006 / Page 13 of 17
`
`

`

`US 2008/0030592 Al
`
`Feb. 7, 2008
`
`find objects that are close to the camera and to process these
`regions in a fashion that does not produce a discontinuity
`artifact. For example,
`the pixel values of the composite
`image 208, for objects that are close to the camera and span
`the transition region, can be determined by interpolating the
`wide image 204. A true depth map can also be created and
`used by the image resampler 214 to sample the appropriate
`locations within the telephoto image 206 and the wide image
`204. In this case, the registration modelis no longer a simple
`scale translation model.
`
`[0049] A further feature of the present inventionis that the
`composite image 208 can be stored on the camera without
`digital zooming. Therefore, digital zooming of the compos-
`ite image 208 can be (lonelater, during post processing, to
`create an image for use by the operator for printing or
`sharing. The composite image 208 can be formed during
`image processing on the camera or later during post pro-
`cessing when the images have been offloaded from the
`camera. To enable the composite image to be formed during
`post processing, the wide image 204 and the telephoto image
`206 must both be stored at the time of image capture.
`[0050] The invention can also be applied to a series of
`sequential images as in a video. Referring to FIG. 7, in the
`case of video, two sets of video images, wide video images
`220 and telephoto video images 222 are captured substan-
`tially simultaneously from the two lenses 2 and 3 or 2 and
`4 and the two image sensors 12 and 14 providing video
`images from a short focal length lens 2 and a zoom lens 3
`or a longer second focal length lens 4. The composite video
`224 is formed by combining the two sets of video images
`220 and 222. The composite video 224 can be formed during
`image processing on the camera and stored on the camera or
`the composite video 224 can be formed later during post
`processing when the images have been offloaded from the
`camera. To enable the composite video 224 to be formed
`during post processing, the wide video images 220 from the
`short focal length lens 2 and the telephoto video images 222
`from the zoom lens 3 or the longer focal length lens 4 must
`both be stored at the time of image capture. Digital zoom of
`the video images can be accomplished on the camera during
`capture, or on the camera after capture, or during post
`processing after the composite video 224 has been offloaded
`from the camera or during post processing when the com-
`posite video 224 is being formed.
`[0051] The processed digital image file is provided to a
`memory card interface 52, which stores the digital imagefile
`on the removable memory card 54. Removable memory
`cards 54 are one type of removable digital image storage
`medium, and are available in several different physical
`formats. For example, the removable memory card 54 can
`include (without limitation) memory cards adapted to well-
`known formats, such as the Compact Flash, SmartMedia.
`MemoryStick, MMC, SD, or XD memory card formats.
`Other types of removable digital image storage media, such
`as magnetic hard drives, magnetic tape, or optical disks, can
`alternatively be used to store the still and motion digital
`images. Alternatively, the digital camera 10A can use inter-
`nal non-volatile memory (not shown), such as internal flash
`EPROM memory to store the processed digital imagefiles.
`In such an embodiment, the memory card interface 52 and
`the removable memory card 54 are not needed.
`[0052] The image processor 50 performs various image
`processing functions, including colorinterpolation followed
`by color and tone correction, in order to produce rendered
`
`sRGB image data. The rendered sRGB image data is then
`JPEG compressed and stored as a JPEG imagefile on the
`removable memory card 54. The rendered sRGB image data
`can also be provided to a host PC 66 via a host interface 62
`communicating over a suitable interconnection, such as a
`SCSI connection, a USB connection or a Firewire connec-
`tion. The JPEGfile uses the so-called “Exif” image format
`defined in “Digital Still Camera Image File Format (Exit)”
`version 2.1, July 1998 by the Japan Electronics Industries
`Development Association (JEIDA), Tokyo, Japan. This for-
`mat includes an Exif application segment that stores par-
`ticular image metadata, including the date or time the image
`was captured, as well as the lens f/number and other camera
`settings.
`It should be noted that the image processor 50,
`[0053]
`although typically a programmable image processor, can
`alternatively be a hard-wired custom integrated circuit (IC)
`processor, a general purpose microprocessor, or a combina-
`tion of hard-wired custom IC and programmable processors.
`[0054] The image processor 50 also creates a low-resolu-
`tion “thumbnail” size image, which can be created as
`described in commonly-assigned U.S. Pat. No. 5,164,831,
`entitled “Electronic Still Camera Providing Multi-Format
`Storage Of Full And Reduced Resolution Images”issued in
`the name of Kuchta,et al., the disclosure of which is herein
`incorporated by refere