`(12) Patent Application Publication (10) Pub. No.: US 2008/0030592 A1
`(43) Pub. Date:
`Feb. 7, 2008
`BOrder et al.
`
`US 20080030592A1
`
`(54)
`
`PRODUCING DIGITAL MAGE WITH
`DIFFERENT RESOLUTION PORTIONS
`
`(75)
`
`Inventors:
`
`John N. Border, Walworth, NY
`S.S.",'',EP".
`Rochester, NY (US)
`s
`
`ESSAS RoSR coMPANY
`PATENT LEGAL STAFF
`343 STATE STREET
`ROCHESTER NY 146SO-22O1
`9
`Eastman Kodak Company
`11/461,574
`
`Assignee:
`Appl. No.:
`
`Filed:
`
`Aug. 1, 2006
`
`(73)
`(21)
`(22)
`
`Publication Classification
`
`(51) Int. Cl.
`(2006.01)
`G06T 5/50
`(52) U.S. Cl. ............................. 348/218.1: 348/E05.028
`
`ABSTRACT
`(57)
`A method of producing a digital image with improved
`resolution during digital Zooming, including simultaneously
`capturing a first low resolution digital image of a scene and
`a second higher resolution digital image of a portion of
`Substantially the same scene. A composite image is then
`formed by combining the first low-resolution digital image
`and a corresponding portion of the high resolution digital
`image. Digital Zooming of the composite image produces a
`Zoomed image with high resolution throughout the Zoom
`range and improved image quality.
`
`2
`in S.
`1st IMAGE
`FOCAL |
`LENGTH U SENSOR
`LENS
`
`22
`
`12e
`ASP 1
`" ' '
`
`3
`al,
`f3
`LENS
`
`A.
`
`13
`
`COCK
`s
`DRIVERS
`14
`14e
`S.
`\ \
`22 MAGE
`SENSori". ASP 2.
`f DIGITAL
`15
`J7
`MUY
`CLOCK 24
`ZOOMAND r | 33- DEAM.
`FOCUS MOTORS
`5:E
`5 J
`CONTROL
`40-r-PROCESSOR AND
`AUTOFOCUS AND -- Ge.R
`46-AUTOEXPOSURE
`| --
`DETECTORS
`
`5.
`--- 50
`is 4/0,
`3
`h
`| CONVERTER
`36
`TS
`2nd A/D
`CONWERIER
`
`m
`
`66
`
`---
`HOST 1. 10A
`PC
`54
`
`RAF
`MEMORY
`CARD
`
`62----,
`-n
`| HOST
`Nier ACE
`64
`52
`ever
`| MEMORY
`--- CARD
`INTERFACE
`-------
`56
`S
`RAM
`MEMORY
`
`IMAGE
`PROCESSOR
`
`H-e
`
`COLOR LCD
`70-N- IMAGE
`DISPLAY
`USER -
`CONTROS
`42
`
`
`
`48-N- FLASH - - -
`
`58
`FIRMWARE
`MEMORY
`
`90- - -
`
`CELULAR
`PROCESSOR
`-->
`
`92
`
`94
`critjar
`CELLULAR
`|
`MODEM
`Li Yu-ri -
`
`APPL-1011 / Page 1 of 17
`APPLE INC. v. COREPHOTONICS LTD.
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 1 of 8
`
`US 2008/0030592 A1
`
`
`
`
`
`
`
`
`
`
`
`
`
`APPL-1011 / Page 2 of 17
`APPLE INC. v. COREPHOTONICS LTD.
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 2 of 8
`
`US 2008/0030592 A1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`APPL-1011 / Page 3 of 17
`APPLE INC. v. COREPHOTONICS LTD.
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 3 of 8
`
`US 2008/0030592 A1
`
`
`
`FIG. 2B
`
`APPL-1011 / Page 4 of 17
`APPLE INC. v. COREPHOTONICS LTD.
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 4 of 8
`
`US 2008/0030592 A1
`
`
`
`APPL-1011 / Page 5 of 17
`APPLE INC. v. COREPHOTONICS LTD.
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 5 of 8
`
`US 2008/0030592 A1
`
`
`
`
`
`
`
`
`
`
`
`| | | | | |
`
`+------|----
`
`APPL-1011 / Page 6 of 17
`APPLE INC. v. COREPHOTONICS LTD.
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 6 of 8
`
`US 2008/0030592 A1
`
`371/SOd/WOO
`
`
`
`579 7/W/
`
`ZOZ
`
`
`
`G (OICH
`
`| N/QO/W/
`
`/WOOZ
`
`90Z
`
`012
`
`APPL-1011 / Page 7 of 17
`APPLE INC. v. COREPHOTONICS LTD.
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 7 of 8
`
`US 2008/0030592 A1
`
`2O4.
`
`2O6
`
`2O6
`
`
`
`APPL-1011 / Page 8 of 17
`APPLE INC. v. COREPHOTONICS LTD.
`
`
`
`Patent Application Publication
`
`Feb. 7, 2008 Sheet 8 of 8
`
`US 2008/0030592 A1
`
`371/SOC/WOO
`
`O B7C7///|
`
`ZOZ
`
`
`
`
`
`Z {0/ H.
`
`[ N/TOW /
`
`WOOZ
`
`O IZ
`
`OZZ
`
`ZZZ
`
`APPL-1011 / Page 9 of 17
`APPLE INC. v. COREPHOTONICS LTD.
`
`
`
`US 2008/0030592 A1
`
`Feb. 7, 2008
`
`PRODUCING DIGITAL IMAGE WITH
`DIFFERENT RESOLUTION PORTIONS
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`Reference is made to commonly assigned U.S.
`0001
`Patent Application Serial No. 2002/0075258, filed Nov. 23,
`2001, entitled “Camera System with High Resolution Image
`Inside a Wide Angle View' by Park et al. and U.S. patent
`application Ser. No. 11/062,174, filed Feb. 18, 2005, entitled
`“Digital Camera Using Multiple Lenses And Image Sensors
`To Provide An Extended Zoom Range' by Peter Laba
`Ziewicz, et al., the disclosures of which are incorporated
`herein.
`
`FIELD OF THE INVENTION
`0002 The present invention relates to a digital camera
`that uses multiple lenses and image sensors to provide an
`extended Zoom range and the method used to produce a
`digital image that combines the multiple images produced
`by the digital camera.
`
`BACKGROUND OF THE INVENTION
`0003 Currently, most digital cameras use a Zoom lens
`and a single color image sensor to capture still and motion
`images. The captured images are then digitally processed to
`produce digital image files, which are stored in a digital
`memory in the camera. The digital image files can then be
`transferred to a computer, displayed, and shared via the
`Internet. The digital camera can be included as part of a
`mobile telephone, to form a so-called “camera phone.” The
`camera phone can transmit the digital image files to another
`camera phone, or to service providers, via a mobile tele
`phone network.
`0004 Small camera size and a large Zoom range are two
`very important features of digital cameras. Users prefer to
`have a large Zoom range (e.g. 5:1 or greater) rather than a
`limited Zoom range (e.g. 3:1 or Smaller). The Zoom range is
`typically composed of both optical Zoom which is provided
`by variable focal length lenses and digital Zoom which is
`provided by a magnification of the digital image after
`capture. Variable focal length lenses for large Zoom range
`are expensive and they increase the size of the digital
`camera. Thus, there are trade-offs between small camera
`size, large Zoom range, and low camera cost which must be
`made when designing a digital camera. With higher cost
`cameras, such as single lens reflex cameras, these problems
`are sometimes addressed by using multiple interchangeable
`Zoom lenses, such as two 3:1 Zoom lenses, e.g., a 28-70 mm
`Zoom and a 70-210 Zoom. This arrangement has user incon
`venience problems and is presently not available for low
`cost digital cameras.
`0005. A different solution that has been offered by Kodak
`in the V570 and the V610 cameras is to include two different
`lens assemblies in the camera with two different focal
`lengths and two separate image sensors. In this case, each of
`the lens assemblies can be either a fixed focal length lens or
`can have a moderate optical Zoom range to reduce the size
`and cost of each of the lens assemblies. Together, the two
`lens assemblies provide a wide Zoom range and a small
`overall size at a lower cost. However, a problem arises when
`the focal length of the first lens does not match the focal
`length of the second lens so that the optical Zoom is not
`
`continuous over the entire Zoom range. In this case, digital
`Zoom must be used for Zoom between the maximum Zoom
`of the first lens and the minimum Zoom of the second lens.
`0006 Digital Zoom based on increased magnification of
`the image with a corresponding decrease in resolution is
`well known in the art. Although digital Zoom is very fast and
`simple, the decrease in resolution can produce a perceived
`decrease in image quality.
`0007. In U.S. Pat. No. 5,657,402, a method is described
`in which a plurality of digital images are combined to form
`an image. U.S. Pat. No. 5,657,402 addresses the use of
`multiple images captured at different times wherein “the
`plurality of images of various focal lengths, such as a Zoom
`Video sequence' (col. 1, lines, 21-22) are captured from the
`same lens. U.S. Pat. No. 5,657,402 does not address two lens
`assemblies simultaneously capturing images of the same
`SCCC.
`0008. In US Publication No. 2002/0075258, a panoramic
`camera system is described in which a moveable telephoto
`camera is additionally used to capture a high-resolution
`portion of the scene which is then overlaid onto the pan
`oramic image. US Publication No. 2002/0075258 describes
`the use of a moveable telephoto camera to enable a higher
`resolution of a portion of the image wherein the moveable
`telephoto camera can be moved to the region of the pan
`oramic image where the higher resolution is desired. US
`Publication No. 2002/0075.258 does not address the case
`wherein a wide-angle camera and a telephoto camera are
`affixed together for simultaneous capture of the same scene.
`In addition, US Publication No. 2002/0075258 does not
`disclose the use of a composite image for improved image
`quality in a digital Zoom system.
`
`SUMMARY OF THE INVENTION
`0009. The present invention provides a sufficiently com
`pact, low cost, optical system with a large Zoom range for a
`Small, lightweight and relatively inexpensive consumer digi
`tal camera.
`0010 What is therefore needed is a digital camera that
`provides a rapidly-operating extended Zoom range without
`unduly increasing the size or cost of the digital camera while
`providing good perceived image quality throughout the
`ZOOm range.
`0011. An object of the invention is to provide a method
`of producing a digital image having portions with different
`resolutions comprising:
`0012 a. simultaneously capturing first and second digital
`images of the same scene wherein the first digital image is
`of a larger portion of the scene than the second digital image
`wherein the second digital image has a higher resolution
`than the resolution in the first digital image corresponding to
`the second digital image; and
`0013 b. combining at least a portion of the second digital
`image into the corresponding portion of the first digital
`image to thereby provide a digital image having portions
`with different resolutions.
`0014. The present invention is directed to overcoming the
`problems set forth above. Briefly summarized, the invention
`includes an electronic camera for producing an image of a
`scene, wherein the camera includes a first image sensor for
`generating a first sensor output, a first lens with a first focal
`length for forming a first image of the scene on the first
`image sensor, a second image sensor for generating a second
`sensor output, and a second lens with a second focal length
`
`APPL-1011 / Page 10 of 17
`APPLE INC. v. COREPHOTONICS LTD.
`
`
`
`US 2008/0030592 A1
`
`Feb. 7, 2008
`
`that is longer than the focal length of the first lens for
`forming a second image of the same scene on the second
`image sensor. The first lens or the second lens can be either
`fixed focal length lenses or multiple focal length lenses as in
`a Zoom lens wherein, the first and second lenses are directed
`at Substantially the same scene and image sets are captured
`Substantially simultaneously by the first image sensor and
`the second image sensor. Portions of the image set captured
`by the first image sensor and the second image sensor are
`then combined to produce a composite image with a higher
`resolution in the portion of the composite image that is
`provided by the second image sensor due to the longer focal
`length of the second lens. Subsequent images produced
`during a digital Zooming process are composed largely of the
`lower resolution image captured by the first image sensor at
`low digital Zoom values and largely of the higher resolution
`image as captured by the second image sensor at high digital
`Zoom values.
`0015. By forming a composite image with portions of the
`image from the short focal length lens and portions of the
`image from the longer focal length lens, perceived image
`quality is improved throughout the Zoom range while lens
`complexity is reduced, since a continuous Zoom ratio can be
`produced with unmatched lens focal lengths. By capturing
`images from the two image sensors Substantially simulta
`neously, complexities in the image processing are reduced
`since differences between the two images due to motion of
`the camera or motion within the scene are avoided. It is an
`additional advantage, that the present invention can avoid
`the slow response that is typical of an optical Zoom system
`when traversing a large Zoom range.
`0016. These and other aspects, objects, features and
`advantages of the present invention will be more clearly
`understood and appreciated from a review of the following
`detailed description of the preferred embodiments and
`appended claims, and by reference to the accompanying
`drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`0017 FIGS. 1A and 1B depict a block diagram of a
`digital camera using a fixed focal length wide-angle lens
`with a first image sensor and a Zoom lens, or a longer second
`fixed focal length lens, with a second image sensor accord
`ing to the present invention;
`0018 FIGS. 2A and 2B show front and rear perspective
`views of the digital camera;
`0019 FIGS. 3A and 3B are perspective views of the front
`and back of a cell phone including a camera with multiple
`lenses and multiple sensors according to the present inven
`tion;
`0020 FIGS. 4A and 4B show two views of the capture
`assembly used in the cell phone shown in FIGS. 3A and 3B
`0021
`FIG. 5 is a block diagram of the stitching process
`to create the composite image:
`0022 FIG. 6 depicts a wide angle image as captured, a
`telephoto image as captured, and a composite image as
`created by the invention; and
`
`0023 FIG. 7 is a block diagram of the stitching process
`with video images to create a composite video.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`0024. Because digital cameras employing imaging
`devices and related circuitry for signal capture, correction,
`and exposure control are well known, the present description
`will be directed in particular to elements forming part of, or
`cooperating more directly with, method and apparatus in
`accordance with the present invention. Elements not spe
`cifically shown or described herein are selected from those
`known in the art. Certain aspects of the embodiments to be
`described are provided in software. Given the system as
`shown and described according to the invention in the
`following materials, Software not specifically shown,
`described or Suggested herein that is useful for implemen
`tation of the invention is conventional and within the ordi
`nary skill in Such arts.
`0025. In the image capture device that is the subject of the
`invention, two or more lens systems are associated with a
`respective number of image sensors. The lenses have dif
`ferent focal lengths and different fields of view within the
`same scene wherein the field of view of the longer focal
`length lenses contains at least a portion of the field of view
`of the shorter focal length lens. In addition, the image
`captured by the image sensor associated with the longer
`focal length lens has a higher resolution than the image
`captured by the image sensor associated with the lens with
`the shorter focal length.
`0026. In the embodiment of the invention, the image
`capture done by the two or more image sensors is done
`Substantially simultaneously so that motion artifacts from
`motion of the camera or motion within the scene, do not
`cause differences in the two or more images that are cap
`tured. The invention discloses the use of the two or more
`images to form a composite image that includes portions of
`each of the two or more images for the purpose of providing
`a digitally Zoomed image with uniformly high resolution.
`0027. Each of the several embodiments of the present
`invention include an image capture assembly having mul
`tiple lenses and multiple image sensors mounted within a
`digital camera wherein the multiple lenses have different
`focal lengths and portions of the fields of view are substan
`tially the same and the multiple image sensors can capture
`images simultaneously. The invention describes an arrange
`ment for producing an image that is formed by combining
`the images from the multiple image sensors in a way that
`provides increased resolution in a digitally Zoomed image.
`0028. In each embodiment, the camera captures images
`from the multiple image sensors simultaneously. Each mul
`tiple lens system contains at least one fixed focal length lens
`or variable focal length lens as in an optical Zoom lens.
`Moreover, each embodiment includes some type of user
`control that allows a user to select a Zoom amount, which
`controls both the digital Zoom and the optical Zoom lens if
`present. In some embodiments, a single "Zoom lens' user
`control is used. e.g., where the “wide' setting selects a wide
`angle fixed focal length lens and the “tele' setting(s) select
`various positions of a Zoom lens. In any case, digital
`Zooming is used along with any optical Zoom that is present
`to provide a continuous Zoom 'up' from the image obtained
`with the short focal length lens to the maximum focal length
`of the multiple lenses. All this, of course, can be transparent
`
`APPL-1011 / Page 11 of 17
`APPLE INC. v. COREPHOTONICS LTD.
`
`
`
`US 2008/0030592 A1
`
`Feb. 7, 2008
`
`to the user, who simply manipulates the "Zoom' user control
`between the “wide' and “tele' settings.
`0029. The composite image can be formed during image
`processing on the camera or later during post processing
`when the images have been offloaded from the camera. In
`either case, the two images must be matched to locate the
`high-resolution image accurately into the low-resolution
`image and then Stitched into place so the edge between the
`two images in the composite image is not discernible. To
`enable the composite image to be formed during post
`processing, both images in the image set must be stored at
`the time of image capture. In the case of video, by storing the
`low-resolution video and the high resolution video, the
`Zoom ratio can be selected after image capture and adjusted
`as desired at that time.
`0030 Turning now to FIG. 1A, a digital camera 10A is
`described which includes an image capture assembly,
`including a fixed focal length lens 2 that focuses an image
`of a scene (not shown) onto a first image sensor 12, and a
`Zoom lens 3 which focuses an image of the scene onto a
`second image sensor 14. The image capture assembly 1
`provides a first image output signal 12e from the first image
`sensor 12 and a second image output signal 14e from the
`second image sensor 14.
`0031. The focal length of the fixed focal length lens 2
`generates a wide-angle field of view and has a fixed focus set
`to a distance near the lens hyperfocal distance of 8 feet so
`that objects from 4 feet to infinity are in focus. Therefore, the
`fixed focal length lens 2 does not need to include a focus
`adjustment. The fixed focal length lens 2 includes an adjust
`able aperture and shutter assembly to control the exposure of
`the first image sensor 12. The Zoom lens 3 includes an
`optical Zoom and autofocus controlled by Zoom and focus
`motors 5 and an adjustable aperture and shutter assembly to
`control the exposure of the image sensor.
`0032. In a preferred embodiment, the image sensors 12
`and 14 are single-chip color Megapixel CCD sensors, using
`the well-known Bayer color filter pattern to capture color
`images. The image sensors 12 and 14 can have, for example,
`a 4:3 image aspect ratio and a total of 3.1 effective mega
`pixels (million pixels), with 2048 active columns of pixelsx
`1536 active rows of pixels. The image sensors 12 and 14 can
`use a /2" type optical format, so that each pixel is approxi
`mately 3.1 microns tall by 3.1 microns wide. A control
`processor and timing generator 40 controls the first image
`sensor 12 by Supplying signals to clock drivers 13, and
`controls the second image sensor 14 by Supplying signals to
`clock drivers 15.
`0033. The control processor and timing generator 40 also
`controls the Zoom and focus motors 5 for Zoom lens 3, and
`a flash 48 for emitting light to illuminate the scene. The
`control processor and timing generator 40 also receives
`signals from automatic focus and automatic exposure detec
`tors 46. In an alternative embodiment, instead of using the
`automatic focus and automatic exposure detectors 46, the
`image sensor 14 could be used to provide exposure detection
`and “through-the-lens' autofocus, as described in com
`monly-assigned U.S. Pat. No. 5,668.597 entitled “Electronic
`Camera with Rapid Automatic Focus of an Image upon a
`Progressive Scan Image Sensor' which issued Sep. 26, 1997
`in the names of Kenneth A. Parulski, Masaki Izumi, Seiichi
`Mizukoshi and Nobuyuki Mori, incorporated herein by
`reference. User controls 42 are used to control the operation
`of the digital camera 10A.
`
`0034. The first image output signal 12e from the first
`image sensor 12 is amplified by a first analog signal pro
`cessor (ASP 1) 22 and provided to a first analog-to-digital
`(A/D) converter 34. The second image output signal 14e
`from the second image sensor 14 is amplified by a second
`analog signal processor (ASP2) 24 and provided to a second
`A/D converter 36.
`0035. The digital data from the A/D converters 34 and 36
`is provided to digital multiplexer 37. The digital multiplexer
`37 is used to select which one of the outputs of the two A/D
`converters 34 and 36 is connected to the DRAM buffer
`memory 38. The digital data is stored in DRAM buffer
`memory 38 and Subsequently processed by an image pro
`cessor 50. The processing performed by the image processor
`50 is controlled by firmware stored in fin-ware memory 58,
`which can be flash EPROM memory. The image processor
`50 processes the input digital image file, which is buffered
`in a RAM memory 56 during the processing stage. The
`image processor 50 combines the digital data from the A/D
`converters 34 and 36 to form a composite image with areas
`of high resolution and areas of lower resolution using a
`method, which constitutes the invention.
`0036) As shown in FIG. 5, the image processor 50 of
`FIGS. 1A and 1B contains an image compositor 202 that
`receives both the wide image 204 from the fixed focal length
`lens 2 and the telephoto image 206 from the Zoom lens 3.
`The telephoto image 206 is of a smaller portion of the scene
`than the wide image 204, but captures this smaller portion
`with greater resolution than the resolution of the wide image
`204. The image compositor 202 generates a composite
`image 208 using image data from both the wide image 204
`and the telephoto image 206. Also, the image compositor
`202 receives a Zoom amount 210 that can be adjusted by the
`camera user as will be described below.
`0037. It is desirable for the image compositor 202 to
`generate a composite image 208 that has the highest possible
`quality. For illustration, assume that the wide image 204 and
`the telephoto image 206 have the same number of rows R
`and columns C of pixels, for example, R=1000 and C=1500
`and that the relative magnification ratio M of the telephoto
`image 206 to the wide image 204 is M=3.
`0038. The image registration determiner 212 determines
`the registration between the wide image 204 and the tele
`photo image 206. The coordinate transformation is simply a
`translation and a scale because the image sensors that
`capture the wide image 204 and the telephoto image 206 are
`coplanar. A convenient way to represent the registration
`between the images is to find the mapping of the four corner
`pixels of the telephoto image 206 onto the wide image 204.
`For example,
`
`Telephoto Image Coordinates
`
`Wide Image Coordinates
`
`(0, 0)
`(999, O)
`(0, 1499)
`(999, 1499)
`
`(333, 499.7)
`(666, 499.7)
`(333,999.3)
`(666, 999.3)
`
`The registration can also be stored in the form of the
`homography H. that transforms the coordinates of the
`telephoto image 206 to the wide image 204.
`
`APPL-1011 / Page 12 of 17
`APPLE INC. v. COREPHOTONICS LTD.
`
`
`
`US 2008/0030592 A1
`
`Feb. 7, 2008
`
`0039
`
`XT
`W
`y = HTwyT
`1
`1
`
`W.W
`
`|
`
`1
`
`= How
`
`XC
`
`1
`
`where
`
`Where coordinates of the telephoto image 206 are in (row,
`column) notation (y,x) and coordinates of the wide image
`204 are (y, X). For example,
`0040
`
`499.7
`1 ? M O
`H = 0
`1 1 M 333
`O
`O
`1
`
`0041. The correspondences between the coordinate sys
`tems represent the registration between the wide image 204
`and the telephoto image 206. The correspondences are
`preferably determined at the time of manufacture by shoot
`ing test targets, as is well known in the art. If one or both of
`the lenses were a Zoom lens rather than a fixed lens... the
`registration correspondences could still be determined at the
`time of manufacture as a function of the Zoom position of the
`lenses. It should be further noted that while the example
`shows a pure translate and Scale transformation, it may be
`necessary to correct for a difference in tilt between the two
`imaging systems.
`0042. Alternatively, the registration between images can
`be determined using the image information contained in the
`wide image 204 and telephoto image 204. This is well
`known in the art of image processing (for example, image
`registration is described in U.S. Pat. No. 6,078,701) and
`generally includes the steps of finding interest points in each
`image, making guesses at corresponding points (i.e. a scene
`feature that appears in both images), determining an initial
`guess at the registration, using that initial guess to refine the
`correspondence point guess, and so on based on comparing
`pixel values or contrast in the two images.
`0043. The image resampler 214 uses the registration
`information and the Zoom amount 210 to produce the
`composite image 208. Preferably, the composite image has
`the same number of rows and columns of pixels as the wide
`image 204 and the telephoto image 206. However, it is well
`known to those skilled in the art that modifying the number
`of rows and columns of pixels (interpolating the image) can
`easily be done so that the image contains the desired number
`of pixels.
`0044) The Zoom amount 210 Z specifies the desired
`relative Zoom amount of the produced composite image 208.
`Preferably, when the value of Z=1, then the composite image
`is the wide image 204. On the other hand, when Z-M, then
`the composite image 208 is the telephoto image 206. When
`the Zoom amount is between 1 and M, data from both the
`wide image 204 and the telephoto image 206 are used by the
`image resampler 214 to produce the composite image 208.
`0045. The image resampler 214 applies the Zoom amount
`Z as follows: Each pixel position (y,x) of the composite
`image 208 is mapped to the coordinates of wide image 204
`according to:
`
`1f Z 0
`(1 - M)(C-1)
`How = 0
`1 f Z 89.
`O
`O
`1
`
`In a similar manner, the position (y, X) of the composite
`image 208 is mapped to the coordinates of the telephoto
`image 206 using the equation:
`
`XT
`yT
`1
`
`= (Hrw). How
`
`XC
`
`1
`
`Then, the pixel value of the composite image at position (y,
`X) is found by interpolation. If the mapped position of (y.
`X) in the telephoto image 206 lands within the limits of the
`existing pixels (i.e. 0<=x<=C-1), the pixel value of the
`composite image 208 at position (y,x) is found by inter
`polating pixel values of the telephoto image 206. Otherwise,
`the pixel value of the composite image 208 at position (y,
`X) is found by interpolating pixel values of the wide image
`204.
`0046 Those skilled in the art will recognize that the
`above description of producing the values of the composite
`image 208 using pixel values of the wide image 204 and the
`telephoto image 206 can be accomplished in many ways. For
`example, it is easy to pre-calculate the region of pixel
`locations of the composite image 208 for which the pixel
`values will be produced by interpolating the telephoto image
`206 and the region for which the pixel values will be
`produced by interpolating the wide image 204. This saves
`computational cost but produces the same image data.
`0047 FIG. 6 shows an example set of images. The wide
`image 204 covers a wide portion of the scene and the
`telephoto image 206 covers a smaller portion of the scene,
`but with greater resolution. The produced composite image
`208 uses pixel data from the telephoto image 206 for those
`portions (i.e. the region within the dashed line 220) that are
`in the view of the telephoto image 206 and uses pixel data
`from the wide image 204 otherwise (i.e. the region outside
`the dashed line 220). The dashed line 220 shows where the
`transition is. Thus, the composite image 208 has higher
`resolution in the interior and lower resolution on the edges.
`Since the Subject of a photograph, especially in consumer
`photography, is likely to be near the center of the scene, the
`subject of the composite image 208 is likely to have the
`highest resolution. It has also been experimentally deter
`mined that the transition within the composite image 208
`between pixels derived by interpolating the wide image 204
`versus the telephoto image 206 does not product visually
`objectionable artifacts.
`0048 Since lenses 2 and 3 are separated by some dis
`tance, it is possible that objects very close to the camera will
`appear to have a discontinuity at the transition. In this case,
`it is possible to use standard image processing techniques to
`
`APPL-1011 / Page 13 of 17
`APPLE INC. v. COREPHOTONICS LTD.
`
`
`
`US 2008/0030592 A1
`
`Feb. 7, 2008
`
`find objects that are close to the camera and to process these
`regions in a fashion that does not produce a discontinuity
`artifact. For example, the pixel values of the composite
`image 208, for objects that are close to the camera and span
`the transition region, can be determined by interpolating the
`wide image 204. A true depth map can also be created and
`used by the image resampler 214 to sample the appropriate
`locations within the telephoto image 206 and the wide image
`204. In this case, the registration model is no longer a simple
`scale translation model.
`0049. A further feature of the present invention is that the
`composite image 208 can be stored on the camera without
`digital Zooming. Therefore, digital Zooming of the compos
`ite image 208 can be (lone later, during post processing, to
`create an image for use by the operator for printing or
`sharing. The composite image 208 can be formed during
`image processing on the camera or later during post pro
`cessing when the images have been offloaded from the
`camera. To enable the composite image to be formed during
`post processing, the wide image 204 and the telephoto image
`206 must both be stored at the time of image capture.
`0050. The invention can also be applied to a series of
`sequential images as in a video. Referring to FIG. 7, in the
`case of video, two sets of video images, wide video images
`220 and telephoto video images 222 are captured substan
`tially simultaneously from the two lenses 2 and 3 or 2 and
`4 and the two image sensors 12 and 14 providing video
`images from a short focal length lens 2 and a Zoom lens 3
`or a longer second focal length lens 4. The composite video
`224 is formed by combining the two sets of video images
`220 and 222. The composite video 224 can be formed during
`image processing on the camera and stored on the camera or
`the composite video 224 can be formed later during post
`processing when the images have been offloaded from the
`camera. To enable the composite video 224 to be formed
`during post processing, the wide video images 220 from the
`short focal length lens 2 and the telephoto video images 222
`from the Zoom lens 3 or the longer focal length lens 4 must
`both be stored at the time of image capture. Digital Zoom of
`the video images can be accomplished on the camera during
`capture, or on the camera after capture, or during post
`processing after the composite video 224 has been offloaded
`from the camera or during post processing when the com
`posite video 224 is being formed.
`0051. The processed digital image file is provided to a
`memory card interface 52, which stores the digital image file
`on the removable memory card 54. Removable memory
`cards 54 are one type of removable digital image storage
`medium, and are available in several different physical
`formats. For example, the removable memory card 54 can
`include (without limitation) memory cards adapted to well
`known formats, such as the Compact Flash, SmartMedia.
`Memory.Stick, MMC, SD, or XD memory card formats.
`Other types of removable digital image storage media, Such
`as magnetic hard drives, magnetic tape, or optical disks, can
`alternatively be used to store the still and motion digital
`images. Alternatively, the digital camera 10A can use inter
`nal non-volatile memory (not shown). Such as internal flash
`EPROM memory to store the processed digital image files.
`In such an embodiment, the memory card interface 52 and
`the removable memory card 54 are not needed.
`0052. The image processor 50 performs various image
`processing functions, inc