US 7,683,950 B2
`(10) Patent No.:
`a2) United States Patent
`Kelly et al.
`(45) Date of Patent:
`Mar.23, 2010
`
`
`US007683950B2
`
`(54) METHOD AND APPARATUS FOR
`CORRECTING A CHANNEL DEPENDENT
`COLOR ABERRATIONIN A DIGITAL IMAGE
`Inventors: Sean C. Kelly, Rochester, NY (US);
`Peter D. Burns, Fairport, NY (US)
`
`(75)
`
`5,565,931 A
`5,696,850 A
`5,721,694 A
`5,729,631 A
`
`10/1996 Girod
`12/1997 Parulski et al.
`2/1998 Graupe
`3/1998 Wober et al.
`
`(73) Assignee: Eastman Kodak Company,Rochester,
`NY (US)
`Subject to any disclaimer, the term ofthis
`ee 1546)by1336foe under 35
`
`(*) Notice:
`
`wo
`
`(Continued)
`FOREIGN PATENT DOCUMENTS
`WO 98/31142
`12/1997
`
`(21) Appl. No.: 11/114,841
`
`(22)
`
`Filed:
`
`Apr. 26, 2005
`
`(65)
`
`(51)
`
`Prior Publication Data
`US 2006/0239549 Al
`Oct. 26, 2006
`Int.
`Cl
`nt.
`Cl.
`(2006.01)
`HOAN 5/208
`(2006.01)
`G06K 9/40
`(52) US. CM.
`eccccccccsccesceeesseseesseeeeeeee 348/252; 382/255
`(58) Field of Classification Search ................. 348/241,
`348/252; 360/255, 260-264, 266, 269, 275
`See application file for complete search history.
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`4,418,358 A
`4,536,885 A
`4,616,252 A
`4,677,465 A
`4,937,720 A
`4,961,139 A
`4,970,593 A
`5,010,401 A
`5,047,968 A
`5,291,286 A
`5,293,258 A
`5,461,440 A
`5,561,611 A
`
`11/1983 Poetschet al.
`8/1985 Loughlin
`10/1986 Schiff
`6/1987 Alkofer
`6/1990 Kirchberg
`10/1990 Honget al.
`11/1990 Cantrell
`4/1991 Murakami etal.
`9/1991 Carrington et al.
`3/1994 Murakami et al.
`3/1994 Dattilo
`10/1995 Toyodaetal.
`10/1996 Avinash
`
`OTHER PUBLICATIONS
`
`XP000443551, M.Doyle et al., “Adaptive Fourier Threshold Filter-
`ing: A Method to Reduce Noise and Incoherent Artifacts in High
`Resolution Cardiac Images”, 8306 Magnetic Resonance in Medicine
`31 May 1999, No.5, Baltimore, Maryland, US, pp. 546-550.
`(Continued)
`
`;
`;
`;
`Primary Examiner—Timothy J Henn
`(74) Attorney, Agent, or Firm—Robert L. Walker
`
`(57)
`
`ABSTRACT
`
`A method,and digital capture apparatus for use therewith, is
`described for correcting a channel dependentcolor aberration
`in a digital image, where the digital image is composedof a
`plurality of color channels. The method includes capturing an
`image comprising the color channels, where oneofthe color
`channels is a blurred color channel due to a channel depen-
`dent color aberration affecting that channel. Then, one of the
`other color channels, other than the blurred color channel, is
`used as an indication of an aim sharpness, and the sharpness
`of the blurred color channel is adjusted, at least partially,
`toward the aim sharpness.
`
`27 Claims, 14 Drawing Sheets
`
`
`IMAGE CAPTURE SYSTEM
`{E.G., OPTICS)
`IMPARTS.
`CHANNEL DEPENDENT
`+710
`COLOR ABERRATION
`
`Jt
`CHANNEL||CHANNEL
`BLURRED
`L
`A
`B
`CHANNEL
`
`4 12
`
`
`
`
`
`PETITIONERS EX1023
`Page 1
`
`PETITIONERS EX1023
`Page 1
`
`

`

`US 7,683,950 B2
`
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`5,801,854 A
`5,900,952 A
`5,939,246 A
`6,009,209 A
`6,055,340 A *
`6,628,329 Bl
`6,728,003 Bl
`2002/0097439 Al
`2004/0047514 Al
`2004/0081364 Al*
`2004/0218071 Al
`2004/0218803 Al
`2004/0240750 Al
`2004/0247195 Al
`2004/0247201 Al
`2006/0093234 Al*
`2006/0181620 Al*
`
`9/1998 Naylor, Jr.
`5/1999 Fan
`8/1999 Breweret al.
`12/1999 Ackeret al.
`4/2000 Nagao oe eeeeeeeeeeee 382/261
`9/2003 Kelly etal.
`4/2004 Gallagheret al.
`7/2002 Braica
`3/2004 Gallagher etal.
`4/2004 Murphy «0.0.0... eee 382/260
`11/2004 Chauville et al.
`11/2004 Chanaset al.
`12/2004 Chauville et al.
`12/2004 Chauville et al.
`12/2004 Arazaki
`5/2006 Silverstein ...........0..... 382/255
`8/2006 Kimbell ........0000000.. 348/241
`
`OTHER PUBLICATIONS
`
`XP002146062, B Aiazzi et al., “A Robust Method For Parameter
`Estimation of Signal-Dependent Noise Models in Digital Images”,
`0-7803-4137-Jun. 1997, IEEE, pp. 601-604.
`
`XP000225283, Aggelos K. Katsaggelos et al., “A Regularized Itera-
`tive Image Restoration Algorithm”, IEEE Transactions on Signal
`Processing, vol. 39, No. 4, Apr. 1991, pp. 914-929.
`XP000780652, Til Aach et al., “Anisotropic Spectral Magnitude Esti-
`mation Filters For Noise Reduction and Image Enhancement”,
`0-7803-3258-X96, 1996, IEEE, Philips GmbH Research Laborato-
`ties, Weisshausstr.2, D-52066 Aachen, Germany, pp. 335-338.
`XP002146454, Rangaraj M. Rangayyan et al., “Adaptive-neighbor-
`hoodFiltering of Images Corrupted By Signal-Dependent Noise”,
`0003-6935/98/204477-11, Jul. 10, 1998, vol. 37, No. 20, Applied
`Optics, 1998 Optical Society of America, Department of Electrical
`and Computer Engineering, University of Calgary, 2500 University
`Drive, N.W. Calgary, Alberta T2N 1N4, Canada, pp. 4477-4487.
`XP000896196, Mei Yuet al., “New Adaptive Vector Filter Based on
`Noise Estimate”, IEICE Transactions Fundamentals vol. E-82A No.
`6, Jun. 1999, Paper, Sepcial Section of Papers Selected from ITC-
`CSCC 98, pp. 911-919.
`XP000280610, J.N. Lin et al., “2-D Adaptive Volterra Filter For 2-D
`Nonlinear Channel Equalization and Image Restoration”, 8030 Elec-
`tronic Letters, vol. 28, No. 2, Jan. 16, 1992, Stevenage, Herts, pp.
`180-182.
`
`* cited by examiner
`
`PETITIONERS EX1023
`Page 2
`
`PETITIONERS EX1023
`Page 2
`
`

`

`U.S. Patent
`
`Mar.23, 2010
`
`Sheet 1 of 14
`
`US 7,683,950 B2
`
` IMAGE CAPTURE SYSTEM
`(E.G., OPTICS) IMPARTS
`CHANNEL DEPENDENT
`COLOR ABERRATION
`
`10
`
`CHANNEL||CHANNEL||cyANNEL
`
`|capruneCHANNELSA,B,CCHANNELSA,B, C 12
`
`=peBeBLURRED
`
`
`
`
`
`
`
`USE
`
`CHANNEL
`
`BFOR
`AIM
`
`SHARPNESS
`
`
`
`ADJUST
`
`SHARPNESSOF|_146
`
`CHANNELC
`
`TOWARDAIM
`
`
`
`
`COMBINE COLOR CHANNELS
`
`
`
`18
`
`FIG. 1
`
`PETITIONERS EX1023
`Page 3
`
`PETITIONERS EX1023
`Page 3
`
`

`

`U.S. Patent
`
`Mar.23, 2010
`
`Sheet 2 of 14
`
`US 7,683,950 B2
`
`DETERMINE MTF
`OF AIM SHARPNESS
`CHANNEL B
`
`CHANNEL C
`
`DETERMINE MTF
`OF BLURRED
`CHANNEL C
`
`DETERMINE RATIO:
`
`MTF OF CHANNEL B
`MTF OF CHANNEL C
`
`GENERATEFILTER
`FROM RATIO
`(FREQ. RESPONSE)
`
`APPLY FILTER TO
`
`FIG. 2
`
`PETITIONERS EX1023
`Page 4
`
`PETITIONERS EX1023
`Page 4
`
`

`

`U.S. Patent
`
`Mar.23, 2010
`
`Sheet 3 of 14
`
`US 7,683,950 B2
`
`
`
`OGREENCHANNEL4REDCHANNEL©BLUE
`CHANNEL
`
`FIG.3
`
`CYCLES/SAMPLE
`
`PETITIONERS EX1023
`Page 5
`
`PETITIONERS EX1023
`Page 5
`
`

`

`U.S. Patent
`
`Mar.23, 2010
`
`Sheet 4 of 14
`
`US 7,683,950 B2
`
`G
`
`<
`©
`
`=
`<<
`“
`3 ©
`a oO Ww
`o>
`
`my
`
`$8
`QS
`
`‘ Y >“
`
`J
`QO
`“
`
`y
`
`PETITIONERS EX1023
`Page 6
`
`_
`S
`
`oO
`
`0.5
`
`oO
`
`»
`
`y ~
`
`»
`
`Pind
`
`-
`
`S
`
`D>
`
`Os
`eS
`
`oY
`
`a
`
`QD
`
`VR
`
`OY
`
`ay
`
`wt
`
`3.5
`
`PETITIONERS EX1023
`Page 6
`
`

`

`U.S. Patent
`
`
`
`TANSNOILMIOANOD
`
`‘970Z00'0'81S820'0-'8Z8790'0-'S9Z1.20'0-'828790'0-'81S8Z0'0-'9r0Z00'0
`
`
`
`‘8LS8Z0'0-'826180'0-'LO9EP0'0-'S29910'0-'L09EH0'0-'8Z6180°0-'8-S8Z0'0-
`
`
`
`
`
`
`
`‘BL8790'0-LO9EP0'0-'BS9Z1LL'0'LZ81722'0'8S97Z1LL'0'LO9EP0'0-'8Z8P790'0-
`
`Mar. 23, 2010
`
`
`
`'G9Z1L.20°0-'S29910'0-'2281220S9SEZr'L'L£781-22'0'SL991L0'0-'S921.20'0-
`
`
`
`
`
`
`
`'8Z8790'0-LO9E70'0-'8S9Z11'0'2Z8122'08597110LO9EH0'0-'828790'0-
`
`
`
`
`
`Sheet 5 of 14
`
`
`
`'81S8Z0'0-'826180'-'L09E70'0-'S29910°0-"L09E70'0-'8Z6180'0-'8S8Z0'0-
`
`
`
`970200'0'81S8Z0'0-'8Z8790°0-'S9Z120'0-'8Z890'0-'81S8Z0'0-'970200'0
`
`US 7,683,950 B2
`
`SOld
`
`PETITIONERS EX1023
`Page 7
`
`PETITIONERS EX1023
`Page 7
`
`

`

`U.S. Patent
`
`Mar. 23, 2010
`
`Sheet 6 of 14
`
`US 7,683,950 B2
`
`WINOZIMOH
`
`WOILYSA
`
`WIV
`
`LXZAZISHSLC109APYALWOUIEy
`
`S'0 Sv'0
`vose0€0S2020StO4190
`
`S00
`
`ASNOdS3Y
`
`AODNSNDAYS
`
`9‘S/d
`
`PETITIONERS EX1023
`Page 8
`
`PETITIONERS EX1023
`Page 8
`
`

`

`U.S. Patent
`
`Mar. 23, 2010
`
`Sheet 7 of 14
`
`US 7,683,950 B2
`
`4dy¥Os4Aa
`
`ti ernenShoe baa a
`Soa
`cae hoes
`a
`te
`oe
`ee
`eeSa
`
`PETITIONERS EX1023
`Page 9
`
`PETITIONERS EX1023
`Page 9
`
`
`

`

`U.S. Patent
`
`oo]
`
`US 7,683,950 B2
`
`
`
`LAaecopePll
`ATTRAEeSFAOTBeTdtakeAtihahehehALAS
`
`
` —YeomeaeSTis=DNRiteiaisanidetkbhierd
` Oo=VITOTUNEafetentainaetd
`
`«enN=ieneneed2Beeeaoeeelod>eneeee
`wt=Sooperermrrrerreesrt
`
`YyaLlsvaYOs3¢d
`
`8Sl
`
`Nya}
`
`Taek
`
`PETITIONERS EX1023
`Page 10
`
`PETITIONERS EX1023
`Page 10
`
`
`
`
`

`

`U.S. Patent
`
`Mar. 23, 2010
`
`Sheet 9 of 14
`
`US 7,683,950 B2
`
`
`
`
`
` COMPUTE MTFs
`
`FOR DIFFERENT
`LENS POSITIONS
`
`
`FOR LENS POSITIONS
`
`
`
`COMPUTE AN
`MTF CORRECTION
`
`
`
`INTERPOLATE MTF {
`' CORRECTIONS FOR +——54
`!
`ADDITIONAL
`;
`!
`LENS POSITIONS
`|
`
` '
`
` GENERATEFILTER
`
`THAT VARIES WITH
`LENS POSITION
`
`
`
`
`
`APPLY FILTER TO
`BLURRED CHANNEL
`
`PER LENS POSITION
`
`
`FIG. 9
`
`PETITIONERS EX1023
`Page 11
`
`PETITIONERS EX1023
`Page 11
`
`

`

`U.S. Patent
`
`Mar. 23, 2010
`
`Sheet 10 of 14
`
`US 7,683,950 B2
`
`dalsiqow
`
`WLISIG
`
`AVI
`
`“ONDIN
`
`YOSS300Ud
`
`Of}
`
`GayOls
`
`ONINSACYVHS
`
`TANYA
`
`OF‘Sila
`
`AXOWAINCv
`
`YALYSANOD
`YOSS39D0ud
`
`SOIWNY
`
`TWNOIS
`
`belcol
`
`Ol
`
`ch
`
`ASV
`
`YOSNAS
`
`YAaLLNHS(|}OL}
`
`8LbLbL
`
`PETITIONERS EX1023
`Page 12
`
`PETITIONERS EX1023
`Page 12
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Mar.23, 2010
`
`Sheet 11 of 14
`
`US 7,683,950 B2
`
`FIG.11
`
`PETITIONERS EX1023
`Page 13
`
`PETITIONERS EX1023
`Page 13
`
`

`

`U.S. Patent
`
`Mar. 23, 2010
`
`Sheet 12 of 14
`
`US 7,683,950 B2
`
`AYOWSW
`
`“OYSIN
`
`YOSS3II0Nd
`
`anv
`
`YALYSANOD
`yOSS3A00Nd
`
`SOTWNV
`
`WNOIS
`
`belCobOctchtObl
`
`Y3SLLAHS(|}SbbLZbb
`
`LSO008
`
`dV
`
`Qay0LSQZ
`
`ONINSdYeVHS
`
`TANYSy
`
`Qay¥OLSOf}
`
`Ob‘Old
`
`PETITIONERS EX1023
`Page 14
`
`PETITIONERS EX1023
`Page 14
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Mar. 23, 2010
`
`Sheet 13 of 14
`
`US 7,683,950 B2
`
`SEL
`
`1LSOH
`
`YaLNdNOd
`
`JOVSYSLNI
`
`cel
`
`gaisIgoW
`
`9El
`
`WLISI
`av
`YOSS300ud
`
`“OYDIN
`
`WALID
`
`YOSS300ud
`
`vr
`
`LSNY3SLNI
`
`iLL
`
`YALYSANOD
`yOossao00dd
`
`WNDIS
`
`YOSNAS
`
`AXOWSWNGN
`
`OvLYALNAWODLSOH=gg,
`velcob
`
`YSLNdWODLSOH
`SO1WNYaoviNI
`OZLohL
`
`Vivdvlaw
`
`3a¥OLS
`
`cPh.
`
`G3aYyOLS
`
`ONINSdYVHS
`
`TANYSx
`
`OcL
`
`cb
`‘Old
`
`ObL
`
`YAaLLNHS
`)BLLLbb
`
`PETITIONERS EX1023
`Page 15
`
`PETITIONERS EX1023
`Page 15
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Mar. 23, 2010
`
`Sheet 14 of 14
`
`US 7,683,950 B2
`
`YOSS3z00dd
`
`AVI
`
`YVINTISS
`
`W3Q0W
`
`002002
`
`OldLhe
`
`8Lc82
`
`90¢
`
`COOO|
`
`OCO
`
`yVINTISS
`
`YyOSss00Nd
`
`Q3yOLS
`
`ONINAdYVHS
`
`TANYSy
`
`Gb‘OldVelSid
`
`PETITIONERS EX1023
`Page 16
`
`PETITIONERS EX1023
`Page 16
`
`
`
`

`

`US 7,683,950 B2
`
`1
`METHOD AND APPARATUS FOR
`CORRECTING A CHANNEL DEPENDENT
`COLOR ABERRATIONIN A DIGITAL IMAGE
`
`FIELD OF THE INVENTION
`
`The present invention relates generally to thefield of pho-
`tography, and in particular to imaging systems producing
`electronically derived images that have channel dependent
`color aberrations, such as longitudinal colorartifacts or aber-
`rations in captured images.
`
`BACKGROUNDOF THE INVENTION
`
`Imaging apparatus, such as photographic film cameras and
`electronic cameras, andin particular their optical assemblies,
`have inherent aberrations which can degrade the quality of
`images captured by such apparatus. One kindofaberration is
`a distortion, which refers to a change in the geometric repre-
`sentation of an object in the image plane. For instance, a
`rectangle might be reproduced with a pincushionor a barrel
`shape—hencethe reference to pincushiondistortion or barrel
`distortion. Another type of aberration, referred to as chro-
`matic aberration, results from the fact that different wave-
`lengths or colors of light are refracted by different amounts by
`an optical assembly. A further type of aberration is a field
`dependentaberration, where somecharacteristic, such as the
`brightness, of an imagepixel is changed in the image plane in
`proportionto its position in the field, such as its distance from
`the center of the image.
`Chromatic aberration appears whena lens is transmitting
`polychromatic light (many colors). Since the indexofrefrac-
`tion of optical glass is wavelength dependent, the red, green
`and blue components bend differently at an optical interface
`in the lens. This leads to longitudinal (axial) and/orlateral
`chromatic aberration effects. When a lens fails to focus vari-
`ous colors sharply in the sameplane,the lensis said to exhibit
`longitudinal (axial) chromatic aberration. In longitudinal
`chromatic aberration, the three components are brought to
`focus on different planes in the image space, which gives a
`color blurring effect. Thus, longitudinal chromatic aberration
`arises due to the focal length varying with wavelength (color).
`In lateral chromatic aberration, color components from a
`single point are brought to focus to different points on the
`same image plane, resulting in a lateral shift of the image.
`This has the effect of magnifying the three colors differently
`and can be visually seen as color fringing. Thus lateral chro-
`matic aberration can be seen asan effect due to magnification
`varying with wavelength.
`A great deal of the complexity of modern lenses is due to
`efforts on the part of optical designers to reduce optical aber-
`rations. In certain cases, such as with single use film cameras
`or inexpensive digital cameras, it may be economically diffi-
`cult to avoid usage of inexpensive optics. Unfortunately, as
`explained above, such optics possess inherentaberrationsthat
`degrade the quality of images formed by the optics. Conse-
`quently, it is desirable to compensate for these aberrations in
`the reproduction process (either in the capture device or ina
`host computer) so that final imagesfree of aberrations may be
`obtained. In order to characterize these aberrations, the ability
`of a lens to transfer information from the object to an image
`plane is represented as a modulation transfer function (MTF).
`A lens MTFis a measure of how well the original frequency-
`dependent contrast of the object is transferred to the image.
`Ina typical camera, in addition to distortion and chromatic
`aberrations, the image formedat a focal plane (where thefilm
`or image sensor is located) can be blurred as a function of
`
`2
`proximity to the optical axis ofthe optical assembly. For such
`field dependentaberrations, the further away from the optical
`axis (normally, the center ofthe image), the more the imageis
`blurred. The resultant image therefore has an MTFthatis a
`function of radial distance from the center of the image. The
`problem is exaggerated with images originating from inex-
`pensive cameras, such as single use film cameras. Because of
`their simple optics or becausethe film may notbe located in
`the position of best focus throughout the focal plane, single
`use film cameras tend to have significant sharpness loss with
`movement away from the optical axis toward the edges ofthe
`frame. Consequently, it is also desirable to compensate for
`these aberrations in the reproduction process (either in the
`capture device or in a host computer) so that final images free
`of field dependentaberrations may be obtained.
`In one example, a camera system described in U.S. Pat. No.
`5,461,440, entitled “Photographing Image Correction Sys-
`tem”and issued Oct. 24, 1995 in the names of Toyodaet al.,
`does not require an expensive optical assembly that is cor-
`rected for marginal attenuation (light amount irregularity)
`and distortion (pincushion andbarreldistortion). Instead, the
`curvature of field data and the light amountirregularity data
`corresponding to the optical assembly are identified in
`advance, and stored either in the camera or separately at a
`downstream scanning and processing station. Either way, the
`data is linked to the specific camera and then used in subse-
`quentfilm processing and scanningto correct the imagesig-
`nal for the image quality degradation imparted by the optical
`assembly.
`The image quality of captured images can be improved by
`the selection of appropriate filters for the input imaging
`device and subsequent devices that process the captured
`images. For instance, in U.S. Pat. No. 4,970,593, entitled
`“Video Image Enhancement Utilizing a Two-dimensional
`Digital Aperture Correction Filter” and issued Nov. 13, 1990
`in the name of C. Cantrell, the modulation transfer function
`(MTF)of the uncorrected optical system is measured and an
`aperture correction function is created from an inverse of the
`MTFfunction to correct an image captured through the opti-
`cal system. In commonly-assigned U.S. Pat. No. 5,696,850,
`entitled “Automatic Image Sharpening in an Electronic Imag-
`ing System”and issued Dec. 9, 1997 in the names of Kenneth
`Parulski and Michael Axman,a digital image produced by a
`digital camera is improvedby using a sharpeningfilter thatis
`produced as a function of the system MTF. Although these
`arrangements produce an improved image, there are still
`problemswith image quality. For example, the imagecanstill
`suffer from position dependent blur and channel dependent
`blur.
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`Commonly assigned U.S. Pat. No. 6,628,329, entitled
`“Correction of Position Dependent Blur in a Digital Image”
`and issued Sep. 30, 2003 in the names of Sean C. Kelly,
`Donald Williams and David Jasinski, describes the correction
`of position dependent blur in a digital camera, where the
`position dependenceis a function of the proximity of a pixel
`to the optical axis. Typically, the camera manufacturer mea-
`sures the MTFat various locations in the image, and then
`creates a boost map that is applied to a sharpening kernal to
`adjustfor position blur ofthe captured image. The boost value
`at each of the pixels of the imageis inversely proportional to
`the actual MTF,i.e., equal to a desired MTFvalue divided by
`the actual MTF value for that pixel. It is desirable that this
`technique be used to spatially equalize the sharpness, to cor-
`rect for lens sharpnessroll off. This techniqueis also useful in
`purposefully modifying the local MTF to some different aim
`(either inducing local blur or enhanced sharpness).
`PETITIONERS EX1023
`Page 17
`
`PETITIONERS EX1023
`Page 17
`
`

`

`US 7,683,950 B2
`
`3
`Some aberrations, specifically chromatic aberrations, are
`channel dependent aberrations in the sense that each color
`channel, e.g., red, green and blue channels, providesa differ-
`ent amountofthe aberration artifact in the imageplane. It has
`also been observed that some field dependent aberrations,
`such as position dependentblur, are also channel dependent.
`Consequently, a different amountof correction wouldideally
`be provided for each color channel at the image plane. For
`instance, lens designers typically provide complicated, and
`therefore expensive, designsto differentially contral the light
`rays according to wavelength in order to minimize such arti-
`facts.
`
`Especially if they are intended for consumeruse, digital
`cameras, which are inherently more complex and expensive
`than simple film cameras, such as single use film cameras,
`must control cost in any way possible. The camera optics is a
`typical candidate for cost reduction, and channel-dependent
`artifacts thus becomesa concern. Despite such image quality
`concerns, it is usually desirable to provide a finished image
`file that is corrected for camera-related influences. What is
`neededis a simple correction for channel dependentaberra-
`tions, such as channel dependentblur and sharpness fall-off,
`that does not require a more complex, or more expensive,
`optical system, as well as a correction that can be imple-
`mented in the processor of a digital camera, or in the down-
`stream scanning and processing ofa film system. More spe-
`cifically, a simple correction is neededfor the kind of channel
`dependentblurring caused by longitudinal chromatic aberra-
`tion and field dependenteffects.
`Channel dependent corrections for a printing process are
`addressed in commonly assigned U.S. Pat. No. 6,728,003,
`entitled “Method of Compensating for MTF in a Digital
`Image Channel,” and issued Apr. 27, 2004 in the names of
`Andrew Gallagher and Robert Parada. In this patent, a digital
`image comprises a plurality of digital image channels, suchas
`red, green and blue channels. A degradation in the MTF of a
`device in an imaging chain is compensated by using the MTF
`and a gain factor to provide an aim response, generating a
`filter from the aim response, and then using the filter to
`process the image channel. In published U.S. Patent Appli-
`cation 2004/0218071, entitled “Method and System for Cor-
`recting the Chromatic Aberrations of a Color Image Produced
`by Meansof an Optical System” and published Nov. 4, 2004
`in the names of Chauville et al., a system and method is
`described for correcting the chromatic aberrations ofa digital
`image composedofa plurality of color planes. The geometric
`anomalies, especiallydistortions, ofthe digitized color planes
`are modeled andcorrected, at least partly, in such a wayas to
`obtain corrected color planes. The corrected color planes are
`then combined in such a manneras to obtain a color image
`corrected completely or partly for the distortion-based chro-
`matic aberrations. Neither Gallagheret al. nor Chauvilleetal.
`address longitudinal chromatic aberrationsorfield dependent
`artifacts.
`
`What is therefore needed is a method for removing the
`aberration of longitudinalcolor in captured digital images. In
`particular, there is need for a restoration algorithm that can
`remove the negative imaging artifacts associated with the
`aberration of longitudinal color. These negative artifacts
`include color fringing wherein the three color planes do not
`line up on top of each other, and sub-optimal sharpnessin at
`least one ofthe color channels.
`
`SUMMARYOF THE INVENTION
`
`The object of the invention is to remove a channel depen-
`dent color aberration from a digital image.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`A further object ofthe invention 1s to remove a longitudinal
`color aberration from a captured digital image.
`A further object ofthe inventionis to remove a channel and
`field dependent color aberration from a captured digital
`image.
`A further object of the invention is to remove a channel
`dependentcolor aberration from a digital image created by a
`zoom lens system.
`The present invention is directed to overcoming one or
`more of the problemsset forth above. Briefly summarized,
`according to one aspectof the invention, the invention com-
`prises a method for correcting a channel dependent color
`aberration in a digital image, where the digital image is com-
`posed of a plurality of color channels. The method comprises
`the steps of capturing an image comprising the color chan-
`nels, where one of the color channels is a blurred color chan-
`nel due to a channel dependentcoloraberration affecting that
`channel; using another color channel, other than the blurred
`color channel, as an indication of an aim sharpness; and
`adjusting the sharpness of the blurred color channel, at least
`partially, toward the aim sharpness.
`In a further aspect of the invention, the channel dependent
`color aberration is a longitudinal color aberration,the blurred
`color channelis a blue channel, and the other color channel
`used as an indication of an aim sharpness is a green color
`channel.
`
`In yet a further aspectofthe invention,the step of adjusting
`the sharpness ofthe blurred channel toward the aim sharpness
`comprises determining the MTFofthe blurred color channel
`and the MTFof the other color channel used as an indication
`
`of the aim sharpness; determining a ratio of the MTFofthe
`other color channel to the MTFofthe blurred color channel;
`using the ratio to generate a filter, wherein theratio is the aim
`frequency responseofthefilter; and applying thefilter to the
`blurred color channel to adjust the sharpness of the blurred
`color channel, at least partially, toward the aim sharpness.
`In yet a further aspect of the invention, where the optical
`system is a zoom system with a plurality of zoom lens posi-
`tions and the MTFofthe optical system for the color channels
`varies for different lens positions, the method further com-
`prises the steps of computing an MTFcorrection as a ratio of
`the MTF ofthe other color channel to the MTFof the blue
`
`color channelfor at least someof the lens positions; using the
`computed corrections to generate a filler that varies its filter-
`ing effect with lens position; and applyingthefilter to the blue
`color channel to equalize the sharpness of the blue color
`channel, at least partially, to the aim sharpness dependent
`upon the zoom lensposition.
`The advantageofthis inventionis that it allows design and
`use of a significantly lower cost lens, since a color channel
`dependentaberration (such as a longitudinalcolor aberration)
`can be allowed into the design and later corrected by the
`image processing method according to the invention. With
`such a lens design, a camera is able to have an inexpensive
`lens while maintaining acceptable image quality and at the
`same time reducing fringing and blurring through subsequent
`image processing. This improvement in image processing is
`particularly useful with inexpensive digital cameras, and
`especially with camera phones, which mustbe extremely low
`cost and may have compromisedlens designs.
`These and other aspects, objects, features and advantages
`of the present invention will be more clearly understood and
`appreciated from a review ofthe following detailed descrip-
`PETITIONERS EX1023
`Page 18
`
`PETITIONERS EX1023
`Page 18
`
`

`

`US 7,683,950 B2
`
`5
`tion of the preferred embodiments and appendedclaims, and
`by reference to the accompanying drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG.1 is a flow diagram showing the methodfor correcting
`a channel dependent color aberration according to the inven-
`tion.
`FIG. 2 is a flow diagram showing further details of a sharp-
`ness correction of a color dependentblur dueto a longitudinal
`color aberration, as generally shownin FIG.1.
`FIG. 3 shows the modulation transfer function (MTF) of
`the three color channels, red, green and blue, typically pro-
`cessed according to the invention.
`FIG. 4 showsthe ratio of the green MTF to the blue MTF.
`FIG. 5 shows an example of a least squares 7x7 convolu-
`tion kernel that achieves an adequate approximation of the
`filter frequency response, as characterized by the ratio shown
`in FIG.4, that is required to equalize the sharpness ofthe blue
`channelto the green.
`FIG. 6 showsthe frequency response aim andactualfilter
`frequency response ofthe filter characterized by the convo-
`lution kernel shownin FIG. 5.
`
`FIG. 7 showsbefore and after examples of an image sub-
`section, with the technique of FIG. 1 applied accordingto the
`invention.
`
`FIG. 8 shows before and after examples of the blue image
`plane ofthe image subsection shownin FIG.7, showing more
`obviously the blue intra-channel blur and its correction
`according to the invention.
`FIG. 9 is a flow diagram showing details of a sharpness
`correction of a color dependent blur due to a longitudinal
`color aberration that is dependent upon a zoomlensposition.
`FIG. 10 is a block diagram of a digital camera having an
`arrangement for applying a channel dependent sharpening
`kernel to the blurred blue color channel in accordance with
`the present invention.
`FIG. 11 is a diagram of an exemplary test target having
`multiple edges, which can be used to determine the system
`MTF.
`
`FIG. 12 is a block diagram of a digital camera having a
`general arrangement for modifying a channel dependent
`sharpening kernel by applying a boost mapto the sharpening
`kernel to accountfor a field dependent aberration.
`FIG. 13 is a block diagram ofa digital capture and process-
`ing system having a digital camera for capturing a digital
`imagethat is subsequently processed by a host computer for
`applying a channel dependentcorrection to the blurred color
`channelin accordance with the present invention.
`FIGS. 14A and 14B are perspective viewsofthe front and
`back of a cell phone including a camera having an arrange-
`ment for applying a sharpening kernel to a blurred color
`channel in accordance with the present invention.
`
`20
`
`30
`
`35
`
`40
`
`45
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`55
`
`Because digital cameras employing imaging devices and
`related circuitry for signal capture and correction are well
`known,the present description will be directed in particular to
`elements forming part of, or cooperating more directly with,
`method and apparatus in accordance with the present inven-
`tion. Elements not specifically shown or described herein
`maybeselected from those knownin theart. Certain aspects
`of the embodiments to be described may be provided in
`software. Given the system as shown anddescribed according
`to the invention in the following materials, software not spe-
`cifically shown, described or suggested herein thatis useful
`
`6
`for implementation of the invention is conventional and
`within the ordinary skill in sucharts.
`Oneof the most important characteristics of an electronic
`imaging system is the ability of its imaging device to capture
`fine detail found in an original scene. This ability to resolve
`detail is determined by a numberof factors, including the
`performanceofthe optical system, the numberof addressable
`photo elements in the optical imaging device, andthe electri-
`cal circuits in the camera, which may include image compres-
`sion and gammacorrection functions. Different measurement
`methods can provide different metrics to quantify the resolu-
`tion of an imaging system, or a component of an imaging
`system, such as a lens. Resolution measurement metrics
`include resolving power, limiting resolution (at some speci-
`fied contrast), modulation transfer function (MTF), and opti-
`cal transfer function (OTF). Mathematically, the modulation
`transfer function is the modulusofthe optical transfer func-
`tion, which is the two-dimensional Fourier transform of the
`point spread function of the imaging system under consider-
`ation. The OTF is a complex function whose modulus (MTF)
`has the value unity at zero spatial frequency. Although the
`focus in this application is on use of the modulation transfer
`function to characterize the resolution of the capture and
`output devices, other metrics could be used, for example the
`OTF,spatial frequency response or depth ofmodulation level
`at various spatial frequencies. These are all various forms of
`spatial transfer functions that can be used to characterize the
`sharpness of an image from an imaging device.
`The advantage ofthe spatial transfer functions is that they
`provide information about image quality over a range of
`frequencies rather than just at the limiting frequency as does
`resolving power. Moreparticularly, the modulation transfer
`function is a graph(i.e., a set of discrete modulation factors)
`that represents the image contrastrelative to the object con-
`trast on the vertical axis over a range of spatial frequencies on
`the horizontal axis, where high frequency corresponds to
`small detail in an object. If it were possible to produce a
`facsimile image, the contrast of the image would be the same
`as the contrast of the object at all frequencies, and the MTF
`would bea straight horizontalline at a level of 1.0. In practice,
`the lines always slope downward to the right, since image
`contrast decreases as the spatial frequency increases. Even-
`tually the lines reach the baseline, representing zero contrast,
`whenthe image-forming system is no longer able to detect the
`luminance variations in the object. The MTF can be deter-
`minedfor each componentin an image-forming system orfor
`combinations of components. The MTFcan also be deter-
`mined for each color component being imaged, such as red,
`green and blue, in a given image plane in an image-forming
`system. The MTFfor a system can be calculated by multiply-
`ing the modulation factors of the components at each spatial
`frequency. Since the MTF curvesofall of the devices in a
`system are multiplied together point by point to provide the
`system MTFcurve, the system curve is also a downwardly
`sloping function diminishing to zero resolution as the spatial
`frequency increases.
`This downwardly sloping characteristic results in a gradual
`loss of contrastin the detail ofthe imageas the detail becomes
`finer and finer. For example, all optical devices have a non-
`ideal MTF response curve because of the finite size of the
`optical aperture associated therewith. The MTF curve of such
`optical devices is normally a monotonically decreasing func-
`tion such as a downwardly sloping diagonalline,1.e., a set of
`diminishing modulation factors, that intersects the spatial
`frequencyaxis at a point of frequencyless than or equal to the
`diffraction limit—the point at which contrast or resolution
`diminishes to zero. A filter can be designed with a transfer
`PETITIONERS EX1023
`Page 19
`
`PETITIONERS EX1023
`Page 19
`
`

`

`US 7,683,950 B2
`
`7
`function to compensate forthe diffraction effects of the finite
`size of the optical aperture of the system.If the filter curve is
`the inverse ofthe system MTFcurve, the composite curve will
`be substantially flat out to the diffraction limit. Thefilter thus
`boosts the high spatial frequency contrast to compensate for
`the downwardly sloping characteristic of the system MTF.
`As mentioned above, a chromatic aberration results from
`the fact that different wavelengths or colors of light are
`refracted by different amounts by an optical assembly. In a
`longitudinal chromatic aberration, the three components are
`brought to focus ondifferent planes in the image space, which
`gives a colorblurring effect. In other words, axial chromatic
`aberration arises due to the focal length varying with wave-
`length (color). A further type ofaberration is a field dependent
`aberration, where somecharacteristic, such as the brightness,
`of an imagepixel is changedin the imageplanein proportion
`to its position in thefield, such asits distanc