`US 6,862,373 B2
`(10) Patent No.:
`Mar. 1, 2005
`(45) Date of Patent:
`Enomoto
`
`US006862373B2
`
`(54)
`
`IMAGE PROCESSING METHOD AND
`APPARATUS
`
`6/2001 Norita et al... 356/337
`6,243,165 Bl *
`6,476,869 B1 * 11/2002 Sekine et al. 0.0.0.0... 348/335
`
`(75)
`
`Inventor:
`
`Jun Enomoto, Kanagawa (JP)
`
`(73) Assignee: Fuji Photo Film Co., Ltd., Kanagawa
`(JP)
`
`(*) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`US.C. 154(b) by 2 days.
`
`(21) Appl. No.: 10/201,447
`
`FOREIGN PATENT DOCUMENTS
`
`JP
`JP
`
`6-311425
`9-281613
`
`11/1994
`10/1997
`
`* cited by examiner
`
`Primary Examiner—Thomas D Lee
`Assistant Examiner—Stephen Brinich
`(74) Attorney, Agent, or Firm—Sughrue Mion, PLLC
`
`(57)
`
`ABSTRACT
`
`(22)
`
`(65)
`
`Filed:
`
`Jul. 25, 2002
`
`Prior Publication Data
`US 2002/0196472 Al Dec. 26, 2002
`
`Related U.S. Application Data
`
`
`
`The image processing method and apparatus acquire input
`image data from the image recorded optically with a taking
`lens, acquire the information about the lens used to record
`the image and perform image processing schemes on the
`input image data using the acquired lens information, pro-
`vided that the type of the lens used is identified from the
`acquired lens information and the intensity of sharpness
`(62) Division of application No. 09/301,538, filed on Apr. 29,
`enhancementof the corresponding imageis altered in accor-
`1999, now Pat. No. 6,603,885.
`dance with the identified lens type. The characteristics of the
`taking lens may also be acquired from the lens information
`Foreign Application Priority Data
`(30)
`and using the obtained lens characteristics as well as the
`Apr. 30, 1998
`(IP)
`vessessesssessesssessssssesssesssssseseees 10-121263
`position information for the recorded image, the input image
`Aug. 27, 1998
`(JP)
`...
`. 10-241495
`data is subjected to aberration correction for correcting the
`Sep. 16, 1998
`(JP)
`...
`. 10-261304
`deterioration in image quality due to the lens characteristics,
`Sep. 16,1998
`GIP)
`...
`. 10-261305
`provided that the information about the focal length of the
`Sep. 30, 1998
`(IP) woe eee cee ceesseeeecnneecnnaeeenes 10-276679
`taking lens effective at
`the time of image recording is
`(SU)
`Inte C17 aa cceccseescesseestesteneesssessn HON 5/225
`additionally used, or image processing schemesare per-
`(52) US. Che cccccecnsscnesseecnesenenees 382/263; 382/260
`formed in two crossed directions of the recorded image, or
`58) Field of Search
`382/260-269
`
`(58)FieldofSearch«ose 382/276 parameters for correcting aberrations in the imaging plane of
`
`/
`the taking lens are scaled such that they are related to the
`(56)
`References Cited
`output image data on a pixel basis. High-quality prints
`reproducing high-quality images can be obtained from origi-
`U.S. PATENT DOCUMENTS
`nal images that were recorded with compact cameras, digital
`250/550
`4806,774 A *
`2/1980 Linet al
`cameras and other conventional inexpensive cameras using
`5,103,490 A *
`4/1992 McMillin .....cssesssssse 382/284
`[OW-Performancelenses.
`5,956,184 A *
`9/1999 Sato w..ceceecccseeeceeee 359/683
`6,097,430 A *
`8/2000 Komiya et al.
`.......... 348/218.1
`
`12
`
`16 Claims, 24 Drawing Sheets
`
`0
`SCANNER
`
`640
`ad
`
`
`
`DATA
`.
`PRESCAN a ‘|—crOTHER IMAGE]|°&_Tata<i
`
`[Eee me| ts]pon/
`|
`64 PROCESSING
`
`52
`SUBSECTION
`paescANNED Pa,
`18
`12
`DATA
`LOG
`PROCESSING
`(MAGE
`
`:
`74
`
`‘ Seon|Aces
`
`
`LENS TYPE|:
`CONDITION
`
`SETTING
`IDENTIFYING
`|
`if]
`SUBSECTION
`|
`SECTION 62°
`
`
`20
`
`
`
`
`
` A6
`
`56 ABERRATION|,[OTHER IMAGE
`
`PRINTER
`|:
`FINE
`“ISCAN
`
`CORRECTING||PROCESSING
`CONVERTING +!
`MEMORY| |
`SUBSECTION||
`PORTION_|[BLOCK
`
`“GOFINE SCANNED
`DATA PROCESSING
`SECTION
`
`50
`
`68IMAGE
`PROCESSING
`SUBSECTION
`
`)
`68A
`
`PETITIONERS EX1024
`Page 1
`
`PETITIONERS EX1024
`Page 1
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 1 of 24
`
`US 6,862,373 B2
`
` 34
`
`32
`
`fo
`Ay
`fT
`
`fie" 30
`
`i: p==[- an CONV
`
`(
`
`/D
`
`A
`
`ERTER |
`
`36
`
`14
`
`16
`
`L
`PROCESSING
`aK)
`
`IMAGE
`
`APPARATUS
`
`6
`
`DISPLAY
`
`20
`
`26
`
`
`
`
`
`RECORDING|
`MEDIUM
`DRIVE
`
`
`UNIT
`
`
`|:
`KEY.
`:
`(ole)
`|
`:
`| cohopo
`18 18a
`!
`18
`MANIPULATING UNIT
`
`
`
`
` 29
`IMAGE
`
`RECORDING
`
`MEDIUM
`
`
`PETITIONERS EX1024
`Page 2
`
`PETITIONERS EX1024
`Page 2
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 2 of 24
`
`US 6,862,373 B2
`
`FIG.2A
`
`
`
`FIG.2B
`
`
`
`34R==34G 34B
`
`PETITIONERS EX1024
`Page 3
`
`PETITIONERS EX1024
`Page 3
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 3 of 24
`
`OL
`
`US 6,862,373 B2
`
`ONLLOSIYNOD]|SNISSIOONOILD3SLslanvaveKDIin|[USLWBANOD
`
`
`
`O¢VivaNVOSsud
`
`91NOLLOaSEns||:poSNLLaS
`
`J90¥donioss|:|SNLLWINdINVIN
`
`
`AV1dS10ONILYSANODINISSAIONdAYOWANY,
`herZLgt!glaGaNNV2Saud
`
`ONISSIOOUdVLGv89ONISSIOOUd
`
` GANNVOS
`3NI409._ADVNI89
`
`NOLSNOLLOSSENS
`
`
`NOLLOASSNS19014
`
`N8sOLLOASENSanLas
`
`
`
`
`i43dALSN3T]i:[NOIasSans]}SNILVNIGUOOD}-1\eee‘[NOIOaSans]'
`
`
`ONIASILNAG][0p-----neneneepene“< NOLLIGNOD
`
`39|LSDVWIY3HLO
`
`
`€‘OI
`
`YSLNidd
`ONILYFANOD
`
`NOILO3S8Ns
`
`Viva
`
`AYOWAWN
`
`NVOS
`
`EINE
`
`INISSADOUd+9
`
`NOILOASAENS
`
`PETITIONERS EX1024
`Page 4
`
`PETITIONERS EX1024
`Page 4
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 4 of 24
`
`US 6,862,373 B2
`
`
`ADVWIYSHLO
`
`INIONGAY/ONIDNVINA
`
`ALVNIGYOOD
`
`SSa00ud
`
`Pp’Sia
`
`1201eelssap0ud
`
`ONIDNGAY/ONIDYVINAONILOFNYOD22ONINYOISNVYL
`
`(NOLLW10dYSLNI)SSI0OUdONINMOISNVELL
`
`N/41:SASSANAVa+V+44]
`
`-(NOILV10dY4S.LNI)SSI9OUd
`ONIONAIY/ONIDYWINA3LWNIGHOOD
`
`
`oNNaLanvavd|AiddNs:| ONIASLLNSGI; MALANVaWd|WUWHO:IdALSNF
`LuvdNOILYOdLuvd
`
`NoiLSasans|:—Nowoasansi|Nnouwoasans
`9oZLbae_—bl
`d¢Z)NOILVYuNsaVW2ZALVNIGdOOD
`
`
`ONINYOISNVYLL
`
`ONISSAIONd(}
`NOLLW10dY¥S.LNI)SSIOOUd
`ONINYOASNVYL
`
`
`
`N/5]::SaSSauav
`
`a+61=6
`
`
`
`
`
`N/dl:S3SS3YdV
`
`ag+9Vv+qI=qI
`
`SSADO8d
`
`
`
`ONIONGSY/ONIDYVINA
`
`JLVNIGYOOOD
`
`PETITIONERS EX1024
`Page 5
`
`PETITIONERS EX1024
`Page 5
`
`
`
`
`
`
`
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 5 of 24
`
`US 6,862,373 B2
`
`G°*SId
`
`
`
`19'19‘Ty
`
`06(8:9"u)sS
`
`c6
`
`
`
`SINILLAdWODJONVNINM
`
`aLlL°O+96S°0+HEO=A
`
`O=0=1
`
`XLW
`
`96
`
`
`
`Hg‘HIS‘HWY
`
`
`
`1419'TY
`
`(a°9"u)3S
`
`PETITIONERS EX1024
`Page 6
`
`PETITIONERS EX1024
`Page 6
`
`
`
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 6 of 24
`
`US 6,862,373 B2
`
`46
`
`bagogjo00000
`
`FIG.6A
`
`46
`
`46
`
`FIG.6B
`
`FIG.6C
`
`FIG.6D
`
`PETITIONERS EX1024
`Page 7
`
`PETITIONERS EX1024
`Page 7
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 7 of 24
`
`US 6,862,373 B2
`
`S1
`
`FIG./7
`
`FC
`
`PETITIONERS EX1024
`Page 8
`
`PETITIONERS EX1024
`Page 8
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 8 of 24
`
`US 6,862,373 B2
`
`LNIYdO.LOHdSe
`
`OLL
`
`WLINIQ~~_”
`
`8°Old
`
`NOILLOASNs
`
`
`
`QNISSSOOUd+9dns}:
`JVM!
`
`
`
`
`
`OINISSIOOUdVIVAINISSADOUd
`
`
`
`NOILOASNOILOASANS
`
`
`
`
`
`
`
`OGANNVOSSNIA09..V89ADVWI89
`
`YALNldd
`OINILYFANOD
`
`NOILOASaNs
`
`Vivd
`
`ONLLIFYYOO
`
`NOILYOd
`
`
`
`AYOWAWN
`
`NVOS
`
`INIS
`
`QVYVHD|SNLVUWddV¥
`
`ONISSIOOUdNOLLD3SAINAL|YSaLYIANOD
`
`
`
`AYOWIW|ONISSADOUdO2VivaNVOSAudViv
`
`J\AwidsiaQNILYFANOD
`NOLLDAS9NOILOasans||’voNSKAdNS
`
`OVNIONISSIDOUd=vLONLLWINdINV901
`rlViva;!St
`
`
`aapoervsnnececaecceceeeaeeat
`
`
`GaNNV
`
`
`osaudNOLLOASANsNOILOaSans|:
`
`
`
`
`
`
`
`
`[oipenseeeneneeeeefeseeeteetersfl,
`
`
`
`cl
`
`NOILOAS
`
`PETITIONERS EX1024
`Page 9
`
`PETITIONERS EX1024
`Page 9
`
`
`
`
`
`
`
`
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 9 of 24
`
`US 6,862,373 B2
`
`02
`
`YILNIdOLOHd
`
`WAIDIO
`
`O2l
`
`NOLLOaSans
`
`ONISSIDOUNd8SNOILOASENS
`
`Viva*.INISSSADOUd£9
`
`SNISSIOOUd|421921ONLINdINV901
`
`
`
`SALVUVddV‘|WALaWVuvdINAL|WaLYSANOD
`eaepoonerofeosesessens NOLLOASOLNowoasans}iios,|29ONILLISforsesenooseseonenesessesesneeed'
`
`
`
`
`
`eckNOLLOSAS
`
`AWdsIdONILYSANODeeeidviva
`
`ADVNIztSLgLBL
`
` GANNVOS
`
`
`ONISSADOUdVLVGONISSIOOUd
`WELWWY[beeenfeefo'INOLLDaYYOONOLLDaNNOD|NOLLSSEns][NOlLoasans]:bone
`
`
`
`PoensPoccnessenonenesesnePons!SNILVNIGUOOD
`
`
`
`GANNVOSTUdae_fuginbecinetn
`JANIS09-.v9JOVNI8Y
`!viva
`WaLAWvuvd
`
`NOLLOaSENS:NLLYIANODINILITHYOS:96
`
`
`MddNsONINYOISNVUL“~~NOLLIGNOD
`
`
`
`
`
`NOILDASNOILOASans
`
`UaLNIdd
`
`
`
`
`
`NOILUOd:
`
`cl
`
`
`
`PETITIONERS EX1024
`Page 10
`
`PETITIONERS EX1024
`Page 10
`
`
`
`
`
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 10 of 24
`
`US 6,862,373 B2
`
`ONILLYIANOD
`
`vivd
`
`YAHLO4O4
`
`JOVANI
`
`
`YaLNIdOLOHdrria-
`
`
`
`ONISSADONdNOLLDASONILLISNOLLSS8ns
`
` NOILOaSans]|ONILDAYYOD|:SNLVYVddvAlay!
`olOEFO
`YALNIdd'ONLINOSONISSIDOUdQNINNOINadNOILYOd
`
`
`OZNOLLYOd
`WIIG.———_NOLLOASENS6p|SaNNYS
`
`
`
`
`
`
`fon53sans]:IDYWNINOILD35AN3ID144300|||SNLLYNIGUOODbl—|onissao0udVivaNOHLOFHHOS|[LHL]!
`
`
`;|Awtasial.|:|NOtLoasansS3W3HOS
`
`CANNWOSANIg¢ADYWIOSL
`
`'TNouSAWSHOSNOILLYOd
`QANNVIS3ud|gel;dNLas]:
`:ONISSIDOUd
`Vel
`
`
`VIVdADVINOLLVuadOONILOANHOD
`VLWO
`9S1pee
`
`
`
`
`YSHLOYO4SJOVWIXLW/LAI
`OT°SIaONISSIOONd“JOVIél
`
`
` NOILOAS
`|_|YSLYSANOD
`
`NOILDaSONISS300ud
`
`NOLLOaSENS
`ONISSAI0Ud
`
`PETITIONERS EX1024
`Page 11
`
`907
`
`NOILIGNOOD
`
`\
`
`
`
`29ONILLASjOS
`
`
`
`NVOS3ud
`AYOWIAN
`INISSIDONd
`
`NOILO3S
`
`Vivd
`
`
`
`PETITIONERS EX1024
`Page 11
`
`
`
`
`
`
`
`
`
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 11 of 24
`
`US 6,862,373 B2
`
`pre nnn ee ee ee ee nee eee ees
`
` NEGATIVE FILM|—
`
`
`SCANNER
`READBAR |)
`CODE
`
`|
`
`YES
`
`‘[
`‘|
`
`!
`
`NO |
`
`RECOGNIZE:
`NG
`;
`
`‘
`
`:
`
`CORRECTION
`COEFFICIENT
`
`RECOGNIZE OR SETTING SECTION
`Po
`"1388[CAPTURE LENS TYPE INFORMATION] C
`138A
`{cn
`a ban nn ben nnneen nnn en ne fe nn pe eee eee eee
`é
`
`'} CORRECTION COEF.
`
`CORRECTION PARAMETER
`COMPUTING
`‘FICIENT STORTING
`
`
`i!
`LOAD
`CORRECTION 1 SUBSECTION
`SUBSECTION
`
`COEFFICIENT f=
`CORRECTION
` PRINT SIZE AND
`COEFFICIENT
`
`
`
`OTHER PARAMETERS
`
`DETECT IMAGE
`
`|
`
`
`
`
`
`‘bag
`wee ee eee ene eee de
`a
`,
`rt
`t
`Cc
`
`
`: |[COMPUTE.1|
`OMPUTE|i p---zzszeeeefeeeeeeseo ‘
`
`'! CENTER |)!|AMOUNT OF
`FINAL '!|CORRECTION
`
`
`:! POSITION 1:1:]PIXEL LOSS
`COEFFICIENT|
`‘|
`COEFFICIENT
`'! COMPUTING|{i
`
`
`oe ‘ CORRECTION COEF-+|'j
`“FICIENT SUPPLY
`|:SUBSECTION|
`
`
`
`
`
`moa oF3%Po
`
`=>uU0
`
`54
`
`
`
` 20 :
`
`
`
`
`
`INFORMATION
`ABOUT CENTER
`
`
`POSITION
`
`
`138C
`
`a@ K TIMES
`
`K TIMES
`
`LENS ABERRATION
`CORRECTING PART
`THROUGH
`
`
`
`CORRECT CHROMATIC ABERRATION
`OF MA
`GNIFICATION
`
`¥
`THROUGH
`
`
`
`!.b2oss02250
`ENLARGING/
`REDUCING PART
`
`
`
`CALCULATE FINE
`
`|}
`
`
`‘| ADJUSTMENT
`
`| COEFFICIENT
`
`
`OTHER IMAGE PROCESSING
`MONITOR
`
`
`|:
`DISPLAYON
`
`DATA CONVERTING[~66(70)
`MONITOR
`
`PERFORM ELECTRONIC SCALING
`
`||
`
`148(156)
`
`INDEX PRINT
`
`
`
`
`16
`
`PETITIONERS EX1024
`Page 12
`
`PETITIONERS EX1024
`Page 12
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 12 of 24
`
`US 6,862,373 B2
`
`FIG.12
`
`CORRECT
`CHROMATIC ABERRATION OF
`MAGNIFICATION AND
`DISTORTION
`<MAIN SCANNING DIRECTION>
`
`<AUXILIARY SCANNING DIRECTION>
`
`PERFORM
`ELECTRONIC SCALING
`<MAIN SCANNING DIRECTION>
`
`CORRECT
`CHROMATIC ABERRATION OF
`MAGNIFICATION AND DISTORTION
`
`
`
`CORRECT
`
`COLOR DIVERGENCE IN SCANNER
`
`
`<AUXILIARY SCANNING DIRECTION>
`
`
`
`
`PERFORM
`
`ELECTRONIC MAGNIFICATION
`
`
`<AUXILIARY SCANNING DIRECTION>
`
`
`
`PETITIONERS EX1024
`Page 13
`
`PETITIONERS EX1024
`Page 13
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 13 of 24
`
`US 6,862,373 B2
`
`
`
`((°A'OX)AG+0A'((0AOX)AQ+240K)xQ-F0X)Cy
`
`
`
`
`
`
`
`(NOLLOAUICONINNVDSAYVIIXN)x
`
`
`
`AGHAOX)+OX)CY (COAX)A404‘(94OK)xQ+0X)LY]((°A‘X)AG+24'((oAOX
`
`
`
`
`
`
`
`
`
`
`((°AX)AG+24‘(9,40X)xQ+0X)Ly
`
`
`
`GOHLAWONISS3D0UdJOVI
`
`(OAOX)OV:LUVYOMd4O
`
`
`
`((°A‘xAt,ox)2
`
`
`(0,'0X)OV:NOLLNSANILNIS3Yd4O
`
`
`GOHLAWONISSIIONdJOVI
`
`
`
`
`
`(NOLLDJYIGONINNVOSNIVIDA
`
`el°Ola
`
`PETITIONERS EX1024
`Page 14
`
`PETITIONERS EX1024
`Page 14
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 14 of 24
`
`US 6,862,373 B2
`
`O22812Pile
`
`PI°OLld
`
`
`
` SNWALSASAYOLVEOREVTWISIOL2
`
`9L2
`
`
`
`QNIGUOD3Y)
`
`GSNNVO2S)
`
`
`
`YOSSADONYSLNIddONISSSDOUdann
`LINNYANNVOS
`
`Yyasv71FOV!do9
`
`LINNLIND
`
`
`
`(VLVdJ9VNI
`
`
`
`(VLVdJ9OVNI
`
`
`
`(314JOVWI)(VLVGS9VWI314)
`
`
`
`
`
`
` S3NIHOWW
`
`
`
` ONISSAD0¥d|' NOILVWYOANIYSHLO1YOVIGAWONIGYODSY;
`
`PETITIONERS EX1024
`Page 15
`
`PETITIONERS EX1024
`Page 15
`
`
`
`
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 15 of 24
`
`
`
`US 6,862,373 B2
`
`PETITIONERS EX1024
`Page 16
`
`PETITIONERS EX1024
`Page 16
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 16 of 24
`
`US 6,862,373 B2
`
`
`ONISSIOOUdJOVI|q99ANN
`
`
` NOISN3LX3 YaAING:||C|CuWOBAaDIwoud|||sng|jbleeZOLE|gZE:89eirIiSIO|lAyowan||Add|[|AWOWSNJeve|Z9E”OIE|||Pauowan[|
`fuassao0ud!LINNMSLNRd|_|veei||YaSVI]"|ocg|avi|YATIOULNOD|||o/||NOLLYOd
`
`JOVI||812:!iFoOEE
`
`||WNOSuWdC4:0|||a||:|——-OSEJWOU||WY|ti
`
`iae|||ponteNOILDASa|gQce|}OUVOSAaYTOULNOS||
`atelojnee|bSEn86,OPESEEE|SS|;|AYOWSW
`||\u2ze|b1Z
`91274
`Y3SS3IOOUd
`fya‘||PPEanionaanias-oinyfj|4
`c-a=---=ae
`
`
`
`
`rwow|[wve][nao]
`
`
`ASNOWO/|yaldyvo
`99¢a4wisBEe
`
`YOLIATASa¢’e
`
`9T°OI
`
`LINN
`
`YANNWOS
`
`
`
`
`
`
`
`PETITIONERS EX1024
`Page 17
`
`PETITIONERS EX1024
`Page 17
`
`
`
`
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 17 of 24
`
`US 6,862,373 B2
`
`
`
`LT°Ord
`
`
`
`lNOLLYOdONISSIDOYdJDVWITWNOISNAWIG-JNO
`
`
`
`
`
`
`(4aTNOYLNOD
`OALOL): YOSSADOUd
`
`ADVI
`
`JAILO3454
`VNOISNSWIC-3NO
`
`MYNOISNAWIG-3NO
`
`NOILVOIZINDVN
`
`NOILWYuNagEV
`
`NILOIUNOO
`
`DILWANOUHD
`
`LNNOWV
`
`
`
`ONLLVINDWd
`
`(4)Luvd
`
`“IWNOISNSWIG-3NO
`
`NOLLVOLINSVA
`
`NOLLVEYSEV
`
`NILDINNOD
`
`OLLWWOYHD
`
`LNNOWY
`
`ONILWAINDAWO
`
`(a)Luvd
`
`Zve
`
`AYOWSIN
`
`awed
`
`
`
`gee~_|YaTIOULNODANOWSWN
`
`
`
`|||||||||
`
`YOL9373S
`
`WOud
`
`OVNI
`
`
`_NOILLIGNOO
`
`“TeQNISSIIOUd
`
`
`
`(ANIDNIdA.L3s
`
`-O.LNVYWOYS)
`
`PETITIONERS EX1024
`Page 18
`
`PETITIONERS EX1024
`Page 18
`
`
`
`
`
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 18 of 24
`
`US 6,862,373 B2
`
`FIG.18A
`
`COORDINATE SYSTEM
`
`”
`
`DIRECTION OF
`f ~~ TRANSPORT OF
`PHOTOGRAPHIC FILM
`
`ORIGINE (0,0)
`
`OOOOUOUd
`LO
`x
`
`
`
`CENTER OF IMAGE
`
`apes
`
` FILM IMAGE
`
`OOOOUOOOO
`\ PHOTOGRAPHIC
`
`FILM
`
`FIG.18B
`
`Xp Yo COORDINATE SYSTEM
`
`DIRECTION OF
`TRANSPORT OF
`PHOTOGRAPHIC FILM
`
`OOOOROOOO
`
` ORIGINE (0,Q)
`
`
`(=CENTER
`OF IMAGE)
` Yp
`
`OOOUQRUVRUUOE
`
`FILM IMAGE
`
`PHOTOGRAPHIC
`FILM
`
`PETITIONERS EX1024
`Page 19
`
`PETITIONERS EX1024
`Page 19
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 19 of 24
`
`US 6,862,373 B2
`
`FIG.19
`
`ELECTRONIC SCALING
`RATIO COMPUTING ROUTINE
`
`400
`
`N
`
`iS THERE
`LE IMAGE DATA?
`
`402
`
`Y
`
`POSITION OF IMAGE
`
`IDENTIFY AND STORE
`LENS TYPE USED TO TAKING IMAGE
`
`406
`
`CAPTURE DISTORTION
`CORRECTING DATA CORRESPONDING
`TO IDENTIFIED LENS TYPE
`
`CALCULATE AMOUNT OF
`PIXEL LOSS A x IN x-DIRECTION
`AND AMOUNTOF PIXEL LOSS
`Avy IN y -DIRECTION BASED ON
`
`DISTORTION CORRECTING DATA
`
`408
`
`410
`
`412
`
`CALCULATE PERCENTAGES OF VIGNETTING
`N x-AND y-DIRECTIONS
`2*Ax/XX).
`(2: Ay/YY
`
`414
`
`416
`
`( y -DIRECTION IS GREATER THAN
`
`THAT IN x -DIRECTION)
`
`PERCENTAGES
`
`OF VIGNETTING I" x -AND
`
`( x-DIRECTION IS
`
`GREATER y-DIRECTION)
`CALCULATE FINE ADJUSTMENT
`CALCULATE FINE ADJUSTMENT
`COEFFICIENT OF
`COEFFICIENT OF
`ELECTRONIC SCALING A m
`
`ELECTRONIC SCALING A m
`Am=YY/(YY—2*A
`
`
`
`418
`
`CORRECT ELECTRONIC
`SCALING RATIO
`mx '=mx-:Am
`my
`'=my:-Am
`
`CALCULATE CORRECTED
`AMOUNTSOF PIXEL LOSS
`Ax', Ay'
`
`420
`
`422
`
`CALCULATE PROCE“SING CONDITION
`OF OTHER IMAGE PROCESSING
`
`424
`
`PETITIONERS EX1024
`Page 20
`
`PETITIONERS EX1024
`Page 20
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 20 of 24
`
`US 6,862,373 B2
`
`FIG.20
`
`ELECTRONIC SCALING
`RETIO COMPUTING ROUTINE
`
`430
`
`434
`
`
`
`
`IS APS FILM
`INSERTRD INTO FILM
`
`CARRIER?
`
`
`
`Y
`
`
`CAPTURE PRINT SIZE
`ax ANDay
`
`
`432
`
`
`
`
`
`
`CALCULATE ELECTRONIC
`SCALING RATIO
`
`
`mx, my
`
`INPUT PRINT SIZE a x AND a y
`
`DOES INPUT END?
`
`438
`
`RETURN
`
`FIG.23A
`x -DIRECTION
`-
`
`es
`
`310°
`/
`_
`
`y -DIRECTION
`
`
`
`PERCENTAGE OF REDUCTION
`: x DIRECTION > y DIRECTION
`
`ORIGINAL IMAGE
`
`OUTPUT IMAGE
`
`x BA
`
`—__>
`
`y -DIRECTION {
`
`FIG.23B
`
`310.
`
`4
`
`.
`
`310
`
`reese ence
`
`PERCENTAGE OF ENLARGEMENT
`- y DIRECTION > x DIRECTION
`
`
`
`ORIGINAL IMAGE
`
`[aT
`OUTPUT IMAGE
`
`PETITIONERS EX1024
`Page 21
`
`PETITIONERS EX1024
`Page 21
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 21 of 24
`
`US 6,862,373 B2
`
`FIG.21A
`
`ORIGINAL IMAGE
`
`
`
`FIG.21B
`
`IMAGE THAT HAS BEEN
`CORRECTED FOR LF ABERRATIONS
`AND SUBJECTED TO ELECTRONIC
`SCALING IN y -DIRECTION
`
`YY (=Y-my)
`SCALING x -DIRECTION
`
`FIG.21C
`
`IMAGE THAT HAS BEEN
`CORRECTED FOR LF ABERRATIONS
`AND SUBJECTED TO ELECTRONIC
`
`PETITIONERS EX1024
`Page 22
`
`PETITIONERS EX1024
`Page 22
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 22 of 24
`
`US 6,862,373 B2
`
`FIG.22A
`
`FIG.22B
`
`ORIGINAL IMAGE
`
`IMAGE SUBJECTED TO LF ABERRATION
`CORRECTION AND ELECTRONIC
`SCALING IN y -DIRECTION
`
`
`
`f
`
`\
`
`PL Ay! LnyYY
`
`To yy!
`Xx
`(=Y -my')
`
`FIG.22C
`
`FIG.22D
`
`IMAGE SUBJECTED TO LF ABERRATION
`CORRECTION AND ELECTRONIC
`SCALING IN x -DIRECTION
`
`IMAGE AFTER REMOVAL
`OF PIXEL LOSSES
`IN y -DIRECTION
`
`YY
`
`
`
`YY
`
`
`
`(=X-mx')
`
`FIG.22E
`
`OUTPUT IMAGE
`(IMAGE AFTER REMOVAL OF
`PIXEL LOSSES IN x -DIRECTION)
`
`YY
`
`~—_
`
`PETITIONERS EX1024
`Page 23
`
`PETITIONERS EX1024
`Page 23
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 23 of 24
`
`US 6,862,373 B2
`
`(YITIOULNOD|
`
`O/|OL)
`
`|||
`
`YOSSdD0Nd
`
`Ove
`
`|||
`=
`
`JAIVNWI
`
`
`INNOWY|ONILIIYUOD8S2|NOLLVuuaaV
`
`JILWWONHO]«=HOY|NOLLVI4INDVW||L
`||(u)Luvd
`
`
`
`NOLLYOdSNISSIIOUdJOVITWNOISNJWIG-3NO
`
`
`gee-~_]YSTIOULNOODANOWAN
`TWNOISN3WIG-3NO
`“IWNOISNAWNIG-ANO
`
`9S2
`
`Do°OLd
`
`eve
`
`AYOWAWN
`
`dWwvVds
`
`
`
`ONILWINDIW)
`
`NOILVIISINDVN
`ONILVINDWO
`
`VNOISNAWIG-JNO
`
`NOILYOLSIC
`
`ONILOAWYOD
`
`LNNOWV
`
`NOLLWudaav
`
`ONILOSNNOD
`
`DILVWOYHD
`
`LNNOWY
`
`
`
`ONILVANDO
`
`(a)Luwd
`
`Luvd
`
`YOL9373S
`
`WOU
`
`ONISSADOUd
`
`JDVWI
`
`“onyWOU)
`NOILIGNOD
`
`|
`
`
`
`(ANIDNJdALas
`
`PETITIONERS EX1024
`Page 24
`
`PETITIONERS EX1024
`Page 24
`
`
`
`
`
`
`
`
`U.S. Patent
`
`Mar.1, 2005
`
`Sheet 24 of 24
`
`US 6,862,373 B2
`
`aeee
`
`FIG.25B
`
`G
`FIG.25A
`
`PETITIONERS EX1024
`Page 25
`
`PETITIONERS EX1024
`Page 25
`
`
`
`US 6,862,373 B2
`
`1
`IMAGE PROCESSING METHOD AND
`APPARATUS
`
`Thisis a division of application Ser. No. 09/301,538filed
`Apr. 29, 1999; now U.S. Pat. No. 6,603,885 the disclosure
`of which is incorporated herein by reference.
`BACKGROUND OF THE INVENTION
`
`2
`apparatus) that illuminates the film with reading light and
`captures the projected light to read the image on the film
`photoelectrically; an image processing apparatus that per-
`formsspecified image processing on the image data captured
`with the scanner or the image data supplied from a digital
`cameraor the like, thereby producing image data for image
`recording and exposing conditions; a printer (image record-
`ing apparatus) that scans and exposures a light-sensitive
`material to record a latent image on it, for example, by
`This invention relates to the field of digital image pro-
`scanning with optical beams in accordance with the image
`ecessing method and apparatus technologics. More
`data output from the image processing apparatus; and a
`particularly, the invention relates to image processing meth-
`processor (developing apparatus) that performs develop-
`ods and image processing apparatus which can applythis
`ment processing on the printer exposedlight-sensitive mate-
`image processing methods for typical use with digital pho-
`rial to produce a print reproducing the input image.
`toprinters that read film images photoelectrically to produce
`Users in general who intend to take ordinary pictures
`prints (photographs) reproducing the images and which are
`seldom use expensive, high-performance cameras such as a
`capable of achieving one of the following results: producing
`single-lens reflex camera but they normally usc so-called
`a high-quality image even if the input image is taken with
`“compact cameras” that are inexpensive and which are
`low-performance lenses as in films with lens, inexpensive
`capable of automatic exposing and focusing. Most recently,
`compact cameras and low-cost digital cameras; correcting
`there are a lot of users whoprefer using so-called “films with
`aberrations such as chromatic aberration of magnification
`lens” by the reason of easy handling.
`and distortion that develop in the images taken with those
`In cameras such as a single-lens reflex camera that needs
`low-performance lenses; producing a high-quality image
`a cost to some extent, high-precision lens elements are used,
`free from the image deterioration due to lens aberrations
`and a plurality of lens elements are combined to record
`even if the input image is taken with those low-performance
`images of very high quality.
`lenses; and particularly performing image processing on the
`In contrast, films with lens and inexpensive compact
`image data representing the image recorded on an image
`cameras cannotafford the use of costly lenses and only one
`recording medium.
`or two lens elements may be adopted. With such lens design,
`Heretofore, the images recorded on photographic films
`images of adequate quality cannot be taken and the image
`such as negatives and reversals (which are hereunder
`reproduced on prints does not necessarily have high quality.
`referred to simply as “films”) have been commonly printed
`If the image recorded on films is deteriorated in quality,
`on light-sensitive materials (photographic paper) by means
`there is a case that the quality of the output image on prints
`of direct (analog) exposure in which the film image is
`cannot be adequately improved by the aforementioned cor-
`optically projected onto the light-sensitive material
`to
`rections. Major causes of the deterioration of the image
`achieve its areal exposure.
`reproduced from films to be output on prints are lens
`A printer which relies upon digital exposure has recently
`aberrations such as “chromatic aberration of magnification”
`been commercialized as a digital photoprinter, in which the
`and “distortion” that originate from the low performance of
`image recorded ona film is read photoelectrically, converted
`the lenses mounted in the camera used to take the input
`to digital signals and subjected to various image processing
`image.
`operations to produce image data for recording purposes;
`Color images are formed of three primary colors, for
`recording light that has been modulated in accordance with
`example, red (R), green (G) and blue (BR). The refractive
`the image data is used to scan and exposealight-sensitive
`index (imaging magnification) ofa lens, evenif it is a single
`material to record a latent image, which is subsequently
`element, varies subtly with wavelength and differing refrac-
`developed to produce a finished print (photograph).
`tive indices occur with R, G and B lights. In other words,
`In the digital photoprinter, images are handled as digital
`even though the same position in a particular scene, a
`image data and the exposing conditions for printing can be
`focused position on a film are slipped off and differ among
`determined by image (data) processing. Hence, processing
`the R, G and B lights. This is the phenomenon generally
`operations suchas the correction of washed-out highlights or
`called “chromatic aberration of magnification” and the
`flat (dull) shadows due to the taking of pictures with rear
`image reproduced from the film has a definite color diver-
`gence.
`light or an electronic flash and sharpness enhancement
`(sometimes referred to simply as “sharpening”) can be
`to obtain a satisfactory and appropriately
`In order
`effectively performed to produce prints of the high quality
`recorded image, a plane of a scene of interest
`that
`is
`that has not been attainable by the conventional direct
`perpendicular to the optical axis must be focused on the
`exposure technique. In addition, the synthesizing of images
`sameplane as the imaging plane perpendicularto the optical
`and characters can be accomplished by image (data) pro-
`axis. In fact however, ordinary lenses have the imaging
`cessing and, as a result, prints can be output after editing
`plane displaced along the optical axis and the resulting
`and/or processing operations have been performedfreely in
`displacement of the focused position in the axial direction
`accordance with specific uses.
`causes a distortion of the focused object. As a natural
`the digital
`Aside from the images recorded on films,
`consequence, the reproduction of the image on the film is
`distorted.
`photoprinter can also output prints of images recorded with
`digital cameras or processed with computers. Other than
`being output as prints, images can be supplied to computers
`and so forth or stored in recording media such as floppy
`disks; hence, the image data can be put to various non-
`photographic uses.
`Having these features, the digital photoprinter is basically
`composed of the following units: a scanner (image reading
`
`Other causes of the image deterioration are the reduction
`of the brightness at the edge of image field which means a
`phenomenonin whichthe peripheral area of the image looks
`darker than the central area whichiscloserto the optical axis
`corresponding to the performanceof the lens used, and the
`point spread function (PSF) whichis attributable to differing
`focal positions in the plane of the film.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`PETITIONERS EX1024
`Page 26
`
`PETITIONERS EX1024
`Page 26
`
`
`
`US 6,862,373 B2
`
`3
`As noted above,if one uses a camera such as a single-lens
`reflex camera that needs a cost
`to some extent, high-
`precision lens elements may be used and a plurality of lens
`elements combined to correct various aberrations including
`chromatic aberration of magnification, distortion, deteriora-
`tion of marginal lumination and PSF and an appropriate
`image can be recorded on the film.
`However, cameras such as films with lens and compact
`cameras required to be a low cost can not use high-cost
`lenses and aberrations will develop in the images recorded
`on films. As a result, the images reproduced on prints will
`eventually have color divergence and distortion.
`To deal with this problem of image deterioration involv-
`ing the difficulty in improving the quality of output images
`onprints, techniques have been proposed in connection with
`image processing methods and apparatus that correct image
`aberrations in accordance with the characteristics of lens
`aberrations that are obtained via certain image acquisition
`means and two typical examples of such technology are
`disclosed in Unexamined Published Japanese Patent Appli-
`cation (kokai) Nos. 311425/1994 and 281613/1997,
`the
`latter being assigned to the present Applicant. According to
`these patents, the proposed technology can correct aberra-
`tions due to lenses and prevent the deterioration of image
`quality in the marginalarea, thereby ensuring the production
`of high-quality imagesat all times.
`Specifically, Unexamined Published Japanese Patent
`Application (kokai) No. 281613/1997 proposes a process of
`correcting the problem of deterioration of marginal lumina-
`tion in a photographic processing apparatus and method. In
`the process,the quantity oflight f(i,j) on an image in a given
`pixel position (i,j) is multiplied by a correction coefficient
`g(i,j) based on lens characteristics and the obtained product
`f' (i,j) is substituted as the corrected quantity oflight in the
`pixel position (i,j). To make the correction for the entire part
`of the image, j is first moved with i held constant and then
`i is moved, or alternatively,
`i is first moved with j held
`constant and then j is moved;in either way, the whole image
`can be corrected. When correcting distortion and chromatic
`aberration of magnification by this method, the position of
`the subject in an image ofinterest is changed so that a huge
`frame memoryis required to store the information about all
`pixel positions of not only before correction but also after
`the correction. In addition, a circuit is necessary that per-
`forms two processing schemesfor i and j as described above
`and this not only increases the cost of the apparatus but also
`causes a significant drop in the correction speed, thus there
`is a problem the practical use of the apparatus is difficult.
`Onthe other hand, unexamined Published Japanese Patent
`Application (kokai) No. 311425/1994 discloses an image
`correcting apparatus capable of rapid image correction in
`accordance with the characteristics of lens aberrations. In
`
`this apparatus, the subjects of the correction are the amount
`of defocusing, the decrease in the quantity of light, and the
`degree of unsharpness in the hue and chromasof a color
`image. According to the disclosure, a quantity of deteriora-
`tion in each of these correction subjects increases as it goes
`from the center of the imageto the peripheral area sothat the
`data of specified patterns that are increased as going from the
`center toward the peripheral in an imagearea that is formed
`by concentric circles or squares extending radially outward
`from the center of the imageare only used for each of given
`lens characteristics as correction enhancement coefficients
`
`for correcting these correction subjects. This approach is
`capable of rough correction but not image corrections aber-
`rant from the patterns that is prepared beforehand. Hence,
`there is a problem that it is impossible to perform appropri-
`ate corrections according to the characteristics of individual
`taking lenses.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`If one wants to accomplish the appropriate correction by
`this technology, correction enhancement coefficients must
`be made available for all patterns that are predicted for the
`given lens characteristics and to meet this need, a memory
`of sufficiently large capacity is required. Whatis more, if the
`available patterns are not simple concentric circles or
`squares whose center coincides with that of the image, an
`increased amount of data has to be calculated to correct
`matrices and the overall image processing speed is lowered.
`Thesepractical problems with cost and processing speed are
`particularly serious in the case of print services that involve
`the volume reproduction of images.
`As mentioned hereinabove, an image processing system is
`conventionally knownthat is able to perform various image
`processing schemes on the image data obtained by reading
`the film image recorded on photographic films or on the
`image data input from a digital camera or the like and which
`then outputs an image in various output modes such as by
`recording the image on recording materials such as photo-
`graphic paper or storing the image data in information
`recording media. Compared to a conventional photographic
`processing system that records the film image by areal
`exposure, the image processing system just described above
`can control the output image quality by image processing
`onto the image data, thereby output images of high quality
`are realized.
`
`Speaking of films with lens, the lens is usually composed
`of an inexpensive plastic lens that inevitably suffers from
`great amounts of aberrations such as distortion and chro-
`matic aberration of magnification. Hence, the film image
`recorded on a photographic film by exposure with the film
`with lens has a comparatively high level of geometric
`distortion according to the lens distortion (so-called
`“pincushion-type distortions) as typically shown in FIG.
`25A (FIGS. 25A and 25Billustrate how an image consisting
`of multiple lines in a grid pattern appearsif it is recorded on
`a photographic film by shooting with a film with lens); at the
`same time, a color divergence occurs at a comparatively
`high level due to chromatic aberration of magnification. To
`deal with this problem, distortion correction for correcting
`the geometric distortion of an image due to the distortion of
`the lens on the film with lens and magnification chromatic
`aberration correction for correcting the color divergence in
`an imagedueto the chromatic aberration of magnification of
`the same lens are being review in order to ensure that the
`image processing system described above can produce an
`output image of high quality from the original (input) image
`recorded with the film with lens.
`
`With a view to increasing the speed of the various
`processing schemes to be performed with the above-
`mentioned image processing system, the contents of image
`processing and the value of a parameter to be applied in a
`specific image processing schemehave beenset identical in
`specified directions, such as vertical and horizontal, of the
`image represented by the image data. For instance, if the
`image processing to be done is electronic scaling,
`the
`electronic scaling ratio is set identical in both the vertical
`and horizontal directions of the image represented by the
`image data to be processed.
`What is unique about image data is that its volume is
`tremendousand that it represents an image having a two-
`dimensional extent. Hence, image processing schemes such
`as ones for correcting distortion and chromatic aberration of
`magnification are so much complicated in contents that they
`not only take prolonged time but also require storage means
`of large capacity. As a result, the image processing section
`capable of performing image processing schemessuchas the
`PETITIONERS EX1024
`Page 27
`
`PETITIONERS EX1024
`Page 27
`
`
`
`US 6,862,373 B2
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`5
`correction of distortion and chromatic aberration of magni-
`fication is considerably complicated in configuration and,
`what
`is more,
`the processing performance of the image
`processing system is eventually deteriorated.
`Consider, for example, the correction of distortion. First,
`distortion correcting data representing the direction and the
`amountofthe shift in the position of each of the pixels in the
`original film image due to the distortion of a lens, as
`referencedto the inherentposition of each pixel (the position
`of its grid point) composingthe film image, are preliminarily
`measured and stored for each of the lens types used; given
`the image data to be processed,the distortion correcting data
`associated with the lens type used in actual shooting are
`captured; on the basis of the captured distortion correcting
`data, the positions of