(12) United States Patent
`Tsuda
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,808,533 B2
`Oct. 5, 2010
`
`US007808533B2
`
`(54) ELECTRONIC CAMERA HAVING SIGNAL
`PROCESSING UNITS THAT PERFORM
`SIGNAL PROCESSING ON MAGE DATA
`
`(75) Inventor: Yutaka Tsuda, Kawasaki (JP)
`(73) Assignee: Nikon Corporation, Tokyo (JP)
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 346 days.
`(21) Appl. No.: 11/819,994
`
`(22) Filed:
`
`Jun. 29, 2007
`
`(65)
`
`Prior Publication Data
`US 2007/0252903A1
`Nov. 1, 2007
`
`-
`
`-
`
`-
`
`Related U.S. Application Data
`(60) Division of application No. 09/497,482, filed on Feb.
`4. 2000. now Pat. No. 7.253,836, which is a continua-
`tion-in-part of application No. 09/342,512, filed on
`Jun. 29, 1999, now abandoned.
`O
`O
`Foreign Application Priority Data
`(30)
`Jun. 30, 1998
`(JP)
`................................. 10-183918
`Jun. 30, 1998
`(JP)
`... 10-183919
`Jun. 30, 1998
`(JP) .
`... 10-183920
`Jun. 30, 1998
`(JP)
`................................. 10-183921
`Aug. 24, 1998 (JP)
`................................. 10-237321
`Jul. 28, 1999
`(JP)
`................................. 11-213299
`(51) Int. Cl.
`(2006.01)
`H04N 5/76
`(52) U.S. Cl. ................................. 348/231.6, 348/2312
`(58) Field of Classification Search .............. 348/220.1,
`348/231.99, 231.2, 231.6, 231.8, 254, 333.11
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`4,774,565 A
`9, 1988 Freeman
`(Continued)
`FOREIGN PATENT DOCUMENTS
`
`JP
`
`A-S62-185489
`8, 1987
`(Continued)
`OTHER PUBLICATIONS
`Aug. 6, 2009 Non-final Rejection for copending U.S. Appl. No.
`1 1/819,666.
`
`(Continued)
`Primary Examiner Timothy J Henn
`(74) Attorney, Agent, or Firm Oliff & Berridge, PLC
`
`(57)
`
`ABSTRACT
`
`A CCD captures a Subject image having passed through a
`taking lens and an image processing circuit performs various
`types of image pre-treatment including gamma correction
`and white balance on image data corresponding to n linesxm
`rows output by the CCD. The image processing circuit also
`performs format processing on the data. The data are then
`compressed at a compression circuit. The white balance
`adjustment and the like are implemented in line sequence at a
`line processing circuit which engages in signal processing in
`pixel sequence in units of individual lines in the output from
`the CCD. The image data having undergone the pre-treatment
`are then subjected to format processing prior to JPEG com
`pression, at a block processing circuit that engages in signal
`processing in units of individual blocks each ranging over an
`nxm (N>n, MDm) block. In other words, the signal process
`ing is performed in block sequence.
`
`4 Claims, 21 Drawing Sheets
`
`3. 1
`
`31
`
`l
`
`--ORMA. IAGE DE
`
`
`
`328
`
`ATA TERNA329
`
`Qualcomm, Exh. 2007, p. 1
`Apple v. Qualcomm, 2018-01277
`
`

`

`US 7,808,533 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`5,016,107 A
`5/1991 Sasson et al.
`5,153,730 A 10/1992 Nagasaki et al.
`5,206,730 A
`4, 1993 Sakai
`5,305.096 A
`4, 1994 Yamagami et al.
`5,319,451 A
`6, 1994 Sasaki et al.
`5,343,243 A
`8, 1994 Maeda
`5,373.322 A 12/1994 Laroche et al.
`5,402,171 A
`3/1995 Tagami et al.
`5,414.464 A
`5, 1995 Sasaki
`5,525,957 A * 6/1996 Tanaka .................... 348/220.1
`5,579,047 A 1 1/1996 Yamagami et al.
`5,581,357. A 12/1996 Sasaki et al.
`5,631,701 A
`5/1997 Miyake
`5,640,619 A
`6/1997 Takayama et al.
`5,778, 106 A
`7/1998 Juenger et al.
`5,867,214 A * 2, 1999 Anderson et al. ........ 348,231.6
`5,917,556 A
`6, 1999 Katayama
`5,920,343 A
`7, 1999 Watanabe et al.
`6,005,613 A * 12/1999 Endsley et al. ........... 348,231.6
`6,075,563 A
`6/2000 Hung
`6, 130,710 A * 10/2000 Yasuda .................... 348,220.1
`6.288,743 B1* 9/2001 Lathrop ................... 348,231.9
`6,292,218 B1* 9/2001 Parulski et al. ........... 348,220.1
`6,359,643 B1* 3/2002 Visvanathan et al. ..... 348/14.14
`6.421,083 B1
`7/2002 Takakura
`6,529,238 B1
`3/2003 Mahant-Shetti et al.
`6,532,039 B2
`3/2003 Anderson
`6,661.451 B1
`12/2003 Kijima et al.
`6,954.228 B1
`10/2005 Acharya et al.
`
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`P
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`WO
`
`2002/0176009 A1 11/2002 Johnson et al.
`FOREIGN PATENT DOCUMENTS
`A-S63-20993
`1, 1988
`A-2-251.05
`1, 1990
`A-HO2-213292
`8, 1990
`A-02-278983
`11, 1990
`A-HO3-148990
`6, 1991
`A-3-154487
`7, 1991
`A-HO4-23576
`1, 1992
`A-4-88782
`3, 1992
`A-4-501645
`3, 1992
`A-HO4-107083
`4f1992
`A-5-176333
`7, 1993
`A-05-252522
`9, 1993
`A-HO7-107505
`4f1995
`A-7-222178
`8, 1995
`A-7-288761
`10, 1995
`A-08-214145
`8, 1996
`A-9-18773
`1, 1997
`A-9-261670
`10, 1997
`A-9-270991
`10, 1997
`A-HO9-322191
`12/1997
`A-10-4858
`1, 1998
`A-10-108116
`4f1998
`A-2000-23184
`1, 2000
`2000-293.145
`10, 2000
`WO 90/O5425
`5, 1990
`OTHER PUBLICATIONS
`-- n:
`Office Action in U.S. Appl. No. 1 1/819,666; mailed May 27, 2010.
`* cited by examiner
`
`Qualcomm, Exh. 2007, p. 2
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 1 of 21
`
`US 7,808,533 B2
`
`FIG. 1
`
`
`
`Qualcomm, Exh. 2007, p. 3
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 2 of 21
`
`US 7,808,533 B2
`
`WG8898098
`
`s
`Y
`
`1.
`a.
`C9
`C9
`d
`
`O
`1V.
`
`+92
`
`£862821292
`
`000
`
`
`
`Qualcomm, Exh. 2007, p. 4
`Apple v. Qualcomm, 2018-01277
`
`

`

`US. Patent
`
`Oct. 5, 2010
`
`Sheet 3 of 21
`
`US 7,808,533 B2
`
`vo—
`
`ma—
`
`Na.
`
`—o_
`
`Mw.mu_nm
`
`ammmzmo»
`
`om>¢o=m2
`
`z_<omz_<om
`
`<h<omom
`
`mu<xm><
`
`m24<>
`
`\zo_p<4=ug<o
`
`=<zoOHm_:
`
`zo_p<4204<o
`
`xu<4m
`
`4m>m4
`
`NoF
`
`mo_
`
`mo_
`
`oc_
`
`mp_:;
`
`muz<4<m
`
`
`
`z_<u4<__o_a
`
`m=<go
`
`Hummus
`
`zo_hum¢moo
`
`<h<o
`
`momo\<2cm;
`
`“a?
`
`Qualcomm, Exh. 2007, p. 5
`Apple v. Qualcomm, 2018-01277
`
`Qualcomm, Exh. 2007, p. 5
`Apple v. Qualcomm, 2018-01277
`
`
`
`

`

`U.S. Patent
`
`Sheet 4 of 21
`
`US 7,808,533 B2
`
`co
`
`002
`
`
`
`
`
`
`
`
`
`Qualcomm, Exh. 2007, p. 6
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 5 of 21
`
`US 7,808,533 B2
`
`
`
`11 12 13 14 15 16 17 18
`
`i.
`3.
`w-r-
`
`...
`S.
`t
`g
`x.
`::
`
`...
`
`X."
`.
`it.
`:
`. . .
`
`':
`
`. .
`s
`a.
`Sales.
`. . ."
`.
`f
`
`r
`
`g
`
`ADD PXEL, DATA N SHADED AREAS
`
`Qualcomm, Exh. 2007, p. 7
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 6 of 21
`
`US 7,808,533 B2
`
`F.G. 6
`
`A U
`
`A C
`
`AD
`
`Qualcomm, Exh. 2007, p. 8
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 7 of 21
`
`US 7,808,533 B2
`
`FIG 7
`
`
`
`801
`
`Qualcomm, Exh. 2007, p. 9
`Apple v. Qualcomm, 2018-01277
`
`

`

`FG. 8
`
`
`
`BER 0
`
`
`
`.
`
`.
`
`.
`
`.
`.
`.
`.
`.
`LEVEL (
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`h
`. 255
`
`FIG. 8B
`
`
`
`
`
`
`
`BER OF
`
`MBER OF DOTS f
`
`
`
`
`
`-
`
`-
`
`-
`
`-
`
`FIG. 8
`
`Hh.
`
`-
`-
`-
`-
`-
`-
`-
`-
`LEVEL (8-bit)
`
`-
`
`-
`
`-
`
`- 255
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`. 255
`
`Qualcomm, Exh. 2007, p. 10
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 9 of 21
`
`US 7,808,533 B2
`
`FG 9
`
`
`
`<; [GEZEGETETTTTTTTTTTLLLL
`
`•[GIG GEDEELD:D:D:D:D:D:D_L_I_I_I_I_I_
`
`
`
`•EEEEEERTTTTTTTTTTILL —[ETETETETTIGTVTTTTTTTILLLLL
`
`Qualcomm, Exh. 2007, p. 11
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 10 of 21
`
`US 7,808,533 B2
`
`3
`4
`5
`6
`7
`8
`9
`O
`1
`2
`3
`
`
`
`
`
`FIG 10
`D5
`
`|
`
`|
`
`|
`
`| | | | | |
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|| |
`
`|
`
`|
`
`|
`
`|
`
`G
`G
`G
`G
`G
`G
`
`D8
`(8X8 BPF OUTPUT)
`D12
`(12X12 BPF INPUT)
`
`Qualcomm, Exh. 2007, p. 12
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 11 of 21
`
`US 7,808,533 B2
`
`
`
`(8x8 LPF OUTPUT)
`(12X12 LPF INPUT)
`
`Qualcomm, Exh. 2007, p. 13
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 12 of 21
`
`US 7,808,533 B2
`
`
`
`D 16-4
`
`D 16-5
`
`Qualcomm, Exh. 2007, p. 14
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 13 of 21
`
`US 7,808,533 B2
`
`
`
`R-G (i-2, j+2)
`
`Qualcomm, Exh. 2007, p. 15
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 14 of 21
`
`US 7,808,533 B2
`
`F.G. 14
`
`D 5
`
`
`
`
`
`4 5 6
`
`8 9
`
`D16 (16X 16 INPUT DATA)
`
`
`
`
`
`I
`1
`|
`| |
`| |
`| | | | | | |
`|
`| | |
`2 | |
`3
`TTTTTTTTTTTT
`5
`||
`|
`|
`|
`|
`|
`|
`|
`|
`| |
`|
`|
`|
`|
`| |
`| |
`6
`7
`8
`9
`10
`11
`12
`13
`III || | | |
`14
`15
`|
`|
`|
`|
`|
`|
`|
`16 T |
`17
`II
`18
`19 || |
`20
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`|
`
`| || ||
`|
`|
`|
`|
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`|
`
`
`
`
`
`
`
`D8
`(8x8 Y OUTPUT DATA)
`(12x12 CrCb OUTPUT DATA)
`
`Qualcomm, Exh. 2007, p. 16
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 15 of 21
`
`US 7,808,533 B2
`
`
`
`2
`
`| |
`
`|
`
`|
`
`|
`
`| |
`
`|
`
`|
`
`|
`
`|
`
`(8X8 OUTPUT DATA)
`D12
`(12X12 NPUT DATA)
`
`Qualcomm, Exh. 2007, p. 17
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 16 of 21
`
`US 7,808,533 B2
`
`FIG. 16
`
`RELEASE HALF PRESSED
`
`S20A
`DETECT FOCAL POINT
`
`S2OB
`RELEASE FULLY PRESSED 2
`Y
`
`S21
`STORE AND READ OUT
`
`S22
`ANALOG SIGNAL PROCESSING,
`A/D CONVERSION
`
`S23
`
`MAGE PROCESSING
`
`S24
`TEMPORARLY RECORD
`N BUFFER MEMORY
`
`S25
`
`COMPRESS
`
`S26
`STORE IN PC CARD
`
`RETURN
`
`Qualcomm, Exh. 2007, p. 18
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 17 of 21
`
`US 7,808,533 B2
`
`
`
`
`
`
`
`
`
`Qualcomm, Exh. 2007, p. 19
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 18 of 21
`
`US 7,808,533 B2
`
`FIG. 18
`
`91
`
`92
`
`IMAGE DATA
`PROGRAM
`
`
`
`HARD DISK
`
`PERSONAL
`COMPUTER
`
`
`
`MONITOR
`
`PRINTER
`
`
`
`
`
`
`
`Qualcomm, Exh. 2007, p. 20
`Apple v. Qualcomm, 2018-01277
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 19 of 21
`
`US 7,808,533 B2
`
`
`
`118
`
`||N/A
`TOHINOO A
`
`
`
`LIn)HIO AVIASIG HOLINOM |~| HOLINOM I VERW9 9INOH105TH 018
`
`300W928ZZE © ?ÆINI , KOE),
`
`
`
`
`
`
`
`Qualcomm, Exh. 2007, p. 21
`Apple v. Qualcomm, 2018-01277
`
`

`

`US. Patent
`
`0
`
`1
`
`n_Ew.11...h_snn............................“f_o.__mWIIIIJLIIIIraIIi...‘_2,n__wu__cua...0,3Iua0'.,—
`mhm
`
`EI'.
`
`,—
`.—
`on
`
`2B3
`
`U
`
`amassing:38,Ew,mus5:8oz.7<._.<n35.S18%5&2
`
`Qualcomm, Exh. 2007, p. 22
`Apple v. Qualcomm, 2018-01277
`
`Qualcomm, Exh. 2007, p. 22
`Apple v. Qualcomm, 2018-01277
`
`
`

`

`U.S. Patent
`
`Oct. 5, 2010
`
`Sheet 21 of 21
`
`US 7,808,533 B2
`
`
`
`p-------?--- --
`
`a - - - - - - - - - - - - - - - - - -
`
`----
`
`Qualcomm, Exh. 2007, p. 23
`Apple v. Qualcomm, 2018-01277
`
`

`

`US 7,808,533 B2
`
`1.
`ELECTRONIC CAMERA. HAVING SIGNAL
`PROCESSING UNITS THAT PERFORM
`SIGNAL PROCESSING ON MAGE DATA
`
`This is a Division of U.S. patent application Ser. No.
`09/497,482 filed Feb. 4, 2000 (now U.S. Pat. No. 7,253.836),
`which in turn is a Continuation-In-Part of U.S. patent appli
`cation Ser. No. 09/342,512 filed Jun. 29, 1999 (now aban
`doned). The disclosure of each of the prior applications is
`hereby incorporated by reference herein in its entirety.
`
`INCORPORATION BY REFERENCE
`
`The disclosures of the following priority applications are
`herein incorporated by reference:
`Japanese Patent Application No. 10-183918, filed Jun. 30.
`1998
`Japanese Patent Application No. 10-183919, filed Jun. 30.
`1998
`Japanese Patent Application No. 10-183920, filed Jun. 30.
`1998
`Japanese Patent Application No. 10-183921, filed Jun. 30.
`1998
`Japanese Patent Application No. 10-237321, filed Aug. 24.
`1998
`Japanese Patent Application No. 11-213299, filed Jul. 28,
`1999
`
`5
`
`10
`
`15
`
`25
`
`BACKGROUND OF THE INVENTION
`
`2
`individual lines in correspondence to the read performed at
`the CCD. Because of this, in a high image quality electronic
`still camera with the number of pixels at the CCD exceeding
`two million, the capacity of the line buffer memory employed
`for pipeline operation and the like, is bound to be very large,
`resulting in the camera becoming expensive. This problem
`may be explained as follows.
`When performing signal processing on the output from a
`Solid image-capturing element, NXM sets of image data cor
`responding to one screen output by the image-capturing ele
`ment are output in point sequence in units of individual lines.
`Thus, when performing signal processing including pixel
`interpolation processing and filtering processing, a line buffer
`memory Supporting four lines, for instance, is required if the
`filtering processing is to be performed in sets of 5x5. In other
`words, the processing can be performed only when image
`data corresponding to four lines have been accumulated in the
`memory. Such a line buffer memory Supporting four lines is
`required for each of the various types of processing Such as
`filtering processing and interpolation processing.
`If a line buffer memory that supports four lines is provided
`at a 1-chip processing IC for each of the various types of
`processing required, such as the filtering processing and the
`interpolation processing described above, the ratio of the area
`occupied by the memory increases, which leads to an increase
`in the number of gates at the 1-chip processing IC, resulting in
`higher cost. In particular, in a high resolution type image
`capturing element having more than two million pixels with a
`large number of pixels per line, the cost will be especially
`high. In addition, if the line buffer memory is provided out
`side the 1-chip processing IC, twenty 10-bit input/output
`pins, for instance, will be required. This means that 20 input/
`output pins will be necessary for each line buffer memory to
`result in an increase in the package size of the 1-chip process
`ing IC.
`(2) In the image-capturing device in an electronic still
`camera in the prior art, the interpolation processing for an
`(R-G) signal and a (B-G) signal, matrix processing through
`which a Y signal, a Cr signal and a Cb signal are generated
`using the (R-G) signal, the (B-G) signal and the G signal and
`LPF processing through which low frequency signals are
`extracted from the Y signal, the Cr signal and the Cb signal is
`performed in time sequence to format the data for JPEG
`compression and to Suppress false colors and color moire
`from occurring. As a result, particularly in the case of a high
`image quality electronic still camera with the number of
`pixels at the CCD exceeding two million, the length of time of
`processing is bound to be large, resulting in poor operability.
`(3) In the image-capturing device in an electronic still
`camera in the prior art, a single primary color type CCD, two
`CCDs (one for G and the other for R/B) or three CCDs (one
`each for R, G and B) are employed. When using a single CCD,
`since an RGB color filter is provided at the front surface of
`each pixel at the CCD, an R signal, a G signal or a B signal is
`missing from a given pixel. Thus, interpolation is performed
`for pixels without a G signal component by using the G
`signals of pixels that have been actually obtained to generate
`G signals for all the pixels, and interpolation in regard to the
`(R-G) signal and the (B-G) signals is likewise performed. The
`same principle applies when using two CCDs, as well.
`However, depending upon the nature of the image that has
`been captured or the characteristics of the low pass filter
`employed, false colors or moire may occur after the interpo
`lation processing, which results in a great degree of degrada
`tion in the image quality. While the Cr signal and the Cb
`signal among the Y signal, the Cr signal and the Cb signal
`described above created using the R, G and B signals are
`
`30
`
`35
`
`40
`
`45
`
`1. Field of the Invention
`The present invention relates to a digital camera that stores
`in memory a subject as image data that are electronically
`compressed, and it also relates to a storage medium that stores
`an image signal processing program. Furthermore, the
`present invention relates to a carrier wave that is encoded to
`transmit a control program for white balance adjustment on
`image data. It also relates to an electronic camera that allows
`selection to be made between recording of irreversible image
`data and recording of raw data.
`2. Description of the Related Art
`Electronic still cameras in the known art include the type
`provided with a viewfinder device to which a subject image
`having passed through a taking lens is guided by a quick
`return mirror, an image-capturing device Such as a CCD that
`is provided at a rearward of the quick return mirror to capture
`an image of the Subject image and output image data, an
`image processing circuit that performs image processing Such
`as white balance and gamma correction on the image data
`output by the image-capturing device, a compression circuit
`that compresses the data which have undergone image pro
`cessing through a method such as JPEG and stores the data in
`a storage medium Such as a flash memory and a monitor that
`displays the data having undergone the image processing. In
`the image processing circuit, parameters such as the R gain
`and the B gain for white balance adjustment or the gradation
`curve for gamma correction are calculated based upon a pre
`set algorithm. In addition, the image data are converted to
`16x8 sets of brightness dataY and 8x8 sets of Crand Cb color
`difference data for JPEG compression.
`60
`The image-capturing device in Such an electronic still cam
`era in the prior art structured as described above presents the
`following problems.
`(1) Both the image pre-treatment Such as white balance or
`gamma correction and the image post-treatment, in which the
`data that have undergone the image pre-treatment are format
`ted for the JPEG compression, are performed in units of
`
`50
`
`55
`
`65
`
`Qualcomm, Exh. 2007, p. 24
`Apple v. Qualcomm, 2018-01277
`
`

`

`US 7,808,533 B2
`
`10
`
`15
`
`25
`
`30
`
`35
`
`3
`processed through the low pass filter to suppress false colors
`or moire in the prior art, this means does not achieve satisfac
`tory results in a high image quality electronic still camera
`having more than two million pixels at the CCD.
`(4) Since the white balance adjustment is achieved using
`predetermined white balance adjustment coefficients in the
`image-capturing device in an electronic still camera in the
`prior art, there is the likelihood of a color-cast image being
`generated if the white balance adjustment coefficients are set
`erroneously. This problem tends to occur more readily in a
`high image quality electronic still camera with the number of
`pixels at the CCD exceeding two million.
`Electronic cameras in which selection between the two
`different data formats described below can be performed
`when recording image data obtained through image-captur
`ing have been known in the prior art.
`(1) Irreversible compressed data obtained through JPEG or
`the like that have undergone a sequence of image processing
`(2) Raw data output by the image-capturing device
`The first type of data, i.e., irreversible compressed data are
`advantageous in that since the code Volume is relatively
`Small, a large number of images can be stored in an external
`recording medium such as a memory card. In addition, they
`are recorded in a general-purpose format which allows data
`decoded by using a common image viewing software pro
`gram or the like to be printed or displayed directly.
`The second type of data, i.e., raw data are image data
`faithful to the output signal from the image-capturing device.
`A data recording format of raw data facilitates external pro
`cessing. Since raw data which, undergo very little irreversible
`gradation conversion or data compression, contain a large
`Volume of information Such as the number of quantization
`bits, they have a wide dynamic range as image information.
`Thus, they provide an advantage in that they can be processed
`in an ideal manner without the tendency to lose fine gradation
`components. For this reason, highly advanced data process
`ing and a higher quality are required in this type of raw data.
`The data format of raw data is particularly suited for printing
`and design applications.
`40
`Normally, an electronic camera requires a greater length of
`time for image processing compared to cameras using a silver
`halide film. In order to achieve a degree of operability in an
`electronic camera comparable to that of cameras using a
`silver halide film, it is crucial to minimize the length of time
`required for image processing. However, in an electronic
`camera in the prior art, a raw data read/write operation per
`formed via an image memory is always necessary. This tends
`to lead to a delay occurring in signal processing performed on
`irreversible compressed data by a length of time correspond
`ing to the length of time spent on the raw data read/write.
`In addition, processing circuits that perform relatively
`complex processing, a prime example of which is pixel value
`matrix operation, are concentrated in a processing unit at a
`stage preceding the stage for gamma control operation. Since
`raw data with a large number of quantization bits are handled
`in this state at these processing circuits, the circuit structures
`of the processing circuits tend to be complex and there is also
`a problem of a greater length of time required for signal
`processing.
`
`4
`A second object of the present invention is to provide a
`storage medium that stores a program for implementing sig
`nal processing which achieves a reduction in the required
`capacity of the buffer memory even when processing image
`data for which image-capturing has been performed using an
`image-capturing device having a large number of pixels.
`A third object of the present invention is to provide a digital
`camera that achieves a reduction in the length of time required
`for data formatting or processing through which false colors
`or moire is prevented even when the number of pixels is large.
`A fourth object of the present invention is to provide a
`storage medium that stores a program for implementing sig
`nal processing in which data formatting and processing for
`preventing false colors and moire can be 3-4 performed
`within a short period of time even when handling image data
`for which image-capturing has been performed using an
`image-capturing device with a great number of pixels.
`A fifth object of the present invention is to provide a digital
`camera that Suppresses the color-cast phenomenon occurring
`due to an error manifesting following the white balance
`adjustment performed by an external sensor to a satisfactory
`degree.
`A sixth object of the present invention is to provide a
`storage medium that stores a program for implementing sig
`nal processing through which the color-cast phenomenon
`occurring due to an error manifesting after the white balance
`adjustment performed by an external sensor can be Sup
`pressed to a satisfactory degree.
`The digital camera according to the present invention com
`prises an image-capturing device that captures a subject
`image having passed through a taking lens and outputs image
`data, a recording processing circuit that performs recording
`processing on the image data and an image processing circuit
`that performs a pre-treatment (includes gamma correction
`and white balance correction) on the image data correspond
`ing to N linesXM rows output by the image-capturing device
`in units of individual lines in line sequence and then performs
`format processing (includes interpolation processing, LPF
`processing, BPF processing and color difference signal cal
`culation processing) that corresponds to the type of recording
`performed at the recording processing circuit on the image
`data having undergone the pre-treatment in units of individual
`blocks corresponding to n linesxm rows (NDn, MDm) in
`block sequence.
`The image processing performed in this digital camera
`may be implemented on a computer. The program stored in a
`storage medium for this purpose implements signal process
`ing including format processing through which the image
`data of an image captured at an image-capturing device are
`formatted for recording, various types of pre-treatment that
`are implemented prior to the format processing and recording
`processing through which the image data having undergone
`format processing are recorded, with signal processing dur
`ing the pre-treatment performed in units of individual lines in
`line sequence on image data corresponding to N linesXM
`rows and signal processing during the format processing per
`formed in units of individual blocks corresponding to n
`linesxm rows (NDn, MDm) in block sequence on the image
`data having undergone the pre-treatment.
`Alternatively, the digital camera according to the present
`invention may comprise an image-capturing device that cap
`tures a subject image having passed through a taking lens and
`outputs image data, a recording processing circuit that per
`forms recording processing on the image data and an image
`processing circuit that, with the image data output by the
`image-capturing device input as data corresponding to n
`linesxm rows, calculates a color difference signal based upon
`
`45
`
`50
`
`55
`
`60
`
`SUMMARY OF THE INVENTION
`
`A first object of the present invention is to provide a digital
`camera that does not necessitate any increase the capacity of
`buffer memory and thus, achieves a reduction in cost even
`when the number of pixels is great.
`
`65
`
`Qualcomm, Exh. 2007, p. 25
`Apple v. Qualcomm, 2018-01277
`
`

`

`US 7,808,533 B2
`
`10
`
`15
`
`5
`the image data thus input, performs interpolation processing
`and low pass filtering processing at once on the color differ
`ence signal using filter coefficients for interpolation/low pass
`filtering and then generates a formatted signal by performing
`matrix processing corresponding to the type of recording
`implemented at the recording processing circuit.
`The image processing performed in this digital camera
`may be implemented on a computer. The program stored in a
`storage medium for this purpose executes format processing
`that formats the image data of an image captured at an image
`capturing device for recording, in which color difference
`signals corresponding to n linesxm rows are calculated based
`upon the image data that are input, interpolation processing
`and low pass filtering processing are executed at once on the
`color difference signals corresponding to n linesxm rows
`using filter coefficients for interpolation/low pass filtering
`and then a formatted signal is generated through matrix pro
`cessing and recording processing through which the image
`data having undergone the format processing are recorded.
`Furthermore, in the digital camera according to the present
`invention, which may comprise an image-capturing device
`that captures a Subject image having passed through a taking
`lens and outputs image data, an image processing circuit that
`performs image processing including data format processing
`appropriate for data compression on the image data output by
`the image-capturing device and a compression circuit that
`compresses image data output by the image processing cir
`cuit, the image processing circuit engages in median process
`ing on image data corresponding to nxm pixel areas to
`execute the format processing.
`The image processing performed in this digital camera
`may be implemented on a computer. The program stored in a
`storage medium for this purpose implements signal process
`ing including format processing through which the image
`data of an image captured at the image-capturing device are
`formatted for compression, various types of signal processing
`that are implemented prior to the format processing and com
`pression processing through which image data having under
`gone the format processing are compressed, with median
`processing performed on image data corresponding to nxm
`40
`pixel areas during the format processing.
`By extracting (n-i)x(m-) sets of image data among the
`image data corresponding to the nxm pixel areas and per
`forming median processing on them, the length of time
`required for the median processing can be reduced.
`Moreover, the digital camera according to the present
`invention may comprise an image-capturing device that cap
`tures a subject image having passed through a taking lens and
`outputs image data and an image processing circuit that
`executes image processing on the image data output by the
`image-capturing device, in which median processing is
`implemented on (n-i)x(m-j) sets of image data extracted from
`image data corresponding to an inxm pixel area.
`The image processing performed in this digital camera
`may be implemented on a computer. The program stored in a
`storage medium for this purpose implements a specific type
`of image processing on the image data of an image captured
`at the image-capturing device, in which median processing is
`executed on (n-i)x(m-) sets of image data extracted from
`image data corresponding to an inxm pixel area.
`The digital camera according to the present invention may
`comprise an image-capturing device that captures a subject
`image that passes through a taking lens and outputs image
`data, a white balance adjustment circuit that performs white
`balance adjustment on the image data output by the image
`capturing device, a white balance fine adjustment coefficient
`calculation circuit that calculates a white balance fine adjust
`
`30
`
`6
`ment coefficients based upon image data having undergone
`the white balance adjustment, output by the white balance
`adjustment circuit and a white balance fine adjustment circuit
`that performs white balance fine adjustment using the white
`balance fine adjustment coefficients on image data having
`undergone the white balance adjustment output by the white
`balance adjustment circuit.
`The image processing implemented in this digital camera
`may be executed by a computer. The program stored in a
`storage medium for this purpose implements white balance
`adjustment processing in which white balance adjustment is
`performed on the image data of an image-captured at an
`image-capturing device, white balance fine adjustment coef
`ficient calculation processing in which white balance fine
`adjustment coefficients are calculated using image data hav
`ing undergone the white balance adjustment through the
`white balance adjustment processing and white balance fine
`adjustment processing in which white balance fine adjust
`ment is performed using the white balance fine adjustment
`coefficients on the image data having undergone the white
`balance adjustment.
`The white balance fine adjustment coefficients are calcu
`lated based upon the average values calculated for the R. B
`and G signals in the image data having undergone the white
`balance adjustment. Alternatively, it may be calculated based
`upon the histograms of the brightness levels calculated for the
`R, B and G signals of the image data having undergone the
`white balance adjustment.
`In addition, the digital camera according to the present
`invention may comprise an image-capturing device that cap
`tures a Subject image that passes through a taking lens and
`outputs image data, a white balance adjustment circuit that
`performs white balance adjustment on the image data output
`by the image-capturing device, an image area selection appa
`ratus that selects one image area among a preset plurality of
`image areas, a white balance fine adjustment coefficient cal
`culation circuit that calculates white balance fine adjustment
`coefficients using image data within an area set in relation
`with the one image area selected by the image area selection
`apparatus, among the image data having undergone the white
`balance adjustment output by the white balance adjustment
`circuit, and a white balance fine adjustment circuit that per
`forms white balance fine adjustment using the white balance
`fine adjustment coefficients calculated at the white balance
`fine adjustment coefficient calculation circuit.
`If the digital camera is provided with a focal point detection
`device that detects the state of focal adjustment relative to the
`subject for each of a preset plurality of focal point detection
`areas and a focal point detection area selection apparatus that
`selects one of the plurality of focal point detection areas based
`upon focal adjustment statuses, the white balance fine adjust
`ment coefficients are calculated by selecting image data in an
`image area related to the focal point detection area selected by
`the focal point detection area selection apparatus.
`The image processing performed in the digital camera may
`be executed on a computer. The program stored in a storage
`medium for this purpose implements white balance adjust
`ment processing in which white balance adjustment is per
`formed on an image captured at the image-capt