I IIIII IIIIIIII Ill lllll lllll lllll lllll lllll lllll lllll lllll 111111111111111111
`US008059177B2
`
`c12) United States Patent
`Nakano et al.
`
`(IO) Patent No.:
`(45) Date of Patent:
`
`US 8,059,177 B2
`*Nov. 15, 2011
`
`(54) ELECTRIC CAMERA
`
`(56)
`
`References Cited
`
`(75)
`
`Inventors: Takahiro Nakano, Hitachinaka (JP);
`Ryuji Nishimura, Yokohama (JP);
`Toshiro Kinugasa, Hiratsuka (JP)
`
`(73) Assignee: Hitachi, Ltd., Tokyo (JP)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 2149 days.
`
`This patent is subject to a terminal dis(cid:173)
`claimer.
`
`(21) Appl. No.: 10/660,710
`
`(22) Filed:
`
`Sep.12,2003
`
`(65)
`
`Prior Publication Data
`
`US 2004/0046884 Al
`
`Mar. 11, 2004
`
`Related U.S. Application Data
`
`(62) Division of application No. 09/520,836, filed on Mar.
`8, 2000, now Pat. No. 6,765,616.
`
`(30)
`
`Foreign Application Priority Data
`
`Jan. 11, 2000
`
`(JP) ..................................... 00-006064
`
`(51)
`
`Int. Cl.
`H04N 5/335
`(2011.01)
`(52) U.S. Cl. ......................... 348/294; 348/322; 348/312
`(58) Field of Classification Search ........................ None
`See application file for complete search history.
`
`U.S. PATENT DOCUMENTS
`4,054,915 A *
`10/1977 Sugihara ....................... 348/283
`5,170,249 A *
`12/ 1992 Ohtsubo et al. . . . . . . . . . . . . 348/222 .1
`5,187,569 A *
`2/1993 Tani .............................. 348/277
`5,828,406 A *
`10/1998 Parulski et al.
`............ 348/220.1
`6,195,125 Bl*
`2/2001 Udagawa et al.
`.......... 348/222.1
`6,661,451 Bl*
`12/2003 Kijima et al. .............. 348/220.1
`7,154,539 Bl*
`12/2006 Nishimura et al.
`........ 348/220.1
`7,403,226 B2 *
`7/2008 Nakano et al. ................ 348/294
`
`FOREIGN PATENT DOCUMENTS
`JP
`04-323973
`11/1992
`JP
`9-270959
`10/1997
`JP
`11-004456
`1/1999
`JP
`11-187306
`7 /1999
`JP
`11-355665
`12/1999
`* cited by examiner
`Primary Examiner - Luong T Nguyen
`(74) Attorney, Agent, or Firm -Antonelli, Terry, Stout &
`Kraus, LLP.
`
`ABSTRACT
`(57)
`An electric camera includes an image sensing device with a
`light receiving surface having N vertically arranged pixels
`and an arbitrary number of pixels arranged horizontally, N
`being equal to or more than three times the number of effec(cid:173)
`tive scanning lines M of a display screen of a television
`system, a driver to drive the image sensing device to vertically
`mix or cull signal charges accumulated in individual pixels of
`K pixels to produce, during a vertical effective scanning
`period of the television system, a number of lines of output
`signals which corresponds to 1/K the number of vertically
`arranged pixels N of the image sensing device, K being an
`integer equal to or less than an integral part of a quotient ofN
`divided by M, and a signal processing unit having a function
`of generating image signals by using the output signals of the
`image sensing device.
`6 Claims, 8 Drawing Sheets
`
`14
`
`15
`
`MODE
`SELECTOR
`SWITCH
`
`RECORD
`BUTION
`
`11
`
`CONTROL
`CIRCUIT
`
`RECORDING
`UNIT
`
`10
`
`16a
`
`GYRO
`SENSOR
`
`GYRO
`SENSOR
`
`17
`
`IMAGE
`UNSTABILITY
`DECISION
`CIRCUIT
`
`16b
`
`4
`
`DRIVE
`CIRCUIT
`
`I\ IMAGE
`
`SENSING
`DEVICE
`
`~ I
`
`3
`LENS 2 APERTURE
`
`GAIN
`ADJUST
`CIRCUIT
`
`AID
`
`SIGNAL
`PROCESSING
`CIRCUIT
`
`VERTICAL
`INTER(cid:173)
`POLATION
`CIRCUIT
`
`HORIZONTAL
`INTER(cid:173)
`POLATION
`CIRCUIT
`
`ENCODER
`CIRCUIT
`
`D/A
`
`OUTPUT
`
`5
`
`6
`
`7
`
`8
`
`9
`
`12
`
`13
`
`Olympus, Exhibit 1002
`
`

`

`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`FIG.1
`
`13
`7
`
`D/A ~OUTPUT
`
`I-+
`
`12
`(
`
`9
`
`8
`
`7
`
`6
`
`5
`
`CIRCUIT
`POLATION ri CIRCUIT
`I L I ENCODER
`INTER·
`HORIZONTAL
`
`'
`
`CIRCUIT
`POLATION
`
`INTER-
`
`VERTICAL
`
`PROCESSING
`
`CIRCUIT
`
`SIGNAL
`
`AID
`
`~
`
`~
`
`CIRCUIT
`ADJUST
`
`GAIN
`
`r,
`
`J
`
`CONTROL H RECORDING 1-1 O
`
`' -
`
`UNIT
`
`CIRCUIT
`
`11 \_, .1...t ____;t..____t I
`
`I
`
`BUITON
`RECORD
`
`t
`15
`
`SWITCH
`
`SELECTOR
`
`MODE
`t
`14
`
`UNSTABILITY
`
`IMAGE
`
`CIRCUIT
`DECISION
`
`(
`17
`
`-....l = N
`
`00 = UI
`d r.,;_
`
`-....l
`""'"'
`'"'..c
`
`0 ....
`....
`.....
`rJJ =(cid:173)
`
`('D
`('D
`
`QO
`
`~ ....
`z 0
`
`....
`0 ....
`
`N
`'"Ul
`
`LENS 2 APERTURE
`
`3
`
`SENSING
`
`CIRCUIT
`DRIVE
`
`1 \
`I DEVICE
`(\ I IMAGE
`•
`
`t +
`4
`
`16b
`(
`
`i-.
`
`f-.
`
`SENSOR
`GYRO
`
`SENSOR
`GYRO
`(
`16a
`
`Olympus, Exhibit 1002
`
`

`

`U.S. Patent
`
`Nov. 15, 2011
`
`Sheet 2 of 8
`
`US 8,059,177 B2
`
`FIG.2
`
`V2 V3
`
`OUTPUT
`
`34
`
`..........
`
`33
`
`FIG.4
`
`- 1 A FIELD (n+ 1 )TH LINE
`
`I
`I
`; .. B FIELD nTH LINE
`
`I
`I
`
`-' A FIELD nTH LINE
`
`Olympus, Exhibit 1002
`
`

`

`-....l = N
`
`"""'
`-....l
`_,.-...c
`_,.0/0 = UI
`d r.,;_
`
`~
`
`('D
`('D
`
`0 ....
`.....
`rJJ =(cid:173)
`
`QO
`
`~ ....
`z 0
`
`....
`0 ....
`
`N
`"'Ul
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`VERTICAL BLANKING PERIOD
`
`VERTICAL EFFECTIVE SCANNING PERIOD
`
`VERTICAL BLANKING PERIOD
`
`VERTICAL SCANNING PERIOD
`
`-1 · .,
`
`T6
`
`T4
`
`,.
`
`SCANNING PERIOD
`
`HORIZONTAL
`BLANKING PERIOD
`HORIZONTAi!
`
`TS
`
`T3
`
`__ Il_ ________________ _flJ1Jl_J"l_ ____________________ n.__~ __ rl_
`
`--u----------------~--------------------u-~--~
`
`__ Il_ ________________ _flJ1Jl_J"l_ ____________________ n.__~ __ rl_
`
`T1
`
`T2
`
`FIG.3
`
`·-----
`
`-
`
`LOW LEVEL
`
`V
`
`LOW LEVEL
`
`[
`
`3 ( MIDDLE LEVEL rL Jl
`V2 MIDDLE LEVEL u-u
`( MIDDLE LEVEL n._
`
`-
`
`HIGH LEVEL
`
`LOW LEVEL
`
`1
`
`V
`
`Olympus, Exhibit 1002
`
`

`

`U.S. Patent
`
`Nov. 15, 2011
`
`Sheet 4 of 8
`
`US 8,059,177 B2
`
`FIG.5
`
`AREAS
`
`AREA A
`
`1
`960 J
`
`FIG.6
`
`-AREAA
`
`-AREAS
`
`,,
`
`I
`I
`I
`
`;..
`
`- -A FIELD GRAVITY CENTER OF (n+1)TH LINE
`
`-------· B FIELD GRAVITY CENTER OF nTH LINE
`
`I
`I
`
`I _, - -A FIELD GRAVITY CENTER OF nTH LINE
`
`Olympus, Exhibit 1002
`
`

`

`-....l = N
`
`00 = UI
`d r.,;_
`
`"""'
`-....l
`'"'..c
`
`0 .....
`Ul
`.....
`rJJ =(cid:173)
`
`('D
`('D
`
`QO
`
`~ ....
`z 0
`
`....
`0 ....
`
`N
`'"Ul
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`OUTPUT
`
`~ DIA
`
`-• CIRCUIT
`ENCODER
`
`HORIZONTAL
`
`CIRCUIT
`POLATION
`
`INTER-
`
`....
`
`CIRCUIT
`POLATION
`
`INTER-
`
`VERTICAL
`
`PROCESSING ....
`
`SIGNAL
`
`CIRCUIT
`
`,.
`
`'
`
`J
`
`I
`
`r-10
`
`RECORDING
`
`CIRCUIT ~ UNIT
`CONTROL
`'
`
`11 ""\.
`
`I
`
`BUTION
`RECORD
`
`SELECTOR
`
`SWITCH
`
`MODE
`
`VIEW ANGLE
`
`SWITCH
`CHANGE
`
`I
`
`FIG.7
`
`15
`
`14
`
`18
`
`.... AID f-+
`
`CIRCUIT
`ADJUST
`
`GAIN
`
`....
`
`.~
`
`CIRCUIT
`DRIVE
`
`t +
`4
`
`I
`
`1 6b
`(
`
`UNSTABILITY
`
`CIRCUIT
`DECISION
`
`IMAGE
`
`l ' t
`
`f-+
`
`f-+
`
`SENSOR
`GYRO
`
`SENSOR
`GYRO
`t
`
`10a
`
`13
`
`12
`)
`
`9
`)
`
`8
`)
`
`7
`)
`
`6
`)
`
`5
`)
`
`2 APERTURE
`
`3
`
`\
`
`LENS
`
`1
`
`)
`
`SENSING
`
`IMAGE
`
`\) I DEVICE
`•
`
`I
`
`I'
`
`Olympus, Exhibit 1002
`
`

`

`U.S. Patent
`
`Nov. 15, 2011
`
`Sheet 6 of 8
`
`US 8,059,177 B2
`
`FIG.8
`
`~,
`- - -A FIELD GRAVITY CENTER OF (n+1)TH LINE
`l .. --------- B FIELD GRAVITY CENTER OF nTH LINE
`r
`)
`A FIELD GRAVITY CENTER OF nTH LINE
`
`FIG.9
`
`AREA A
`
`AREAS
`
`AREAC
`
`r 1200 D
`_L....____
`
`AREA A : 960HIGH x 1280WIDE
`
`AREA B : 720HIGH x 960WIDE
`
`AREA C : 480HIGH x 640WIDE
`
`I'""" 4t-----1600------1 .. ~I
`
`Olympus, Exhibit 1002
`
`

`

`U.S. Patent
`
`Nov. 15, 2011
`
`Sheet 7 of 8
`
`US 8,059,177 B2
`
`32
`V1 V2 V3 V4 VS V
`6
`
`-
`
`,___
`
`I
`
`30
`3
`~\1
`ct -
`
`IC[
`
`-r
`
`FIG.10
`
`-
`fvc
`
`IYC
`
`-
`IGC
`
`IGC
`
`IGC
`. ,__
`.
`.
`.
`.
`.
`IGC
`
`IGC
`
`IGI
`
`ICC . -. . . . .
`
`ICC
`
`let
`
`IC!
`
`. . . . . . . . . .
`
`IYC
`. ,__
`.
`.
`.
`.
`.
`
`IYC
`
`IYt
`iYl
`
`~
`33
`
`FIG.12
`
`~
`
`IYC
`
`IYC
`
`,__
`
`IYC . . . . . .
`
`IYC
`
`Ive
`
`IYC
`
`OUT(cid:173)
`PUT
`
`34
`
`- - - - nTH LINE
`
`- - - - (n+1)TH LINE
`
`- - - - (n+2)TH LINE
`
`- - - - nTH LINE
`- - - - (n'+1)TH LINE
`- - - - (n'+2)TH LINE
`- - - - (n'+3)TH LINE
`- - - - (n'+4)TH LINE
`
`(BEFORE INTERPOLATION)
`
`(AFTER INTERPOLATION)
`
`FIG.13A
`
`FIG.138
`
`Ye w Cy
`Ye w Cy
`Ye w Cy
`Ye w Cy
`Ye w Cy
`
`R G B
`R G B
`R G B
`R G B
`R G B
`
`Olympus, Exhibit 1002
`
`

`

`-....l = N
`
`00 = UI
`d r.,;_
`
`"""'
`-....l
`'"'..c
`
`0 ....
`QO
`.....
`rJJ =(cid:173)
`
`('D
`('D
`
`QO
`
`~ ....
`z 0
`
`....
`0 ....
`
`N
`'"Ul
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`~
`
`VERTICAL BLANKING PERIOD
`
`VERTICAL SCANNING PERIOD
`
`VERTICAL EFFECTIVE SCANNING PERIOD
`
`VERTICAL BLANKING PERIOD
`
`~
`
`--u----------------~----------------------u----~-~---~
`--u----------------~---------------------~----u--~--~
`
`--u-----------------~---------------------~----u--~---ir-
`-u-----------------------u------------------------~---u---~--ir-
`
`n ___ n __ ___ fl ________________ _____Il____ _____________________ .fl. ___ f"l _ _____J-iJ"1 ___ f"l
`n ___ ,L
`___ fl ________________ _____Il____ _____________________ .fl. ___ fl_~ ___ f1
`1r---f
`1r--u---
`r--u--
`11--u--(cid:173)
`
`n ~
`.1 .. I
`
`1.
`
`SCANNING PERIOD
`
`HORIZONTAL
`BLANKING PERIOD
`HORIZONTAL
`
`TS
`
`T3
`
`VS MIDDLE LEVEL
`
`[
`
`LOWLEVEL
`MIDDLE LEVEL
`LOW LEVEL
`
`(
`
`VG
`
`V3 MIDDLE LEVEL
`
`HIGH LEVEL
`LOW LEVEL
`
`MIDDLE LEVEL
`LOW LEVEL
`
`MIDDLE LEVEL
`
`HIGH LEVEL
`
`(
`V2
`
`[
`
`V1
`
`HIGH LEVEL
`LOW LEVEL
`
`MIDDLE LEVEL
`LOW LEVEL
`
`(
`V4
`
`[
`
`FIG.11
`
`T2
`
`Olympus, Exhibit 1002
`
`

`

`US 8,059,177 B2
`
`1
`ELECTRIC CAMERA
`
`CROSS REFERENCE TO RELATED
`APPLICATION
`
`This is a divisional ofU.S. application Ser. No. 09/520,836,
`filed Mar. 8, 2000 now U.S. Pat. No. 6,765,616, the subject
`matter of which is incorporated by reference herein.
`
`BACKGROUND OF THE INVENTION
`
`The present invention relates to a photography related to
`video cameras, camcorders, digital still cameras and others
`using a solid-state image sensing device, and more particu(cid:173)
`larly to an electric camera using a solid-state image sensing
`device with a large number of pixels.
`Electric cameras using solid-state image sensors such as
`CCDs (charge-coupled devices) include a so-called video
`camera or camcorder for taking moving images and a so(cid:173)
`called digital still camera for taking still images. In recent
`years, video cameras with a still image taking function and
`digital still cameras with a moving image taking function
`have become available.
`In a video camera to photograph moving images, it is
`generally assumed that the video is viewed on a display such
`as television monitor and thus the camera is designed to
`produce output signals conforming to a television system
`such as NTSC and PAL. Therefore, the effective number of
`vertically arranged pixels or picture elements on the image
`sensing device used in such a camera is determined to enable
`television signals to be generated. The NTSC system, for
`example, performs interlaced scanning on two fields, each of
`which has an effective scanning line number of about 240
`lines (the number of scanning lines actually displayed on the
`monitor which is equal to the number of scanning lines in the
`vertical blanking period subtracted from the total number of
`scanning lines in each field). To realize this, the image sensing
`device has about 480 pixel rows as the standard effective
`number of vertically arranged pixels. That is, the signals of
`two vertically adjoining pixels in each field are mixed
`together inside or outside the image sensing device to gener(cid:173)
`ate about 240 scanning lines, and the combinations of pixels
`to be cyclically mixed together are changed from one field to
`another to achieve the interlaced scanning.
`Some image sensing devices to take moving images
`according to the NTSC system have an area of pixels for
`image stabilization added to the area of effective pixel area,
`thus bringing the effective number of vertically arranged
`pixels to about 480 or more. In this case, an area beyond 480th
`pixels is read out at high speed during the vertical blanking 50
`period and therefore the signals thus read out are not used as
`effective signals. Therefore, the video signals can only be
`generated from those signals coming from the area of about
`480 vertically arranged pixels. When such a camera is used to
`photograph a still image, it is relatively easy to generate a 55
`static image signal conforming to, for example, JPEG (Joint
`Photographic Expert Group) from the signals coming from
`the same pixel area that is used to take a moving image. A
`problem remains, however, that the number of vertically
`arranged pixels obtained is limited to around 480, making it 60
`impossible to produce more detailed static image signals.
`In a camera having an image sensing device with the area of
`pixels for image stabilization mentioned above, a method of
`alleviating this problem may involve using the entire area of
`effective pixels including the area of image stabilization pix- 65
`els in photographing a still image. Even when photographing
`a still image, however, the photographed image needs to be
`
`2
`monitored for check and, for that purpose, it is necessary to
`generate signals conforming to the television system from
`signals read out from all effective pixels.
`An example of such a conventional camera has been pro-
`5 posed in JP-A-11-187306. In the camera disclosed in this
`publication, signals from all the effective pixels are read out
`taking two or more times the field period of the television
`system, stored in a memory means such as a field memory,
`and then subjected to interpolation processing for transfor-
`lO mation into signals conforming to the field cycle and hori(cid:173)
`zontal scan cycle of television.
`This conventional camera, however, requires a large pro(cid:173)
`cessing circuit, such as field memory, for signal conversion.
`15 Another drawback is that the image sensing device readout
`cycle is a plurality of times the field cycle, degrading the
`dynamic resolution. Even with the use of this circuit, the
`number of pixels obtained as the static image signals is lim(cid:173)
`ited to the number of effective pixels used for moving videos
`20 plus the area of image stabilization pixels.
`In a digital still camera designed for taking still images,
`there has been a trend in recent years toward an increasing
`number of pixels used on the moving video image sensing
`device in order to obtain higher resolution static image sig-
`25 nals. When taking a moving image or monitoring the video, it
`is necessary to generate signals that conform to the television
`system. The number of pixels on such an image sensing
`device, however, does not necessarily match the number of
`scanning lines of the television system and therefore some
`30 form of conversion means is required.
`The conversion means may involve, as in the video camera
`with the area of image stabilization pixels, reading out signals
`from the image sensing device taking a longer time than the
`field cycle and interpolating them to generate television sig-
`35 nals. This method has, in addition to the problem described
`above, a drawback that the readout cycle increases as the
`number of pixels increases, further degrading the dynamic
`resolution.
`To mitigate this problem, JP-A-9-270959 discloses an
`40 apparatus which mixes together or culls the pixel signals
`inside the image sensing device to reduce the number of
`signals to be read and therefore the read cycle. Although this
`apparatus alleviates the problem of the degraded dynamic
`resolution, it requires a large processing circuit such as field
`45 memory to perform time-axis transformation to generate sig(cid:173)
`nals conforming to the television system and the image sens(cid:173)
`ing device itself needs to have a special structure for perform(cid:173)
`ing desired mixing and culling.
`
`SUMMARY OF THE INVENTION
`
`The present invention relates to a photography of video
`cameras, camcorders, digital still cameras and others using a
`solid-state image sensing device, and more particularly to an
`electric camera using a solid-state image sensing device with
`a large number of pixels.
`The conventional electric cameras, as described above,
`have drawbacks that when taking a still picture by using a
`video camera, the number of pixels is not sufficient and that
`when taking a moving image with a still camera, the associ(cid:173)
`ated circuit inevitably increases and the dynamic image qual(cid:173)
`ity deteriorates. Taking both moving and static images of
`satisfactory quality with a single camera is difficult to
`achieve. In addition to solving the above problems, to obtain
`good dynamic picture quality by using an image sensing
`device having a large number of pixels intended for still
`images requires extracting a pixel area that is used to realize
`
`Olympus, Exhibit 1002
`
`

`

`US 8,059,177 B2
`
`4
`FIGS. 13A and 13B are schematic diagrams showing the
`arrangement of color filters in the image sensing device in a
`fourth embodiment of the electric camera according to the
`present invention.
`
`DESCRIPTION OF THE EMBODIMENTS
`
`3
`an image stabilizing function. The conventional art and cam(cid:173)
`eras do not offer a means to accomplish this function.
`An object of the present invention is to provide an electric
`camera that solves these problems and which uses an image
`sensing device with a sufficient number of pixels for still 5
`images and enables taking of highly detailed still images and
`a moving video taking with reduced image quality degrada(cid:173)
`tion without increasing circuitry such as field memory. It is
`also an object of the present invention to provide an electric
`camera that can also realize the image stabilizing function.
`According to one aspect of this invention, the electric cam(cid:173)
`era to realize the above objectives has: an image sensing
`device with a light receiving surface having N vertically
`arranged pixels and an arbitrary number of pixels arranged 15
`horizontally, N being equal to or more than three times the
`number of effective scanning lines M of a display screen of a
`television system; a driver to drive the image sensing device
`to vertically mix or cull signal charges accumulated in indi(cid:173)
`vidual pixels of every K pixels to produce a number oflines of 20
`output signals which corresponds to the number of effective
`scanning lines M, K being at least one of integers equal to or
`less than an integral part of a quotient of N divided by M ( a
`number of lines of output signals corresponds to 1/K the
`number of vertically arranged pixels N of the image sensing 25
`device); and a signal processing unit to generate image sig(cid:173)
`nals by using the output signals of the image sensing device.
`As explained above, since this invention eliminates the
`limit on the number of vertically arranged pixels, an electric
`camera can be provided which enables taking of highly 30
`detailed still images and a satisfactory moving video taking
`by using an image sensing device with a large enough pixel
`number even for still images.
`
`10
`
`Now embodiments of the present invention will be
`described by referring to the accompanying drawings. FIG. 1
`is a block diagram showing the configuration of one embodi(cid:173)
`ment of an electric camera according to the invention.
`In FIG. 1, reference number 1 represents a lens, 2 an
`aperture, 3 an image sensing device, 4 a drive circuit, 5 a gain
`adjust circuit, 6 an analog-digital (AID) conversion circuit, 7
`a signal processing circuit, 8 a vertical interpolation circuit to
`perform interpolation in a vertical direction, 9 a horizontal
`interpolation circuit to perform interpolation in a horizontal
`direction, 10 a recording unit including recording media such
`as magnetic tape, semiconductor memory and optical disk to
`record a video signal, 11 a control circuit to control these
`constitutional elements according to the operating state, 12 an
`encoder circuit to modulate the video signal into a standard
`television signal, 13 a digital-analog (D/A) conversion cir(cid:173)
`cuit, 14 a mode selector switch to change over the operation
`mode between the moving video taking and the still image
`taking, 15 a record button to start or stop the recording, 16a
`and 16b gyro sensors to detect vertical image-unstability and
`lateral image-unstability, respectively, and 17 an image-un(cid:173)
`stability decision circuit to determine the image-instability
`from signals output from the gyro sensors.
`In the above configuration, light coming from the lens 1
`through the aperture 2 is focused on a light receiving surface
`of the image sensing device 3 where it is converted into an
`electric signal. In this embodiment the image sensing device
`35 3 is of a CCD type. FIG. 2 shows the structure of this image
`sensing device 3. In FIG. 2, denoted 30 are pixels each formed
`of a photodiode, which are arranged horizontally and verti(cid:173)
`cally in a grid pattern. On these grid-arrayed pixels three types
`of color filters that pass yellow (Ye), green (G) and cyan (Cy),
`40 respectively, are arranged in such a way that the combination
`of these three colors is repeated horizontally every three pix(cid:173)
`els and that the filters of the same colors are lined vertically in
`so-called vertical stripes. Although an arbitrary number of
`pixels may be used, this embodiment has an array of 1200
`45 pixels vertically and 1600 pixels horizontally. A vertical
`transferunit 32 is a CCD which is driven by three phase pulses
`Vl, V2, V3. This CCD has a three-gate structure in which
`each pixel corresponds to three phase pulses and thus can
`vertically transfer a signal charge of each pixel independently.
`50 Transfer gates 31 for transferring the charge of each pixel to
`the vertical transfer unit 3 2 are commonly connected to a gate
`of the vertical transfer unit 32 that corresponds to the V2
`pulse. An operation to transfer the charge from each pixel to
`the vertical transfer unit 32 in response to a peak value of the
`55 pulse applied to the commonly connected gate and an opera(cid:173)
`tion to transfer the charge vertically are performed separately.
`A horizontal transfer unit 33 horizontally transfers the
`charges supplied from the vertical transfer units 32 and out(cid:173)
`puts them successively through an output amplifier 34 from
`60 the output terminal.
`Referring back to FIG. 1, the operation performed when
`the moving video mode is selected by the mode selector
`switch 14 will be explained. The number of vertically
`arranged pixels on the image sensing device in this embodi-
`65 ment is 1200, so if the number of effective scanning lines in
`the field of the NTSC system is assumed to be 240 lines, then
`vertically mixing five pixels ( = 1200 pixel rows/240 scanning
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a block diagram showing the configuration of a
`first embodiment of an electric camera according to the
`present invention.
`FIG. 2 is a schematic diagram showing the structure of an
`image sensing device in the first embodiment of the electric
`camera of the invention.
`FIG. 3 is a drive pulse timing diagram in the first embodi(cid:173)
`ment of the electric camera of the invention.
`FIG. 4 is a schematic diagram showing a mixing operation
`in the first embodiment of the electric camera of the invention.
`FIG. 5 is a schematic diagram showing a readout area in the
`first embodiment of the electric camera of the invention.
`FIG. 6 is a schematic diagram showing a mixing operation
`in the first embodiment of the electric camera of the invention.
`FIG. 7 is a block diagram showing the configuration of a
`second embodiment of an electric camera according to the
`present invention.
`FIG. 8 is a schematic diagram showing a mixing operation
`in the second embodiment of the electric camera of the inven(cid:173)
`tion.
`FIG. 9 is a schematic diagram showing a readout area in the
`second embodiment of the electric camera of the invention.
`FIG.10 is a schematic diagram showing the structure of an
`image sensing device in a third embodiment of the electric
`camera according to the present invention.
`FIG. 11 is a drive pulse timing diagram in the third embodi(cid:173)
`ment of the electric camera of the invention.
`FIG. 12 is a schematic diagram showing an interpolation
`operation in the third embodiment of the electric camera of
`the invention.
`
`Olympus, Exhibit 1002
`
`

`

`US 8,059,177 B2
`
`6
`two rows between the two fields, thus allowing the interlaced
`scanning to be performed as shown in FIG. 4. (FIG. 4 shows
`the light receiving surface of the image sensing device and its
`relation to the displayed screen is vertically inverted.)
`Let us return to FIG. 1. The output signal from the image
`sensing device 3 is adjusted in gain by the gain adjust circuit
`5 and then converted by the AID conversion circuit 6 into a
`digital signal. The digital signal is then processed by the
`signal processing circuit 7 that performs color signal process-
`10 ing and luminance signal processing, such as generation of
`color signals, gamma correction, white balance processing
`and outline enhancement. The image sensing device in this
`embodiment has an array of vertical stripes of yellow (Ye),
`green (G) and cyan (Cy) color filters, so the color signals for
`15 Ye, G and Cy are obtained as a series of color points from one
`line of output signals at all times no matter how many pixels
`are vertically combined. From these color signals three pri(cid:173)
`mary color signals R, G, B can be obtained from the following
`calculations.
`
`5
`lines) can match the number of lines of output signals from
`the image sensing device to the number of effective scanning
`lines.
`However, in this embodiment, to realize the image stabi(cid:173)
`lizing function described later, four vertically arranged pixels 5
`are mixed together during motion image taking mode. When
`four vertically arranged pixels are to be cyclically mixed
`together, the signals from the ar a of 960 pixels (=240 scan(cid:173)
`ning linesx4 pixels) out of the 1200 vertically arranged pixels
`are used as effective signals and the remaining 240 pixels
`(=1200 (all pixels) 960 (effective pixels)) are not used for
`image forming. FIG. 3 shows the timing of a vertical drive
`pulse for the image sensing device in this operation mode,
`with Vl, V2 and V3 representing three phase drive pulses
`applied to each gate of the CCD or vertical transfer unit 32.
`In FIG. 3, in a period Tl included in the vertical blanking
`period, the drive pulse V2 is held high to transfer the signal
`charge accumulated in each pixel to under the V2 gate of the
`vertical CCD. Next, in a period T2, while the V2 pulse is still
`at middle level, the V3 pulse is raised from low level to middle 20
`level; next, while the V3 pulse is at middle level, the V2 pulse
`is changed from middle level to low level, after which the Vl
`pulse is changed from low level to middle level; next, while
`the Vl pulse is at middle level, the V3 pulse is changed from
`middle level to low level, after which the V2 pulse is changed 25
`to middle level and finally the Vl pulse is changed from
`middle level to low level. With this sequence of pulse opera(cid:173)
`tions, the signal charges under the V2 gate for one pixel row
`are transferred and held again under the V2 gate.
`By repeating this series of operations, the signal charges 30
`for a desired number of pixel rows can be transferred. In FIG.
`3, during a period T3 included in the vertical blanking period
`before the vertical effective scanning period (the vertical
`scanning period minus the vertical blanking period which
`corresponds to the actually displayed image) and during a 35
`period T4 included in the vertical blanking period after the
`vertical effective scanning period, the above transfer opera(cid:173)
`tion for one pixel row is repeated a total of 240 times to
`transfer the signal charges of the 240 pixel rows not used for
`image generation to the horizontal transfer unit 33 during the 40
`vertical blanking period. For example, if this transfer opera(cid:173)
`tion is performed 120 times during the period T3 and 120
`times during the period T 4, the signal charges from upper 120
`pixel rows and lower 120 pixel rows on the light receiving
`surface are transferred to the horizontal transfer unit 33 dur- 45
`ing the period T3 and period T4 within the vertical blanking
`period. During each of the subsequent periods TS and T6 in
`the vertical blanking period, the horizontal transfer unit 33 is
`driven for a predetermined period to output the charges trans(cid:173)
`ferred to the horizontal transfer unit 33 from the output ter- 50
`minal. These charges are not used as valid signals as they are
`output during the vertical blanking period.
`Next, in the vertical effective scanning period of FIG. 3, the
`above one-pixel-row transfer operation is performed four
`times during each horizontal blanking period to transfer the 55
`signal charges of four pixel rows to the horizontal transfer unit
`33 where they are mixed together. Then, during a horizontal
`effective scanning period (the horizontal scanning period
`minus the horizontal blanking period which corresponds to
`the actually displayed image), the horizontal transfer unit 33 60
`is driven to read out the signal charges from the horizontal
`transfer unit to produce an output signal conforming to the
`television system. If the above operation is performed on the
`A field and if, on the B field, the number of pixel rows
`transferred during the period T3 is set to 122 rows and that 65
`during the period T 4 is set to 118 rows, then the combination
`of four pixel rows to be cyclically mixed together shifts by
`
`G~G
`
`The R, G and B signals undergoes the white balance pro(cid:173)
`cessing and gamma correction processing in the signal pro(cid:173)
`cessing circuit 7 and are then converted into color difference
`signals such as R-Y, B-Y or U and V. The luminance signals
`and the color difference signals are then entered through the
`vertical interpolation circuit 8 into the horizontal interpola(cid:173)
`tion circuit 9. In this operation state the signals just pass
`through the vertical interpolation circuit 8 without being pro(cid:173)
`cessed. The horizontal interpolation circuit 9 performs inter(cid:173)
`polation on the signals in the horizontal direction.
`FIG. 5 shows the light receiving surface of the image
`sensing device. As described above, in the operating state of
`this embodiment, the signals read out during the vertical
`effective scanning period correspond to an area having 960 of
`the 1200 vertically arranged pixels and a horizontal width of
`1600 pixels, as shown shaded at A in FIG. 5. If the entire light
`receiving surface of the image sensing device has a 4-to-3
`(width to height) aspect ratio, the shaded area A is more
`laterally elongate than this aspect ratio. Hence, if the signals
`of all horizontal pixels of the light receiving surface are dis(cid:173)
`played, for example, on an NTSC standard television monitor
`with the 4-to-3 aspect ratio, the image displayed is com(cid:173)
`pressed horizontally and looks vertically elongate, compared
`with the original image. It is therefore necessary to output
`during the horizontal effective scanning period only those
`signals coming from a pixel area with the horizontal width
`conforming to the aspect ratio of the television system, as
`shown by a shaded area B. When the television system has an
`4-to-3 aspect ratio, the number of pixels in the horizontal
`width of the shaded area Bis 1280 (=960 (vertical effective
`pixels )x4/3 ).
`Returning back to FIG. 1, the horizontal interpolation cir(cid:173)
`cuit 9 performs interpolation processing on the signals from
`the horizontal 1280 pixels to expand the signals so that they
`can be output over the entire horizontal effective scanning
`period. It also performs switching among different clocks as
`required. With the above operation, an area having 960 pixels
`in height and 1280 pixels in width is demarcated from the
`light receiving surface as signals conforming to the television
`system. Then, the luminance signal and the color difference
`signal are encoded by the encoder circuit 12 into television
`signals, which are then converted by the DIA conversion
`
`Olympus, Exhibit 1002
`
`

`

`US 8,059,177 B2
`
`7
`circuit 13 into analog signals for output. When the recording
`is specified by the record button 15, the signals are recorded
`by the recording unit 10. At this time, the signals may be
`compressed in the MPEG (Moving Picture Expert Group)
`format.
`Next, the image stabilizing operation will be explained.
`Image-unstability information obtained by the gyro sensors
`l6a, l6b that detect vertical and horizontal image-unstabili(cid:173)
`ties is entered into the image-unstability decision circuit 17,
`which checks the received information for the amount and
`direction of the image-unstability and converts them into the
`number of pixels in vertical and horizontal directions on the
`light receiving surface of the image sensing device. Based on
`the converted pixel numbers, the position of an extracted area
`(effective pixel area) on the light receiving surface is shifted
`in a direction that cancels the image-unstability. This can
`correct the image-unstabil