Case 5:19-cv-00036-RWS Document 1-4 Filed 03/15/19 Page 1 of 19 PageID #: 103
`Case 5:19-cv-00036—RWS Document 1-4 Filed 03/15/19 Page 1 of 19 PageID #: 103
`
`EXHIBIT 3
`
`EXHIBIT 3
`
`

`

`Case 5:19-cv-00036-RWS Document 1-4 Filed 03/15/19 Page 2 of 19 PageID #: 104
`
`US0083394.93B2
`
`(12) United States Patent
`Nakano et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,339,493 B2
`*Dec. 25, 2012
`
`(54) ELECTRIC CAMERA
`
`(75) Inventors: Takahiro Nakano, Hitachinaka (JP);
`Ryuji Nishimura, Yokohama (JP);
`:
`s
`Toshiro Kinugasa, Hiratsuka (JP)
`
`(73) Assignee: Hitachi, Ltd., Tokyo (JP)
`(*) Notice:
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 136 days.
`This patent is Subject to a terminal dis-
`claimer.
`
`(21) Appl. No.: 12/845,266
`(22) Filed:
`Jul. 28, 2010
`(65)
`Prior Publication Data
`US 2010/02899.07 A1
`Nov. 18, 2010
`
`Related U.S. Application Data
`(60) Continuation of application No. 10/660,710, filed on
`Sep. 12, 2003, now Pat. No. 8,059,177, which is a
`division of application No. 09/520,836, filed on Mar. 8,
`2000, now Pat. No. 6,765,616.
`
`(30)
`
`Foreign Application Priority Data
`
`Jan. 11, 2000 (JP) ................................. 2000-006064
`(51) Int. Cl.
`(2011.01)
`H04N 5/335
`(52) U.S. Cl. ......... 34829. 348/32. 348/312. 348,273
`(58) Field of Classification Search ........................ None
`S
`lication file f
`let
`h historv.
`ee appl1cauon Ille Ior complete searcn n1Story
`References Cited
`
`(56)
`
`U.S. PATENT DOCUMENTS
`4,054,915 A 10/1977 Sugihara
`5,170,249 A 12/1992 Ohtsubo et al.
`5,187,569 A
`2, 1993 Tani
`
`5,828.406 A 10, 1998 Parulski et al.
`5,847,758. A * 12/1998 Iizuka ........................... 348,317
`6,195,125 B1
`2/2001 Udagawa et al.
`6,519,000 B1* 2/2003 Udagawa ................... 348,220.1
`6,661.451 B1
`12/2003 Kijima et al.
`6,765,616 B1* 7/2004 Nakano et al. ................ 348,322
`6,906,746 B2 *
`6/2005 Hijishiri et al. ............ 348,240.2
`7,154,539 B1
`12/2006 Nishimura et al.
`7.403.226 B2
`7/2008 Nakano et al.
`FOREIGN PATENT DOCUMENTS
`JP
`04-323973
`11, 1992
`JP
`9-270959
`10, 1997
`JP
`11-004456
`1, 1999
`JP
`11-1873.06
`7, 1999
`JP
`11-355.665
`12/1999
`* cited by examiner
`
`Primary Examiner — Luong TNguyen
`(74) Attorney, Agent, or Firm — Antonelli, Terry, Stout &
`Kraus, LLP.
`
`(57)
`
`ABSTRACT
`
`An electric camera includes an image sensing device with a
`light receiving Surface having N vertically arranged pixels
`and an arbitrary number of pixels arranged horizontally, N
`being equal to or more than three times the number of effec
`tive Scanning lines M of a display screen of a television
`system, a driver to drive the image sensing device to vertically
`mix or cull signal charges accumulated in individual pixels of
`K pixels to produce, during a vertical effective scanning
`period of the television system, a number of lines of output
`signals which corresponds to 1/K the number of vertically
`arranged pixels N of the image sensing device, K being an
`integer equal to or less than an integral part of a quotient of N
`divided by M, and a signal processing unit having a function
`of generating image signals by using the output signals of the
`image sensing device.
`
`14 Claims, 8 Drawing Sheets
`
`16a
`2
`YRC
`SENSOR
`
`Gyo
`SENSOR
`
`
`
`17
`2
`IMAGE
`UNSTABy
`ECSO
`CCUT
`
`1i
`1:
`MOE
`SLECTOR ER
`SWITCH
`
`11
`
`Comto
`
`RECORONS
`-> "G"h-10
`
`MAge
`8AM
`SiGNAL
`SENSING - AEIST - AD PRESSNG- E - EN NEP DA Hs oUTPUT
`CRCUIT
`WCe
`CCU
`CRC ji
`CRC
`
`WERTCA
`
`HORIZONTA
`
`N 3
`i
`ENs 2 APERTURE
`
`5
`
`6
`
`7
`
`8
`
`9
`
`2
`
`13
`
`

`

`V
`a
`apv
`Case 5:19-cv-00036-RWS Document 1-4 Filed 03/15/19 Page 3 of 19 PageID #: 105
`41
`3
`5O
`m5
`
`,mU
`
`pm
`
`12m.m9mM,
`
`100/fmmdtmwH%
`
`5.
`
`cM.m:EmPPGE
`
`0n0.e
`
`Aomkzoou2,SmEHSOEODc29280SmE3352:Wm2;R10:26%ummwmm
`m0F0m4mm04;moo:NF
`m2or:23toogommozamoomm
`
`.229m2205:22PmmoOQZmmez_mez_
`
`
`
`4<onumoI4<o:mm>
`
`
`
`
`Enemae26%tnomawSE30<5booms6:586:38ozammooE
`
`:aogo
`
`m.w.2EmmNom
`
`ME.
`
`Om>o
`
`momzmm
`
`Om>o
`
`mOmzmm
`
`am:
`
`m>zuo
`
`m0<§__F=40w=o
`
`mmsEma<mmzm:
`m/mQ>wD_0252mm
`
`
`
`

`

`Case 5:19-cv-00036-RWS Document 1-4 Filed 03/15/19 Page 4 of 19 PageID #: 106
`
`U.S. Patent
`
`Dec. 25, 2012
`
`Sheet 2 of 8
`
`US 8,339,493 B2
`
`FIG.2
`
`303
`
`32
`V1 V2 V3
`
`
`
`34
`
`33
`
`FIG.4
`
`A FIELD (n+1)TH LINE
`
`> BFIELD nTHINE
`
`A FIELD nTH LINE
`
`

`

`Case 5:19-cv-00036-RWS Document 1-4 Filed 03/15/19 Page 5 of 19 PageID #: 107
`
`U.S. Patent
`
`US 8,339,493 B2
`
`
`
`

`

`Case 5:19-cv-00036-RWS Document 1-4 Filed 03/15/19 Page 6 of 19 PageID #: 108
`
`U.S. Patent
`
`Dec. 25, 2012
`
`Sheet 4 of 8
`
`US 8,339,493 B2
`
`FIG.5
`
`AREAB
`
`AREA A
`
`
`
`
`
`1280
`1600
`
`FIG.6
`
`
`
`A FIELD GRAVITY CENTER OF (n+1)TH LINE
`
`~~ ~ - ~ ~ ~ ~ ~ ~ ~ .
`
`- - - - - - - - BFIELD GRAVITY CENTER OF nTH LNE
`
`A FIELD GRAVITY CENTER OF nTH LINE
`
`

`

`Case 5:19-cv-00036-RWS Document 1-4 Filed 03/15/19 Page 7 of 19 PageID #: 109
`
`U.S. Patent
`
`US 8,339,493 B2
`
`
`
`EOHAECI|
`
`

`

`Case 5:19-cv-00036-RWS Document 1-4 Filed 03/15/19 Page 8 of 19 PageID #: 110
`
`U.S. Patent
`
`Dec. 25, 2012
`
`Sheet 6 of 8
`
`US 8,339,493 B2
`
`FIG.8
`
`- A FIELD GRAVITY CENTER OF (n+1)TH LINE
`
`- -
`J
`
`w w m - w w BFIELD GRAVITY CENTER OF nTH LINE
`A FIELD GRAVITY CENTER OF nTH LINE
`
`FIG.9
`
`AREA A
`AREAB
`AREA C
`
`\
`
`1200
`
`AREA A : 96OHGH x 128OWDE
`
`AREA B : 72OHIGH x 96OWDE
`
`AREA C : 48OHIGH X 64OWDE
`
`— o—
`
`

`

`Case 5:19-cv-00036-RWS Document 1-4 Filed 03/15/19 Page 9 of 19 PageID #: 111
`
`U.S. Patent
`
`Dec. 25, 2012
`
`Sheet 7 of 8
`
`US 8,339,493 B2
`
`FIG.10
`
`T
`
`303 32 V1 V2 V3 V4 V5 V6
`G (CH
`Yi
`is
`a
`YH (GE CH
`(YH (GH CH
`
`l I
`
`Y
`Y
`|Y
`
`I
`
`C
`G
`Y
`Y
`C
`G
`Y
`Y
`F PUT
`34
`33
`
`FIG. 12
`
`nTH INE - n'TH LINE
`- (n'+1)TH LINE
`(n+1)TH LINE
`race - (n'+2)TH LINE
`- (n'+3)TH LINE
`(n+2)TH LINE - (n'+4)TH LINE
`
`(BEFORE INTERPOLATION)
`
`(AFTER INTERPOLATION)
`
`
`
`

`

`m
`0le
`aP9l:/u
`Case 5:19-cv-00036-RWS Document 1-4 Filed 03/15/19 Page 10 of 19 PageID #: 1122ll
`4lt
`R6
`C
`9
`g
`
`5Se0xU
`
`wtOnOwwmya
`1P.
`
`onDcemD
`w2,
`
`n2em
`
`8nf3008dtmwn%
`
`I.2#B3m9e4g9a9P393,100mm
`
`
`
`DOEmmOZEZSm._<o_._.mm>
`
`
`Gz_zz<0mm>_.5m_n_u_w._<O_.Em>
`
`
` DOEmEozizjm4<o_._.mm>
`
`4m>m._mEQQE
`
`.E>mjmien—=2Qm>
`
`4m>mfimime—2_.>
`
`d>m._IQ:
`
`.mSm:>>O._
`
`..m>m._>>04w©>
`
`\___/L____/
`CO
`>
`
`E
`
`N>
`
`4<._.ZONEOI
`
`
`
`09meoz_zz<om
`
`
`
`DOEmEOzuxzfim
`
`._<_.20N_mOI
`
`._<o_._.mm> oOEmm
`
`
`
`QO_mwn_oz_zz<ow
`
`
`
`._m_>m._IQ:
`
`4m>mjmime:
`
`.m_>m:304
`
`
`
`4m>mj>>O._
`
`
`
`4m>m4IQ:
`
`
`
`4m>m._Eco—.2
`
`
`
`4m>m3509.2
`
`
`
`4m>mj>>o._
`
`
`
`._m>m_._>>OA
`
`
`

`

`Case 5:19-cv-00036-RWS Document 1-4 Filed 03/15/19 Page 11 of 19 PageID #: 113
`
`US 8,339,493 B2
`
`1.
`ELECTRIC CAMERA
`
`CROSS REFERENCE TO RELATED
`APPLICATION
`
`This application is a continuation of U.S. application Ser.
`No. 10/660,710 Aug. 12, 2003 U.S. Pat. No. 8,059,177, and is
`related to U.S. application Ser. No. 10/660,711, filed Sep. 12,
`2003, now U.S. Pat. No. 7,403.226, issued Jul. 22, 2008, both
`of which are divisional applications of U.S. application Ser.
`No. 09/520,836, filed Mar. 8, 2000, now U.S. Pat. No. 6,765,
`616, issued Jul. 20, 2004, the subject matter of which is
`incorporated by reference herein.
`
`BACKGROUND OF THE INVENTION
`
`10
`
`15
`
`25
`
`30
`
`35
`
`The present invention relates to a photography related to
`Video cameras, camcorders, digital still cameras and others
`using a solid-state image sensing device, and more particu
`larly to an electric camera using a solid-state image sensing
`device with a large number of pixels.
`Electric cameras using solid-state image sensors such as
`CCDs (charge-coupled devices) include a so-called video
`camera or camcorder for taking moving images and a so
`called digital still camera for taking still images. In recent
`years, video cameras with a still image taking function and
`digital still cameras with a moving image taking function
`have become available.
`In a video camera to photograph moving images, it is
`generally assumed that the video is viewed on a display Such
`as television monitor and thus the camera is designed to
`produce output signals conforming to a television system
`such as NTSC and PAL. Therefore, the effective number of
`Vertically arranged pixels or picture elements on the image
`sensing device used in Such a camera is determined to enable
`television signals to be generated. The NTSC system, for
`example, performs interlaced scanning on two fields, each of
`which has an effective scanning line number of about 240
`lines (the number of scanning lines actually displayed on the
`monitor which is equal to the number of scanning lines in the
`vertical blanking period subtracted from the total number of
`scanning lines in each field). To realize this, the image sensing
`device has about 480 pixel rows as the standard effective
`number of Vertically arranged pixels. That is, the signals of
`45
`two vertically adjoining pixels in each field are mixed
`together inside or outside the image sensing device to gener
`ate about 240 Scanning lines, and the combinations of pixels
`to be cyclically mixed together are changed from one field to
`another to achieve the interlaced scanning.
`Some image sensing devices to take moving images
`according to the NTSC system have an area of pixels for
`image stabilization added to the area of effective pixel area,
`thus bringing the effective number of vertically arranged
`pixels to about 480 or more. In this case, an area beyond 480th
`pixels is read out at high speed during the vertical blanking
`period and therefore the signals thus read out are not used as
`effective signals. Therefore, the video signals can only be
`generated from those signals coming from the area of about
`480 vertically arranged pixels. When such a camera is used to
`photograph a still image, it is relatively easy to generate a
`static image signal conforming to, for example, JPEG (Joint
`Photographic Expert Group) from the signals coming from
`the same pixel area that is used to take a moving image. A
`problem remains, however, that the number of vertically
`65
`arranged pixels obtained is limited to around 480, making it
`impossible to produce more detailed Static image signals.
`
`40
`
`50
`
`55
`
`60
`
`2
`In a camera having an image sensing device with the area of
`pixels for image stabilization mentioned above, a method of
`alleviating this problem may involve using the entire area of
`effective pixels including the area of image stabilization pix
`els in photographing a still image. Even when photographing
`a still image, however, the photographed image needs to be
`monitored for check and, for that purpose, it is necessary to
`generate signals conforming to the television system from
`signals read out from all effective pixels.
`An example of Such a conventional camera has been pro
`posed in JP-A-11-187306. In the camera disclosed in this
`publication, signals from all the effective pixels are read out
`taking two or more times the field period of the television
`system, stored in a memory means such as a field memory,
`and then subjected to interpolation processing for transfor
`mation into signals conforming to the field cycle and hori
`Zontal scan cycle of television.
`This conventional camera, however, requires a large pro
`cessing circuit, such as field memory, for signal conversion.
`Another drawback is that the image sensing device readout
`cycle is a plurality of times the field cycle, degrading the
`dynamic resolution. Even with the use of this circuit, the
`number of pixels obtained as the static image signals is lim
`ited to the number of effective pixels used for moving videos
`plus the area of image stabilization pixels.
`In a digital still camera designed for taking still images,
`there has been a trend in recent years toward an increasing
`number of pixels used on the moving video image sensing
`device in order to obtain higher resolution static image sig
`nals. When taking a moving image or monitoring the video, it
`is necessary to generate signals that conform to the television
`system. The number of pixels on such an image sensing
`device, however, does not necessarily match the number of
`scanning lines of the television system and therefore some
`form of conversion means is required.
`The conversion means may involve, as in the video camera
`with the area of image stabilization pixels, reading out signals
`from the image sensing device taking a longer time than the
`field cycle and interpolating them to generate television sig
`nals. This method has, in addition to the problem described
`above, a drawback that the readout cycle increases as the
`number of pixels increases, further degrading the dynamic
`resolution.
`To mitigate this problem, JP-A-9-270959 discloses an
`apparatus which mixes together or culls the pixel signals
`inside the image sensing device to reduce the number of
`signals to be read and therefore the read cycle. Although this
`apparatus alleviates the problem of the degraded dynamic
`resolution, it requires a large processing circuit such as field
`memory to perform time-axis transformation to generate sig
`nals conforming to the television system and the image sens
`ing device itself needs to have a special structure for perform
`ing desired mixing and culling.
`
`SUMMARY OF THE INVENTION
`
`The present invention relates to a photography of video
`cameras, camcorders, digital still cameras and others using a
`Solid-state image sensing device, and more particularly to an
`electric camera using a Solid-state image sensing device with
`a large number of pixels.
`The conventional electric cameras, as described above,
`have drawbacks that when taking a still picture by using a
`video camera, the number of pixels is not sufficient and that
`when taking a moving image with a still camera, the associ
`ated circuit inevitably increases and the dynamic image qual
`ity deteriorates. Taking both moving and static images of
`
`

`

`Case 5:19-cv-00036-RWS Document 1-4 Filed 03/15/19 Page 12 of 19 PageID #: 114
`
`US 8,339,493 B2
`
`3
`satisfactory quality with a single camera is difficult to
`achieve. In addition to solving the above problems, to obtain
`good dynamic picture quality by using an image sensing
`device having a large number of pixels intended for still
`images requires extracting a pixel area that is used to realize
`an image stabilizing function. The conventional art and cam
`eras do not offer a means to accomplish this function.
`An object of the present invention is to provide an electric
`camera that solves these problems and which uses an image
`sensing device with a sufficient number of pixels for still
`images and enables taking of highly detailed still images and
`a moving video taking with reduced image quality degrada
`tion without increasing circuitry such as field memory. It is
`also an object of the present invention to provide an electric
`camera that can also realize the image stabilizing function.
`According to one aspect of this invention, the electric cam
`era to realize the above objectives has: an image sensing
`device with a light receiving surface having N vertically
`arranged pixels and an arbitrary number of pixels arranged
`horizontally, N being equal to or more than three times the
`number of effective scanning lines Mofa display screen of a
`television system; a driver to drive the image sensing device
`to vertically mix or cull signal charges accumulated in indi
`vidual pixels of every Kpixels to produce a number of lines of
`output signals which corresponds to the number of effective
`scanning lines M. K being at least one of integers equal to or
`less than an integral part of a quotient of N divided by M (a
`number of lines of output signals corresponds to 1/K the
`number of vertically arranged pixels N of the image sensing
`device); and a signal processing unit to generate image sig
`nals by using the output signals of the image sensing device.
`As explained above, since this invention eliminates the
`limit on the number of Vertically arranged pixels, an electric
`camera can be provided which enables taking of highly
`detailed still images and a satisfactory moving video taking
`by using an image sensing device with a large enough pixel
`number even for still images.
`
`4
`FIG. 12 is a schematic diagram showing an interpolation
`operation in the third embodiment of the electric camera of
`the invention.
`FIGS. 13A and 13B are schematic diagrams showing the
`arrangement of color filters in the image sensing device in a
`fourth embodiment of the electric camera according to the
`present invention.
`
`DESCRIPTION OF THE EMBODIMENTS
`
`Now embodiments of the present invention will be
`described by referring to the accompanying drawings. FIG. 1
`is a block diagram showing the configuration of one embodi
`ment of an electric camera according to the invention.
`In FIG. 1, reference number 1 represents a lens, 2 an
`aperture, 3 an image sensing device, 4 a drive circuit, 5 again
`adjust circuit, 6 an analog-digital (A/D) conversion circuit, 7
`a signal processing circuit, 8 a vertical interpolation circuit to
`perform interpolation in a vertical direction, 9 a horizontal
`interpolation circuit to perform interpolation in a horizontal
`direction, 10 a recording unit including recording media Such
`as magnetic tape, semiconductor memory and optical disk to
`record a video signal, 11 a control circuit to control these
`constitutional elements according to the operating State, 12 an
`encoder circuit to modulate the video signal into a standard
`television signal, 13 a digital-analog (D/A) conversion cir
`cuit, 14 a mode selector Switch to change over the operation
`mode between the moving video taking and the still image
`taking, 15 a record button to start or stop the recording, 16a
`and 16b gyro sensors to detect vertical image-unstability and
`lateral image-unstability, respectively, and 17 an image-un
`stability decision circuit to determine the image-instability
`from signals output from the gyro sensors.
`In the above configuration, light coming from the lens 1
`through the aperture 2 is focused on a light receiving Surface
`of the image sensing device 3 where it is converted into an
`electric signal. In this embodiment the image sensing device
`3 is of a CCD type. FIG. 2 shows the structure of this image
`sensing device 3. In FIG.2, denoted 30 are pixels each formed
`of a photodiode, which are arranged horizontally and verti
`cally in a grid pattern. On these grid-arrayed pixels three types
`of color filters that pass yellow (Ye), green (G) and cyan (Cy),
`respectively, are arranged in Such a way that the combination
`of these three colors is repeated horizontally every three pix
`els and that the filters of the same colors are lined vertically in
`so-called vertical stripes. Although an arbitrary number of
`pixels may be used, this embodiment has an array of 1200
`pixels vertically and 1600 pixels horizontally. A vertical
`transfer unit 32 is a CCD which is driven by three phase pulses
`V1,V2, V3. This CCD has a three-gate structure in which
`each pixel corresponds to three phase pulses and thus can
`Vertically transfera signal charge of each pixel independently.
`Transfer gates 31 for transferring the charge of each pixel to
`the vertical transfer unit 32 are commonly connected to a gate
`of the vertical transfer unit 32 that corresponds to the V2
`pulse. An operation to transfer the charge from each pixel to
`the vertical transfer unit 32 in response to a peak value of the
`pulse applied to the commonly connected gate and an opera
`tion to transfer the charge vertically are performed separately.
`A horizontal transfer unit 33 horizontally transfers the
`charges supplied from the vertical transfer units 32 and out
`puts them Successively through an output amplifier 34 from
`the output terminal.
`Referring back to FIG. 1, the operation performed when
`the moving video mode is selected by the mode selector
`switch 14 will be explained. The number of vertically
`arranged pixels on the image sensing device in this embodi
`
`10
`
`15
`
`25
`
`30
`
`35
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a block diagram showing the configuration of a
`first embodiment of an electric camera according to the
`present invention.
`FIG. 2 is a schematic diagram showing the structure of an
`image sensing device in the first embodiment of the electric
`camera of the invention.
`FIG. 3 is a drive pulse timing diagram in the first embodi
`ment of the electric camera of the invention.
`FIG. 4 is a schematic diagram showing a mixing operation
`in the first embodiment of the electric camera of the invention.
`FIG.5 is a schematic diagram showing a readout area in the
`first embodiment of the electric camera of the invention.
`FIG. 6 is a schematic diagram showing a mixing operation
`in the first embodiment of the electric camera of the invention.
`FIG. 7 is a block diagram showing the configuration of a
`second embodiment of an electric camera according to the
`present invention.
`FIG. 8 is a schematic diagram showing a mixing operation
`in the second embodiment of the electric camera of the inven
`tion.
`FIG.9 is a schematic diagram showing a readout area in the
`second embodiment of the electric camera of the invention.
`FIG. 10 is a schematic diagram showing the structure of an
`image sensing device in a third embodiment of the electric
`camera according to the present invention.
`FIG.11 is a drive pulse timing diagram in the third embodi
`ment of the electric camera of the invention.
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`

`

`Case 5:19-cv-00036-RWS Document 1-4 Filed 03/15/19 Page 13 of 19 PageID #: 115
`
`US 8,339,493 B2
`
`6
`transferred during the period T3 is set to 122 rows and that
`during the period T4 is set to 118 rows, then the combination
`of four pixel rows to be cyclically mixed together shifts by
`two rows between the two fields, thus allowing the interlaced
`scanning to be performed as shown in FIG. 4. (FIG. 4 shows
`the light receiving Surface of the image sensing device and its
`relation to the displayed screen is vertically inverted.)
`Let us return to FIG.1. The output signal from the image
`sensing device 3 is adjusted in gain by the gain adjust circuit
`5 and then converted by the A/D conversion circuit 6 into a
`digital signal. The digital signal is then processed by the
`signal processing circuit 7 that performs color signal process
`ing and luminance signal processing. Such as generation of
`color signals, gamma correction, white balance processing
`and outline enhancement. The image sensing device in this
`embodiment has an array of vertical stripes of yellow (Ye).
`green (G) and cyan (Cy) color filters, so the color signals for
`Ye, G and Cy are obtained as a series of color points from one
`line of output signals at all times no matter how many pixels
`are vertically combined. From these color signals three pri
`mary color signals R,G,B can be obtained from the following
`calculations.
`
`10
`
`15
`
`5
`ment is 1200, so if the number of effective scanning lines in
`the field of the NTSC system is assumed to be 240 lines, then
`vertically mixing five pixels (=1200 pixel rows/240 scanning
`lines) can match the number of lines of output signals from
`the image sensing device to the number of effective scanning
`lines.
`However, in this embodiment, to realize the image stabi
`lizing function described later, four vertically arranged pixels
`are mixed together during motion image taking mode. When
`four vertically arranged pixels are to be cyclically mixed
`together, the signals from the area of 960 pixels(=240 scan
`ning linesx4 pixels) out of the 1200 vertically arranged pixels
`are used as effective signals and the remaining 240 pixels
`(=1200 (all pixels)-960 (effective pixels)) are not used for
`image forming. FIG. 3 shows the timing of a vertical drive
`pulse for the image sensing device in this operation mode,
`with V1,V2 and V3 representing three phase drive pulses
`applied to each gate of the CCD or vertical transfer unit 32.
`In FIG. 3, in a period T1 included in the vertical blanking
`period, the drive pulse V2 is held high to transfer the signal
`charge accumulated in each pixel to under the V2 gate of the
`vertical CCD. Next, in a period T2, while the V2 pulse is still
`at middle level, the V3 pulse is raised from low level to middle
`level; next, while the V3 pulse is at middle level, the V2 pulse
`is changed from middle level to low level, after which the V1
`pulse is changed from low level to middle level; next, while
`the V1 pulse is at middle level, the V3 pulse is changed from
`middle level to low level, after which the V2 pulse is changed
`to middle level and finally the V1 pulse is changed from
`middle level to low level. With this sequence of pulse opera
`tions, the signal charges under the V2 gate for one pixel row
`are transferred and held again under the V2 gate.
`By repeating this series of operations, the signal charges
`for a desired number of pixel rows can be transferred. In FIG.
`3, during a period T3 included in the vertical blanking period
`before the vertical effective scanning period (the vertical
`scanning period minus the vertical blanking period which
`corresponds to the actually displayed image) and during a
`period T4 included in the vertical blanking period after the
`Vertical effective scanning period, the above transfer opera
`tion for one pixel row is repeated a total of 240 times to
`transfer the signal charges of the 240 pixel rows not used for
`image generation to the horizontal transfer unit 33 during the
`Vertical blanking period. For example, if this transfer opera
`tion is performed 120 times during the period T3 and 120
`times during the period T4, the signal charges from upper 120
`pixel rows and lower 120 pixel rows on the light receiving
`surface are transferred to the horizontal transfer unit 33 dur
`ing the period T3 and period T4 within the vertical blanking
`period. During each of the subsequent periods T5 and T6 in
`the vertical blanking period, the horizontal transfer unit 33 is
`driven for a predetermined period to output the charges trans
`ferred to the horizontal transfer unit 33 from the output ter
`minal. These charges are not used as valid signals as they are
`output during the vertical blanking period.
`Next, in the vertical effective scanning period of FIG.3, the
`above one-pixel-row transfer operation is performed four
`times during each horizontal blanking period to transfer the
`signal charges of four pixel rows to the horizontal transfer unit
`33 where they are mixed together. Then, during a horizontal
`effective Scanning period (the horizontal scanning period
`minus the horizontal blanking period which corresponds to
`the actually displayed image), the horizontal transfer unit 33
`is driven to read out the signal charges from the horizontal
`transfer unit to produce an output signal conforming to the
`television system. If the above operation is performed on the
`A field and if, on the B field, the number of pixel rows
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`The R, G and B signals undergoes the white balance pro
`cessing and gamma correction processing in the signal pro
`cessing circuit 7 and are then converted into color difference
`signals such as R-Y. B-Y or U and V. The luminance signals
`and the color difference signals are then entered through the
`vertical interpolation circuit 8 into the horizontal interpola
`tion circuit 9. In this operation state the signals just pass
`through the vertical interpolation circuit 8 without being pro
`cessed. The horizontal interpolation circuit 9 performs inter
`polation on the signals in the horizontal direction.
`FIG. 5 shows the light receiving surface of the image
`sensing device. As described above, in the operating state of
`this embodiment, the signals read out during the vertical
`effective scanning period correspond to an area having 960 of
`the 1200 vertically arranged pixels and a horizontal width of
`1600 pixels, as shown shaded at A in FIG. 5. If the entire light
`receiving Surface of the image sensing device has a 4-to-3
`(width to height) aspect ratio, the shaded area A is more
`laterally elongate than this aspect ratio. Hence, if the signals
`of all horizontal pixels of the light receiving surface are dis
`played, for example, on an NTSC standard television monitor
`with the 4-to-3 aspect ratio, the image displayed is com
`pressed horizontally and looks vertically elongate, compared
`with the original image. It is therefore necessary to output
`during the horizontal effective scanning period only those
`signals coming from a pixel area with the horizontal width
`conforming to the aspect ratio of the television system, as
`shown by a shaded area B. When the television system has an
`4-to-3 aspect ratio, the number of pixels in the horizontal
`width of the shaded area B is 1280 (=960 (vertical effective
`pixels)x4/3).
`Returning back to FIG. 1, the horizontal interpolation cir
`cuit 9 performs interpolation processing on the signals from
`the horizontal 1280 pixels to expand the signals so that they
`can be output over the entire horizontal effective scanning
`period. It also performs Switching among different clocks as
`required. With the above operation, an area having 960 pixels
`in height and 1280 pixels in width is demarcated from the
`light receiving Surface as signals conforming to the television
`
`

`

`Case 5:19-cv-00036-RWS Document 1-4 Filed 03/15/19 Page 14 of 19 PageID #: 116
`
`7
`system. Then, the luminance signal and the color difference
`signal are encoded by the encoder circuit 12 into television
`signals, which are then converted by the D/A conversion
`circuit 13 into analog signals for output. When the recording
`is specified by the record button 15, the signals are recorded
`by the recording unit 10. At this time, the signals may be
`compressed in the MPEG (Moving Picture Expert Group)
`format.
`Next, the image stabilizing operation will be explained.
`Image-unstability information obtained by the gyro sensors
`16a, 16b that detect vertical and horizontal image-unstabili
`ties is entered into the image-unstability decision circuit 17,
`which checks the received information for the amount and
`direction of the image-unstability and converts them into the
`number of pixels in vertical and horizontal directions on the
`light receiving Surface of the image sensing device. Based on
`the converted pixel numbers, the position of an extracted area
`(effective pixel area) on the light receiving surface is shifted
`in a direction that cancels the image-unstability. This can
`correct the image-unstability. The positional shifting of the
`extracted area is performed as follows. The shifting in the
`Vertical direction can be made by changing the number of
`pixel rows transferred during the periods T3 and T4 in FIG.3
`and the shifting in the horizontal direction made by changing
`the interpolation start position in the horizontal interpolation
`circuit 9.
`The operation during the moving video mode has been
`described above. Next, the operation performed when the
`static image mode is selected by the mode selector Switch 14
`will be explained.
`In the static image mode, too, until the recording is
`requested by the recordbutton 15, the camera outputs signals
`compatible with the television system to monitor the angle of
`view. Unlike the moving video photographing, all of the
`effective pixels on the image sensing device are used in this
`embodiment during the still image photographing to produce
`signals with as high a resolution as possible. Hence, during
`the monitoring the television signals need to be generated
`from the signals coming from the entire pixel area.
`The image sensing device of this embodiment has 1200
`Vertically arranged pixels, and the number of lines of output
`signals from the image sensing device can be made to match
`the number of effective scanning lines of NTSC system,
`which is assumed to have 240 scanning lines, by Vertically
`mixing five pixels (=1200/240). To make the image sensing
`device operate in this manner, the one-pixel-row transfer
`operation is performed five times during each horizontal
`blanking period in the vertical effective scanning period
`shown in the pulse timing diagram of FIG. 3. With this opera
`tion, the signal charges offive pixel rows can be mixed by the
`horizontal transfer unit 33. As for the transfer operations
`during the periods T3 and T4 in the vertical blanking period,
`because the interlaced scanning is carried out, only two pixel
`rows are transferred during the period T3 on the B field, with
`no transfer operations performed in other vertical blanking
`periods (In this e