`
`Confirmation No.: TBD
`
`Inventor(s)
`
`Serial No.
`
`Filed
`
`For
`
`Group
`
`Examiner
`
`Docket No.
`
`:
`
`:
`
`:
`
`:
`
`:
`
`:
`
`:
`
`T. NAKANO et al.
`
`To be assigned
`
`December 21, 2016
`
`ELECTRIC CAMERA
`
`To be assigned
`
`To be assigned
`
`ASA-9606—08
`
`Customer No.:
`
`24956
`
`PRELIMINARY AMENDMENT
`
`Mail Stop Amendments
`Commissioner for Patents
`PO. Box 1450
`
`Alexandria, VA 22313-1450
`
`Sir:
`
`December 21, 2016
`
`Prior to examination, please amend the above-identified application as follows.
`
`Amendments to the Specification begin on page 2;
`
`Remarks are included following the amendments.
`
`IPR2020-00597
`
`Apple EX1024 Page 1
`
`IPR2020-00597
`Apple EX1024 Page 1
`
`
`
`Serial No. To be assigned
`Preliminary Amendment filed December 21 , 2016
`
`ASA-9606-08
`
`IN THE SPECIFICATION:
`
`Please amend the first paragraph on page 1 of the specification following the
`
`heading "CROSS REFERENCE TO RELATED APPLICATION" as follows:
`
`CROSS REFERENCE TO RELATED APPLICATION
`
`This application is a continuation of U.S. patent application No.
`
`
`14/661 227 filed March 18 2015 which is a continuation of U.S. patent
`
`application No. 14/264,243, filed April 29, 2014, now U.S. Patent No.
`
`9,100,604, which is a continuation of U.S. application serial no. 13/681 ,495,
`
`filed November 20, 2012, now U.S. Patent No. 8,736,729, which is a
`
`continuation of U.S. application serial no. 12/845,266, filed July 28, 2010, now
`
`U.S. Patent No. 8,339,493, issued December 25, 2012, which is a
`
`continuation of U.S. application serial no. 10/660,710, filed September 12,
`
`2003, now U.S. Patent No. 8,059,177, issued November 15, 2011, and is
`
`related to U.S. application serial no. 10/660,711, filed September 12, 2003,
`
`now U.S. Patent No. 7,403,226, issued July 22, 2008, both of which are
`
`divisional applications of U.S. application Serial No. 09/520,836, filed March 8,
`
`2000, now U.S. Patent No. 6,765,616, issued July 20, 2004, the subject
`
`matter of all the above is incorporated by reference herein.
`
`Page 2 of 3
`
`|PR2020-00597
`
`Apple EX1024 Page 2
`
`IPR2020-00597
`Apple EX1024 Page 2
`
`
`
`Serial No. To be assigned
`Preliminary Amendment filed December 21 , 2016
`
`ASA-9606-08
`
`REMARKS
`
`Entry of the above amendments prior to examination is respectfully requested.
`
`No new matter is added by the amendments.
`
`Please charge any shortage in fees due in connection with the filing of this
`
`paper, or credit any overpayment of fees, to the deposit account of MATTING LY &
`
`MALUR, PC, Deposit Account No. 50-1417.
`
`Respectfully submitted,
`
`MATTINGLY & MALUR, PC
`
`/John R. Mattin l /
`
`John R. Mattingly, Reg. No. 30,293
`(703) 684-1 120
`
`Page 3 of 3
`
`|PR2020-00597
`
`Apple EX1024 Page 3
`
`IPR2020-00597
`Apple EX1024 Page 3
`
`
`
`
`CROSS REFERENCE TO RELATED APPLICATION
`
`_1_
`
`ELECTRIC CAMERA
`
`This application is a continuation of U.S. application serial no. 13/681,495,
`
`filed November 20, 2012, which is a continuation of U.S. application serial no.
`
`12/845,266, filed July 28, 2010, now U.S. Patent No. 8,339,493,
`
`issued December 25,
`
`2012, which is a continuation of U.S. application serial no. 10/660,710, filed
`
`September 12, 2003, now U.S. Patent No. 8,059,177,
`
`issued November 15, 2011, and is
`
`10
`
`related to U.S. application serial no. 10/660,711, filed September 12, 2003, now U.S.
`
`Patent No. 7,403,226,
`
`issued July 22, 2008, both of which are divisional applications
`
`of U.S. application Serial No. 09/520,836, filed March 8, 2000, now U.S. Patent No.
`
`6,765,616,
`
`issued July 20, 2004,
`
`the subject matter of all the above is incorporated
`
`by reference herein.
`
`15
`
`BACKGROUND OF THE INVENTION
`
`The present
`
`invention relates to a photography related to video cameras,
`
`camcorders, digital still cameras and others using a solid—state image sensing device,
`
`and more particularly to an electric camera using a solid—state image sensing device
`
`with a large number of pixels.
`
`20
`
`
`Electric cameras using solid—state image sensors such as CCDs
`
`(charge—coupled
`
`devices)
`
`include a soecalled video camera or camcorder for taking moving images and a
`
`so—called digital still camera for taking still images.
`
`In recent years, video cameras
`
`with a still image taking function and digital still cameras with a moving image
`
`taking function have become available.
`
`25
`
`In a video camera to photograph moving images, it is generally assumed that the
`
`video is viewed on a display such as television monitor and thus the camera is
`
`designed to produce output signals conforming to a television system such as NTSC and
`
`|PR2020—OO597
`
`Apple EX1024 Page 4
`
`IPR2020-00597
`Apple EX1024 Page 4
`
`
`
`_1b_
`
`PAL. Therefore,
`
`the effective number of vertically arranged pixels or picture elements
`
`on the image sensing device used in such a camera is determined to enable
`
`television signals to be generated. The NTSC system, for example, performs
`
`interlaced scanning on two fields, each of which has an effective scanning
`
`line number of about 240 lines (the number of scanning lines actually
`
`|PR2020-00597
`
`Apple EX1024 Page 5
`
`IPR2020-00597
`Apple EX1024 Page 5
`
`
`
`-2—
`
`displayed on the monitor which is equal to the number of
`
`scanning lines in the vertical blanking period subtracted
`
`from the total number of scanning lines in each field).
`
`To
`
`realize this,
`
`the image sensing device has about 480 pixel
`
`rows as the standard effective number of vertically
`
`arranged pixels. That is, the signals of two vertically
`
`adjoining pixels in each field are mixed together inside or
`
`outside the image sensing device to generate about 240
`
`scanning lines, and the combinations of pixels to be
`
`10
`
`cyclically mixed together are changed from one field to
`
`another to achieve the interlaced scanning.
`
`15
`
`20
`
`25
`
`Some image sensing devices to take moving images
`
`according to the NTSC system have an area of pixels for
`
`image stabilization added to the area of effective pixel
`
`area,
`
`thus bringing the effective number of vertically
`
`arranged pixels to about 480 or more.
`
`In this case, an
`
`area beyond 480th pixels is read out at high speed during
`
`the vertical blanking period and therefore the signals thus
`
`read out are not used as effective signals. Therefore,
`
`the
`
`Video signals can only be generated from these signals
`
`coming from the area of about 480 vertically arranged
`
`pixels. When such a camera is used to photograph a still
`
`image, it is relatively easy to generate a static image
`
`signal conforming to, for example,
`
`JPEG (Joint Photographic
`
`Expert Group)
`
`from the signals coming from the same pixel
`
`area that is used to take a moving image.
`
`A problem
`
`remains, however, that the number of vertically arranged
`
`pixels obtained is limited to around 480, making it
`
`|PR2020-00597
`
`Apple EX1024 Page 6
`
`IPR2020-00597
`Apple EX1024 Page 6
`
`
`
`_ 3 _
`
`impossible to produce more detailed static image signals.
`
`In a camera having an image sensing device with
`
`the area of pixels for image stabilization mentioned above,
`
`a method of alleviating this problem may involve using the
`
`entire area of effective pixels including the area of image
`
`stabilization pixels in photographing a still image.
`
`Even
`
`when photographing a still image, however,
`
`the photographed
`
`image needs to be monitored for check and, for that
`
`purpose, it is necessary to generate signals conforming to
`
`the television system from signals read out from all effec-
`
`tive pixels.
`
`An example of such a conventional camera has been
`
`proposed in JP—A—11—187306.
`
`In the camera disclosed in
`
`this publication, signals from all the effective pixels are
`
`read out taking two or more times the field period of the
`
`television system, stored in a memory means such as a field
`
`memory, and then subjected to interpolation processing for
`
`transformation into signals conforming to the field cycle
`
`and horizontal scan cycle of television.
`
`This conventional camera, however, requires a
`large processing circuit, such as field memory, for signal
`
`conversion. Another drawback is that the image sensing
`
`device readout cycle is a plurality of times the field
`
`cycle, degrading the dynamic resolution.
`
`Even with the use
`
`of this circuit,
`
`the number of pixels obtained as the
`
`static image signals is limited to the number of effective
`
`pixels used for moving videos plus the area of image
`
`stabilization pixels.
`
`5
`
`10
`
`15
`
`20
`
`25
`
`|PR2020-00597
`
`Apple EX1024 Page 7
`
`IPR2020-00597
`Apple EX1024 Page 7
`
`
`
`_ 4 _
`
`In a digital still camera designed for taking
`
`still images,
`
`there has been a trend in recent years toward
`
`an increasing number of pixels used on the moving video
`
`image sensing device in order to obtain higher resolution
`
`5
`
`static image signals. When taking a moving image or
`
`monitoring the video, it is necessary to generate signals
`
`that conform to the television system.
`
`The number of
`
`pixels on such an image sensing device, however, does not
`
`necessarily match the number of scanning lines of the
`
`10
`
`television system and therefore some form of conversion
`
`means is required.
`
`The conversion means may involve, as in the video
`
`camera with the area of image stabilization pixels, reading
`
`out signals from the image sensing device taking a longer
`
`15
`
`time than the field cycle and interpolating them to
`
`generate television signals. This method has,
`
`in addition
`
`to the problem described above, a drawback that the readout
`
`cycle increases as the number of pixels increases, further
`
`degrading the dynamic resolution.
`
`20
`
`To mitigate this problem, JPuAu9=270959 discloses
`
`an apparatus which mixes together or culls the pixel
`
`signals inside the image sensing device to reduce the
`
`number of signals to be read and therefore the read cycle.
`
`Although this apparatus alleviates the problem of the
`
`25
`
`degraded dynamic resolution, it requires a large processing
`
`circuit such as field memory to perform time—axis transfor-
`
`mation to generate signals conforming to the television
`
`system and the image sensing device itself needs to have a
`
`|PR2020-00597
`
`Apple EX1024 Page 8
`
`IPR2020-00597
`Apple EX1024 Page 8
`
`
`
`_ 5 _
`
`special structure for performing desired mixing and
`
`culling.
`
`SUMMARY OF THE INVENTION
`
`The present invention relates to a photography of
`
`5 video cameras, camcorders, digital still cameras and others
`
`using a solid—state image sensing device, and more particu—
`
`larly to an electric camera using a solid—state image
`
`sensing device with a large number of pixels.
`
`The conventional electric cameras, as described
`
`10
`
`above, have drawbacks that when taking a still picture by
`
`using a video camera,
`
`the number of pixels is not suffi—
`
`cient and that when taking a moving image with a still
`
`camera,
`
`the associated circuit inevitably increases and the
`
`dynamic image quality deteriorates. Taking both moving and
`
`15
`
`static images of satisfactory quality with a single camera
`
`is difficult to achieve.
`
`In addition to solving the above
`
`problems,
`
`to obtain good dynamic picture quality by using
`
`an image sensing device having a large number of pixels
`
`intended for still images requires extracting a pixel area
`
`20
`
`that is used to realize an image stabilizing function.
`
`The
`
`conventional art and cameras do not offer a means to
`
`accomplish this function.
`
`An object of the present invention is to provide
`
`an electric camera that solves these problems and which
`
`25
`
`uses an image sensing device with a sufficient number of
`
`pixels for still images and enables taking of highly
`
`detailed still images and a moving video taking with
`
`|PR2020-00597
`
`Apple EX1024 Page 9
`
`IPR2020-00597
`Apple EX1024 Page 9
`
`
`
`_5__.
`
`reduced image quality degradation without
`
`increasing circuitry
`
`such as field memory. It is also an object of the present
`
`invention to provide an electric camera that can also realize
`
`the image stabilizing function.
`
`According to one aspect of this invention,
`
`the
`
`electric camera to realize the above objectives has: an image
`
`sensing device with a light receiving surface having N
`
`vertically arranged pixels and an arbitrary number of pixels
`
`arranged horizontally, N being equal to or more than three times
`
`the number of effective scanning lines M of a display screen of
`
`a television system; a driver to drive the image sensing device
`
`to vertically mix or cull signal charges accumulated in
`
`individual pixels of every K pixels to produce a number of lines
`
`of output signals which corresponds to the number of effective
`
`scanning lines M, K being at least one of integers equal to or
`
`less than an integral part of a quotient of N divided by M (a
`
`number of lines of output signals corresponds to l/K the number
`
`of vertically arranged pixels N of the image sensing device);
`
`and a signal processing unit to generate image signals by using
`
`the output signals of the image sensing device.
`
`As explained above, since this invention eliminates
`
`the limit on the number of vertically arranged pixels, an
`
`electric camera can be provided which enables taking of highly
`
`detailed still images and a satisfactory moving video taking by
`
`using an image sensing device with a large enough pixel number
`
`even for still images.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Figure l is a block diagram showing the configu
`
`|PR2020-00597
`
`Apple EX1024 Page 10
`
`IPR2020-00597
`Apple EX1024 Page 10
`
`
`
`_ 7 _
`
`ration of a first embodiment of an electric camera accord-
`
`ing to the present invention.
`
`Figure 2 is a schematic diagram showing the
`
`structure of an image sensing device in the first embodi—
`
`ment of the electric camera of the invention.
`
`Figure 3 is a drive pulse timing diagram in the
`
`first embodiment of the electric camera of the invention.
`
`Figure 4 is a schematic diagram showing a mixing
`
`operation in the first embodiment of the electric camera of
`
`10
`
`the invention.
`
`Figure 5 is a schematic diagram showing a readout
`
`area in the first embodiment of the electric camera of the
`
`invention.
`
`15
`
`20
`
`Figure 6 is a schematic diagram showing a mixing
`
`operation in the first embodiment of the electric camera of
`
`the invention.
`
`Figure 7 is a block diagram showing the configu-
`
`ration of a second embodiment of an electric camera accord-
`
`ing to the present invention.
`
`Figure 8 is a schematic diagram showing a mixing
`
`operation in the second embodiment of the electric camera
`
`of the invention.
`
`Figure 9 is a schematic diagram showing a readout
`
`area in the second embodiment of the electric camera of the
`
`25
`
`invention.
`
`Figure 10 is a schematic diagram showing the
`
`structure of an image sensing device in a third embodiment
`
`of the electric camera according to the present invention.
`
`|PR2020-00597
`
`Apple EX1024 Page 11
`
`IPR2020-00597
`Apple EX1024 Page 11
`
`
`
`- 3 _
`
`Figure 11 is a drive pulse timing diagram in the
`
`third embodiment of the electric camera of the invention.
`
`Figure 12 is a schematic diagram showing an
`
`interpolation operation in the third embodiment of the
`
`electric camera of the invention.
`
`Figures 13A and 13B are schematic diagrams show-
`
`ing the arrangement of color filters in the image sensing
`
`device in a fourth embodiment of the electric camera
`
`according to the present invention.
`
`10
`
`DESCRIPTION OF THE EMBODIMENTS
`
`Now embodiments of the present invention will be
`
`described by referring to the accompanying drawings.
`
`Figure l is a block diagram showing the configuration of
`
`one embodiment of an electric camera according to the
`
`15
`
`invention.
`
`In Figure 1, reference number 1 represents a
`
`lens,
`
`2 an aperture,
`
`3 an image sensing device, 4 a drive
`
`circuit,
`
`5 a gain adjust circuit,
`
`6 an analog-digital
`
`(A/D)
`
`conversion circuit, 7 a signal processing circuit,
`
`8 a
`
`vertical interpolation circuit to perform interpolation in
`
`a vertical direction,
`
`9 a horizontal interpolation circuit
`
`to perform interpolation in a horizontal direction, 10 a
`
`recording unit including recording media such as magnetic
`
`tape, semiconductor memory and optical disk to record a
`
`video signal, 11 a control circuit to control these
`
`constitutional elements according to the operating state,
`
`12 an encoder circuit to modulate the video signal into a
`
`20
`
`25
`
`|PR2020-00597
`
`Apple EX1024 Page 12
`
`IPR2020-00597
`Apple EX1024 Page 12
`
`
`
`-9-
`
`standard television signal, 13 a digital—analog (D/A)
`
`conversion circuit, 14'a mode selector switch to change
`
`over the operation mode between the moving video taking and
`
`the still image taking, 15 a record button to start or stop
`
`the recording, 16a and 16b gyro sensors to detect vertical
`
`image—unstability and lateral image—unstability, respec-
`
`tively, and 17 an image—unstability decision circuit to
`
`determine the image—instability from signals output from
`
`the gyro sensors.
`
`10
`
`15
`
`20
`
`25
`
`In the above configuration,
`
`light coming from the
`
`lens 1 through the aperture 2 is focused on a light receiv—
`
`ing surface of the image sensing device 3 where it is
`
`converted into an electric signal.
`
`In this embodiment the
`
`image sensing device 3 is of a CCD type. Figure 2 shows
`
`the structure of this image sensing device 3.
`
`In Figure 2,
`
`denoted 30 are pixels each formed of a photodiode, which
`
`are arranged horizontally and vertically in a grid pattern.
`
`On these grid—arrayed pixels three types of color filters
`
`that pass yellow (Ye), green (G) and cyan (CY),
`
`respec—
`
`tively, are arranged in such a way that the combination of
`
`these three colors is repeated horizontally every three
`
`pixels and that the filters of the same colors are lined
`
`vertically in so—called vertical stripes. Although an
`
`arbitrary number of pixels may be used, this embodiment has
`
`an array of 1200 pixels vertically and 1600 pixels horizon—
`
`tally.
`
`A vertical transfer unit 32 is a CCD which is
`
`driven by three phase pulses V1, V2, V3. This CCD has a
`
`three—gate structure in which each pixel corresponds to
`
`|PR2020-00597
`
`Apple EX1024 Page 13
`
`IPR2020-00597
`Apple EX1024 Page 13
`
`
`
`-10-
`
`three phase pulses and thus can vertically transfer a
`
`signal charge of each pixel independently. Transfer gates
`
`31 for transferring the charge of each pixel to the
`
`vertical transfer unit 32 are commonly connected to a gate
`
`of the vertical transfer unit 32 that corresponds to the V2
`
`pulse. An operation to transfer the charge from each pixel
`
`to the vertical transfer unit 32 in reSponse to a peak
`
`value of the pulse applied to the commonly connected gate
`
`and an operation to transfer the charge vertically are
`
`performed separately.
`
`A horizontal transfer unit 33
`
`horizontally transfers the charges supplied from the
`
`vertical transfer units 32 and outputs them successively
`
`through an output amplifier 34 from the output terminal.
`
`Referring back to Figure l, the operation
`
`performed when the moving video mode is selected by the
`
`mode selector switch 14 will be explained. The number of
`
`vertically arranged pixels on the image sensing device in
`
`this embodiment is 1200, so if the number of effective
`
`scanning lines in the field of the NTSC system is assumed
`
`to be 240 lines,
`
`then vertically mixing five pixels (= 1200
`
`pixel rows/240 scanning lines) can match the number of
`
`lines of output signals from the image sensing device to
`
`the number of effective scanning lines.
`
`However,
`
`in this embodiment,
`
`to realize the image
`
`stabilizing function described later, four vertically
`
`arranged pixels are mixed together during motion image
`
`taking mode. When four vertically arranged pixels are to
`
`be cyclically mixed together,
`
`the signals from the area of
`
`10
`
`15
`
`20
`
`25
`
`|PR2020-00597
`
`Apple EX1024 Page 14
`
`IPR2020-00597
`Apple EX1024 Page 14
`
`
`
`_ 11 -
`
`960 pixels (= 240 scanning lines X 4 pixels) out of the
`
`1200 vertically arranged pixels are used as effectiVe
`
`signals and the remaining 240 pixels (= 1200 (all pixels) —
`
`960 (effective pixels)) are not used for image forming.
`
`Figure 3 shows the timing of a vertical drive pulse for the
`
`image sensing device in this operation mode, with V1, V2
`
`and V3 representing three phase drive pulses applied to
`
`each gate of the CCD or vertical transfer unit 32.
`
`In Figure 3,
`
`in a period T1 included in the
`
`10
`
`15
`
`20
`
`vertical blanking period,
`
`the drive pulse V2 is held high
`
`to transfer the signal charge accumulated in each pixel to
`
`under the V2 gate of the vertical CCD. Next,
`
`in a period
`
`T2, while the V2 pulse is still at middle level,
`
`the V3
`
`pulse is raised from low level to middle level; next, while
`
`the V3 pulse is at middle level,
`
`the V2 pulse is changed
`
`from middle level to low level, after which the V1 pulse is
`
`changed from low level to middle level; next, while the V1
`
`pulse is at middle level, the V3 pulse is changed from
`
`middle level to low level, after which the V2 pulse is
`
`changed to middle level and finally the V1 pulse is changed
`
`from middle level to low level. With this sequence of
`
`pulse operations, the signal charges under the V2 gate for
`
`one pixel row are transferred and held again under the V2
`
`gate.
`
`25
`
`By repeating this series of operations,
`
`the
`
`signal charges for a desired number of pixel rows can be
`
`transferred.
`
`In Figure 3, during a period T3 included in
`
`the vertical blanking period before the vertical effective
`
`|PR2020-00597
`
`Apple EX1024 Page 15
`
`IPR2020-00597
`Apple EX1024 Page 15
`
`
`
`- 12 _
`
`scanning period (the vertical scanning period minus the
`
`vertical blanking period which corresponds to the actually
`
`displayed image) and during a period T4 included in the
`
`vertical blanking period after the vertical effective
`
`scanning period,
`
`the above transfer operation for one pixel
`
`row is repeated a total of 240 times to transfer the signal
`
`charges of the 240 pixel rows not used for image generation
`
`to the horizontal transfer unit 33 during the vertical
`
`blanking period.
`
`For example, if this transfer operation
`
`10
`
`is performed 120 times during the period T3 and 120 times
`
`during the period T4,
`
`the signal charges from upper 120
`
`pixel rows and lower 120 pixel rows on the light receiving
`
`surface are transferred to the horizontal transfer unit 33
`
`during the period T3 and period T4 within the vertical
`
`15
`
`blanking period. During each of the subsequent periods T5
`
`and T6 in the vertical blanking period,
`
`the horizontal
`
`transfer unit 33 is driven for a predetermined period to
`
`output the charges transferred to the horizontal transfer
`
`unit 33 from the output terminal. These charges are not
`used as valid signals as they are output during the
`
`2O
`
`vertical blanking period.
`
`Next,
`
`in the vertical effective scanning period
`
`of Figure 3,
`
`the above one—pixel-row transfer operation is
`
`performed four times during each horizontal blanking period
`
`25
`
`to transfer the signal charges of four pixel rows to the
`
`horizontal transfer unit 33 where they are mixed together.
`
`Then, during a horizontal effective scanning period (the
`
`horizontal scanning period minus the horizontal blanking
`
`|PR2020-00597
`
`Apple EX1024 Page 16
`
`IPR2020-00597
`Apple EX1024 Page 16
`
`
`
`_ 13 _
`
`period which corresponds to the actually displayed image),
`
`the horizontal transfer unit 33 is driven to read out the
`
`signal charges from the horizontal transfer unit to produce
`
`an output signal conforming to the television system.
`
`If
`
`the above operation is performed on the A field and if, on
`
`the B field,
`
`the number of pixel rows transferred during
`
`the period T3 is set to 122 rows and that during the period
`
`T4 is set to 118 rows,
`
`then the combination of four pixel
`
`rows to be cyclically mixed together shifts by two rows
`
`between the two fields,
`
`thus allowing the interlaced scan~
`
`ning to be performed as shown in Figure 4.
`
`(Figure 4 shows
`
`the light receiving surface of the image sensing device and
`
`its relation to the displayed screen is vertically
`
`inverted.)
`
`Let us return to Figure l.
`
`The output signal
`
`from the image sensing device 3 is adjusted in gain by the
`
`gain adjust circuit 5 and then converted by the A/D conver—
`
`sion circuit 6 into a digital signal.
`
`The digital signal
`
`is then processed by the signal processing circuit 7 that
`
`performs color signal processing and luminance signal
`
`processing, such as generation of color signals, gamma
`
`correction, white balance processing and outline enhance—
`
`ment.
`
`The image sensing device in this embodiment has an
`
`array of vertical stripes of yellow (Ye), green (G) and
`
`cyan (Cy) color filters, so the color signals for Ye, G and
`
`Cy are obtained as a series of color points from one line
`
`of output signals at all times no matter how many pixels
`
`are vertically combined.
`
`From these color signals three
`
`10
`
`15
`
`20
`
`25
`
`|PR2020-00597
`
`Apple EX1024 Page 17
`
`IPR2020-00597
`Apple EX1024 Page 17
`
`
`
`_ 14 -
`
`primary color signals R, G, B can be obtained from the
`
`following calculations.
`
`R = Ye - G
`
`B = Cy - G
`
`G = G
`
`The R, G and B signals undergoes the white
`
`balance processing and gamma correction processing in the
`
`signal processing circuit 7 and are then converted into
`
`color difference signals such as R—Y, B—Y or U and V.
`
`The
`
`luminance signals and the color difference signals are then
`entered through the vertical interpolation circuit 8 into
`
`the horizontal interpolation circuit 9.
`
`In this operation
`
`state the signals just pass through the vertical interpola—
`
`tion circuit 8 without being processed.
`
`The horizontal
`
`interpolation circuit 9 performs interpolation on the
`
`signals in the horizontal direction.
`
`Figure 5 shows the light receiving surface of the
`
`image sensing device. As described above,
`
`in the operating
`
`state of this embodiment,
`
`the signals read out during the
`
`vertical effective scanning period correspond to an area
`
`having 960 of the 1200 vertically arranged pixels and a
`
`horizontal width of 1600 pixels, as shown shaded at A in
`
`Figure 5.
`
`If the entire light receiving surface of the
`
`image sensing device has a 4-to—3 (width to height) aspect
`
`ratio,
`
`the shaded area A is more laterally elongate than
`
`this aspect ratio. Hence, if the signals of all horizontal
`
`pixels of the light receiving surface are displayed, for
`
`example, on an NTSC standard television monitor with the
`
`10
`
`15
`
`20
`
`25
`
`|PR2020-00597
`
`Apple EX1024 Page 18
`
`IPR2020-00597
`Apple EX1024 Page 18
`
`
`
`_ 15 _
`
`4—to—3 aspect ratio,
`
`the image displayed is compressed
`
`horizontally and looks vertically elongate, compared with
`
`the original image.
`
`It is therefore necessary to output
`
`during the horizontal effective scanning period only those
`
`signals coming from a pixel area with the horizontal width
`
`conforming to the aspect ratio of the television system, as
`
`shown by a shaded area B. When the television system has
`
`an 4-to-3 aspect ratio,
`
`the number of pixels in the
`
`horizontal width of the shaded area B is 1280 (= 960
`
`(vertical effective pixels) X 4/3).
`
`Returning back to Figure l,
`
`the horizontal
`
`interpolation circuit 9 performs interpolation processing
`
`on the signals from the horizontal 1280 pixels to expand
`
`the signals so that they can be output over the entire
`
`horizontal effective scanning period.
`
`It also performs
`
`switching among different clocks as required. With the
`
`above operation, an area having 960 pixels in height and
`
`1280 pixels in width is demarcated from the light receiving
`
`surface as signals conforming to the television system.
`
`Then,
`
`the luminance signal and the color difference signal
`
`are encoded by the encoder circuit 12 into television
`
`signals, which are then converted by the D/A conversion
`
`circuit 13 into analog signals for output. When the
`
`recording is specified by the record button 15,
`
`the signals
`
`are recorded by the recording unit 10. At this time, the
`
`signals may be compressed in the MPEG (Moving Picture
`
`Expert Group) format.
`
`Next,
`
`the image stabilizing operation will be
`
`10
`
`15
`
`20
`
`25
`
`|PR2020-00597
`
`Apple EX1024 Page 19
`
`IPR2020-00597
`Apple EX1024 Page 19
`
`
`
`-16..
`
`explained.
`
`Image—unstability information obtained by the
`
`gyro sensors 16a, 16b that detect vertical and horizontal
`
`image-unstabilities is entered into the image-unstability
`
`decision circuit 17, which checks the received information
`
`for the amount and direction of the image—unstability and
`
`converts them into the number of pixels in vertical and
`
`horizontal directions on the light receiving surface of the
`
`image sensing device. Based on the converted pixel
`
`numbers,
`
`the position of an extracted area (effective pixel
`
`area) on the light receiving surface is shifted in a
`
`direction that cancels the image—unstability. This can
`
`correct the image—unstability.
`
`The positional shifting of
`
`the extracted area is performed as follows.
`
`The shifting
`
`in the vertical direction can be made by changing the
`
`number of pixel rows transferred during the periods T3 and
`
`T4 in Figure 3 and the shifting in the horizontal direction
`
`made by changing the interpolation start position in the
`
`horizontal interpolation circuit 9.
`
`The operation during the moving video mode has
`
`been described above. Next,
`
`the operation performed when
`
`the static image mode is selected by the mode selector
`
`switch 14 will be explained.
`
`In the static image mode,
`
`too, until the record—
`
`ing is requested by the record button 15,
`
`the camera
`
`outputs signals compatible with the television system to
`
`monitor the angle of view. Unlike the moving video photo—
`
`graphing, all of the effective pixels on the image sensing
`
`device are used in this embodiment during the still image
`
`10
`
`15
`
`20
`
`25
`
`|PR2020-00597
`
`Apple EX1024 Page 20
`
`IPR2020-00597
`Apple EX1024 Page 20
`
`
`
`_ 17 _
`
`photographing to produce signals with as high a resolution
`
`as possible. Hence, during the monitoring the television
`
`signals need to be generated from the signals coming from
`
`the entire pixel area.
`
`The image sensing device of this embodiment has
`
`1200 vertically arranged pixels, and the number of lines of
`
`output signals from the image sensing device can be made to
`
`match the number of effective scanning lines of NTSC
`
`system, which is assumed to have 240 scanning lines, by
`
`vertically mixing five pixels (= 1200/240).
`
`To make the
`
`image sensing device operate in this manner,
`
`the one—pixel-
`
`row transfer operation is performed five times during each
`
`horizontal blanking period in the vertical effective
`
`scanning period shown in the pulse timing diagram of Figure
`
`3. With this operation,
`
`the signal charges of five pixel
`
`rows can be mixed by the horizontal transfer unit 33. As
`
`for the transfer operations during the periods T3 and T4 in
`
`the vertical blanking period, because the interlaced scan—
`
`ning is carried out, only two pixel rows are transferred
`
`during the period T3 on the B field, with no transfer
`
`operations performed in other vertical blanking periods (In
`
`this embodiment, 1200/240
`
`5 with no remainder produced,
`
`so no further transfer is necessary; if, however, a
`
`remainder occurs,
`
`the remaining pixels need only be trans—
`
`ferred during the periods T3 and T4).
`
`The charges mixed by the horizontal transfer unit
`
`33 are read out by driving the horizontal transfer unit 33
`
`during the horizontal effective scanning period. With the
`
`10
`
`15
`
`20
`
`25
`
`|PR2020-00597
`
`Apple EX1024 Page 21
`
`IPR2020-00597
`Apple EX1024 Page 21
`
`
`
`- 18 -
`
`above operations,
`
`the signal charges of all pixels on the
`
`image sensing device can be read out in a manner conforming
`
`to the television system.
`
`The output signal from the image
`
`sensing device 3 is, as during the moving image photo-
`
`graphing, adjusted in gain by the gain adjust circuit 5 and
`
`converted by the A/D conversion circuit 6 into a digital
`
`signal, which is then subjected to the color signal
`
`processing and the luminance signal processing in the
`
`signal processing circuit 7 before being entered into the
`
`vertical interpolation circuit 8. During the static image
`
`monitoring,
`
`the vertical interpolation circuit 8 performs a
`
`vertical gravity center correction on the received signals.
`
`Figure 6 shows combinations of pixels to be
`
`cyclically mixed on the A field and the B field and also
`
`the vertical position of the gravity center of the mixed
`
`signals.
`
`In the interlaced scanning, scanning lines of the
`
`A field and the B field are located at the centers of
`
`adjoining scanning lines on other field. Hence,
`
`the signal
`
`samplings in the camera system for the two fields must be
`
`180 degrees out of phase in the vertical direction.
`
`In the
`
`operating state of this embodiment, however, because five
`
`pixels are mixed together,
`
`the gravity centers of the
`
`output signals for the A field and the B field are deviated
`
`36 degrees (= 1/2 pixel or l/lO the line-to—line distance
`
`on the same field) from the ideal sampling phase difference
`
`of 180 degrees, as shown in Figure 6.
`
`To correct this
`
`requires generating a signal from two adjoining output
`
`lines by interpolation.
`
`For example, if we let an nth
`
`10
`
`15
`
`20
`
`25
`
`|PR2020-00597
`
`Apple EX1024 Page 22
`
`IPR2020-00597
`App