`(10) Patent N0.:
`(12) Unlted States Patent
`
`Dunton et al.
`(45) Date of Patent:
`*Jan. 28, 2003
`
`U5006512541B2
`
`(54)
`
`INCREASING IMAGE FIELD OF VIEW AND
`FRAME RATE IN AN IMAGING APPARATUS
`
`(75)
`
`Inventors: Randy R. Dunton; Lawrence A.
`BOOth, JR, both Of Phoenix, AZ (US)
`
`(73) Assignee:
`
`Intel Corporation, Santa Clara, CA
`(US)
`
`( * ) Notice:
`
`This patent issued on a continued pros-
`ecution application filed under 37 CFR
`1.53(d), and is subject to the twenty year
`patent
`term provisions of 35 U.S.C.
`154009).
`.
`.
`.
`.
`Sutbjetcttto artly (353121111112; thf tierm5f 0:1:
`13186113 1153?;an 0:); a Jus e un er
`'
`'
`'
`( )
`y
`ays.
`
`5,822,625 A * 10/1998 Leidig et al.
`................. 396/77
`5,920,657 A *
`7/1999 Bender et al.
`..
`..... 382/284
`6,005,609 A * 12/1999 Cheong ................... 348/169
`
`............. 348/231
`6,005,613 A * 12/1999 Endsley et al.
`OTHER PUBLICATIONS
`
`“Canon’s Optura: A digital Camcorder That Performs Like
`and SLR Camera,” HyperZine Manufacture’s View, original
`posting: 1997—08—25; revised: 1997—09—03, pp. 1—3.
`“From Consumer Electronics Online News: Aug. 25, 1997,”
`HyperZine Expert’s View, Original posting: 1997—08—25,
`pp. 1_3.
`.
`.
`* Cited by examiner
`Primary Examiner—Wendy R. Garber
`Assistant Examiner—Mitchell White
`(74) Attorney, Agent, or Firm—Blakely, Sokoloff, Taylor &
`Zafman LLP
`
`(21) Appl. No.: 08/986,754
`
`(57)
`
`ABSTRACT
`
`An imaging apparatus that is configurable to operate in at
`least two modes. One mode is particularly suitable for still
`image capture, whereas the second mode is suitable for
`Video image capture and other rapid frame rate applications.
`The image data in the second mode is smaller (lower
`resolution) than the image data obtained in the first mode.
`The reduction is accomplished by either dlgltal scaling,
`cropping, or by a combination of optical scaling and selec-
`tive readout of sensor signals. The simple digital scaling
`provides a fixed angular field of View for both modes of
`operation, while cropping alone gives a smaller field of view.
`Using the combination of optical scaling and selective
`sensor signal readout, however, provides a wider field of
`View for the second mode of operation while at the same
`time providing lower resolution images,
`thus improving
`frame rate in the second mode of operation. The embodi-
`ments can be used in a wide range of imaging applications,
`including digital cameras used for both still image capture
`and video.
`
`17 Claims, 6 Drawing Sheets
`
`(22)
`
`(65)
`
`Filed:
`
`Dec. 8, 1997
`_
`.
`_
`Pr10r Publlcatlon Data
`US 2001/0050712 A1 Dec. 13; 2001
`7
`
`H04N 5/235
`Int. Cl.
`(51)
`348/230, 348/240, 348/239
`(52) U.S. Cl.
`348/222, 224,
`(58) Field of Search
`348/229, 230, 240, 239, 14, 15, 18, 19,
`340, 135, 169, 172; 250/2031, 203.6
`
`(56)
`
`References CitEd
`U.S. PATENT DOCUMENTS
`
`348/352
`Toyama et al.
`
`382/3 1 9
`Parulski ............
`Sato ........................... 396/123
`Buchner et a1.
`395/139
`Ueda .......................... 348/222
`Sasaki
`Sato ........................... 359/687
`Kimura et al.
`348/240
`
`7/1991
`4/1994
`4/1994
`11/1995
`11/1996
`3/1998
`3/1998
`9/1998
`
`*****
`
`5,031,049
`5,301,244
`5,305,046
`5,471,572
`5,572,253
`5,734,424
`5,734,508
`5,812,189
`
`>>>>>>>>
`
`DISTANT
`SCENE
`
`STROBE
`
`W112
`
`
`
`
`
`
`
`VIEWING
`ANGLE
`
`NEAR IMAGE
`DATA 170
`
`
`
`COMM.
`INTERFACE
`
`LOCAL USER
`INTERFACE
`
`LOCAL
`STORAGE
`
`
`
`
`
`
`154
`
`\158
`
`122
`
`K DISTANT
`IMAGE
`DATA 172
`
`
`
`|PR2020-00597
`
`Apple EX1020 Page 1
`
`IPR2020-00597
`Apple EX1020 Page 1
`
`
`
`
`
`1119ch'S'fl
`
`H
`g
`N
`.3
`g
`
`()3
`
`g
`ES.
`H
`a,
`ax
`
`1:
`CI)
`
`9U
`
`]
`
`
` 100
`
`
`NEAR IMAGE
`STROBE
`—
`DATA 17o
`H2
`
`
`DBTANT
`SCENE
`I
`K103
`I
`'
`
`
`I
`\j
`I
`”TI
`I
`1083- ——
`104
`
`SCENE
`
`\L
`
`,r’
`
`A
`
`114
`
`110
`
`m
`g
`(33
`
`CD
`
`:
`
`SIGNAL
`PROCESSING
`160
`
`SYSTEM
`CONTROLLER
`
`
`
`
`
`
`
`
`
`VIEWIN
`ANGLE G
`
`COMM.
`
`INTERFACE
`
`LOCAL USER
`
`INTERFACE
`
`LOCAL
`
`STORAGE
`
`154
`
`158
`
`122
`
`
`
`KOISTANT
`IMAGE
`DATA 172
`
`
`
`IPR2020-00597
`Apple EX1020 Page 2
`
`
`
`108
`
`1 06A
`
`a;¢
`,’,,,/,,,
`
`:2?
`
`7,5,‘:2I:w:/.................___=,%///////////
`
`105
`
`106D
`
`I?
`‘72
`522%mgave/n1,
`a,I.y
`.._._.._..,_.g
`
`ga..I(5...;I?)
`aM“,
`gr
`
`,i
`
`
`cm.5.8—:
`Has.mm.25m933Na.a
`
`Cmavmwmwmn:an
`
`
`
`PROJECTED
`
`IPR2020-00597
`Apple EX1020 Page 3
`
`
`
`
`106A
`
`105
`
`1068
`
`1060
`
`104
`
`=2,5.5II_Ii=l—i2Lug/é.
`
`,flVi7.
`,//
`57Z,:4.
`
`,.,,
`
`
`
`2/6
`
`|______.________________.__._.__...________..___.._._________________._.___________.___.._.____
`
`PROJECTED
`
`
`
`cm.5.8—:
`
`ha.a.38
`
`mramnu3.a
`
`Cmavmwmwmn:an
`
`IPR2020-00597
`Apple EX1020 Page 4
`
`
`
`
`
`US. Patent
`
`Jan. 28, 2003
`
`Sheet 4 0f 6
`
`US 6,512,541 B2
`
`SELECTED COLUMNS FOR NEAR IMAGES
`
`SENSOR
`
`IIIIIEIIIII-"III
`
`IIIIIIIIIII-"III
`
`IIIIIHIIIII-"III
`
`
`
`I'lll_fl____u-IIIIIIIIII—IIIIIIIIIII
`
`I'lllillllfi-"III
`
`IIIIIEIIIV‘E-"III
`
`IIIIIEIIIii-"III
`
`IIIII_IEEli_-IIIII
`
`I'llliiymflifi-IIIII
`
`IIIII_IIII_I-IIIII
`
`I'lllilllrfll-II'II
`
`IIIIIEIIIII-I'I"
`
`IIIIIIIIIIIIIIII
`
`IIIIIIIIIIIIIIII
`
`IIIIIIIIIIIIIIII
`
`IIIIIIIIIIIIIIII
`
`IIIIIIIIIIIIIIII
`
`IIIIIIIIIIIIIIII
`
`SELECTED
`ROWS FOR
`NEAR IMAGE
`
`I-
`
`SENSOR SIGNALS FOR VIDEO IMAGE
`
`SENSOR SIGNALS FOR DISTANT IMAGES
`
`FIG. 4
`
`IPR2020-00597
`
`Apple EX1020 Page 5
`
`IPR2020-00597
`Apple EX1020 Page 5
`
`
`
`
`
`111mm'90
`
`3‘
`F
`3.:
`15
`
`3
`
`%
`
`2 OI
`
`—g,
`
`a
`
`C1
`ch
`
`m
`
`b]
`
`
`PIXEL SUBSTITUTION
`O
`'G-
`
`IMAGE
`COMPANmNG
`
`DAV\ GAMMACORRECNON
`
`
`ENCONNGLOG”
`
`DECORRELATION AND
`
`RAM
`
`SCALED
`DATA
`—————————————————————
`
`_____2§8_____
`
`
`
`DATA-LOW.ONTROLLER
`
`LUT
`
`_
`
`CORRECTED
`DATA g
`L——
`
`_ —5_ — _ — _
`
`“0
`
`II
`
`|
`I
`I_
`
`_ _
`
`
`>
`>
`>
`
`R
`(/3
`
`1%IEIIIiI'
`M-MORY
`-0NTR0LLER
`
`so
`a",
`
`:lafllll%
`INTERFACE
`
`246
`
`—L
`O)
`
`>
`U
`
`>
`O
`
`LOCAL
`STORAGE
`
`?\350
`
`HOST/PC
`COMMUNICATION
`
`INTERFACE
`
`SYSTEM
`CONTROLLER
`
`UNTT
`
`8
`fi
`
`g
`
`no
`3",
`
`Ed
`rn
`
`E;
`LOCAL USER
`'NTEREACE
`
`IPR2020-00597
`Apple EX1020 Page 6
`
`
`
`US. Patent
`
`Jan. 28, 2003
`
`Sheet 6 0f 6
`
`US 6,512,541 B2
`
`IDENTIFY MODE OF OPERATION
`(VIDEO OR STILL)
`
`SELECT SPATIAL SCALING RATIO,
`DECORRELATION AND ENCODING SCHEMES,
`AND PACKING FORMAT FOR THE IDENTIFIED
`
`MODE OF OPERATION
`
`SCALE IMAGE DATA TO OBTAIN SCALED
`IMAGE DATA BASED ON SELECTED SPATIAL
`
`SCALING RATIO
`
`
`
`DECORRELATE THE SCALED IMAGE DATA IN
`PREPARATION FOR ENCODING ACCORDING TO
`
`THE SELECTED DECORRELATION SCHEME
`
`COMPRESS THE DECORRELATED IMAGE DATA
`ACCORDING TO ENTROPY ENCODING SCHEME
`SELECTED
`
`PACK THE COMPRESSED DATA INTO THE
`
`SELECTED FORMAT
`
`FIG. 6
`
`|PR2020-00597
`
`Apple EX1020 Page 7
`
`IPR2020-00597
`Apple EX1020 Page 7
`
`
`
`US 6,512,541 B2
`
`1
`INCREASING IMAGE FIELD OF VIEW AND
`FRAME RATE IN AN IMAGING APPARATUS
`
`BACKGROUND
`
`This invention is generally related to electronic imaging,
`and more particularly to changing an image field of View and
`image frame rate in an imaging apparatus.
`Modern electronic imaging systems have become an
`important part of every household and business, from tra-
`ditional applications such as video cameras and copiers to
`more modern ones such as the facsimile machine, scanner,
`medical
`imaging devices, and more recently,
`the digital
`camera. The digital camera has been developed as a portable
`system that acquires and stores detailed still
`images in
`electronic form. The images may be used in a number of
`different ways such as being displayed in an electronic
`photo-album or used to embellish graphical computer appli-
`cations such as letters and greeting cards. The still images
`may also be shared with friends via modem anywhere in the
`world within minutes of being taken.
`Most purchasers of digital cameras have access to a
`desktop computer for Viewing the still images. Therefore,
`such purchasers might also enjoy using the digital camera to
`communicate with another person via videoconferencing or
`to View images of motion in a scene. Using a digital camera
`as a video camera or Videoconferencing tool, however,
`presents requirements that may conflict with those for cap-
`turing still images. For instance, due to the limited trans-
`mission bandwidth between the camera and a host computer
`used for Viewing video images, the transmitted frames of
`video images must be of a typically lower resolution than
`still images.
`To meet a given image frame rate over a limited trans-
`mission bandwidth, one solution is to simply electronically
`scale the detailed still image frames into lower resolution
`image frames prior to transmitting them. Alternatively, the
`detailed image can be “cropped” to a smaller size, and
`therefore lower resolution image. In this way, the amount of
`spatial data per image frame is reduced, so that a greater
`frame rate can be achieved between the digital camera and
`the host computer.
`Electronic scaling and/or cropping of the detailed image,
`however, does not address another problem posed by
`videoconferencing, namely that due to close proximity of
`the object (a person’s face or body) to the digital camera
`during the Video phone or Videoconferencing session, a
`wider field of View is required of the images. The field of
`view can loosely be thought of as relating to the fraction of
`the scene included in the transmitted image frame.
`Digital cameras typically use an optical system with a
`fixed efiective focal length. Although a detailed still image
`using such a camera could have an acceptable field of View
`for distant scenes, electronically scaling the image for video
`operation does not increase the field of view, while cropping
`actually decreases the field of View. Therefore, what is
`needed is a mechanism that allows a digital camera to
`capture images of close-up scenes having a wider field of
`view but with lower resolution, in order to increase frame
`rate for rapid frame rate applications such as video phones
`and Videoconferencing.
`
`SUMMARY
`
`The invention in one embodiment is directed at a circuit
`
`for processing first sensor signals to yield first digital image
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`
`data, where the first signals are generated by an image sensor
`in response to a first image of a scene projected on the
`sensor. The circuit is further configured to process second
`sensor signals to yield second digital image data having a
`lower resolution than the first data. The second signals are
`also generated by the image sensor but this time in response
`to a second image projected on the sensor, where the second
`image has a greater angular field of View but is smaller than
`the first image.
`The circuit may be incorporated into an imaging appara-
`tus such as a digital camera as a different embodiment of the
`invention. The imaging apparatus includes the image sensor
`coupled to an optical system, where the optical system has
`an adjustable effective focal length, such as in a zoom lens,
`in order to focus light from a scene onto the sensor to create
`the first and second images. The second image data is
`obtained through a combination of (1) the optical system
`being adjusted to project the second image having a wider
`field of view than the first image on the image sensor, and
`(2) the circuit processing the second sensor signals which
`are generated in response to the second image. The first
`image data is generated while the camera operates in “still”
`mode to capture detailed images of distant scenes, whereas
`the second image data results while operating in “Video”
`mode to capture less detailed but wider angle images of near
`scenes typically encountered during, for example, videocon-
`ferencing.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`These and other features as well as advantages of the
`different embodiments of the invention will be apparent by
`referring to the drawings, detailed description, and claims
`below, where:
`FIG. 1 illustrates a digital image capture apparatus pro-
`viding images in dual mode according to a first embodiment
`of the invention.
`FIG. 2 illustrates the detail
`
`in an embodiment of the
`
`optical system used in the imaging apparatus to generate a
`near image.
`FIG. 3 illustrates detail in the optical system that gener-
`ates a smaller field of View but larger image, according to
`another embodiment of the invention.
`
`FIG. 4 is a diagram of an image sensor with a projected
`near image and the associated sensor signals to be processed
`by a circuit embodiment of the invention.
`FIG. 5 shows a data flow diagram of the path taken by
`image data for video and still modes of operation.
`FIG. 6 illustrates a flow diagram of imaging operations
`that can be performed by the embodiment of FIG. 5.
`
`DETAILED DESCRIPTION
`
`As briefly summarized above, the embodiments of the
`invention are directed at an apparatus and associated method
`for capturing images having increased angular field of View
`and at the same allowing an increased frame rate due to their
`lower resolution. The techniques are particularly suitable for
`an imaging system such as a digital camera that operates in
`at least two modes to provide still and video images. The
`video images have lower resolution but a greater angular
`field of view than the still images. The greater angular field
`allows the video images to capture close-up scenes that are
`typically found in Videoconferencing sessions, while their
`lower resolution permits the transmission of images at a
`higher frame rate to a host processor over a limited trans-
`mission bandwidth. The method and apparatus embodiments
`
`|PR2020-00597
`
`Apple EX1020 Page 8
`
`IPR2020-00597
`Apple EX1020 Page 8
`
`
`
`3
`
`4
`
`US 6,512,541 B2
`
`of the invention achieve such a result by a combination of
`optical scaling to obtain a smaller image size for video
`mode, and selectively reading only those sensor signals
`which are generated in response to the smaller image.
`For purposes of explanation, specific embodiments are set
`forth below to provide a thorough understanding of the
`invention. However, as understood by one skilled in the art,
`from reading this disclosure, the invention may be practiced
`without such details. Furthermore, well-known elements,
`devices, process steps, and the like, are not set forth in detail
`in order to avoid obscuring the invention.
`FIG. 1 shows a digital
`image capture apparatus 100
`according to a first embodiment of the invention. The
`apparatus 100 has an optical system 108, including a lens
`system 106 and aperture 104, for being exposed to incident
`light reflected from a scene whose image is to be captured.
`For this embodiment, two scenes in particular are identified,
`a near scene 102 and a distant scene 103. The near scene
`
`may be, for instance, a videoconferencing session where a
`user is sitting at a desk with the apparatus 100 positioned
`approximately 2 feet in front of the user. The distant scene
`103 includes the presence of objects that are located farther
`away from the apparatus 100, e.g., 8—10 feet, such as when
`taking still images.
`The apparatus 100 may also include a strobe 112 or
`electronic flash for generating supplemental light to further
`illuminate the scenes when the apparatus 100 is operating
`under low light conditions.
`The optical system 108 channels the incident light rays
`onto an electronic image sensor 114. The image sensor 114
`has a number of pixels or photocells (not shown) which are
`electrically responsive to incident
`light
`intensity, and,
`optionally, to color. Each of the pixels in the sensor 114
`generates a sensor signal that together represent a captured
`image with sufficient resolution to be acceptable as a still
`image. Contemplated resolutions include 640x480 and
`higher for acceptable quality still images.
`The sensor 114 generates sensor signals in response to an
`image of a scene formed on the sensor. The signal processing
`block 110 then processes the sensor signals into captured
`digital image data representing the image projected on the
`sensor 114. An analog-to-digital (A/D) converter (not
`shown) may be included in the sensor 114, as part of the
`same single integrated circuit die, to generate digital sensor
`signals (one per pixel) that define a digital image of the
`exposed scene.
`The captured image data include near image data 170 and
`distant
`image data 172. These are obtained in part by
`adjusting the optical system 108 to change its focal length
`and, more generally, its modulation transfer function (MTF),
`to focus images of either the near scene 102 or distant scene
`103 onto the sensor 114 at the focal plane of the optical
`system. The signal processing unit 110 processes the sensor
`signals according to image processing methodologies to
`yield the near image data 170 or the distant image data 172.
`The near image data may be provided as Video images which
`are streamed to an image processing system such as a host
`computer (not shown) Via the communication interface 154.
`The larger and greater resolution distant image data may also
`be transferred to the host via the interface 154, but at a lower
`frame rate. The image data is then decompressed (if
`necessary), rendered, and/or displayed in the host computer.
`The image data, particularly the distant (or still mode)
`image data, may optionally be stored in a local storage 122
`aboard the apparatus 100. The local storage 122 may include
`a FLASH semiconductor memory and/or a rotating media
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`device such as a hard disk. The FLASH memory may be
`removable, such as the Intel® Miniature Card. The rotating
`media may also be removable or fixed, and may be of the
`magnetic disk or other type suitable for storing image data
`files.
`
`The apparatus 100 can be configured to operate in at least
`two modes. A first mode generates near image data 170,
`suitable for Video operation. A second mode generates
`distant
`image data 172, suitable for still
`image capture.
`Mode selection can be made by the user Via mechanical
`control (not shown) on the apparatus 100. Mechanical knob
`settings can be received and translated by a local user
`interface 158 into control signals and control data that is
`processed by a system controller 160. Alternatively,
`the
`apparatus 100 can be tethered to the host computer (not
`shown) such as a personal computer (PC) via the commu-
`nication interface 154. The user can then make the mode
`
`selection through software running on the host which in turn
`communicates the proper control signals and data to the
`system controller 160.
`The system controller 160 orchestrates the capture of
`images in both modes of operation in response to the mode
`selection made by the user. In particular, the system con-
`troller configures the signal processing block 110 to provide
`the near or distant image data as described in greater detail
`below.
`
`the effective
`In the first embodiment of the invention,
`focal
`length of the optical system 108 must be altered
`between the different modes. FIGS. 2 and 3 illustrate two
`
`different settings of the optical system 108 corresponding to
`the two different modes of operation for the apparatus 100.
`The optical system 108 as shown includes a lens system 106
`consisting of four lens elements 106a—d that are positioned
`in front of the sensor 114. An adjustable or movable lens and
`aperture combination 105, such as a zoom lens,
`is also
`included. The zoom lens or combination 105 can be moved
`
`by the user actuating a lever, ring, or by an electromechani-
`cal mechanism such as a solenoid or motor. The lens
`combination 105 includes lens elements 106b, 106C as well
`as the aperture 104. For clarity, only the light rays from the
`lower half of the scenes are shown in the figures. Although
`optical system 108 is shown as a lens system having four
`separate lenses and a fixed aperture 104, one skilled in the
`art will recognize that other variations are possible which
`yield a smaller near image with greater angular field of view.
`By simply adjusting the position of combination 105 from
`a near position in FIG. 2 to a distant position in FIG. 3, the
`size of the image projected onto sensor 114 can be increased.
`This optical scaling feature results in a larger image size for
`the still image (distant image data) mode of operation.
`The other significant characteristic of the optical scaling
`is the change in angular field of view. The angular field of
`View can loosely be thought of as relating to the fraction of
`the scene included in the image projected onto the sensor
`114. Thus, although the projected image in FIG. 2 is smaller
`than that of FIG. 3, a greater fraction of the scene is included
`in the near image of FIG. 2 as shown by the additional light
`rays that enter the optical system 108 through the first lens
`106a.
`
`In order to obtain the smaller near image data 170 and
`greater frame rate when the apparatus 100 is configured with
`the optical system 108 in the near position,
`the signal
`processing block 110 and the sensor 114 are configured (by
`control signals and data received from the system controller
`160, see FIG. 1) to process only the pixel signals originating
`from those rows and columns of the sensor 114 that define
`
`|PR2020-00597
`
`Apple EX1020 Page 9
`
`IPR2020-00597
`Apple EX1020 Page 9
`
`
`
`5
`
`6
`
`US 6,512,541 B2
`
`the region on which the smaller image is formed. This can
`be seen in FIG. 4 which shows a near image being formed
`on an array of pixels in the sensor 114 and using fewer
`bitlines than the maximum resolution of the array. Instead of
`reading all of the sensor signals (which can be read for
`obtaining the maximum resolution of the image sensor
`array) shown in FIG. 4, only those sensor signals coming
`from the pixels in those rows and columns which define the
`region of the near image are read. The fewer sensor signals
`result in both lower processing times in the signal processing
`block 110, and greater image frame rate through a limited
`bandwidth interface between the sensor 114 and signal
`processing block 110. This in turn results in greater image
`frame rate through the host communication interface 154
`(see FIG. 1).
`The image data can be further compressed and/or scaled
`by the signal processing unit 110 as discussed below in
`connection with FIGS. 5 and 6 in order to increase the frame
`
`10
`
`15
`
`rate while transmitting the images through a bandwidth-
`limited communication interface 154.
`
`20
`
`To further reduce the cost of manufacturing an apparatus
`100 that operates in both still capture and video modes,
`optical system 108 can be fixed to project images on the
`sensor 114 having approximately 55° of angular field of
`view. Such a field of view may be an acceptable compromise
`for both the near and distant scenes. The distant scene would
`
`be captured as a detailed still image, while the near scene
`(e.g., videoconferencing session) would be handled by digi-
`tally scaling (using the signal processing unit 110)
`the
`detailed still
`image to reduce its resolution. This gives
`greater frame rate for the Video images, but no increase in
`the field of view as compared to still images. Embodiments
`of the signal processing unit 110 are shown in FIGS. 5 and
`6 and described below.
`
`To summarize, the above-described embodiments of the
`invention are an imaging apparatus (such as a digital
`camera), a method performed using the imaging apparatus,
`and a circuit that provides analog and digital processing, for
`increasing image field of view while at
`the same time
`increasing the image frame rate. The embodiments of the
`invention are, of course, subject
`to some variations in
`structure and implementation. For instance, the embodiment
`of the invention as signal processing block 110 can be
`implemented entirely in analog and hardwired logic cir-
`cuitry. Alternatively,
`the digital scaling and compression
`functions of the signal processing block 110 can be per-
`formed by a programmed processor. Therefore, the scope of
`the invention should be determined not by the embodiments
`illustrated but by the appended claims and their legal equiva-
`lents.
`
`Signal Processing Architecture
`The image capture apparatus 100 can be electronically
`configured for dual mode operation by configuring the signal
`processing block 110 to provide either still image data or a
`sequence of video images using the logical block diagram
`and architecture of FIG. 5. In one embodiment, the block
`110 implements digital signal and image processing func-
`tions as logic circuitry and/or a programmed data processor
`to generate compressed image data having a predefined
`resolution and compression ratio from detailed, original
`image data received from the sensor 114.
`FIG. 5 shows a data flow diagram for an embodiment of
`the invention of the path taken by image data for both video
`and still modes of operation. The processing block 110
`includes a chain of imaging functions which may begin with
`a correction block 210. The correction block 210 is used
`
`whenever the quality of the original image data received
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`from the sensor 114 warrants some sort of pre-processing
`before the image is scaled and compressed. In certain cases,
`the correction block 210 performs pixel substitution,
`companding, and gamma correction on the original image
`data received from the image sensor. The original image data
`should be of sufficient detail (e.g., 768x576 spatial resolu-
`tion or higher is preferred) to yield still images of acceptable
`quality.
`Pixel substitution may be performed in block 210 to
`replace invalid pixel data with valid data to provide a more
`deterministic input to subsequent imaging functions. Com-
`panding may be performed to lower the resolution of each
`pixel (the number of bits per pixel). For example,
`the
`original image data can arrive as 10 bits per pixel, whereas
`a preferred pixel resolution for the logic circuitry may be 8
`bits (1 byte). Conventional gamma correction may also be
`performed to conform the information content of the image
`to that expected by the host computer where the image will
`be ultimately displayed.
`Other functions that may be performed in block 210 on
`each received original image frame include fixed pattern
`noise reduction which is often needed before compressing
`an image. Once again, whether or not any correction func-
`tions are performed by block 210 in general depends on the
`quality of the original image data received from the sensor
`114 and any subsequent image processing such as scaling or
`compression to be performed before the image data is ready
`for storage or transmission to the host computer.
`Once the original
`image data has been corrected or
`otherwise processed into the desired size or format by
`correction block 510, the corrected data may be scaled and
`compressed if needed to meet the transmission and storage
`requirements of the communication interface 154 and the
`optional
`local storage 122 (see FIG. 1). To meet such
`requirements, the processing block 110 can include scaling
`and compression logic 514 to perform any necessary image
`scaling and compression prior to transmission and storage.
`For instance, the scaling and compression logic 214 may
`be configured to reduce image size and resolution to yield
`smaller, less detailed video images, as compared to larger
`and more detailed still images. Smaller and less detailed
`image data may be required in order to transmit a rapid
`sequence of video images that are to be decompressed and
`viewed in a host/PC. However,
`if the transmission link
`between the apparatus 100 and the host/PC has sufficient
`bandwidth to transmit a sequence of detailed original image
`data at the needed rate to the host/PC, then the scaling and
`compression logic 514 can be simplified or even eliminated
`for both still or video operation.
`A number of digital image processing functions are con-
`templated for the logic 514. These or others similar in
`function may be configured as described below by one
`skilled in the art depending on the performance (speed of
`rendering the compressed image data) and image quality
`desired. The imaging functions have been implemented in
`one embodiment as separate units of logic circuitry as seen
`in FIG. 5. The functions are described as follows in con-
`
`junction with the flow diagram of FIG. 6.
`The logic 514 can perform a 2-D spatial scaling of the
`corrected image data in order to yield smaller images that
`may be easier to store or transmit. The scaling is done
`according to a selected scaling ratio using conventional
`known techniques. The scaling ratio may be integer or
`fractional. The scaling can be performed in a 2-dimensional
`fashion by,
`for
`instance, utilizing two separate
`1-dimensional scaling processes.
`The logic 514 can be used for both video and still image
`capture simply by selecting the appropriate scaling ratio, as
`
`|PR2020-00597
`
`Apple EX1020 Page 10
`
`IPR2020-00597
`Apple EX1020 Page 10
`
`
`
`7
`
`8
`
`US 6,512,541 B2
`
`indicated in step 614. For instance, a 4: 1 sub-sampling of the
`corrected image may be performed in video mode so that 16
`pixels from the corrected image data are averaged together
`to produce 1 pixel
`in the scaled image data. Based on
`standard sampling theory, and assuming uncorrelated noise
`sources, the sub-sampling may also improve the signal to
`noise ratio by Vfi, or a factor of 4. Lower scaling ratios such
`as 2:1 may also be used, where 4 pixels are averaged to
`generate a single pixel in the scaled image data, resulting in
`a signal to noise ratio (SNR) improvement of 2. By scaling
`the more detailed corrected image data in this way during
`operation in video mode, the imaging system compensates
`for the increased noise due to lower light levels that are
`typically encountered with video operation, such as during
`videoconferencing. The scaling step, if needed, appears as
`step 618 in FIG. 6.
`Next in the chain of imaging function blocks in FIG. 5 is
`the decorrelation and encoding logic 522. The scaled image
`data received from the logic 514 is decorrelated by logic 522
`in preparation for entropy encoding as indicated in step 622,
`according to a selected one of a number of decorrelation
`methodologies. Once again, the user may select a particular
`decorrelation methodology that is suitable for obtaining the
`normally smaller size Video images, as indicated in step 614.
`The decorrelation function can generate error image data
`as differences between neighboring pixels. One particular
`method that can be used for image decorrelation is digital
`pulse code modulation (DPCM). To obtain more compres-
`sion of the image data, if needed, for example, in transmit-
`ting a large number of video image frames, “loss” may be
`introduced in the form of “quantization” (mapping a first set
`of data to a smaller set of values) errors using DPCM.
`The next stage in the chain of imaging function blocks is
`entropy encoding, also performed by logic 522. The tech-
`nique uses a variable length encoding technique to compress
`the decorrelated image data, if needed, in step 626. For
`instance, a commonly known entropy encoding methodol-
`ogy that may be used is Huffman encoding. Entropy encod-
`ing involves replacing symbols in the decorrelated image
`data by bit strings in such a way that different symbols are
`represented by binary strings of different variable lengths,
`with the most commonly occurring symbols being repre-
`sented by the shortest binary strings. The logic 522 thus
`provides compressed image data having variable size, for
`instance as seen in FIG. 5 where the scaled 8-bit data is
`
`encoded into compressed data having variable size of 3—16
`bits.
`
`the encoding methodologies for obtaining
`Once again,
`video and still images can be different and may be selected,
`as indicated in step 614, depending on the mode of operation
`identified earlier in step 610. For instance, a larger set of
`symbols (having variable binary string lengths) may be used
`for encoding still image data as compared to video image
`data. This is because there may be more time allocated in the
`host/PC to decompress still
`images than to decompress
`video images. In contrast, for encoding video images, a more
`limited set of symbols having uniform binary string lengths
`should be employed to obtain faster decompression of a
`series of video image frames. In addition, having a uniform
`binary string length allows usage of a fixed amount of
`bandwidth to transmit the image data that is specifically
`suitable for a host/PC interface such as the Universal Serial
`Bus (USE).
`The image processing system shown in FIG. 5 includes
`additional
`logic that facilitates the dual mode operation
`described above. In particular, the logic circuitry in blocks
`510, 514, and 522 use programmable look-up tables (LUTs)
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`533, 534, and 535 and random access memories (RAMs)
`535 for flexibility in performing their respective imaging
`functions. Each LUT or RAM provides information to its
`respective imaging function logic as specified by the
`selected methodology for the particular mode of operation.
`For instance, the scaling logic 514 uses a RAM 235 as a
`storage area to store intermediate scaling computations.
`Also, the LUT 534 for the decorrelation and encoding logic
`522 can be loaded with different rules and data required for
`performing decorrelation and encoding as known in the art,
`depending on whether a still or a video image is desired. In
`a particular embodiment,
`two look-up ta