(12) United States Patent
`LeGall et a].
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 8,243,171 B2
`*Aug. 14, 2012
`
`US008243171B2
`
`(54)
`
`(75)
`
`HIGH RESOLUTION ZOOM: A NOVEL
`DIGITAL ZOOM FOR DIGITAL VIDEO
`CAMERA
`
`Inventors: Didier LeGall, Los Altos, CA (US);
`Leslie D. Kohn, Fremont, CA (US);
`Elliot N. Linzer, Suffem, NY (US)
`
`(73)
`
`Assignee: Ambarella, Inc., Santa Clara, CA (U S)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`USC 154(b) by 112 days.
`This patent is subject to a terminal dis
`claimer.
`
`JP
`
`(21)
`
`Appl. N0.: 12/956,232
`
`(22)
`
`Filed:
`
`Nov. 30, 2010
`
`(65)
`
`(63)
`
`(51)
`
`(52)
`(58)
`
`Prior Publication Data
`
`US 2011/0069206 A1
`
`Mar. 24, 2011
`
`Related US. Application Data
`
`Continuation of application No. 12/716,525, ?led on
`Mar. 3, 2010, noW Pat. No. 7,880,776, Which is a
`continuation of application No. 11/010,032, ?led on
`Dec. 10, 2004, noW Pat. No. 7,688,364.
`
`Int. Cl.
`(2006.01)
`H04N 5/262
`(2006.01)
`H04N 5/228
`US. Cl. ............ .. 348/240.99; 348/2086; 348/2401
`
`Field of Classi?cation Search ............. .. 348/2401,
`348/2402, 240.3, 240.99, 208.6
`See application ?le for complete search history.
`
`120 \
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5/1995 YamagiWa ............... .. 348/2403
`5,420,632 A
`3/1999 MacDougallet a1. .
`423/700
`5,882,625 A
`11/2003 Luo et a1. ............. ..
`382/282
`6,654,506 B1
`6,654,507 B2 11/2003 Luo ....... ..
`382/282
`6,876,386 B1
`4/2005 Ito ..... ..
`348/2401
`6,982,755 B1
`1/2006 KikuZaWa ................... .. 348/241
`7,015,941 B2 *
`3/2006 Malloy Desormeaux .... .. 348/64
`7,221,386 B2
`5/2007 Thacher et a1. .......... .. 348/14.02
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`06-203148
`7/1994
`(Continued)
`
`OTHER PUBLICATIONS
`
`Micron Technology, Inc., Boise, ID, “l/z-lnch 3-MegapiXel CMOS
`Active-Pixel Digital Image Sensor” data sheet, Rev C, Sep. 2004.
`(MT9T001i3100iDSi1.fm-Rev.C9/04EN 2003 Micron Technol
`ogy, Inc. All rights reserved.)
`
`Primary Examiner * Hung Lam
`(74) Attorney, Agent, or Firm * Christopher P. Maiorana,
`PC
`
`ABSTRACT
`(57)
`A camera system and a method for Zooming the camera
`system is disclosed. The method generally includes the steps
`of (A) generating an electronic image by sensing an optical
`image received by the camera, the sensing including elec
`tronic cropping to a Window siZe to establish an initial reso
`lution for the electronic image, (B) generating a ?nal image
`by decimating the electronic image by a decimation factor to
`a ?nal resolution smaller than the initial resolution and (C)
`changing a Zoom factor for the ?nal image by adjusting both
`of the decimation factor and the Window siZe.
`
`20 Claims, 8 Drawing Sheets
`
`154
`
`130
`
`LENS
`ASSEMBLY
`
`152
`
`ASSEMBLY l
`
`MOTOR
`
`/128
`/12s
`l USERINPUT
`
`L
`
`CMD
`
`147
`
`146
`
`124 \
`
`122 \
`/
`
`MAIN CIRCUIT
`
`FRMCNT
`
`SENSOR
`ARRAY
`
`150 /
`
`A
`
`SCNT
`V50
`148
`
`LENS MOTOR
`CONTROLLER
`
`DECIMATION
`FILTER
`
`PROCESS
`
`D
`
`*
`
`162
`
`DETECTOR
`
`135
`
`126
`
`138
`v 140
`
`MEMORY
`
`Qomo_1004
`
`

`
`U.S. PATENT DOCUMENTS
`7,227,573
`7,401,007
`7,417,670
`7,477,297
`2002/0126208
`2004/0189830
`2005/0046710
`2005/0052646
`
`6/2007 Stavely ................. ..
`7/2008 Su ............. ..
`8/2008 LinZer et 31.
`1/2009 Pollard ..... ..
`9/2002 Misue et al.
`9/2004 Pollard
`3/ 2005 Miyazaki ......... ..
`3/ 2005 Wohlstadter et a1. .
`
`B2
`B1 *
`B1
`B2
`A1 *
`A1
`A1
`A1
`
`US 8,243,171 B2
`Page 2
`
`34786510;
`""348/222l
`, 348/240'1
`348/211
`. 348/240.1
`348/239
`. 356/311
`
`2005/0078205 A1
`2005/0083556 A1
`2005/0093982 A1
`2006/0077269 A1
`2009/0028413 A1
`
`4/2005
`4/2005
`5/2005
`4/2006
`1/2009
`
`Hynecek ..................... .. 348/294
`Carlson .
`358/474
`Kuroki
`348/20799
`Kindt et al. ................. .. 348/294
`Goodwin et al. ........... .. 382/133
`
`FOREIGN PATENT DOCUMENTS
`6/2000
`
`JP
`2000-184259
`* cited by examiner
`
`Qomo_1004
`
`

`
`U.S. Patent
`
`Aug. 14, 2012
`
`Sheet 1 of8
`
`US 8,243,171 B2
`
`80\
`
`
`
`FIG. 1
`
`82a
`
`82b
`
`82e
`
`82f
`
`82i
`
`82'
`
`96
`
`82c
`
`82d
`
`82
`
`82h
`
`82k
`
`82|
`
`82m
`
`82n
`
`82
`
`82r
`
`82u
`
`82v
`
`82o
`
`82
`
`82s
`
`82t
`
`82w
`
`82x
`
`82
`
`822
`
`82cc
`
`82dd 82 82hh
`
`82aa
`
`82bb
`
`82ee
`
`82ff
`
`82ii
`
`82'
`
`FIG. 2
`
`98a
`
`98b
`
` 98C
`
`98d
`
`Qomo_1004
`
`Qomo_1004
`
`

`
`US. Patent
`
`Aug. 14, 2012
`
`Sheet 2 of8
`
`US 8,243,171 B2
`
`102
`104
`106
`108
`110
`112
`MAW
`
`CROPPING
`BIN A
`| BIN B
`
`OPTICAL
`BIN B
`
`CROPPING
`BIN B | BIN c
`
`BIN c
`
`DECIMATE
`
`INTERPOLATE
`
`ZOOM OUT
`
`ZOOM IN
`
`b
`
`164 \
`
`DECIMATION
`FILTER
`
`INITIAL
`ASPECT
`RATIo
`(E.G.,1:1)
`
`198 \
`196 \
`D ’ HORIZONTAL _> VERTICAL
`FILTER
`FILTER
`
`FINAL
`G ’ ASPECT
`RATIo
`(E.G., 4:3)
`
`FIG. 6
`
`Qomo_1004
`
`

`
`US. Patent
`
`Aug. 14, 2012
`
`Sheet 3 of8
`
`US 8,243,171 B2
`
`120 \
`
`154
`
`130 \
`
`r 128
`
`/- 129
`
`LENS
`ASSEMBLY \ 152
`
`MOTOR
`ASSEMBLY
`
`USER INPUT
`
`122\
`I \
`
`124\
`
`\
`147
`CMD
`/146
`MAIN CIRCUIT
`
`L
`
`\
`144
`/142
`/170
`+
`LENS MOTOR
`CONTROLLER
`
`FRMCNT
`
`\166
`
`SENSOR
`
`/
`160
`
`A
`
`PROCESS
`162 /
`
`DETECTOR
`
`\ D
`132 /
`134
`
`CONTROLLER
`-\<SCNT
`150 /
`148 DFCNT+
`|NTCNT+
`DECIMATION
`FILTER
`Q T
`
`\ 164
`
`INTERPOLATOR
`+ I
`H
`
`168 \
`
`G
`—> FORMAT
`
`OUT
`\ >
`136
`
`‘138
`F
`126 \ / 140
`
`MEMORY
`
`FIG. 4
`
`Qomo_1004
`
`

`
`US. Patent
`
`Aug. 14, 2012
`
`Sheet 4 as
`
`US 8,243,171 B2
`
`176 I»
`
`USER COMMAND
`
`I
`ADJUST CROPPING,
`178%
`BINNING AND/OR OPTICS
`I
`CONVERT OPTICAL
`TO ELECTRICAL
`I
`
`180'»
`
`182 m ANALOG PROCESSING
`
`184/»
`
`186~
`
`188
`
`I
`DIGITIZE ELECTRICAL
`SIGNAL
`I
`PICTURE QUALITY
`DIGITAL PROCESSING
`
`INTERPOLATION
`FLAG = ON?
`
`194\
`INTERPOLATION
`
`190 w DECIMATION
`
`192 w FORMATTING
`
`@
`
`FIG. 5
`
`Qomo_1004
`
`

`
`US. Patent
`
`Aug. 14, 2012
`
`Sheet 5 018
`
`US 8,243,171 B2
`
`mm;
`
`f SN
`
`wow
`
`Qomo_1004
`
`

`
`US. Patent
`
`Aug. 14, 2012
`
`Sheet 6 018
`
`US 8,243,171 B2
`
`4 3 2
`
`236
`
`242
`
`236
`
`8 4 2
`
`246
`
`252
`
`246
`
`2 5 2
`
`246
`
`{-244
`
`r254
`
`Qomo_1004
`
`

`
`US. Patent
`
`Aug. 14, 2012
`
`Sheet 7 018
`
`US 8,243,171 B2
`
`262 m
`
`260 \
`
`USER LOW LIGHT
`COMMAND
`
`I
`
`ADJUST WINDOW SIZE TO
`264 A, MAXIMIZE AREA UTILIZED
`ON SENSOR
`
`I
`
`ADJUST OPTICAL ZOOM
`FACTOR TO PRODUCE
`COMMANDED ZOOM
`
`266 N
`
`I
`
`ADJUST BIN RATIO TO
`268 N MINIMUM VALUE THAT
`ALLOWS SUFFICIENT TIME
`TO READ WINDOW
`
`I
`
`ADJUST DECIMATION
`FACTOR
`
`270 N
`
`END
`
`FIG. 9
`
`Qomo_1004
`
`

`
`US. Patent
`
`Aug. 14, 2012
`
`Sheet 8 0f 8
`
`US 8,243,171 B2
`
`272 \
`
`274 N
`
`USER LOW DISTORTION
`COMMAND
`
`COMMAND BINNING OFF
`
`I
`I
`
`ADJUST WINDOW SIZE TO
`MAXIMUM VALUE THAT
`278 N
`ALLOWS SUFFICIENT TIME
`TO READ WINDOW
`
`I
`
`ADJUST OPTICAL ZOOM
`FACTOR TO PRODUCE
`COMMANDED ZOOM
`
`I
`
`ADJUST DECIMATION
`FACTOR
`
`280 N
`
`282 w
`
`FIG. 10
`
`Qomo_1004
`
`

`
`US 8,243,171 B2
`
`1
`HIGH RESOLUTION ZOOM: A NOVEL
`DIGITAL ZOOM FOR DIGITAL VIDEO
`CAMERA
`
`This is a continuation of US. Ser. No. 12/716,525, ?led
`Mar. 3, 2010 now US. Pat. No. 7,880,776, Which is a con
`tinuation of US. Ser. No. 11/010,032, ?led Dec. 10, 2004,
`now US. Pat. No. 7,688,364, Which are each incorporated by
`reference.
`
`FIELD OF THE INVENTION
`
`The present invention relates to video image processing
`generally and, more particularly, to a digital Zoom for digital
`video cameras.
`
`BACKGROUND OF THE INVENTION
`
`Functionality of conventional Digital Still Cameras (DSC)
`and conventional camcorders are converging. The DSCs
`implement sensors (i.e., CCD or CMOS) With at least 4 to 5
`million pixels. A video signal in a typical camcorder is
`acquired at 30 to 60 frames per seconds With a resolution
`varying from 720x480 (i.e., standard de?nition) to 1920><
`1080 (i.e., high de?nition) vieWable pixels. The availability of
`sensors that can combine both a high pixel number to accom
`modate DSCs and a transfer rate to accommodate video
`alloWs an introduction of a neW digital Zoom function that is
`quite different from the current digital Zoom function used in
`conventional cameras and camcorders.
`A conventional digital Zoom operation, also called “inter
`polated” Zoom, is achieved by calculating an up-conversion
`of a WindoW in existing image data to generate an enlarged
`version. Interpolated Zoom is achieved by cropping a WindoW
`in a standard resolution picture and enlarging the WindoW by
`interpolation. The resulting image has a progressively
`decreasing resolution as the cropping factor increases. The
`decreasing spatial resolution has created a feeling among
`users that digital Zoom is a technique inferior to a true optical
`Zoom.
`
`SUMMARY OF THE INVENTION
`
`The present invention concerns a camera system and a
`method for Zooming the camera. The method generally com
`prises the steps of (A) generating an electronic image by
`sensing an optical image received by the camera, the sensing
`including electronic cropping to a WindoW siZe to establish an
`initial resolution for the electronic image, (B) generating a
`?nal image by decimating the electronic image by a decima
`tion factor to a ?nal resolution smaller than the initial resolu
`tion and (C) changing a Zoom factor for the ?nal image by
`adjusting both of the decimation factor and the WindoW siZe.
`The objects, features and advantages of the present inven
`tion include providing a Zooming method and a camera sys
`tem that may provide (i) a high resolution digital Zoom capa
`bility, (ii) a substantially constant output image resolution at
`different Zoom levels, (iii) a loW light mode, (iv) a loW dis
`tortion mode, (v) a digital Zoom capable of operating With
`camcorder speed data and/or (vi) a loW-cost alternative to
`high Zoom optics.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`These and other objects, features and advantages of the
`present invention Will be apparent from the folloWing detailed
`description and the appended claims and draWings in Which:
`
`65
`
`2
`FIG. 1 is a block diagram of a ?rst example binning pro
`cess;
`FIG. 2 is a block diagram of a second example binning
`process;
`FIG. 3 is a block diagram illustration a Zooming operation;
`FIG. 4 is a block diagram of an example implementation of
`a system in accordance With a preferred embodiment of the
`present invention;
`FIG. 5 is a How diagram of an example method of process
`ing an optical image;
`FIG. 6 is a block diagram of an example implementation of
`a decimation ?lter circuit;
`FIG. 7 is a How diagram of an example Zoom-in process;
`FIGS. 8A-8E are block diagrams illustrating various Win
`doW siZe and bin ratio settings;
`FIG. 9 is a How diagram of an example method for con?g
`uring a loW light mode; and
`FIG. 10 is a How diagram of an example method for con
`?guring a loW distortion mode.
`
`DETAILED DESCRIPTION OF THE PREFERRED
`EMBODIMENTS
`
`The present invention described herein may be referred to
`as a High Resolution (HR) Zoom. HR Zoom generally alloWs
`a user to control a Zoom factor (or level) by electronically
`cropping (or WindoWing) a sensor area detecting an optical
`image, digitiZing a resulting electrical image and doWn-sam
`pling the electronic image to a ?nal resolution. An amount of
`cropping and a level of doWn-sampling may be controlled so
`that the ?nal resolution is substantially constant over a range
`of different Zoom factors. In HR Zoom, a subjective digital
`Zoom effect may appear much like an optical Zoom in that
`pictures generally remain sharp throughout Zoom-in and
`Zoom-out operations.
`In video applications Where a standard resolution video
`(e.g., International Telecommunications Union-Radiocom
`munications Sector, Recommendation BT.656-4 (February
`1998), Geneva, SWitZerland) may be implemented, the HR
`Zoom process generally alloWs a high resolution electronic
`Zoom-in ratio up to a square root of a raW data capture rate
`divided by a video resolution rate. For example, a sensor
`element capture rate of 60 megahertz (MHZ) may yield about
`2 million pixels per frame at 30 frames per second. Decimat
`ing the image data doWn to a 720x480 pixel video resolution
`at 30 frames per second (e.g., 10.37 MHZ) generally alloWs a
`2.4 maximum Zoom factor. In practice, the raW data input data
`rate may be limited by a speed of an optical sensor array. The
`high image resolution available in modern optical sensor
`arrays generally permits acquisition of raW image data at a
`resolution signi?cantly higher than standard video resolution,
`thus alloWing the implementation of the HR Zoom process.
`Sensor arrays for Digital Still Cameras (DCS) and cam
`corders generally have a large number of individual optical
`to-electrical sensor elements. Therefore, reading all the sen
`sor elements (e.g., 5 million elements or pixels) in the sensor
`array may not be possible in a video frame period (e.g., 1/3oth
`to 1/6oth of a second). One or more of the folloWing techniques
`may be used to reduce an output rate from a sensor array. A
`?rst technique may be to skip some sensor element roWs
`and/or columns to cover the Whole sensor array area, but not
`present all of the available data. A second technique, called
`binning, may be implemented to reduce (i) a total amount of
`data presented and (ii) an impact of aliasing due to sub
`sampling. In binning, multiple sensor element sets may be
`combined to create a binned set. The binning technique gen
`erally has multiple advantages over sub-sampling (e.g., skip
`
`Qomo_1004
`
`

`
`US 8,243,171 B2
`
`3
`ping) since both aliasing and photon noise inherent to optical
`detection may be reduced by the combination of the collected
`samples.
`A sensor array (e.g., a CCD array or a CMOS array) may be
`either a monochrome sensor array or a color sensor array. In
`the case of a monochrome sensor array, three monochrome
`sensor arrays may be implemented to generate a color image.
`In the case of the color sensor array, a mosaic of red, green and
`blue color ?lters is generally applied on the sensor surface.
`The most common mosaic pattern is called a Bayer pattern
`consisting of tWo green cells and one each of a red cell and a
`blue cell. Applications of the HR Zoom process With Bayer
`patterned sensor arrays generally include a conversion step to
`red-green-blue (RGB) data before ?ltered decimation in RGB
`space. The conversion step is commonly called de-mosaicing.
`Referring to FIG. 1, a block diagram of a ?rst example
`binning process 80 is shoWn. The ?rst binning process 80
`generally illustrates a bin ratio of 2:1 in each of a horizontal
`and a vertical direction. In particular, individual image ele
`ments 82a-82p generated in a set 84 of adjoining sensor
`elements (e.g., a 4x4 set) may be combined to form a binned
`set 86 de?ning feWer image elements 88a-88d. The binned set
`86 generally de?nes R, B and tWo G values.
`The original set 84 generally comprises four Bayer sets of
`sensor elements. Each Bayer set generally comprises a sensor
`element de?ning a red value and a location (e.g., R), a sensor
`element de?ning a blue value and a location (e.g., B) and tWo
`sensor elements de?ning tWo green values and tWo locations
`(e. g., Gr and Gb). The green sensor element Gr may be
`located on a same roW as the red sensor element R. The green
`sensor element Gb may be located on a same roW as the blue
`sensor element B. The binned set 86 may folloW the Bayer
`pattern de?ning a virtual red sensor element (e.g., Rs), a
`virtual blue sensor element (e.g., Bs) and tWo virtual green
`sensor elements (e.g., Grs and Gbs). Other sensor element
`layouts and color patterns may be implemented to meet the
`criteria of a particular application.
`An effect of the 2:1 bin ratio in each direction may be to
`reduce an image data rate from the sensor array by a factor of
`four While still alloWing all of the photons striking the original
`set 84 to contribute to the binned set 86. Therefore, the maxi
`mum Zoom factor may increase since the raW data capture rate
`is generally reduced. For example, the 2:1 bin ratio in each
`direction and a 5 million pixel sensor array generally yields a
`3.80 maximum Zoom factor as long as the raW capture rate is
`greater than 40 MHZ.
`Referring to FIG. 2, a block diagram of a second example
`binning process 90 is shoWn. The second binning process 90
`generally illustrates a bin ratio of 3:1 in each direction. In
`particular, individual image elements 82a-82jj' generated
`Within a set 94 may be combined to form a binned set 96. Each
`binned image element 98a-98d may be generated from the
`original set 94. An effect of the 3 :1 bin ratio in each direction
`may be to reduce an image data rate from the sensor array by
`a factor of nine While still alloWing all of the photons striking
`the original set 94 to contribute to the binned set 96. Other bin
`ratios and/or combinations of different bin ratios in each
`direction may be implemented to meet the criteria of a par
`ticular application.
`Referring to FIG. 3, a block diagram illustrating a Zooming
`operation 100 is shoWn. The operation 100 may comprise a
`section (or range) 102, a section (or range) 104, a section (or
`range) 106, a section (or range) 108, a section (or range) 110
`and a section (or range) 112.Any given horiZontal point in the
`?gure may be considered a unique Zoom factor. Movement
`toWard the left (e.g., Zoom out) in the ?gure generally pro
`duces a Wider ?eld of vieW (e.g., a smaller Zoom factor).
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`Movement toWard the right (e.g., Zoom in) in the ?gure gen
`erally produces a narroWer ?eld of vieW (e. g., a larger Zoom
`factor).
`From a given Zoom factor, a cropping operation and an
`optional optical Zooming operation may be used to increase
`and decrease the Zoom factor. A full range of Zoom factors
`may be achieved using cropping over a loW Zoom range (e. g.,
`sections 102 and 104), optical Zooming over a medium Zoom
`range (e. g., section 106) and cropping over a high Zoom range
`(e.g., sections 108 and 110). Zooming beyond the high Zoom
`range may be achieved using an interpolation operation (e. g.,
`section 112).
`Zooming may be achieved by a user pressing controls to
`Zoom in or Zoom out. The pressed control generally causes the
`camera to utiliZe a cropping setting and an optical Zoom
`setting corresponding to a position in FIG. 3. If the current
`amount of Zoom corresponds to the Zoom level in one of the
`sections 102, 104,108,110 or 112, more or less Zoom may be
`achieved by more or less cropping. For example, if the user
`commands “Zoom in”, more cropping may be used. If the user
`commands “Zoom out”, less cropping is used. If the current
`amount of Zoom corresponds to a Zoom level in the section
`106, Zooming may be achieved by optical Zooming. For
`example, if the user commands “Zoom in” Within the section
`106, more optical Zoom may be used. If the user commands
`“Zoom out” Within the section 106, less optical Zoom may be
`used.
`When the camera changes sections, the method of Zooming
`may also change. For example, if the amount of Zoom corre
`sponds to the section 104 and the user selects to Zoom further
`in, more and more cropping (e.g., less and less sensor area)
`may be used until the section 106 is reached. Thereafter, the
`amount of cropping may remain constant and increasing opti
`cal Zooming may be used until the section 108 is reached.
`Thereafter, more and more cropping may be used.
`Conversely, if the amount of Zoom corresponds to the sec
`tion 108 and the user selects to Zoom further out, less and less
`cropping may be user until the section 106 is reached. There
`after, the amount of cropping may remain constant and
`decreasing optical Zooming may be used until the section 104
`is reached. Thereafter, less and less cropping may be used
`While the optical Zooming remains constant.
`Other arrangements of the cropping, optical Zooming and
`interpolation may be implemented to meet the criteria of a
`particular implementation. In a ?rst example of another
`arrangement, the sections 102 and 104 may be eliminated
`such that the optical Zooming in section 106 covers a loWest
`range of Zoom factors. In a second example of another
`arrangement, optical Zooming in the section 106 may be
`eliminated such that the camera only implements electronic
`Zooming by cropping and possibly interpolating. In a third
`example, the sections 108 and 110 may be eliminated such
`that interpolating in section 112 adjoins the optical Zooming
`in section 106.
`For sensors implementing a binning capability, a bin ratio
`in one or both directions may be changed one or more times.
`For example, in the section 102, a ?rst bin ratio (e.g., BIN A)
`may be utiliZed. In the sections 104, 106 and 108 a second bin
`ratio (e.g., BIN B) may be used. In the sections 110 and 112,
`a third bin ratio (e. g., BIN C) may be used. Generally, BIN A
`has a higher bin ratio than BIN B. BIN B generally has a
`higher bin ratio than BIN C. BIN C may have a 1:1 bin ratio
`in each direction (e.g., binning off or no binning).
`The sections 102-110 may use various cropping WindoWs
`to achieve various Zoom factors. Zooming out generally
`means decreasing a WindoW siZe (e.g., increasing sensor
`area). Zooming in generally means increasing the WindoW
`
`Qomo_1004
`
`

`
`US 8,243,171 B2
`
`5
`size (e.g., decreasing sensor area). More cropping generally
`alloWs for less binning. The section 112 may also implement
`cropping to a WindoW siZe matching the ?nal resolution.
`Referring to FIG. 4, a block diagram of an example imple
`mentation of a system 120 is shoWn in accordance With a
`preferred embodiment of the present invention. The system
`120 may be suitable for applications in digital still cameras
`and camcorders. The system 120 generally comprises a cir
`cuit (or module) 122, a circuit (or module) 124, a circuit (or
`module) 126, a circuit (or module) 128, a circuit (or module)
`129 and an assembly (or module) 130.
`A signal (e.g., D) may be presented at an output 132 of the
`circuit 122 to an input 134 of the circuit 124. A signal (e.g.,
`OUT) may be generated at an output 13 6 of the circuit 124 . An
`interface 138 of the circuit 124 may be connected to an
`interface 140 of the circuit 128 to transfer a signal (e.g., F). A
`signal (e.g., L) may be presented from an output 142 of the
`circuit 124 to an input 144 of the circuit 128. A signal (e.g.,
`CMD) may be received at an input 146 of the circuit 124 from
`an output 147 ofthe circuit 129. A signal (e.g., SCNT) may be
`presented from an output 148 of the circuit 124 to an input 150
`of the circuit 122. A mechanical linkage 152 may be disposed
`betWeen the circuit 128 and the assembly 130. The assembly
`130 may focus an optical signal (e.g., light) 154 onto a surface
`of the circuit 122 to form an optical image.
`The circuit 122 may be referred to as a detector circuit. The
`detector circuit 122 may be operational to convert the optical
`image received from the assembly 130 into the digital signal
`D in response to the control signal SCNT. The digital signal D
`may convey one or more optical images as one or more
`electronic images. The control signal SCNT may carry Win
`doWing, binning, read rate, offset, scaling, color correction
`and other information for use by the detector circuit 122. The
`electronic images may be con?gured to have an initial reso
`lution (e. g., a horizontal number of image elements by a
`vertical number of image elements) and an initial data rate.
`The circuit 124 may be referred to as a main circuit. The
`main circuit 124 may be con?gured to generate the signal
`OUT by processing the one or more electronic images
`received in the digital signal D as instructed by a user via the
`command signal CMD. The main circuit 124 may be opera
`tional to generate the control signal SCNT and the signal L in
`response to the command signal CMD. The signal OUT gen
`erally comprises a still image (e.g., JPEG) or a video bit
`stream (e.g., ITU-R BT.656-4) having a sequence of images
`(or pictures). The picture or pictures carried by the signal
`OUT may be con?gured to have a ?nal resolution smaller
`than the initial resolution of the electronic images in the
`digital signal D. The command signal CMD may carry Zoom
`factor commands and optional mode commands from the
`user. In one embodiment, the detector circuit 122 and the
`main circuit 124 may be fabricated on separate dies. In
`another embodiment, the detector circuit 122 and the main
`circuit 124 may be fabricated on the same die.
`The circuit 126 may be referred to as a memory circuit. The
`memory circuit 126 may be operational to temporarily store
`image data (e.g., luminance and chrominance) for the main
`circuit 124. In one embodiment, the memory circuit 126 may
`be fabricated as one or more dies separate from the main
`circuit 124 fabrication. In another embodiment, the memory
`circuit 126 may be fabricated on the same die as the main
`circuit 124.
`The circuit 128 may be referred to as a motor assembly. The
`motor assembly 128 may be operational to actuate the linkage
`152 in response to the signal L. The linkage 152 generally
`comprises a ?rst mechanical element for focus control and an
`independent second mechanical element for Zoom control.
`
`50
`
`55
`
`60
`
`65
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`6
`The signal L generally carries command information for the
`focus operation and command information for the optical
`Zoom operation. The signal L may also carry position and/or
`velocity feedback data from the motor assembly 128 back to
`the main circuit 124.
`The circuit 129 may be referred to as a user input circuit.
`The user input circuit 129 may be operational to generate the
`signal CMD based on commands received from a user. The
`commands received may include, but are not limited to, a
`Zoom in command, a Zoom out command, a normal mode, a
`loW light mode and a loW distortion mode. In one embodi
`ment, the signal CMD may comprise multiple discrete signals
`(e.g., one signal for each sWitch implemented in the user input
`circuit 129). In another embodiment, the signal CMD may
`carry the user entered commands in a multiplexed fashion as
`one or a feW signals.
`The assembly 130 may be referred to as a lens assembly.
`The lens assembly 130 may be operational to optically Zoom
`and optically focus the optical signal 154 onto a surface of the
`detector circuit 122 to form the optical image. The optical
`image may vary over time and thus may be considered a
`sequence of optical images. Focusing may be controlled by
`the ?rst mechanical element of the linkage 152. Zooming may
`be controlled by the second mechanical element of the link
`age 152.
`The detector circuit 122 generally comprises a sensor array
`160 and a circuit (or module) 162. The sensor array 160 may
`be operational to convert the optical image generated by the
`assembly 130 into a series of values in a signal (e.g., A). The
`values conveyed in the signal A may be analog voltages
`representing a luminance value at a predetermined color for
`each individual sensor element of the sensor array 160. The
`sensor array 160 may include an electronic cropping (or Win
`doWing) capability. The electronic cropping capability may
`be operational to limit readout of image elements in a WindoW
`(or an active area) of the sensor array 160. The circuit 162 may
`be operational to process and convert the analog signal A to
`generate the digital signal D.
`Processing of the electronic images may include, but is not
`limited to, analog gain for color corrections, analog offset
`adjustments for black level calibrations, digital gain for color
`corrections and digital offsets for color corrections. The con
`version generally comprises an analog to digital conversion
`(e.g., 10-bit) and color space conversion (e.g., Bayer to RGB).
`An example implementation of the detector circuit 122 may
`be an MT9T001 3-megapixel digital image sensor available
`from Micron Technology, Inc., Bosie, Id. Operations of the
`MT9T001 sensor are generally described in a document,
`“Micron, 1/2-inch, 3-megapixels CMOS Active-Pixel Digital
`Image Sensor”, Preliminary Datasheet, MT9T001, Septem
`ber 2004, by Micron Technology Inc ., hereby incorporated by
`reference in its entirety.
`The main circuit 124 generally comprises a circuit (or
`module) 164, an optional circuit (or module) 166, a circuit (or
`module) 168, a circuit (or module) 170 and a circuit (or
`module) 172. The circuits 164 and 166 may receive the digital
`signal D from the circuit 162 and exchange the signal F with
`the memory 126. The circuit 168 may generate the signal
`OUT. The circuit 170 may generate the signal L. The circuit
`172 may receive the command signal CMD and generate the
`control signal SCNT.
`A signal (e. g., G) may be provided by the circuit 164 to the
`circuit 168. A signal (e.g., H) may be provided by the circuit
`166 to the circuit 168. The circuit 172 may provide a control
`signal (e. g., LMCNT) to the circuit 170. The circuit 172 may
`also provide a control signal (e.g., DFCNT) to the circuit 164.
`A control signal (e.g., INTCNT) may be transferred from the
`
`Qomo_1004
`
`

`
`US 8,243,171 B2
`
`7
`circuit 172 to the circuit 166. A control signal (e.g., FRM
`CNT) may be presented from the circuit 172 to the circuit 168.
`The circuit 164 may be referred to as a decimation ?lter
`circuit. The decimation ?lter circuit 164 may be operational
`to generate one or more intermediate (decimated) images in
`the signal G by decimating the one or more electronic images
`in the digital signal D. An amount of horizontal decimation
`and/ or vertical decimation may be determined by the control
`signal DFCNT. The intermediate images in the signal G may
`be con?gured to have a ?nal resolution smaller than the initial
`resolution conveyed in the signal D.
`The circuit 166 may be referred to as an interpolation
`circuit. The interpolation circuit 166 may be operational to
`generate one or more intermediate (interpolated) electronic
`images in the signal H by interpolating the one or more
`images in the signal D. The interpolated images may be
`con?gured to have the ?nal resolution, similar to the deci
`mated images.
`The circuit 168 may be referred to as a format circuit. The
`format circuit 168 may be operational to generate the video
`signal OUT by formatting the one or more intermediate
`images in the signal G or the signal H, one signal at a time.
`The format circuit 168 may generate the signal OUT in fur
`ther response to the control signal FRMCNT. The signal
`FRMCNT may also command the format circuit 168 to select
`betWeen the signal G and the signal H.
`The circuit 170 may be referred to as a lens motor control
`ler circuit. The lens motor controller circuit 170 may be
`operational to generate the signal L in response to control
`information received the control signal LMCNT and any
`feedback data received from the motor assembly 128.
`The circuit 172 may be referred to as a controller circuit.
`The controller circuit 172 may be operational to generate the
`control signals LMCNT, DFCNT, INTCNT, FRMCNT and
`SCNT in response to the command signal CMD and limita
`tions of the detector circuit 122. In particular, the controller
`circuit 172 may be con?gured to generate the signal DFCNT,
`SCNT and LMCNT to implement a normal operating mode,
`a loW light mode and a loW distortion mode of the HR Zoom
`process. The control signal INTCNT may control interpola
`tion operations. The control signal FRMCNT may control
`formatting operations.
`Referring to FIG. 5, a How diagram of an example method
`174 of processing an optical image is shoWn. The method (or
`process) 174 generally comprises a step (or block) 176, a step
`(or block) 178, a step (or block) 180, a step (or block) 182, a
`step (or block) 184, a step (or block) 186, a step (or block)
`188, a step (or block) 190, a step (or block) 192 and a step (or
`block) 194. The method 174 is generally applied during the
`normal mode.
`A description of the method 174 generally starts With a
`reception of one or more user commands at the user input
`circuit 129 (e.g., step 176). The controller circuit 172 may
`generate (i) the control signal SCNT to adjust cropping and
`binning in the detector circuit 122 and/or (ii) the control
`signal LMCNT to adjust an optical zoom of the lens assembly
`130 (e.g., step 178) in response to commands received in the
`command signal CMD. The sensor array 160 may convert the
`optical image to an electrical image (e.g., step 180). The
`processing circuit 162 may perform analog processing (e.g.,
`step 182), analog to digital conversion (e.g., step 184) and
`picture quality digital processing (e.g., step 186) to generate
`the signal D.
`A check may be performed on an interpolation ?ag (e.g.,
`step 188) to determine if the system 120 should include inter
`polation (e.g., ON) or not (e.g., OFF). If the interpolation ?ag
`is off (e.g., the NO branch), the