Exhibit 1004
`Exhibit 1004
`
`

`
`(12) United States Patent
`LeGall et a].
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 8,243,171 B2
`*Aug. 14, 2012
`
`US008243171B2
`
`(54)
`
`(75)
`
`HIGH RESOLUTION ZOOM: A NOVEL
`DIGITAL ZOOM FOR DIGITAL VIDEO
`CAMERA
`
`Inventors: Didier LeGall, Los Altos, CA (US);
`Leslie D. Kohn, Fremont, CA (US);
`Elliot N. Linzer, Suffem, NY (US)
`
`(73)
`
`Assignee: Ambarella, Inc., Santa Clara, CA (U S)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`USC 154(b) by 112 days.
`This patent is subject to a terminal dis
`claimer.
`
`JP
`
`(21)
`
`Appl. N0.: 12/956,232
`
`(22)
`
`Filed:
`
`Nov. 30, 2010
`
`(65)
`
`(63)
`
`(51)
`
`(52)
`(58)
`
`Prior Publication Data
`
`US 2011/0069206 A1
`
`Mar. 24, 2011
`
`Related US. Application Data
`
`Continuation of application No. 12/716,525, ?led on
`Mar. 3, 2010, noW Pat. No. 7,880,776, Which is a
`continuation of application No. 11/010,032, ?led on
`Dec. 10, 2004, noW Pat. No. 7,688,364.
`
`Int. Cl.
`(2006.01)
`H04N 5/262
`(2006.01)
`H04N 5/228
`US. Cl. ............ .. 348/240.99; 348/2086; 348/2401
`
`Field of Classi?cation Search ............. .. 348/2401,
`348/2402, 240.3, 240.99, 208.6
`See application ?le for complete search history.
`
`120 \
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5/1995 YamagiWa ............... .. 348/2403
`5,420,632 A
`3/1999 MacDougallet a1. .
`423/700
`5,882,625 A
`11/2003 Luo et a1. ............. ..
`382/282
`6,654,506 B1
`6,654,507 B2 11/2003 Luo ....... ..
`382/282
`6,876,386 B1
`4/2005 Ito ..... ..
`348/2401
`6,982,755 B1
`1/2006 KikuZaWa ................... .. 348/241
`7,015,941 B2 *
`3/2006 Malloy Desormeaux .... .. 348/64
`7,221,386 B2
`5/2007 Thacher et a1. .......... .. 348/14.02
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`06-203148
`7/1994
`(Continued)
`
`OTHER PUBLICATIONS
`
`Micron Technology, Inc., Boise, ID, “l/z-lnch 3-MegapiXel CMOS
`Active-Pixel Digital Image Sensor” data sheet, Rev C, Sep. 2004.
`(MT9T001i3100iDSi1.fm-Rev.C9/04EN 2003 Micron Technol
`ogy, Inc. All rights reserved.)
`
`Primary Examiner * Hung Lam
`(74) Attorney, Agent, or Firm * Christopher P. Maiorana,
`PC
`
`ABSTRACT
`(57)
`A camera system and a method for Zooming the camera
`system is disclosed. The method generally includes the steps
`of (A) generating an electronic image by sensing an optical
`image received by the camera, the sensing including elec
`tronic cropping to a Window siZe to establish an initial reso
`lution for the electronic image, (B) generating a ?nal image
`by decimating the electronic image by a decimation factor to
`a ?nal resolution smaller than the initial resolution and (C)
`changing a Zoom factor for the ?nal image by adjusting both
`of the decimation factor and the Window siZe.
`
`20 Claims, 8 Drawing Sheets
`
`154
`
`130
`
`LENS
`ASSEMBLY
`
`152
`
`ASSEMBLY l
`
`MOTOR
`
`/128
`/12s
`l USERINPUT
`
`L
`
`CMD
`
`147
`
`146
`
`124 \
`
`122 \
`/
`
`MAIN CIRCUIT
`
`FRMCNT
`
`SENSOR
`ARRAY
`
`150 /
`
`A
`
`SCNT
`V50
`148
`
`LENS MOTOR
`CONTROLLER
`
`DECIMATION
`FILTER
`
`PROCESS
`
`D
`
`*
`
`162
`
`DETECTOR
`
`135
`
`126
`
`138
`v 140
`
`MEMORY
`
`

`
`US 8,243,171 B2
`US 8,243,171 B2
`Page 2
`Page 2
`
`4/2005
`2005/0078205 A1
`Hynecek ..................... .. 348/294
`4/2005
`2005/0078205 A1
`Carlson .
`358/474
`Hynecek ..................... .. 348/294
`4/2005
`2005/0083556 A1
`Kuroki
`.... ..
`348/207.99
`Carlson .
`358/474
`ggggg
`3333333333; :1
`
`5/2005
`2005/0093982 A1
`Kuroki
`348/20799
`Kindt et al.
`................. .. 348/294
`4/2006
`2006/0077269 A1
`4/2006
`2006/0077269 A1
`Kindt et al. ................. .. 348/294
`Goodwin et al.
`........... .. 382/133
`1/2009
`2009/0028413 A1
`Goodwin et al. ........... .. 382/133
`1/2009
`2009/0028413 A1
`FOREIGN PATENT DOCUMENTS
`FOREIGN PATENT DOCUMENTS
`6/2000
`6/2000
`
`JP
`2000-184259
`JP
`2000484259
`* cited by examiner
`* cited by examiner
`
`U.S. PATENT DOCUMENTS
`U.S. PATENT DOCUMENTS
`7,227,573
`B2
`6/2007 Stavely ,,,,,,,,,,,,,,,,, :0
`7,227,573
`B2
`6/2007 Stavely ................. ..
`7,401,007
`131*
`7/2008 Su ............. ..
`7,401,007
`B1 *
`7/2008 Su ............. ..
`7,417,670
`7,417,670
`B1
`8/2008 LinZer et 31.
`B1
`8/2008 Linzeretal
`B2
`1/2009 Pollard ..... ..
`7,477,297
`7,477,297
`2002/0126208
`A1*
`9/2002 Misue etal
`B2
`1/2009 Pollard ..... ..
`2002/0126208
`A1 *
`9/2002 Misue et al.
`2004/0189830
`A1
`9/2004 Pollard
`2004/0189830
`A1
`9/2004 Pollard
`2005/0046710
`A1
`3/2005 Miyazaki
`......... ..
`2005/0046710
`A1
`3/ 2005 Miyazaki ......... ..
`2005/0052646
`
`A1
`3/2005 Wohlstadter et a1.
`.
`2005/0052646
`A1
`3/ 2005 Wohlstadter et a1. .
`
`
`
`348/240.2
`34786510;
`702/189
`""348/222l
`.348/222.1
`. 348/240.1
`, 348/240'1
`348/211
`348/211
`. 348/240.1
`. 348/240.1
`348/239
`348/239
`. 356/311
`. 356/311
`
`

`
`U.S. Patent
`
`Aug. 14, 2012
`
`Sheet 1 of8
`
`US 8,243,171 B2
`
`80\
`
`
`
`FIG. 1
`
`82a
`
`82b
`
`82e
`
`82f
`
`82i
`
`82'
`
`96
`
`82c
`
`82d
`
`82
`
`82h
`
`82k
`
`82|
`
`98a
`
`98b
`
` 98C
`
`98d
`
`82m
`
`82n
`
`82
`
`82r
`
`82u
`
`82v
`
`82o
`
`82
`
`82s
`
`82t
`
`82w
`
`82x
`
`82
`
`822
`
`82cc
`
`82dd 82 82hh
`
`82aa
`
`82bb
`
`82ee
`
`82ff
`
`82ii
`
`82'
`
`FIG. 2
`
`

`
`US. Patent
`U.S. Patent
`
`Aug. 14, 2012
`Aug. 14, 2012
`
`Sheet 2 of8
`Sheet 2 of8
`
`US 8,243,171 B2
`US 8,243,171 B2
`
`102
`104
`106
`108
`110
`112
`102
`104
`106
`108
`110
`112
`MAW
`
`CROPP|N(3
`CROPPING
`BIN A
`| BIN B
`
`OPTICAL
`OPTICAL
`BIN B
`
`CROPPING
`CROPPING
`BIN B | BIN c
`
`DECIMATE
`DECIMATE
`
`BIN c
`
`INTERPOLATE
`
`INTERPOLATE
`
`b
`
`ZOOM OUT
`ZOOM OUT
`
`ZOOM IN
`ZOOM IN
`
`FIG. 3
`
`164
`164 \
`
`DECIMATION
`DECIMATION
`FILTER
`
`FILTER 193
`
`INITIAL
`INITIAL
`ASPECT
`ASPECT
`RATIO
`RATIo
`(E.G.,1:1)
`(E.G.,1:1)
`
`195
`198 \
`196 \
`VERTICAL
`HORIZONTAL
`D ’ HORIZONTAL _> VERTICAL
`FILTER
`FILTER
`FILTER
`FILTER
`
`FINAL
`FINAL
`ASPECT
`G ’ ASPECT
`RATIO
`RATIo
`(E.G., 4:3)
`(E.G., 4:3)
`
`FIG. 6
`FIG. 6
`
`

`
`US. Patent
`U.S. Patent
`
`Aug. 14, 2012
`Aug. 14, 2012
`
`Sheet 3 of8
`Sheet 3 of8
`
`US 8,243,171 B2
`US 8,243,171 B2
`
`120 \
`
`154
`
`130
`130 \
`
`r 128
`
`/- 129
`
`LENS
`ASSEMBLY \ 152
`
`MOTOR
`ASSEMBLY
`
`USER INPUT
`
`129 MOTOR
`ASSEMBLY \
`144
`\
`\
`147
`144
`CMD
`L
`
` 142/
`/146
`122\
`124\
`/142
`I \
`+
`MAIN CIRCUIT
`/170
`LENS MOTOR
`CONTROLLER
`
`SENSOR
`ARRAY
`
`FRMCNT
`
`A
`
` SENSOR
`/
`CONTROLLER
`-\<SCNT
`160
`150 /
`148 DFCNT+
`|NTCNT+
`DECIMATION
`FILTER
`Q T
`
`INTERPOLATOR
`+ I
`H
`
`\166
`
`
`
`
`PROCESS
`2
`162 /
`16
`
`\ D
`132 /
`134
`
`\ 164
`
` PROCESS
` /140
`
`DETECTOR
`DETECTOR
`
`168 \
`
`G
`—> FORMAT
`
`OUT
`\ >
`136
`
`‘138
`F
`126 \ / 140
`
`MEMORY
`MEMORY
`
`
`
`FIG. 4
`FIG. 4
`
`

`
`US. Patent
`U.S. Patent
`
`Aug. 14, 2012
`Aug. 14, 2012
`
`Sheet 4 as
`Sheet 4 of8
`
`US 8,243,171 B2
`US 8,243,171 B2
`
`174 \
`
`176
`176 I»
`
`USER COMMAND
`USER COMMAND
`
`I
`ADJUST CROPPING,
`178
`ADJUST CROPPING,
`178%
`BINNING AND/OR OPTICS
`BINNING AND/OR OPTICS
`I
`CONVERT OPTICAL
`CONVERT OPTICAL
`TO ELECTRICAL
`TO ELECTRICAL
`I
`
`180
`180'»
`
`182
`ANALOG PROCESSING
`182 m ANALOG PROCESSING
`
`I
`DIGITIZE ELECTRICAL
`DIGITIZE ELECTRICAL
`SIGNAL
`SIGNAL
`I
`PICTURE QUALITY
`PICTURE QUALITY
`DIGITAL PROCESSING
`DIGITAL PROCESSING
`
`184
`184/»
`
`186
`186~
`
`188
`188
`INTERPOLATION
`FLAG = ON?
`FLAG = ON?
`
` INTERPOLATION
`
`YES
`
`194
`194\
`INTERPOLATION
`INTERPOLATION
`
`NO
`
`190
`DECIMATION
`190 w DECIMATION
`
`192
`FORMATTING
`192 w FORMATTING
`
`@
`
`END
`
`FIG. 5
`FIG. 5
`
`

`
`US. Patent
`U.S. Patent
`
`Aug. 14, 2012
`Aug. 14, 2012
`
`Sheet 5 018
`Sheet 5 of8
`
`US 8,243,171 B2
`US 8,243,171 B2
`
`N.®_u_
`
`mm;
`
`
`
`zo:<._on_mm:z_z_-_>_ooN
`
`
`
`ZOnO<._n_n_z<_>__>_oo
`
`Z_-_2OON
`
`oz<s_s_oo
`
`
`
`~..m_>:.o<«m_>:.o<
`
`
`
`
`
`
`
`z_mmoo_>_OON._<o_Eoz_m_moo
`
`N50
`
`:_<O_._.n_O
`
`X<_>_vS_OON
`
`n..._.__>__:_
`
`m_m<m_moz_
`
`
`n..._.__>__._.Es__._
`
`
`z__2AoP<mommm22¢pm,Ao:<m
`
`
`
`museummoaommmoaommmuseum
`
`
`
`
`
`oz<s_s_oo
`
`Z_-_2OON
`f SN
`mom
`
`._.Xm_Z
`
`Amum._._Z_
`
`|_<Z_n_«ex
`
`«mum
`
`wow
`
`
`
`
`
`
`
`O_._.<w_Z_m_m_N_w>>On_Z_>>O_._.<N_Z_m_m_N_m>>On_Z_>>
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`
`US. Patent
`U.S. Patent
`
`Aug. 14, 2012
`Aug. 14, 2012
`
`Sheet 6 018
`Sheet 6 of8
`
`US 8,243,171 B2
`US 8,243,171 B2
`
`234
`4 3 2
`
`236
`235
`
`I
`IIIIIIII
`I
`JIIIIIII
`I
`II’ VIIIIIII
`II;
`
`{-244
`
`r254
`
`242
`242
`
`236
`236
`
`8 4 2
`248
`
`246
`246
`
`252
`252
`
`246
`246
`
`252
`2 5 2
`
`246
`246
`
`

`
`US. Patent
`U.S. Patent
`
`Aug. 14, 2012
`Aug. 14, 2012
`
`Sheet 7 018
`Sheet 7 of8
`
`US 8,243,171 B2
`US 8,243,171 B2
`
`260 \
`260 \
`
`262
`262 m
`
`USER LOW LIGHT
`USER LOW LIGHT
`COMMAND
`COMMAND
`
`I
`
`ADJUST WINDOW SIZE TO
`ADJUST WINDOW SIZE TO
`264 A, MAXIMIZE AREA UTILIZED
`MAXIMIZE AREA UTILIZED
`264
`ON SENSOR
`ON SENSOR
`
`I
`
`ADJUST OPTICAL ZOOM
`ADJUST OPTICAL ZOOM
`FACTOR TO PRODUCE
`FACTOR TO PRODUCE
`COMMANDED ZOOM
`COMMANDED ZOOM
`
`266 N
`266
`
`
`
`I
`
`ADJUST BIN RATIO TO
`ADJUST BIN RATIO TO
`268 N MINIMUM VALUE THAT
`MINIMUM VALUE THAT
`268
`ALLOWS SUFFICIENT TIME
`ALLOWS SUFFICIENT TIME
`TO READ WINDOW
`TO READ WINDOW
`
`I
`
`
`270
`270 N
`
`ADJUST DECIMATION
`ADJUST DECIMATION
`FACTOR
`FACTOR
`
`END
`
`FIG. 9
`FIG. 9
`
`

`
`US. Patent
`U.S. Patent
`
`Aug. 14, 2012
`Aug. 14, 2012
`
`Sheet 8 0f 8
`Sheet 8 of 8
`
`US 8,243,171 B2
`US 8,243,171 B2
`
`272 \
`272 \
`
`USER LOW DISTORTION
`USER LOW DISTORTION
`COMMAND
`COMMAND
`
`
`
`COMMAND BINNING OFF
`COMMAND BINNING OFF
`
`274 N
`
`274
`
`
`276
`
`I
`I
`
`
`
`ADJUST WINDOW SIZE TO
`ADJUST WINDOW SIZE TO
`MAXIMUM VALUE THAT
`MAXIMUM VALUE THAT
`278 N
`278
`ALLOWS SUFFICIENT TIME
`ALLOWS SUFFICIENT TIME
`TO READ WINDOW
`TO READ WINDOW
`
`280 N
`280
`
`282 w
`282
`
`I
`
`ADJUST OPTICAL ZOOM
`ADJUST OPTICAL ZOOM
`FACTOR TO PRODUCE
`FACTOR TO PRODUCE
`COMMANDED ZOOM
`COMMANDED ZOOM
`
`ADJUST DECIMATION
`ADJUST DECIMATION
`FACTOR
`FACTOR
`
`I
`
`
`FIG. 10
`FIG. 10
`
`

`
`US 8,243,171 B2
`US 8,243,171 B2
`
`1
`1
`HIGH RESOLUTION ZOOM: A NOVEL
`HIGH RESOLUTION ZOOM: A NOVEL
`DIGITAL ZOOM FOR DIGITAL VIDEO
`DIGITAL ZOOM FOR DIGITAL VIDEO
`CAMERA
`CAMERA
`
`This is a continuation of U.S. Ser. No. 12/716,525, filed
`This is a continuation of US. Ser. No. 12/716,525, ?led
`Mar. 3, 2010 now U.S. Pat. No. 7,880,776, which is a con-
`Mar. 3, 2010 now US. Pat. No. 7,880,776, Which is a con
`tinuation of U.S. Ser. No. 11/010,032, filed Dec. 10, 2004,
`tinuation of US. Ser. No. 11/010,032, ?led Dec. 10, 2004,
`now U.S. Pat. No. 7,688,364, which are each incorporated by
`now US. Pat. No. 7,688,364, Which are each incorporated by
`reference.
`reference.
`
`10
`
`FIELD OF THE INVENTION
`FIELD OF THE INVENTION
`
`The present invention relates to video image processing
`The present invention relates to video image processing
`generally and, more particularly, to a digital zoom for digital
`generally and, more particularly, to a digital Zoom for digital
`video cameras.
`video cameras.
`
`15
`
`BACKGROUND OF THE INVENTION
`BACKGROUND OF THE INVENTION
`
`Functionality of conventional Digital Still Cameras (DSC)
`Functionality of conventional Digital Still Cameras (DSC)
`and conventional camcorders are converging. The DSCs
`and conventional camcorders are converging. The DSCs
`implement sensors (i.e., CCD or CMOS) with at least 4 to 5
`implement sensors (i.e., CCD or CMOS) With at least 4 to 5
`million pixels. A video signal
`in a typical camcorder is
`million pixels. A video signal in a typical camcorder is
`acquired at 30 to 60 frames per seconds with a resolution
`acquired at 30 to 60 frames per seconds With a resolution
`varying from 720><480 (i.e., standard definition) to 1920><
`varying from 720x480 (i.e., standard de?nition) to 1920><
`1080 (i.e., high definition) viewable pixels. The availability of
`1080 (i.e., high de?nition) vieWable pixels. The availability of
`sensors that can combine both a high pixel number to accom-
`sensors that can combine both a high pixel number to accom
`modate DSCs and a transfer rate to accommodate video
`modate DSCs and a transfer rate to accommodate video
`allows an introduction of a new digital zoom function that is
`alloWs an introduction of a neW digital Zoom function that is
`quite different from the current digital zoom function used in
`quite different from the current digital Zoom function used in
`conventional cameras and camcorders.
`conventional cameras and camcorders.
`A conventional digital zoom operation, also called “inter-
`A conventional digital Zoom operation, also called “inter
`polated” zoom, is achieved by calculating an up-conversion
`polated” Zoom, is achieved by calculating an up-conversion
`of a window in existing image data to generate an enlarged
`of a WindoW in existing image data to generate an enlarged
`version. Interpolated zoom is achieved by cropping a window
`version. Interpolated Zoom is achieved by cropping a WindoW
`in a standard resolution picture and enlarging the window by
`in a standard resolution picture and enlarging the WindoW by
`interpolation. The resulting image has a progressively
`interpolation. The resulting image has a progressively
`decreasing resolution as the cropping factor increases. The
`decreasing resolution as the cropping factor increases. The
`decreasing spatial resolution has created a feeling among
`decreasing spatial resolution has created a feeling among
`users that digital zoom is a technique inferior to a true optical
`users that digital Zoom is a technique inferior to a true optical
`zoom.
`Zoom.
`
`SUMMARY OF THE INVENTION
`SUMMARY OF THE INVENTION
`
`The present invention concerns a camera system and a
`The present invention concerns a camera system and a
`method for zooming the camera. The method generally com-
`method for Zooming the camera. The method generally com
`prises the steps of (A) generating an electronic image by
`prises the steps of (A) generating an electronic image by
`sensing an optical image received by the camera, the sensing
`sensing an optical image received by the camera, the sensing
`including electronic cropping to a window size to establish an
`including electronic cropping to a WindoW siZe to establish an
`initial resolution for the electronic image, (B) generating a
`initial resolution for the electronic image, (B) generating a
`final image by decimating the electronic image by a decima-
`?nal image by decimating the electronic image by a decima
`tion factor to a final resolution smaller than the initial resolu-
`tion factor to a ?nal resolution smaller than the initial resolu
`tion and (C) changing a zoom factor for the final image by
`tion and (C) changing a Zoom factor for the ?nal image by
`adjusting both of the decimation factor and the window size.
`adjusting both of the decimation factor and the WindoW siZe.
`The objects, features and advantages of the present inven-
`The objects, features and advantages of the present inven
`tion include providing a zooming method and a camera sys-
`tion include providing a Zooming method and a camera sys
`tem that may provide (i) a high resolution digital zoom capa-
`tem that may provide (i) a high resolution digital Zoom capa
`bility, (ii) a substantially constant output image resolution at
`bility, (ii) a substantially constant output image resolution at
`different zoom levels, (iii) a low light mode, (iv) a low dis-
`different Zoom levels, (iii) a loW light mode, (iv) a loW dis
`tortion mode, (v) a digital zoom capable of operating with
`tortion mode, (v) a digital Zoom capable of operating With
`camcorder speed data and/or (vi) a low-cost alternative to
`camcorder speed data and/or (vi) a loW-cost alternative to
`high zoom optics.
`high Zoom optics.
`BRIEF DESCRIPTION OF THE DRAWINGS
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`These and other objects, features and advantages of the
`These and other objects, features and advantages of the
`present invention will be apparent from the following detailed
`present invention Will be apparent from the folloWing detailed
`description and the appended claims and drawings in which:
`description and the appended claims and draWings in Which:
`
`20
`20
`
`25
`25
`
`30
`30
`
`35
`35
`
`40
`40
`
`45
`45
`
`50
`50
`
`55
`55
`
`60
`60
`
`65
`65
`
`2
`2
`FIG. 1 is a block diagram of a first example binning pro-
`FIG. 1 is a block diagram of a ?rst example binning pro
`cess;
`cess;
`FIG. 2 is a block diagram of a second example binning
`FIG. 2 is a block diagram of a second example binning
`process;
`process;
`FIG. 3 is a block diagram illustration a zooming operation;
`FIG. 3 is a block diagram illustration a Zooming operation;
`FIG. 4 is a block diagram of an example implementation of
`FIG. 4 is a block diagram of an example implementation of
`a system in accordance with a preferred embodiment of the
`a system in accordance With a preferred embodiment of the
`present invention;
`present invention;
`FIG. 5 is a flow diagram of an example method ofprocess-
`FIG. 5 is a How diagram of an example method of process
`ing an optical image;
`ing an optical image;
`FIG. 6 is a block diagram of an example implementation of
`FIG. 6 is a block diagram of an example implementation of
`a decimation filter circuit;
`a decimation ?lter circuit;
`FIG. 7 is a flow diagram of an example zoom-in process;
`FIG. 7 is a How diagram of an example Zoom-in process;
`FIGS. 8A-8E are block diagrams illustrating various win-
`FIGS. 8A-8E are block diagrams illustrating various Win
`dow size and bin ratio settings;
`doW siZe and bin ratio settings;
`FIG. 9 is a flow diagram of an example method for config-
`FIG. 9 is a How diagram of an example method for con?g
`uring a low light mode; and
`uring a loW light mode; and
`FIG. 10 is a flow diagram of an example method for con-
`FIG. 10 is a How diagram of an example method for con
`figuring a low distortion mode.
`?guring a loW distortion mode.
`DETAILED DESCRIPTION OF THE PREFERRED
`DETAILED DESCRIPTION OF THE PREFERRED
`EMBODIMENTS
`EMBODIMENTS
`
`The present invention described herein may be referred to
`The present invention described herein may be referred to
`as a High Resolution (HR) Zoom. HR Zoom generally allows
`as a High Resolution (HR) Zoom. HR Zoom generally alloWs
`a user to control a zoom factor (or level) by electronically
`a user to control a Zoom factor (or level) by electronically
`cropping (or windowing) a sensor area detecting an optical
`cropping (or WindoWing) a sensor area detecting an optical
`image, digitizing a resulting electrical image and down-sam-
`image, digitiZing a resulting electrical image and doWn-sam
`pling the electronic image to a final resolution. An amount of
`pling the electronic image to a ?nal resolution. An amount of
`cropping and a level of down-sampling may be controlled so
`cropping and a level of doWn-sampling may be controlled so
`that the final resolution is substantially constant over a range
`that the ?nal resolution is substantially constant over a range
`of different zoom factors. In HR Zoom, a subjective digital
`of different Zoom factors. In HR Zoom, a subjective digital
`zoom effect may appear much like an optical zoom in that
`Zoom effect may appear much like an optical Zoom in that
`pictures generally remain sharp throughout zoom-in and
`pictures generally remain sharp throughout Zoom-in and
`zoom-out operations.
`Zoom-out operations.
`In video applications where a standard resolution video
`In video applications Where a standard resolution video
`(e.g., International Telecommunications Union-Radiocom-
`(e.g., International Telecommunications Union-Radiocom
`munications Sector, Recommendation BT.656-4 (February
`munications Sector, Recommendation BT.656-4 (February
`1998), Geneva, Switzerland) may be implemented, the HR
`1998), Geneva, SWitZerland) may be implemented, the HR
`Zoom process generally allows a high resolution electronic
`Zoom process generally alloWs a high resolution electronic
`zoom-in ratio up to a square root of a raw data capture rate
`Zoom-in ratio up to a square root of a raW data capture rate
`divided by a video resolution rate. For example, a sensor
`divided by a video resolution rate. For example, a sensor
`element capture rate of 60 megahertz (MHz) may yield about
`element capture rate of 60 megahertz (MHZ) may yield about
`2 million pixels per frame at 30 frames per second. Decimat-
`2 million pixels per frame at 30 frames per second. Decimat
`ing the image data down to a 720x480 pixel video resolution
`ing the image data doWn to a 720x480 pixel video resolution
`at 30 frames per second (e.g., 10.37 MHz) generally allows a
`at 30 frames per second (e.g., 10.37 MHZ) generally alloWs a
`2.4 maximum zoom factor. In practice, the raw data input data
`2.4 maximum Zoom factor. In practice, the raW data input data
`rate may be limited by a speed of an optical sensor array. The
`rate may be limited by a speed of an optical sensor array. The
`high image resolution available in modern optical sensor
`high image resolution available in modern optical sensor
`arrays generally permits acquisition of raw image data at a
`arrays generally permits acquisition of raW image data at a
`resolution significantly higher than standard video resolution,
`resolution signi?cantly higher than standard video resolution,
`thus allowing the implementation of the HR Zoom process.
`thus alloWing the implementation of the HR Zoom process.
`Sensor arrays for Digital Still Cameras (DCS) and cam-
`Sensor arrays for Digital Still Cameras (DCS) and cam
`corders generally have a large number of individual optical-
`corders generally have a large number of individual optical
`to-electrical sensor elements. Therefore, reading all the sen-
`to-electrical sensor elements. Therefore, reading all the sen
`sor elements (e.g., 5 million elements or pixels) in the sensor
`sor elements (e.g., 5 million elements or pixels) in the sensor
`array may not be possible in a video frame period (e.g., 1/soth
`array may not be possible in a video frame period (e.g., 1/3oth
`to 1/soth of a second). One or more ofthe following techniques
`to 1/6oth of a second). One or more of the folloWing techniques
`may be used to reduce an output rate from a sensor array. A
`may be used to reduce an output rate from a sensor array. A
`first technique may be to skip some sensor element rows
`?rst technique may be to skip some sensor element roWs
`and/or columns to cover the whole sensor array area, but not
`and/or columns to cover the Whole sensor array area, but not
`present all of the available data. A second technique, called
`present all of the available data. A second technique, called
`binning, may be implemented to reduce (i) a total amount of
`binning, may be implemented to reduce (i) a total amount of
`data presented and (ii) an impact of aliasing due to sub-
`data presented and (ii) an impact of aliasing due to sub
`sampling. In binning, multiple sensor element sets may be
`sampling. In binning, multiple sensor element sets may be
`combined to create a binned set. The binning technique gen-
`combined to create a binned set. The binning technique gen
`erally has multiple advantages over sub-sampling (e.g., skip-
`erally has multiple advantages over sub-sampling (e.g., skip
`
`

`
`US 8,243,171 B2
`US 8,243,171 B2
`
`3
`3
`ping) since both aliasing and photon noise inherent to optical
`ping) since both aliasing and photon noise inherent to optical
`detection may be reduced by the combination ofthe collected
`detection may be reduced by the combination of the collected
`samples.
`samples.
`A sensor array (e.g., a CCD array or a CMOS array) may be
`A sensor array (e.g., a CCD array or a CMOS array) may be
`either a monochrome sensor array or a color sensor array. In
`either a monochrome sensor array or a color sensor array. In
`the case of a monochrome sensor array, three monochrome
`the case of a monochrome sensor array, three monochrome
`sensor arrays may be implemented to generate a color image.
`sensor arrays may be implemented to generate a color image.
`In the case ofthe color sensor array, a mosaic ofred, green and
`In the case of the color sensor array, a mosaic of red, green and
`blue color filters is generally applied on the sensor surface.
`blue color ?lters is generally applied on the sensor surface.
`The most common mosaic pattern is called a Bayer pattern
`The most common mosaic pattern is called a Bayer pattern
`consisting of two green cells and one each of a red cell and a
`consisting of tWo green cells and one each of a red cell and a
`blue cell. Applications of the HR Zoom process with Bayer
`blue cell. Applications of the HR Zoom process With Bayer
`patterned sensor arrays generally include a conversion step to
`patterned sensor arrays generally include a conversion step to
`red-green-blue (RGB) data before filtered decimation in RGB
`red-green-blue (RGB) data before ?ltered decimation in RGB
`space. The conversion step is commonly called de-mosaicing.
`space. The conversion step is commonly called de-mosaicing.
`Referring to FIG. 1, a block diagram of a first example
`Referring to FIG. 1, a block diagram of a ?rst example
`binning process 80 is shown. The first binning process 80
`binning process 80 is shoWn. The ?rst binning process 80
`generally illustrates a bin ratio of 2:1 in each of a horizontal
`generally illustrates a bin ratio of 2:1 in each of a horizontal
`and a vertical direction. In particular, individual image ele-
`and a vertical direction. In particular, individual image ele
`ments 82a-82p generated in a set 84 of adjoining sensor
`ments 82a-82p generated in a set 84 of adjoining sensor
`elements (e.g., a 4x4 set) may be combined to form a binned
`elements (e.g., a 4x4 set) may be combined to form a binned
`set 86 defining fewer image elements 88a-88d. The binned set
`set 86 de?ning feWer image elements 88a-88d. The binned set
`86 generally defines R, B and two G values.
`86 generally de?nes R, B and tWo G values.
`The original set 84 generally comprises four Bayer sets of
`The original set 84 generally comprises four Bayer sets of
`sensor elements. Each Bayer set generally comprises a sensor
`sensor elements. Each Bayer set generally comprises a sensor
`element defining a red value and a location (e.g., R), a sensor
`element de?ning a red value and a location (e.g., R), a sensor
`element defining a blue value and a location (e.g., B) and two
`element de?ning a blue value and a location (e.g., B) and tWo
`sensor elements defining two green values and two locations
`sensor elements de?ning tWo green values and tWo locations
`(e.g., Gr and Gb). The green sensor element Gr may be
`(e. g., Gr and Gb). The green sensor element Gr may be
`located on a same row as the red sensor element R. The green
`located on a same roW as the red sensor element R. The green
`sensor element Gb may be located on a same row as the blue
`sensor element Gb may be located on a same roW as the blue
`sensor element B. The binned set 86 may follow the Bayer
`sensor element B. The binned set 86 may folloW the Bayer
`pattern defining a virtual red sensor element (e.g., Rs), a
`pattern de?ning a virtual red sensor element (e.g., Rs), a
`virtual blue sensor element (e.g., Bs) and two virtual green
`virtual blue sensor element (e.g., Bs) and tWo virtual green
`sensor elements (e.g., Grs and Gbs). Other sensor element
`sensor elements (e.g., Grs and Gbs). Other sensor element
`layouts and color patterns may be implemented to meet the
`layouts and color patterns may be implemented to meet the
`criteria of a particular application.
`criteria of a particular application.
`An effect of the 2:1 bin ratio in each direction may be to
`An effect of the 2:1 bin ratio in each direction may be to
`reduce an image data rate from the sensor array by a factor of
`reduce an image data rate from the sensor array by a factor of
`four while still allowing all ofthe photons striking the original
`four While still alloWing all of the photons striking the original
`set 84 to contribute to the binned set 86. Therefore, the maxi-
`set 84 to contribute to the binned set 86. Therefore, the maxi
`mum zoom factor may increase since the raw data capture rate
`mum Zoom factor may increase since the raW data capture rate
`is generally reduced. For example, the 2:1 bin ratio in each
`is generally reduced. For example, the 2:1 bin ratio in each
`direction and a 5 million pixel sensor array generally yields a
`direction and a 5 million pixel sensor array generally yields a
`3.80 maximum zoom factor as long as the raw capture rate is
`3.80 maximum Zoom factor as long as the raW capture rate is
`greater than 40 MHz.
`greater than 40 MHZ.
`Referring to FIG. 2, a block diagram of a second example
`Referring to FIG. 2, a block diagram of a second example
`binning process 90 is shown. The second binning process 90
`binning process 90 is shoWn. The second binning process 90
`generally illustrates a bin ratio of 3:1 in each direction. In
`generally illustrates a bin ratio of 3:1 in each direction. In
`particular, individual image elements 82a-82jj' generated
`particular,
`individual
`image elements 82a-82]] generated
`within a set 94 may be combined to form a binned set 96. Each
`Within a set 94 may be combined to form a binned set 96. Each
`binned image element 98a-98d may be generated from the
`binned image element 98a-98d may be generated from the
`original set 94. An effect of the 3 :1 bin ratio in each direction
`original set 94. An effect of the 3 :1 bin ratio in each direction
`may be to reduce an image data rate from the sensor array by
`may be to reduce an image data rate from the sensor array by
`a factor of nine while still allowing all of the photons striking
`a factor of nine While still alloWing all of the photons striking
`the original set 94 to contribute to the binned set 96. Other bin
`the original set 94 to contribute to the binned set 96. Other bin
`ratios and/or combinations of different bin ratios in each
`ratios and/or combinations of different bin ratios in each
`direction may be implemented to meet the criteria of a par-
`direction may be implemented to meet the criteria of a par
`ticular application.
`ticular application.
`Referring to FIG. 3, a block diagram illustrating a zooming
`Referring to FIG. 3, a block diagram illustrating a Zooming
`operation 100 is shown. The operation 100 may comprise a
`operation 100 is shoWn. The operation 100 may comprise a
`section (or range) 102, a section (or range) 104, a section (or
`section (or range) 102, a section (or range) 104, a section (or
`range) 106, a section (or range) 108, a section (or range) 110
`range) 106, a section (or range) 108, a section (or range) 110
`and a section (or range) 112.Any given horizontal point in the
`and a section (or range) 112.Any given horiZontal point in the
`figure may be considered a unique zoom factor. Movement
`?gure may be considered a unique Zoom factor. Movement
`toward the left (e.g., zoom out) in the figure generally pro-
`toWard the left (e.g., Zoom out) in the ?gure generally pro
`duces a wider field of view (e.g., a smaller zoom factor).
`duces a Wider ?eld of vieW (e.g., a smaller Zoom factor).
`
`10
`
`15
`
`20
`20
`
`25
`25
`
`30
`30
`
`35
`35
`
`40
`40
`
`45
`45
`
`50
`50
`
`55
`55
`
`60
`60
`
`65
`65
`
`4
`4
`Movement toward the right (e.g., zoom in) in the figure gen-
`Movement toWard the right (e.g., Zoom in) in the ?gure gen
`erally produces a narrower field of view (e.g., a larger zoom
`erally produces a narroWer ?eld of vieW (e. g., a larger Zoom
`factor).
`factor).
`From a given zoom factor, a cropping operation and an
`From a given Zoom factor, a cropping operation and an
`optional optical zooming operation may be used to increase
`optional optical Zooming operation may be used to increase
`and decrease the zoom factor. A full range of zoom factors
`and decrease the Zoom factor. A full range of Zoom factors
`may be achieved using cropping over a low zoom range (e.g.,
`may be achieved using cropping over a loW Zoom range (e. g.,
`sections 102 and 104), optical zooming over a medium zoom
`sections 102 and 104), optical Zooming over a medium Zoom
`range (e.g., section 106) and cropping over a high zoom range
`range (e. g., section 106) and cropping over a high Zoom range
`(e.g., sections 108 and 110). Zooming beyond the high zoom
`(e.g., sections 108 and 110). Zooming beyond the high Zoom
`range may be achieved using an interpolation operation (e.g.,
`range may be achieved using an interpolation operation (e. g.,
`section 112).
`section 112).
`Zooming may be achieved by a user pressing controls to
`Zooming may be achieved by a user pressing controls to
`zoom in or zoom out. The pressed control generally causes the
`Zoom in or Zoom out. The pressed control generally causes the
`camera to utilize a cropping setting and an optical zoom
`camera to utiliZe a cropping setting and an optical Zoom
`setting corresponding to a position in FIG. 3. If the current
`setting corresponding to a position in FIG. 3. If the current
`amount of zoom corresponds to the zoom level in one of the
`amount of Zoom corresponds to the Zoom level in one of the
`sections 102, 104,108,110 or 112, more or less zoom may be
`sections 102, 104,108,110 or 112, more or less Zoom may be
`achieved by more or less cropping. For example, if the user
`achieved by more or less cropping. For example, if the user
`commands “zoom in”, more cropping may be used. Ifthe user
`commands “Zoom in”, more cropping may be used. If the user
`commands “zoom out”, less cropping is used. If the current
`commands “Zoom out”, less cropping is used. If t