Exhibit 1006
`Exhibit 1006
`
`

`
`(12)
`(12)
`
`United States Patent
`United States Patent
`Novak
`Novak
`
`(10) Patent N0.:
`(10) Patent No.:
`(45) Date of Patent:
`(45) Date of Patent:
`
`US 7,071,968 B2
`US 7,071,968 B2
`Jul. 4, 2006
`Jul. 4, 2006
`
`US00707l968B2
`US007071968B2
`
`(54) SYSTEM AND METHOD FOR A SOFTWARE
`(54) SYSTEM AND METHOD FOR A SOFTWARE
`STEERABLE WEB CAMERA WITH
`STEERABLE WEB CAMERA WITH
`MULTIPLE IMAGE SUBSET CAPTURE
`MULTIPLE IMAGE SUBSET CAPTURE
`
`-
`_
`75
`Inventor: R°b°rt E‘ Nwak’ Kirkland’ WA (US)
`(75)
`Inventor‘ Robert E‘ Novak’ Klrkland’ WA (Us)
`(
`)
`(73) Assigneez Dig” I“°" Kirk1a“d= WA (US)
`-
`_
`.
`.
`(73) Asslgnee' Dlgeo’ Inc" Klrkland’ WA (Us)
`
`( * ) Notice:
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`patent is extended or adjusted under 35
`U_S_C_ 154(1)) by 718 days_
`USC. 15403) by 718 days.
`
`(21) Appl. No.: 09/923,820
`(21) Appl. No.: 09/923,820
`
`(22)
`Ffled:
`(22) Flled:
`65
`65
`(
`)
`(
`)
`
`Aug' 6’ 2001
`Aug' 6’ 2001
`.
`.
`.
`D t
`P
`P bl
`t
`.
`.
`.
`D t
`t
`P
`P bl
`nor
`11
`lea Ion
`a a
`nor u lea Ion a a
`US 2002/0141658 A1
`Oct. 3, 2002
`US 2002/0141658 A1
`Oct. 3, 2002
`
`Related U.S. Application Data
`Related US. Application Data
`(63) Continuation-in-part of application No. 09/823,804,
`(63) Continuation-in-part of application No. 09/823,804,
`filed on Mar. 30, 2001, now abandoned.
`?led on Mar. 30, 2001, noW abandoned.
`
`(51)
`Int- Cl-
`(51) Int- Cl-
`(2006.01)
`H04N 5/225
`(2006.01)
`H04N 5/225
`(2006.01)
`H04N 5/232
`(2006.01)
`H04N 5/232
`(52) U.S. Cl.
`............................... .. 348/207.1;348/211.9
`(52) US. Cl. ............................... .. 348/207.1;348/2ll.9
`(58) Field of Classification Search ........... .. 348/207.1,
`(58) Field of Classi?cation Search ........... .. 348/207.l,
`348/211.9
`348/2ll.9
`See application file for complete search history.
`See application ?le for complete search history.
`
`(56)
`(56)
`
`Referenees Cited
`References Cited
`U.S. PATENT DOCUMENTS
`U.S. PATENT DOCUMENTS
`4,310,222 A
`1/1982 Ikemori
`4,310,222 A
`V1982 lkemofi
`4,772,107 A
`9/1988 Friedman
`4,772,107 A
`9/1988 Fnedman
`4,831,438 A
`5/1989 Bellman, Jr. et al.
`..... .. 358/108
`i
`genman’t ‘III’ et a1‘ """ "
`5,128,776 A
`7/1992 Scorse et al.
`............. .. 358/426
`5,353,392 A
`10/1994 Luquet et al.
`..
`395/135
`5,353,392 A 10/1994 Luquet et a1. ..
`395/135
`
`5,359,363 A
`10/1994 Kuban et al.
`348/36
`5,359,363 A 10/1994 Kuban et a1.
`348/36
`5,444,478 A
`8/1995 Lelong et al.
`.............. .. 348/39
`5,444,478 A
`8/1995 Lelong et a1. .............. .. 348/39
`
`,
`
`,
`
`corse e a .
`
`............. ..
`
`2/1997 Kim ......................... .. 348/159
`5,606,364 A
`2/1997 Kim ......................... .. 348/159
`5,606,364 A
`8/1997 Henley ........ ..
`348/38
`5,657,073 A
`8/1997 Henley ........ ..
`.
`. 348/38
`5,657,073 A
`
`3/1999 Newlin et al.
`.
`348/724
`5,877,821 A
`3/1999 NeWlin et a1.
`348/724
`5,877,821 A
`1/2000 Hsieh et al.
`.............. .. 345/435
`6,011,558 A
`l/2000 Hsieh et a1. .............. .. 345/435
`6,011,558 A
`2/2000 Golin et a1.
`6,031,540 A
`A21/2333
`................. .. 386/117
`8:332:63 :
`4/2000 Abe ......................... .. 386/117
`6,052,509 A
`11/233? §1(i1cT1ri0e]t:t1.a1................. 348/208
`81
`6,144,773 A 11/2000 Kolarov et a1.
`6,172,707 B1
`1/2001 Ouchi et a1. .............. .. 348/208
`
`(Continued)
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`FOREIGN PATENT DOCUMENTS
`WOWO PCT/US99/02122 Al
`2/2000
`WOWO PCT/US99/02l22 Al
`2/2000
`OTHER PUBLICATIONS
`OTHER PUBLICATIONS
`Ken Turkowski, Making Environment Maps From Fisheye
`Ken TurkoWski, Making Environment Maps From Fisheye
`Photographs
`[online]
`[retrieved on Oct.
`10
`2001].
`’
`’
`’
`Photographs [online]
`[retrieved on Oct. 10 2001].
`’
`’
`’
`Retrieved from the Internet: <URL:http://www.iqtvra.org./
`Retrieved from the Internet: <URL:http://WWW.iqtvra.org./
`De?sh/Fisheye.html>.
`Defish/Fjgheye,htm1>,
`
`(Continued)
`(Continued)
`Primary ExamineriNgoc-Yen Vu
`Primary Examiner—Ngoc—Yen Vu
`Assistant Examiner—James M. Hannett
`Assistant Examinerilames M. Hannett
`Home ,
`em, or
`zrm— o
`.
`istenseng
`toe
`74A
`yAg
`F’
`KryDChr'
`S
`l
`(74) Attorney, Agent, or F irmiKory D. Christensen; Stoel
`Rives LLP
`Rives LLP
`
`(57)
`(57)
`
`ABSTRACT
`ABSTRACT
`
`An apparatus for controlling the capture of an image of an
`An apparatus for controlling the capture of an image of an
`object, includes: a lens capable to capture a scene within a
`object, includes: a lens capable to capture a scene Within a
`wide field of vision of the lens; an image collection array
`Wide ?eld of vision of the lens; an image collection array
`communicatively coupled to the lens and capable to store
`communicatively coupled to the lens and capable to store
`data of the scene within the wide field of vision; a memory
`data of the scene Within the Wide ?eld of vision; a memory
`communicatively coupled to the image collection array and
`communicatively coupled to the image collection array and
`capable to store digitized data of the scene within the wide
`capable to store digitized data of the scene Within the Wide
`field of vision; and a processing stage communicatively
`?eld of vision; and a processing stage communicatively
`coupled to the memory and capable to select a plurality of
`coupled to the memory and capable to select a plurality of
`subsets of the digitized data of the scene in order to generate
`subsets of the digitized data of the scene in order to generate
`an image of the captured scene.
`an Image Ofthe Captured Scene‘
`
`-
`
`22 Claims, 18 Drawing Sheets
`22 Claims, 18 Drawing Sheets
`
`
`"Fish-Eye" Capture
`Living Room Scene Viewed by Camera
`1000
`
`Living Room Scene Viewed by Camera (1000
`1020
`uFishjsgegzpmre
`
`[1042
`
`1050
`
`i
`
`.
`
`1030/
`Skew Corrected
`Skew Corrected
`Scene Ready
`for Transmission
`Scene Ready
`for Transmission
`
`
`
`Selected
`Subset #2
`01 Scene
`1055
`
`1
`005
`
`1015
`
`1010
`
`1045A
`
`Motion
`Detector
`1005
`1040
`
`83323:;
`of Scene
`1025
`
`1o6o\
`Skew Corrected
`Scene Ready
`352223;?“
`for Transmission
`for Transmissizn
`
`

`
`US 7,071,968 B2
`US 7,071,968 B2
`Page 2
`Page 2
`
`Panoscan Home. Panoscan offers a choice of new cameras!
`Panoscan Home. Panoscan offers a choice of neW cameras!
`Webpage [online]. Panoscan, Inc., 2001 [retrieved on Oct.
`Webpage [online]. Panoscan, Inc., 2001 [retrieved on Oct.
`10, 2001]. Retrieved from the Internet:<URL:http://www.
`10, 2001]. Retrieved from the Internet:<URL:http://WWW.
`panoscan.com>.
`panoscan.com>.
`Internet monitoring,
`live streaming video, and webcam
`Internet monitoring, live streaming video, and Webcam
`softWare from Surveyor Corporation. Webpage [online].
`software from Surveyor Corporation. Webpage [online].
`Surveyor Corporation, 2001 [retrieved on Oct. 11, 2001].
`Surveyor Corporation, 2001 [retrieved on Oct. 11, 2001].
`Retrieved
`from the
`Internet:
`<URL:http://www.
`Retrieved
`from
`the
`Internet:
`<URL:http://WWW.
`surveyorcorp.com/products/productsiinfo.html>.
`surveyorcorp.con1/products/productsiinfo.html>.
`Robotic Camera Mounts Info, Transit RCM. Webpage
`Robotic Camera Mounts Info, Transit RCM. Webpage
`[online]. Surveyor Corporation, 2001 [retrieved on Oct. 10,
`[online]. Surveyor Corporation, 2001 [retrieved on Oct. 10,
`2001]. Retrieved from the
`Internet:<URL:http://www.
`2001]. Retrieved from the Internet:<URL:http://WWW.
`surveyorcorp.com/products/transit/rcmiinfo.html>.
`surveyorcorp.con1/products/transit/rcm_info.html>.
`Product Selector, Product Overview/Selector. Webpage
`Product Selector, Product Overview/Selector. Webpage
`[online]. Surveyor Corporation, 2001. [retrieved on Oct. 10,
`[online]. Surveyor Corporation, 2001. [retrieved on Oct. 10,
`2001]. Retrieved from the
`Internet:<URL:http://www.
`2001]. Retrieved from the Internet:<URL:http://WWW.
`daZZle.com/products/select£ut.html>.
`dazzle.con1/products/select$ut.html>.
`Company News. IMove granted new patent for panoramic
`Company NeWs. IMove granted neW patent for panoramic
`Imaging: The patent extends iMove’ s intellectual property in
`Imaging: The patent extends iMove’ s intellectual property in
`panoramic Image seaming. Webpage [online]. IMove Incor
`panoramic Image seaming. Webpage [online]. IMove Incor-
`porated, Mar. 25, 2000 [retrieved on Mar. 26, 2000],
`porated, Mar. 25, 2000 [retrieved on Mar. 26, 2000],
`Retrieved
`from
`the
`Internet:<URL:http://www.
`Retrieved
`from
`the
`Internet:<URL:http://WWW.
`smoothmove.com/01companyiinfo/.html>.
`smoothmove.con1/01companyiinfo/.html>.
`* cited by examiner
`* cited by examiner
`
`U.S. PATENT DOCUMENTS
`US. PATENT DOCUMENTS
`6,223,213 B1
`4/2001 Cleron et al.
`............. .. 709/206
`6,223,213 B1
`4/2001 Cleron et a1. ............. .. 709/206
`6,297,846 B1* 10/2001 Edanami . . . . . . . . .
`. . . .. 348/239
`10/2001 Matsumoto ............... .. 396/106
`6,308,015 B1
`10/2001 Matsumoto ............... .. 396/106
`6,308,015 B1
`4/2002 Monroe .................... .. 348/148
`6,366,311 B1
`4/2002 Monroe .................... .. 348/148
`6,366,311 B1
`6,470,498 B1* 10/2002 Reber et al.
`.
`725/139
`6,470,498 B1* 10/2002 Reber et a1. .
`725/139
`6,507,366 B1*
`1/2003 Lee . . . . . . . . . . . . .
`. . . .. 348/352
`.... .. 725/60
`4/2003 Ogasawara ..
`6,543,052 B1
`.... .. 725/60
`4/2003 Ogasawara ..
`6,543,052 B1
`5/2003 Kuno .................... .. 348/211.3
`6,567,121 B1*
`6,567,121 B1* 5/2003 Kuno .................... .. 348/211.3
`6,606,422 B1 *
`8/2003 Dulin et al.
`.............. .. 382/282
`6,606,422 B1 *
`8/2003 Dulin et a1. .............. .. 382/282
`6,727,940 B1*
`4/2004 Oka et al.
`......... ..
`.. 348/211.3
`6,727,940 B1* 4/2004 Oka et a1. ......... ..
`.. 348/211.3
`2001/0019355 A1
`9/2001 Koyanagi et al.
`........... .. 348/36
`2001/0019355 A1
`9/2001 Koyanagi et a1. ........... .. 348/36
`2003/0025803 A1*
`2/2003 Nakamura et al.
`..... .. 348/218.1
`2003/0025803 A1* 2/2003 Nakamura et a1. ..... .. 348/218.1
`OTHER PUBLICATIONS
`OTHER PUBLICATIONS
`
`
`
`6,297,846 B1* 10/2001 Edanami . . . . . . . . .
`
`. . . .. 348/239
`
`6,507,366 B1* 1/2003 Lee . . . . . . . . . . . . .
`
`. . . .. 348/352
`
`Welcome To BeHere.com. Webpage [online]. BeHere Tech-
`Welcome To BeHere.com. Webpage [online]. BeHere Tech
`nologies, 2001 [retrieved on Oct. 10, 2001], Retrieved from
`nologies, 2001 [retrieved on Oct. 10, 2001], Retrieved from
`the Internet:<URL:http://www.BeHere.con1/1.html>.
`the Internet:<URL:http://WWW.BeHere.com/1.html>.
`Leegomes, BeHere takes a 360-degree Turn In Imaging.
`Leegomes, BeHere takes a 360-degree Tum In Imaging.
`Press Releases
`[online],
`[retrieved on Oct. 11, 2001].
`Press Releases [online], [retrieved on Oct. 11, 2001].
`Retrieved from the Internet: <URL:http://www.BeHere.con1/
`Retrieved from the Internet: <URL:http://WWW.BeHere.com/
`neWsipressi031501.html>.
`news,press,031501.html>.
`Products. Webpage [online]. SPheron VR, 2000 [retrieved
`Products. Webpage [online]. SPheron VR, 2000 [retrieved
`on Oct. 10, 2001]. Retrieved from the Internet:<URL:http://
`on Oct. 10, 2001]. Retrieved from the Intemet:<URL:http://
`WWW.Spheron.com/products/productsiportal.html>.
`www.Spheron.com/products/products_portal.html>.
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4,2006
`Jul. 4, 2006
`
`Sheet 1 of 18
`Sheet 1 0f 18
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`Object
`L130
`
`130
`
`( 160
`
`Cable Network
`
`100
`
`110
`
`©_\/ |_(1_75
`
`165
`§
`
`Printer
`
`140 / Set Top BOX
`|
`|
`
`Display
`
`A Computer
`6%
`
`@ 170
`
`Internet
`L150
`
`L120
`
`200
`
`210 220 230
`
`240
`
`245
`
`250
`
`
`
`Image
`Image
`[110
`.
`—>—> —> Collection
`Collection —
`
`Array
`Array
`:
`To
`____________________ __.'
`STB
`Z100
`140
`
`Object
`
`’
`
`130
`
`FIG. 2
`FIG. 2
`
`

`
`U.S. Patent
`
`Mmm
`
`US 7,071,968 B2
`
`:o_mmmaEoo
`
`mmc_m=m_
`
`
`
`:o=8:oo59:E88>> E8no>>mmosmo8mtmE_Eaves.mocacm
`
`
`
`v.®_u_
`
`E88>>.oH
`
`559:.8.
`
`2:
`
`S_
`
`md:
`
`8.0.am
`
`2982
`
`as$2
`
`_otm>:oo
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2006
`Jul. 4,2006
`
`Sheet 3 0f 18
`Sheet 3 of 18
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`480
`
`\
`
`482
`
`/'
`
`484
`
`\\
`481—/\\
`
`\
`
`I
`I/\-481
`
`I
`I
`\\ Scene I
`
`200
`
`486:
`
`K
`Input
`Device
`365
`
`488
`T’
`495
`
`/ Webcam
`100
`
`~/ 110
`
`STB
`
`FIG. 5A
`
`485
`
`FIG. 5B
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2006
`Jul. 4,2006
`
`Sheet 4 0f 18
`Sheet 4 of 18
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`490
`490
`\
`
`430
`480
`
`490
`_____ 490
`
`f 487
`
`FIG. 6A
`FIG. 6A
`
`480
`480
`
`FIG. 6B
`FIG. 6B
`
`487
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2006
`Jul. 4,2006
`
`Sheet 5 of 18
`Sheet 5 of 18
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`600
`
`600 \
`\
`
`Send Pan, Tilt or Zoom Command \/ 605
`Send Pan, Tilt or Zoom Command
`605
`
`I
`
`Capture Scene in Field of Vision of
`Capture Scene in Field of Vision of
`Webcam Lens
`Webcam Lens
`
`610
`
`I
`
`Store Captured Scene as Image Data
`Store Captured Scene as Image Data
`in an Image Collection Array
`in an Image Collection Array
`
`615
`V615
`
`I
`
`Process and Store the Scene Image __\
`Process and Store the Scene Image
`Data as a Digitized Scene Image Data
`620
`Data as a Digitized Scene Image Data
`620
`
`I
`
`Select Subset of Digitized Scene V 625
`Select Subset of Digitized Scene
`625
`Image Data
`Image Data
`
`630
`
`S
`
`_/ 635
`535
`
`I
`
`Perform Distortion Compensation
`Perform Distortion Compensation
`and Compression of Subset
`and Compression of Subset
`
`Transmit Distortion-compensated
`Transmit Distortion-compensated
`and Compressed Subset
`and Compressed Subset
`
`I
`@
`
`End
`
`FIG. 7
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4,2006
`Jul. 4, 2006
`
`Sheet 6 of 18
`Sheet 6 0f 18
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`
`
`200
`
`/ Webcam
`100
`
`q 110
`
`STB
`
`U 710
`
`200
`
`700
`
`705
`
`k
`Input
`Device
`365
`365
`
`740
`
`FIG. 8A
`FIG. 8A
`
`4 f 85
`
`FIG. 88
`
`Ilnlll'nll.
`
`FIG. 8C
`FIG. 8C
`
`f 330
`
`Frame
`Buffer
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2006
`Jul. 4,2006
`
`Sheet 7 0f 18
`Sheet 7 of 18
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`r 485
`
`Scene Image
`Scene Image
`Data
`
`Data
`
`FIG. 9
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2006
`Jul. 4,2006
`
`Sheet 8 of 18
`Sheet 8 of 18
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`8°°\
`
`Send Zoom In or Zoom Away Command
`Send Zoom In or Zoom Away Command
`
`\/ 805
`805
`
`Capture Scene in Field of Vision of Webcam Lens
`Capture Scene in Field of Vision of Webcam Lens
`
`'-\ 310
`310
`
`Store Captured Scene as Image Data in an Image
`Store Captured Scene as Image Data in an Image
`Collection Array
`Collection Array
`
`i
`
`Process and Store the Scene Image Data as a Digitized
`Process and Store the Scene Image Data as a Digitized
`Scene Image Data
`Scene Image Data
`
`V 815
`815
`
`820
`A 820
`
`i
`
`Select a Subset of Digitized Scene Image Data
`Select a Subset of Digitized Scene Image Data
`
`J 825
`325
`
`i
`
`Perform Processing of Subset Based on the Zoom Command f‘ 827
`Perform Processing of Subset Based on the Zoom Command
`827
`
`i
`
`Perform Distortion Compensation and Compression of Subset -/ 830
`Perform Distortion Compensation and Compression of Subset
`330
`
`i
`
`Transmit Distortion-compensated and Compressed Subset
`Transmit Distortion-compensated and Compressed Subset
`
`"\ 835
`835
`
`FIG. 10
`FIG. 10
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2006
`60024,MJ
`
`81f0
`Sheet 9 0f 18
`S
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`hHamuH_mflux’
`
`M3mp:mwmm
`
`S.O_u_
`: 6E
`
`
`CA 562522. he
`:o_ww_Ew:m:..8
`
`zvmomocmom
`56% 38m
`
`
`
`c9oo:oo;9_w
`86260 296
`
`
`
`2:38__.a;m-fi_u_._
`
`8m
`
`Eoemo38>>m_>88mEoom9%:
`
`
`
`

`
`U.S. Patent
`
`Jul. 4,2006
`
`Sheet 10 of 18
`
`US 7,071,968 B2
`
`mcmowB
`
`urge.I.o\.uI»..:.
`‘\4:
`,h..«,«\W§
`.l«_
`
`ufloemm.s\\‘\.,m.,..u,..¢9M
`
`82
`
`f.|Jr».fl
`
`x“mwem
`
`
`
`uBoo_mm.%L.__.m£.<o82
`
`‘:o_mw_Em.§._..2
`
`38¢ocmom
`
`
`
`umfimtoo.s$_m
`
`$2
`
`as
`
`28mE9.2
`
`82
`
`umsgoo395
`
`%8m9.8m
`
`
`
`:o_$_Ew:m:..2
`
`NF.o_u_
`
`
`
`2:38..§m-fi:..
`
`ocmom3oNo_.
`
`
`
`
`
`89m.mEmoE?...§m_>mcmowEoommc_>_._
`
`
`
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2006
`Jul. 4,2006
`
`Sheet 11 0f 18
`Sheet 11 of 18
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`1042?.
`
`1110
`
`Scene
`/ 1000
`
`i 1050
`
`1110
`
`1010
`
`365 J
`
`1040 J
`
`1125
`
`FIG. 13A
`FIG. 13A
`
`1020v
`
`Scene Image
`Scene Image
`Data
`
`FIG. 13B
`
`l
`l
`
`\/~1_____
`1055
`
`1100 0;
`Data
`
`1050
`
`:__?____
`
`

`
`U.S. Patent
`
`Jul. 4,2006
`
`Sheet 12 of 18
`
`US 7,071,968 B2
`
`pfléfl
`
`u9om_mm
`.xx.r7T.:
`
`.IA,,.m.s_
`xx‘...¢.Nfi,ml
`V942."
`
`..1.,..y/7
`
`mcmoms
`
`I
`
`E83
`
`88m8E535
`
`82
`
`89
`
`
`
`U$om._._oO>>mv_mcmswcooamxw
`
`
`
`__,M_MM_,m__w_,,,m,,
`
`co_wm_Em:m#.8
`
`3.O_u_
`
`mcmowB
`
`>o2230._m>m-;w_n_..Smemonu9sm_>ocoowEoomc_>_._
`
`
`
`
`
`
`
`
`
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2006
`Jul. 4,2006
`
`Sheet 13 0f 18
`Sheet 13 of 18
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2006
`Jul. 4,2006
`
`Sheet 14 0f 18
`Sheet 14 of 18
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`22
`
`99
`
`3E
`
`0;;
`
`moo?
`
`no?
`
`Wow?
`
`
`
`"Fish-Eye"CaptureofScene
`
`moo?
`
`1000
`
`
`
`
`
`LivingRoomSceneViewedbyCamera
`
`QM
`
`/ own?
`
`m: .QI
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4,2006
`Jul. 4, 2006
`
`Sheet 15 of 18
`Sheet 15 0f 18
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`
`
`FIG. 17
`FIG. 17
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2006
`Jul. 4,2006
`
`Sheet 16 6f 18
`Sheet 16 of 18
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`1364
`[
`
`1352
`
`1354
`
`Scene
`1356 K 1350
`
`I-— ———1- --- l- -"——|
`I
`1
`1
`1
`:
`1
`1
`‘1\
`1
`1
`1362\_/I|'>
`1
`1
`1
`\1366
`1
`J.
`1
`I
`1
`1
`1
`I
`1
`1
`1
`I
`1
`1
`1
`I
`1
`1
`1
`l
`|
`I
`l
`
`I _ _ _ _ __1 _ _ _ _ __1 _ _ _ _ __1
`
`F|G.18A
`
`1364
`\
`
`1360
`(
`
`F-“--1-----T-----|
`1
`1
`1
`
`’—1366
`
`1362~\1\€> 1
`
`1
`1
`
`1
`1
`
`1 (3)1’
`
`1
`1
`
`1
`1
`
`)
`
`1368 \
`
`t
`
`1
`
`l1370 / 1372
`
`
`
`
`
`Image Reconstruction V 1374
`Image Reconstruction
`1374
`Stage
`Stage
`
` 1376
`
`FIG. 18B
`FIG. 18B
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2006
`Jul. 4,2006
`
`Sheet 17 0f 18
`Sheet 17 of 18
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`\
`
`Camera
`1015
`1015
`
`:
`
`
`
`>
`
`Destination Device
`Destination Device
`(e.g., Printer, Computer,
`(e.g., Printer, Computer,
`Server)
`Server)
`
`/ 1320
`
`1325
`
`1320
`
`53 r r en
`
`1405
`
`K1410
`
`FIG. 19A
`FIG. 19A
`
`1400
`
`Camera
`Camera
`1015
`
`Destination Device
`
`Destination Device D
`1330
`
`k1410
`
`1400
`
`/ CPE —>
`1415)
`
`1320
`1320
`
`1325
`1325
`
`1330
`
`1405
`
`FIG. 19B
`FIG. 19B
`
`

`
`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2006
`Jul. 4,2006
`
`Sheet 18 of 18
`Sheet 18 of 18
`
`US 7,071,968 B2
`US 7,071,968 B2
`
`Capture Scene in Field of Vision of
`Capture Scene in Field of Vision of
`Camera Lens
`Camera Lens
`
`I
`
`Store Captured Scene as Image Data
`Store Captured Scene as Image Data
`in an Image Collection Array
`in an Image Collection Array
`
`I
`
`Process and Store the Scene Image Data
`Process and Store the Scene Image Data
`as a Digitized Scene Image Data
`as a Digitized Scene Image Data
`
`\_/ 1500
`1500
`
`A1505
`1505
`
`1510
`1510
`
`I
`
`Select a Plurality of Sublets of the Digitized A 1515
`Select a Plurality of Sublets of the Digitized
`1515
`Scene Image Data
`Scene Image Data
`
`I
`
`Perform Distortion Compensation and v 1520
`Perform Distortion Compensation and
`1520
`Compression of Sublets
`Compression of Sublets
`
`I
`
`Transmit the Distortion-compensated
`Transmit the Distortion-compensated
`and Compressed Sublets
`and Compressed Sublets
`
`/\ 1525
`1525
`
`FIG. 20
`
`

`
`US 7,071,968 B2
`
`1
`SYSTEM AND METHOD FOR A SOFTWARE
`STEERABLE WEB CAMERA WITH
`MULTIPLE IMAGE SUBSET CAPTURE
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a continuation-in-part of U.S. appli-
`cation Ser. No. 09/823,804, by common inventor Robert
`Novak, filed Mar. 30, 2001 now abandoned, and entitled
`“SYSTEM AND METHOD FOR A SOFTWARE STEER-
`
`ABLE WEB CAMERA”. application Ser. No. 09/823,804 is
`fully incorporated herein by reference.
`
`TECHNICAL FIELD
`
`This disclosure relates generally to digital imaging, digital
`video or web cameras, and more particularly but not exclu-
`sively, to systems and methods for capturing camera images
`by use of software control.
`
`BACKGROUND
`
`Conventional digital imaging, digital video or web cam-
`eras (“webcams”) can be used for teleconferencing, surveil-
`lance, and other purposes. One of the problems with con-
`ventional webcams is that they have a very restricted field of
`vision. This restricted vision field is due to the limitations in
`
`the mechanism used to control the webcam and in the optics
`and other components in the webcam.
`In order to increase the vision field of a webcam, the user
`might manually control the webcam to pan and/or tilt in
`various directions (e.g., side-to-side or up-and-down) and/or
`to zoom in or away from an image to be captured. However,
`this manual technique is inconvenient, as it requires the user
`to stop whatever he/she is doing, to readjust the webcam,
`and to then resume his/her previous activity.
`Various other schemes have been proposed to increase the
`webcam vision field, such as adding complex lens assem-
`blies and stepper motors to the webcams to permit the
`camera to perform the pan and zoom functions. However,
`complex lens assemblies are expensive and will make web-
`cams unaffordable for many consumers. Additionally, step-
`per motors use moving or mechanical parts that may fail
`after a certain amount of time, thus requiring expensive
`repairs or the need to purchase a new webcam. Stepper
`motors may also disadvantageously suffer from hysterisis, in
`which repeated pan,
`tilt or zooming operations lead to
`slightly inconsistent settings during each operation.
`Furthermore,
`repairs for webcams on set
`top boxes
`(STBs) are particularly expensive because of the required
`service call for repairing the STB webcam.
`Accordingly, there is need for a new system and method
`to allow webcams to increase their vision field. There is also
`
`a need for a new system and method to permit webcams to
`perform particular operations, such as panning, tilting, and/
`or zooming, without using stepper motors or requiring the
`user to physically adjust the webcam.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Non-limiting and non-exhaustive embodiments of the
`present invention are described with reference to the fol-
`lowing figures, wherein like reference numerals refer to like
`parts throughout the various views unless otherwise speci-
`fied.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`
`FIG. 1 is a block diagram showing a webcam coupled to
`a set top box according to an embodiment of the invention.
`FIG. 2 is a block diagram of an embodiment of the
`webcam of FIG. 1.
`
`FIG. 3 is a block diagram of an embodiment of the set top
`box of FIG. 1.
`
`FIG. 4 is a block diagram of one example of a memory
`device of the set top box.
`FIG. 5A is an illustrative example block diagram showing
`a function of the webcam of FIG. 1 in response to particular
`pan and/or tilt commands.
`FIG. 5B is an illustrative example block diagram of
`selected subsets in a digitized scene image data in response
`to particular pan and/or tilt commands.
`FIG. 6A is an illustrative example block diagram of a
`selected subset image data with distortions.
`FIG. 6B is an illustrative example block diagram of a
`selected subset image data that has been distortion compen-
`sated.
`
`FIG. 7 is a flowchart of a method according to an
`embodiment of the invention.
`
`FIG. 8A is an illustrative example block diagram showing
`a function of the webcam of FIG. 1 in response to particular
`pan and zoom commands.
`FIG. 8B is an illustrative example block diagram of a
`selected subset in the digitized scene image data in response
`to a particular pan command.
`FIG. 8C is an illustrative example block diagram of the
`selected subset in FIG. 8B in response to a particular zoom
`command.
`
`FIG. 9 is an illustrative example block diagram of the
`selected subset in FIG. 9 in response to another particular
`zoom command.
`
`FIG. 10 is a flowchart of a method according to another
`embodiment of the invention.
`
`FIG. 11 is another diagram shown to further assist in
`describing an operation of an embodiment of the invention.
`FIG. 12 is a diagram illustrating an operation of an
`embodiment of the invention.
`
`FIG. 13A is an illustrative example block diagram show-
`ing a function of the camera of FIG. 12 in response to
`particular pan, tilt, and/or zoom commands.
`FIG. 13B is an illustrative example block diagram of
`selected subsets in a digitized scene image data in response
`to particular pan, tilt, and/or zoom commands.
`FIG. 14 is a diagram illustrating an operation of another
`embodiment of the invention.
`
`FIG. 15 is an illustrative example block diagram of
`selected particular subsets a digitized scene image data
`related to FIG. 14.
`
`FIG. 16 is a diagram illustrating another operation of an
`embodiment of the invention where selected image data
`subsets overlap.
`FIG. 17 is an illustrative example block diagram of
`selected subsets in a digitized scene image data where at
`least some of the selected subsets overlap.
`FIG. 18A is a diagram illustrating another operation of an
`embodiment of the invention.
`
`FIG. 18B is an illustrative example block diagram of
`selected particular subsets a digitized scene image data
`related to FIG. 18A.
`
`FIG. 19A is a diagram illustrating an operation of an
`embodiment of the invention where image data subsets are
`transmitted from a camera to a destination device.
`
`

`
`US 7,071,968 B2
`
`3
`FIG. 19B is a diagram illustrating an operation of an
`embodiment of the invention where image data subsets are
`transmitted from a customer premise equipment to a desti-
`nation device.
`
`FIG. 20 is a flowchart of a method according to another
`embodiment of the invention.
`
`5
`
`DETAILED DESCRIPTION OF THE
`ILLUSTRATED EMBODIMENTS
`
`Embodiments of a system and method for a software
`steerable camera are disclosed herein. As an overview, an
`embodiment of the invention provides a system and method
`that capture camera images by use of software control. As an
`example, the camera may be web camera or other types of
`camera that can support a wide angle lens. The wide angle
`lens is used to capture a scene or image in the wide field of
`vision. The captured scene or image data is then stored in an
`image collection array and then digitized and stored in
`memory. In one embodiment, the image collection array is
`a relatively larger sized array to permit the array to store
`image data from the wide vision field. Processing is per-
`formed for user commands to effectively pan or tilt the
`webcam in particular directions and/or to zoom the webcam
`toward or away from an object to be captured as an image.
`However,
`instead of physically moving the webcam in
`response to the user commands, a particular subset of the
`digitized data is selected and processed so that selected
`subset data provides a simulated panning,
`tilting, and/or
`zooming of the image of the captured object. A compression/
`correction engine can then compensate the selected subset
`data for distortion and compress the selected subset data for
`transmission.
`
`In another embodiment, a plurality of subsets in the
`digitized data are selected and processed prior to transmit-
`ting the data subsets to a destination device. Particular
`subsets may be overlapping or non-overlapping in the digi-
`tized data. A motion detector may, for example, be used to
`determine the location of at least one of the data subsets.
`
`This embodiment may permit a single camera to simulate
`multiple virtual cameras, since images from multiple focus
`areas can be serially captured and integrated into a single,
`integrated output image.
`The invention advantageously permits a camera, such as
`a webcam, to have a wide vision field. The invention may
`also advantageously provide a wide vision field for cameras
`that have short depth fields. The invention also advanta-
`geously avoids the use of stepper motors to obtain particular
`images based on pan and zoom commands from the user.
`In the description herein, numerous specific details are
`provided, such as the description of system components in
`FIGS. 1 through 20, to provide a thorough understanding of
`embodiments of the invention. One skilled in the relevant art
`
`will recognize, however, that the invention can be practiced
`without one or more of the specific details, or with other
`methods, components, materials, parts, and the like. In other
`instances, well-known structures, materials, or operations
`are not shown or described in detail to avoid obscuring
`aspects of the invention.
`Reference throughout this specification to “one embodi-
`ment”, “an embodiment”, or “a specific embodiment” means
`that a particular feature, structure, or characteristic described
`in connection with the embodiment is included in at least
`
`one embodiment of the present invention. Thus, the appear-
`ances of the phrases “in one embodiment”, “in an embodi-
`ment”, or “in a specific embodiment” in various places
`throughout this specification are not necessarily all referring
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`
`to the same embodiment. Furthermore, the particular fea-
`tures, structures, or characteristics may be combined in any
`suitable manner in one or more embodiments.
`
`FIG. 1 is a block diagram showing a webcam 100 coupled
`to a set top box (“STB”) 140 according to an embodiment of
`the invention. The webcam 100 can capture an image of an
`object 130 that is in the webcam field of vision. Webcam 100
`is coupled to STB 140 via,
`for example, a cable 110.
`Webcam 100 may also be coupled to STB 140 by use of
`other suitable connections or methods, such as IR beams,
`radio signals, suitable wireless transmission techniques, and
`the like. Typically, STB 140 is coupled to a cable network
`160 and receives TV broadcasts, as well as other data, from
`the cable network 160. Typically, STB 140 is also coupled
`to the Internet 150 or other networks for sending and
`receiving data. Data received from the Internet 150 or cable
`network 160 may be displayed on a display 120. STB 140
`may also transmit images that are captured by the webcam
`100 to other computers via the Internet 150. STB may also
`transmit the captured webcam images to a printer 165 and/or
`to other devices 170 such as a computer in a local area
`network.
`
`It is noted that embodiments of the invention may also be
`implemented in other types of suitable cameras that can
`support a wide angle lens. For example, an embodiment of
`the invention may be implemented in, for example, security
`cameras, ATM cash machine cameras, spy cameras, portable
`cameras, or pin-hole type cameras. It is further noted that the
`invention is not
`limited to the use of STB 140. Other
`
`processing device may be used according to embodiments of
`the invention to perform image distortion compensation,
`image compression, and/or other functions that will be
`described below.
`
`FIG. 2 is a block diagram of an embodiment of the
`webcam 100 of FIG. 1. Webcam 100 comprises a lens 210;
`a shutter 220; a filter 230; an image collection array 240; a
`sample stage 245; and an analog to digital converter
`(“ADC”) 250. The lens 210 may be a wide angle lens, such
`as a fish-eye lens, that has angular field of, for example, at
`least about 140 degrees, as indicated by lines 200. Using a
`wide-angle lens allows webcam 100 to capture a larger
`image area than a conventional webcam. Shutter 220 opens
`and closes at a pre-specified rate, allowing light into the
`interior of webcam 100 and onto a filter 230. Filter 230
`
`allows for image collection array 240 to capture different
`colors of an image and may include a static filter, such as a
`Bayer filter, or may include a spinning disk filter. In another
`embodiment, the filter may be replaced with a beam splitter
`or other color differentiation device. In another embodiment,
`webcam 100 does not include a filter or other color differ-
`entiation device.
`
`In one embodiment, the image collection array 240 can
`include charge coupled device (“CCD”) sensors or comple-
`mentary metal oxide semiconductor (“CMOS”) sensors,
`which are generally much less expensive than CCD sensors
`but may be more susceptible to noise. Other types of sensors
`may be used in the image collection array 240. The size of
`the image collection array 240 is relatively larger in size
`such as, for example, 1024 by 768, 1200 by 768, or 2000 by
`1000 sensors. The large sized array permits the array 240 to
`capture images in the wide vision field 200 that is viewed by
`the webcam 200.
`
`A sample stage 245 reads the image data from the image
`collection array 240 when shutter 220 is closed, and an
`analog-to-digital converter (ADC) 250 converts the image
`data from an analog to digital form, and feeds the digitized
`image data to STB 140 via cable 110 for processing and/or
`
`

`
`US 7,071,968 B2
`
`5
`transmission. In an alternative embodiment, the image data
`may be processed entirely by components of the webcam
`100 and transmitted from webcam 100 to other devices such
`
`as the printer 165 or computer 170.
`For purposes of explaining the functionality of embodi-
`ments of the invention, other conventional components that
`are included in the webcam 100 have been omitted in the
`
`figures and are not discussed herein.
`FIG. 3 is a block diagram of an embodiment of the set top
`box (STB) 140. STB 140 includes a network interface 300;
`a processor 310; a memory device 320; a frame bulfer 330;
`a converter 340; a modem 350; a webcam interface 360, and
`an