(12) United States Patent
`Lipton et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,932,923 B2
`Apr. 26, 2011
`
`US007932923B2
`
`(54) VIDEO SURVEILLANCE SYSTEM
`EMPLOYINGVIDEO PRIMITIVES
`
`(75) Inventors: Alan J. Lipton, Falls Church, VA (US);
`Thomas M. Strat, Pakton, VA (US);
`Peter L. Venetianer, McLean, VA (US);
`Mark C. Allmen, Morrison, CO (US);
`William E. Severson, Littleton, CO
`(US); Niels Haering, Arlington, VA
`(US); Andrew J. Chosak, McLean, VA
`(US); Zhong Zhang, Herndon, VA (US);
`Matthew F. Frazier, Arlington, VA
`(US); James S. Seekas, Arlington, VA
`(US); Tasuki Hirata, Silver Spring, MD
`(US); John Clark, Leesburg, VA (US)
`(73) Assignee: ObjectVideo, Inc., Reston, VA (US)
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`(21) Appl. No.: 12/569,116
`(22) Filed:
`Sep. 29, 2009
`
`(65)
`
`Prior Publication Data
`US 201O/OO13926A1
`Jan. 21, 2010
`
`(51) Int. Cl.
`(2006.01)
`H04N 7/8
`(52) U.S. Cl. ....................................................... 348/143
`(58) Field of Classification Search .................. 375/143,
`375/144, 145, 148; H04N 7/18
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`3,812,287 A
`5, 1974 Lemelson
`4,249,207 A
`2, 1981 Harman et al.
`4,257,063 A
`3/1981 Loughry et al.
`
`4, 1988 Araki et al.
`4,737,847 A
`3/1990 Fujioka et al.
`4,908,704 A
`9, 1995 Soohoo
`5.448,315 A
`2, 1996 Odle
`5,491,511 A
`5/1996 Hennessey et al.
`5,515,453 A
`3, 1997 Abecassis
`5,610,653 A
`4/1997 Camire
`5,623,249 A
`5,696,503 A 12/1997 Nasburg
`5,801,943 A
`9/1998 Nasburg
`5,802.361 A
`9/1998 Wang et al.
`5,850,352 A 12/1998 MoeZZi et al.
`5,860,086 A
`1/1999 Crump et al.
`5,872,865 A
`2f1999 Normile et al.
`5,886,701 A
`3, 1999 Chauvin et al.
`5,912,980 A
`6/1999 Hunke
`5,926,210 A
`7/1999 Hackett et al.
`5,956,081 A
`9, 1999 Katz et al.
`5,959,690 A
`9/1999 Toebes, VIII et al.
`(Continued)
`
`EP
`
`FOREIGN PATENT DOCUMENTS
`O293,189 B1
`T 1994
`(Continued)
`
`OTHER PUBLICATIONS
`International Search Report for International Application No. PCT/
`US08/09073, dated Nov. 3, 2008.
`(Continued)
`
`Primary Examiner — Tung Vo
`(74) Attorney, Agent, or Firm — Muir Patent Consulting,
`PLLC
`
`ABSTRACT
`(57)
`A video Surveillance system is set up, calibrated, tasked, and
`operated. The system extracts video primitives and extracts
`event occurrences from the video primitives using event dis
`criminators. The system can undertake a response. Such as an
`alarm, based on extracted event occurrences.
`
`41 Claims, 7 Drawing Sheets
`
`15
`
`14
`
`Video
`SeSOS
`
`video
`recorders
`
`
`
`
`
`
`
`other
`SeSOS
`
`17
`
`11
`
`Computer system
`
`
`
`Computer-readable
`medium
`
`16
`
`I/O devices
`
`Canon Ex. 1001 Page 1 of 23
`
`

`

`US 7,932,923 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`10, 1999
`Polish
`5,963,202
`Goldberg et al.
`10, 1999
`5,963,203
`11, 1999
`5,983,147
`Krumm
`11, 1999
`Abecassis
`5,987,211
`Kajiya et al.
`12, 1999
`5.999,189
`Hennessey et al.
`1, 2000
`6,014,461
`2, 2000
`MacCormacket al.
`6,031,573
`5, 2000
`Hudson et al.
`6,069,653
`6, 2000
`6,075,560
`Katz
`T/2000
`Mead
`6,088,484
`Seeley et al.
`T/2000
`6,091,771
`Seeley et al.
`8, 2000
`6,097.429
`9, 2000
`Carder et al.
`6,123,123
`11, 2000
`Jain et al.
`6,144,375
`Jang
`11, 2000
`6,151413
`12, 2000
`Jaszlics et al.
`6,166,744
`Billington et al.
`1, 2001
`6,177,886
`Chang et al.
`2, 2001
`6,025,877
`3, 2001
`Schaffer
`6,201.473
`4, 2001
`Scaman et al.
`6,211,907
`Qian et al.
`5, 2001
`6,226,388
`10, 2001
`Schatz et al.
`6,297,844
`10, 2001
`Moon et al.
`6,307,885
`10, 2001
`6,310,916
`Han
`Snyder et al.
`12, 2001
`6,326,964
`2, 2002
`Bulman
`6,351,265
`2, 2002
`Kim
`6,351,492
`3, 2002
`Jain et al.
`6,360,234
`6, 2002
`Ito et al.
`6,404,455
`Vaithilingam et al.
`6, 2002
`6,411,724
`Courtney
`T/2002
`6.424,370
`1, 2003
`Lemons et al.
`6,504,479
`2, 2003
`Streetman et al.
`6,525,658
`4, 2003
`Barker et al.
`6,542,075
`4, 2003
`Okamoto et al.
`6,542,840
`4, 2003
`Adler et al.
`6,552,826
`Tsering
`5/2003
`6,570,608
`6, 2003
`Madrane
`6,573,907
`Murray et al.
`T/2003
`6,597,800
`9, 2003
`Brill et al.
`6,628,835
`11, 2003
`DaGraca et al.
`6,646,676
`2, 2004
`Venetianer et al.
`6,696,945
`Wang
`3, 2004
`6,707,852
`Qian et al. .................... 382,224
`4, 2004
`6,721.454
`4, 2004
`Toklu et al.
`6,724.915
`4, 2004
`Randall
`6,727,938
`5, 2004
`Allmen et al.
`6,738,424
`Nagaya
`5, 2004
`6,741,977
`10, 2004
`Owechko et al.
`6,801.662
`11, 2004
`Ito et al.
`6,812,835
`11, 2004
`Brill et al.
`6,816, 184
`12, 2004
`Nichani et al.
`6,829,371
`1/2005
`Grech-Clini
`6,844,818
`3, 2005
`Bush
`6,865,580
`8, 2005
`Dorbie
`6,924,801
`9, 2005
`6,940,998
`Garoutte
`Lipton
`10, 2005
`6,954.498
`Nagahisa
`1, 2006
`6,987,528
`Lipton et al.
`1, 2006
`6,987,883
`4, 2006
`Olson
`7,023,469
`1/2007
`Comaniciu et al.
`7,167,519
`3, 2007
`Hsu et al.
`7,197,072
`5/2007
`Ito et al.
`7,215,795
`6, 2007
`Srinivasa et al.
`7,227,893
`Ellenby
`11/2007
`7,301,536
`12, 2007
`Broemmelsiek
`7,307,652
`4, 2008
`Dimitrova
`7,356,830
`10, 2008
`Yeredor et al.
`7,436,887
`11, 2008
`Brown et al.
`7.447,331
`1/2009
`Merheim et al.
`7,479,980
`1, 2010
`Paek et al. ............................. 1f1
`7,653,635
`2, 2010
`Lu et al.
`7,660,439
`2001/OO19357
`9, 2001
`Ito et al.
`2001/OO33330
`10, 2001
`Garoutte
`2001/0035907
`11, 2001
`Broemmelsiek
`2002fOOO8758
`1, 2002
`Broemmelsiek et al.
`2002fOO24446
`2, 2002
`Grech-Clini
`2002/0051058
`5, 2002
`Ito et al.
`
`6/2002 Church et al.
`2002fOO82769 A1
`7/2002 Barker et al.
`2002fOO95490 A1
`2002/0135483 A1
`9, 2002 Merheim et al.
`2002/0163521 A1 1 1/2002 Ellenby
`2002/019 1851 A1 12/2002 Keinan
`2003.0043.160 A1
`3/2003 Elfving et al.
`2003/005.1255 A1
`3/2003 Bulman et al.
`2003/0053659 A1
`3/2003 Pavlidis et al.
`2003/0O85992 A1
`5/2003 Arpa et al.
`2003/0231769 A1 12/2003 Bole et al.
`2004/O113933 A1
`6/2004 Guler
`2004/O130620 A1
`7/2004 Buehler et al.
`2004O161133 A1
`8, 2004 Elazar et al.
`2004/0240542 A1 12/2004 Yeredor et al.
`2005, 0146605 A1
`7/2005 Lipton et al.
`2005. O157169 A1
`7/2005 Brodsky et al.
`2005/0162515 A1
`7/2005 Venetianer et al.
`2005/O168574 A1
`8/2005 Lipton et al.
`2005, 0169367 A1
`8, 2005 Venetianer et al.
`2005, 0198063 A1
`9, 2005 Thomas et al.
`2006/0232673 Al 10/2006 Lipton et al.
`2006/0279630 Al 12/2006 Aggarwal et al.
`2007,0002141 A1
`1/2007 Lipton et al.
`2007/OO 13776 A1
`1/2007 Venetianer et al.
`2007/00528O3 A1
`3/2007 Chosak et al.
`2007/O127774 A1
`6/2007 Zhang et al.
`2008. O100704 A1
`5/2008 Venetianer et al.
`
`FOREIGN PATENT DOCUMENTS
`O893.923 A1
`1, 1999
`EP
`O967584 A2 12/1999
`EP
`1024666 A2
`8, 2000
`EP
`1120746
`8, 2001
`EP
`1333682 A1
`8, 2003
`EP
`2009-247654 A
`9, 1997
`JP
`10-048008
`2, 1998
`JP
`10-290449
`10, 1998
`JP
`2000-175174
`6, 2000
`JP
`2000-339.923
`8, 2000
`JP
`2000-224.542
`11, 2000
`JP
`2001-175868
`6, 2001
`JP
`2001-285681
`10, 2001
`JP
`94.03.014 A1
`2, 1994
`WO
`O1/62005
`8, 2001
`WO
`WO-03/044727 A1
`5, 2003
`WO
`WO WO-2004/006184 A2
`1, 2004
`
`OTHER PUBLICATIONS
`Written Opinion for International Application No. PCT/US08/
`09073, dated Nov. 3, 2008.
`A. Selinger and L. Wixson, "Classifying Moving Objects as Rigid or
`Non-Rigid Without Correspondences.” Proceedings of DARPA
`Image Understanding Workshop, 1, Nov. 1998, pp. 341-347.
`Alan J. Lipton, Virtual Postman—An Illustrative Example of Virtual
`Video, International Journal of Robotics and Automation, vol. 15,
`No. 1, Jan. 2000, pp. 9-16.
`Alan J. Lipton, Virtual Postman Real Time, Interactive Virtual
`Video, IASTED Conference on Computer Graphics and Imaging
`(CGIM 99), Palm Springs, Oct. 25-27, 1999.
`Robert T. Collins et al., “A System for Video Surveillance and Moni
`toring.” Technical Report CMU-RI-TR-00-12, Robotics Institute,
`Carnegie Mellon University, May 2000.
`Jemez Technology Corp., Variant iD Web-Site, www.variantid.com,
`printed Aug. 25, 2003.
`JP Office Action issued in PCT/US2002/22688 dated Oct. 9, 2007.
`A. J. Lipton et al., “MovingTarget Classifications and Tracking from
`Real-time Video.” IUW, pp. 159-175, May 1997.
`A. J. Lipton, “Local Application of Optic Flow to Analyze Rigid
`Versus Non-Rigid Motion.” International Conference on Computer
`Vision, Corfu, Greece, Sep. 1999.
`A. J. Lipton, H. Fujiyoshi and R. S. Patil, “Moving Target Classifi
`cation and Tracking for Real-time Video.” Proceedings of IEEE
`WACV98, Princeton, NJ, 1998, pp. 8-14.
`C. R. Wren et al., “Pfinder: Real-time Tracking of the Human Body.”
`PAMI, vol. 19, pp. 780-784, 1997.
`
`Canon Ex. 1001 Page 2 of 23
`
`

`

`US 7,932,923 B2
`Page 3
`
`C. R. Wren et al., “Pfinder: Real-time Tracking of the Human Body.”
`Vismod, 1995.
`CN Office Action for CN 0282.2772.7 on Oct. 14, 2005.
`D. M. Gavrila, “The Visual Analysis of Human Movement: A Sur
`vey.” CVIU, 73(1):82-98, Jan. 1999.
`F. Bartolini et al., “Counting People Getting in and Out of a Bus by
`Real-time Image-sequence Processing.” IVC, 12(1):36-41, Jan.
`1994.
`H. Fujiyoshi and A. J. Lipton, “Real-time Human Motion Analysis by
`Image Skeletonization.” Proceedings of IEEE WACV'98, Princeton,
`NJ, 1998, pp. 15-21.
`International Search Report for International Application No. PCT/
`2001/32614 on May 6, 2002.
`International Search Report for International Application No. PCT/
`2006/25196, mailed Jan. 16, 2008.
`International Search Report for International Application No. PCT/
`US06/45625, mailed on Sep. 24, 2007.
`International Search Report for International Application No. PCT/
`US2002/22688 on Dec. 11, 2002.
`International Search Report for International Application No. PCT/
`US2006/012556, mailed on Feb. 12, 2008.
`International Search Report for International Application No. PCT/
`US2006/02700 dated Apr. 13, 2007.
`J.P. Deparis et al., “A Device for Counting Passenger Making Use of
`Two Active Linear Cameras: Comparison of Algorithms.” IEEE, pp.
`1629-1634, 1996.
`L. Khoudour et al., “Real-time Pedestrian Counting by Active Linear
`Cameras,” JEI, 5(4):452-459, Oct. 1996.
`L. Wixson et al., “Detecting Salient Motion by Accumulating
`Directionally-Consistent Flow.” IEEE 1999.
`L. Wixson et al., “Detecting Salient Motion by Accumulating
`Directionally-Consistent Flow.” IEEE Trans Pattern Anal. Mach.
`Intell., vol. 22, pp. 774-781, Aug. 2000.
`
`M. Rossiet al., “Tracking and Counting Moving People.” ICIP94, pp.
`212-216, 1994.
`M. Allmen et al., “Long-Range Spatiotemporal Motion Understand
`ing Using Spatiotemporal Flow Curves.” Proc. IEEE CVPR,
`Lahaina, Maui, Hawaii. pp. 303-309, 1991.
`M. Isard et al., “BraMBLe: A Bayesian Multiple-Blob Tracker.”
`ICCV, 2001.
`N. Haering et al., “Visual Event Detection.” Video Computing Series,
`Editor Mubarak Shah, 2001.
`Notification for IL Application No. 161777 issued Feb. 21, 2008.
`R.T. Collins, Y. Tsin, J. R. Miller and A. J. Lipton, “Using a DEM to
`Determine Geospatial Object Trajectories' CMU-RI-TR-98-19,
`1998.
`S. Ioffe et al., “Probabilistic Methods for Finding People.” IJCV.
`43(1):45-68, Jun. 2001.
`Shio et al., "Segmentation of People in Motion.” Proc. IEEE, 1991,
`vol. 79, pp. 325-332.
`T.J. Olsen et al., “Moving Object Detection and Event Recognition
`Algorithm for Smart Cameras.” IUW, pp. 159-175, May 1997.
`W.E.L. Grimson et al., “Using Adaptive Tracking to Classify and
`Monitor Activities in a Site.” CVPR, pp. 22-29, Jun. 1998.
`Written Opinion for International Application No. PCT/US2006/
`25196, mailed on Jan. 16, 2008.
`Written Opinion issued in PCT Application No. PCT/US2006/
`0 12556, mailed on Feb. 12, 2008.
`Written Opinion of the International Searching Authority issued for
`PCT Application No. PCT/US2006/45625, mailed on Sep. 24, 2007.
`A. J. Lipton et al., “MovingTarget Classifications and Tracking from
`Real-time Video.” IUW, pp. 129-136, 1998.
`* cited by examiner
`
`Canon Ex. 1001 Page 3 of 23
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 1 of 7
`
`US 7,932,923 B2
`
`
`
`| ||
`
`
`
`
`
`
`
`
`
`/ |
`
`Canon Ex. 1001 Page 4 of 23
`
`

`

`U.S. Patent
`U.S. Patent
`
`A
`
`11026,2
`
`f02
`
`US 7,932,923 B2
`299
`
`m
`
`
`
`
`
`82890::820%»:
`
`mmcm
`
`3mGE
`
`
`
`
`
`7vGE
`
`
`S9653::Hombxmmmooom263Uxwfl
`
`flmpmmaoaammm389508$228.55996
`
`
`.0/v.wmcoamw.Em>m82>82:0685:53.5
`
`«M69532Em>m82>82>850m
`
`
`w9&5?anmm882500032:83mm>:_E_.a82>
`
`
`
`wxmtmuczH‘09;me@2506HombxoEwfio
`
`Canon EX. 1001 Page 5 of 23
`
`3:823:52
`
`£52£82
`
`
`
`.98895:QO
`
`333:3.»wme
`
`>5ch
`
`£3.30
`
`Canon Ex. 1001 Page 5 of 23
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 3 of 7
`
`US 7,932,923 B2
`
`GG
`
`
`
`
`
`
`
`‘?suodsou
`
`e?e?udoudde se
`
`Canon Ex. 1001 Page 6 of 23
`
`

`

`U.S. Patent
`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 4 of 7
`
`US 7,932,923 B2
`US 7,932,923 B2
`
`62%582. REm»EEE
`69%E330 mmvwNwrm
`
`
` 60.3080:5BmEo829:9.8.30805:89:m_>82>858BwoN_w
`
`
`
`BEBExomb998mm33.3089%£550
`BEBE mmcmco«3
`w.9“.
`
`ENE—Bo$3.30
`
`mnoEcouoEm_>00U_>woSom
`
`
`
`
`
`
`
`
`
`
`
`
`5.GE
`
`
`
`$00.30“02va
`
`mmcmcom_>
`
`
`
`xow:
`
`$00.50
`
`
`
`593032%..
`
`€5235:52
`
`988meBmmflmB033
`
`
`
`mmmB$8.30:85?
`
`
`
`£8.3069%
`
`Canon EX. 1001 Page 7 of 23
`
`Canon Ex. 1001 Page 7 of 23
`
`
`
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 5 of 7
`
`US 7,932,923 B2
`
`
`
`
`
`Canon Ex. 1001 Page 8 of 23
`
`

`

`U.S. Patent
`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 6 of 7
`
`US 7,932,923 B2
`US 7,932,923 B2
`
`
`
`umfiom cozothn.
`
`.N:2?\
`
`
`
`
`
`Sock?"$08230.><
`
`GEE...33
`
`
`
`
`
`.505:”emEQmao.><
`
`
`
`”2on.28-$qu
`
`GEE...26..
`
`
`
`gmm”met:23.><
`
`
`
`
`
`SoEN“QmEBmso.><
`
`
`
`555
`
`820
`
`wxomcm
`
`mvom
`
`Nr.OE
`
`".ongoo.$55
`
`
`
`.30qu“mLoEoumso.><
`
`
`.owwm68:.=0>>D.><§§E:
`
`
`
`
`59:3”whwEBmao.><,
`6%wm6:5=95.><r,
`
`
`
`582°:-820,
`
`
`
`”can;:9...
`
`
`
`
`
`._:o£mmuflmESwzo.><
`
`scamBI-wmnow
`
`
`
`
`
`59:9.”mLmEBmzo.><
`
`S,.07.
`
`2
`.07.
`
`
`
`.8wNNH95...:95.><
`
`Canon EX. 1001 Page 9 of 23
`
`Canon Ex. 1001 Page 9 of 23
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 7 of 7
`
`US 7,932,923 B2
`
`GQUE50:EQO......
`
`“50%QEgo,0E86§Emam1
`
`ammo”n::03an.
`
`coema0+02
`
`
`
`_,ESQ#0mm;Emama
`
`
`
`090EmmmEQO..
`
`”Nmowfl:COmLmumo
`
`m?.07.
`
`Canon EX. 1001 Page 10 of 23
`
`Canon Ex. 1001 Page 10 of 23
`
`
`

`

`US 7,932,923 B2
`
`1.
`VIDEO SURVELLANCE SYSTEM
`EMPLOYINGVIDEO PRIMITIVES
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application claims the priority to U.S. patent applica
`tion Ser. No. 09/987,707, filed Nov. 15, 2001, which claims
`priority to U.S. patent application Ser. No. 09/694,712, now
`U.S. Pat. No. 6,954,498, each of which is incorporated herein
`by reference in their entirety.
`
`BACKGROUND OF THE INVENTION
`
`Field of the Invention
`
`The invention relates to a system for automatic video Sur
`veillance employing video primitives.
`
`REFERENCES
`
`5
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`For the convenience of the reader, the references referred to
`herein are listed below. In the specification, the numerals
`within brackets refer to respective references. The listed ref.
`erences are incorporated herein by reference.
`The following references describe moving target detection:
`{1} A. Lipton, H. Fujiyoshi and R. S. Patil, "Moving Target
`Detection and Classification from Real-Time Video. Pro
`ceedings of IEEE WACV '98, Princeton, N.J., 1998, pp.
`8-14.
`{2} W. E. L. Grimson, et al., “Using Adaptive Tracking to
`Classify and Monitor Activities in a Site'. CVPR, pp.
`22-29, June 1998.
`{3}. A. J. Lipton, H. Fujiyoshi, R. S. Patil, “Moving Target
`Classification and Tracking from Real-time Video.” IUW,
`pp. 129-136, 1998
`{4} T.J. Olson and F. Z. Brill, “Moving Object Detection and
`Event Recognition Algorithm for Smart Cameras.” IUW,
`pp. 159-175, May 1997.
`The following references describe detecting and tracking
`humans:
`{5} A.J. Lipton, “Local Application of Optical Flow to Anal
`yse Rigid Versus Non-Rigid Motion.” International Con
`ference on Computer Vision, Corfu, Greece, September
`45
`1999.
`{6} F. Bartolini, V. Cappellini, and A. Mecocci, "Counting
`people getting in and out of a bus by real-time image
`sequence processing.” IVC, 12(1):36-41, January 1994.
`{7} M. Rossi and A. Bozzoli, "Tracking and counting moving
`people.” ICIP94, pp. 212-216, 1994.
`{8 C. R. Wren, A. Azarbayejani, T. Darrell, and A. Pentland,
`“Pfinder: Real-time tracking of the human body. Vismod,
`1995.
`{9} L. Khoudour, L. Duvieubourg, J. P. Deparis, “Real-Time
`Pedestrian Counting by Active Linear Cameras. JEI, 5(4):
`452-459, October 1996.
`{10}. S. Ioffe, D. A. Forsyth, “Probabilistic Methods for Find
`ing People.” IJCW 43(1):45-68, June 2001.
`{11} M. Isard and J. MacCormick, “BraMBLe: A Bayesian
`Multiple-Blob Tracker,” ICCV, 2001.
`The following references describe blob analysis:
`{12} D. M. Gavrila, “The Visual Analysis of Human Move
`ment: A Survey. CVIU, 73(1):82-98, January 1999.
`{13 Niels Haering and Niels da Vitoria Lobo, “Visual Event
`Detection.” Video Computing Series, Editor Mubarak
`Shah, 2001.
`
`50
`
`55
`
`60
`
`65
`
`2
`The following references describe blob analysis for trucks,
`cars, and people:
`{14 Collins, Lipton, Kanade, Fujiyoshi, Duggins, Tsin, Tol
`liver, Enomoto, and Hasegawa, “A System for Video Sur
`veillance and Monitoring: VSAM Final Report. Technical
`Report CMU-RI-TR-00-12, Robotics Institute, Carnegie
`Mellon University, May 2000.
`{15 Lipton, Fujiyoshi, and Patil, “Moving Target Classifi
`cation and Tracking from Real-time Video. 98 Darpa
`IUW, Nov. 20-23, 1998.
`The following reference describes analyzing a single-per
`son blob and its contours:
`{16} C. R. Wren, A. Azarbayejani, T. Darrell, and A. P.
`Pentland. “Pfinder: Real-Time Tracking of the Human
`Body.” PAMI, vol 19, pp. 780-784, 1997.
`The following reference describes internal motion of
`blobs, including any motion-based segmentation:
`{17 M. Allmen and C. Dyer, “Long-Range Spatiotemporal
`Motion Understanding Using Spatiotemporal Flow
`Curves.” Proc. IEEE CVPR, Lahaina, Maui, Hi., pp. 303
`309, 1991.
`{18} L. Wixson, “Detecting Salient Motion by Accumulating
`Directionally Consistent Flow, IEEE Trans. Pattern Anal.
`Mach. Intell. Vol. 22, pp. 774-781, August, 2000.
`
`BACKGROUND OF THE INVENTION
`
`Video surveillance of public spaces has become extremely
`widespread and accepted by the general public. Unfortu
`nately, conventional video Surveillance systems produce Such
`prodigious Volumes of data that an intractable problem results
`in the analysis of video Surveillance data.
`A need exists to reduce the amount of video surveillance
`data so analysis of the video Surveillance data can be con
`ducted.
`A need exists to filter video surveillance data to identify
`desired portions of the video surveillance data.
`
`SUMMARY OF THE INVENTION
`
`An object of the invention is to reduce the amount of video
`Surveillance data so analysis of the video Surveillance data
`can be conducted.
`An object of the invention is to filter video surveillance data
`to identify desired portions of the video surveillance data.
`An object of the invention is to produce a real time alarm
`based on an automatic detection of an event from video Sur
`veillance data.
`An object of the invention is to integrate data from surveil
`lance sensors other than video for improved searching capa
`bilities.
`An object of the invention is to integrate data from surveil
`lance sensors other than video for improved event detection
`capabilities
`The invention includes an article of manufacture, a
`method, a system, and an apparatus for video Surveillance.
`The article of manufacture of the invention includes a
`computer-readable medium comprising Software for a video
`Surveillance system, comprising code segments for operating
`the video surveillance system based on video primitives.
`The article of manufacture of the invention includes a
`computer-readable medium comprising Software for a video
`Surveillance system, comprising code segments for accessing
`archived video primitives, and code segments for extracting
`event occurrences from accessed archived video primitives.
`
`Canon Ex. 1001 Page 11 of 23
`
`

`

`US 7,932,923 B2
`
`3
`The system of the invention includes a computer system
`including a computer-readable medium having Software to
`operate a computer in accordance with the invention.
`The apparatus of the invention includes a computer includ
`ing a computer-readable medium having Software to operate
`the computer in accordance with the invention.
`The article of manufacture of the invention includes a
`computer-readable medium having Software to operate a
`computer in accordance with the invention.
`Moreover, the above objects and advantages of the inven
`tion are illustrative, and not exhaustive, of those that can be
`achieved by the invention. Thus, these and other objects and
`advantages of the invention will be apparent from the descrip
`tion herein, both as embodied herein and as modified in view
`of any variations which will be apparent to those skilled in the
`art.
`
`DEFINITIONS
`
`4
`a carrier wave used to carry computer-readable electronic
`data, Such as those used in transmitting and receiving e-mail
`or in accessing a network.
`“Software” refers to prescribed rules to operate a computer.
`Examples of software include: Software; code segments;
`instructions; computer programs; and programmed logic.
`A “computer system” refers to a system having a computer,
`where the computer comprises a computer-readable medium
`embodying Software to operate the computer.
`A "network” refers to a number of computers and associ
`ated devices that are connected by communication facilities.
`A network involves permanent connections such as cables or
`temporary connections such as those made through telephone
`or other communication links. Examples of a network
`include: an internet, Such as the Internet; an intranet; a local
`area network (LAN); a wide area network (WAN); and a
`combination of networks, such as an internet and an intranet.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Embodiments of the invention are explained in greater
`detail by way of the drawings, where the same reference
`numerals refer to the same features.
`FIG. 1 illustrates a plan view of the video surveillance
`system of the invention.
`FIG. 2 illustrates a flow diagram for the video surveillance
`system of the invention.
`FIG. 3 illustrates a flow diagram for tasking the video
`Surveillance system.
`FIG. 4 illustrates a flow diagram for operating the video
`Surveillance system.
`FIG.5 illustrates a flow diagram for extracting video primi
`tives for the video surveillance system.
`FIG. 6 illustrates a flow diagram for taking action with the
`Video Surveillance system.
`FIG. 7 illustrates a flow diagram for semi-automatic cali
`bration of the video surveillance system.
`FIG. 8 illustrates a flow diagram for automatic calibration
`of the video surveillance system.
`FIG. 9 illustrates an additional flow diagram for the video
`Surveillance system of the invention.
`FIGS. 10-15 illustrate examples of the video surveillance
`system of the invention applied to monitoring a grocery store.
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`The automatic video surveillance system of the invention is
`for monitoring a location for, for example, market research or
`security purposes. The system can be a dedicated video Sur
`veillance installation with purpose-built Surveillance compo
`nents, or the system can be a retrofit to existing video Surveil
`lance equipment that piggybacks off the Surveillance video
`feeds. The system is capable of analyzing video data from live
`Sources or from recorded media. The system can have a
`prescribed response to the analysis, such as record data, acti
`vate an alarm mechanism, or active another sensor System.
`The system is also capable of integrating with other Surveil
`lance system components. The system produces security or
`market research reports that can be tailored according to the
`needs of an operator and, as an option, can be presented
`through an interactive web-based interface, or other reporting
`mechanism.
`An operator is provided with maximum flexibility in con
`figuring the system by using event discriminators. Event dis
`criminators are identified with one or more objects (whose
`descriptions are based on video primitives), along with one or
`more optional spatial attributes, and/or one or more optional
`
`10
`
`15
`
`25
`
`30
`
`35
`
`A “video” refers to motion pictures represented in analog
`and/or digital form. Examples of video include: television,
`movies, image sequences from a video camera or other
`observer, and computer-generated image sequences.
`A“frame' refers to a particular image or other discrete unit
`within a video.
`An “object” refers to an item of interest in a video.
`Examples of an object include: a person, a vehicle, an animal,
`and a physical Subject.
`An “activity” refers to one or more actions and/or one or
`more composites of actions of one or more objects. Examples
`of an activity include: entering; exiting; stopping; moving:
`raising: lowering; growing; and shrinking.
`A "location” refers to a space where an activity may occur.
`A location can be, for example, Scene-based or image-based.
`Examples of a scene-based location include: a public space; a
`store; a retail space; an office; a warehouse; a hotel room; a
`hotel lobby; a lobby of a building; a casino; a bus station; a
`train station; an airport; a port; abus; a train; an airplane; and
`a ship. Examples of an image-based location include: a video
`image; a line in a Video image; an area in a Video image; a
`rectangular section of a video image; and a polygonal section
`of a video image.
`An "event” refers to one or more objects engaged in an
`activity. The event may be referenced with respect to a loca
`tion and/or a time.
`A “computer refers to any apparatus that is capable of
`accepting a structured input, processing the structured input
`according to prescribed rules, and producing results of the
`processing as output. Examples of a computer include: a
`computer, a general purpose computer, a Supercomputer, a
`mainframe; a Super mini-computer; a mini-computer; a work
`station; a micro-computer; a server; an interactive television;
`a hybrid combination of a computer and an interactive tele
`vision; and application-specific hardware to emulate a com
`55
`puter and/or Software. A computer can have a single processor
`or multiple processors, which can operate in parallel and/or
`not in parallel. A computer also refers to two or more com
`puters connected together via a network for transmitting or
`receiving information between the computers. An example of
`Such a computer includes a distributed computer system for
`processing information via computers linked by a network.
`A “computer-readable medium” refers to any storage
`device used for storing data accessible by a computer.
`Examples of a computer-readable medium include: a mag
`65
`netic hard disk; a floppy disk; an optical disk, Such as a
`CD-ROM and a DVD; a magnetic tape; a memory chip; and
`
`40
`
`45
`
`50
`
`60
`
`Canon Ex. 1001 Page 12 of 23
`
`

`

`US 7,932,923 B2
`
`5
`temporal attributes. For example, an operator can define an
`event discriminator (called a “loitering event in this
`example) as a “person’ object in the “automatic teller
`machine' space for “longer than 15 minutes' and “between
`10:00 p.m. and 6:00 a.m.”
`Although the video surveillance system of the invention
`draws on well-known computer vision techniques from the
`public domain, the inventive video Surveillance system has
`several unique and novel features that are not currently avail
`able. For example, current video Surveillance systems use
`large Volumes of video imagery as the primary commodity of
`information interchange. The system of the invention uses
`Video primitives as the primary commodity with representa
`tive video imagery being used as collateral evidence. The
`system of the invention can also be calibrated (manually,
`semi-automatically, or automatically) and thereafter auto
`matically can infer video primitives from video imagery. The
`system can further analyze previously processed video with
`out needing to reprocess completely the video. By analyzing
`previously processed video, the system can perform inference
`analysis based on previously recorded video primitives,
`which greatly improves the analysis speed of the computer
`system.
`As another example, the system of the invention provides
`unique system tasking. Using equipment control directives,
`current video systems allow a user to position video sensors
`and, in Some Sophisticated conventional systems, to mask out
`regions of interest or disinterest. Equipment control direc
`tives are instructions to control the position, orientation, and
`focus of video cameras. Instead of equipment control direc
`tives, the system of the invention uses event discriminators
`based on video primitives as the primary tasking mechanism.
`With event discriminators and video primitives, an operator is
`provided with a much more intuitive approach over conven
`tional systems for extracting useful information from the
`system. Rather than tasking a system with an equipment
`control directives, such as "camera A pan 45 degrees to the
`left, the system of the invention can be tasked in a human
`intuitive manner with one or more event discriminators based
`on video primitives, such as “a person enters restricted area
`A.
`Using the invention for market research, the following are
`examples of the type of video surveillance that can be per
`formed with the invention: counting people in a store; count
`ing people in a part of a store; counting people who stop in a
`particular place in a store; measuring how long people spend
`in a store; measuring how long people spend in a part of a
`store; and measuring the length of a line in a store.
`Using the invention for security, the following are
`examples of the type of video surveillance that can be per
`formed with the invention: determining when anyone enters a
`restricted area and storing associated imagery; determining
`when a person enters an area at unusual times; determining
`when changes to shelf space and storage space occur that
`might be unauthorized; determining when passengers aboard
`an aircraft approach the cockpit; determining when people
`tailgate through a secure portal; determining if there is an
`unattended bag in an airport; and determining if there is a theft
`of an asset.
`FIG. 1 illustrates a plan view of the video surveillance
`system of the invention. A computer system 11 comprises a
`computer 12 having a computer-readable medium 13
`embodying software to operate the computer 12 according to
`the invention. The computer system 11 is coupled to one or
`more video sensors 14, one or more video recorders 15, and
`one or more input/output (I/O) devices 16. The video sensors
`14 can also be optionally coupled to the video recorders 15 for
`
`40
`
`45
`
`6
`direct recording of video surveillance data. The computer
`system is optionally coupled to other sensors 17.
`The video sensors 14 provide source video to the computer
`system 11. Each video sensor 14 can be coupled to the com
`puter system 11 using, for example, a direct connection (e.g.,
`a firewire digital camera interface) or a network. The video
`sensors 14 can exist prior to installation of the invention or
`can be installed as part of the invention. Examples of a video
`sensor 14 include: a video camera; a digital video camera; a
`color camera; a monochrome camera; a camera; a camcorder,
`a PC camera; a webcam: an infra-red video camera; and a
`CCTV camera.
`The video recorders 15 receive video surveillance data
`from the computer system 11 for recording and/or provide
`source video to the computer system 11. Each video recorder
`15 can be coupled to the computer system 11 using, for
`example, a direct connection or a network. The video record
`ers 15 can exist prior to installation of the invention or can be
`installed as part of the invention. Examples of a video
`recorder 15 include: a video tape recorder; a digital video
`recorder; a video disk; a DVD; and a computer-readable
`medium.
`The I/O devices 16 provide input to and receive output
`from the computer system 11. The I/O devices 16 can be used
`to task the computer system 11 and produce reports from the
`computer system 11. Examples of I/O devices 16 include: a
`keyboard; a mouse; a stylus; a monitor; a printer; another
`computer system; a network; and an alarm.
`The other sensors 17 provide additional input to the com
`puter system 11. Each other sensor 17 can be coupled to the
`computer system 11 using, for example, a direct connection
`or a network. The other sensors 17 can exit prior to installa
`tion of the invention or can be installed as part of the inven
`tion. Examples of another sensor 17 include: a motion sensor;
`an optical tripwire; a biometric sensor, and a card-based or
`keypad-based authorization system. The outputs of the other
`sensors 17 can be recorded by the computer system 11,
`recording