I 1111111111111111 11111 111111111111111 IIIII IIIII IIIII 11111 111111111111111111
`US007932923B2
`
`c12) United States Patent
`Lipton et al.
`
`(IO) Patent No.:
`(45) Date of Patent:
`
`US 7,932,923 B2
`Apr. 26, 2011
`
`(54) VIDEO SURVEILLANCE SYSTEM
`EMPLOYING VIDEO PRIMITIVES
`
`(75)
`
`Inventors: Alan J. Lipton, Falls Church, VA (US);
`Thomas M. Strat, Pakton, VA (US);
`Peter L. Venetianer, McLean, VA (US);
`Mark C. Allmen, Morrison, CO (US);
`William E. Severson, Littleton, CO
`(US); Niels Haering, Arlington, VA
`(US); Andrew J. Chosak, McLean, VA
`(US); Zhong Zhang, Herndon, VA (US);
`Matthew F. Frazier, Arlington, VA
`(US); James S. Seekas, Arlington, VA
`(US); Tasuki Hirata, Silver Spring, MD
`(US); John Clark, Leesburg, VA (US)
`
`(73) Assignee: ObjectVideo, Inc., Reston, VA (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days.
`
`EP
`
`4,737,847 A
`4,908,704 A
`5,448,315 A
`5,491,511 A
`5,515,453 A
`5,610,653 A
`5,623,249 A
`5,696,503 A
`5,801,943 A
`5,802,361 A
`5,850,352 A
`5,860,086 A
`5,872,865 A
`5,886,701 A
`5,912,980 A
`5,926,210 A
`5,956,081 A
`5,959,690 A
`
`Araki et al.
`4/1988
`Fujioka et al.
`3/1990
`Soohoo
`9/1995
`Odle
`2/1996
`5/1996
`Hennessey et al.
`3/1997
`Abecassis
`4/1997
`Camire
`12/1997
`Nasburg
`9/1998
`Nasburg
`Wang eta!.
`9/1998
`Moezzi et al.
`12/1998
`1/1999
`Crump et al.
`2/1999
`Normile et al.
`3/1999
`Chauvin et al.
`6/1999
`Hunke
`7/1999
`Hackett et al.
`9/1999
`Katz et al.
`9/1999
`Toebes, VIII et al.
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`0293189 Bl
`7 /1994
`(Continued)
`
`OTHER PUBLICATIONS
`
`(21) Appl. No.: 12/569,116
`
`(22) Filed:
`
`Sep.29,2009
`
`(65)
`
`Prior Publication Data
`
`US 2010/0013926 Al
`
`Jan.21,2010
`
`(51)
`
`Int. Cl.
`H04N 7118
`(2006.01)
`(52) U.S. Cl. ....................................................... 348/143
`(58) Field of Classification Search .................. 375/143,
`375/144, 145,148; H04N 7/18
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`3,812,287 A
`5/1974 Lemelson
`4,249,207 A
`2/ 1981 Harman et al.
`4,257,063 A
`3/1981 Loughry et al.
`
`International Search Report for International Application No. PCT/
`US08/09073, dated Nov. 3, 2008.
`
`(Continued)
`
`Primary Examiner - Tung Vo
`(74) Attorney, Agent, or Firm - Muir Patent Consulting,
`PLLC
`
`(57)
`
`ABSTRACT
`
`A video surveillance system is set up, calibrated, tasked, and
`operated. The system extracts video primitives and extracts
`event occurrences from the video primitives using event dis(cid:173)
`criminators. The system can undertake a response, such as an
`alarm, based on extracted event occurrences.
`
`41 Claims, 7 Drawing Sheets
`
`15
`
`14
`
`video
`sensors
`
`video
`recorders
`
`other
`sensors
`
`17
`
`11
`
`computer system
`
`computer
`
`computer-readable
`medium
`
`12
`
`13
`
`16
`
`1/0 devices
`
`

`

`US 7,932,923 B2
`Page 2
`
`2002/0082769 Al
`2002/0095490 Al
`2002/0135483 Al
`2002/0163521 Al
`2002/0191851 Al
`2003/0043160 Al
`2003/0051255 Al
`2003/0053659 Al
`2003/0085992 Al
`2003/0231769 Al
`2004/0113933 Al
`2004/0130620 Al
`2004/0161133 Al
`2004/0240542 Al
`2005/0146605 Al
`2005/0157169 Al
`2005/0162515 Al
`2005/0168574 Al
`2005/0169367 Al
`2005/0198063 Al
`2006/0232673 Al
`2006/0279630 Al
`2007/0002141 Al
`2007/0013776 Al
`2007/0052803 Al
`2007/0127774 Al
`2008/0100704 Al
`
`6/2002 Church et al.
`7 /2002 Barker et al.
`9/2002 Merheim et al.
`11/2002 Ellenby
`12/2002 Keinan
`3/2003 Elfving et al.
`3/2003 Bulman et al.
`3/2003 Pavlidis et al.
`5/2003 Arpa et al.
`12/2003 Bolle et al.
`6/2004 Guler
`7 /2004 Buehler et al.
`8/2004 Elazar et al.
`12/2004 Yeredor et al.
`7 /2005 Lipton et al.
`7 /2005 Brodsky et al.
`7 /2005 Venetianer et al.
`8/2005 Lipton et al.
`8/2005 Venetianer et al.
`9/2005 Thomas et al.
`10/2006 Lipton et al.
`12/2006 Aggarwal et al.
`1/2007 Lipton et al.
`1/2007 Venetianer et al.
`3/2007 Chosak et al.
`6/2007 Zhang et al.
`5/2008 Venetianer et al.
`
`EP
`EP
`EP
`EP
`EP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`WO
`WO
`WO
`WO
`
`FOREIGN PATENT DOCUMENTS
`0893923 Al
`1/1999
`0967584 A2
`12/1999
`1024666 A2
`8/2000
`1120746
`8/2001
`1333682 Al
`8/2003
`2009-247654 A
`9/1997
`10-048008
`2/1998
`10-290449
`10/1998
`2000-175174
`6/2000
`2000-339923
`8/2000
`2000-224542
`11/2000
`2001-175868
`6/2001
`10/2001
`2001-285681
`2/1994
`94/03014 Al
`8/2001
`01/62005
`5/2003
`WO-03/044727 Al
`1/2004
`WO-2004/006184 A2
`
`OTHER PUBLICATIONS
`
`U.S. PATENT DOCUMENTS
`5,963,202 A
`10/1999 Polish
`5,963,203 A
`10/1999 Goldberg et al.
`5,983,147 A
`11/1999 Krumm
`5,987,211 A
`11/1999 Abecassis
`5,999,189 A
`12/1999 Kajiya et al.
`6,014,461 A
`1/2000 Hennessey et al.
`6,031,573 A
`2/2000 Maccormack et al.
`6,069,653 A
`5/2000 Hudson et al.
`6,075,560 A
`6/2000 Katz
`6,088,484 A
`7/2000 Mead
`6,091,771 A
`7/2000 Seeley et al.
`6,097,429 A
`8/2000 Seeley et al.
`6,123,123 A
`9/2000 Carder et al.
`6,144,375 A
`11/2000 Jain et al.
`6,151,413 A
`11/2000 Jang
`6,166,744 A
`12/2000 Jaszlics et al.
`6,177,886 Bl
`1/2001 Billington et al.
`6,025,877 Al
`2/2001 Chang et al.
`6,201,473 Bl
`3/2001 Schaffer
`6,211,907 Bl
`4/2001 Scaman et al.
`6,226,388 Bl
`5/2001 Qian et al.
`6,297,844 Bl
`10/2001 Schatz et al.
`6,307,885 Bl
`10/2001 Moon eta!.
`6,310,916 Bl
`10/2001 Han
`6,326,964 Bl
`12/2001 Snyder et al.
`6,351,265 Bl
`2/2002 Bulman
`6,351,492 Bl
`2/2002 Kim
`6,360,234 B2
`3/2002 Jain et al.
`6,404,455 Bl
`6/2002 Ito et al.
`6,411,724 Bl
`6/2002 Vaithilingam et al.
`6,424,370 Bl
`7/2002 Courtney
`6,504,479 Bl
`1/2003 Lemons et al.
`6,525,658 B2
`2/2003 Streetman et al.
`6,542,075 B2
`4/2003 Barker et al.
`6,542,840 B2
`4/2003 Okamoto et al.
`6,552,826 B2
`4/2003 Adler et al.
`6,570,608 Bl
`5/2003 Tserng
`6,573,907 Bl
`6/2003 Madrane
`6,597,800 Bl
`7/2003 Murray et al.
`6,628,835 Bl
`9/2003 Brill et al.
`6,646,676 Bl
`11/2003 DaGraca et al.
`6,696,945 Bl
`2/2004 Venetianer et al.
`6,707,852 Bl
`3/2004 Wang
`6,721,454 Bl*
`4/2004 Qian et al.
`Toklu et al.
`6,724,915 Bl
`4/2004
`Randall
`6,727,938 Bl
`4/2004
`Allmen et al.
`6,738,424 Bl
`5/2004
`Nagaya
`6,741,977 Bl
`5/2004
`Owechko et al.
`6,801,662 Bl
`10/2004
`Ito et al.
`6,812,835 B2
`11/2004
`Brill et al.
`6,816,184 Bl
`11/2004
`Nichani et al.
`6,829,371 Bl
`12/2004
`Grech-Cini
`6,844,818 B2
`1/2005
`Bush
`6,865,580 Bl
`3/2005
`Dorbie
`6,924,801 Bl
`8/2005
`Garoutte
`6,940,998 B2
`9/2005
`Lipton
`6,954,498 Bl
`10/2005
`6,987,528 Bl
`1/2006
`Nagahisa
`6,987,883 B2
`1/2006
`Lipton et al.
`7,023,469 Bl
`Olson
`4/2006
`Comaniciu et al.
`7,167,519 B2
`1/2007
`Hsu et al.
`7,197,072 Bl
`3/2007
`Ito et al.
`7,215,795 B2
`5/2007
`7,227,893 Bl
`6/2007
`Srinivasa et al.
`7,301,536 B2
`11/2007
`Ellenby
`7,307,652 B2
`12/2007
`Broemmelsiek
`7,356,830 Bl
`4/2008
`Dimitrova
`7,436,887 B2
`10/2008
`Yeredor et al.
`7,447,331 B2
`11/2008
`Brown eta!.
`7,479,980 B2
`1/2009
`Merheim et al.
`7,653,635 Bl*
`1/2010
`Paek et al.
`7,660,439 Bl
`2/2010
`Lu et al.
`2001/0019357 Al
`9/2001
`Ito et al.
`2001/0033330 Al
`10/2001
`Garoutte
`2001/0035907 Al
`11/2001
`Broemmelsiek
`2002/00087 58 Al
`1/2002
`Broemmelsiek et al.
`2002/0024446 Al
`2/2002
`Grech-Cini
`2002/0051058 Al
`5/2002
`Ito et al.
`
`.................... 382/224
`
`Written Opinion for International Application No. PCT/US08/
`09073, dated Nov. 3, 2008.
`A. Selinger and L. Wixson, "Classifying Moving Objects as Rigid or
`Non-Rigid Without Correspondences," Proceedings of DARPA
`Image Understanding Workshop, 1, Nov. 1998, pp. 341-347.
`Alan J. Lipton, Virtual Postman-An Illustrative Example of Virtual
`Video, International Journal of Robotics and Automation, vol. 15,
`No. 1, Jan. 2000, pp. 9-16.
`Alan J. Lipton, Virtual Postman-Real Time, Interactive Virtual
`Video, IASTED Conference on Computer Graphics and Imaging
`(CGIM '99), Palm Springs, Oct. 25-27, 1999.
`.
`Robert T. Collins et al., "A System for Video Surveillance and Mom(cid:173)
`toring," Technical Report CMU-RI-TR-00-12, Robotics Institute,
`Carnegie Mellon University, May 2000.
`Jemez Technology Corp., Variant iD Web-Site, www.variantid.com,
`printed Aug. 25, 2003.
`JP Office Action issued in PCT/US2002/22688 dated Oct. 9, 2007.
`A. J. Lipton et al., "Moving Target Classifications and Tracking from
`Real-time Video," IUW, pp. 159-175, May 1997.
`A. J. Lipton, "Local Application of Optic Flow to Analyze Rigid
`Versus Non-Rigid Motion," International Conference on Computer
`.
`Vision, Corfu, Greece, Sep. 1999.
`A. J. Lipton, H. Fujiyoshi and R. S. Patil, "Moving Target Classifi(cid:173)
`cation and Tracking for Real-time Video," Proceedmgs of IEEE
`WACV'98, Princeton, NJ, 1998, pp. 8-14.
`C.R. Wren et al., "Pfinder: Real-time Tracking of the Human Body,"
`PAMI, vol. 19, pp. 780-784, 1997.
`
`1/1
`
`

`

`US 7,932,923 B2
`Page 3
`
`C.R. Wren et al., "Pfinder: Real-time Tracking of the Human Body,"
`Vismod, 1995.
`CN Office Action for CN 02822772.7 on Oct. 14, 2005.
`D. M. Gavrila, "The Visual Analysis of Human Movement: A Sur(cid:173)
`vey," CVIU, 73(1):82-98, Jan. 1999.
`F. Bartolini et al., "Counting People Getting in and Out of a Bus by
`Real-time Image-sequence Processing," IVC, 12(1):36-41, Jan.
`1994.
`H. Fujiyoshi and A. J. Lipton, "Real-time Human Motion Analysis by
`Image Skeletonization," Proceedings ofIEEE WACV'98, Princeton,
`NJ, 1998, pp. 15-21.
`International Search Report for International Application No. PCT/
`2001/32614 on May 6, 2002.
`International Search Report for International Application No. PCT/
`2006/25196, mailed Jan. 16, 2008.
`International Search Report for International Application No. PCT/
`US06/45625, mailed on Sep. 24, 2007.
`International Search Report for International Application No. PCT/
`US2002/22688 on Dec. 11, 2002.
`International Search Report for International Application No. PCT/
`US2006/012556, mailed on Feb. 12, 2008.
`International Search Report for International Application No. PCT/
`US2006/02700 dated Apr. 13, 2007.
`J.P. Deparis et al., "A Device for Counting Passenger Making Use of
`Two Active Linear Cameras: Comparison of Algorithms," IEEE, pp.
`1629-1634, 1996.
`L. Khoudour et al., "Real-time Pedestrian Counting by Active Linear
`Cameras," JEI, 5(4):452-459, Oct. 1996.
`L. Wixson et al., "Detecting Salient Motion by Accumulating
`Directionally-Consistent Flow," IEEE 1999.
`L. Wixson et al., "Detecting Salient Motion by Accumulating
`Directionally-Consistent Flow," IEEE Trans Pattern Anal. Mach.
`Intell., vol. 22, pp. 774-781, Aug. 2000.
`
`M. Rossi et al., "Tracking and Counting Moving People," ICIP94, pp.
`212-216, 1994.
`M. Allmen et al., "Long-Range Spatiotemporal Motion Understand(cid:173)
`ing Using Spatiotemporal Flow Curves," Proc. IEEE CVPR,
`Lahaina, Maui, Hawaii, pp. 303-309, 1991.
`M. Isard et al., "BraMBLe: A Bayesian Multiple-Blob Tracker,"
`ICCV, 2001.
`N. Haering et al., "Visual Event Detection,"Video Computing Series,
`Editor Mubarak Shah, 2001.
`Notification for IL Application No. 161777 issued Feb. 21, 2008.
`R .T. Collins, Y. Tsin, J. R. Miller and A. J. Lipton, "Using a DEM to
`Determine Geospatial Object Trajectories" CMU-RI-TR-98-19,
`1998.
`S. Ioffe et al., "Probabilistic Methods for Finding People," IJCV,
`43(1):45-68, Jun. 2001.
`Shio et al., "Segmentation of People in Motion," Proc. IEEE, 1991,
`vol. 79, pp. 325-332.
`T.J. Olsen et al., "Moving Object Detection and Event Recognition
`Algorithm for Smart Cameras," IUW, pp. 159-175, May 1997.
`W.E.L. Grimson et al., "Using Adaptive Tracking to Classify and
`Monitor Activities in a Site," CVPR, pp. 22-29, Jun. 1998.
`Written Opinion for International Application No. PCT/US2006/
`25196, mailed on Jan. 16, 2008.
`Written Opinion issued in PCT Application No. PCT/US2006/
`012556, mailed on Feb. 12, 2008.
`Written Opinion of the International Searching Authority issued for
`PCT Application No. PCT/US2006/45625, mailed on Sep. 24, 2007.
`A. J. Lipton et al., "Moving Target Classifications and Tracking from
`Real-time Video," IUW, pp. 129-136, 1998.
`
`* cited by examiner
`
`

`

`N w = N
`\0
`N
`\0 w
`d r.,;_
`
`-....l
`
`0 ....
`....
`.....
`rJJ =- ('D
`
`('D
`
`-....J
`
`N
`O'I
`N
`:-:
`> "e
`
`~
`
`....
`0 ....
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`I
`
`1/0 devices
`
`\....
`16
`
`~1
`
`I I
`
`,~
`
`I
`
`I
`
`FIG.2
`
`system
`
`surveillance
`
`video
`operate
`
`system
`
`surveillance
`
`video
`task
`
`system
`
`surveillance
`
`video
`
`calibrate
`
`system
`
`surveillance
`
`video
`setup
`
`24
`
`23
`
`22
`
`21
`
`computer-readable
`
`1
`
`11 "
`
`computer
`
`►1
`I computer system
`
`1◄
`
`h
`
`recorders
`
`video
`
`15-. I
`
`sensors
`video
`
`14
`
`FIG. 1
`
`13
`
`\
`·s:
`
`medium
`
`-,
`
`7 12
`
`' 17
`
`I
`
`sensors
`other~
`
`

`

`N w = N
`\0
`N
`\0 w
`-....l
`d r.,;_
`
`as appropriate
`
`response,
`undertake
`
`94
`
`FIG. 9
`
`occurrences
`
`primitives
`
`event
`
`IJi,I archived video
`
`I
`
`system
`
`surveillance
`
`video
`task
`
`access H extract
`
`93
`
`92
`
`91
`
`0 ....
`N
`.....
`rJJ =- ('D
`
`('D
`
`-....J
`
`N
`~Cl's
`N
`:-:
`> "e
`
`....
`0 ....
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`as appropriate I
`
`response,
`
`occurrences
`
`event
`
`FIG.4
`
`primitives
`
`video
`
`45
`
`44
`
`43
`
`extract H archive H extract H undertake
`
`primitives
`
`video
`
`H
`
`42
`
`video
`source
`obtain
`
`41
`
`FIG.3
`
`I
`
`I
`
`interactions
`
`identify
`
`...
`
`r
`
`responses
`
`identify
`
`....
`
`~
`
`attributes
`temporal
`identify
`
`identify H
`
`areas
`spatial
`
`H
`
`objects
`identify
`
`35
`
`34
`
`33
`
`32
`
`31
`
`

`

`N w = N
`\0
`N
`\0 w
`d r.,;_
`
`-....l
`
`FIG. 6
`
`output
`generate
`
`activity record
`
`generate
`
`as appropriate
`
`response,
`undertake
`
`63
`
`62
`
`61
`
`-....J
`
`0 ....
`.....
`rJ'1 =(cid:173)
`
`(,H
`
`('D
`('D
`
`N
`~Cl's
`N
`:-:
`> "e
`
`....
`0 ....
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`primitives
`
`video
`identify
`
`objects
`classify
`
`57
`
`56
`
`is salient
`
`object
`
`foreground
`trajectory of
`determine if
`
`55
`
`FIG.5
`
`52
`
`____ ___,
`
`via change
`
`detect objects
`
`objects
`track
`
`blobs
`
`generate
`
`detect objects
`
`..... 1 via motion
`
`....... -
`
`-
`
`54
`
`53
`
`51
`
`

`

`N w = N
`\0
`N
`\0 w
`d r.,;_
`
`-....l
`
`-....J
`
`0 ....
`.....
`rJJ =(cid:173)
`
`.i;...
`
`('D
`('D
`
`N
`~Cl's
`N
`:-:
`> "e
`
`....
`0 ....
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`FIG.8
`
`typical object
`
`monitor
`
`areas
`
`trackable
`identify
`
`I
`
`typical objects
`typical sizes of 11111
`
`identify
`
`86
`
`87
`
`83
`
`88
`
`detect objects 1---------'
`
`via change
`
`objects
`track
`
`blobs
`
`generate
`
`►I
`
`via motion
`detect objects 1
`
`►I
`
`1
`
`source video
`
`obtain
`
`85
`
`84
`
`FIG. 7
`
`82
`
`73
`
`via change
`detect objects -------
`
`81
`
`typical object
`
`sizes of
`
`identify typical
`
`77
`
`typical object
`
`monitor
`
`objects
`track
`
`blobs
`
`generate
`
`1
`
`~
`
`1
`
`detect objects
`
`~ 1 via motion
`
`1
`
`source video
`
`obtain
`
`76
`
`75
`
`74
`
`72
`
`71
`
`

`

`U.S. Patent
`U.S. Patent
`
`Apr. 26, 2011
`Apr. 26, 2011
`
`Sheet 5 of 7
`Sheet 5 of 7
`
`US 7,932,923 B2
`US 7,932,923 B2
`
`uosied
`
`V10ld
`
`
`
`yy6iay2UuO
`
`uosJad
`
`ebeloay
`
`Q) C:
`C) 0
`~ Cl)
`Q)
`...
`~~
`
`C:
`.Q
`...
`0)
`Q)
`::.c
`.....
`C:
`
`Cl)
`
`Cl)
`~
`u
`~
`.....
`Q)
`.0
`-~ Q)
`...
`-
`
`Q)
`0)
`:--
`C:
`('lj
`()
`
`Cl)
`~
`u
`~
`"C
`0
`0
`C)
`
`-
`...
`~
`Q)
`E
`('lj
`...
`C)
`J§
`-~
`C:
`.Q
`0)
`~
`-~
`..c
`I-
`
`CJ
`0
`a::
`
`aauL
`ajdoed
`
`
`
` Ood0ldOm
`
`syed}poobJo}ewebseys|uolBessuf
`
`
`
`uolBessiy}ulsyoes}ajqeljou396},ueD
`
`
` OL “‘Sld
`
`UY}PIA
`
`0 .....
`
`Axis Exhibit 1034, Page 8 of 23
`
`
`

`

`N w = N
`\0
`N
`\0 w
`d r.,;_
`
`-....l
`
`FIG.14
`
`FIG. 13
`
`Av. Dwell Time: 22 Sec.
`Av. Customers: 15/hour
`
`Sodas -Hot Spot:
`
`0 ....
`('D a
`rJJ =(cid:173)
`
`O'I
`
`-....J
`
`Av. Dwell Time: 5 Sec.
`Av. Customers: 2/hour
`
`Coffee -Cold Spot:
`
`Av. Dwell Time: 3 Sec.
`Av. Customers: 4/hour
`
`Water -Cold Spot:
`
`Av. Dwell Time: 34 Sec.
`Av. Dwell Time: 34 Sec.
`Av. Customers: 10/hour
`Av. Customers: 10/hour
`
`Chips -Hot Spot:
`Chips -Hot Spot:
`
`""'I
`
`> "e :-:
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`....
`0 ....
`
`~
`
`N
`O'I
`N
`
`

`

`U.S. Patent
`U.S. Patent
`
`Apr. 26, 2011
`Apr. 26, 2011
`
`Sheet 7 of 7
`Sheet 7 of 7
`
`US 7,932,923 B2
`US 7,932,923 B2
`
`V')
`
`
`
`
`
`DsIoulszgjueds—-
`
`
`
`-CF01C|UOSIed
`
`r--
`
`V")
`
`
`
`
`
`jueds- -SCOLGC]UOSIed©julodjosg,
`
`r--
`
`(/) (/)
`
`0
`-+-
`-1-
`(l) C
`0 C
`·-
`!.- ·-
`~ 0
`0
`0
`0 Q
`Q
`. . C
`·- -+-
`C
`. . V') 0 0
`. . C 0
`-+-
`·-
`0
`Cf) co
`N
`~
`V") 0
`. . N N
`(")
`(V') 0
`co 0
`(l)
`0 N
`---
`(V') Q.
`L{)
`r--
`r--
`-+- 0
`-+-
`-+- 0 +-
`0
`0
`C C
`C C
`(I)
`-+-
`(I)
`(])
`(I)
`-+-
`C Q_ Q_ C Q_ Q. u 0
`(1) z
`0
`(/) (/) 0
`•--.
`~
`~
`..0
`(l)
`(I)
`0
`o_
`(L
`•
`
`SbOld
`
`49E8IqO©julod40szyusds- :LO0ZEAl
`
`I'""'"""
`
`
`
`UOSIOdDJON
`
`
`
`
`
`DSIDUlSgaQ:|jueds—
`
`•
`
`•
`
`I.(')
`T""
`
`(!)
`u.
`
`Axis Exhibit 1034, Page 10 of 23
`
`
`

`

`US 7,932,923 B2
`
`1
`VIDEO SURVEILLANCE SYSTEM
`EMPLOYING VIDEO PRIMITIVES
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application claims the priority to U.S. patent applica(cid:173)
`tion Ser. No. 09/987,707, filed Nov. 15, 2001, which claims
`priority to U.S. patent application Ser. No. 09/694,712, now
`U.S. Pat. No. 6,954,498, each of which is incorporated herein 10
`by reference in their entirety.
`
`BACKGROUND OF THE INVENTION
`
`Field of the Invention
`
`The invention relates to a system for automatic video sur(cid:173)
`veillance employing video primitives.
`
`REFERENCES
`
`15
`
`20
`
`2
`The following references describe blob analysis for trucks,
`cars, and people:
`{14} Collins, Lipton, Kanade, Fujiyoshi, Duggins, Tsin, Tol(cid:173)
`liver, Enomoto, and Hasegawa, "A System for Video Sur(cid:173)
`veillance and Monitoring: VSAM Final Report," Technical
`Report CMU-RI-TR-00-12, Robotics Institute, Carnegie
`Mellon University, May 2000.
`{15} Lipton, Fujiyoshi, and Patil, "Moving Target Classifi(cid:173)
`cation and Tracking from Real-time Video," 98 Darpa
`IUW, Nov. 20-23, 1998.
`The following reference describes analyzing a single-per(cid:173)
`son blob and its contours:
`{16} C. R. Wren, A. Azarbayejani, T. Darrell, and A. P.
`Pentland. "Pfinder: Real-Time Tracking of the Human
`Body," PAM!, vol 19, pp. 780-784, 1997.
`The following reference describes internal motion of
`blobs, including any motion-based segmentation:
`{ 17} M. Allmen and C. Dyer, "Long-Range Spatiotemporal
`Motion Understanding Using Spatiotemporal Flow
`Curves," Proc. IEEE CVPR, Lahaina, Maui, Hi., pp. 303-
`309, 1991.
`{ 18} L. Wixson, "Detecting Salient Motion by Accumulating
`Directionally Consistent Flow", IEEE Trans. Pattern Anal.
`Mach. Intel!., vol. 22, pp. 774-781, August, 2000.
`
`BACKGROUND OF THE INVENTION
`
`Video surveillance of public spaces has become extremely
`widespread and accepted by the general public. Unfortu(cid:173)
`nately, conventional video surveillance systems produce such
`prodigious volumes of data that an intractable problem results
`in the analysis of video surveillance data.
`A need exists to reduce the amount of video surveillance
`data so analysis of the video surveillance data can be con-
`ducted.
`A need exists to filter video surveillance data to identify
`desired portions of the video surveillance data.
`
`SUMMARY OF THE INVENTION
`
`35
`
`40
`
`For the convenience of the reader, the references referred to
`herein are listed below. In the specification, the numerals
`within brackets refer to respective references. The listed ref- 25
`erences are incorporated herein by reference.
`The following references describe moving target detection:
`{ 1} A. Lipton, H. Fujiyoshi and R. S. Patil, "Moving Target
`Detection and Classification from Real-Time Video," Pro(cid:173)
`ceedings of IEEE WACV '98, Princeton, N.J., 1998, pp. 30
`8-14.
`{2} W. E. L. Grimson, et al., "Using Adaptive Tracking to
`Classify and Monitor Activities in a Site", CVPR, pp.
`22-29, June 1998.
`{3} A. J. Lipton, H. Fujiyoshi, R. S. Patil, "Moving Target
`Classification and Tracking from Real-time Video," IUW,
`pp. 129-136, 1998
`{ 4} T. J. Olson and F. Z. Brill, "Moving Object Detection and
`Event Recognition Algorithm for Smart Cameras," IUW,
`pp. 159-175, May 1997.
`The following references describe detecting and tracking
`humans:
`{ 5} A. J. Lipton, "Local Application of Optical Flow to Anal(cid:173)
`yse Rigid Versus Non-Rigid Motion," International Con(cid:173)
`ference on Computer Vision, Corfu, Greece, September
`1999.
`{ 6} F. Bartolini, V. Cappellini, and A. Mecocci, "Counting
`people getting in and out of a bus by real-time image(cid:173)
`sequence processing," IVC, 12(1):36-41, January 1994.
`{7} M. Rossi andA. Bozzoli, "Tracking and counting moving
`people," ICIP94, pp. 212-216, 1994.
`{8} C.R. Wren,A.Azarbayejani, T. Darrell, andA. Pentland,
`"Pfinder: Real-time tracking of the human body," Vismod,
`1995.
`{9} L. Khoudour, L. Duvieubourg, J.P. Deparis, "Real-Time
`Pedestrian Counting by Active Linear Cameras," JEI, 5( 4):
`452-459, October 1996.
`{10} S. Ioffe, D.A. Forsyth, "Probabilistic Methods for Find(cid:173)
`ing People," IJCV, 43(1):45-68, June 2001.
`{ 11} M. Isard and J. MacCormick, "BraMBLe: A Bayesian 60
`Multiple-Blob Tracker," !CCV, 2001.
`The following references describe blob analysis:
`{12} D. M. Gavrila, "The Visual Analysis of Human Move(cid:173)
`ment: A Survey," CVIU, 73(1):82-98, January 1999.
`{13} Niels Haering and Niels da Vitoria Lobo, "Visual Event 65
`Detection," Video Computing Series, Editor Mubarak
`Shah, 2001.
`
`An object of the invention is to reduce the amount of video
`surveillance data so analysis of the video surveillance data
`45 can be conducted.
`An object of the invention is to filter video surveillance data
`to identify desired portions of the video surveillance data.
`An object of the invention is to produce a real time alarm
`based on an automatic detection of an event from video sur-
`50 veillance data.
`An object of the invention is to integrate data from surveil(cid:173)
`lance sensors other than video for improved searching capa(cid:173)
`bilities.
`An object of the invention is to integrate data from surveil-
`55 lance sensors other than video for improved event detection
`capabilities
`The invention includes an article of manufacture, a
`method, a system, and an apparatus for video surveillance.
`The article of manufacture of the invention includes a
`computer-readable medium comprising software for a video
`surveillance system, comprising code segments for operating
`the video surveillance system based on video primitives.
`The article of manufacture of the invention includes a
`computer-readable medium comprising software for a video
`surveillance system, comprising code segments for accessing
`archived video primitives, and code segments for extracting
`event occurrences from accessed archived video primitives.
`
`

`

`US 7,932,923 B2
`
`3
`The system of the invention includes a computer system
`including a computer-readable medium having software to
`operate a computer in accordance with the invention.
`The apparatus of the invention includes a computer includ(cid:173)
`ing a computer-readable medium having software to operate
`the computer in accordance with the invention.
`The article of manufacture of the invention includes a
`computer-readable medium having software to operate a
`computer in accordance with the invention.
`Moreover, the above objects and advantages of the inven- 10
`tion are illustrative, and not exhaustive, of those that can be
`achieved by the invention. Thus, these and other objects and
`advantages of the invention will be apparent from the descrip(cid:173)
`tion herein, both as embodied herein and as modified in view
`of any variations which will be apparent to those skilled in the
`art.
`
`4
`a carrier wave used to carry computer-readable electronic
`data, such as those used in transmitting and receiving e-mail
`or in accessing a network.
`"Software" refers to prescribed rules to operate a computer.
`5 Examples of software include: software; code segments;
`instructions; computer programs; and progranmied logic.
`A "computer system" refers to a system having a computer,
`where the computer comprises a computer-readable medium
`embodying software to operate the computer.
`A "network" refers to a number of computers and associ-
`ated devices that are connected by communication facilities.
`A network involves permanent connections such as cables or
`temporary connections such as those made through telephone
`or other communication links. Examples of a network
`15 include: an internet, such as the Internet; an intranet; a local
`area network (LAN); a wide area network (WAN); and a
`combination of networks, such as an internet and an intranet.
`
`DEFINITIONS
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`20
`
`A "video" refers to motion pictures represented in analog
`and/or digital form. Examples of video include: television,
`movies, image sequences from a video camera or other
`observer, and computer-generated image sequences.
`A "frame" refers to a particular image or other discrete unit 25
`within a video.
`An "object" refers to an item of interest in a video.
`Examples of an object include: a person, a vehicle, an animal,
`and a physical subject.
`An "activity" refers to one or more actions and/or one or 30
`more composites of actions of one or more objects. Examples
`of an activity include: entering; exiting; stopping; moving;
`raising; lowering; growing; and shrinking.
`A "location" refers to a space where an activity may occur.
`A location can be, for example, scene-based or image-based.
`Examples of a scene-based location include: a public space; a
`store; a retail space; an office; a warehouse; a hotel room; a
`hotel lobby; a lobby of a building; a casino; a bus station; a
`train station; an airport; a port; a bus; a train; an airplane; and
`a ship. Examples of an image-based location include: a video 40
`image; a line in a video image; an area in a video image; a
`rectangular section of a video image; and a polygonal section
`of a video image.
`An "event" refers to one or more objects engaged in an
`activity. The event may be referenced with respect to a loca- 45
`tion and/or a time.
`A "computer" refers to any apparatus that is capable of
`accepting a structured input, processing the structured input
`according to prescribed rules, and producing results of the
`processing as output. Examples of a computer include: a 50
`computer; a general purpose computer; a supercomputer; a
`mainframe; a super mini-computer; a mini-computer; a work(cid:173)
`station; a micro-computer; a server; an interactive television;
`a hybrid combination of a computer and an interactive tele(cid:173)
`vision; and application-specific hardware to emulate a com- 55
`puter and/or software. A computer can have a single processor
`or multiple processors, which can operate in parallel and/or
`not in parallel. A computer also refers to two or more com(cid:173)
`puters connected together via a network for transmitting or
`receiving information between the computers. An example of 60
`such a computer includes a distributed computer system for
`processing information via computers linked by a network.
`A "computer-readable medium" refers to any storage
`device used for storing data accessible by a computer.
`Examples of a computer-readable medium include: a mag- 65
`netic hard disk; a floppy disk; an optical disk, such as a
`CD-ROM and a DVD; a magnetic tape; a memory chip; and
`
`Embodiments of the invention are explained in greater
`detail by way of the drawings, where the same reference
`numerals refer to the same features.
`FIG. 1 illustrates a plan view of the video surveillance
`system of the invention.
`FIG. 2 illustrates a flow diagram for the video surveillance
`system of the invention.
`FIG. 3 illustrates a flow diagram for tasking the video
`surveillance system.
`FIG. 4 illustrates a flow diagram for operating the video
`surveillance system.
`FIG. 5 illustrates a flow diagram for extracting video primi(cid:173)
`tives for the video surveillance system.
`FIG. 6 illustrates a flow diagram for taking action with the
`35 video surveillance system.
`FIG. 7 illustrates a flow diagram for semi-automatic cali(cid:173)
`bration of the video surveillance system.
`FIG. 8 illustrates a flow diagram for automatic calibration
`of the video surveillance system.
`FIG. 9 illustrates an additional flow diagram for the video
`surveillance system of the invention.
`FIGS. 10-15 illustrate examples of the video surveillance
`system of the invention applied to monitoring a grocery store.
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`The automatic video surveillance system of the invention is
`for monitoring a location for, for example, market research or
`security purposes. The system can be a dedicated video sur(cid:173)
`veillance installation with purpose-built surveillance compo(cid:173)
`nents, or the system can be a retrofit to existing video surveil(cid:173)
`lance equipment that piggybacks off the surveillance video
`feeds. The system is capable ofanalyzing video data from live
`sources or from recorded media. The system can have a
`prescribed response to the analysis, such as record data, acti(cid:173)
`vate an alarm mechanism, or active another sensor system.
`The system is also capable of integrating with other surveil(cid:173)
`lance system components. The system produces security or
`market research reports that can be tailored according to the
`needs of an operator and, as an option, can be presented
`through an interactive web-based interface, or other reporting
`mechanism.
`An operator is provided with maximum flexibility in con(cid:173)
`figuring the system by using event discriminators. Event dis(cid:173)
`criminators are identified with one or more objects (whose
`descriptions are based on video primitives), along with one or
`more optional spatial attributes, and/or one or more optional
`
`

`

`US 7,932,923 B2
`
`5
`temporal attributes. For example, an operator can define an
`event discriminator ( called a "loitering" event in this
`example) as a "person" object in the "automatic teller
`machine" space for "longer than 15 minutes" and "between
`10:00 p.m. and 6:00 a.m."
`Although the video surveillance system of the invention
`draws on well-known computer vision techniques from the
`public domain, the inventive video surveillance system has
`several unique and novel features that are not currently avail(cid:173)
`able. For example, current video surveillance systems use
`large volumes of video imagery as the primary commodity of
`information interchange. The system of the invention uses
`video primitives as the primary commodity with representa(cid:173)
`tive video imagery being used as collateral evidence. The
`system of the invention can also be calibrated (manually, 15
`semi-automatically, or automatically) and thereafter auto(cid:173)
`matically can infer video primitives from video imagery. The
`system can further analyze previously processed video with(cid:173)
`out needing to reprocess completely the video. By analyzing
`previously processed video, the system can perform inference 20
`analysis based on previously recorded video primitives,
`which greatly improves the analysis speed of the computer
`system.
`As another example, the system of the invention provides
`unique system tasking. Using equipment control directives, 25
`current video systems allow a user to position video sensors
`and, in some sophisticated conventional systems, to mask out
`regions of interest or disinterest. Equipment control direc(cid:173)
`tives are instructions to control the position, orientation, and
`focus of video cameras. Instead of equipment control direc- 30
`tives, the system of the invention uses event discriminators
`based on video primitives as the primary tasking mechanism.
`With event discriminators and video primitives, an operator is
`provided with a much more intuitive approach over conven(cid:173)
`tional systems for extracting useful information from the 35
`system. Rather than tasking a system with an equipment
`control directives, such as "camera A pan 45 degrees to the
`left," the system of the invention can be tasked in a human(cid:173)
`intuitive manner with one or more event discriminators based
`on video primitives, such as "a person enters restricted area 40
`A."
`Using the invention for market research, the following are
`examples of the type of video surveillance that can be per(cid:173)
`formed with the invention: counting people in a store; count(cid:173)
`ing people in a part of a store; counting people who stop in a
`particular place in a store; measuring how long people spend
`in a store; measuring how long people spend in a part of a
`store; and measuring the length of a line in a store.
`Using the invention for security, the following are
`examples of the type of video surveillance that can be per- 50
`formed with the invention: determining when anyone enters a
`restricted area and storing associated imagery; determining
`when a person enters an area at unusual times; determining
`when changes to shelf space and storage space occur t