I 1111111111111111 11111 1111111111 11111 1111111111 1111111111111111 IIII IIII IIII
`
`US007868912B2
`
`US 7,868,912 B2
`c12) United States Patent
`(IO) Patent No.:
`(45)Date of Patent:
`Jan. 11, 2011
`
`Venetianer et al.
`
`(54)VIDEO SURVEILLANCE SYSTEM
`
`
`EMPLOYING VIDEO PRIMITIVES
`
`(58)Field of Classification Search ................. 348/143,
`
`
`
`348/148, 150,149,166,169,170; 382/103,
`
`
`382/115; 375/240.02, 240.08;H04N 7/18
`
`
`
`See application file for complete search history.
`
`References Cited
`
`
`
`U.S. PATENT DOCUMENTS
`
`(75)Inventors: Peter L. Venetianer, McLean, VA (US);
`
`
`
`
`
`
`Alan J. Lipton, Herndon, VA (US);
`(56)
`
`
`Andrew J. Chosak, Arlington, VA (US);
`
`
`Matthew F. Frazier, Arlington, VA
`
`
`(US); Niels Haering, Reston, VA (US);
`
`
`GaryW. Myers, Ashburn, VA (US);
`
`
`Weihong Yin, Herndon, VA (US);
`
`Zhong Zhang, Herndon, VA (US)
`
`3,812,287 A 5/1974 Lemelson
`
`
`
`
`4,249,207 A 2/1981 Harman et al.
`
`
`4,257,063 A 3/1981 Loughry et al.
`
`
`4/1988 4,737,847 A Araki et al.
`
`
`4,908,704 A 3/1990 Fujioka et al.
`5,448,315 A
`9/1995 Soohoo
`
`
`5,491,511 A 2/1996 Odle
`
`
`5,515,453 A 5/1996 Hennessey et al.
`
`5,610,653 A 3/1997 Abecassis
`( *) Notice: Subject to any disclaimer, the term ofthis
`
`
`
`
`5,623,249 A 4/1997 Camire
`
`
`
`patent is extended or adjusted under 35
`
`
`5,696,503 A 12/1997 Nasburg
`
`U.S.C. 154(b) by 1612 days.
`
`
`
`(73)Assignee: ObjectVideo, Inc., Reston, VA (US)
`
`
`
`
`
`(Continued)
`
`(21)Appl. No.: 11/098,385
`
`
`
`FOREIGN PATENT DOCUMENTS
`
`
`
`(22)Filed:Apr. 5, 2005
`
`EP
`
`
`
`0293189 Bl 7 /1994
`
`(Continued)
`
`(65)
`
`
`
`Prior Publication Data
`
`OTHER PUBLICATIONS
`
`
`
`US 2005/0169367 Al Aug. 4, 2005
`
`International Search Report for International Application No. PCT/
`
`
`
`
`
`
`
`
`
`US08/09073, dated Nov. 3, 2008.
`
`
`
`
`
`Related U.S. Application Data
`
`(Continued)
`
`
`
`Primary Examiner-Tung Vo
`
`
`
`
`(63) Continuation-in-part of application No. 11/057,154,
`
`
`
`filed on Feb. 15, 2005, which is a continuation-in-part
`PLLC
`
`
`of application No. 09/987, 707, filed on Nov. 15, 2001,
`
`
`now abandoned, which is a continuation-in-part of
`(57)
`ABSTRACT
`
`
`application No. 09/694,712, filed on Oct. 24, 2000,
`A video surveillance system extracts video primitives and
`
`
`now Pat. No. 6,954,498.
`
`
`
`
`extracts event occurrences from the video primitives using
`
`
`event discriminators. The system can undertake a response,
`
`
`such as an alarm, based on extracted event occurrences.
`
`(51) Int. Cl.
`
`H04N 7118 (2006.01)
`
`(74) Attorney, Agent, or Firm-Muir Patent Consulting,
`
`
`
`
`22 Claims, 19 Drawing Sheets
`
`(52)U.S. Cl. ...................................................... 348/143
`
`14
`
`video
`sensors
`
`15
`
`video
`recorders
`
`other
`sensors
`
`17
`
`11
`
`
`
`computer system
`
`computer
`
`computer-readable
`medium
`
`12
`
`13
`
`16
`
`1/0 devices
`
`

`

`US 7,868,912 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`7,436,887 B2
`7,447,331 B2 *
`7,660,439 Bl*
`2001/0019357 Al
`2001/0033330 Al
`2001/0035907 Al
`2002/00087 58 Al
`2002/0024446 Al
`2002/0051058 Al
`2002/0082769 Al
`2002/0095490 Al
`2002/0135483 Al
`2002/0163521 Al
`2002/0191851 Al
`2003/0043160 Al
`2003/0051255 Al
`2003/0053659 Al
`2003/0085992 Al
`2003/0231769 Al*
`2004/0113933 Al
`2004/0161133 Al*
`2004/0240542 Al
`2005/0146605 Al
`2005/0157169 Al
`2005/0162515 Al
`2005/0168574 Al
`2006/0232673 Al
`2006/0279630 Al *
`2007/0002141 Al
`2007/0013776 Al
`2007/0052803 Al
`2007/0127774 Al
`2008/0100704 Al
`
`10/2008 Yeredor et al.
`11/2008 Brown et al. ................ 382/103
`2/2010 Lu et al ...................... 382/107
`9/2001 Itoetal.
`10/2001 Garoutte
`11/2001 Broemmelsiek
`1/2002 Broemmelsiek et al.
`2/2002 Grech-Cini
`5/2002 Ito et al.
`6/2002 Church et al.
`7 /2002 Barker et al.
`9/2002 Merheim et al.
`11/2002 Ellenby et al.
`12/2002 Keinan
`3/2003 Elfving et al.
`3/2003 Bulman et al.
`3/2003 Pavlidis et al.
`5/2003 Arpa et al.
`12/2003 Bolle et al .................. 380/210
`6/2004 Guler
`8/2004 Elazar et al ................. 382/115
`12/2004 Yeredor et al.
`7 /2005 Lipton et al.
`7 /2005 Brodsky et al.
`7 /2005 Venetianer et al.
`8/2005 Lipton et al.
`10/2006 Lipton et al.
`12/2006 Aggarwal et al.
`1/2007 Lipton et al.
`1/2007 Venetianer et al.
`3/2007 Chosak et al.
`6/2007 Zhang et al.
`5/2008 Venetianer et al.
`
`........... 348/143
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`EP
`EP
`EP
`EP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`WO
`WO
`WO
`WO
`
`0893823 Al
`0893923 Al
`0967584 A2
`1024666 A2
`1120746 A2
`1333682 Al
`09-247654 A
`10-048008
`10290449 A
`2000-175174
`2000-339923
`2000-224542
`2001-175868
`2001-285681
`WO 94/03014 Al
`WO 01/62005
`WO-03/044727 Al
`WO-2004/006184 A2
`
`1/1999
`1/1999
`12/1999
`8/2000
`8/2001
`8/2003
`9/1997
`2/1998
`10/1998
`6/2000
`8/2000
`11/2000
`6/2001
`10/2001
`2/1994
`8/2001
`5/2003
`1/2004
`
`OTHER PUBLICATIONS
`
`Written Opinion for International Patent Application No. PCT /US08/
`09073, dated Nov. 3, 2008.
`International Search Report issued for PCT Application No. PCT/
`US06/25196, mailed on Jan. 16, 2008.
`Written Opinion issued for PCT Application No. PCT/US06/25196,
`mailed on Jan. 16, 2008.
`Shio et al., "Segmentation of People in Motion", IEEE 1991, p.
`325-332.
`International Search Report issued in PCT Application No. PCT/
`US2006/012556, mailed on Feb. 12, 2008.
`Written Opinion issued in PCT Application No. PCT/US2006/
`012556, mailed on Feb. 12, 2008.
`Notification for IL App. No. 161777 issued February 21, 2008 and
`English translation thereof.
`CN Office Action for CN 02822772.7 on Oct. 14, 2005 in English.
`International Search Report issued for PCT Application No. PCT/
`US06/45625, mailed on Sep. 24, 2007.
`
`5,801,943 A
`5,802,361 A
`5,850,352 A *
`5,860,086 A
`5,872,865 A
`5,886,701 A
`5,912,980 A *
`5,926,210 A
`5,956,081 A
`5,959,690 A
`5,963,202 A
`5,963,203 A
`5,983,147 A
`5,987,211 A
`5,999,189 A
`6,014,461 A
`6,025,877 A *
`6,031,573 A
`6,069,653 A
`6,075,560 A
`6,088,484 A
`6,091,771 A
`6,097,429 A *
`6,123,123 A
`6,144,375 A
`6,151,413 A
`6,166,744 A
`6,177,886 Bl
`6,201,473 Bl
`6,211,907 Bl
`6,226,388 Bl
`6,297,844 Bl
`6,307,885 Bl
`6,310,916 Bl
`6,326,964 Bl
`6,351,265 Bl
`6,351,492 Bl
`6,360,234 B2 *
`6,404,455 Bl
`6,411,724 Bl
`6,424,370 Bl
`6,504,479 Bl
`6,525,658 B2
`6,542,840 B2
`6,552,826 B2
`6,570,608 Bl
`6,573,907 Bl
`6,597,800 Bl
`6,628,835 Bl
`6,646,676 Bl
`6,696,945 Bl
`6,707,852 Bl
`6,721,454 Bl*
`6,724,915 Bl
`6,727,938 Bl
`6,738,424 Bl
`6,741,977 Bl
`6,801,662 Bl
`6,816,184 Bl
`6,829,371 Bl
`6,844,818 B2
`6,865,580 Bl
`6,924,801 Bl
`6,954,498 Bl
`6,987,528 Bl
`6,987,883 B2
`7,023,469 Bl
`7,167,519 B2
`7,197,072 Bl*
`7,227,893 Bl*
`7,356,830 Bl*
`
`.............. 345/419
`
`9/1998 Nasburg
`9/1998 Wang et al.
`12/1998 Moezzi et al.
`1/1999 Crump et al.
`2/1999 Normile et al.
`3/ 1999 Chauvin et al.
`6/1999 Hunke ........................ 382/103
`7 / 1999 Hackett et al.
`9/1999 Katz et al.
`9/ 1999 Toebes, VIII et al.
`10/ 1999 Polish
`10/ 1999 Goldberg et al.
`11/1999 Krumm
`1111999 Abecassis
`12/1999 Kajiya et al.
`1/2000 Hennessey et al.
`2/2000 Chang et al ............ 375/240.01
`2/2000 Maccormack et al.
`5/2000 Hudson et al.
`6/2000 Katz
`7/2000 Mead
`7 /2000 Seeley et al.
`8/2000 Seeley et al. ................ 348/154
`9/2000 Carder et al.
`11/2000 Jain et al.
`11/2000 Jang
`12/2000 Jaszlics et al.
`1/2001 Billington et al.
`3/2001 Schaffer
`4/2001 Scaman et al.
`5/2001 Qian et al.
`10/2001 Schatz et al.
`10/2001 Moon et al.
`10/2001 Han
`12/2001 Snyder et al.
`2/2002 Bulman
`2/2002 Kim
`3/2002 Jain et al.
`6/2002 Ito et al.
`6/2002 Vaithilingam et al.
`7 /2002 Courtney
`1/2003 Lemons et al.
`2/2003 Streetman et al.
`4/2003 Okamoto et al.
`4/2003 Adler et al.
`5/2003 Tsergn
`6/2003 Madrane et al.
`7/2003 Murray et al.
`9/2003 Brill et al.
`11/2003 DaGraca et al.
`2/2004 Venetianer et al.
`3/2004 Wang
`4/2004 Qian et al.
`4/2004 Toklu et al.
`4/2004 Randall
`5/2004 Allmen et al.
`5/2004 Nagaya
`10/2004 Owechko et al.
`11/2004 Brill et al.
`12/2004 Nichani et al.
`1/2005 Grech-Cini
`3/2005 Bush
`8/2005 Dorbie
`10/2005 Lipton
`1/2006 N agahisa et al.
`1/2006 Lipton et al.
`4/2006 Olson
`1/2007 Comaniciu et al.
`3/2007 Hsu et al.
`.............. 375/240.02
`6/2007 Srinivasa et al. ....... 375/240.08
`4/2008 Dimitrova ... .. ... ... ... ... .. . 725/51
`
`................... 715/201
`
`.................. 382/224
`
`

`

`US 7,868,912 B2
`Page 3
`
`International Search Report issued for PCT Application No. PCT/
`US0l/32614 on May 6, 2002.
`International Search Report issued for PCT Application No. PCT/
`US02/22688 on Dec. 11, 2002.
`Written Opinion of the International Searching Authority issued for
`PCT Application No. PCT/US06/45625, mailed on Sep. 24, 2007.
`H. Fujiyoshi and A. J. Lipton, "Real-time Human Motion Analysis by
`Image Skeletonization, "Proceedings of IEEE WACV'98, Princeton,
`NJ, 1998, pp. 15-21.
`A. J. Lipton, H. Fujiyoshi and R. S. Patil, "Moving Target Classifi(cid:173)
`cation and Tracking from Real-time Video," Proceedings of IEEE
`WACV'98, Princeton, NJ, 1998, pp. 8-14.
`A. J. Lipton, "Local Application of Optic Flow to Analyse Rigid
`Versus Non-Rigid Motion," International Conference on Computer
`Vision, Corfu, Greece, Sep. 1999.
`R. T. Collins, Y. Tsin, J. R. Miller, and A. J. Lipton "Using a DEM to
`Determine Geospatial Object Trajectories," CMU-RI-TR-98-19,
`1998.
`A. Selinger and L. Wixson, "Classifying Moving Objects as Rigid or
`Non-Rigid Without Correspondences," Proceedings of DARPA
`Image Understanding Workshop, Nov. 1, 1998, pp. 341-347.
`Jemez Technology Corp., Variant iD Web-Site, www.variantid.com,
`printed Aug. 25, 2003.
`Alan J. Lipton "Virtual Postman-An Illustrative Example ofVirtual
`Video," International Journal of Robotics and Automation, vol. 15,
`No. 1, Jan. 2000, pp. 9-16.
`Alan J. Lipton, "Virtual Postman-Real-Time, Interactive Virtual
`Video," IASTED Conference on Computer Graphics and Imaging
`(CGIM '99), Palm Springs, Oct. 25-27, 1999.
`Robert T. Collins et al., "A System for Video Surveillance and Moni(cid:173)
`toring," Technical Report CMU-RI-TR-00-12, Robotics Institute,
`Carnegie Mellon University, May 2000.
`L. Wixson et al., "Detecting Salient Motion by Accumulating
`Directionally-Consistent Flow," IEEE, 1999.
`W.E.L. Grimson et al., "Using Adaptive Tracking to Classify and
`Monitor Activities in a Site," CVPR, pp. 22-29, Jun. 1998.
`A.J. Lipton et al., "Moving Target Classification and Tracking from
`Real-time Video," IUW, pp. 129-136, 1998.
`T.J. Olsen et al., "Moving Object Detection and Event Recognition
`Algorithm for Smart Cameras," IUW, pp. 159-175, May 1997.
`
`A. J. Lipton, "Local Application of Optical Flow to Analyse Rigid
`Versus Non-Rigid Motion," International Conference on Computer
`Vision Frame Rate Workshop, Corfu, Greece, Sep. 1999.
`F. Bartolini et al., "Counting people getting in and out of a bus by
`real-time image-sequence processing," IVC, 12(1):36-41, Jan. 1994.
`M. Rossi et al., "Tracking and counting moving people," ICIP94, pp.
`212-216, 1994.
`C.R. Wren et al., "Pfinder: Real-time tracking of the human body,"
`Vismod, 1995.
`L. Khoudour et al., "Real-Time Pedestrian Counting by Active Linear
`Cameras," JEI, 5(4):452-459, Oct. 1996.
`S. Ioffe et al., "Probabilistic Methods for Finding People," IJCV,
`43(1):45-68, Jun. 2001.
`M. Isard et al., "BraMBLe: A Bayesian Multiple-Blob Tracker,"
`ICCV, 2001.
`D.M. Gavrila, "The Visual Analysis of Human Movement: A Sur(cid:173)
`vey," CVIU, 73(1):82-98, Jan. 1999.
`N. Haering et al., "Visual Event Detection,"Video Computing Series,
`Editor Mubarak Shah, 2001.
`Collins et al., "A System for Video Surveillance and Monitoring:
`VSAM Final Report," Technical Report CMU-RI-TR-00-12, Robot(cid:173)
`ics Institute, Carnegie Mellon University, May 2000.
`J.P. Deparis et al., "A Device for Counting Passengers Making Use of
`Two Active Linear Cameras: Comparison of Algorithms," IEEE, pp.
`1629-1634, 1996.
`C.R. Wren et al.. "Pfinder: Real-Time Tracking of the Human Body,"
`PAMI, vol. 19, pp. 780-784, 1997.
`M. Allmen et al., "Long-Range Spatiotemporal Motion Under(cid:173)
`standing Using Spatiotemporal Flow Curves," Proc. IEEE CVPR,
`Lahaina, Maui, Hawaii, pp. 303-309, 1991.
`L. Wixson, "Detecting Salient Motion by Accumulating Direction(cid:173)
`ally Consistent Flow", IEEE Trans. Pattern Anal. Mach. Intell., vol.
`22, pp. 774-781, Aug. 2000.
`International Search Report and Written Opinion in PCT/US06/
`02700, Apr. 13, 2007.
`JP Office Action issued in PCT/US02/22688, along with an English
`Translation, Oct. 9, 2007.
`
`* cited by examiner
`
`

`

`system
`
`surveillance
`
`video
`operate
`
`"
`
`24
`
`..
`
`-
`
`system
`
`surveillance
`
`video
`task
`
`"
`
`23
`
`FIG. 2
`
`..
`
`-
`
`system
`
`surveillance
`
`video
`
`calibrate
`
`"
`
`22
`
`.._
`
`..
`
`1/0 devices
`
`"
`
`16
`
`~
`
`-
`
`~
`
`...
`
`FIG. 1
`
`13
`
`\
`
`'-
`l
`
`I
`
`12
`(
`
`medium
`
`computer-readable
`
`computer
`
`computer system
`
`"
`
`11
`
`~
`
`.._
`
`~
`
`-
`
`~
`
`-
`
`-
`
`...
`
`~
`
`...
`
`~
`
`...
`
`system
`
`surveillance
`
`video
`set up
`
`\
`
`21
`
`"'
`
`17
`
`sensors
`other
`
`recorders
`
`video
`
`15 '-
`
`sensors
`video
`
`14 "
`
`

`

`as appropriate
`
`response,
`undertake
`
`...
`
`-
`
`"
`
`94
`
`occurrences
`
`event
`extract
`
`'
`
`93
`
`FIG. 9
`
`...
`
`-
`
`primitives
`
`...
`~ archived video
`
`access
`
`"
`
`92
`
`system
`
`surveillance
`
`video
`task
`
`"
`
`91
`
`.... -c
`0 -.
`N
`.....
`=-
`
`rJJ
`
`('D
`('D
`
`....
`0 ....
`....
`? ....
`""' ~
`
`N
`
`~
`
`=
`
`~
`~
`~
`~
`•
`00
`~
`
`~
`
`as appropriate
`
`occurrences
`
`response,
`undertake
`
`event
`extract
`
`FIG.4
`
`primitives
`
`video
`archive
`
`primitives
`
`video
`extract
`
`video
`source
`obtain
`
`45
`
`44
`
`43
`
`42
`
`interactions
`
`identify
`
`responses
`
`identify
`
`FIG. 3
`
`attributes
`temporal
`identify
`
`areas
`spatial
`identify
`
`objects
`identify
`
`35
`
`34
`
`33
`
`32
`
`41
`
`31
`
`

`

`FIG.6
`
`output
`
`generate
`
`activity record
`
`generate
`
`as appropriate
`
`response,
`undertake
`
`63
`
`62
`
`61
`
`FIG. 5
`
`52
`
`via change
`
`L--~ detect objects .,__ ___ ___.
`
`primitives
`
`video
`identify
`
`objects
`classify
`
`57
`
`56
`
`is salient
`
`object
`
`foreground
`trajectory of
`determine if
`
`55
`
`objects
`track
`
`blobs
`
`generate
`
`via motion
`
`___ _.,. detect objects
`
`54
`
`53
`
`51
`
`

`

`typical object
`
`monitor
`
`FIG. 8
`
`areas
`
`trackable
`identify
`
`86
`
`87
`
`83
`
`via change
`
`detect objects _____ _
`
`typical objects
`typical sizes of
`
`identify
`
`88
`
`objects
`track
`
`blobs
`
`generate
`
`detect objects 1-----1"
`
`t----.r--...i via motion
`
`source video
`
`obtain
`
`85
`
`84
`
`FIG. 7
`
`82
`
`73
`
`detect objects ._ ____ .......
`
`via change
`
`81
`
`typical object
`
`sizes of
`
`identify typical
`
`77
`
`typical object
`
`monitor
`
`objects
`track
`
`blobs
`
`generate
`
`detect objects 1---.....
`
`.----..--..... via motion
`
`source video
`
`obtain
`
`76
`
`75
`
`74
`
`72
`
`71
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jan. 11,2011
`Jan. 11, 2011
`
`Sheet 5 of 19
`Sheet 5 of 19
`
`US 7,868,912 B2
`US 7,868,912 B2
`
`C:
`.Q
`C)
`~
`.S!l:
`.....
`..c
`. 5
`Cl)
`~
`0
`e!
`
`-
`-
`
`Q)
`:0
`-~
`~
`Q)
`C)
`
`~
`C:
`cu
`(.)
`
`E
`cu ... C)
`
`0 -
`-~
`E
`...
`I
`-
`(U
`C)
`.....
`0
`-~
`I
`...
`,g
`
`Cl)
`~
`0
`~
`'C
`0
`0
`C)
`
`Q) C:
`C) 0
`e! Cl)
`Q)
`...
`> Q)
`~ a.
`
`Q)
`E
`cu
`...
`C)
`~
`-~
`C:
`.Q
`Q) ...
`C)
`-~
`..c
`I-
`
`0
`..-
`0
`u:::
`
`HEX“? Hi3" AII‘IIII
`L-Xifi’.WPMQV HIII‘I II
`IJEMLJADII
`.
`
`..-
`..-
`FIG.11
`0
`u:::
`
`
`
`
` Histogram Can’tgetreliabletracksinthisregion
`
`
`
`Thisregionisfairgameforgoodtracks
`
`
`
`
`I I I
`
`
`
`I
`
`ElE.13mm‘
`
` LAE;‘
`
`
`WI"3
`WEI"
`I fill _L}|“I
`I -|
`all}
`
`
`.6 A :1VII
`
`;_§!LflEugu;,;;
`
`a IEUELQQ’IDEZM.
`
`
`21W E ”"..:n
`
`
`
`
`HL‘EL‘E:
`'
`HESS a
`
` i
`
`Axis Exhibit 1001, Page 8 of 39
`
`

`

`FIG. 14
`
`FIG. 13
`
`Av. Dwell Time: 22 Sec.
`Av. Customers: 15/hour
`
`Sodas• Hot Spot:
`
`II Time: 5 Sec.
`ustomers: 2/hour
`
`Av. Owe
`Av. C
`
`t"
`Coffee • Cold Spo .
`
`Av. Dwell Time: 3 Sec.
`Av. Customers: 4/hour
`
`Water • Cold Spot:
`
`FIG. 12
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jan. 11, 2011
`Jan. 11, 2011
`
`Sheet 7 of 19
`Sheet 7 of 19
`
`US 7,868,912 B2
`US 7,868,912 B2
`
`(Q)
`0
`-+-
`-+-
`(1) C
`~ ·-
`C
`0
`·-
`(1) 0
`0
`0
`~ Q_
`Q_
`0
`C
`·- -+-
`• • C
`. . V') 0 0
`-+-
`0
`• • C
`·-
`0
`N
`CV')
`CX)
`V')
`V') 0
`r0
`. . N N
`CV') 0
`V')
`~
`(1)
`0 N
`CX) 0
`CV') Q_
`I..()
`r--
`-+- 0
`-+- 0
`-+-
`-+-
`0
`C C -
`(1) -
`C C
`(1)
`-+-
`(1)
`(1)
`-+-
`Q_ Q_ u 0
`0.. Q_ C
`(1) z
`(/) (/) 0
`,---,
`_Q
`I
`0
`•
`
`r--
`
`”mm?m:coemm
`
`
`
`0
`-
`C
`0
`
`(/')
`~
`(1)
`a_
`•
`
`
`
`(Q)
`
`Hmm?Q:03me©286mm:EmamI
`
`V')
`
`r--
`
`r--
`
`I
`
`V')
`~
`(1)
`a_
`•
`
`090EmmmEQOI
`
`I
`
`r--
`
`r--
`
`”50%Q3.50QE8.6mg.5QOI
`
`I
`
`0902£0;:53I
`
`I
`
`C./) C./)
`
`
`
`coema0.62I
`
`LO
`
`
`
`T"" m_‘.07.
`
`Axis Exhibit 1001, Page 10 of 39
`
`

`

`U.S. Patent
`
`Jan. 11, 2011
`
`Sheet 8 of 19
`
`US 7,868,912 B2
`
`LO
`
`0 co
`
`T"""
`
`(]) Q)
`.2: C)
`:!::::: m
`E "(cid:173)
`"- .....
`·- 0
`a..
`tn
`
`0 Q)
`Q) C)
`"C ~
`·- 0 > 1n
`
`~
`:::J
`0)
`·-LL
`
`tn
`0 ·(cid:173)
`Q) ~
`:'Q co 1 - - - - - -1
`> C co
`
`o E
`~~m
`> 1n
`(
`N
`0 co
`
`T"""
`
`T"""
`0
`(C
`T"""
`
`('I')
`
`0 co
`
`T"""
`
`

`

`Figure 16b
`
`storage
`Video
`
`168
`
`response
`
`Event
`
`occurrences
`
`Event
`
`occurrences
`Extract event
`
`167
`
`166'\
`
`Responses
`
`Rules
`
`Primitives
`non-video
`Video and
`
`165
`
`161 \
`
`(1617
`
`interface
`definition
`response
`Rule and
`
`163\
`
`162
`
`

`

`.... -c
`0 -.
`0
`....
`.....
`=- ('D
`
`rJJ
`
`('D
`
`Figure 17b
`
`Figure 17a
`
`Stopped
`
`=
`
`Started
`
`=
`
`CameraMotion
`
`Camera Motion
`
`50% Red
`
`>
`
`Color
`
`Vehicle
`
`=
`
`Classification
`
`178
`
`177
`
`174
`
`173
`
`~
`
`....
`0 ....
`.... ~
`? ....
`""'
`
`N
`
`= ~
`
`~
`~
`~
`~
`•
`00
`~
`
`n
`
`OR
`
`starts/sto s"
`when camera
`uery: "Show me
`
`176
`
`n
`
`AND
`
`any red vehicle"
`Query: "Show me
`
`172
`
`175
`
`171
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jan. 11,2011
`Jan. 11, 2011
`
`Sheet 11 of 19
`Sheet 11 of 19
`
`US 7,868,912 B2
`US 7,868,912 B2
`
`o o
`
`o
`\—
`
`(DL
`~
`3
`:::J
`·-LL
`C)
`.9)
`LI.
`
`..c
`.0
`00
`CX)
`T-
`~
`(DL
`~
`3
`:::J
`C)
`.9
`LL
`LI.
`
`co
`as
`co
`CX)
`~
`T-
`~
`2
`:::J
`3
`C)
`.9)
`LL
`LI.
`
`Axis Exhibit 1001, Page 14 of 39
`
`-
`
`
`
`'
`
`@
`E
`,,©
`1%
`l@
`
`u
`3
`«var/1)
`1%
`15
`0
`
`

`

`U.S. Patent
`U.S. Patent
`
`11m1,1n.
`Jan. 11, 2011
`nJa
`
`.m2
`Sheet 12 of 19
`
`US 7,868,912 B2
`19,
`0068,7SU
`
`("')
`C)
`
`.,...
`
`m9‘23mm2-
`
`..h..”is_,sgawgggggo
`am?.w_
`9A1560
`cosmoEmmgo
`
`
`
`Em9:262m:595
`
`H9:92:9R:
`
`
`
`29:53:3390
`
`F2
`.,...
`
`C z
`
`<(
`
`N
`N2
`.,...
`0)
`
`C z 1-----1
`
`<(
`
`...
`0
`0 A
`u
`
`"C
`Q)
`c:::
`;::R
`0 0
`LO
`
`uwm$8
`
`.2029
`
`("')
`
`r---.,...
`
`Axis Exhibit 1001, Page 15 of 39
`
`
`

`

`202 D 20~3--'-----.-2-0_4 _ _,_ __ --,
`
`an illegal left turn"
`
`red vehicle that makes
`Query: "Show me any
`
`201
`
`193
`
`205
`
`2nd within 1 Os of 1st
`
`same
`
`modifier:
`Temporal
`
`modifier:
`Object
`
`Figure 20
`
`AND
`
`50% Red
`
`>
`
`Color
`
`Vehicle
`
`=
`
`Classification
`
`174
`
`173
`
`AND
`
`172
`
`AND
`
`192
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jan. 11,2011
`Jan. 11, 2011
`
`Sheet 14 of 19
`Sheet 14 of 19
`
`US 7,868,912 B2
`US 7,868,912 B2
`
`N
`0
`~
`N
`
`2102
`
`~
`0
`~
`N
`
`C"')
`0
`~
`N
`
`21062104
`
`
`
`21032101
`
`LO
`0
`~
`N
`
`2105
`
`~
`0
`~
`N
`
`co
`0
`~
`N
`
`ro
`N
`Q)
`I....
`::J
`C)
`■-
`LL
`
`
`
`~
`
`Figure21a
`
`Axis Exhibit 1001, Page 17 of 39
`
`

`

`""'"' N = N
`
`"' \C
`QC
`0-,
`"' QC
`-....l
`
`d r.,;_
`
`Figure 21 b
`
`2116
`
`2115
`
`2113
`
`2112
`
`crossed
`
`Tripwire 2106
`
`crossed
`
`Tripwire 2105
`
`crossed
`
`Tripwire 2103
`
`crossed
`
`Tripwire 2102
`
`2115 before 2116
`
`modifier:
`Temporal
`
`same
`
`modifier:
`Object
`
`AND
`
`2112 before 2113
`
`modifier:
`Temporal
`
`same
`
`modifier:
`Object
`
`AND
`
`2120
`
`2119
`
`2118
`
`2117
`
`2114
`
`2111
`
`2111 before 2114
`
`modifier:
`Temporal
`
`same
`
`modifier:
`Object
`
`AND
`
`Vehicle
`
`=
`
`Classification
`
`2121
`
`2124
`
`2123
`
`2122
`
`AND
`
`2125
`
`

`

`Alerts, ~
`
`2212
`
`Alert Console
`
`Applications
`Consoles and
`
`Back-End
`
`Pri~itives, "orensiO
`
`· Data
`
`Video
`
`Rules
`
`I
`I
`I
`I
`I
`I
`I
`I
`
`Figure 22
`•
`I
`I
`I
`I
`I
`I
`I
`I
`
`Alerts
`
`Data
`
`~orensiO
`
`CD -,
`'<
`m
`r
`en
`3
`3
`0
`()
`
`228
`
`Event Inference
`
`Rules
`
`226
`
`Video Primitives
`
`Analysis
`
`Video Content
`
`225
`
`Com ressed video
`
`Encode
`
`224
`
`Decode)
`
`(Digitization
`
`or IP
`
`Input
`Video
`
`222
`
`Video Processor
`
`In-Device Components -: 1•~ Off-Board Components
`
`

`

`Data
`
`Alerts
`
`"'"orensiO
`
`Video
`
`Primitives, 1,...
`
`2313
`
`Alert Console
`
`Alerts
`
`Event Inference
`
`2311
`
`rimitives
`Video
`
`Rules
`
`Figure 23
`
`■
`
`I
`I
`I
`I
`I
`I
`I
`
`Primitives
`
`Video
`
`CD -,
`'<
`ll)
`r
`3
`3
`0
`0
`
`C/)
`
`Data
`
`~orensiO
`
`236
`
`Primitives
`
`Video
`
`Analysis
`
`Video Content
`
`235
`
`Encode
`
`r-2-34-----, video ,-.....,..,.,.
`
`Compressed
`
`(Digitization
`
`Decode)
`
`or IP
`
`Input
`31 Video
`
`232
`
`12330S I
`Video Processor
`
`Back End
`
`239
`
`Back End Components
`
`Analysis Device Components
`
`

`

`c$
`
`~~
`
`5,ness lntell19e11
`
`• Pattern learning
`Consumer behaviour tracking
`Queue control
`
`Stock tracking
`Warehouse monitoring
`Customer theft detection
`Loading dock monitoring
`
`241 "-~
`
`Figure 24
`
`Departments
`Marketing / Operations
`
`Archive
`
`egal Department/
`
`Primitives
`
`t?>Primitives
`
`fll6J
`
`_,
`
`Crowd monitoring
`Slip-and-fall detection
`Parking lot monitoring
`
`Critical asset Protection
`Vandalism detection
`Perimeter protection
`
`Force
`Mobile Security
`
`Facility
`Remote Monitoring
`
`

`

`Set of Algorithms
`
`Analysis Block with Complete
`
`258
`
`User Interfaces
`
`Figure 25
`Inference Algorithms
`Platform with CA and
`IP Video Management
`
`256
`
`l l
`
`Alerts primitives
`
`..
`
`Video
`Analog
`
`Analog Camera
`
`254
`
`Inference Algorithms
`
`IP Camera with CA and
`
`Inference Algorithms
`Wireless PDA with
`
`Alerts
`
`255
`
`primitives l
`
`DVR with Inference Algorithms
`
`Alerts
`
`Rules
`
`253
`
`Algorithms
`
`IP Camera with CA
`
`primitives
`
`252
`
`

`

`US 7,868,912 B2
`
`1
`VIDEO SURVEILLANCE SYSTEM
`EMPLOYING VIDEO PRIMITIVES
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a continuation-in-part of U.S. patent
`application Ser. No.11/057,154, filedonFeb.15, 2005, which
`is a continuation-in-part of U.S. patent application Ser. No.
`09/987,707, filed on Nov. 15, 2001, which claims the priority
`of U.S. patent application Ser. No. 09/694,712, filed on Oct.
`24, 2000, all of which are incorporated herein by reference.
`
`BACKGROUND OF THE INVENTION
`
`Field of the Invention
`
`The invention relates to a system for automatic video sur(cid:173)
`veillance employing video primitives.
`
`REFERENCES
`
`2
`The following references describe blob analysis for trucks,
`cars, and people:
`{14} Collins, Lipton, Kanade, Fujiyoshi, Duggins, Tsin,
`Tolliver, Enomoto, and Hasegawa, "A System for Video Sur-
`5 veillance and Monitoring: VSAM Final Report," Technical
`Report CMU-RI-TR-00-12, Robotics Institute, Carnegie
`Mellon University, May 2000.
`{15} Lipton, Fujiyoshi, and Patil, "Moving Target Classi(cid:173)
`fication and Tracking from Real-time Video," 98 Darpa IUW,
`10 November 20-23, 1998.
`The following reference describes analyzing a single-per(cid:173)
`son blob and its contours:
`{16} C. R. Wren, A. Azarbayejani, T. Darrell, and A. P.
`Pentland. "Pfinder: Real-Time Tracking of the Human Body,"
`15 PAM!, vol 19, pp. 780-784, 1997.
`The following reference describes internal motion of
`blobs, including any motion-based segmentation:
`{ 17} M. Allmen and C. Dyer, "Long-Range Spatiotem-
`20 poral Motion Understanding Using Spatiotemporal Flow
`Curves," Proc. IEEE CVPR, Lahaina, Maui, Hawaii, pp. 303-
`309, 1991.
`{18} L. Wixson, "Detecting Salient Motion by Accumu(cid:173)
`lating Directionally Consistent Flow", IEEE Trans. Pattern
`25 Anal. Mach. Intel!., vol. 22, pp. 774-781, August 2000.
`
`BACKGROUND OF THE INVENTION
`
`Video surveillance of public spaces has become extremely
`widespread and accepted by the general public. Unfortu(cid:173)
`nately, conventional video surveillance systems produce such
`prodigious volumes of data that an intractable problem results
`in the analysis of video surveillance data.
`A need exists to reduce the amount of video surveillance
`35 data so analysis of the video surveillance data can be con(cid:173)
`ducted.
`A need exists to filter video surveillance data to identify
`desired portions of the video surveillance data.
`
`SUMMARY OF THE INVENTION
`
`For the convenience of the reader, the references referred to
`herein are listed below. In the specification, the numerals
`within brackets refer to respective references. The listed ref(cid:173)
`erences are incorporated herein by reference.
`The following references describe moving target detection:
`{ 1} A. Lipton, H. Fujiyoshi and R. S. Patil, "Moving Target
`Detection and Classification from Real-Time Video," Pro(cid:173)
`ceedings of IEEE WACV '98, Princeton, N.J., 1998, pp. 8-14. 30
`{ 2} W. E. L. Grimson, et al., "Using Adaptive Tracking to
`Classify and Monitor Activities in a Site", CVPR, pp. 22-29,
`June 1998.
`{3} A. J. Lipton, H. Fujiyoshi, R. S. Patil, "Moving Target
`Classification and Tracking from Real-time Video," IUW, pp.
`129-136, 1998.
`{ 4} T. J. Olson and F. Z. Brill, "Moving Object Detection
`and Event Recognition Algorithm for Smart Cameras," IUW,
`pp. 159-175, May 1997.
`The following references describe detecting and tracking 40
`humans:
`{ 5} A. J. Lipton, "Local Application of Optical Flow to
`Analyse Rigid Versus Non-Rigid Motion," International
`Conference on Computer Vision, Corfu, Greece, September
`1999.
`{ 6} F. Bartolini, V. Cappellini, and A. Mecocci, "Counting
`people getting in and out of a bus by real-time image-se(cid:173)
`quence processing," IVC, 12(1):36-41, January 1994.
`{7} M. Rossi and A. Bozzoli, "Tracking and counting
`moving people," ICIP94, pp. 212-216, 1994.
`{8} C.R. Wren, A. Azarbayejani, T. Darrell, and A. Pent(cid:173)
`land, "Pfinder: Real-time tracking of the human body," Vis(cid:173)
`mod, 1995.
`{9} L. Khoudour, L. Duvieubourg, J. P. Deparis, "Real(cid:173)
`Time Pedestrian Counting by Active Linear Cameras," JEI,
`5(4):452-459, October 1996.
`{10} S. Ioffe, D. A. Forsyth, "Probabilistic Methods for
`Finding People," IJCV 43(1 ):45-68, June 2001.
`{ 11} M. Isard and J. MacCormick, "BraMBLe: A Baye- 60
`sian Multiple-Blob Tracker," !CCV, 2001.
`The following references describe blob analysis:
`{12} D. M. Gavrila, "The Visual Analysis of Human
`Movement: A Survey," CVIU, 73(1):82-98, January 1999.
`{13} Niels Haering and Niels da Vitoria Lobo, "Visual 65
`Event Detection," Video Computing Series, Editor Mubarak
`Shah, 2001.
`
`45
`
`50
`
`An object of the invention is to reduce the amount of video
`surveillance data so analysis of the video surveillance data
`can be conducted.
`An object of the invention is to filter video surveillance data
`to identify desired portions of the video surveillance data.
`An object of the invention is to produce a real time alarm
`based on an automatic detection of an event from video sur(cid:173)
`veillance data.
`An object of the invention is to integrate data from surveil(cid:173)
`lance sensors other than video for improved searching capa(cid:173)
`bilities.
`An object of the invention is to integrate data from surveil-
`55 lance sensors other than video for improved event detection
`capabilities
`The invention includes an article of manufacture, a
`method, a system, and an apparatus for video surveillance.
`The article of manufacture of the invention includes a
`computer-readable medium comprising software for a video
`surveillance system, comprising code segments for operating
`the video surveillance system based on video primitives.
`The article of manufacture of the invention includes a
`computer-readable medium comprising software for a video
`surveillance system, comprising code segments for accessing
`archived video primitives, and code segments for extracting
`event occurrences from accessed archived video primitives.
`
`

`

`US 7,868,912 B2
`
`3
`The system of the invention includes a computer system
`including a computer-readable medium having software to
`operate a computer in accordance with the invention.
`The apparatus of the invention includes a computer includ(cid:173)
`ing a computer-readable medium having software to operate
`the computer in accordance with the invention.
`The article of manufacture of the invention includes a
`computer-readable medium having software to operate a
`computer in accordance with the invention.
`Moreover, the above objects and advantages of the inven- 10
`tion are illustrative, and not exhaustive, of those that can be
`achieved by the invention. Thus, these and other objects and
`advantages of the invention will be apparent from the descrip(cid:173)
`tion herein, both as embodied herein and as modified in view
`of any variations which will be apparent to those skilled in the
`art.
`
`4
`a carrier wave used to carry computer-readable electronic
`data, such as those used in transmitting and receiving e-mail
`or in accessing a network.
`"Software" refers to prescribed rules to operate a computer.
`5 Examples of software include: software; code segments;
`instructions; computer programs; and progranimed logic.
`A "computer system" refers to a system having a computer,
`where the computer comprises a computer-readable medium
`embodying software to operate the computer.
`A "network" refers to a number of computers and associ-
`ated devices that are connected by communication facilities.
`A network involves permanent connections such as cables or
`temporary connections such as those made through telephone
`or other communication links. Examples of a network
`15 include: an internet, such as the Internet; an intranet; a local
`area network (LAN); a wide area network (WAN); and a
`combination of networks, such as an internet and an intranet.
`
`DEFINITIONS
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`25
`
`30
`
`A "video" refers to motion pictures represented in analog 20
`and/or digital form. Examples of video include: television,
`movies, image sequences from a video camera or other
`observer, and computer-generated image sequences.
`A "frame" refers to a particular image or other discrete unit
`within a video.
`An "object" refers to an item of interest in a video.
`Examples of an object include: a person, a vehicle, an animal,
`and a physical subject.
`An "activity" refers to one or more actions and/or one or
`more composites of actions of one or more objects. Examples
`of an activity include: entering; exiting; stopping; moving;
`raising; lowering; growing; and shrinking.
`A "location" refers to a space where an activity may occur.
`A location can be, for example, scene-based or image-based.
`Examples of a scene-based location include: a public space; a
`store; a retail space; an office; a warehouse; a hotel room; a
`hotel lobby; a lobby of a building; a casino; a bus station; a
`train station; an airport; a port; a bus; a train; an airplane; and
`a ship. Examples of an image-based location include: a video 40
`image; a line in a video image; an area in a video image; a
`rectangular section of a video image; and a polygonal section
`of a video image.
`An "event" refers to one or more objects engaged in an
`activity. The event may be referenced with respect to a loca(cid:173)
`tion and/or a time.
`A "computer" refers to any apparatus that is capable of
`accepting a structured input, processing the structured input
`according to prescribed rules, and producing results of the
`processing as output. Examples of a computer include: a 50
`computer; a general purpose computer; a supercomputer; a
`mainframe; a super mini-computer; a mini-computer; a work(cid:173)
`station; a micro-computer; a server; an interactive television;
`a hybrid combination of a computer and an interactive tele(cid:173)
`vision; and application-specific hardware to emulate a com(cid:173)
`puter and/or software. A computer can have a single processor
`or multiple processors, which can operate in parallel and/or
`not in parallel. A computer also refers to two or more com(cid:173)
`puters connected together via a network for transmitting or
`receiving information between the computers. An example of
`such a computer includes a distributed computer system for
`processing information via computers linked by a network.
`A "computer-readable medium" refers to any storage
`device used for storing data accessible by a computer.
`Examples of a computer-readable medium include: a mag- 65
`netic hard disk; a floppy disk; an optical disk, such as a
`CD-ROM and a DVD; a magnetic tape; a memory chip; and
`
`Embodiments of the invention are explained in greater
`detail by way of the drawings, where the same reference
`numerals refer to the same features.
`FIG. 1 illustrates a plan view of the video surveillance
`system of the invention.
`FIG. 2 illustrates a flow diagram for the video surveillance
`system of the invention.
`FIG. 3 illustrates a flow diagram for tasking the video
`surveillance system.
`FIG. 4 illustrates a flow diagram for operating the video
`surveillance system.
`FIG. 5 illustrates a flow diagram for extracting video primi(cid:173)
`tives for the video surveillance system.
`FIG. 6 illustrates a flow diagram fo