`US008564661B2
`
`c12) United States Patent
`Lipton et al.
`
`(IO) Patent No.:
`(45) Date of Patent:
`
`US 8,564,661 B2
`Oct. 22, 2013
`
`(54) VIDEO ANALYTIC RULE DETECTION
`SYSTEM AND METHOD
`
`(75)
`
`Inventors: Alan J. Lipton, Austin, TX (US); John
`I. W. Clark, Flamborough (CA); Zhong
`Zhang, Great Falls, VA (US); Peter L.
`Venetianer, McLean, VA (US); Thomas
`Strat, Oakton, VA (US); MarkAllmen,
`Morrison, CO (US); William Severson,
`Centennial, CO (US); Niels Haering,
`Reston, VA (US); Andrew Chosak,
`Arlington, VA (US); Matthew Frazier,
`New York, NY (US); James Sfekas,
`Seattle, WA (US); Tasuki Hirata,
`Cambridge, MA (US)
`
`(51)
`
`(2006.01)
`
`Int. Cl.
`H04N9/47
`(52) U.S. Cl.
`USPC ............................ 348/143; 382/103; 382/115
`(58) Field of Classification Search
`None
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`3,812,278 A
`3,812,287 A
`3,988,533 A
`
`5/1974 Aker
`5/1974 Lemelson
`10/1976 Mick et al.
`(Continued)
`
`(73) Assignee: ObjectVideo, Inc., Reston, VA (US)
`
`FOREIGN PATENT DOCUMENTS
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1602 days.
`
`DE
`DE
`
`2/1996
`4430016 Al
`19848490 Al
`4/2000
`(Continued)
`
`OTHER PUBLICATIONS
`
`(21) Appl. No.: 11/828,842
`
`(22) Filed:
`
`Jul. 26, 2007
`(Under 37 CFR 1.47)
`
`(65)
`
`Prior Publication Data
`
`US 2010/0026802 Al
`
`Feb. 4, 2010
`
`Related U.S. Application Data
`
`(63) Continuation-in-part of application No. 11/057,154,
`filed on Feb. 15, 2005, which is a continuation-in-part
`of application No. 09/987, 707, filed on Nov. 15, 2001,
`now abandoned, which is a continuation-in-part of
`application No. 09/694,712, filed on Oct. 24, 2000,
`now Pat. No. 6,954,498, application No. 11/828,842,
`which is a continuation-in-part of application No.
`11/167,218, filed on Jun. 28, 2005, which is a
`continuation-in-part of application No. 11/098,382,
`filedonApr. 5, 2005, now Pat. No. 7,868,912, which is
`a continuation-in-part of application No. 11/057,154.
`
`Notification of Defects for IL 161777 (original and translation), Feb.
`21, 2008.
`
`(Continued)
`
`Primary Examiner - Ario Etienne
`Assistant Examiner - Clayton R Williams
`(74) Attorney, Agent, or Firm - Muir Patent Consulting,
`PLLC
`
`ABSTRACT
`(57)
`A video surveillance system is set up, calibrated, tasked, and
`operated. The system extracts video primitives and extracts
`event occurrences from the video primitives using event dis(cid:173)
`criminators. The extracted video primitives and event occur(cid:173)
`rences may be used to create and define additional video
`analytic rules. The system can undertake a response, such as
`an alarm, based on extracted event occurrences.
`
`32 Claims, 25 Drawing Sheets
`
`14
`
`VIDEO
`SENSORS
`
`15
`
`VIDEO
`RECORDERS
`
`OTHER
`SENSORS
`
`17
`
`11
`
`COMPUTER SYSTEM
`
`COMPUTER
`
`COMPUTER-READABLE
`MEDIUM
`
`12
`
`13
`
`16
`
`1/ODEVICES
`
`
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,160,998 A
`4,198,653 A
`4,249,207 A
`4,257,063 A
`4,458,266 A
`4,593,274 A
`4,737,847 A
`4,764,971 A
`4,857,912 A
`4,908,704 A
`4,941,182 A
`4,943,854 A
`4,949,389 A
`5,048,095 A
`5,091,780 A
`5,099,322 A
`5,159,647 A
`5,161,107 A
`5,229,850 A
`5,296,852 A
`5,321,396 A
`5,448,315 A
`5,448,651 A
`5,465,115 A
`5,485,611 A
`5,491,511 A
`5,515,453 A
`5,519,789 A
`5,521,634 A
`5,526,133 A
`5,576,972 A
`5,586,200 A
`5,602,585 A
`5,610,653 A
`5,621,889 A
`5,623,249 A
`5,666,157 A
`5,671,294 A
`5,684,715 A
`5,696,503 A
`5,696,551 A
`5,708,767 A
`5,721,692 A
`5,724,456 A
`5,764,306 A
`5,768,413 A
`5,774,569 A
`5,798,787 A
`5,801,943 A
`5,802,203 A
`5,802,361 A
`5,844,603 A
`5,850,352 A
`5,860,086 A
`5,872,865 A
`5,875,304 A
`5,875,305 A
`5,886,701 A
`5,912,980 A
`5,923,365 A
`5,926,210 A
`5,937,092 A
`5,956,081 A
`5,959,690 A
`5,963,202 A
`5,963,203 A
`5,969,755 A
`5,983,147 A
`5,983,251 A
`5,987,211 A
`5,990,955 A
`5,999,189 A
`5,999,877 A
`6,008,865 A
`6,014,461 A
`
`7/1979 Kamin
`4/1980 Kamin
`2/1981 Harman et al.
`3/1981 Loughry et al.
`7/1984 Mahoney
`6/1986 Rizzo
`4/1988 Araki et al.
`8/1988 Sullivan
`8/1989 Everett, Jr. et al.
`3/1990 Fujioka et al.
`7/1990 Patel
`7/1990 Shiota et al.
`8/1990 Allebach et al.
`9/1991 Bhanu et al.
`2/1992 Pomerleau
`3/1992 Gove
`10/1992 Burt
`11/1992 Mayeaux et al.
`7/1993 Toyoshima
`3/1994 Rathi
`6/1994 Lamming et al.
`9/1995 Soohoo
`9/1995 Sakou et al.
`11/1995 Conrad et al.
`1/1996 Astle
`2/1996 Odle
`5/1996 Hennessey et al.
`5/1996 Etoh
`5/1996 McGary
`6/1996 Paff
`11/1996 Harrison
`12/1996 Devaney et al.
`2/1997 Dickinson et al.
`3/1997 Abecassis
`4/1997 Lermuzeaux et al.
`4/1997 Camire
`9/1997 Aviv
`9/1997 Rogers et al.
`11/1997 Palmer
`12/1997 Nasburg
`12/1997 Katto
`1/1998 Yeo et al.
`2/1998 Nagaya et al.
`3/1998 Boyack et al.
`6/1998 Steffano
`6/1998 Levin et al.
`6/1998 Waldenmaier
`8/1998 Yamaguchi et al.
`9/1998 Nasburg
`9/1998 Black et al.
`9/1998 Wang et al.
`12/1998 Ogata
`12/1998 Moezzi et al.
`1/1999 Crump eta!.
`2/1999 Normile et al.
`2/1999 Winter et al.
`2/1999 Winter et al.
`3/1999 Chauvin et al.
`6/1999 Hunke
`7/1999 Tamir et al.
`7/1999 Hackett et al.
`8/1999 Wootton et al.
`9/1999 Katz et al.
`9/1999 Toebes, VIII et al.
`10/1999 Polish
`10/1999 Goldberg et al.
`10/1999 Courtney
`11/1999 Krumm
`11/1999 Martens et al.
`11/1999 Abecassis
`11/1999 Koz
`12/1999 Kajiya et al.
`12/1999 Takahashi et al.
`12/1999 Fogel
`1/2000 Hennessey et al.
`
`US 8,564,661 B2
`Page 2
`
`2/2000 Chang et al.
`6,025,877 A
`6,028,626 A * 2/2000 Aviv ............................. 348/152
`6,031,573 A
`2/2000 Maccormack et al.
`6,044,166 A
`3/2000 Bassman et al.
`6,049,363 A
`4/2000 Courtney et al.
`6,058,210 A
`5/2000 de Queiroz et al.
`6,067,112 A
`5/2000 Wellner et al.
`6,069,653 A
`5/2000 Hudson
`6,069,655 A
`5/2000 Seeley et al.
`6,075,560 A
`6/2000 Katz
`6,078,619 A
`6/2000 Monro eta!.
`6,084,912 A
`7/2000 Reitmeier et al.
`6,088,484 A
`7/2000 Mead
`6,091,771 A
`7/2000 Seeley et al.
`6,091,846 A
`7/2000 Lin et al.
`6,097,429 A
`8/2000 Seeley et al.
`6,108,033 A
`8/2000 Ito et al.
`6,123,123 A
`9/2000 Carder et al.
`6,128,396 A
`10/2000 Hasegawa et al.
`6,128,398 A * 10/2000 Kuperstein et al. ........... 382/118
`6,144,375 A
`11/2000 Jain et al.
`6,151,413 A
`11/2000 Jang et al.
`6,166,744 A
`12/2000 Jaszlics et al.
`6,167,143 A
`12/2000 Badique
`6,177,886 Bl
`1/2001 Billington et al.
`6,182,022 Bl
`1/2001 Mayle et al.
`6,188,381 Bl
`2/2001 van der Wal et al.
`6,188,777 Bl
`2/2001 Darrell et al.
`6,195,458 Bl
`2/2001 Warnick et al.
`6,201,473 Bl
`3/2001 Schaffer
`6,201,476 Bl
`3/2001 Depeursinge et al.
`6,205,239 Bl
`3/2001 Lin et al.
`6,211,907 Bl
`4/2001 Scaman et al.
`6,226,388 Bl
`5/2001 Qian et al.
`6,249,613 Bl
`6/2001 Crinon et al.
`6,278,466 Bl
`8/2001 Chen
`6,297,844 Bl
`10/2001 Schatz et al.
`6,304,669 Bl
`10/2001 Kaneko et al.
`6,307,885 Bl
`10/2001 Moon et al.
`6,310,916 Bl
`10/2001 Han et al.
`6,326,964 Bl
`12/2001 Snyder et al.
`6,337,917 Bl
`1/2002 Onural et al.
`6,349,113 Bl
`2/2002 Mech eta!.
`6,351,265 Bl
`2/2002 Bulman
`6,351,492 Bl
`2/2002 Kim et al.
`6,360,234 B2
`3/2002 Jain et al.
`6,385,772 Bl
`5/2002 Courtney
`6,393,054 Bl
`5/2002 Altunbasak et al.
`6,396,535 Bl
`5/2002 Waters
`6,396,876 Bl
`5/2002 Babonneau et al.
`6,396,961 Bl
`5/2002 Wixson et al.
`6,404,455 Bl
`6/2002 Ito et al.
`6,404,925 Bl
`6/2002 Foote et al.
`6,408,293 Bl
`6/2002 Aggarwal et al.
`6,411,209 Bl
`6/2002 Lyons et al.
`6,411,724 Bl
`6/2002 Vaithilingam et al.
`6,424,370 Bl * 7/2002 Courtney ...................... 348/143
`6,437,819 Bl
`8/2002 Loveland
`6,441,734 Bl
`8/2002 Gutta et al.
`9/2002 Kuwano eta!.
`6,456,320 B2
`6,469,734 Bl
`10/2002 Nichani et al.
`6,490,370 Bl
`12/2002 Krasinski et al.
`6,504,479 Bl
`1/2003 Lemons et al.
`6,509,926 Bl
`1/2003 Mills et al.
`6,515,615 B2
`2/2003 Burchett et al.
`6,525,658 B2
`2/2003 Streetman et al.
`6,525,663 B2
`2/2003 Colmenarez et al.
`3/2003 Wildes eta!.
`6,535,620 B2
`6,539,396 Bl
`3/2003 Bowman-Amuah
`6,542,840 B2
`4/2003 Okamoto et al.
`6,545,706 Bl
`4/2003 Edwards et al.
`6,546,115 Bl
`4/2003 Ito et al.
`6,546,135 Bl
`4/2003 Lin et al.
`6,552,826 B2
`4/2003 Adler et al.
`6,570,496 B2
`5/2003 Britton
`6,570,608 Bl
`5/2003 Tseng
`6,570,610 Bl
`5/2003 Kipust
`6,573,907 Bl
`6/2003 Madrane
`6,584,211 Bl
`6/2003 Hashimoto et al.
`
`
`
`US 8,564,661 B2
`Page 3
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`6,597,800 Bl
`6,625,310 B2
`6,628,323 Bl
`6,628,835 Bl
`6,646,676 Bl
`6,658,136 Bl
`6,696,945 Bl
`6,697,103 Bl
`6,697,104 Bl
`6,698,021 Bl
`6,707,486 Bl
`6,707,852 Bl
`6,721,454 Bl
`6,724,915 Bl
`6,727,938 Bl
`6,738,424 Bl
`6,741,977 Bl
`6,754,367 Bl
`6,757,328 Bl
`6,792,319 Bl
`6,801,662 Bl
`6,807,361 Bl
`6,816,184 Bl
`6,816,186 B2
`6,829,371 Bl
`6,844,818 B2
`6,859,803 B2
`6,865,580 Bl
`6,924,801 Bl
`6,930,689 Bl
`6,940,397 Bl
`6,940,998 B2
`6,954,498 Bl
`6,954,859 Bl
`6,963,658 B2
`6,970,083 B2
`6,975,220 Bl
`6,975,346 B2
`6,985,620 B2
`6,987,451 B2
`6,987,528 Bl
`6,987,883 B2
`7,023,469 Bl
`7,035,430 B2
`7,082,209 B2
`7,110,569 B2
`7,146,286 B2
`7,167,519 B2
`7,167,575 Bl
`7,184,777 B2
`7,197,072 Bl
`7,218,756 B2
`7,224,852 B2
`7,227,893 Bl
`7,283,034 B2
`7,308,443 Bl
`7,319,479 Bl*
`7,356,830 Bl
`7,358,976 B2
`7,373,395 B2
`7,400,344 B2
`7,423,666 B2
`7,424,167 Bl
`7,436,887 B2
`7,440,589 B2
`7,447,331 B2
`7,468,662 B2
`7,487,072 B2
`7,522,745 B2
`7,595,815 B2
`7,602,413 B2
`7,613,324 B2
`7,623,677 B2
`7,633,520 B2
`7,643,653 B2
`
`7/2003 Murray et al.
`9/2003 Lipton et al.
`9/2003 Wegmann
`9/2003 Brill et al.
`11/2003 DaGraca et al.
`12/2003 Brumitt
`2/2004 Venetianer et al.
`2/2004 Fernandez et al.
`2/2004 Yakobi et al.
`2/2004 Amini et al.
`3/2004 Millet et al.
`3/2004 Wang
`4/2004 Qian et al.
`4/2004 Toklu et al.
`4/2004 Randall
`5/2004 Allmen et al.
`5/2004 Nagaya et al.
`6/2004 Ito et al.
`6/2004 Huang et al.
`9/2004 Bilger
`10/2004 Owechko et al.
`10/2004 Girgensohn et al.
`11/2004 Brill et al.
`11/2004 Luke et al.
`12/2004 Nichani et al.
`1/2005 Grech-Cini et al.
`2/2005 Dagtas et al.
`3/2005 Bush
`8/2005 Dorbie
`8/2005 Giacalone et al.
`9/2005 Le Mire
`9/2005 Garoutte
`10/2005 Lipton
`10/2005 Simerly et al.
`11/2005 Hagihara et al.
`11/2005 Venetianer et al.
`12/2005 Foodman et al.
`12/2005 Kurnhyr
`1/2006 Sawhney et al.
`1/2006 McKeown et al.
`1/2006 Nagahisa et al.
`1/2006 Lipton et al.
`4/2006 Olson
`4/2006 Ito et al.
`7/2006 Ito et al.
`9/2006 Brodsky et al.
`12/2006 Takeda et al.
`1/2007 Comaniciu et al.
`1/2007 Nichani et al.
`2/2007 Diener et al.
`3/2007 Hsu et al.
`5/2007 Garoutte
`5/2007 Lipton et al.
`6/2007 Srinivasa et al.
`10/2007 Nakamura et al.
`12/2007 Lee et al.
`1/2008 Crabtree et al.
`4/2008 Dimitrova
`4/2008 Hampshire, II et al.
`5/2008 Brailean et al.
`7/2008 Ito et al.
`9/2008 Sakakibara et al.
`9/2008 Chosak et al.
`10/2008 Yeredor et al.
`10/2008 Garoutte
`11/2008 Brown eta!.
`12/2008 Velipasalar et al.
`2/2009 Semple et al.
`4/2009 Grasso et al.
`9/2009 Donovan et al.
`10/2009 Kondo et al.
`11/2009 Venetianer et al.
`11/2009 Girgensohn et al.
`12/2009 Samarasekera et al.
`1/2010 Garoutte
`
`.............. 348/169
`
`1/2010 Garoutte
`7,650,058 Bl
`1/2010 Paek et al.
`7,653,635 Bl
`2/2010 Whelan et al.
`7,657,914 B2
`2/2010 Lu et al.
`7,660,439 Bl
`3/2010 Elazar et al.
`7,683,929 B2
`6/2010 Yin et al.
`7,733,369 B2
`6/2010 Rezvani et al.
`7,734,724 B2
`6/2010 Donovan et al.
`7,737,837 B2
`8/2010 Cofer
`7,768,549 B2
`8/2010 Arrouye et al.
`7,774,326 B2
`9/2010 Lipton et al.
`7,796,780 B2
`10/2010 Garoutte et al.
`7,822,224 B2
`11/2010 Zhang et al.
`7,825,954 B2
`11/2010 Primot et al.
`7,826,066 B2
`1/2011 Venetianer et al.
`7,868,912 B2
`2/2011 Yin et al.
`7,884,849 B2
`4/2011 Lipton et al.
`7,932,923 B2
`6/2011 Jackson et al.
`7,956,735 B2
`2001/0004739 Al
`6/2001 Sekiguchi et al.
`8/2001 Takagi et al.
`2001/0010542 Al
`2001/0019357 Al
`9/2001
`Ito et al.
`10/2001 Garoutte
`2001/0033330 Al
`2001/0035907 Al
`11/2001 Broemmelsiek
`2002/00087 58 Al
`1/2002 Broemmelsiek et al.
`2002/0024446 Al
`2/2002 Grech-Cini
`2002/0048388 Al
`4/2002 Hagihara et al.
`5/2002 Ito et al.
`2002/0051058 Al
`2002/0063154 Al
`5/2002 Hoyos et al.
`2002/0082769 Al
`6/2002 Church et al.
`7/2002 Barker et al.
`2002/0095490 Al
`2002/0135483 Al
`9/2002 Merheim et al.
`2002/0141637 Al * 10/2002 Brodsky et al. ............... 382/165
`10/2002 Nakamura
`2002/0150308 Al
`2002/0159634 Al
`10/2002 Lipton et al.
`2002/0163521 Al
`11/2002 Ellenby et al.
`2002/0163577 Al
`11/2002 Myers
`2002/0171546 Al
`11/2002 Evans et al.
`2003/0010345 Al
`1/2003 Koblasz et al.
`2003/0020808 Al
`1/2003 Luke et al.
`2003/0025599 Al
`2/2003 Monroe
`2003/0043160 Al
`3/2003 Elfving et al.
`2003/0051255 Al
`3/2003 Bulman et al.
`2003/0053659 Al
`3/2003 Pavlidis et al.
`2003/0058111 Al
`3/2003 Lee et al.
`2003/0058340 Al
`3/2003 Lin et al.
`2003/0058341 Al
`3/2003 Brodsky et al.
`2003/0085992 Al
`5/2003 Arpa et al.
`2003/0093260 Al
`5/2003 Dagtas et al.
`2003/0126622 Al
`7/2003 Cohen et al.
`2003/0163289 Al
`8/2003 Whelan et al.
`2003/0231769 Al
`12/2003 Bolle et al.
`2004/0113933 Al
`6/2004 Guler
`2004/0130620 Al *
`7/2004 Buehler et al. ................ 348/143
`2004/0137915 Al
`7/2004 Diener et al.
`2004/0143602 Al
`7/2004 Ruiz et al.
`2004/0151374 Al
`8/2004 Lipton et al.
`2004/0161133 Al
`8/2004 Elazar et al.
`2004/0225681 Al
`11/2004 Chaney et al.
`2004/0240542 Al
`12/2004 Yeredor et al.
`2004/0240546 Al
`12/2004 Wells
`2005/0002561 Al
`1/2005 Monachino et al.
`2005/0146605 Al
`7/2005 Lipton et al.
`2005/0157169 Al
`7/2005 Brodsky et al.
`2005/0162515 Al
`7/2005 Venetianer et al.
`2005/0168574 Al
`8/2005 Lipton et al.
`2005/0169367 Al
`8/2005 Venetianer et al.
`2005/0198063 Al
`9/2005 Thomas et al.
`2005/0275549 Al
`12/2005 Barclay et al.
`2006/0066722 Al
`3/2006 Yin et al.
`2006/0117356 Al
`6/2006 Jojic et al.
`2006/0200842 Al
`9/2006 Chapman et al.
`2006/0222209 Al
`10/2006 Zhang et al.
`2006/0232673 Al
`10/2006 Lipton et al.
`2006/0268111 Al
`11/2006 Zhang et al.
`2006/0279630 Al
`12/2006 Aggarwal et al.
`2007/0002141 Al
`1/2007 Lipton et al.
`2007/0013776 Al
`1/2007 Venetianer et al.
`2007 /0035623 Al
`2/2007 Garoutte et al.
`2007/0052803 Al
`3/2007 Chosak et al.
`
`
`
`US 8,564,661 B2
`Page 4
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2007/0122000 Al
`2007/0127774 Al
`2007/0182818 Al
`2008/0292140 Al
`2008/0317286 Al
`2009/0244291 Al
`2010/0020172 Al
`
`5/2007 Venetianer et al.
`6/2007 Zhang et al.
`8/2007 Buehler
`11/2008 Morris et al.
`12/2008 Thorpe et al.
`10/2009 Saptharishi et al.
`1/2010 Mariadoss
`
`FOREIGN PATENT DOCUMENTS
`
`KR
`KR
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`
`100248374
`1020000026757
`WO-9403014
`WO 98/19450
`WO 00/28498
`WO 01/33503
`WO 01/62005
`WO 03/044727
`WO 2004/006184 A2
`WO 2004/047039 Al
`WO 2008/054489 A2
`WO 2009/120616 Al
`WO 2010/076261 Al
`WO 2011/036661 Al
`
`12/1999
`5/2000
`2/1994
`5/1998
`5/2000
`5/2001
`8/2001
`5/2003
`1/2004
`6/2004
`5/2008
`10/2009
`7/2010
`3/2011
`
`DE
`DE
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`FR
`GB
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`KR
`
`10153484 Al
`10252731 Al
`0293189
`0893923
`0913799 A2
`0 967 584 A2
`0967584 A2
`1024666
`1120746
`1333682
`1916618 Al
`2944934 Al
`2337146 A
`01-244598
`03-242592
`1991-035399
`05-014891
`1993-284501
`1994-52311
`1994-76047
`07-160891
`1995-302328
`1996-171681
`1996-63599
`08-123935
`08-136251
`1996-136251
`08-272926
`09-050585
`09-130783
`09-252467
`09247654
`1997-265539
`09-330415
`Hl0-048008
`10-116359
`1998-116359
`10-283584
`Hl0-290449
`1999-15982
`11-177973
`11-203568
`1999-355759
`2000-175174
`11-009618
`11-009619
`2000-207664
`2000-207665
`2000-224542
`11-48585
`2000-253382
`11-88337
`2000-285223
`2000-287173
`2000-295600
`2000-339923
`2001-045465
`2001-145092
`2001-175868
`2001-189925
`2001-204007
`2001-266131
`2001-285681
`1999-0042072
`
`5/2003
`5/2004
`11/1988
`1/1999
`5/1999
`12/1999
`12/1999
`8/2000
`8/2001
`8/2003
`4/2008
`10/2010
`10/1999
`9/1989
`2/1990
`2/1991
`1/1993
`10/1993
`2/1994
`3/1994
`6/1995
`11/1995
`2/1996
`3/1996
`5/1996
`5/1996
`5/1996
`10/1996
`2/1997
`5/1997
`9/1997
`9/1997
`10/1997
`12/1997
`2/1998
`5/1998
`5/1998
`10/1998
`10/1998
`1/1999
`7 /1999
`7 /1999
`12/1999
`6/2000
`7/2000
`7/2000
`7/2000
`7/2000
`8/2000
`9/2000
`9/2000
`10/2000
`10/2000
`10/2000
`10/2000
`12/2000
`2/2001
`5/2001
`6/2001
`7/2001
`7/2001
`9/2001
`10/2001
`5/1998
`
`OTHER PUBLICATIONS
`
`Ivanov Yet al., "Video Surveillance oflnteractions," Visual Surveil(cid:173)
`lance, 1999. Second IEEE Workshop on, (VS '99) Fort Collins, CO,
`USA, Jun. 26, 1999, Los Alamitos CA, USA, IEEE Computer Soci(cid:173)
`ety US.
`Lipton et al.: "Objectvideo Forensics: Activity-Based Video Indexing
`and Retrieval for Physical Security Applications", IEE Intelligent
`Distributed Surveillance Systems, Feb. 23, 2004.
`Heartwell et al: "Critical Asset Protection, Perimeter Monitoring and
`Threat Detection using Automated Video Surveillance-A Technol(cid:173)
`ogy Overview with Case Studies", Proceedings 36th Annual 2002
`International Carnahan Conference on Security Technology, 2002.
`Ellen McCarthy "A Firm That Lets Guards Keep an Electronic Eye
`Open", Oct. 21, 2002.
`International Search Report for International Application No. PCT/
`US08/09073, dated Nov. 3, 2008.
`Written Opinion for International Patent Application No. PCT /US08/
`09073, dated Nov. 3, 2008.
`A. Lipton, H. Fujiyoshi and R.S. Patil, "Moving Target Classification
`and Tracking from Real-Time Video," Proceedings of IEEE WACV
`'98, Princeton, NJ, 1998, pp. 8-14.
`W.E.L. Grimson, et al., "Using Adaptive Tracking to Classify and
`Monitor Activities in a Site," CVPR, pp. 22-29, Jun. 1998.
`A.J. Lipton, H. Fujiyoshi, R.S. Patil, "Moving Target Classification
`and Tracking from Real-time Video," IUW, pp. 129-136, 1998.
`T. J. Olson and F. Z. Brill, "Moving Object Detection and Event
`Recognition Algorithm for Smart Cameras," IUW, pp. 159-17 5, May
`1997.
`A. J. Lipton, "Local Application of Optic Flow to Analyse Rigid
`Versus Non-Rigid Motion," International Conference on Computer
`Vision, Corfu, Greece, Sep. 1999.
`F. Bartolini, V. Cappellini, and A. Mecocci, "Counting people getting
`in and out of a bus by real-time image-sequence processing," IVC,
`12(1):36-41, Jan. 1994.
`M. Rossi and A. Bozzoli, "Tracking and counting moving people,"
`ICIP94,pp.212-216, 1994.
`C. R. Wren, A. Azarbayejani, T. Darrell, and A. Pentland, "Pfinder:
`Real-time tracking of the human body,"Vismod, 1995.
`L. Khoudour, L. Duvieubourg, J.P. Deparis, "Real-Time Pedestrian
`Counting by Active Linear Cameras," JEI, 5(4):452-459, Oct. 1996.
`S. Ioffe, D. A. Forsyth, "Probabilistic Methods for Finding People,"
`IJCV, 43(1):45-68, Jun. 2001.
`M. Isard and J. MacCormick, "BraMBLe: A Bayesian Multiple-Blob
`Tracker," ICCV, 2001.
`D. M. Gavrila, "The Visual Analysis of Human Movement: A Sur(cid:173)
`vey," CVIU, 73(1):82-98, Jan. 1999.
`Niels Haering and Niels da Vitoria Lobo, "Visual Event Detection,"
`Video Computing Series, Editor Mubarak Shah, 2001.
`Collins, Lipton, Kanade, Fujiyoshi, Duggins, Tsin, Tolliver,
`Enomoto, and Hasegawa, "A System for Video Surveillance and
`Monitoring: VSAM Final Report," Technical Report CMU-RI-TR-
`00-12, Robotics Institute, Carnegie Mellon University, May 2000.
`C.R. Wren, A. Azarbayejani, T. Darrell, and A. P. Pentland. "Pfinder:
`Real-Time Tracking of the Human Body," PAMI, vol. 19, pp. 780-
`784, 1997.
`M. Allmen and C. Dyer, "Long-Range Spatiotemporal Motion
`Understanding Using Spatiotemporal Flow Curves," Proc. IEEE
`CVPR, Lahaina, Maui, Hi., pp. 303-309, 1991.
`
`
`
`US 8,564,661 B2
`Page 5
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`L. Wixson, "Detecting Salient Motion by Accumulating Direction(cid:173)
`ally Consistent Flow", IEEE Trans. Pattern Anal. Mach. Intell., vol.
`22, pp. 774-781, Aug. 2000.
`JP Office Action issued for JP 2003-546290 on Oct. 9, 2007, with an
`English translation.
`Jemez Technology Corp., Variant iD Web-Site, www.variantid.com,
`printed Aug. 25, 2003.
`L. Wixson et al., "Detecting Salient Motion by Accumulating
`Directionally-Consistent Flow," IEEE, 1999.
`J.P. Deparis et al., "A Device for Counting Passengers Making Use of
`Two Active Linear Cameras: Comparison of Algorithms," IEEE, pp.
`1629-1634, 1996.
`Alan J. Lipton "Virtual Postman-An Illustrative Example ofVirtual
`Video," International Journal of Robotics and Automation, vol. 15,
`No. 1, Jan. 2000, pp. 9-16.
`Alan J. Lipton, "Virtual Postman-Real Time, Interactive Virtual
`Video," IASTED Conference on Computer Graphics and Imaging
`(CGIM'99), Palm Springs, Oct. 25-27, 1999.
`H. Fujiyoshi and A. J. Lipton, "Real-time Human Motion Analysis by
`Image Skeletonization, "Proceedings of IEEE WACV'98, Princeton,
`NJ, 1998, pp. 15-21.
`R. T. Collins, Y. Tsin, J. R. Miller, and A. J. Lipton "Using a DEM to
`Determine Geospatial Object Trajectories," CMU-RI-TR-98-19,
`1998.
`International Search Report issued for PCT Application No. PCT/
`US06/45625, mailed on Sep. 24, 2007.
`Written Opinion of the International Searching Authority issued for
`PCT Application No. PCT/US06/45625, mailed on Sep. 24, 2007.
`International Search Report and Written Opinion in PCT/US06/
`02700.
`A. Selinger and L. Wixson, "Classifying Moving Objects as Rigid
`and Non-Rigid Without Correspondences," Proceedings of DARPA
`Image Understanding Workshop, Nov. 1, 1998, pp. 341-347.
`International Search Report issued for PCT Application No. PCT/
`US06/25196, mailed Jan. 16, 2008.
`Written Opinion issued for PCT Application No. PCT /US06/25 l 96,
`mailed Jan. 16, 2008.
`Written Opinion of the International Searching Authority issued for
`PCT Application No.PCT/US06/012556, mailed on Feb. 12, 2008.
`International Search Report of the International Searching Authority
`issued for PCT Application No. PCT/US06/012556, mailed on Feb.
`12, 2008.
`CN Office Action for CN 02822772.7 on Oct. 14, 2005 in English.
`Collins, et al. "A System for Video Surveillance and Monitoring".
`Carnegie Mellon University, 2000.
`Kanade, et al. "Advances in Cooperative Multi-Sensor Video Surveil(cid:173)
`lance". Robotics Institute, Carnegie Mellon University.
`Lipton, et al. "Automated Video Protection, Monitoring & Detec(cid:173)
`tion". IEEE AESS Systems Magazine, May 2003.
`Cao, et al. "Automatic Geo-Registration for Port Surveillance". Inter(cid:173)
`national Journal of Pattern Recognition and Artificial Intelligence,
`Nov. 19, 2008.
`Haering, et al. "A Semantic Event-Detection Approach and Its Appli(cid:173)
`cation to Detecting Hunts in Wildlife Video". IEEE Transactions on
`Circuits and Systems for Video Technology, vol. 10, No. 6, Sep. 2000.
`Cao, et al. "Automatic Geo-Registration of Maritime Video Feeds".
`IEEE, 2008.
`Haering, et al. "Content-Based Access of Image and Video Librar(cid:173)
`ies". Proceedings on IEEE Workshop, pp. 15-25, Jun. 20, 1997.
`Baker, et al. "Autonomous Vehicle Video Aided Navigation-Cou(cid:173)
`pling INS and Video Approaches".
`Lipton, Alan. "Keynote: Intelligent Video as a Force Multiplier for
`Crime Detection and Prevention".
`Ali, et al. "Interactive Retrieval of Targets for Wide Area Surveil(cid:173)
`lance". MM' 10, Firenze, Italy, Oct. 2010.
`Rice, et al."Maritime Surveillance in the Intracoastal Waterway using
`Networked Underwater Acoustic Sensors integrated with a Regional
`Command Center". 2nd International Conference on Waterside
`Security, Marina di Carrara, Italy, Nov. 2010.
`
`Heartwell, et al. "Critical Asset Protection, Perimeter Monitoring and
`Threat Detection using Automated Video Surveillance-A Technol(cid:173)
`ogy Overview with Case Studies", IEEE, 2002.
`Choe, et al. "Image Transformation for Object Tracking in High(cid:173)
`Resolution Video". IEEE, 2008.
`Deng, et al. "Fast Forensic Video Event Retrieval Using Geospatial
`Computing".
`Venetianer, et al. "Performance evaluation of an intelligent video
`surveillance system-A case study". Computer Vision and Image
`Understanding 114, pp. 1292-1302, 2010.
`Hakeem, et al. "Semantic Video Search using Natural Language
`Queries" MM'09, Beijing, China, Oct. 2009.
`Rasheed, et al. "Rapidly Deployable Video Analysis Sensor Units for
`Wide Area Surveillance".
`Haering, Niels. "What Would You Pay for Automated Video Analy(cid:173)
`sis?". 2009 Advanced Video and Signal Based Surveillance. IEEE,
`2009.
`Office Action ofNov. 25, 2011 inrelatedEPOApplication No. 06 749
`276.9-1232.
`Econolite, "Auto scope Image Sensor-AIS-IV," pp. 1-2, 2011.
`Ghanem et al., Representation and Recognition of Events in Surveil(cid:173)
`lance Video Using Petri Nets, Proceedings of the 2004 IEEE Com(cid:173)
`puter Society Conference on Computer Vision and Pattern Recogni(cid:173)
`tion Workshops, pp. 1-9, 2004.
`Computer & Information Systems Abstracts, Keeping an Eye Out,
`2005.
`Computer & Information Systems Abstracts, ObjectVideo Extends
`its Capabilities for Border Applications as Part of Multi-Million
`Dollar Homeland Security Advanced Research Project Agency
`Grant, 2006.
`Computer & Information Systems Abstracts, QA: Security Grants Up
`for Grabs, 2004.
`CSA, Veni, Vidi, Vici Analytics: But Who Will Really Conquer?,
`2007.
`Dialog, Ex FCC Chief Makes Cisco Board an Even Dozen, Mar. 26,
`2007.
`Dialog, Iridium Names VP for Data Business, Jul. 9, 2001.
`Diamond Back, (2) Digest, Jan. 27, 2000.
`Elsevier Eng. Info. Inc., Threat Detection Expands Role, Jul. 20,
`2006.
`Gale/ Cengage, Airport Selects Object Video for Intrusion Detection,
`Jun. 29, 2005.
`Gale/ Cengage, Appian Names Novak Biddle and MapQuest Exec to
`Its Board, Oct. 23, 2008.
`Gale/ Cengage, Avinor Selects ESP Group and Object Video for Sur(cid:173)
`veillance at Norwegian Airports, May 5, 2008.
`Gale/ Cengage, Bosch Security Systems Utilizes ObjectVideo's
`Intelligent Video Surveillance Software, Apr. 6, 2005.
`Gale/ Cengage, Chubb Security Taps Object Video as Approved Sup(cid:173)
`plier, Jul. 8, 2005.
`Gale/ Cengage, Companies in the News Today: About Chubb Secu(cid:173)
`rity, Jul. 8, 2005.
`Gale/ Cengage, Companies in the News Today: About ObjectVideo
`Inc., Jul. 8, 2005.
`Gale/ Cengage, comScore Names Kenneth Tarpey as CFO, Apr. 26,
`2009.
`Gale/ Cengage, Cornet Offers iVDO SmartEdge Video Analytics for
`Its Ruggedized iVDO IP Encoders, May 4, 2008.
`Gale/ Cengage, Co Vi Technologies Adds Digital Video Analytics to
`Analog Video Surveillance Camera, Sep. 30, 2007.
`Gale/ Cengage, Digiop Inks OEM Partnership with ObjectVideo,
`Apr. 30, 2008.
`Gale/ Cengage, Does the eye spy? Around-the-clock video surveil(cid:173)
`lance is the Holy Grail of K-12 safety efforts. But it raises many
`questions, including whether or not the cameras are a wholly benign
`presence., Nov. 1, 2007.
`Gale/ Cengage, Fibridge's New HD, IP Camera Leverages Reference
`Design from TI and ZOT, Dec. 13, 2008.
`Gale/ Cengage, Former FCC Chairman Powell Joins ObjectVideo
`Board of Directors, Sep. 9, 2005.
`Gale/ Cengage, Intelligent IP Camera Features Onboard Analytics,
`Apr. 13, 2007.
`
`
`
`US 8,564,661 B2
`Page 6
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Gale/ Cengage, Intelligent Video Server Promotes OEM Camera
`Development, Apr. 30, 2007.
`Gale/ Cengage, Lanaccess' Encoders to Include ObjectVideo's
`Analytics, Sep. 21, 2005.
`Gale/ Cengage, MangoDSP and ObjectVideo to Provide Integrated
`Intelligent Video Surveillance Platforms, Apr. 10, 2005.
`Gale/ Cengage, MangoDSP, ObjectVideo Join Forces on Surveil(cid:173)
`lance Solutions, Apr. 7, 2005.
`Gale/ Cengage, Next Steps for Technology: How U.S. R&D Efforts
`Have Pushed Development Forward, Sep. 11, 2006.
`Gale/ Cengage, Object Video and TI Get Win with Hangzhou Hikvi(cid:173)
`sion Digital Technology, Apr. 1, 2007.
`Gale/ Cengage, ObjectVideo Develops New Scanning Camera Tech(cid:173)
`nology to Protect U.S. Borders, Jul. 19, 2006.
`Gale/ Cengage, Object Video Gets $3 .2 Million Pacts from Homeland
`Security Advanced Research Project Agency (HSARPA), May 13,
`2005.
`Gale/ Cengage, Object Video Gets DARPA Contract to Support Next
`Generation Surveillance System, Jan. 28, 2008.
`Gale/ Cengage, ObjectVideo Named 2005 "Technology Pioneer,"
`Dec. 6, 2004.
`Gale/ Cengage, Object Video Named a Red Herring 100 Company for
`Second Year in a Row, May 13, 2005.
`Gale/ Cengage, Object Video Offers Intelligent Video Network Pro(cid:173)
`tocol for Standardized Analytics Communication and Integration,
`Sep. 25, 2007.
`Gale/ Cengage, Object Video Provides Intelligent Video Software to
`Port, Jul. 13, 2005.
`Gale/ Cengage, Object Video Releases OnBoard Analytics on ARM
`Processors, Apr. 5, 2011.
`Gale/ Cengage, ObjectVideo Releases Software for Intel Architec(cid:173)
`ture Systems, Apr. 21, 2008.
`Gale/ Cengage, ObjectVideo Signs Distributor for Asia, Dec. 14,
`2005.
`Gale/ Cengage, ObjectVideo Software Part of Security System at
`Memphis/Shelby County Airport, Jun. 21, 2005.
`Gale/ Cengage, Object Video to Showcase the Latest Intelligent Solu(cid:173)
`tions at ASIS, Sep. 23, 2007.
`Gale/ Cengage, Object Video to Spotlight Intelligent Solutions at ISC
`West 2007, Mar. 26, 2007.
`Gale/ Cengage, ObjectVideo Touts Early Adoption of its Protocol,
`Mar. 31, 2008.
`Gale/ Cengage, ObjectVideo, Security Services & Technologies
`Announce Integration Partnership, Apr. 6, 2005.
`Gale/ Cengage, People in Wireless News: Peter Currie Joins Critical
`Path Board, Feb. 2, 2004.
`Gale/ Cengage, Phoenix IVS, ObjectVideo and MangoDSP to
`Launch Intelligent Video Module, Aug. 10, 2005.
`Gale/ Cengage, Proximex Upgrades Surveillint, Dec. 21, 2008.
`Gale/ Cengage, Puerto Rico Public Housing Beefs Up Security Via
`Intelligent Surveillance System from Avant Technologies and
`ObjectVideo, Sep. 19, 2007.
`Gale/ Cengage, Software Creates Intelligent Devices for Video Sys(cid:173)
`tems, Mar. 31, 2006.
`Gale/ Cengage, Software Delivers Industry-Specific Intelligent
`Video, May 10, 2006.
`Gale/ Cengage, Synectic Systems Group and ObjectVideo Ink OEM
`Pact, Feb. 28, 2008.
`Gale/ Cengage, Video-DSPs Focus on Vision Systems, May 21,
`2007.
`Gale/ Cengage, VidSys Partners with Security Companies for PSIM
`Thought Leadership Site, Oct. 23, 2008.
`Gale/ Cengage, Wireless Channel: Chubb Security to Resell
`ObjectVideo Technology, Jul. 9, 2005.
`Information Week, Curtis Joins CMGI, May 8, 2000.
`Lipton, Stationary Target Detection Using the ObjectVideo Surveil(cid:173)
`lance System, 2007.
`Lipton, The Intelligent Vision Sensor: Turning Video into Informa(cid:173)
`tion, 2007.
`
`Lipton, Video Verification of Point of Sale Transactions, 2007.
`Abstract and documents related to the contract for Department of
`Defense Award No. 81276, A Unified Framework for False Alarm
`Reduction Using Scene Context from Airborne Sensors, Identifica(cid:173)
`tion No. 07SB2-0308, W31P4Q-08-C-0108, 2007.
`Abstract and documents related to the contract for Department of
`Defense Award No. 63547, Automated Video Surveillance at Night,
`Identification No. 03ST1-004, DAAH0103CR276, 2003.
`Abstract and documents related to the contract for Department of
`Defense Award No. 82715, Context-Sensitive Content Extraction and
`Scene Understanding, Identification No. N071-085-0199, N00014-
`08-C-0639, 2007.
`Abstract and documents related to the contract for Department of
`Defense Award No. 65022, Human Activity Recognition, Identifica(cid:173)
`tion No. N031-l 723, N00014-03-M-0256, 2003.
`Abstract and documents related to the contract for Department of
`Defense Award No. 76719, Identify and Track Important Assets,
`Identification No. S052-007-0063, H92222-06-P-0012, 2005.
`Abstract and documents related to the contract for Department of
`Defense Award No. 78523, Intelligent Retrieval of Surveillance
`Imagery, Identification No. N062-147-0567, N00014-08-C0110,
`2006.
`Abstract and documents related to the contract for Department of
`Defense Award No. 59579, Mast-Mounted In-Port Video Surveil(cid:173)
`lance System, Identification No. N021-1610; N00014-03-C-0232,
`2002.
`Abstract and documents related to the contract for Department of
`Defense Award No. 83229, Robust Multiple Target Tracking, Iden(cid:173)
`tification No. A074-007-009 l, W9 l 1NF-08-C-0068, 2007.
`Abstract and documents related to the contract for National Science
`Foundation Award No. 69404, SBIR Phase I: Learning Context,
`Identification No. 0340175, 0340175, 2004.
`Abstract and documents related to the contract for Department of
`Defense Award No. 5 84 72, Streaming Multimedia for the Warfighter,
`Identification No. 99SB1-0357, DAAH01-02-C-Rl93, 2002.
`Abstract and documents related to the contract for Department of
`Defense Award No. 55489, Video Detection, Tracking and Clas