throbber
I 1111111111111111 11111 lllll lllll lllll lllll lllll lllll lllll 111111111111111111
`US008564661B2
`
`c12) United States Patent
`Lipton et al.
`
`(IO) Patent No.:
`(45) Date of Patent:
`
`US 8,564,661 B2
`Oct. 22, 2013
`
`(54) VIDEO ANALYTIC RULE DETECTION
`SYSTEM AND METHOD
`
`(75)
`
`Inventors: Alan J. Lipton, Austin, TX (US); John
`I. W. Clark, Flamborough (CA); Zhong
`Zhang, Great Falls, VA (US); Peter L.
`Venetianer, McLean, VA (US); Thomas
`Strat, Oakton, VA (US); MarkAllmen,
`Morrison, CO (US); William Severson,
`Centennial, CO (US); Niels Haering,
`Reston, VA (US); Andrew Chosak,
`Arlington, VA (US); Matthew Frazier,
`New York, NY (US); James Sfekas,
`Seattle, WA (US); Tasuki Hirata,
`Cambridge, MA (US)
`
`(51)
`
`(2006.01)
`
`Int. Cl.
`H04N9/47
`(52) U.S. Cl.
`USPC ............................ 348/143; 382/103; 382/115
`(58) Field of Classification Search
`None
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`3,812,278 A
`3,812,287 A
`3,988,533 A
`
`5/1974 Aker
`5/1974 Lemelson
`10/1976 Mick et al.
`(Continued)
`
`(73) Assignee: ObjectVideo, Inc., Reston, VA (US)
`
`FOREIGN PATENT DOCUMENTS
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1602 days.
`
`DE
`DE
`
`2/1996
`4430016 Al
`19848490 Al
`4/2000
`(Continued)
`
`OTHER PUBLICATIONS
`
`(21) Appl. No.: 11/828,842
`
`(22) Filed:
`
`Jul. 26, 2007
`(Under 37 CFR 1.47)
`
`(65)
`
`Prior Publication Data
`
`US 2010/0026802 Al
`
`Feb. 4, 2010
`
`Related U.S. Application Data
`
`(63) Continuation-in-part of application No. 11/057,154,
`filed on Feb. 15, 2005, which is a continuation-in-part
`of application No. 09/987, 707, filed on Nov. 15, 2001,
`now abandoned, which is a continuation-in-part of
`application No. 09/694,712, filed on Oct. 24, 2000,
`now Pat. No. 6,954,498, application No. 11/828,842,
`which is a continuation-in-part of application No.
`11/167,218, filed on Jun. 28, 2005, which is a
`continuation-in-part of application No. 11/098,382,
`filedonApr. 5, 2005, now Pat. No. 7,868,912, which is
`a continuation-in-part of application No. 11/057,154.
`
`Notification of Defects for IL 161777 (original and translation), Feb.
`21, 2008.
`
`(Continued)
`
`Primary Examiner - Ario Etienne
`Assistant Examiner - Clayton R Williams
`(74) Attorney, Agent, or Firm - Muir Patent Consulting,
`PLLC
`
`ABSTRACT
`(57)
`A video surveillance system is set up, calibrated, tasked, and
`operated. The system extracts video primitives and extracts
`event occurrences from the video primitives using event dis(cid:173)
`criminators. The extracted video primitives and event occur(cid:173)
`rences may be used to create and define additional video
`analytic rules. The system can undertake a response, such as
`an alarm, based on extracted event occurrences.
`
`32 Claims, 25 Drawing Sheets
`
`14
`
`VIDEO
`SENSORS
`
`15
`
`VIDEO
`RECORDERS
`
`OTHER
`SENSORS
`
`17
`
`11
`
`COMPUTER SYSTEM
`
`COMPUTER
`
`COMPUTER-READABLE
`MEDIUM
`
`12
`
`13
`
`16
`
`1/ODEVICES
`
`

`

`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,160,998 A
`4,198,653 A
`4,249,207 A
`4,257,063 A
`4,458,266 A
`4,593,274 A
`4,737,847 A
`4,764,971 A
`4,857,912 A
`4,908,704 A
`4,941,182 A
`4,943,854 A
`4,949,389 A
`5,048,095 A
`5,091,780 A
`5,099,322 A
`5,159,647 A
`5,161,107 A
`5,229,850 A
`5,296,852 A
`5,321,396 A
`5,448,315 A
`5,448,651 A
`5,465,115 A
`5,485,611 A
`5,491,511 A
`5,515,453 A
`5,519,789 A
`5,521,634 A
`5,526,133 A
`5,576,972 A
`5,586,200 A
`5,602,585 A
`5,610,653 A
`5,621,889 A
`5,623,249 A
`5,666,157 A
`5,671,294 A
`5,684,715 A
`5,696,503 A
`5,696,551 A
`5,708,767 A
`5,721,692 A
`5,724,456 A
`5,764,306 A
`5,768,413 A
`5,774,569 A
`5,798,787 A
`5,801,943 A
`5,802,203 A
`5,802,361 A
`5,844,603 A
`5,850,352 A
`5,860,086 A
`5,872,865 A
`5,875,304 A
`5,875,305 A
`5,886,701 A
`5,912,980 A
`5,923,365 A
`5,926,210 A
`5,937,092 A
`5,956,081 A
`5,959,690 A
`5,963,202 A
`5,963,203 A
`5,969,755 A
`5,983,147 A
`5,983,251 A
`5,987,211 A
`5,990,955 A
`5,999,189 A
`5,999,877 A
`6,008,865 A
`6,014,461 A
`
`7/1979 Kamin
`4/1980 Kamin
`2/1981 Harman et al.
`3/1981 Loughry et al.
`7/1984 Mahoney
`6/1986 Rizzo
`4/1988 Araki et al.
`8/1988 Sullivan
`8/1989 Everett, Jr. et al.
`3/1990 Fujioka et al.
`7/1990 Patel
`7/1990 Shiota et al.
`8/1990 Allebach et al.
`9/1991 Bhanu et al.
`2/1992 Pomerleau
`3/1992 Gove
`10/1992 Burt
`11/1992 Mayeaux et al.
`7/1993 Toyoshima
`3/1994 Rathi
`6/1994 Lamming et al.
`9/1995 Soohoo
`9/1995 Sakou et al.
`11/1995 Conrad et al.
`1/1996 Astle
`2/1996 Odle
`5/1996 Hennessey et al.
`5/1996 Etoh
`5/1996 McGary
`6/1996 Paff
`11/1996 Harrison
`12/1996 Devaney et al.
`2/1997 Dickinson et al.
`3/1997 Abecassis
`4/1997 Lermuzeaux et al.
`4/1997 Camire
`9/1997 Aviv
`9/1997 Rogers et al.
`11/1997 Palmer
`12/1997 Nasburg
`12/1997 Katto
`1/1998 Yeo et al.
`2/1998 Nagaya et al.
`3/1998 Boyack et al.
`6/1998 Steffano
`6/1998 Levin et al.
`6/1998 Waldenmaier
`8/1998 Yamaguchi et al.
`9/1998 Nasburg
`9/1998 Black et al.
`9/1998 Wang et al.
`12/1998 Ogata
`12/1998 Moezzi et al.
`1/1999 Crump eta!.
`2/1999 Normile et al.
`2/1999 Winter et al.
`2/1999 Winter et al.
`3/1999 Chauvin et al.
`6/1999 Hunke
`7/1999 Tamir et al.
`7/1999 Hackett et al.
`8/1999 Wootton et al.
`9/1999 Katz et al.
`9/1999 Toebes, VIII et al.
`10/1999 Polish
`10/1999 Goldberg et al.
`10/1999 Courtney
`11/1999 Krumm
`11/1999 Martens et al.
`11/1999 Abecassis
`11/1999 Koz
`12/1999 Kajiya et al.
`12/1999 Takahashi et al.
`12/1999 Fogel
`1/2000 Hennessey et al.
`
`US 8,564,661 B2
`Page 2
`
`2/2000 Chang et al.
`6,025,877 A
`6,028,626 A * 2/2000 Aviv ............................. 348/152
`6,031,573 A
`2/2000 Maccormack et al.
`6,044,166 A
`3/2000 Bassman et al.
`6,049,363 A
`4/2000 Courtney et al.
`6,058,210 A
`5/2000 de Queiroz et al.
`6,067,112 A
`5/2000 Wellner et al.
`6,069,653 A
`5/2000 Hudson
`6,069,655 A
`5/2000 Seeley et al.
`6,075,560 A
`6/2000 Katz
`6,078,619 A
`6/2000 Monro eta!.
`6,084,912 A
`7/2000 Reitmeier et al.
`6,088,484 A
`7/2000 Mead
`6,091,771 A
`7/2000 Seeley et al.
`6,091,846 A
`7/2000 Lin et al.
`6,097,429 A
`8/2000 Seeley et al.
`6,108,033 A
`8/2000 Ito et al.
`6,123,123 A
`9/2000 Carder et al.
`6,128,396 A
`10/2000 Hasegawa et al.
`6,128,398 A * 10/2000 Kuperstein et al. ........... 382/118
`6,144,375 A
`11/2000 Jain et al.
`6,151,413 A
`11/2000 Jang et al.
`6,166,744 A
`12/2000 Jaszlics et al.
`6,167,143 A
`12/2000 Badique
`6,177,886 Bl
`1/2001 Billington et al.
`6,182,022 Bl
`1/2001 Mayle et al.
`6,188,381 Bl
`2/2001 van der Wal et al.
`6,188,777 Bl
`2/2001 Darrell et al.
`6,195,458 Bl
`2/2001 Warnick et al.
`6,201,473 Bl
`3/2001 Schaffer
`6,201,476 Bl
`3/2001 Depeursinge et al.
`6,205,239 Bl
`3/2001 Lin et al.
`6,211,907 Bl
`4/2001 Scaman et al.
`6,226,388 Bl
`5/2001 Qian et al.
`6,249,613 Bl
`6/2001 Crinon et al.
`6,278,466 Bl
`8/2001 Chen
`6,297,844 Bl
`10/2001 Schatz et al.
`6,304,669 Bl
`10/2001 Kaneko et al.
`6,307,885 Bl
`10/2001 Moon et al.
`6,310,916 Bl
`10/2001 Han et al.
`6,326,964 Bl
`12/2001 Snyder et al.
`6,337,917 Bl
`1/2002 Onural et al.
`6,349,113 Bl
`2/2002 Mech eta!.
`6,351,265 Bl
`2/2002 Bulman
`6,351,492 Bl
`2/2002 Kim et al.
`6,360,234 B2
`3/2002 Jain et al.
`6,385,772 Bl
`5/2002 Courtney
`6,393,054 Bl
`5/2002 Altunbasak et al.
`6,396,535 Bl
`5/2002 Waters
`6,396,876 Bl
`5/2002 Babonneau et al.
`6,396,961 Bl
`5/2002 Wixson et al.
`6,404,455 Bl
`6/2002 Ito et al.
`6,404,925 Bl
`6/2002 Foote et al.
`6,408,293 Bl
`6/2002 Aggarwal et al.
`6,411,209 Bl
`6/2002 Lyons et al.
`6,411,724 Bl
`6/2002 Vaithilingam et al.
`6,424,370 Bl * 7/2002 Courtney ...................... 348/143
`6,437,819 Bl
`8/2002 Loveland
`6,441,734 Bl
`8/2002 Gutta et al.
`9/2002 Kuwano eta!.
`6,456,320 B2
`6,469,734 Bl
`10/2002 Nichani et al.
`6,490,370 Bl
`12/2002 Krasinski et al.
`6,504,479 Bl
`1/2003 Lemons et al.
`6,509,926 Bl
`1/2003 Mills et al.
`6,515,615 B2
`2/2003 Burchett et al.
`6,525,658 B2
`2/2003 Streetman et al.
`6,525,663 B2
`2/2003 Colmenarez et al.
`3/2003 Wildes eta!.
`6,535,620 B2
`6,539,396 Bl
`3/2003 Bowman-Amuah
`6,542,840 B2
`4/2003 Okamoto et al.
`6,545,706 Bl
`4/2003 Edwards et al.
`6,546,115 Bl
`4/2003 Ito et al.
`6,546,135 Bl
`4/2003 Lin et al.
`6,552,826 B2
`4/2003 Adler et al.
`6,570,496 B2
`5/2003 Britton
`6,570,608 Bl
`5/2003 Tseng
`6,570,610 Bl
`5/2003 Kipust
`6,573,907 Bl
`6/2003 Madrane
`6,584,211 Bl
`6/2003 Hashimoto et al.
`
`

`

`US 8,564,661 B2
`Page 3
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`6,597,800 Bl
`6,625,310 B2
`6,628,323 Bl
`6,628,835 Bl
`6,646,676 Bl
`6,658,136 Bl
`6,696,945 Bl
`6,697,103 Bl
`6,697,104 Bl
`6,698,021 Bl
`6,707,486 Bl
`6,707,852 Bl
`6,721,454 Bl
`6,724,915 Bl
`6,727,938 Bl
`6,738,424 Bl
`6,741,977 Bl
`6,754,367 Bl
`6,757,328 Bl
`6,792,319 Bl
`6,801,662 Bl
`6,807,361 Bl
`6,816,184 Bl
`6,816,186 B2
`6,829,371 Bl
`6,844,818 B2
`6,859,803 B2
`6,865,580 Bl
`6,924,801 Bl
`6,930,689 Bl
`6,940,397 Bl
`6,940,998 B2
`6,954,498 Bl
`6,954,859 Bl
`6,963,658 B2
`6,970,083 B2
`6,975,220 Bl
`6,975,346 B2
`6,985,620 B2
`6,987,451 B2
`6,987,528 Bl
`6,987,883 B2
`7,023,469 Bl
`7,035,430 B2
`7,082,209 B2
`7,110,569 B2
`7,146,286 B2
`7,167,519 B2
`7,167,575 Bl
`7,184,777 B2
`7,197,072 Bl
`7,218,756 B2
`7,224,852 B2
`7,227,893 Bl
`7,283,034 B2
`7,308,443 Bl
`7,319,479 Bl*
`7,356,830 Bl
`7,358,976 B2
`7,373,395 B2
`7,400,344 B2
`7,423,666 B2
`7,424,167 Bl
`7,436,887 B2
`7,440,589 B2
`7,447,331 B2
`7,468,662 B2
`7,487,072 B2
`7,522,745 B2
`7,595,815 B2
`7,602,413 B2
`7,613,324 B2
`7,623,677 B2
`7,633,520 B2
`7,643,653 B2
`
`7/2003 Murray et al.
`9/2003 Lipton et al.
`9/2003 Wegmann
`9/2003 Brill et al.
`11/2003 DaGraca et al.
`12/2003 Brumitt
`2/2004 Venetianer et al.
`2/2004 Fernandez et al.
`2/2004 Yakobi et al.
`2/2004 Amini et al.
`3/2004 Millet et al.
`3/2004 Wang
`4/2004 Qian et al.
`4/2004 Toklu et al.
`4/2004 Randall
`5/2004 Allmen et al.
`5/2004 Nagaya et al.
`6/2004 Ito et al.
`6/2004 Huang et al.
`9/2004 Bilger
`10/2004 Owechko et al.
`10/2004 Girgensohn et al.
`11/2004 Brill et al.
`11/2004 Luke et al.
`12/2004 Nichani et al.
`1/2005 Grech-Cini et al.
`2/2005 Dagtas et al.
`3/2005 Bush
`8/2005 Dorbie
`8/2005 Giacalone et al.
`9/2005 Le Mire
`9/2005 Garoutte
`10/2005 Lipton
`10/2005 Simerly et al.
`11/2005 Hagihara et al.
`11/2005 Venetianer et al.
`12/2005 Foodman et al.
`12/2005 Kurnhyr
`1/2006 Sawhney et al.
`1/2006 McKeown et al.
`1/2006 Nagahisa et al.
`1/2006 Lipton et al.
`4/2006 Olson
`4/2006 Ito et al.
`7/2006 Ito et al.
`9/2006 Brodsky et al.
`12/2006 Takeda et al.
`1/2007 Comaniciu et al.
`1/2007 Nichani et al.
`2/2007 Diener et al.
`3/2007 Hsu et al.
`5/2007 Garoutte
`5/2007 Lipton et al.
`6/2007 Srinivasa et al.
`10/2007 Nakamura et al.
`12/2007 Lee et al.
`1/2008 Crabtree et al.
`4/2008 Dimitrova
`4/2008 Hampshire, II et al.
`5/2008 Brailean et al.
`7/2008 Ito et al.
`9/2008 Sakakibara et al.
`9/2008 Chosak et al.
`10/2008 Yeredor et al.
`10/2008 Garoutte
`11/2008 Brown eta!.
`12/2008 Velipasalar et al.
`2/2009 Semple et al.
`4/2009 Grasso et al.
`9/2009 Donovan et al.
`10/2009 Kondo et al.
`11/2009 Venetianer et al.
`11/2009 Girgensohn et al.
`12/2009 Samarasekera et al.
`1/2010 Garoutte
`
`.............. 348/169
`
`1/2010 Garoutte
`7,650,058 Bl
`1/2010 Paek et al.
`7,653,635 Bl
`2/2010 Whelan et al.
`7,657,914 B2
`2/2010 Lu et al.
`7,660,439 Bl
`3/2010 Elazar et al.
`7,683,929 B2
`6/2010 Yin et al.
`7,733,369 B2
`6/2010 Rezvani et al.
`7,734,724 B2
`6/2010 Donovan et al.
`7,737,837 B2
`8/2010 Cofer
`7,768,549 B2
`8/2010 Arrouye et al.
`7,774,326 B2
`9/2010 Lipton et al.
`7,796,780 B2
`10/2010 Garoutte et al.
`7,822,224 B2
`11/2010 Zhang et al.
`7,825,954 B2
`11/2010 Primot et al.
`7,826,066 B2
`1/2011 Venetianer et al.
`7,868,912 B2
`2/2011 Yin et al.
`7,884,849 B2
`4/2011 Lipton et al.
`7,932,923 B2
`6/2011 Jackson et al.
`7,956,735 B2
`2001/0004739 Al
`6/2001 Sekiguchi et al.
`8/2001 Takagi et al.
`2001/0010542 Al
`2001/0019357 Al
`9/2001
`Ito et al.
`10/2001 Garoutte
`2001/0033330 Al
`2001/0035907 Al
`11/2001 Broemmelsiek
`2002/00087 58 Al
`1/2002 Broemmelsiek et al.
`2002/0024446 Al
`2/2002 Grech-Cini
`2002/0048388 Al
`4/2002 Hagihara et al.
`5/2002 Ito et al.
`2002/0051058 Al
`2002/0063154 Al
`5/2002 Hoyos et al.
`2002/0082769 Al
`6/2002 Church et al.
`7/2002 Barker et al.
`2002/0095490 Al
`2002/0135483 Al
`9/2002 Merheim et al.
`2002/0141637 Al * 10/2002 Brodsky et al. ............... 382/165
`10/2002 Nakamura
`2002/0150308 Al
`2002/0159634 Al
`10/2002 Lipton et al.
`2002/0163521 Al
`11/2002 Ellenby et al.
`2002/0163577 Al
`11/2002 Myers
`2002/0171546 Al
`11/2002 Evans et al.
`2003/0010345 Al
`1/2003 Koblasz et al.
`2003/0020808 Al
`1/2003 Luke et al.
`2003/0025599 Al
`2/2003 Monroe
`2003/0043160 Al
`3/2003 Elfving et al.
`2003/0051255 Al
`3/2003 Bulman et al.
`2003/0053659 Al
`3/2003 Pavlidis et al.
`2003/0058111 Al
`3/2003 Lee et al.
`2003/0058340 Al
`3/2003 Lin et al.
`2003/0058341 Al
`3/2003 Brodsky et al.
`2003/0085992 Al
`5/2003 Arpa et al.
`2003/0093260 Al
`5/2003 Dagtas et al.
`2003/0126622 Al
`7/2003 Cohen et al.
`2003/0163289 Al
`8/2003 Whelan et al.
`2003/0231769 Al
`12/2003 Bolle et al.
`2004/0113933 Al
`6/2004 Guler
`2004/0130620 Al *
`7/2004 Buehler et al. ................ 348/143
`2004/0137915 Al
`7/2004 Diener et al.
`2004/0143602 Al
`7/2004 Ruiz et al.
`2004/0151374 Al
`8/2004 Lipton et al.
`2004/0161133 Al
`8/2004 Elazar et al.
`2004/0225681 Al
`11/2004 Chaney et al.
`2004/0240542 Al
`12/2004 Yeredor et al.
`2004/0240546 Al
`12/2004 Wells
`2005/0002561 Al
`1/2005 Monachino et al.
`2005/0146605 Al
`7/2005 Lipton et al.
`2005/0157169 Al
`7/2005 Brodsky et al.
`2005/0162515 Al
`7/2005 Venetianer et al.
`2005/0168574 Al
`8/2005 Lipton et al.
`2005/0169367 Al
`8/2005 Venetianer et al.
`2005/0198063 Al
`9/2005 Thomas et al.
`2005/0275549 Al
`12/2005 Barclay et al.
`2006/0066722 Al
`3/2006 Yin et al.
`2006/0117356 Al
`6/2006 Jojic et al.
`2006/0200842 Al
`9/2006 Chapman et al.
`2006/0222209 Al
`10/2006 Zhang et al.
`2006/0232673 Al
`10/2006 Lipton et al.
`2006/0268111 Al
`11/2006 Zhang et al.
`2006/0279630 Al
`12/2006 Aggarwal et al.
`2007/0002141 Al
`1/2007 Lipton et al.
`2007/0013776 Al
`1/2007 Venetianer et al.
`2007 /0035623 Al
`2/2007 Garoutte et al.
`2007/0052803 Al
`3/2007 Chosak et al.
`
`

`

`US 8,564,661 B2
`Page 4
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2007/0122000 Al
`2007/0127774 Al
`2007/0182818 Al
`2008/0292140 Al
`2008/0317286 Al
`2009/0244291 Al
`2010/0020172 Al
`
`5/2007 Venetianer et al.
`6/2007 Zhang et al.
`8/2007 Buehler
`11/2008 Morris et al.
`12/2008 Thorpe et al.
`10/2009 Saptharishi et al.
`1/2010 Mariadoss
`
`FOREIGN PATENT DOCUMENTS
`
`KR
`KR
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`
`100248374
`1020000026757
`WO-9403014
`WO 98/19450
`WO 00/28498
`WO 01/33503
`WO 01/62005
`WO 03/044727
`WO 2004/006184 A2
`WO 2004/047039 Al
`WO 2008/054489 A2
`WO 2009/120616 Al
`WO 2010/076261 Al
`WO 2011/036661 Al
`
`12/1999
`5/2000
`2/1994
`5/1998
`5/2000
`5/2001
`8/2001
`5/2003
`1/2004
`6/2004
`5/2008
`10/2009
`7/2010
`3/2011
`
`DE
`DE
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`FR
`GB
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`KR
`
`10153484 Al
`10252731 Al
`0293189
`0893923
`0913799 A2
`0 967 584 A2
`0967584 A2
`1024666
`1120746
`1333682
`1916618 Al
`2944934 Al
`2337146 A
`01-244598
`03-242592
`1991-035399
`05-014891
`1993-284501
`1994-52311
`1994-76047
`07-160891
`1995-302328
`1996-171681
`1996-63599
`08-123935
`08-136251
`1996-136251
`08-272926
`09-050585
`09-130783
`09-252467
`09247654
`1997-265539
`09-330415
`Hl0-048008
`10-116359
`1998-116359
`10-283584
`Hl0-290449
`1999-15982
`11-177973
`11-203568
`1999-355759
`2000-175174
`11-009618
`11-009619
`2000-207664
`2000-207665
`2000-224542
`11-48585
`2000-253382
`11-88337
`2000-285223
`2000-287173
`2000-295600
`2000-339923
`2001-045465
`2001-145092
`2001-175868
`2001-189925
`2001-204007
`2001-266131
`2001-285681
`1999-0042072
`
`5/2003
`5/2004
`11/1988
`1/1999
`5/1999
`12/1999
`12/1999
`8/2000
`8/2001
`8/2003
`4/2008
`10/2010
`10/1999
`9/1989
`2/1990
`2/1991
`1/1993
`10/1993
`2/1994
`3/1994
`6/1995
`11/1995
`2/1996
`3/1996
`5/1996
`5/1996
`5/1996
`10/1996
`2/1997
`5/1997
`9/1997
`9/1997
`10/1997
`12/1997
`2/1998
`5/1998
`5/1998
`10/1998
`10/1998
`1/1999
`7 /1999
`7 /1999
`12/1999
`6/2000
`7/2000
`7/2000
`7/2000
`7/2000
`8/2000
`9/2000
`9/2000
`10/2000
`10/2000
`10/2000
`10/2000
`12/2000
`2/2001
`5/2001
`6/2001
`7/2001
`7/2001
`9/2001
`10/2001
`5/1998
`
`OTHER PUBLICATIONS
`
`Ivanov Yet al., "Video Surveillance oflnteractions," Visual Surveil(cid:173)
`lance, 1999. Second IEEE Workshop on, (VS '99) Fort Collins, CO,
`USA, Jun. 26, 1999, Los Alamitos CA, USA, IEEE Computer Soci(cid:173)
`ety US.
`Lipton et al.: "Objectvideo Forensics: Activity-Based Video Indexing
`and Retrieval for Physical Security Applications", IEE Intelligent
`Distributed Surveillance Systems, Feb. 23, 2004.
`Heartwell et al: "Critical Asset Protection, Perimeter Monitoring and
`Threat Detection using Automated Video Surveillance-A Technol(cid:173)
`ogy Overview with Case Studies", Proceedings 36th Annual 2002
`International Carnahan Conference on Security Technology, 2002.
`Ellen McCarthy "A Firm That Lets Guards Keep an Electronic Eye
`Open", Oct. 21, 2002.
`International Search Report for International Application No. PCT/
`US08/09073, dated Nov. 3, 2008.
`Written Opinion for International Patent Application No. PCT /US08/
`09073, dated Nov. 3, 2008.
`A. Lipton, H. Fujiyoshi and R.S. Patil, "Moving Target Classification
`and Tracking from Real-Time Video," Proceedings of IEEE WACV
`'98, Princeton, NJ, 1998, pp. 8-14.
`W.E.L. Grimson, et al., "Using Adaptive Tracking to Classify and
`Monitor Activities in a Site," CVPR, pp. 22-29, Jun. 1998.
`A.J. Lipton, H. Fujiyoshi, R.S. Patil, "Moving Target Classification
`and Tracking from Real-time Video," IUW, pp. 129-136, 1998.
`T. J. Olson and F. Z. Brill, "Moving Object Detection and Event
`Recognition Algorithm for Smart Cameras," IUW, pp. 159-17 5, May
`1997.
`A. J. Lipton, "Local Application of Optic Flow to Analyse Rigid
`Versus Non-Rigid Motion," International Conference on Computer
`Vision, Corfu, Greece, Sep. 1999.
`F. Bartolini, V. Cappellini, and A. Mecocci, "Counting people getting
`in and out of a bus by real-time image-sequence processing," IVC,
`12(1):36-41, Jan. 1994.
`M. Rossi and A. Bozzoli, "Tracking and counting moving people,"
`ICIP94,pp.212-216, 1994.
`C. R. Wren, A. Azarbayejani, T. Darrell, and A. Pentland, "Pfinder:
`Real-time tracking of the human body,"Vismod, 1995.
`L. Khoudour, L. Duvieubourg, J.P. Deparis, "Real-Time Pedestrian
`Counting by Active Linear Cameras," JEI, 5(4):452-459, Oct. 1996.
`S. Ioffe, D. A. Forsyth, "Probabilistic Methods for Finding People,"
`IJCV, 43(1):45-68, Jun. 2001.
`M. Isard and J. MacCormick, "BraMBLe: A Bayesian Multiple-Blob
`Tracker," ICCV, 2001.
`D. M. Gavrila, "The Visual Analysis of Human Movement: A Sur(cid:173)
`vey," CVIU, 73(1):82-98, Jan. 1999.
`Niels Haering and Niels da Vitoria Lobo, "Visual Event Detection,"
`Video Computing Series, Editor Mubarak Shah, 2001.
`Collins, Lipton, Kanade, Fujiyoshi, Duggins, Tsin, Tolliver,
`Enomoto, and Hasegawa, "A System for Video Surveillance and
`Monitoring: VSAM Final Report," Technical Report CMU-RI-TR-
`00-12, Robotics Institute, Carnegie Mellon University, May 2000.
`C.R. Wren, A. Azarbayejani, T. Darrell, and A. P. Pentland. "Pfinder:
`Real-Time Tracking of the Human Body," PAMI, vol. 19, pp. 780-
`784, 1997.
`M. Allmen and C. Dyer, "Long-Range Spatiotemporal Motion
`Understanding Using Spatiotemporal Flow Curves," Proc. IEEE
`CVPR, Lahaina, Maui, Hi., pp. 303-309, 1991.
`
`

`

`US 8,564,661 B2
`Page 5
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`L. Wixson, "Detecting Salient Motion by Accumulating Direction(cid:173)
`ally Consistent Flow", IEEE Trans. Pattern Anal. Mach. Intell., vol.
`22, pp. 774-781, Aug. 2000.
`JP Office Action issued for JP 2003-546290 on Oct. 9, 2007, with an
`English translation.
`Jemez Technology Corp., Variant iD Web-Site, www.variantid.com,
`printed Aug. 25, 2003.
`L. Wixson et al., "Detecting Salient Motion by Accumulating
`Directionally-Consistent Flow," IEEE, 1999.
`J.P. Deparis et al., "A Device for Counting Passengers Making Use of
`Two Active Linear Cameras: Comparison of Algorithms," IEEE, pp.
`1629-1634, 1996.
`Alan J. Lipton "Virtual Postman-An Illustrative Example ofVirtual
`Video," International Journal of Robotics and Automation, vol. 15,
`No. 1, Jan. 2000, pp. 9-16.
`Alan J. Lipton, "Virtual Postman-Real Time, Interactive Virtual
`Video," IASTED Conference on Computer Graphics and Imaging
`(CGIM'99), Palm Springs, Oct. 25-27, 1999.
`H. Fujiyoshi and A. J. Lipton, "Real-time Human Motion Analysis by
`Image Skeletonization, "Proceedings of IEEE WACV'98, Princeton,
`NJ, 1998, pp. 15-21.
`R. T. Collins, Y. Tsin, J. R. Miller, and A. J. Lipton "Using a DEM to
`Determine Geospatial Object Trajectories," CMU-RI-TR-98-19,
`1998.
`International Search Report issued for PCT Application No. PCT/
`US06/45625, mailed on Sep. 24, 2007.
`Written Opinion of the International Searching Authority issued for
`PCT Application No. PCT/US06/45625, mailed on Sep. 24, 2007.
`International Search Report and Written Opinion in PCT/US06/
`02700.
`A. Selinger and L. Wixson, "Classifying Moving Objects as Rigid
`and Non-Rigid Without Correspondences," Proceedings of DARPA
`Image Understanding Workshop, Nov. 1, 1998, pp. 341-347.
`International Search Report issued for PCT Application No. PCT/
`US06/25196, mailed Jan. 16, 2008.
`Written Opinion issued for PCT Application No. PCT /US06/25 l 96,
`mailed Jan. 16, 2008.
`Written Opinion of the International Searching Authority issued for
`PCT Application No.PCT/US06/012556, mailed on Feb. 12, 2008.
`International Search Report of the International Searching Authority
`issued for PCT Application No. PCT/US06/012556, mailed on Feb.
`12, 2008.
`CN Office Action for CN 02822772.7 on Oct. 14, 2005 in English.
`Collins, et al. "A System for Video Surveillance and Monitoring".
`Carnegie Mellon University, 2000.
`Kanade, et al. "Advances in Cooperative Multi-Sensor Video Surveil(cid:173)
`lance". Robotics Institute, Carnegie Mellon University.
`Lipton, et al. "Automated Video Protection, Monitoring & Detec(cid:173)
`tion". IEEE AESS Systems Magazine, May 2003.
`Cao, et al. "Automatic Geo-Registration for Port Surveillance". Inter(cid:173)
`national Journal of Pattern Recognition and Artificial Intelligence,
`Nov. 19, 2008.
`Haering, et al. "A Semantic Event-Detection Approach and Its Appli(cid:173)
`cation to Detecting Hunts in Wildlife Video". IEEE Transactions on
`Circuits and Systems for Video Technology, vol. 10, No. 6, Sep. 2000.
`Cao, et al. "Automatic Geo-Registration of Maritime Video Feeds".
`IEEE, 2008.
`Haering, et al. "Content-Based Access of Image and Video Librar(cid:173)
`ies". Proceedings on IEEE Workshop, pp. 15-25, Jun. 20, 1997.
`Baker, et al. "Autonomous Vehicle Video Aided Navigation-Cou(cid:173)
`pling INS and Video Approaches".
`Lipton, Alan. "Keynote: Intelligent Video as a Force Multiplier for
`Crime Detection and Prevention".
`Ali, et al. "Interactive Retrieval of Targets for Wide Area Surveil(cid:173)
`lance". MM' 10, Firenze, Italy, Oct. 2010.
`Rice, et al."Maritime Surveillance in the Intracoastal Waterway using
`Networked Underwater Acoustic Sensors integrated with a Regional
`Command Center". 2nd International Conference on Waterside
`Security, Marina di Carrara, Italy, Nov. 2010.
`
`Heartwell, et al. "Critical Asset Protection, Perimeter Monitoring and
`Threat Detection using Automated Video Surveillance-A Technol(cid:173)
`ogy Overview with Case Studies", IEEE, 2002.
`Choe, et al. "Image Transformation for Object Tracking in High(cid:173)
`Resolution Video". IEEE, 2008.
`Deng, et al. "Fast Forensic Video Event Retrieval Using Geospatial
`Computing".
`Venetianer, et al. "Performance evaluation of an intelligent video
`surveillance system-A case study". Computer Vision and Image
`Understanding 114, pp. 1292-1302, 2010.
`Hakeem, et al. "Semantic Video Search using Natural Language
`Queries" MM'09, Beijing, China, Oct. 2009.
`Rasheed, et al. "Rapidly Deployable Video Analysis Sensor Units for
`Wide Area Surveillance".
`Haering, Niels. "What Would You Pay for Automated Video Analy(cid:173)
`sis?". 2009 Advanced Video and Signal Based Surveillance. IEEE,
`2009.
`Office Action ofNov. 25, 2011 inrelatedEPOApplication No. 06 749
`276.9-1232.
`Econolite, "Auto scope Image Sensor-AIS-IV," pp. 1-2, 2011.
`Ghanem et al., Representation and Recognition of Events in Surveil(cid:173)
`lance Video Using Petri Nets, Proceedings of the 2004 IEEE Com(cid:173)
`puter Society Conference on Computer Vision and Pattern Recogni(cid:173)
`tion Workshops, pp. 1-9, 2004.
`Computer & Information Systems Abstracts, Keeping an Eye Out,
`2005.
`Computer & Information Systems Abstracts, ObjectVideo Extends
`its Capabilities for Border Applications as Part of Multi-Million
`Dollar Homeland Security Advanced Research Project Agency
`Grant, 2006.
`Computer & Information Systems Abstracts, QA: Security Grants Up
`for Grabs, 2004.
`CSA, Veni, Vidi, Vici Analytics: But Who Will Really Conquer?,
`2007.
`Dialog, Ex FCC Chief Makes Cisco Board an Even Dozen, Mar. 26,
`2007.
`Dialog, Iridium Names VP for Data Business, Jul. 9, 2001.
`Diamond Back, (2) Digest, Jan. 27, 2000.
`Elsevier Eng. Info. Inc., Threat Detection Expands Role, Jul. 20,
`2006.
`Gale/ Cengage, Airport Selects Object Video for Intrusion Detection,
`Jun. 29, 2005.
`Gale/ Cengage, Appian Names Novak Biddle and MapQuest Exec to
`Its Board, Oct. 23, 2008.
`Gale/ Cengage, Avinor Selects ESP Group and Object Video for Sur(cid:173)
`veillance at Norwegian Airports, May 5, 2008.
`Gale/ Cengage, Bosch Security Systems Utilizes ObjectVideo's
`Intelligent Video Surveillance Software, Apr. 6, 2005.
`Gale/ Cengage, Chubb Security Taps Object Video as Approved Sup(cid:173)
`plier, Jul. 8, 2005.
`Gale/ Cengage, Companies in the News Today: About Chubb Secu(cid:173)
`rity, Jul. 8, 2005.
`Gale/ Cengage, Companies in the News Today: About ObjectVideo
`Inc., Jul. 8, 2005.
`Gale/ Cengage, comScore Names Kenneth Tarpey as CFO, Apr. 26,
`2009.
`Gale/ Cengage, Cornet Offers iVDO SmartEdge Video Analytics for
`Its Ruggedized iVDO IP Encoders, May 4, 2008.
`Gale/ Cengage, Co Vi Technologies Adds Digital Video Analytics to
`Analog Video Surveillance Camera, Sep. 30, 2007.
`Gale/ Cengage, Digiop Inks OEM Partnership with ObjectVideo,
`Apr. 30, 2008.
`Gale/ Cengage, Does the eye spy? Around-the-clock video surveil(cid:173)
`lance is the Holy Grail of K-12 safety efforts. But it raises many
`questions, including whether or not the cameras are a wholly benign
`presence., Nov. 1, 2007.
`Gale/ Cengage, Fibridge's New HD, IP Camera Leverages Reference
`Design from TI and ZOT, Dec. 13, 2008.
`Gale/ Cengage, Former FCC Chairman Powell Joins ObjectVideo
`Board of Directors, Sep. 9, 2005.
`Gale/ Cengage, Intelligent IP Camera Features Onboard Analytics,
`Apr. 13, 2007.
`
`

`

`US 8,564,661 B2
`Page 6
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Gale/ Cengage, Intelligent Video Server Promotes OEM Camera
`Development, Apr. 30, 2007.
`Gale/ Cengage, Lanaccess' Encoders to Include ObjectVideo's
`Analytics, Sep. 21, 2005.
`Gale/ Cengage, MangoDSP and ObjectVideo to Provide Integrated
`Intelligent Video Surveillance Platforms, Apr. 10, 2005.
`Gale/ Cengage, MangoDSP, ObjectVideo Join Forces on Surveil(cid:173)
`lance Solutions, Apr. 7, 2005.
`Gale/ Cengage, Next Steps for Technology: How U.S. R&D Efforts
`Have Pushed Development Forward, Sep. 11, 2006.
`Gale/ Cengage, Object Video and TI Get Win with Hangzhou Hikvi(cid:173)
`sion Digital Technology, Apr. 1, 2007.
`Gale/ Cengage, ObjectVideo Develops New Scanning Camera Tech(cid:173)
`nology to Protect U.S. Borders, Jul. 19, 2006.
`Gale/ Cengage, Object Video Gets $3 .2 Million Pacts from Homeland
`Security Advanced Research Project Agency (HSARPA), May 13,
`2005.
`Gale/ Cengage, Object Video Gets DARPA Contract to Support Next
`Generation Surveillance System, Jan. 28, 2008.
`Gale/ Cengage, ObjectVideo Named 2005 "Technology Pioneer,"
`Dec. 6, 2004.
`Gale/ Cengage, Object Video Named a Red Herring 100 Company for
`Second Year in a Row, May 13, 2005.
`Gale/ Cengage, Object Video Offers Intelligent Video Network Pro(cid:173)
`tocol for Standardized Analytics Communication and Integration,
`Sep. 25, 2007.
`Gale/ Cengage, Object Video Provides Intelligent Video Software to
`Port, Jul. 13, 2005.
`Gale/ Cengage, Object Video Releases OnBoard Analytics on ARM
`Processors, Apr. 5, 2011.
`Gale/ Cengage, ObjectVideo Releases Software for Intel Architec(cid:173)
`ture Systems, Apr. 21, 2008.
`Gale/ Cengage, ObjectVideo Signs Distributor for Asia, Dec. 14,
`2005.
`Gale/ Cengage, ObjectVideo Software Part of Security System at
`Memphis/Shelby County Airport, Jun. 21, 2005.
`Gale/ Cengage, Object Video to Showcase the Latest Intelligent Solu(cid:173)
`tions at ASIS, Sep. 23, 2007.
`Gale/ Cengage, Object Video to Spotlight Intelligent Solutions at ISC
`West 2007, Mar. 26, 2007.
`Gale/ Cengage, ObjectVideo Touts Early Adoption of its Protocol,
`Mar. 31, 2008.
`Gale/ Cengage, ObjectVideo, Security Services & Technologies
`Announce Integration Partnership, Apr. 6, 2005.
`Gale/ Cengage, People in Wireless News: Peter Currie Joins Critical
`Path Board, Feb. 2, 2004.
`Gale/ Cengage, Phoenix IVS, ObjectVideo and MangoDSP to
`Launch Intelligent Video Module, Aug. 10, 2005.
`Gale/ Cengage, Proximex Upgrades Surveillint, Dec. 21, 2008.
`Gale/ Cengage, Puerto Rico Public Housing Beefs Up Security Via
`Intelligent Surveillance System from Avant Technologies and
`ObjectVideo, Sep. 19, 2007.
`Gale/ Cengage, Software Creates Intelligent Devices for Video Sys(cid:173)
`tems, Mar. 31, 2006.
`Gale/ Cengage, Software Delivers Industry-Specific Intelligent
`Video, May 10, 2006.
`Gale/ Cengage, Synectic Systems Group and ObjectVideo Ink OEM
`Pact, Feb. 28, 2008.
`Gale/ Cengage, Video-DSPs Focus on Vision Systems, May 21,
`2007.
`Gale/ Cengage, VidSys Partners with Security Companies for PSIM
`Thought Leadership Site, Oct. 23, 2008.
`Gale/ Cengage, Wireless Channel: Chubb Security to Resell
`ObjectVideo Technology, Jul. 9, 2005.
`Information Week, Curtis Joins CMGI, May 8, 2000.
`Lipton, Stationary Target Detection Using the ObjectVideo Surveil(cid:173)
`lance System, 2007.
`Lipton, The Intelligent Vision Sensor: Turning Video into Informa(cid:173)
`tion, 2007.
`
`Lipton, Video Verification of Point of Sale Transactions, 2007.
`Abstract and documents related to the contract for Department of
`Defense Award No. 81276, A Unified Framework for False Alarm
`Reduction Using Scene Context from Airborne Sensors, Identifica(cid:173)
`tion No. 07SB2-0308, W31P4Q-08-C-0108, 2007.
`Abstract and documents related to the contract for Department of
`Defense Award No. 63547, Automated Video Surveillance at Night,
`Identification No. 03ST1-004, DAAH0103CR276, 2003.
`Abstract and documents related to the contract for Department of
`Defense Award No. 82715, Context-Sensitive Content Extraction and
`Scene Understanding, Identification No. N071-085-0199, N00014-
`08-C-0639, 2007.
`Abstract and documents related to the contract for Department of
`Defense Award No. 65022, Human Activity Recognition, Identifica(cid:173)
`tion No. N031-l 723, N00014-03-M-0256, 2003.
`Abstract and documents related to the contract for Department of
`Defense Award No. 76719, Identify and Track Important Assets,
`Identification No. S052-007-0063, H92222-06-P-0012, 2005.
`Abstract and documents related to the contract for Department of
`Defense Award No. 78523, Intelligent Retrieval of Surveillance
`Imagery, Identification No. N062-147-0567, N00014-08-C0110,
`2006.
`Abstract and documents related to the contract for Department of
`Defense Award No. 59579, Mast-Mounted In-Port Video Surveil(cid:173)
`lance System, Identification No. N021-1610; N00014-03-C-0232,
`2002.
`Abstract and documents related to the contract for Department of
`Defense Award No. 83229, Robust Multiple Target Tracking, Iden(cid:173)
`tification No. A074-007-009 l, W9 l 1NF-08-C-0068, 2007.
`Abstract and documents related to the contract for National Science
`Foundation Award No. 69404, SBIR Phase I: Learning Context,
`Identification No. 0340175, 0340175, 2004.
`Abstract and documents related to the contract for Department of
`Defense Award No. 5 84 72, Streaming Multimedia for the Warfighter,
`Identification No. 99SB1-0357, DAAH01-02-C-Rl93, 2002.
`Abstract and documents related to the contract for Department of
`Defense Award No. 55489, Video Detection, Tracking and Clas

This document is available on Docket Alarm but you must sign up to view it.


Or .

Accessing this document will incur an additional charge of $.

After purchase, you can access this document again without charge.

Accept $ Charge
throbber

Still Working On It

This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.

Give it another minute or two to complete, and then try the refresh button.

throbber

A few More Minutes ... Still Working

It can take up to 5 minutes for us to download a document if the court servers are running slowly.

Thank you for your continued patience.

This document could not be displayed.

We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.

Your account does not support viewing this document.

You need a Paid Account to view this document. Click here to change your account type.

Your account does not support viewing this document.

Set your membership status to view this document.

With a Docket Alarm membership, you'll get a whole lot more, including:

  • Up-to-date information for this case.
  • Email alerts whenever there is an update.
  • Full text search for other cases.
  • Get email alerts whenever a new case matches your search.

Become a Member

One Moment Please

The filing “” is large (MB) and is being downloaded.

Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.

Your document is on its way!

If you do not receive the document in five minutes, contact support at support@docketalarm.com.

Sealed Document

We are unable to display this document, it may be under a court ordered seal.

If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.


Access Government Site

We are redirecting you
to a mobile optimized page.





Document Unreadable or Corrupt

Refresh this Document
Go to the Docket

We are unable to display this document.

Refresh this Document
Go to the Docket