(12) United States Patent
`Lipton et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,564,661 B2
`Oct. 22, 2013
`
`USOO8564661 B2
`
`(54) VIDEO ANALYTIC RULE DETECTION
`SYSTEMAND METHOD
`
`(75) Inventors: Alan J. Lipton, Austin, TX (US); John
`I. W. Clark, Flamborough (CA); Zhong
`Zhang, Great Falls, VA (US); Peter L.
`Venetianer, McLean, VA (US); Thomas
`Strat, Oakton, VA (US); Mark Allmen,
`Morrison, CO (US); William Severson,
`Centennial, CO (US); Niels Haering,
`Reston, VA (US); Andrew Chosak,
`Arlington, VA (US); Matthew Frazier,
`New York, NY (US); James Sfekas,
`Seattle, WA (US); Tasuki Hirata,
`Cambridge, MA (US)
`(73) Assignee: ObjectVideo, Inc., Reston, VA (US)
`
`(65)
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1602 days.
`(21) Appl. No.: 11/828,842
`(22) Filed:
`Jul. 26, 2007
`(Under 37 CFR 1.47)
`Prior Publication Data
`US 2010/0026802 A1
`Feb. 4, 2010
`O
`O
`Related U.S. Application Data
`(63) Continuation-in-part of application No. 11/057,154,
`filed on Feb. 15, 2005, which is a continuation-in-part
`of application No. 09/987,707, filed on Nov. 15, 2001,
`now. abandoned, which is a continuation-in-part of
`application No. 09/694,712, filed on Oct. 24, 2000,
`now Pat. No. 6,954,498, application No. 11/828,842,
`which is a continuation-in-part of application No.
`11/ 167,218, filed on Jun. 28, 2005, which is a
`continuation-in-part of application No. 1 1/098,382,
`filed on Apr. 5, 2005, now Pat. No. 7,868,912, which is
`a continuation-in-part of application No. 11/057,154.
`
`(2006.01)
`
`(51) Int. Cl.
`H04N 9/47
`(52) U.S. Cl.
`USPC ............................ 348/143: 382/103: 382/115
`(58) Field of Classification Search
`None
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`3,812.278 A
`5, 1974 Aker
`3.812,287 A
`5/1974 Lemelson
`3,988,533 A 10, 1976 Micket al.
`(Continued)
`FOREIGN PATENT DOCUMENTS
`
`DE
`DE
`
`2/1996
`443 001.6 A1
`4, 2000
`19848490 A1
`(Continued)
`OTHER PUBLICATIONS
`Nation of Defects for IL 161777 (original and translation), Feb.
`
`(Continued)
`
`Primary Examiner - Ario Etienne
`Assistant Examiner — Clayton R Williams
`(74) Attorney, Agent, or Firm — Muir Patent Consulting,
`PLLC
`
`ABSTRACT
`(57)
`A video Surveillance system is set up, calibrated, tasked, and
`operated. The system extracts video primitives and extracts
`event occurrences from the video primitives using event dis
`criminators. The extracted video primitives and event occur
`rences may be used to create and define additional video
`analytic rules. The system can undertake a response, such as
`an alarm, based on extracted event occurrences.
`
`32 Claims, 25 Drawing Sheets
`
`
`
`
`
`
`
`
`
`14
`
`WDEO
`NSOR
`SENSORS
`
`15
`
`VIDEO
`RECORDERS
`
`
`
`
`
`
`
`
`
`OTHER
`SENSORS
`
`17
`
`COMPUTERSYSTEM
`
`COMPUTER-READABLE
`MEDIUM
`
`6
`
`
`
`ODEVICES
`
`Canon Ex. 1035 Page 1 of 52
`
`

`

`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`7, 1979 Kamin
`4, 160,998 A
`4, 1980 Kamin
`4,198,653 A
`2, 1981 Harman et al.
`4,249,207 A
`3/1981 Loughry et al.
`4,257,063 A
`7/1984 Mahoney
`4,458.266 A
`6, 1986 RiZZO
`4.593,274 A
`4, 1988 Araki et al.
`4.737,847 A
`8, 1988 Sullivan
`4,764,971 A
`8, 1989 Everett, Jr. et al.
`4,857,912 A
`3/1990 Fujioka et al.
`4,908,704 A
`7, 1990 Patel
`4.941,182 A
`7, 1990 Shiota et al.
`4,943,854 A
`8, 1990 Allebach et al.
`4,949,389 A
`9, 1991 Bhanu et al.
`5,048,095 A
`2f1992 Pomerleau
`5,091,780 A
`3, 1992 Gove
`5,099.322 A
`5,156,647 A 10/1992 Burt
`5,161,107 A 1 1/1992 Mayeaux et al.
`5,229,850 A
`7/1993 Toyoshima
`5.296.852 A
`3, 1994 Rathi
`5.331396 A
`6/1994 Lamming et al.
`5.448,315 A
`9, 1995 Soohoo
`5.448,651 A
`9, 1995 Sakou et al.
`5.465,115 A 11/1995 Conrad et al.
`5,485,611 A
`1/1996 Astle
`5.491.5ii A
`2, 1996 Odle
`5,515,453 A
`5/1996 Hennessey et al.
`5,519,789 A
`5, 1996 Etoh
`5,521,634 A
`5/1996 McGary
`5,526,133 A
`6, 1996 Paff
`5576.972 A 11/1996 Harrison
`5,586.200 A 12/1996 Devaney et al.
`5,602.585 A
`2, 1997 Dickinson et al.
`5,610,653 A
`3, 1997 Abecassis
`5,621,889 A
`4, 1997 Lermuzeaux et al.
`5,623.249. A
`4/1997 Camire
`5,666,157 A
`9, 1997 Aviv
`5,671,294 A
`9/1997 Rogers et al.
`5,684,715 A 11, 1997 Palmer
`5,696,503 A
`12/1997 Nasburg
`5,696,551 A 12, 1997 Katto
`5,708,767 A
`1/1998 Yeo et al.
`5,721,692 A
`2/1998 Nagaya et al.
`5,724.456 A
`3/1998 Boyacket al.
`5,764,306 A
`6, 1998 Steffano
`5,768.413 A
`6, 1998 Levin et al.
`5,774.569 A
`6, 1998 Waldenmaier
`5,798,787 A
`8/1998 Yamaguchi et al.
`5,801,943 A
`9/1998 Nasburg
`5,802,203 A
`9, 1998 Black et al.
`5,802.361 A
`9/1998 Wang et al.
`5,844,603 A 12/1998 Ogata
`5,850,352 A 12/1998 MoeZZi et al.
`5,860.086 A
`1/1999 Crumpet al.
`5,872,865 A
`2f1999 Normile et al.
`5.875.304 A
`2f1999 Winter et al.
`5.875305 A
`2f1999 Winter et al.
`5,886,701 A
`3, 1999 Chauvin et al.
`5,912,980 A
`6, 1999 Hunke
`5.923.365 A
`7, 1999 Tamir et al.
`5,926.30 A
`7, 1999 Hackett et al.
`5.937,092 A
`8, 1999 Wootton et al.
`5,956,081 A
`9, 1999 Katz et al.
`5,959,690 A
`9/1999 Toebes, VIII et al.
`5,963,202 A 10, 1999 Polish
`5,963,203 A 10/1999 Goldberg et al.
`5,969,755 A 10/1999 Courtney
`5,983,147 A 11/1999 Krumm
`5,983,251 A 11/1999 Martens et al.
`5,987.211 A 11/1999 Abecassis
`5.990,955 A 11, 1999 KOZ
`5.999,189 A 12/1999 Kajiya et al.
`5.999,877 A 12/1999 Takahashi et al.
`6,008,865 A 12/1999 Fogel
`6,014,461 A
`1/2000 Hennessey et al.
`
`US 8,564,661 B2
`Page 2
`
`2/2000 Chang et al.
`6,025,877 A
`2/2000 Aviv ............................. 348, 152
`6,028,626 A
`2/2000 MacCormack et al.
`6,031,573 A
`3/2000 Bassman et al.
`6,044,166 A
`4/2000 Courtney et al.
`6,049,363 A
`5/2000 de Queiroz et al.
`6,058,210 A
`5, 2000 Wellner et al.
`6,067,112 A
`5, 2000 Hudson
`6,069,653 A
`5/2000 Seeley et al.
`6,069,655 A
`6/2000 Katz
`6,075,560 A
`6/2000 Monro et al.
`6,078,619 A
`7/2000 Reitmeier et al.
`6,084,912 A
`7, 2000 Mead
`6,088.484 A
`7/2000 Seeley et al.
`6,091,771 A
`7/2000 Lin et al.
`6,091,846 A
`8/2000 Seeley et al.
`6,097.429 A
`8, 2000 Ito et al.
`6,108,033. A
`9, 2000 Carder et al.
`6,123,123 A
`6,128,396 A 10/2000 Hasegawa et al.
`6,128,398 A * 10/2000 Kuperstein et al. ........... 382,118
`6,144,375. A 1 1/2000 Jain et al.
`6,151,413 A 1 1/2000 Jang et al.
`6,166,744 A 12/2000 JaSzlics et al.
`6,167,143 A 12/2000 Badique
`6,177,886 B1
`1/2001 Billington et al.
`6,182,022 B1
`1/2001 Mayle et al.
`6,188,381 B1
`2/2001 van der Wall et al.
`6,188,777 B1
`2/2001 Darrell et al.
`6,195.458 B1
`2/2001 Warnicket al.
`6,201.473 B1
`3/2001 Schaffer
`6,201.476 B1
`3/2001 Depeursinge et al.
`6,205.239 B1
`3/2001 Lin et al.
`6.211.907 B1
`4/2001 Scaman et al.
`6,226,388 B1
`5/2001 Qian et al.
`6,249,613 B1
`6/2001 Crinonet al.
`6,278.466 B1
`8/2001 Chen
`6,297,844 B1 10/2001 Schatz et al.
`6,304,669 B1
`10/2001 Kaneko et al.
`6,307,885 B1
`10/2001 Moon et al.
`6,310,916 B1
`10/2001 Han et al.
`6,326,964 B1
`12/2001 Snyder et al.
`6,337,917 B1
`1/2002 Onural et al.
`6,349,113 B1
`2/2002 Mech et al.
`6,351,265 B1
`2/2002 Bulman
`6,351,492 B1
`2/2002 Kim et al.
`6,360,234 B2
`3/2002 Jain et al.
`6,385,772 B1
`5/2002 Courtney
`6,393,054 B1
`5/2002 Altunbasak et al.
`6,396,535 B1
`5, 2002 Waters
`6,396,876 B1
`5/2002 Babonneau et al.
`6,396.961 B1
`5, 2002 Wixson et al.
`6,404,455 B1
`6/2002 Ito et al.
`6,404,925 B1
`6/2002 Foote et al.
`6,408.293 B1
`6/2002 Aggarwal et al.
`6,411,209 B1
`6/2002 Lyons et al.
`6,411,724 B1
`6/2002 Vaithilingam et al.
`6.424,370 B1* 7/2002 Courtney ...................... 348/143
`6,437,819 B1
`8/2002 Loveland
`6,441,734 B1
`8/2002 Gutta et al.
`6,456,320 B2
`9/2002 Kuwano et al.
`6,469,734 B1
`10/2002 Nichani et al.
`6,490,370 B1
`12/2002 Krasinski et al.
`6,504,479 B1
`1/2003 Lemons et al.
`6,509,926 B1
`1/2003 Mills et al.
`6,515,615 B2
`2/2003 Burchett et al.
`6,525,658 B2
`2/2003 Streetman et al.
`6,525,663 B2
`2/2003 Colmenarez et al.
`6,535,620 B2
`3/2003 Wildes et al.
`6,539,396 B1
`3/2003 Bowman-Amuah
`6,542,840 B2
`4/2003 Okamoto et al.
`6,545,706 B1
`4/2003 Edwards et al.
`6,546,115 B1
`4/2003 Ito et al.
`6,546,135 B1
`4/2003 Lin et al.
`6,552,826 B2
`4/2003 Adler et al.
`6,570,496 B2
`5/2003 Britton
`6,570,608 B1
`5/2003 Tseng
`6,570,610 B1
`5/2003 Kipust
`6,573,907 B1
`6/2003 Madrane
`6,584.211 B1
`6/2003 Hashimoto et al.
`
`Canon Ex. 1035 Page 2 of 52
`
`

`

`US 8,564,661 B2
`Page 3
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`7/2003 Murray et al.
`6,597,800 B1,
`9/2003 Lipton et al.
`6,625,310 B2
`9/2003 Wegmann
`6,628,323 B1
`9, 2003 Brill et al.
`6,628,835 B1
`1 1/2003 DaGraca et al.
`6,646,676 B1
`12/2003 Brumitt
`6,658,136 B1
`2/2004 Venetianer et al.
`6,696,945 B1
`2/2004 Fernandez et al.
`6,697,103 B1
`2/2004 Yakobi et al.
`6,697,104 B1
`2, 2004 Amini et al.
`6.698,021 B1
`3, 2004 Millet et al.
`6,707.4s6 B1
`3/2004 Wang
`6.707852 Bi
`4/2004 Qian et al.
`6,721,454 B1
`4/2004 Toklu et al.
`6.724.915 B1
`4/2004 Randall
`6,727,938 B1
`5, 2004 Allmen et al.
`6.738,424 B1
`5/2004 Nagaya et al.
`6,741977 B1
`6, 2004 Ito et al.
`6.754.367 B1
`62004 Huang et al.
`6,757,328 Bi
`9/2004 Bilger
`6,792.319 B1
`10/2004 Owechko et al.
`6,801,662 B1
`10/2004 Girgensohn et al.
`6,807,361 B1
`11/2004 Brill et al.
`6.816,184 B1
`1/2004 Lakeet al.
`686,186 B2
`12/2004 Nichani et al.
`6.829,371 B1
`1/2005 Grech-Cini et al.
`6 844,818 B2
`2/2005 Dagtaset al.
`6859,803 B2
`3, 2005 Bush
`6.865,580 B1
`82005 Dorbie
`6,924.80 B
`8/2005 Giacalone et al.
`6.930,689 B1
`9/2005 Le Mire
`6,940,397 B1
`9/2005 Garoute
`6,940,998 B2
`10/2005 Lipton
`6,954.498 B1
`10/2005 Simerly et al.
`6,954,859 B1
`6,963,658 B2 11/2005 Hagihara et al.
`6,970,083 B2 11/2005 Venetianer et al.
`6.975,220 B1
`12/2005 Foodman et al.
`6,975,346 B2 12/2005 Kumhyr
`6,985,620 B2
`1/2006 Sawhney et al.
`6.987,451 B2
`1/2006 McKeown et al.
`6987.528 B1
`1/2006 Nagahisaeal
`6,987,883 B2
`1/2006 Lipton et al.
`7.023.469 B1
`4/2006 Olson
`7035.450 B2
`4/2006 to et al.
`7,082,209 B2
`7/2006 foetal.
`70.569 B2
`9/2006 Brodsky et al.
`7,146,286 B2 12/2006 Takeda et al.
`7,167,519 B2
`1/2007 Comaniciu et al.
`7,167.575 B1
`1/2007 Nichani et al.
`7.184777 B3
`2.2007 Dienereal.
`7,197,072 B1
`3/2007 Hsu et al.
`7218756 B2
`5/2007 Garotte
`7224.852 B2
`5 2007 Lipton et al.
`7,227,893 B1
`6/2007 Srinivasa et al.
`7.283,034 B2 10, 2007 Nakamura et al.
`730s. 443 B1
`13/2007 Lee et al.
`7319.479 B1* 1/2008 Crabtree et al. ... 348,169
`7,356,830 B1
`4/2008 Dimitrova
`7,358,976 B2
`4/2008 Hampshire, II et al.
`T.373.395 B2
`5, 2008 Brailean et al.
`7400.344 B2
`7/2008 to et al.
`7.433,666 B2
`9, 2008 Sakakibara et al.
`7.424,167 B1
`9/2008 Chosaketal.
`7.436,887 B3 10/2008 Yeredoretal.
`7.440,589 B2 10/2008 Garoute
`7.447,331 B2 11/2008 Brown et al.
`7.468,662 B2 12/2008 Velipasalar et al.
`7.487,072 B2
`2/2009 Semple et al.
`7,522,745 B2
`4/2009 Grasso et al.
`7,595,815 B2
`9/2009 Donovan et al.
`7,602,413 B2 10/2009 Kondo et al.
`7,613,324 B2 11/2009 Venetianer et al.
`7.623,677 B2 11/2009 Girgensohn et al.
`7,633,520 B2 12/2009 Samarasekera et al.
`7,643,653 B2
`1/2010 Garoute
`
`1/2010 GarOutte
`7,650,058 B1
`1/2010 Paek et al.
`7,653,635 B1
`2/2010 Whelanet al.
`7,657,914 B2
`2/2010 Lu et al.
`7,660,439 B1
`3/2010 Elazaret al.
`7,683,929 B2
`6/2010 Yin et al.
`7,733,369 B2
`6, 2010 Rezvani et al.
`7,734,724 B2
`6/2010 Donovan et al.
`7,737,837 B2
`8/2010 Cofer
`7,768,549 B2
`8/2010 Arrouye et al.
`7,774,326 B2
`9/2010 Lipton et al.
`7,796.780 B2
`7,822.224 B2 10/2010 Garoute et al.
`7,825,954 B2 11/2010 Zhang et al.
`7,826,066 B2 11/2010 Primot et al.
`7,868,912 B2
`1/2011 Venetianer et al.
`7,884,849 B2
`2/2011 Yin et al.
`7,932,923 B2
`4/2011 Lipton et al.
`7.956,735 B2
`6/2011 Jackson et al.
`2001/0004739 A1
`6/2001 Sekiguchi et al.
`2001/0010542 A1
`8/2001 Takagi et al.
`2001 OO19357 A1
`9, 2001 to et al.
`2001/0033330 A1 10, 2001 GarOutte
`2001/0035907 A1 11/2001 Broemmelsiek
`2002fOOO8758 A1
`1/2002 Broemmelsiek et al.
`2002fOO24446 A1
`2/2002 Grech-Cini
`2002fOO48388 A1
`4/2002 Hagihara et al.
`2002/0051058 A1
`5, 2002 to et al.
`2002/0063154 A1
`5/2002 Hoyos et al.
`2002/0082769 A1
`6, 2002 Church et al.
`2002fOO95490 A1
`7/2002 Barker et al.
`2002/0135483 Al
`9, 2002 Merheim et al.
`2002/0141637 A1* 10/2002 Brodsky et al. ............... 382,165
`2002fO150308 A1 10, 2002 Nakamura
`2002/0159634 A1 10/2002 Lipton et al.
`2002/0163521 A1 1 1/2002 Ellenby et al.
`2002/0163577 A1 1 1/2002 Myers
`2002/0171546 A1 11/2002 Evans et al.
`2003, OO 10345 A1
`1/2003 Koblasz et al.
`2003/0020808 A1
`1/2003 Luke et al.
`2003/0025599 A1
`2/2003 Monroe
`2003.0043.160 A1
`3/2003 Elfving et al.
`2003/005.1255 A1
`3/2003 Bulman et al.
`2003/0053659 A1
`3/2003 Pavlidis et al.
`2003/0058111 A1
`3/2003 Lee et al.
`2003, OO5834.0 A1
`3, 2003 Lin et al.
`2003, OO58341 A1
`3/2003 Brodsky et al.
`2003/0O85992 A1
`5/2003 Arpa et al.
`2003/0093260 A1
`5/2003 Dagtas et al.
`2003/O126622 A1
`7/2003 Cohen et al.
`2003,0163289 A1
`8, 2003 Whelanet al.
`2003/0231769 A1 12/2003 Bole et al.
`2004/O113933 A1
`6, 2004 Guler
`2004/O130620 A1
`7/2004 Buehler et al. ................ 348/143
`2004/O137915 A1
`7/2004 Diener et al.
`2004.0143602 A1
`7/2004 Ruiz et al.
`2004/0151374, A1
`8/2004 Lipton et al.
`2004/O161133 A1
`8/2004 Elazar et al.
`2004/0225681 A1 1 1/2004 Chaney et al.
`2004/0240542 A1 12/2004 Yeredor et al.
`2004/0240546 A1 12, 2004 Wells
`2005.0002561 A1
`1/2005 Monachino et al.
`2005, 0146605 A1
`7/2005 Lipton et al.
`2005. O157169 A1
`7/2005 Brodsky et al.
`2005/0162515 A1
`7/2005 Venetianer et al.
`2005/0168574 A1
`8/2005 Lipton et al.
`2005, 0169367 A1
`8, 2005 Venetianer et al.
`2005, 0198063 A1
`9, 2005 Thomas et al.
`2005/0275549 A1 12/2005 Barclay et al.
`2006/0066722 A1
`3, 2006 Yin et al.
`2006/01 17356 A1
`6/2006 Jojic et al.
`2006/0200842 A1
`9/2006 Chapman et al.
`2006/0222209 A1 10/2006 Zhang et al.
`2006/0232673 Al 10/2006 Lipton et al.
`2006/02681 11 A1 1 1/2006 Zhang et al.
`2006/0279630 Al 12/2006 Aggarwal et al.
`2007/0002141 A1
`1/2007 Lipton et al.
`2007/OO 13776 A1
`1/2007 Venetianer et al.
`2007.0035623 A1
`2/2007 Garoute et al.
`2007/0052803 A1
`3, 2007 Chosak et al.
`
`Canon Ex. 1035 Page 3 of 52
`
`

`

`US 8,564,661 B2
`Page 4
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2007/O122000 A1
`2007/0127774 A1
`2007, 01828.18 A1
`
`5/2007 Venetianer et al.
`6/2007 Zhang et al.
`8, 2007 Buehler
`
`2008/0317286 Al 12/2008 Thorpe et al.
`2009/0244291 A1 10/2009 Saptharishi et al.
`2010.0020172 A1
`1/2010 Mariadoss
`
`FOREIGN PATENT DOCUMENTS
`
`KR
`KR
`WO
`WO
`WO
`WO
`WO
`WO
`
`10O248374
`102OOOOO26757
`WO-9403.014
`WO98, 19450
`WOOO,28498
`WOO1/33503
`WOO1? 62005
`WO 03/044727
`
`12/1999
`5, 2000
`2, 1994
`5, 1998
`5, 2000
`5, 2001
`8, 2001
`5, 2003
`
`6, 2004
`WO WO 2004/047O39 A1
`5, 2008
`WO WO 2008/O54489 A2
`WO WO 2009/12061.6 A1 10/2009
`WO WO 2010/076261 A1
`T 2010
`WO WO 2011/036661 A1
`3, 2011
`
`DE
`DE
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`FR
`GB
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`KR
`
`10153484 A1
`10252731 A1
`O293,189
`O893.923
`O913799 A2
`O967 584 A2
`O967584 A2
`1024666
`1120746
`1333682
`1916618 A1
`2944934 A1
`2337146 A
`O1-244598
`O3-242592
`1991-035399
`05-014891
`1993-2845O1
`1994-52311
`1994-76O47
`O7-160891
`1995-3O2328
`1996-171681
`1996-63599
`O8-123935
`08-136251
`1996-136251
`08-272926
`09-050585
`O9-130783
`09-252467
`O9247654
`1997-265539
`O9-33.04.15
`H10-048008
`10-116359
`1998-116359
`10-283.584
`H10-290449
`1999-15982
`11-177973
`11-203568
`1999-3557.59
`2000-175174
`11-009618
`11-0096.19
`2000-207664
`2000-2O7665
`2000-224.542
`11-48585
`2000-253382
`11-88337
`2000-285223
`2000-287.173
`2000-295600
`2000-339.923
`2001-045465
`2001-145092
`2001-175868
`2001-1899.25
`2001-204007
`2001-266131
`2001-285681
`1999-0042O72
`
`5, 2003
`5, 2004
`11, 1988
`1, 1999
`5, 1999
`12/1999
`12/1999
`8, 2000
`8, 2001
`8, 2003
`4/2008
`10, 2010
`10, 1999
`9, 1989
`2, 1990
`2, 1991
`1, 1993
`10, 1993
`2, 1994
`3, 1994
`6, 1995
`11, 1995
`2, 1996
`3, 1996
`5, 1996
`5, 1996
`5, 1996
`10, 1996
`2, 1997
`5, 1997
`9, 1997
`9, 1997
`10, 1997
`12/1997
`2, 1998
`5, 1998
`5, 1998
`10, 1998
`10, 1998
`1, 1999
`7, 1999
`7, 1999
`12/1999
`6, 2000
`T 2000
`T 2000
`T 2000
`T 2000
`8, 2000
`9, 2000
`9, 2000
`10, 2000
`10, 2000
`10, 2000
`10, 2000
`12/2000
`2, 2001
`5, 2001
`6, 2001
`T 2001
`T 2001
`9, 2001
`10, 2001
`5, 1998
`
`OTHER PUBLICATIONS
`
`Ivanov Yet al., “Video Surveillance of Interactions.” Visual Surveil
`lance, 1999. Second IEEE Workshop on. (VS 99) Fort Collins, CO,
`USA, Jun. 26, 1999, Los Alamitos CA, USA, IEEE Computer Soci
`ety US.
`Lipton et al.: "ObjectVideo Forensics: Activity-Based Video Indexing
`and Retrieval for Physical Security Applications”, IEE Intelligent
`Distributed Surveillance Systems, Feb. 23, 2004.
`Heartwelletal: “Critical Asset Protection, Perimeter Monitoring and
`Threat Detection using Automated Video Surveillance—A Technol
`ogy Overview with Case Studies'. Proceedings 36th Annual 2002
`International Carnahan Conference on Security Technology, 2002.
`Ellen McCarthy "A Firm That Lets Guards Keep an Electronic Eye
`Open”, Oct. 21, 2002.
`International Search Report for International Application No. PCT/
`US08/09073, dated Nov. 3, 2008.
`Written Opinion for International Patent Application No. PCT/US08/
`09073, dated Nov. 3, 2008.
`A. Lipton, H. Fujiyoshi and R.S. Patil, "MovingTarget Classification
`and Tracking from Real-Time Video.” Proceedings of IEEE WACV
`98, Princeton, NJ, 1998, pp. 8-14.
`W.E.L. Grimson, et al., “Using Adaptive Tracking to Classify and
`Monitor Activities in a Site.” CVPR, pp. 22-29, Jun. 1998.
`A.J. Lipton, H. Fujiyoshi, R.S. Patil, "Moving Target Classification
`and Tracking from Real-time Video.” IUW, pp. 129-136, 1998.
`T. J. Olson and F. Z. Brill, “Moving Object Detection and Event
`Recognition Algorithm for Smart Cameras.” IUW, pp. 159-175, May
`1997.
`A. J. Lipton, “Local Application of Optic Flow to Analyse Rigid
`Versus Non-Rigid Motion.” International Conference on Computer
`Vision, Corfu, Greece, Sep. 1999.
`F. Bartolini, V. Cappellini, and A. Mecocci, "Counting people getting
`in and out of a bus by real-time image-Sequence processing.” IVC,
`12(1):36-41, Jan. 1994.
`M. Rossi and A. BoZZoli, "Tracking and counting moving people.”
`ICIP94, pp. 212-216, 1994.
`C. R. Wren, A. Azarbayejani, T. Darrell, and A. Pentland, “Pfinder:
`Real-time tracking of the human body.” Vismod, 1995.
`L. Khoudour, L. Duvieubourg, J. P. Deparis, “Real-Time Pedestrian
`Counting by Active Linear Cameras.” JEI, 5(4):452-459, Oct. 1996.
`S. Ioffe, D. A. Forsyth, “Probabilistic Methods for Finding People.”
`IJCV. 43(1):45-68, Jun. 2001.
`M. Isard and J. MacCormick, “BraMBLe: A Bayesian Multiple-Blob
`Tracker. ICCV, 2001.
`D. M. Gavrila, “The Visual Analysis of Human Movement: A Sur
`vey.” CVIU, 73(1):82-98, Jan. 1999.
`Niels Haering and Niels da Vitoria Lobo, “Visual Event Detection.”
`Video Computing Series, Editor Mubarak Shah, 2001.
`Collins, Lipton, Kanade, Fujiyoshi, Duggins, Tsin, Tolliver,
`Enomoto, and Hasegawa, “A System for Video Surveillance and
`Monitoring: VSAM Final Report.” Technical Report CMU-RI-TR
`00-12, Robotics Institute, Carnegie Mellon University, May 2000.
`C. R. Wren, A. Azarbayejani, T. Darrell, and A. P. Pentland. “Pfinder:
`Real-Time Tracking of the Human Body.” PAMI. vol. 19, pp. 780
`784, 1997.
`M. Allmen and C. Dyer, “Long Range Spatiotemporal Motion
`Understanding Using Spatiotemporal Flow Curves.” Proc. IEEE
`CVPR, Lahaina, Maui, Hi. pp. 303-309, 1991.
`
`Canon Ex. 1035 Page 4 of 52
`
`

`

`US 8,564,661 B2
`Page 5
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`L. Wixson, “Detecting Salient Motion by Accumulating Direction
`ally Consistent Flow”, IEEE Trans. Pattern Anal. Mach. Intell., vol.
`22, pp. 774-781, Aug. 2000.
`JP Office Action issued for JP 2003-546290 on Oct. 9, 2007, with an
`English translation.
`Jemez Technology Corp., Variant iD Web-Site, www.variantid.com,
`printed Aug. 25, 2003.
`L. Wixson et al., “Detecting Salient Motion by Accumulating
`Directionally-Consistent Flow.” IEEE, 1999.
`J. P. Deparis et al., “A Device for Counting Passengers Making Use of
`Two Active Linear Cameras: Comparison of Algorithms.” IEEE, pp.
`1629-1634, 1996.
`Alan J. Lipton “Virtual Postman-An Illustrative Example of Virtual
`Video.” International Journal of Robotics and Automation, vol. 15,
`No. 1, Jan. 2000, pp. 9-16.
`Alan J. Lipton, "Virtual Postman Real Time, Interactive Virtual
`Video.” IASTED Conference on Computer Graphics and Imaging
`(CGIM'99), Palm Springs, Oct. 25-27, 1999.
`H. Fujiyoshi and A. J. Lipton, “Real-time Human Motion Analysis by
`Image Skeletonization, 'Proceedings of IEEE WACV'98, Princeton,
`NJ, 1998, pp. 15-21.
`R. T. Collins, Y. Tsin, J. R. Miller, and A. J. Lipton “Using a DEM to
`Determine Geospatial Object Trajectories.” CMU-RI-TR-98-19,
`1998.
`International Search Report issued for PCT Application No. PCT/
`US06/45625, mailed on Sep. 24, 2007.
`Written Opinion of the International Searching Authority issued for
`PCT Application No. PCT/US06/45625, mailed on Sep. 24, 2007.
`International Search Report and Written Opinion in PCT/US06/
`O2700.
`A. Selinger and L. Wixson, “Classifying Moving Objects as Rigid
`and Non-Rigid Without Correspondences.” Proceedings of DARPA
`Image Understanding Workshop, Nov. 1, 1998, pp. 341-347.
`International Search Report issued for PCT Application No. PCT/
`US06/25196, mailed Jan. 16, 2008.
`Written Opinion issued for PCT Application No. PCT/US06/25196,
`mailed Jan. 16, 2008.
`Written Opinion of the International Searching Authority issued for
`PCT Application No. PCT/US06/012556, mailed on Feb. 12, 2008.
`International Search Report of the International Searching Authority
`issued for PCT Application No. PCT/US06/012556, mailed on Feb.
`12, 2008.
`CN Office Action for CN 02822772.7 on Oct. 14, 2005 in English.
`Collins, et al. “A System for Video Surveillance and Monitoring”.
`Carnegie Mellon University, 2000.
`Kanade, et al. "Advances in Cooperative Multi-Sensor Video Surveil
`lance'. Robotics Institute, Carnegie Mellon University.
`Lipton, et al. "Automated Video Protection, Monitoring & Detec
`tion”. IEEE AESS Systems Magazine, May 2003.
`Cao, et al. "Automatic Geo-Registration for Port Surveillance'. Inter
`national Journal of Pattern Recognition and Artificial Intelligence,
`Nov. 19, 2008.
`Haering, et al. “A Semantic Event-Detection Approach and Its Appli
`cation to Detecting Hunts in Wildlife Video'. IEEE Transactions on
`Circuits and Systems for Video Technology, vol. 10, No. 6, Sep. 2000.
`Cao, et al. "Automatic Geo-Registration of Maritime Video Feeds”.
`IEEE, 2008.
`Haering, et al. “Content-Based Access of Image and Video Librar
`ies'. Proceedings on IEEE Workshop, pp. 15-25, Jun. 20, 1997.
`Baker, et al. "Autonomous Vehicle Video Aided Navigation—Cou
`pling INS and Video Approaches'.
`Lipton, Alan. “Keynote: Intelligent Video as a Force Multiplier for
`Crime Detection and Prevention'.
`Ali, et al. “Interactive Retrieval of Targets for Wide Area Surveil
`lance”. MM 10, Firenze, Italy, Oct. 2010.
`Rice, etal."Maritime Surveillance in the Intracoastal Waterway using
`Networked Underwater Acoustic Sensors integrated with a Regional
`Command Center'. 2nd International Conference on Waterside
`Security, Marina di Carrara, Italy, Nov. 2010.
`
`Heartwell, et al. “Critical Asset Protection, Perimeter Monitoring and
`Threat Detection using Automated Video Surveillance—A Technol
`ogy Overview with Case Studies”, IEEE, 2002.
`Choe, et al. “Image Transformation for Object Tracking in High
`Resolution Video'. IEEE, 2008.
`Deng, et al. "Fast Forensic Video Event Retrieval Using Geospatial
`Computing”.
`Venetianer, et al. “Performance evaluation of an intelligent video
`Surveillance system—A case study. Computer Vision and Image
`Understanding 114, pp. 1292-1302, 2010.
`Hakeem, et al. “Semantic Video Search using Natural Language
`Queries' MM’09, Beijing, China, Oct. 2009.
`Rasheed, et al. “Rapidly Deployable Video Analysis Sensor Units for
`Wide Area Surveillance'.
`Haering, Niels. “What Would You Pay for Automated Video Analy
`sis?”. 2009 Advanced Video and Signal Based Surveillance. IEEE,
`2009.
`Office Action of Nov. 25, 2011 in related EPO Application No. 06749
`276.9-1232.
`Econolite, "Autoscope Image Sensor—AIS-IV.” pp. 1-2, 2011.
`Ghanem et al., Representation and Recognition of Events in Surveil
`lance Video Using PetriNets, Proceedings of the 2004 IEEE Com
`puter Society Conference on Computer Vision and Pattern Recogni
`tion Workshops, pp. 1-9, 2004.
`Computer & Information Systems Abstracts, Keeping an Eye Out,
`2005.
`Computer & Information Systems Abstracts, ObjectVideo Extends
`its Capabilities for Border Applications as Part of Multi-Million
`Dollar Homeland Security Advanced Research Project Agency
`Grant, 2006.
`Computer & Information Systems Abstracts, QA: Security Grants Up
`for Grabs, 2004.
`CSA, Veni, Vidi, Vici Analytics: But Who Will Really Conquer?,
`2007.
`Dialog, Ex FCC Chief Makes Cisco Board an Even Dozen, Mar. 26.
`2007.
`Dialog, Iridium Names VP for Data Business, Jul. 9, 2001.
`Diamond Back, (2) Digest, Jan. 27, 2000.
`Elsevier Eng. Info. Inc., Threat Detection Expands Role, Jul. 20.
`2006.
`Gale? Cengage, Airport Selects ObjectVideo for Intrusion Detection,
`Jun. 29, 2005.
`Gale? Cengage, Appian Names Novak Biddle and MapQuest Exec to
`Its Board, Oct. 23, 2008.
`Gale? Cengage, Avinor Selects ESP Group and ObjectVideo for Sur
`veillance at Norwegian Airports, May 5, 2008.
`Gale? Cengage, Bosch Security Systems Utilizes ObjectVideo's
`Intelligent Video Surveillance Software, Apr. 6, 2005.
`Gale? Cengage, Chubb Security Taps ObjectVideo as Approved Sup
`plier, Jul. 8, 2005.
`Gale? Cengage, Companies in the News Today: About Chubb Secu
`rity, Jul. 8, 2005.
`Gale? Cengage, Companies in the News Today: About ObjectVideo
`Inc., Jul. 8, 2005.
`Gale? Cengage, comScore Names Kenneth Tarpey as CFO, Apr. 26.
`2009.
`Gale? Cengage, Cornet Offers iVDO SmartEdge Video Analytics for
`Its Ruggedized iVDO IP Encoders, May 4, 2008.
`Gale? Cengage, CoViTechnologies Adds Digital Video Analytics to
`Analog Video Surveillance Camera, Sep. 30, 2007.
`Gale? Cengage, Digiop Inks OEM Partnership with ObjectVideo,
`Apr. 30, 2008.
`Gale? Cengage, Does the eye Spy? Around-the-clock video Surveil
`lance is the Holy Grail of K-12 safety efforts. But it raises many
`questions, including whether or not the cameras are a wholly benign
`presence. Nov. 1, 2007.
`Gale? Cengage, Fibridge's New HD, IP Camera Leverages Reference
`Design from TI and ZOT, Dec. 13, 2008.
`Gale? Cengage, Former FCC Chairman Powell Joins ObjectVideo
`Board of Directors, Sep. 9, 2005.
`Gale? Cengage, Intelligent IP Camera Features Onboard Analytics,
`Apr. 13, 2007.
`
`Canon Ex. 1035 Page 5 of 52
`
`

`

`US 8,564,661 B2
`Page 6
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Gale? Cengage, Intelligent Video Server Promotes OEM Camera
`Development, Apr. 30, 2007.
`Gale? Cengage, Lanaccess Encoders to Include ObjectVideo's
`Analytics, Sep. 21, 2005.
`Gale? Cengage, MangoDSP and ObjectVideo to Provide Integrated
`Intelligent Video Surveillance Platforms, Apr. 10, 2005.
`Gale? Cengage, MangoDSP. ObjectVideo Join Forces on Surveil
`lance Solutions, Apr. 7, 2005.
`Gale? Cengage, Next Steps for Technology: How U.S. R&D Efforts
`Have Pushed Development Forward, Sep. 11, 2006.
`Gale? Cengage, ObjectVideo and TI Get Win with Hangzhou Hikvi
`sion Digital Technology, Apr. 1, 2007.
`Gale? Cengage, ObjectVideo Develops New Scanning Camera Tech
`nology to Protect U.S. Borders, Jul. 19, 2006.
`Gale? Cengage, ObjectVideo Gets $3.2 Million Pacts from Homeland
`Security Advanced Research Project Agency (HSARPA), May 13,
`2005.
`Gale? Cengage, ObjectVideo Gets DARPA Contract to SupportNext
`Generation Surveillance System, Jan. 28, 2008.
`Gale? Cengage, ObjectVideo Named 2005 "Technology Pioneer.”
`Dec. 6, 2004.
`Gale? Cengage, ObjectVideoNamed a Red Herring 100 Company for
`Second Year in a Row, May 13, 2005.
`Gale? Cengage, ObjectVideo Offers Intelligent Video Network Pro
`tocol for Standardized Analytics Communication and Integration,
`Sep. 25, 2007.
`Gale? Cengage, ObjectVideo Provides Intelligent Video Software to
`Port, Jul 13, 2005.
`Gale? Cengage, ObjectVideo Releases OnBoard Analytics on ARM
`Processors, Apr. 5, 2011.
`Gale? Cengage, ObjectVideo Releases Software for Intel Architec
`ture Systems, Apr. 21, 2008.
`Gale? Cengage, ObjectVideo Signs Distributor for Asia, Dec. 14.
`2005.
`Gale? Cengage, ObjectVideo Software Part of Security System at
`Memphis/Shelby County Airport, Jun. 21, 2005.
`Gale? Cengage, ObjectVideo to Showcase the Latest IntelligentSolu
`tions at ASIS, Sep. 23, 2007.
`Gale? Cengage, ObjectVideo to Spotlight IntelligentSolutions at ISC
`West 2007, Mar. 26, 2007.
`Gale? Cengage, ObjectVideo Touts Early Adoption of its Protocol,
`Mar. 31, 2008.
`Gale? Cengage, ObjectVideo, Security Services & Technologies
`Announce Integration Partnership, Apr. 6, 2005.
`Gale? Cengage, People in Wireless News: Peter Currie Joins Critical
`Path Board, Feb. 2, 2004.
`Gale? Cengage, Phoenix IVS, ObjectVideo and MangoDSP to
`Launch Intelligent Video Module, Aug. 10, 2005.
`Gale? Cengage, Proximex Upgrades Surveillint, Dec. 21, 2008.
`Gale? Cengage, Puerto Rico Public Housing Beefs Up Security Via
`Intelligent Surveillance System from Avant Technologies and
`ObjectVideo, Sep. 19, 2007.
`Gale? Cengage, Software Creates Intelligent Devices for Video Sys
`tems, Mar. 31, 2006.
`Gale? Cengage, Software Delivers Industry-Specific Intelligent
`Video, May 10, 2006.
`Gale? Cengage, Synectic Systems Group and ObjectVideo Ink OEM
`Pact, Feb. 28, 2008.
`Gale? Cengage, Video—DSPs Focus on Vision Systems, May 21,
`2007.
`Gale? Cengage, VidSys Partners with Security Companies for PSIM
`Thought Leadership Site, Oct. 23, 2008.
`Gale? Cengage, Wireless Channel: Chubb Security to Resell
`ObjectVideo Technology, Jul. 9, 2005.
`Information Week. Curtis Joins CMGI, May 8, 2000.
`Lipton, Stationary Target Detection Using the ObjectVideo Surveil
`lance System, 2007.
`Lipton, The Intelligent Vision Sensor: Turning Video into Informa
`tion, 2007.
`
`Lipton, Video Verification of Point of Sale Transactions, 2007.
`Abstract and documents related to the contract for Department of
`Defense Award No. 81276, A Unified Framework for False Alarm
`Reduction Using Scene Context from Airborne Sensors, Identifica
`tion No. 07SB2-0308, W31 P4Q-08-C-0108, 2007.
`Abstract and documents related to the contract for Department of
`Defense Award No. 63547. Automated Video Surveillance at Night,
`Identification No. 03ST1-004, DAAHO 103CR 276, 2003.
`Abstract and documents related to the contract for Department of
`Defense Award No. 82715, Context-Sensitive Content Extraction and
`Scene Understanding, Identification No. N071-085-0199. N00014
`08-C-0639, 2007.
`Abstract and documents related to the contract for Department of
`Defense Award No. 65022, Human Activity Recognition, Identifica
`tion No. NO31-1723, NO0014-03-M-0256, 2003.
`Abstract and documents related to the contract for Department of
`Defense Award No. 76719. Identify and Track Important Assets,
`Identification No. S052-007-0063, H92222-06-P-0012, 2005.
`Abstract and documents related to the contract for Department of
`Defense Award No. 78523, Intelligent Retrieval of Surveillance
`Imagery, Identification No. N062-147-0567, N00014-08-C0110,
`2006.
`Abstract and documents related to the contract for Department of
`Defense Award No. 59579, Mast-Mounted In-Port Video Surveil
`lance System, Identification No. N021-1610; N00014-03-C-0232,
`2002.
`Abstract and documents related to the contract for Department of
`Defense Award No. 83229, Robust Multiple Target Tracking, Iden
`tification No. A074-007-0091, W911NF-08-C-0068, 2007.
`Abstract and documents related to the contract for National Science
`Foundation Award No. 69404, SBIR Phase I: Learning Context,
`Identification No. 0340175, 0340175, 2004.
`Abstract and documents related to the contract for Department of
`Defense Award No. 58472, Streaming Multimedia for the Warfighter,
`Identification No. 99SB1-0357, DAAHO1-02-C-R193, 2002.
`Abstract and documents related to the contract for Department of
`Defense Award No. 55489, Video Detection, Tracking and Classifi
`cation, Identification No. 99ST1-004, 2001-2002.
`Abstract and documents related to the contract for Department of
`Defense Award No. 85007, Wide Area Video Motion Blur Elimina
`tion, Identification No. 08ST1-0147, W31 P4Q-08-C-0470, 2008.
`ObjectVideo, Amtrak Lags in Implementing Security Technologies:
`Madrid