PTO/SB/05 (09-04)
`Approved for use through 07/31/2006. 0MB 0651-0032
`U.S. Patent and Trademark Office. U.S. DEPARTMENT OF COMMERCE
`Under the Paperwork Reduction Act of 1995, no ciersons are reoulred to resnnnd to a collection of Information unless It dis~1~,,., a varld 0MB control number.
`l 37112-213547
`Attorney Docket No.
`UTILITY
`l Peter L. Venetianer
`First Inventor
`PATENT APPLICATION
`TRANSMITTAL
`(ONLY FOR NEW NONPROV/SIONAL APPL/CA T/ONS UNDER
`37 CFR 1.53(8))
`
`Title VIDEO SURVEILLANCE SYSTEM
`Express Mail Label No. l
`
`APPLICATION ELEMENTS
`See MPEP chapter 600 concerning utility patent application contents.
`
`(Submit an original and a dupHcate for fee processing)
`
`·
`
`See 37 CFR 1.27.
`
`1. 0 Fee Transmittal Form (e.g., PTO/SB/17)
`2 0 Applicant claims small entity status.
`3. 0 Specification
`4. 0 Drawlng(s) (35 U.S.C. 113)
`a. 0 Newly executed (original or copy)
`
`[Total Pages
`Both the Claims and <ibstract rrust start on a new page
`(For lnfo,mation on the preferred arrangement, see MPEP 608.01(a)J
`15
`I
`---
`4
`---
`
`48
`
`I
`
`I
`
`[Total Sheets
`
`(Total Sheets
`
`5. Oath or Declaration
`
`Commissioner for Patents
`ADDRESS TO: P.O. Box 1450
`Alexandria, VA 22313-1450
`ACCOMPANYING APPLICATION PARTS
`
`9. 0 Assignment Papers (cover sheet & document(s))
`
`Name of Assignee:
`
`I
`
`ObjectVideo, Inc.
`
`I
`
`10 □ 37 CFR 3.73(b) Statement 0Powerof
`·
`(when there is an assignee)
`Attorney
`
`11. D English Translation Document (if applicable)
`12. D Information Disclosure Statement (PTO/SB/08 or PT0-144g)
`D Coples of citations attached
`13. D Preliminary Amendment
`14. 0 Return Receipt Postcard (MPEP 503)
`15 D Certified Copy of Priority Document(s)
`16. D Nonpublication Request under 35 U.S.C.122 (b)(2)(B)(i).
`
`b DA copy from a prior application (37 CFR 1.63(d))
`·
`(for contlnualionldMsional wllh Box 18 completed)
`1. □ DEbETION QF INVENTOR(l;!)
`Signed statement attached deleting inventor(s)
`named in the prior application,
`see 37 CFR 1.63(d)(2) and 1.33(b).
`
`6. 0 Application Data Sheet. See 37 CFR 1.76
`
`7 □ CD-ROM or CD-R in duplicate, large table or
`·
`Computer Program (Appendix)
`
`8. Nucleotide and/or Amino Acid Sequence Submission
`(If applicable, items a. - c. are required)
`
`D Landscape Table on CD
`a. D Computer Readable Form (CRF)
`i. D CD-ROM or CD-R (2 copies); or
`ii. D Paper
`c. D Statements verifying identity of above copies
`
`b.
`
`Specification Sequence Listing on:
`
`(Should be specif,cafly itemized)
`
`·
`
`(if fore'tf}n priority is claimed)
`
`Applicant must attach form PTO/SB/35 or Its equivalent
`17. □ 0ther. I
`
`I
`
`18. If a CONTINUING APPLICATION, check appropriate box, and supply the requisite information below and In the first sentence of the
`specification following the title, or In an Application Data Sheet under 37 CFR 1. 76:
`of prior application No.:
`
`D Continuation D Divisional 0 Continuation-In-part (CIP)
`
`Art Unit:
`
`09/987707
`N/A
`
`Prior application Information: Examiner
`
`Not Yet Assigned
`19. CORRESPONDENCE ADDRESS
`
`0 The address associated with Customer Number. I
`
`26694
`
`I
`
`OR D Correspondence address below
`
`Name
`
`VENABLE LLP
`
`Address P.O. Box 34385
`
`I State I
`I Zip Code I 20043-9998
`Washington
`City
`D.C.
`I Telephone 202-344-8017
`I Fax I 202-344-8300
`Country U.S.A.
`----
`Signature ~ Date
`February 15, 2005
`-
`Registration No.
`rey
`. Gluck, Ph.D.
`Name,. -..,,,,
`(Attorney/Agent)
`
`44,457
`
`--
`
`

`

`Approved for use through 12/08/2005. 0MB 0651-0032
`U.S. Patent and Trademark Office: U.S. DEPARTMENT OF COMMERCE
`Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of Information unless it displays a valid 0MB control number.
`
`FEE TRANS MITT AL
`for FY 2005
`
`Patent fees are subject to annual revision.
`
`TOTAL AMOUNT OF PAYMENT
`
`1,040
`
`1.
`
`till
`
`Deposit
`Account
`Number
`
`The Commissioner is hereby authorized to charge
`Indicated fees and credit any over payments to:
`
`22-0261
`
`Venable LLP
`
`Deposit
`Account
`Name
`181 Charge Any Additional Fee Required Under 37 CFR 1.16 and 1.17
`181 Applicant claims small entity status. See 37 CFR 1.27
`2. D Payment Enclosed:
`
`0 Check
`
`□ Credit card
`
`D Money
`Order
`FEE CALCULATION
`
`□ Other
`
`1.
`Large
`Fee
`Code
`1011
`1012
`1013
`1014
`1005
`1081
`1082
`1083
`1084
`1085
`
`BASIC FILING FEE
`Entity Small Entity
`Fee
`Fee($)
`Fee
`Code
`($)
`2011
`150
`2012
`100
`2013
`100
`2014
`150
`2005
`100
`2081
`125
`2082
`125
`2083
`125
`2084
`125
`2085
`125
`
`300
`200
`200
`300
`200
`250
`250
`250
`250
`250
`
`Fee Description
`
`Utility filing fee
`Design filing fee
`Plant filing fee
`Reissue filing fee
`Provisional filling fee
`Utility App. Size Fee
`Design App Size Fee
`Plant App. Size Fee
`Reissue App Size Fee
`Prov. App Size Fee
`
`Fee Paid
`150
`
`SUBTOTAL (1)
`
`($)150
`
`2. EXTRA CLAIM FEES
`
`Extra
`Claims
`
`Fee from
`below
`I XI 25
`
`Fee
`Paid
`=QQD
`
`Fee
`Paid
`250
`
`100
`
`Application Number
`
`Complete If Known
`Not Yet Assigned
`
`Filing Date
`
`Concurrently Herewith
`
`First Named Inventor Peter L. Venetianer
`Examiner Name
`Not Yet Assi ned
`Grou
`/Art Unit
`Not Yet Assi ned
`Attorney Docket No.
`37112-213547
`
`FEE CALCULATION continued
`
`Fee Description
`
`3. ADDITIONAL FEES
`Lrg Ent Fee
`Sm Ent
`Fee
`Fee
`Code Fee
`Code
`(Sl
`($)
`500
`2111
`250
`100
`2112
`50
`2113
`150
`300
`500
`2114
`250
`2311
`200
`100
`2312
`130
`65
`160
`2313
`80
`600
`2314
`300
`2051
`130
`65
`50
`2052
`25
`
`1111
`1112
`1113
`1114
`1311
`1312
`1313
`1314
`1051
`1052
`
`1053
`1812
`1804
`
`130
`2,520
`920'
`
`1805
`
`1,840"
`
`1805
`
`Utility Search Fee
`Design Search Fee
`Plant Search Fee
`Reissue Search Fee
`Utility Examination Fee
`Design Examination Fee
`Plant Examination Fee
`Reissue Examination Fee
`Surcharge • late filing fee or oath
`Surcharge • late provisional filing fee
`or cover sheet.
`1053
`130
`Non-English specification
`1812
`2,520 For filing a request for reexamination
`1804 920' Requesting publication of SIR prior to
`Examiner action
`1,840' Requesting publication of SIR after
`Examiner action
`Extension for reply within first month
`
`120
`450
`
`1,020
`1,590
`
`2,160
`500
`500
`1,000
`
`1251
`1252
`
`1253
`1254
`
`1255
`1401
`1402
`1403
`
`1451
`
`2215
`2252
`
`60
`225
`
`2253
`2254
`
`510
`795
`
`2255
`2401
`2402
`2403
`
`1,510
`
`1451
`
`Extension for reply within second
`month
`Extension for reply within third month
`Extension for reply within fourth
`month
`1080 Extension for reply within fifth month
`250
`Notice of Appeal
`250
`Filing a brief in support of an appeal
`500
`Request for oral hearing
`1,510 Petition to institute a public use
`proceeding
`250
`Petition to revive - unavoidable
`750
`Petition to revive - unintentional
`700
`Utility issue fee (or reissue)
`400
`Design Issue fee
`550
`Plant Issue fee
`50
`Processing fee under 37 CFR 1. 17 (q)
`Submission of Information Disclosure
`Stmt
`Recording each patent assignment
`per proper1y (times number of
`properties)
`Filing a submission after final rejection
`(37 CFR § 1.129(a))
`For each additional invention to be
`examined (37 CFR § 1.129(b))
`
`40
`
`395
`
`395
`
`180
`
`40
`
`1452
`1453
`1501
`1502
`1503
`1807
`
`1806
`
`500
`1,500
`1,400
`800
`1,100
`50
`
`180
`
`8021
`
`40
`
`1809
`
`790
`
`1810
`
`790
`
`2452
`2453
`2501
`2502
`2503
`1807
`
`1806
`
`8021
`
`2809
`
`2810
`
`1801
`
`790
`
`2801
`
`395 Request for Continued Examination (RCE)
`
`Other fee (specify)
`
`otal Claims I 40
`
`ultiple
`Dependent
`
`Large
`Fee
`Code
`1202
`1201
`1203
`
`1204
`
`1205
`
`1-20 •• =I 20
`.3•• =EJ X~=~
`xi 180 I=~
`I
`
`D
`I 0
`
`Entity Small
`Fee
`Fee
`Code
`($)
`50
`2202
`200
`2201
`2204
`360
`
`200
`
`50
`
`2204
`
`2205
`
`Fee Description
`
`Entity
`Fee
`($)
`Claims In excess of 20
`25
`Independent claims In excess of 3
`100
`180 Multiple dependent claim, If not paid
`•• Reissue independent claims in
`excess of three
`•• Reissue claims in excess of 20 and
`over original patent
`
`100
`
`25
`
`SUBTOTAL (2)
`
`1 ($)500
`
`•Reduced by Basic Filing Fee Paid
`
`SUBTOTAL (3)
`
`($)390
`
`"or number previously paid, if greater; For Reissues, see above
`
`SUBMITTED BY
`Name (PrinVType)
`
`Com lete if a
`I/cable
`Jeffrey W, Gluck, Ph.D.
`
`Reg No. Attomey/Agent)
`
`44,457
`
`Telephone
`
`202-344-4000
`
`Signature
`
`e February 15, 2005
`
`VENABLE
`PC Docs No. 620537
`
`WARN! G: Information on this form may become public. Credit card Information should not be Included on this form. Provide credit
`card Information and authorization on PTO-2038. SEND TO: Commissioner for Patents, Mall Stop Patent Appllcatlon, Alexandria, VA
`22313-1450.
`
`

`

`PTO/SB/05 (09-04)
`Approved for use through 07/31/2006. 0MB 0651-0032
`U.S. Patent and Trademark Office. U.S. DEPARTMENT OF COMMERCE
`Under the Paperwork Reduction Act of 1995, no ciersons are reoulred to resnnnd to a collection of Information unless It dis~1~,,., a varld 0MB control number.
`l 37112-213547
`Attorney Docket No.
`UTILITY
`l Peter L. Venetianer
`First Inventor
`PATENT APPLICATION
`TRANSMITTAL
`(ONLY FOR NEW NONPROV/SIONAL APPL/CA T/ONS UNDER
`37 CFR 1.53(8))
`
`Title VIDEO SURVEILLANCE SYSTEM
`Express Mail Label No. l
`
`APPLICATION ELEMENTS
`See MPEP chapter 600 concerning utility patent application contents.
`
`(Submit an original and a dupHcate for fee processing)
`
`·
`
`See 37 CFR 1.27.
`
`1. 0 Fee Transmittal Form (e.g., PTO/SB/17)
`2 0 Applicant claims small entity status.
`3. 0 Specification
`4. 0 Drawlng(s) (35 U.S.C. 113)
`a. 0 Newly executed (original or copy)
`
`[Total Pages
`Both the Claims and <ibstract rrust start on a new page
`(For lnfo,mation on the preferred arrangement, see MPEP 608.01(a)J
`15
`I
`---
`4
`---
`
`48
`
`I
`
`I
`
`[Total Sheets
`
`(Total Sheets
`
`5. Oath or Declaration
`
`Commissioner for Patents
`ADDRESS TO: P.O. Box 1450
`Alexandria, VA 22313-1450
`ACCOMPANYING APPLICATION PARTS
`
`9. 0 Assignment Papers (cover sheet & document(s))
`
`Name of Assignee:
`
`I
`
`ObjectVideo, Inc.
`
`I
`
`10 □ 37 CFR 3.73(b) Statement 0Powerof
`·
`(when there is an assignee)
`Attorney
`
`11. D English Translation Document (if applicable)
`12. D Information Disclosure Statement (PTO/SB/08 or PT0-144g)
`D Coples of citations attached
`13. D Preliminary Amendment
`14. 0 Return Receipt Postcard (MPEP 503)
`15 D Certified Copy of Priority Document(s)
`16. D Nonpublication Request under 35 U.S.C.122 (b)(2)(B)(i).
`
`b DA copy from a prior application (37 CFR 1.63(d))
`·
`(for contlnualionldMsional wllh Box 18 completed)
`1. □ DEbETION QF INVENTOR(l;!)
`Signed statement attached deleting inventor(s)
`named in the prior application,
`see 37 CFR 1.63(d)(2) and 1.33(b).
`
`6. 0 Application Data Sheet. See 37 CFR 1.76
`
`7 □ CD-ROM or CD-R in duplicate, large table or
`·
`Computer Program (Appendix)
`
`8. Nucleotide and/or Amino Acid Sequence Submission
`(If applicable, items a. - c. are required)
`
`D Landscape Table on CD
`a. D Computer Readable Form (CRF)
`i. D CD-ROM or CD-R (2 copies); or
`ii. D Paper
`c. D Statements verifying identity of above copies
`
`b.
`
`Specification Sequence Listing on:
`
`(Should be specif,cafly itemized)
`
`·
`
`(if fore'tf}n priority is claimed)
`
`Applicant must attach form PTO/SB/35 or Its equivalent
`17. □ 0ther. I
`
`I
`
`18. If a CONTINUING APPLICATION, check appropriate box, and supply the requisite information below and In the first sentence of the
`specification following the title, or In an Application Data Sheet under 37 CFR 1. 76:
`of prior application No.:
`
`D Continuation D Divisional 0 Continuation-In-part (CIP)
`
`Art Unit:
`
`09/987707
`N/A
`
`Prior application Information: Examiner
`
`Not Yet Assigned
`19. CORRESPONDENCE ADDRESS
`
`0 The address associated with Customer Number. I
`
`26694
`
`I
`
`OR D Correspondence address below
`
`Name
`
`VENABLE LLP
`
`Address P.O. Box 34385
`
`I State I
`I Zip Code I 20043-9998
`Washington
`City
`D.C.
`I Telephone 202-344-8017
`I Fax I 202-344-8300
`Country U.S.A.
`----
`Signature ~ Date
`February 15, 2005
`-
`Registration No.
`rey
`. Gluck, Ph.D.
`Name,. -..,,,,
`(Attorney/Agent)
`
`44,457
`
`--
`
`

`

`Approved for use through 12/08/2005. 0MB 0651-0032
`U.S. Patent and Trademark Office: U.S. DEPARTMENT OF COMMERCE
`Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of Information unless it displays a valid 0MB control number.
`
`FEE TRANS MITT AL
`for FY 2005
`
`Patent fees are subject to annual revision.
`
`TOTAL AMOUNT OF PAYMENT
`
`1,040
`
`1.
`
`till
`
`Deposit
`Account
`Number
`
`The Commissioner is hereby authorized to charge
`Indicated fees and credit any over payments to:
`
`22-0261
`
`Venable LLP
`
`Deposit
`Account
`Name
`181 Charge Any Additional Fee Required Under 37 CFR 1.16 and 1.17
`181 Applicant claims small entity status. See 37 CFR 1.27
`2. D Payment Enclosed:
`
`0 Check
`
`□ Credit card
`
`D Money
`Order
`FEE CALCULATION
`
`□ Other
`
`1.
`Large
`Fee
`Code
`1011
`1012
`1013
`1014
`1005
`1081
`1082
`1083
`1084
`1085
`
`BASIC FILING FEE
`Entity Small Entity
`Fee
`Fee($)
`Fee
`Code
`($)
`2011
`150
`2012
`100
`2013
`100
`2014
`150
`2005
`100
`2081
`125
`2082
`125
`2083
`125
`2084
`125
`2085
`125
`
`300
`200
`200
`300
`200
`250
`250
`250
`250
`250
`
`Fee Description
`
`Utility filing fee
`Design filing fee
`Plant filing fee
`Reissue filing fee
`Provisional filling fee
`Utility App. Size Fee
`Design App Size Fee
`Plant App. Size Fee
`Reissue App Size Fee
`Prov. App Size Fee
`
`Fee Paid
`150
`
`SUBTOTAL (1)
`
`($)150
`
`2. EXTRA CLAIM FEES
`
`Extra
`Claims
`
`Fee from
`below
`I XI 25
`
`Fee
`Paid
`=QQD
`
`Fee
`Paid
`250
`
`100
`
`Application Number
`
`Complete If Known
`Not Yet Assigned
`
`Filing Date
`
`Concurrently Herewith
`
`First Named Inventor Peter L. Venetianer
`Examiner Name
`Not Yet Assi ned
`Grou
`/Art Unit
`Not Yet Assi ned
`Attorney Docket No.
`37112-213547
`
`FEE CALCULATION continued
`
`Fee Description
`
`3. ADDITIONAL FEES
`Lrg Ent Fee
`Sm Ent
`Fee
`Fee
`Code Fee
`Code
`(Sl
`($)
`500
`2111
`250
`100
`2112
`50
`2113
`150
`300
`500
`2114
`250
`2311
`200
`100
`2312
`130
`65
`160
`2313
`80
`600
`2314
`300
`2051
`130
`65
`50
`2052
`25
`
`1111
`1112
`1113
`1114
`1311
`1312
`1313
`1314
`1051
`1052
`
`1053
`1812
`1804
`
`130
`2,520
`920'
`
`1805
`
`1,840"
`
`1805
`
`Utility Search Fee
`Design Search Fee
`Plant Search Fee
`Reissue Search Fee
`Utility Examination Fee
`Design Examination Fee
`Plant Examination Fee
`Reissue Examination Fee
`Surcharge • late filing fee or oath
`Surcharge • late provisional filing fee
`or cover sheet.
`1053
`130
`Non-English specification
`1812
`2,520 For filing a request for reexamination
`1804 920' Requesting publication of SIR prior to
`Examiner action
`1,840' Requesting publication of SIR after
`Examiner action
`Extension for reply within first month
`
`120
`450
`
`1,020
`1,590
`
`2,160
`500
`500
`1,000
`
`1251
`1252
`
`1253
`1254
`
`1255
`1401
`1402
`1403
`
`1451
`
`2215
`2252
`
`60
`225
`
`2253
`2254
`
`510
`795
`
`2255
`2401
`2402
`2403
`
`1,510
`
`1451
`
`Extension for reply within second
`month
`Extension for reply within third month
`Extension for reply within fourth
`month
`1080 Extension for reply within fifth month
`250
`Notice of Appeal
`250
`Filing a brief in support of an appeal
`500
`Request for oral hearing
`1,510 Petition to institute a public use
`proceeding
`250
`Petition to revive - unavoidable
`750
`Petition to revive - unintentional
`700
`Utility issue fee (or reissue)
`400
`Design Issue fee
`550
`Plant Issue fee
`50
`Processing fee under 37 CFR 1. 17 (q)
`Submission of Information Disclosure
`Stmt
`Recording each patent assignment
`per proper1y (times number of
`properties)
`Filing a submission after final rejection
`(37 CFR § 1.129(a))
`For each additional invention to be
`examined (37 CFR § 1.129(b))
`
`40
`
`395
`
`395
`
`180
`
`40
`
`1452
`1453
`1501
`1502
`1503
`1807
`
`1806
`
`500
`1,500
`1,400
`800
`1,100
`50
`
`180
`
`8021
`
`40
`
`1809
`
`790
`
`1810
`
`790
`
`2452
`2453
`2501
`2502
`2503
`1807
`
`1806
`
`8021
`
`2809
`
`2810
`
`1801
`
`790
`
`2801
`
`395 Request for Continued Examination (RCE)
`
`Other fee (specify)
`
`otal Claims I 40
`
`ultiple
`Dependent
`
`Large
`Fee
`Code
`1202
`1201
`1203
`
`1204
`
`1205
`
`1-20 •• =I 20
`.3•• =EJ X~=~
`xi 180 I=~
`I
`
`D
`I 0
`
`Entity Small
`Fee
`Fee
`Code
`($)
`50
`2202
`200
`2201
`2204
`360
`
`200
`
`50
`
`2204
`
`2205
`
`Fee Description
`
`Entity
`Fee
`($)
`Claims In excess of 20
`25
`Independent claims In excess of 3
`100
`180 Multiple dependent claim, If not paid
`•• Reissue independent claims in
`excess of three
`•• Reissue claims in excess of 20 and
`over original patent
`
`100
`
`25
`
`SUBTOTAL (2)
`
`1 ($)500
`
`•Reduced by Basic Filing Fee Paid
`
`SUBTOTAL (3)
`
`($)390
`
`"or number previously paid, if greater; For Reissues, see above
`
`SUBMITTED BY
`Name (PrinVType)
`
`Com lete if a
`I/cable
`Jeffrey W, Gluck, Ph.D.
`
`Reg No. Attomey/Agent)
`
`44,457
`
`Telephone
`
`202-344-4000
`
`Signature
`
`e February 15, 2005
`
`VENABLE
`PC Docs No. 620537
`
`WARN! G: Information on this form may become public. Credit card Information should not be Included on this form. Provide credit
`card Information and authorization on PTO-2038. SEND TO: Commissioner for Patents, Mall Stop Patent Appllcatlon, Alexandria, VA
`22313-1450.
`
`

`

`APPLICATION FOR UNITED STATES PATENT
`
`INVENTORS:
`
`PETER L. VENETIANER, McLean, Virginia
`ALAN J. LIPTON, Herndon, Virginia
`ANDREW J. CHOSAK, Arlington, Virginia
`MATTHEW F. FRAZIER, Arlington, Virginia
`NIELS HAERING, Reston, Virginia
`GARY W. MYERS, Ashburn, Virginia
`WEIHONG YIN, Herndon, Virginia
`ZHONG ZHANG, Herndon, Virginia
`
`TITLE:
`
`VIDEO SURVEILLANCE SYSTEM EMPLOYING
`VIDEO PRIMITIVES
`
`ATTORNEYS/ AGENTS:
`
`VENABLELLP
`Post Office Box 34385
`Washington, DC 20043-9998
`Telephone: (202) 344~4000
`Facsimile: (202) 344-8300
`
`ATTORNEY DOCKET NO.:
`
`37112-213547
`
`

`

`VIDEO SURVEILLANCE SYSTEM EMPLOYING VIDEO PRIMITIVES
`
`CROSS-REFERENCE TO RELATED APPLICATIONS
`
`[1]
`
`This application is a continuation-in-part of U.S. Patent Application No.
`
`09/987,707, filed on November 15, 2001, which claims the priority of U.S. Patent Application
`
`No. 09/694,712, filed on October 24, 2000, both of which are incorporated herein by reference.
`
`BACKGROUND OF THE INVENTION
`
`Field of the Invention
`
`[2]
`
`The invention relates to a system for automatic video surveillance employing
`
`video primitives.
`
`References
`
`[3]
`
`For the convenience of the reader, the references referred to herein are listed
`
`below. In the specification, the numerals within brackets refer to respective references. The
`
`listed references are incorporated herein by reference.
`
`[4]
`
`[5]
`
`The following references describe moving target detection:
`
`{ 1} A. Lipton, H. Fujiyoshi and R. S. Patil, "Moving Target Detection and
`
`Classification from Real-Time Video," Proceedings of IEEE W ACV '98, Princeton, NJ, 1998,
`
`pp. 8-14.
`
`[6]
`
`{2} W.E.L. Grimson, et al., "Using Adaptive Tracking to Classify and Monitor
`
`Activities in a Site", CVPR, pp. 22-29, June 1998.
`
`[7]
`
`{3} A.J. Lipton, H. Fujiyoshi, R.S. Patil, "Moving Target Classification and
`
`Tracking from Real-time Video," IUW, pp. 129-136, 1998.
`
`(37112-213547)
`
`

`

`[8]
`
`{4} T.J. Olson and F.Z. Brill, "Moving Object Detection and Event Recognition
`
`Algorithm for Smart Cameras," IUW, pp. 159-175, May 1997.
`
`[9]
`
`The following references describe detecting and tracking humans:
`
`[ 1 O]
`
`{ 5} A. J. Lipton, "Local Application of Optical Flow to Analyse Rigid Versus
`
`Non-Rigid Motion," International Conference on Computer Vision, Corfu, Greece, September
`
`1999.
`
`[ 11]
`
`{ 6} F. Bartolini, V. Cappellini, and A. Mecocci, "Counting people getting in and
`
`out of a bus by real-time image-sequence processing," IVC, 12(1):36-41, January 1994.
`
`[12]
`
`{7} M. Rossi and A. Bozzoli, "Tracking and counting moving people," ICIP94.
`
`pp. 212-216. 1994.
`
`[13]
`
`{8} C.R. Wren, A. Azarbayejani, T. Darrell, and A. Pentland, "Pfinder: Real-
`
`time tracking of the human body," Vismod, 1995.
`
`[14]
`
`{9} L. Khoudour, L. Duvieubourg, J.P. Deparis, "Real-Time Pedestrian Counting
`
`by Active Linear Cameras," JEI, 5(4):452-459. October 1996.
`
`[15]
`
`{10} S. Joffe, D.A. Forsyth, "Probabilistic Methods for Finding People," IJCV.
`
`43(1):45-68, June 2001.
`
`[16]
`
`{ 11} M. Isard and J. MacCormick, "BraMBLe: A Bayesian Multiple-Blob
`
`Tracker," ICCV. 2001.
`
`[17]
`
`The following references describe blob analysis:
`
`[18]
`
`{12} D.M. Gavrila, "The Visual Analysis of Human Movement: A Survey,"
`
`CVIU, 73(1):82-98, January 1999.
`
`[ 19]
`
`{ 13} Niels Haering and Niels da Vitoria Lobo, "Visual Event Detection," Video
`
`Computing Series. Editor Mubarak Shah, 2001.
`
`- 2 -
`
`(37112-213547)
`
`

`

`[20]
`
`The following references describe blob analysis for trucks, cars, and people:
`
`[21]
`
`{ 14} Collins, Lipton, Kanade, Fujiyoshi, Duggins, Tsin, Tolliver, Enomoto, and
`
`Hasegawa, "A System for Video Surveillance and Monitoring: VSAM Final Report," Technical
`
`Report CMU-RI-TR-00-12, Robotics Institute, Carnegie Mellon University, May 2000.
`
`[22]
`
`{ 15} Lipton, Fujiyoshi, and Patil, "Moving Target Classification and Tracking
`
`from Real-time Video," 98 Dama IUW, Nov. 20-23, 1998.
`
`[23]
`
`The following reference describes analyzing a single-person blob and its contours:
`
`[24]
`
`{16} C.R. Wren, A. Azarbayejani, T. Darrell, and A.P. Pentland. "Pfinder: Real-
`
`Time Tracking of the Human Body," PAMI, vol 19, pp. 780-784, 1997.
`
`[25]
`
`The following reference describes internal motion of blobs, including any motion-
`
`based segmentation:
`
`[26]
`
`{ 17} M. Allmen and C. Dyer, "Long--Range Spatiotemporal Motion
`
`Understanding Using Spatiotemporal Flow Curves," Proc. IEEE CVPR, Lahaina, Maui, Hawaii,
`
`pp. 303-309, 1991.
`
`[27]
`
`{ 18} L. Wixson, "Detecting Salient Motion by Accumulating Directionally
`
`Consistent Flow", IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, pp. 774-781, Aug, 2000.
`
`Background of the Invention
`
`[28] Video surveillance of public spaces has become extremely widespread and
`
`accepted by the general public. Unfortunately, conventional video surveillance systems produce
`
`such prodigious volumes of data that an intractable problem results in the analysis of video
`
`surveillance data.
`
`- 3 -
`
`(37112-213547)
`
`

`

`[29] A need exists to reduce the amount of video surveillance data so analysis of the
`
`video surveillance data can be conducted.
`
`[30] A need exists to filter video surveillance data to identify desired portions of the
`
`video surveillance data.
`
`SUMMARY OF THE INVENTION
`
`[31] An object of the invention is to reduce the amount of video surveillance data so
`
`analysis of the video surveillance data can be conducted.
`
`[32] An object of the invention is to filter video surveillance data to identify desired
`
`portions of the video surveillance data.
`
`[33] An object of the invention is to produce a real time alarm based on an automatic
`
`detection of an event from video surveillance data.
`
`[34] An object of the invention is to integrate data from surveillance sensors other than
`
`video for improved searching capabilities.
`
`[35] An object of the invention is to integrate data from surveillance sensors other than
`
`video for improved event detection capabilities
`
`[36] The invention includes an article of manufacture, a method, a system, and an
`
`apparatus for video surveillance.
`
`[37] The article of manufacture of the invention includes a computer-readable medium
`
`comprising software for a video surveillance system, comprising code segments for operating the
`
`video surveillance system based on video primitives.
`
`[38] The article of manufacture of the invention includes a computer-readable medium
`
`comprising software for a video surveillance system, comprising code segments for accessing
`
`- 4 -
`
`(37112-213547)
`
`

`

`archived video primitives, and code segments for extracting event occurrences from accessed
`
`archived video primitives.
`
`[39]
`
`The system of the invention includes a computer system including a computer•
`
`readable medium having software to operate a computer in accordance with the invention.
`
`[40]
`
`The apparatus of the invention includes a computer including a computer-readable
`
`medium having software to operate the computer in accordance with the invention.
`
`[ 41] The article of manufacture of the invention includes a computer-readable medium
`
`having software to operate a computer in accordance with the invention.
`
`[42] Moreover, the above objects and advantages of the invention are illustrative, and
`
`not exhaustive, of those that can be achieved by the invention. Thus, these and other objects and
`
`advantages of the invention will be apparent from the description herein, both as embodied
`
`herein and as modified in view of any variations which will be apparent to those skilled in the
`
`art.
`
`Definitions
`
`[43] A "video" refers to motion pictures represented in analog and/or digital form.
`
`Examples of video include: television, movies, image sequences from a video camera or other
`
`observer, and computer-generated image sequences.
`
`[44] A "frame" refers to a particular image or other discrete unit within a video.
`
`[ 45] An "object" refers to an item of interest in a video. Examples of an object
`
`include: a person, a vehicle, an animal, and a physical subject.
`
`• 5 -
`
`(37112-213547)
`
`

`

`[ 46] An "activity" refers to one or more actions and/or one or more composites of
`
`actions of one or more objects. Examples of an activity include: entering; exiting; stopping;
`
`moving; raising; lowering; growing; and shrinking.
`
`[47] A "location" refers to a space where an activity may occur. A location can be, for
`
`example, scene-based or image-based. Examples of a scene-based location include: a public
`
`space; a store; a retail space; an office; a warehouse; a hotel room; a hotel lobby; a lobby of a
`
`building; a casino; a bus station; a train station; an airport; a port; a bus; a train; an airplane; and
`
`a ship. Examples of an image-based location include: a video image; a line in a video image; an
`
`area in a video image; a rectangular section of a video image; and a polygonal section of a video
`
`image.
`
`[48] An "event" refers to one or more objects engaged in an activity. The event may
`
`be referenced with respect to a location and/or a time.
`
`[ 49] A "computer" refers to any apparatus that is capable of accepting a structured
`
`input, processing the structured input according to prescribed rules, and producing results of the
`
`processing as output. Examples of a computer include: a computer; a general purpose computer;
`
`a supercomputer; a mainframe; a super mini-computer; a mini-computer; a workstation; a micro(cid:173)
`
`computer; a server; an interactive television; a hybrid combination of a computer and an
`
`interactive television; and application-specific hardware to emulate a computer and/or software.
`
`A computer can have a single processor or multiple processors, which can operate in parallel
`
`and/or not in parallel. A computer also refers to two or more computers connected together via a
`
`network for transmitting or receiving information between the computers. An example of such a
`
`computer includes a distributed computer system for processing information via computers
`
`linked by a network.
`
`- 6 -
`
`(37112-213547)
`
`

`

`[50] A "computer-readable medium" refers to any storage device used for storing data
`
`accessible by a computer. Examples of a computer-readable medium include: a magnetic hard
`
`disk; a floppy disk; an optical disk, such as a CD-ROM and a DVD; a magnetic tape; a memory
`
`chip; and a carrier wave used to carry computer-readable electronic data, such as those used in
`
`transmitting and receiving e-mail or in accessing a network.
`
`[51]
`
`"Software" refers to prescribed rules to operate a computer. Examples of
`
`software include: software; code segments; instructions; computer programs; and programmed
`
`logic.
`
`[52] A "computer system" refers to a system having a computer, where the computer
`
`comprises a computer-readable medium embodying software to operate the computer.
`
`[53] A "network" refers to a number of computers and associated devices that are
`
`connected by communication facilities. A network involves permanent connections such as
`
`cables or temporary connections such as those made through telephone or other communication
`
`links. Examples of a network include: an internet, such as the Internet; an intranet; a local area
`
`network (LAN); a wide area network (WAN); and a combination of networks, such as an internet
`
`and an intranet.
`
`BRIEF DESCRIPTION OF THE ORA WINGS
`
`[54]
`
`Embodiments of the invention are explained in greater detail by way of the
`
`drawings, where the same reference numerals refer to the same features.
`
`[55]
`
`Figure I illustrates a plan view of the video surveillance system of the invention.
`
`[56]
`
`Figure 2 illustrates a flow diagram for the video surveillance system of the
`
`invention.
`
`- 7 -
`
`(37112-213547)
`
`

`

`[57]
`
`Figure 3 illustrates a flow diagram for tasking the video surveillance system.
`
`[58]
`
`Figure 4 illustrates a flow diagram for operating the video surveillance system.
`
`[59]
`
`Figure 5 illustrates a flow diagram for extracting video primitives for the video
`
`surveillance system.
`
`[60]
`
`Figure 6 illustrates a flow diagram for taking action with the video surveillance
`
`system.
`
`[61]
`
`Figure 7 illustrates a flow diagram for semi-automatic calibration of the video
`
`surveillance system.
`
`[62]
`
`Figure 8 illustrates a flow diagram for automatic calibration of the video
`
`surveillance system.
`
`[63]
`
`Figure 9 illustrates an additional flow diagram for the video surveillance system
`
`of the invention.
`
`[ 64]
`
`Figures 10-15 illustrate examples of the video surveillance system of the
`
`invention applied to monitoring a grocery store.
`
`[ 65]
`
`Figure 16a shows a flow diagram of a video analysis subsystem according to an
`
`embodiment of the invention.
`
`[66]
`
`Figure 16b shows the flow diagram of the event occurrence detection and
`
`response subsystem according to an embodiment of the invention.
`
`[67)
`
`Figure 17 shows exemplary database queries.
`
`[ 68]
`
`Figure 18 shows three exemplary activity detectors according to various
`
`embodiments of the invention: detecting tripwire crossings (Figure 18a), loitering (Figure 18b)
`
`and theft (Figure 18c ).
`
`- 8 -
`
`(37112-213547)
`
`

`

`[69]
`
`Figure 19 shows an activity detector query according to an embodiment of the
`
`invention.
`
`[70]
`
`Figure 20 shows an exemplary query using activity detectors and Boolean
`
`operators with modifiers, according to an embodiment of the invention.
`
`[71]
`
`Figures 21 a and 21 b show an exemplary query using multiple levels of
`
`combinators, activity detectors, and property queries.
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`[72]
`
`The automatic video surveillance system of the invention is for monitoring a
`
`location for, for example, market research or security purposes. The system can be a dedicated
`
`video surveillance installation with purpose-built surveillance components, or the system can be
`
`a retrofit to existing video surveillance equipment that piggybacks off the surveillance video
`
`feeds. The system is capable of analyzing video data from live sources or from recorded media.
`
`The system is capable of processing the video data in real-time, and storing the extracted video
`
`primitives to allow very high speed forensic event detection later. The system can have a
`
`prescribed response to the analysis, such as record data, activate an alarm mechanism, or activate
`
`another sensor system. The system is also capable of integrating with other surveillance system
`
`components. The system may be used to produce, for example, security or market research
`
`reports that can be tailored according to the needs of an operator and, as an option, can be
`
`presented through an interactive web-based interface, or other reporting mechanism.
`
`[73] An operator is provided with maximum flexibility in configuring the system by
`
`using event discriminators. Event discriminators are identified with one or more objects (whose
`
`descriptions are based on video primitives), along with one or more optional spatial attributes,
`
`- 9 -
`
`(37112-213547)
`
`

`

`and/or one or more optional temporal attributes. For example, an operator can define an event
`
`discriminator (called a "loitering" event in this example) as a "person" object in the "automatic
`
`teller machine" space for "longer than 15 minutes" and "between 10:00 p.m. and 6:00 a.m."
`
`Event discriminators can be combined with modified Boolean operators to form more complex
`
`queries.
`
`[74] Although the video surveillance system of the invention draws on well-known
`
`computer vision techniques from the public domain, the inventive video surveillance system has
`
`several unique and novel features that are not currently available. For example, current video
`
`surveillance systems use large volumes of