Please type a plus sign (+) inside this box —>
`
`
`UT||_|TY
`
`PTO/SB/OS Revised (11-00)
`
`37112-175340
`
`PATENT APPLICATION
`
`Alani LIPTON
`
`TRANSMITTAL
`
`VIDEO SURVEILLANCE SYSTEM EMPLOYING
`VIDEO PRIMITIVES
`
`(Only for new nonprovisional applications under 37 CFR. 1.53{b)) _xpressMail Label No.
`
`
`T
`APPLICATION ELEMEN S
`See MPEP chapter 600 concerning utility patent application contents.
`
`ADDRESS TO:
`
`Assistant Commissioner for Patents
`Box Patent Application
`Washington, DC 20231
`
`}
`
`IIIIIIIIIII’IIIIIIIIIII
`
`'S'nSETTL‘
`
`0.1.6
`
`
`
`Fee Transmittal Form (e.g., PTO/SB/17)
`(Submit an original and a duplicate for fee processing)
`
`See 37 CFR 1.27.
`[Total Pages IE ]
`Specification
`(preferred arrangement set forth below)
`- Descriptive title of the invention
`— Cross References to Related Applications
`- Statement Regarding Fed sponsored R & D
`_ Reference to sequence listing. a table,
`0‘ a 00ml“!ter Pr°9ram “Sting appendix
`- Background of the Invention
`- Brief Summary of the Invention
`' 32????“th the D'aWings ( “’9‘”
`:Claiiia) escnp'mn
`'AbStraCt “the D'Sdosure
`[Total Sheets 1
`Drawing(s) (35 u.s.c.113)
`4.
`[Total Pages |:| ]
`5. Oath or Declaration
`I] Newly executed (original or copy)
`b. E] Copy from a prior application (37 CFR 1.63 (d))
`f r
`o t'n
`ti n/d' i ' n l w'th Box 18 cm Ieted
`(o a c n I ua O
`"I SIC a
`I
`c
`p
`)
`i- [I DELETION 0F INVENTQR '3
`name In
`e nor a
`reel IO”, 58
`WW?$§®“§M§*M%%$$)
`1763(d)(2)andp1.33(§§
`e
`6. El Application Data Sheet. See 37 CFR 1.76
`
`6a. E Priority is claimed under the provisions of 35 U.S.C. § 120
`Appln No. 09/694,712 filed on October 24 2000.
`Appln No.
`filed in
`on
`
`18. If a CONTINUING APPLICATION, check appropriate box, and supply the requisite information below and in a preliminary amendment,
`or in an Application Data Sheet under 37 CFR 1.76:
`of priorappiication No: O_9 i694 712
`E] Continuation
`E] Divisional
`Group /Art Unit:
`Prior application information:
`Examiner
`For CONTINUATION or DIVISIONAL APPS only: The entire disclosure of the prior application, from which an oath or declaration is supplied
`under Box 5b, is considered a part of the disclosure of the accompanying or divisional application and is hereby incorporated by reference.
`‘ The incorporation can only be relied upon when a portion has been inadvertently omitted from the submitted application parts.
`
`Continuation-impart (CIP)
`
`20. CORRESPONDENCE ADDRESS
`
`E Customer Number or Bar Code Label
`
`26694
`
`...RATWIWEMAEK QEEGE......
`.
`
`Name
`
`Address
`
`
`PO. Box 34385
`
`Washington
`
`U.S.A
`
`DC.
`
`Zip Code
`
`20043-9998
`
`Telephone
`
`202-962—4800
`
`202-962-8300
`
`Name (Print/Type)
`
`Michael A. Sartori, PhD
`
`M.,-4.
`
`#332195 v1 - 37112-175340 - New App. Transmittal for C-I-P (DIAMONDBACK)
`PC Docs No. 332195
`
`Whilst}:
`
`AVIGILON EX. 2007
`IPR2019-00235
`
`Page 1 of 46
`
`7. I] CD—ROM or CD-R in duplicate, large table or
`Computer Program (Appendix)
`8. Nucleotide and/or Amino Acid Sequence Submission
`(if applicable, all necessary)
`a. [3 Computer Readable Form (CRF)
`b. Specification Sequence Listing on:
`i. D CD-ROM or CD-R (2 copies); or
`ii [1 paper
`.
`'
`.
`.
`.
`.
`0. [:I Statements verIfyIng Identity of above copies
`ACCOMPANYING APPLICATIONS PARTS
`
`g3
`=$
`=n
`==:
`EH
`E
`E-
`
`r
`
`9. 1:]
`10. E]
`
`11. El
`12. El
`
`Assignment Papers (cover sheet & document(s))
`37 C.F.R.§3.73(b) Statement
`[I Power of
`(when there is an assignee)
`Attorney
`English Translation Document (if applicable)
`Information Disclosure
`III Copies of IDS
`Statement (IDSVPTO'1449
`C'tat'ons
`PreIImInary Amendment
`Return Receipt 303mm? (MPEP 503)
`(Should be specrfically Itemized)
`Certified Copy of Priority Document(s)
`(if foreign priority is claimed)
`.
`.
`1cm meflwflkaMwMafiUSQQZ
`g)i(t:)(eB)lER’.al:t:piicant must attach form PTO/SB/35
`17 D Other q
`'
`
`13'
`14‘
`15. E]
`
` Applicant claims small entity status.
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 1 of 46
`
`

`

`
`
`Revised PTO/SB/35 (11-00)
`Approved for use through 10/31/2002 OMB 0651-0031
`US. Patent and Trademark Office; U. S. DEPARTMENT OF COMMERCE
`Under the Papemork Reduction Act of 1995. no persons are required to respond to a collection of information unless it displays a valid OMB control number.
`Attorney Docket No. 332238
`
`
`
`First Named inventor
`
`Alan J. LIPTON
`
`
`
`Title
`
`VIDEO SURVEILLANCE SYSTEM EMPLOYING
`VIDEO PRIMITIVES
`
`
`
`
`REQUEST AND CERTIFICATION
`UNDER
`
`
`
`35 u.s.c. 122(b)(2)(B)(i)
`
`I hereby certify that the invention disclosed in the attached application has not and will not be
`the subject of an application filed in another country, or under a multilateral agreement, that
`requires publication at eighteen months after filing. I hereby request that the attached application
`not be published under 35 U.S.C. 122(b).
`
`@m/ee /5/ 255/
`Date
`
`Sig nature
`
`
`
`Michael A. Sartori, Ph.D. Reg. No. 41,289
`
`Typed or printed name
`
`This request must be signed in compliance with 37 CFR 1.33(b) and submitted
`with the application upon filing.
`
`If applicant
`Applicant may rescind this nonpublication request at any time.
`rescinds a request that an application not be published under 35 U.S.C. 122(b),
`the application will be scheduled for publication at eighteen months from the
`earliest claimed filing date for which a benefit is claimed.
`
`If applicant subsequently files an application directed to the invention disclosed in
`the attached application in another country, or under a multilateral international
`agreement, that requires publication of applications eighteen months after filing,
`the applicant must notify the United States Patent and Trademark Office of such
`filing within forty-five (45) days after the date of the filing of such foreign or
`international application. Failure to do so will result in abandonment of this
`application (35 U.S.C. 122(b)(2)(B)(iii)).
`
`SEND TO: Assistant Commissioner for Patents. Washington, DC 20231.
`PO Docs No. 332238
`
`mm
`,
`.
`
`AVIGILON EX. 2007
`IPR2019-00235
`
`Page 2 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 2 of 46
`
`

`

`APPLICATION FOR UNITED STATES PATENT
`
`INVENTORS:
`
`
`
`TITLE:
`
`ALAN J. LIPTON
`THOMAS M. STRAT
`PETER L. VENETJANER
`MARK C. ALLMEN
`
`WILLIAM E. SEVERSON
`
`NIELS HAERING
`
`ANDREW J. CHOSAK
`
`ZHONG ZHANG
`
`MATTHEW F. FRAZIER
`
`JAMES s. SEEMS
`
`TASUKI HJRATA
`
`JOHN CLARK
`
`VIDEO SURVEILLANCE SYSTEM EMPLOYING
`VIDEO PRIMITIVES
`
`ATTORNEYS' ADDRESS:
`
`VENABLE
`
`1201 New York Avenue, N.W., Suite 1000
`Washington, DC. 20005—3917
`Telephone: (202) 962-4800
`Telefax: (202) 962-8300
`
`ADDRESS FOR U.S.P.T.O. CORRESPONDENCE:
`
`VENABLE
`
`Post Office Box 34385
`
`Washington, DC. 20043-9998
`
`ATTORNEY DOCKET NO.:
`
`37112—175340
`
`AVIGILON EX. 2007
`
`IPR2019-00235
`
`Page 3 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 3 of 46
`
`

`

`VIDEO SURVEILLANCE SYSTEM EMPLOYING VIDEO PRIMITIVES
`
`CROSS-REFERENCE TO RELATED APPLICATIONS
`
`[1]
`
`This application claims the priority of US. Patent Application No. 09/694,712
`
`filed October 24, 2000, which is incorporated herein by reference.
`
`BACKGROUND OF THE INVENTION
`
`Field of the Invention
`
`[2]
`
`The invention relates to a system for automatic Video surveillance employing
`
`video primitives.
`
`References
`
`[3]
`
`For the convenience of the reader, the references referred to herein are listed
`
`below. In the specification, the numerals within brackets refer to respective references. The
`
`listed references are incorporated herein by reference.
`
`[4]
`
`[5]
`
`The following references describe moving target detection:
`
`{1} A. Lipton, H. Fujiyoshi and R. S. Patil, “Moving Target Detection and
`
`Classification from Real-Time Video,” Proceedings of IEEE WACV ’98, Princeton, NJ, 1998,
`
`
`
`
`pp. 8—14.
`
`20
`
`[6]
`
`{2} W.E.L. Grimson, et al., “Using Adaptive Tracking to Classify and Monitor
`
`
`Activities in a Site”, CVPR, pp. 22-29, June 1998.
`
`[7]
`
`{3} AI. Lipton, H. Fujiyoshi, R.S. Patil, “Moving Target Classification and
`
`Tracking from Real-time Video,” IUW, pp. 129-136, 1998.
`
`(371124753402LVIGILON EX. 2007
`IPR2019-00235
`
`Page 4 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 4 of 46
`
`

`

`[8]
`
`{4} T.J. Olson and F2. Brill, “Moving Object Detection and Event Recognition
`
`Algorithm for Smart Cameras,” M, pp. 159—175, May 1997.
`
`[9]
`
`The following references describe detecting and tracking humans:
`
`[10]
`
`{5 } A. J. Lipton, “Local Application of Optical Flow to Analyse Rigid Versus
`
`Non-Rigid Motion,” International Conference on Computer Vision, Corfu, Greece, September
`
`1 999.
`
`[11]
`
`{6} F. Bartolini, V. Cappellini, and A. Mecocci, “Counting people getting in and
`
`out of a bus by real-time image-sequence processing,” M, 12(1):36-41, January 1994.
`
`[12]
`
`{7} M. Rossi and A. Bozzoli, “Tracking and counting moving people,” IC_IPQ_4,
`
`pp. 212-216, 1994.
`
`[13]
`
`{8} CR. Wren, A. Azarbayejani, T. Darrell, and A. Pentland, “Pfinder: Real-
`
`time tracking of the human body,” Vismod, 1995.
`
`[14]
`
`{9} L. Khoudour, L. Duvieubourg, J.P. Deparis, “Real-Time Pedestrian Counting
`
`
`by Active Linear Cameras,” JEI, 5(4):452-459, October 1996.
`
`[15]
`
`{10} S. Ioffe, D.A. Forsyth, “Probabilistic Methods for Finding People,” IJCV,
`
`43(1):45-68, June 2001.
`
`[16]
`
`{11} M. Isard and J. MacCorrnick, “BraMBLe: A Bayesian Multiple-Blob
`
`
`
`
`Tracker,” ICCV, 2001 .
`
`[17]
`
`The following references describe blob analysis:
`
`20
`
`[1 8]
`
`{l2} D.M. Gavrila, “The Visual Analysis of Human Movement: A Survey,”
`
`
`CVIU, 73(1):82-98, January 1999.
`
`[19]
`
`
`{13} Niels Haering and Niels da Vitoria Lobo, “Visual Event Detection,” Video
`
`Computing Series, Editor Mubarak Shah, 2001.
`
`- 2 -
`
`(37112-17534‘8?
`AVI
`ILON EX. 2007
`
`IPR2019-00235
`
`Page 5 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 5 of 46
`
`

`

`[20]
`
`The following references describe blob analysis for trucks, cars, and people:
`
`[21]
`
`{14} Collins, Lipton, Kanade, Fujiyoshi, Duggins, Tsin, Tolliver, Enomoto, and
`
`Hasegawa, “A System for Video Surveillance and Monitoring: VSAM Final Report,“ Technical
`
`Report CMU—RI-TR—OO-l2, Robotics Institute, Carnegie Mellon University, May 2000.
`
`5
`
`[22]
`
`{15} Lipton, Fujiyoshi, and Patil, “Moving Target Classification and Tracking
`
`fiom Real—time Video,” 98 Dgpa IUW, Nov. 20-23, 1998.
`
`[23]
`
`The following reference describes analyzing a single-person blob and its contours:
`
`[24]
`
`{16} CR. Wren, A. Azarbayejani, T. Darrell, and AP. Pentland. “Pfinder: Real-
`
`Time Tracking of the Human Body,” PAMI, V01 19, pp. 7 80-784, 1997.
`
`[25]
`
`The following reference describes internal motion of blobs, including any motion-
`
` based segmentation:
`
`
`[26]
`
`{17} M. Allmen and C. Dyer, “Long-Range Spatiotemporal Motion
`
`Understanding Using Spatiotemporal Flow Curves,” Proc. IEEE CVPR, Lahaina, Maui, Hawaii,
`
`pp. 303-309, 1991.
`
`15
`
`[27]
`
`{18} L. Wixson, "Detecting Salient Motion by Accumulating Directionally
`
`Consistent Flow", IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, pp. 774-781, Aug, 2000.
`
`Background of the Invention
`
`[28] Video surveillance of public spaces has become extremely widespread and
`
`20
`
`accepted by the general public. Unfortunately, conventional video surveillance systems produce
`
`such prodigious volumes of data that an intractable problem results in the analysis of video
`
`surveillance data.
`
`- 3 -
`
`(37112-175340
`AVI
`
`ILON EX. 2007
`
`IPR2019-00235
`
`Page 6 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 6 of 46
`
`

`

`[29]
`
`A need exists to reduce the amount of video surveillance data so analysis of the
`
`Video surveillance data can be conducted.
`
`[30] A need exists to filter video surveillance data to identify desired portions of the
`
`video surveillance data.
`
`SUMMARY OF THE INVENTION
`
`
`
`
`[31] An object of the invention is to reduce the amount of video surveillance data so
`
`analysis of the video surveillance data can be conducted.
`
`[32] An object of the invention is to filter video surveillance data to identify desired
`
`portions of the video surveillance data.
`
`[33] An object of the invention is to produce a real time alarm based on an automatic
`
`detection of an event from video surveillance data.
`
`[34] An object of the invention is to integrate data from surveillance sensors other than
`
`video for improved searching capabilities.
`
`[35] An object of the invention is to integrate data from surveillance sensors other than
`
`video for improved event detection capabilities
`
`[3 6]
`
`The invention includes an article of manufacture, a method, a system, and an
`
`apparatus for video surveillance.
`
`[3 7]
`
`The article of manufacture of the invention includes a computer-readable medium
`
`20
`
`comprising software for a video surveillance system, comprising code segments for operating the
`
`video surveillance system based on video primitives.
`
`[3 8]
`
`The article of manufacture of the invention includes a computer-readable medium
`
`comprising software for a video surveillance system, comprising code segments for accessing
`
`-4 -
`
`(3711247336110 ILON EX. 2007
`IPR2019-00235
`
`Page 7 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 7 of 46
`
`

`

`archived video primitives, and code segments for extracting event occurrences from accessed
`
`archived video primitives.
`
`[39]
`
`The system of the invention includes a computer system including a computer-
`
`readable medium having software to operate a computer in accordance with the invention.
`
`[40]
`
`The apparatus of the invention includes a computer including a computer-readable
`
`medium having software to operate the computer in accordance with the invention.
`
`[41]
`
`The article of manufacture of the invention includes a computer-readable medium
`
`having software to operate a computer in accordance with the invention.
`
`[42] Moreover, the above objects and advantages of the invention are illustrative, and
`
`not exhaustive, of those that can be achieved by the invention. Thus, these and other objects and
`
`advantages of the invention will be apparent from the description herein, both as embodied
`
`herein and as modified in View of any variations which will be apparent to those skilled in the
`
`art.
`
`Definitions
`
`
`
`
`[43] A “video” refers to motion pictures represented in analog and/or digital form.
`
`Examples of video include: television, movies, image sequences from a video camera or other
`
`observer, and computer-generated image sequences.
`
`[44] A “frame” refers to a particular image or other discrete unit within a video.
`
`20
`
`[45] An “object” refers to an item of interest in a video. Examples of an object
`
`include: a person, a vehicle, an animal, and a physical subject.
`
`- 5 —
`
`(37112-175340
`AVI
`
`ILON EX. 2007
`
`IPR2019-00235
`
`Page 8 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 8 of 46
`
`

`

`[46] An “activity” refers to one or more actions and/or one or more composites of
`
`actions of one or more objects. Examples of an activity include: entering; exiting; stopping;
`
`moving; raising; lowering; growing; and shrinking.
`
`[47] A “location” refers to a space Where an activity may occur. A location can be, for
`
`example, scene-based or image-based. Examples of a scene-based location include: a public
`
`space; a store; a retail space; an office; a warehouse; a hotel room; a hotel lobby; a lobby of a
`
`building; a casino; a bus station; a train station; an airport; a port; a bus; a train; an airplane; and
`
`a ship. Examples of an image-based location include: a Video image; a line in a Video image; an
`
`area in a video image; a rectangular section of a video image; and a polygonal section of a Video
`
`image.
`
`[48] An “event” refers to one or more objects engaged in an activity. The event may
`
`be referenced with respect to a location and/or a time.
`
`[49] A “computer” refers to any apparatus that is capable of accepting a structured
`
`input, processing the structured input according to prescribed rules, and producing results of the
`
`processing as output. Examples of a computer include: a computer; a general purpose computer;
`
`a supercomputer; a mainframe; a super mini-computer; a mini-computer; a workstation; a micro-
`
`computer; a server; an interactive television; a hybrid combination of a computer and an
`
`interactive television; and application-specific hardware to emulate a computer and/or software.
`
`
`
`A computer can have a single processor or multiple processors, which can operate in parallel
`
`20
`
`and/or not in parallel. A computer also refers to two or more computers connected together via a
`
`network for transmitting or receiving information between the computers. An example of such a
`
`computer includes a distributed computer system for processing information via computers
`
`linked by a network.
`
`_ 6 _
`
`2-
`
`(3711
`
`”Xi/410 ILON EX. 2007
`IPR2019-00235
`
`Page 9 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 9 of 46
`
`

`

`[50] A “computer-readable medium” refers to any storage device used for storing data
`
`accessible by a computer. Examples of a computer—readable medium include: a magnetic hard
`
`disk; a floppy disk; an optical disk, such as a CD-ROM and a DVD; a magnetic tape; a memory
`
`chip; and a carrier wave used to carry computer—readable electronic data, such as those used in
`
`transmitting and receiving e-mail or in accessing a network.
`
`[51]
`
`"Software” refers to prescribed rules to operate a computer. Examples of
`
`software include: software; code segments; instructions; computer programs; and programmed
`
`logic.
`
`
`
`[52] A “computer system” refers to a system having a computer, where the computer
`
`comprises a computer-readable medium embodying software to operate the computer.
`
`[53] A “network” refers to a number of computers and associated devices that are
`
`connected by communication facilities. A network involves permanent connections such as
`
`cables or temporary connections such as those made through telephone or other communication
`
`links. Examples of a network include: an internet, such as the Internet; an intranet; a local area
`
`network (LAN); a Wide area network (WAN); and a combination of networks, such as an intemet
`
`and an intranet.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[54]
`
`Embodiments of the invention are explained in greater detail by way of the
`
`20
`
`drawings, where the same reference numerals refer to the same features.
`
`[55]
`
`Figure 1 illustrates a plan View of the video surveillance system of the invention.
`
`[56]
`
`Figure 2 illustrates a flow diagram for the video surveillance system of the
`
`invention.
`
`' 7 '
`
`(37112'17Xv‘i81LON EX. 2007
`IPR2019-00235
`
`Page 10 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 10 of 46
`
`

`

`[57]
`
`Figure 3 illustrates a flow diagram for tasking the video surveillance system.
`
`[5 8]
`
`Figure 4 illustrates a flow diagram for operating the Video surveillance system.
`
`[59]
`
`Figure 5 illustrates a flow diagram for extracting Video primitives for the Video
`
`surveillance system.
`
`5
`
`[60]
`
`Figure 6 illustrates a flow diagram for taking action with the video surveillance
`
`system.
`
`[61]
`
`Figure 7 illustrates a flow diagram for semi-automatic calibration of the Video
`
`surveillance system.
`
`[62]
`
`Figure 8 illustrates a flow diagram for automatic calibration of the Video
`
`[63]
`
`Figure 9 illustrates an additional flow diagram for the video surveillance system
`
`of the invention.
`
` surveillance system.
`
`
`[64]
`
`Figures 10-15 illustrate examples of the Video surveillance system of the
`
`invention applied to monitoring a grocery store.
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`[65]
`
`The automatic video surveillance system of the invention is for monitoring a
`
`location for, for example, market research or security purposes. The system can be a dedicated
`
`video surveillance installation with purpose-built surveillance components, or the system can be
`
`20
`
`a retrofit to existing Video surveillance equipment that piggybacks off the surveillance Video
`
`feeds. The system is capable of analyzing Video data from live sources or from recorded media.
`
`The system can have a prescribed response to the analysis, such as record data, activate an alarm
`
`mechanism, or active another sensor system. The system is also capable of integrating with
`
`- 8 -
`
`(
`
`37112-17534
`AVI LON EX. 2007
`IPR2019-00235
`
`Page 11 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 11 of 46
`
`

`

`other surveillance system components. The system produces security or market research reports
`
`that can be tailored according to the needs of an operator and, as an option, can be presented
`
`through an interactive web—based interface, or other reporting mechanism.
`
`[66] An operator is provided with maximum flexibility in configuring the system by
`
`using event discriminators. Event discriminators are identified with one or more objects (whose
`
`descriptions are based on video primitives), along with one or more optional spatial attributes,
`
`and/or one or more optional temporal attributes. For example, an operator can define an event
`
`discriminator (called a "loitering" event in this example) as a “person” object in the “automatic
`
`teller machine” space for “longer than 15 minutes” and “between 10:00 pm. and 6:00 am.”
`
`[67] Although the Video surveillance system of the invention draws on well-known
`
`computer vision techniques from the public domain, the inventive Video surveillance system has
`
`several unique and novel features that are not currently available. For example, current video
`
`surveillance systems use large volumes of Video imagery as the primary commodity of
`
`information interchange. The system of the invention uses video primitives as the primary
`
`commodity with representative video imagery being used as collateral evidence. The system of
`
`the invention can also be calibrated (manually, semi-automatically, or automatically) and
`
`thereafier automatically can infer video primitives from video imagery. The system can further
`
`
`
`
`
`
`
`analyze previously processed video without needing to reprocess completely the video. By
`
`analyzing previously processed video, the system can perform inference analysis based on
`
`20
`
`previously recorded video primitives, which greatly improves the analysis speed of the computer
`
`system.
`
`[68] As another example, the system of the invention provides unique system tasking.
`
`Using equipment control directives, current video systems allow a user to position video sensors
`
`- 9 -
`
`(371 12'173ii‘10d1L0N EX. 2007
`IPR2019-00235
`
`Page 12 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 12 of 46
`
`

`

`and, in some sophisticated conventional systems, to mask out regions of interest or disinterest.
`
`Equipment control directives are instructions to control the position, orientation, and focus of
`
`video cameras. Instead of equipment control directives, the system of the invention uses event
`
`discriminators based on video primitives as the primary tasking mechanism. With event
`
`discriminators and video primitives, an operator is provided with a much more intuitive approach
`
`over conventional systems for extracting useful information from the system. Rather than
`
`tasking a system with an equipment control directives, such as “camera A pan 45 degrees to the
`
`left,” the system of the invention can be tasked in a human-intuitive manner with one or more
`
`event discriminators based on video primitives, such as “a person enters restricted area A.”
`
`[69] Using the invention for market research, the following are examples of the type of
`
`video surveillance that can be performed with the invention: counting people in a store; counting
`
`people in a part of a store; counting people who stop in a particular place in a store; measuring
`
`how long people spend in a store; measuring how long people spend in a part of a store; and
`
`measuring the length of a line in a store.
`
`[70] Using the invention for security, the following are examples of the type of video
`
`surveillance that can be performed with the invention: determining when anyone enters a
`
`restricted area and storing associated imagery; determining when a person enters an area at
`
`unusual times; determining when changes to shelf space and storage space occur that might be
`
`
`
`unauthorized; determining when passengers aboard an aircraft approach the cockpit; determining
`
`20
`
`when people tailgate through a secure portal; determining if there is an unattended bag in an
`
`airport; and determining if there is a theft of an asset.
`
`[71]
`
`Figure 1 illustrates a plan view of the video surveillance system of the invention.
`
`A computer system 11 comprises a computer 12 having a computer-readable medium 13
`
`- 10 -
`
`(37112-175340
`AVI LON EX. 2007
`
`IPR2019-00235
`
`Page 13 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 13 of 46
`
`

`

`embodying software to operate the computer 12 according to the invention. The computer
`
`system 11 is coupled to one or more video sensors 14, one or more video recorders 15, and one
`
`or more input/output (l/O) devices 16. The video sensors 14 can also be optionally coupled to
`
`the video recorders 15 for direct recording of video surveillance data. The computer system is
`
`optionally coupled to other sensors 17.
`
`[72]
`
`The video sensors 14 provide source video to the computer system 11. Each
`
`video sensor 14 can be coupled to the computer system 11 using, for example, a direct
`
`connection (e. g., a firewire digital camera interface) or a network. The video sensors 14 can
`
`exist prior to installation of the invention or can be installed as part of the invention. Examples
`
`of a video sensor 14 include: a video camera; a digital video camera; a color camera; a
`
`monochrome camera; a camera; a camcorder, a PC camera; a webcam; an infra-red video
`
`camera; and a CCTV camera.
`
`[73]
`
`The video recorders 15 receive video surveillance data from the computer system
`
`11 for recording and/or provide source video to the computer system 11. Each video recorder 15
`
`can be coupled to the computer system 11 using, for example, a direct connection or a network.
`
`The video recorders 15 can exist prior to installation of the invention or can be installed as part
`
`of the invention. Examples of a video recorder 15 include: a video tape recorder; a digital video
`
`recorder; a video disk; a DVD; and a computer-readable medium.
`
`
`
`
`[74]
`
`The I/O devices 16 provide input to and receive output from the computer system
`
`20
`
`11. The I/O devices 16 can be used to task the computer system 11 and produce reports from the
`
`computer system 11. Examples of I/O devices 16 include: a keyboard; a mouse; a stylus; a
`
`monitor; a printer; another computer system; a network; and an alarm.
`
`— 11 -
`
`(37112-175340]
`AVIG LON EX. 2007
`IPR2019-00235
`
`Page 14 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 14 of 46
`
`

`

`[75]
`
`The other sensors 17 provide additional input to the computer system 11. Each
`
`other sensor 17 can be coupled to the computer system 11 using, for example, a direct connection
`
`or a network. The other sensors 17 can exit prior to installation of the invention or can be
`
`installed as part of the invention. Examples of another sensor 17 include: a motion sensor; an
`
`optical tripwire; a biometric sensor; and a card-based or keypad-based authorization system. The
`
`outputs of the other sensors 17 can be recorded by the computer system 11, recording devices,
`
`and/or recording systems.
`
`
`
`
`[76]
`
`Figure 2 illustrates a flow diagram for the Video surveillance system of the
`
`invention. Various aspects of the invention are exemplified with reference to Figures 10-15,
`
`which illustrate examples of the video surveillance system of the invention applied to monitoring
`
`a grocery store.
`
`[77]
`
`In block 21, the video surveillance system is set up as discussed for Figure 1.
`
`Each video sensor 14 is orientated to a location for video surveillance. The computer system 11
`
`is connected to the Video feeds from the video equipment 14 and 15. The video surveillance
`
`system can be implemented using existing equipment or newly installed equipment for the
`
`location.
`
`[78]
`
`In block 22, the video surveillance system is calibrated. Once the video
`
`surveillance system is in place from block 21, calibration occurs. The result of block 22 is the
`
`ability of the Video surveillance system to determine an approximate absolute size and speed of a
`
`20
`
`particular object (e.g., a person) at various places in the video image provided by the video
`
`sensor. The system can be calibrated using manual calibration, semi-automatic calibration, and
`
`automatic calibration. Calibration is further described after the discussion of block 24.
`
`- 12 -
`
`(37112-175340
`AVI
`
`ILON EX. 2007
`
`IPR2019-00235
`
`Page 15 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 15 of 46
`
`

`

`[79]
`
`In block 23 of Figure 2, the video surveillance system is tasked. Tasking occurs
`
`after calibration in block 22 and is optional. Tasking the video surveillance system involves
`
`specifying one or more event discriminators. Without tasking, the video surveillance system
`
`operates by detecting and archiving video primitives and associated video imagery without
`
`taking any action, as in block 45 in Figure 4.
`
`
`
`
`[80]
`
`Figure 3 illustrates a flow diagram for tasking the video surveillance system to
`
`determine event discriminators. An event discriminator refers to one or more objects optionally
`
`interacting with one or more spatial attributes and/or one or more temporal attributes. An event
`
`discriminator is described in terms of Video primitives. A video primitive refers to an observable
`
`attribute of an object viewed in a Video feed. Examples of video primitives include the
`
`following: a classification; a size; a shape; a color; a texture; a position; a velocity; a speed; an
`
`internal motion; a motion; a salient motion; a feature of a salient motion; a scene change; a
`
`feature of a scene change; and a pre—defined model.
`
`[81] A classification refers to an identification of an object as belonging to a particular
`
`category or class. Examples of a classification include: a person; a dog; a vehicle; a police car;
`
`an individual person; and a specific type of object.
`
`[82] A size refers to a dimensional attribute of an object. Examples of a size include:
`
`large; medium; small; flat; taller than 6 feet; shorter than 1 foot; wider than 3 feet; thinner than 4
`
`feet; about human size; bigger than a human; smaller than a human; about the size of a car; a
`
`20
`
`rectangle in an image with approximate dimensions in pixels; and a number of image pixels.
`
`[83] A color refers to a chromatic attribute of an object. Examples of a color include:
`
`white; black; grey; red; 21 range of HSV values; a range of YUV values; a range of RGB values;
`
`an average RGB value; an average Y U V value; and a histogram of RGB values.
`
`- 13 -
`
`(37112-175340)
`AVIGILON EX. 2007
`
`IPR2019-00235
`
`Page 16 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 16 of 46
`
`

`

`[84]
`
`A texture refers to a pattern attribute of an object. Examples of texture features
`
`include: self-similarity; spectral power; linearity; and coarseness.
`
`[85] An internal motion refers to a measure of the rigidity of an object. An example of
`
`a fairly rigid object is a car, which does not exhibit a great amount of internal motion. An
`
`example of a fairly non-rigid object is a person having swinging arms and legs, which exhibits a
`
`great amount of internal motion.
`
`
`
`
`
`15
`
`[86] A motion refers to any motion that can be automatically detected. Examples of a
`
`motion include: appearance of an object; disappearance of an object; a vertical movement of an
`
`object; a horizontal movement of an object; and a periodic movement of an object.
`
`[87] A salient motion refers to any motion that can be automatically detected and can
`
`be tracked for some period of time. Such a moving object exhibits apparently purposeful
`
`motion. Examples of a salient motion include: moving from one place to another; and moving to
`
`interact with another object.
`
`[88] A feature of a salient motion refers to a property of a salient motion. Examples of
`
`a feature of a salient motion include: a trajectory; a length of a trajectory in image space; an
`
`approximate length of a trajectory in a three-dimensional representation of the environment; a
`
`position of an object in image space as a function of time; an approximate position of an object
`
`in a three-dimensional representation of the environment as a function of time; a duration of a
`
`trajectory; a velocity (e.g., speed and direction) in image space; an approximate velocity (e.g.;
`
`20
`
`speed and direction) in a three-dimensional representation of the environment; a duration of time
`
`at a velocity; a change of velocity in image space; an approximate change of velocity in a three-
`
`dimensional representation of the environment; a duration of a change of velocity; cessation of
`
`motion; and a duration of cessation of motion. A velocity refers to the speed and direction of an
`
`.
`
`'14 '
`
`(37112'17gi/‘l9ilL0N EX. 2007
`IPR2019-00235
`
`Page 17 of 46
`
`AVIGILON EX. 2007
`IPR2019-00235
`Page 17 of 46
`
`

`

`object at a particular time. A trajectory refers a set of (position, velocity) pairs for an object for
`
`as long as the object can be tracked or for a time period.
`
`[89] A scene change refers to any region of a scene that can be detected as changing
`
`over a period of time. Examples of a scene cha