Image siniinai Workshop
`
`Proceedings of aWork
`heldiin
`
`
` AURT F. WENDTLiBkay
`COLLEGE OF ENGINEERIN’
`Volume leet - 4 tag
`
`: ‘Sponsored by:
`:
`DefenseAdvanced ResearchProjects Agency
`
`UW-MADISON. Wt rerne
`
`This documentcontains copies of reports prepared for the DARPA Image Understanding Workshop.
`Included are Principal Investigator reports and technical results from the basic and strategic comput-
`ing programs within DARPA/ISO-sponsored projects and certain technical reports from selected sci-
`entists from other organizations.
`
`APPROVED FOR PUBLIC RELEASE
`DISTRIBUTION UNLIMITED
`
`The views and conclusions contained in this document are those of the authors and should not be
`interpreted as necessarily representing the official policies, either expressed or implied, of the De-
`fense Advanced Research Projects Agency or the Governmentof the United States of America.
`
`Canon Ex. 1051 Page 1 of 19
`
` Canon Ex. 1051 Page 1 of 19
`
`

`

`Kurt F. Wendt Library
`University of Wisconsin - Madisor
`215 .N. Randall Avenue
`Madison, WI 53706-1688
`
`Distributed by:
`Morgan Kaufmann PublishersInc.
`340 Pine Street, 6th Floor
`San Francisco, Calif. 94104-3205
`ISBN: 1-55860-583-5
`Printed in the United States of America
`
`Canon Ex. 1051 Page 2 of 19
`
` Canon Ex. 1051 Page 2 of 19
`
`

`

`This material may be protected by Copyright law (Title 17 U.S. Code)
`
`Event Recognition and Reliability Improvements for
`the AutonomousVideo Surveillance System
`
`Frank Z. Brill, Thomas J. Olson, and Christopher Tserng
`Texas Instruments
`P.O. Box 655303, MS 8374, Dallas, TX 75265
`brill @csc.ti.com, olson @csc.ti.com, tserng @csc.ti.com
`
`Abstract
`
`This report describes recent progress in the devel-
`opment of the Autonomous Video Surveillance
`(AVS) system, a general-purpose system for mov-
`ing object detection and event recognition. AVS
`analyses live video of a scene and builds a descrip-
`tion of the activity in that scene. The recent
`enhancements to AVS described in this report are:
`(1) use of collateral information sources, (2) cam-
`era hand-off, (3) vehicle event recognition, and (4)
`complex-event recognition. Also described is a
`new segmentation and tracking technique and an
`evaluation of AVS performing the best-view selec-
`tion task.
`
`1. Introduction
`
`The Autonomous Video Surveillance (AVS) sys-
`tem processeslive video streams from surveillance
`cameras to automatically produce a real-time map-
`based display of the locations of people, objects
`and events in a monitored region. The system al-
`lows
`a
`user
`to
`specify
`alarm conditions
`interactively, based on the locations of people and
`objects in the scene, the types of objects in the
`scene, the events in which the people and objects
`are involved, and the times at which the events oc-
`cur. Furthermore, the user can specify the action to
`take when an alarm is triggered, e.g., to generate an
`audio alarm or write a log file. For example, the
`user can specify that an audio alarm should betrig-
`gered if a person deposits a briefcase on a given
`table between 5:00pm and 7:00am on a weeknight.
`Section 2 below describes recent enhancements to
`
`This research was sponsored in part by the DARPA Image
`Understanding Program.
`
`the AVS system. Section 3 describes progress in
`improving the reliability of segmentation and
`tracking. Section 4 describes an experiment that
`quantifies the performance of the AVS “best view
`selection”capability.
`
`2. New AVSfunctionality
`
`The structure and function of the AVS system is
`described in detail in a previous IUW paper [Olson
`and Brill, 1997]. The primary purpose of the cur-
`rent paper is to describe recent enhancements to
`the AVS system. These enhancements are de-
`scribed in four sections below:
`(1) collateral
`information sources, (2) camera hand-off, (3) vehi-
`cle event
`recognition, and (4) complex-event
`recognition.
`
`2.1. Collateral information sources
`
`Figure 1 showsa diagram of the AVS system. One
`or more “smart” cameras process the video stream
`to recognize events. The resulting event streams
`are sent
`to a Video Surveillance Shell
`(VSS),
`which integrates the information and displays it on
`a map. The VSScan also generate alarms based on
`the information in the event streams.
`In recent
`work, the VSS was enhancedto accept information
`from other sources, or “recognition devices” which
`can identify the objects being reported on by the
`cameras. For example, a camera may report that
`there is a person near a door. A recognition device
`may report that the person near the door is Joe
`Smith. The recognition device may be a badge
`reader, a keypad in which a person types their PIN,
`a face recognition system, or other recognition sys-
`tem.
`
`267
`
`Canon Ex. 1051 Page 3 of 19
`
` Canon Ex. 1051 Page 3 of 19
`
`

`

`oo Display
`—_oea
`SmartCamera2
`es«8
`audiooutput
`0"
`an snapshots
`(=F
`
`Smart Camera 1 Map
`
`eventfiltering
`
`oe
`
`log files
`
`Figure 1: AVS system diagram
`
`The recognition device we have incorporated is a
`voice verification system. The user stands in a pre-
`defined location in the room, and speaks his or her
`name. The system matchesthe utterance to previ-
`ously captured examples of the person speaking
`their name, and reports to the VSS if there is a
`match. The VSS now knowsthe identity of the per-
`son being observed, and can customize alarms
`based on the person’s identity.
`
`A recognition device could identify things other
`than people, and could classify actions instead of
`objects. For example, the MIT Action Recognition
`System (MARS)recognizes actions of people in
`the scene, such as raising their arms or bending
`over. MARSistrained by observing examples of
`the action to be recognized and forming “temporal
`templates” that briefly describe the action [Davis
`and Bobick, 1997]. At run time, MARS observes
`the motion in the scene and determines when the
`motion matches one of the stored temporal tem-
`plates. TI has obtained an evaluation copy of the
`
`MARSsoftware and usedit as an recognition de-
`vice which identifies actions, and sends theresult
`to the AVS VSS. We successfully trained MARSto
`recognize the actions of opening a door, and open-
`ing the drawer of a file cabinet. When MARS
`recognizes these actions, it sends a message to the
`AVS VSS, which can generate an appropriate
`alarm.
`
`2.2. Camera hand-off
`
`As depicted in Figure 1, the AVS system incorpo-
`rates multiple cameras to enable surveillance of a
`widerarea than can be monitored via a single cam-
`era. If the fields of view of these cameras are
`adjacent, a person can be tracked from one moni-
`tored area to another. When the person leaves the
`field of view of one camera andenters another, the
`process of maintaining the track from one camera
`view to another is termed camera hand-off. Figure
`2 showsan area monitored by two cameras. Cam-
`
`
`
`File
`
`Regions—Moniters
`
`268
`
`Canon Ex. 1051 Page 4 of 19
`
` Canon Ex. 1051 Page 4 of 19
`
`

`

`and Camera-2
`hallway,
`the
`monitors the interior of the room. When a person
`moves through the doorwayto enter the room from
`the hall or vice-versa, camera hand-off is necessary
`to enable the system to know that the person that
`was being monitored in the hall via Camera-1 is
`the same as the person being monitored in the
`room via Camera-2.
`
`backgrounds. The work
`dynamic
`and
`es,
`described here in section 2.3 addresses the first
`problem,to extend the system for vehicle events in
`conditions of uniform overcast with little wind.
`Our approachto handling general outdoorlighting
`conditions is discussed in section 4.
`
`The method is further specialized for imaging con-
`ditions in which:
`
`1. The camera viewscarslaterally.
`2. Cars are unoccluded by othercars.
`3. When cars and people overlap, only one of
`the overlapping objects is moving
`4. The events of interest are people getting
`into and outof cars.
`
`2.3.2. Car detection
`
`The AVSsystem accomplishes camera hand-off by
`integrating the information from the two cameras
`in the map coordinate system. The AVS “smart”
`cameras report the locations of the monitored ob-
`jects and people in map coordinates, so that when
`the VSS receives reports about a person from two
`separate cameras, and both cameras are reporting
`the person’s coordinates at about the same maplo-
`cation, the VSS can deduce that the two separate
`reports refer to the same person.In the example de-
`The first thing that was done to expand the event
`picted in Figure 2, when a personis standing in the
`recognizing capability of the current system was to
`doorway, both cameras can see the person and re-
`give the system the ability to distinguish between
`port his or her location at nearly the same place.
`people and cars. The system classifies objects as
`The VSSreports this as one person, using a mini-
`cars by using their sizes and aspectratios. The size
`mum distance to allow forerrors in location. When
`of an object in feet is obtained using the AVS sys-
`Camera-2 first sees a person at a location near the
`tem’s
`image
`coordinate
`to world coordinate
`doorway and reports this to the VSS,
`the VSS
`mapping. Oncethe system hasdetectedacar, it an-
`checks to see if Camera-1 recently reported a per-
`alyzes the motion graph to recognize new events.
`son near the door. If so, the VSS reports the person
`in the room as the same one that Camera-1 had
`been tracking in the hall.
`
` era-1 monitors
`
`2.3. Vehicle event recognition
`
`This section describes extensions to the existing
`AVS system that enable the recognition of events
`involving interactions of people with cars. These
`new capabilities enable smart security cameras to
`monitor streets, parking lots and drivewaysand re-
`port when suspicious events occur. For example, a
`smart camerasignals an alarm when a person exits
`a car, deposits an object near a building, reenters
`the car, and drives away.
`
`2.3.1. Scope and assumptions
`
`Extending the AVS system to handle human-vehi-
`cle interactions reliably involved two separable
`subproblems. First,
`the system’s vocabulary for
`events and objects must be extended to handle a
`new class of object (vehicle) and new eventtypes.
`Second,
`the AVS moving object detection and
`tracking software must be modified to handle the
`outdoor environment, which features variable
`lighting, strong shadows, atmospheric disturbanc-
`
`2.3.3. Car event recognition
`
`In principle, car exit and car entry events could be
`recognized by detecting characteristic interactions
`of blobs in difference images, in a manner similar
`to the way AVS recognizes DEPOSIT and RE-
`MOVEevents. In early experiments, however, this
`method turned outto be unsatisfactory because the
`underlying motion segmentation method did not
`segment cars from people. Whenever the people
`pass nearthe car they appear to merge with it, and
`track is lost until they walk away from it.
`
`To solve this problem, a new approach involving
`additional image differencing was developed. The
`technique allowsobjects to be detected and tracked
`even when their images overlap the image of the
`car. This method requires two reference images:
`one consists of the original background scene
`(background image), and the other is identical to
`the first except it includes the car. The system takes
`differences between the current video image and
`the original reference image as usual. However,it
`also differences the current video image with the
`reference image containing the car. This allows the
`
`269
`
`Canon Ex. 1051 Page 5 of 19
`
` Canon Ex. 1051 Page 5 of 19
`
`

`

`system to detect objects which may be overlapping
`the car. Using this technique, it is easy to detect
`whenpeople enter and exit a car. If an object disap-
`pears while overlapping with a car,
`it probably
`entered the car. Similarly, if an object appears over-
`lapping a car, it probably exited the car.
`
`2.3.4. Basic method
`
`Whena car comesto rest, the following steps are
`taken.First, the image of the car object is removed
`from its frame and stored. Then, the car image is
`merged with the background image, creating an
`updated reference image containing the car. (Ter-
`minology: a reference car image is the subregion
`of the updated reference image that contains the
`car.) Then, the car background image, the region of
`
`the original background imagethat is replaced by
`the car image, is stored.
`
`For each successive frame, two difference images
`are generated. One difference image,
`the fore-
`ground
`difference
`image,
`is
`calculated
`by
`differencing the current video image with the up-
`dated reference image. The foreground difference
`image will contain all the blobs that represent ob-
`jects other than the car, including ones that overlap
`the car. The second difference image, the car dif-
`ference
`image,
`is
`calculated using the
`car
`background image. The car difference image is
`formed from the difference between the current
`frame and the car background image, and contains
`the large blob for the car itself. Figures 3 and 4
`show the construction and use of these images.
`
`
`
`
`(b)
`
`Figure 3: (a) Background image.(b) Car background image.
`(c) Updated reference image
`
`
`
`=>
`
`
`cco
`
`(a)
`
`Figure 4: (a) Current video image. (b) Foreground difference image
`
`270
`
`Canon Ex. 1051 Page 6 of 19
`
` Canon Ex. 1051 Page 6 of 19
`
`

`

`
`
`
`reference frame
`
`previous frame
`
`current frame
`
`/
`
`|
`
`stored car
`object
`
`
`
`ee er
`
`ee
`
`car object
`
`7 b
`
`frameprior to
`car resting
`
`car resting
`frame
`Figure 5: Creation of the motion graph.
`The starred framerepresents the framepriorto the background image being updated.
`an INCAR eventis reported. If an object disap-
`pears while intersecting a car object, an OUTCAR
`event is reported. Figure 6 showsthe output of the
`system. The system will continue to operate in this
`manneruntil the car in the reference frame begins
`to move again.
`Whenthe car movesagain, the system reverts to its
`normal single-reference-image state. The system
`detects the car’s motion based on the movement of
`its centroid. It compares the position of the cen-
`troid of the stored car object with the centroid of
`the current car object. Figure 7 shows the slight
`movementofthecar.
`
`The blobs in the foreground difference image are
`grouped into objects using the normal grouping
`heuristics and placed in the current frame. The
`blobs in the car difference image necessarily repre-
`sent the car, so they are all grouped into one current
`car object and placed in a special reference frame.
`Normallinks occur between objects in the previous
`frame andobjects in the current frame. Additional-
`ly, the stored car object, which was removed from
`its frame, (from Step 1) is linked to the current car
`object which is in the reference frame. In any given
`sequence, there is only one reference frame.
`Figure 5 demonstrates the creation of this new mo-
`tion graph. As indicated by the dotted lines, all
`objects maintain their tracks using this method.
`Notice that even though the car object disappears
`from future frames (due to the updated reference
`image),it is not detected to have exited becauseits
`track is maintained throughoutevery frame. Using
`this method, the system is able to keep track of the
`car object as well as any objects overlapping the
`car. If an object appears intersecting a car object,
`
`
`
`Figure 6: Final output of system
`
`oe
`
`a
`
`(b)
`
`(c)
`Figure 7: (a) Reference car image. (b) Moving car image.
`(c) Reference car difference image. (d) Moving car difference image
`
`(d)
`
`271
`
`Canon Ex. 1051 Page 7 of 19
`
` Canon Ex. 1051 Page 7 of 19
`
`

`

`transient objects grouped
`aeno ae >i es
`
`
`resting.~ stored car ~ together to form new
`
`“
`moving car object in
`‘\
`f
`object
`transient objects
`current frame
`\
`/
`removed in
`\
`reference frameps
`previous frame
`~— _
`moving car
`object
`
`s
`
`=
`
`previous frame
`
`\
`
`
`
`current frame
`
`next frame
`
`/
`
`Figure 8: Restoration ofnormal differencing. The starred frame represents the last frame prior to the
`original reference image being restored.
`
`If the centroid locations differ by more than a
`threshold, the following sequence of events occur
`to restore the systemtoits original state:
`1. An object representing the moving car is
`created in the current frame.
`2. The stored car objectis linked to this new
`moving car objectin the current frame.
`3. Objects in the previous framethatintersect
`the moving car are removed from that
`frame.
`4. The car background imageis merged with
`the updated reference image to restore the
`original reference image.
`5. Normaldifferencing continues.
`
`Figure 8 demonstrates how the system is restored
`to its original state. Note that there is one continu-
`ous track that
`represents the path of the car
`throughout.
`
`Whenthe car begins to move again,transient blobs
`appear in the foreground difference image due to
`the fact that the car is in the updated reference im-
`age as seen in Figure 9. Therefore, to create a new
`moving car objectin the current frame, these tran-
`sient objects, which are
`identified by their
`intersection with the location of the resting car, are
`
`grouped together as one car object. If there are no
`transient objects, a copy of the stored car objectis
`inserted into the current frame. This way, there is
`definitely a car object in the current frameto link
`with the stored car object. Transient objects might
`also appear in the previous frame when a car is
`moving. Therefore, these transient objects must be
`removed from their frame in order to prevent them
`from being linked to the new moving car object
`that wasjust created in the current frame. After the
`steps described above occur,the system continues
`as usual until another car comesto rest.
`
`2.3.5. Experiments: disk-based sequences
`
`Totest the principles behind the modified AVS sys-
`tem,
`three sequences of video that represented
`interesting events were captured to disk. These se-
`quences represented events which the modified
`system should be able to recognize. Capturing the
`sequences to disk reduces noise and ensures that
`the system processes the same frames on every run,
`making the results deterministic.
`In addition to
`these sequences, longer sequences were recorded
`and run directly from videotapeto test how the sys-
`tem would work underless ideal conditions.
`
`
`
`
`
`(b)
`(c)
`Figure 9: (a) Updated reference image. (b) Current video image. (c) Foreground difference
`
`image
`
`272
`
`Canon Ex. 1051 Page 8 of 19
`
` Canon Ex. 1051 Page 8 of 19
`
`

`

`filmed from the 3rd story of an office building
`overlooking the driveway in front of the building.
`A car drives up and a person exits the car, walks
`away, deposits a briefcase, and finally reenters the
`car. Then, the car drives away. In this segment, the
`system successfully detects the person exiting the
`car. However, the person entering the car is missed
`because the person gets grouped with a second per-
`son walking near the car.
`
` 2.3.5.1. Simple sequence. The first sequence was
`
`Further on in the sequence, the car drives up again
`and a personexits the car, walks away, removes the
`briefcase, and finally reenters the car. Again, the
`car drives away. In this segment, both the person
`entering and exiting the car are recognized. In both
`these sequences, there was only the one false nega-
`tive mentionedearlier and nofalse positives.
`
`es with the car in them. In this sequence, two
`people entered a car. However, both events were
`missed because the car was not recognizedasrest-
`ing dueto the dark lighting conditions on this rainy
`day.
`
`2.3.6.2. Cloudy day. This is a 13 minute sequence
`in the same location as the previous sequence ex-
`cept it is a cloudy day. In this time span, 9 cars
`passed through the camera’s field of view andall of
`them were detected by the system. There were a to-
`tal of 2 people entering a car and 2 peopleexiting a
`car. The system successfully detected them all. Ad-
`ditionally,
`it
`incorrectly reported one person
`walking near a car as an instance of a person exit-
`ing a car.
`
`2.3.6.3. Cloudy day—extended time. This is a 30
`minute sequence in the samelocation as the previ-
`ous two.In this time span, 28 cars pass through and
`was
`sequence
`2.3.5.2. Pickup sequence. This
`all of them were detected. The system successfully
`filmed in front of a house looking at the street in
`detected one person exiting a car but missed two
`front of the house. In the sequence, a person walks
`others. The two people were missed because the
`into the scene and waits at the curb. A car drives
`car was on the edge of the camera’s field of view
`up, picks up the person, and drives away. The sys-
`and so it was not recognized immediately asacar.
`tem correctly detects the person entering the car.
`There are nofalse positives or negatives.
`
`2.3.7. Evaluation of car-event recognition
`
`The modified AVS system performs reasonably
`well on the test data. However,
`it has only been
`tested on a small numberof videotaped sequences,
`in which much of the action was staged. Further
`experiments and further work with live, uncon-
`trolled data will be required to make the system
`handle outdoor vehicle events as well as it handles
`indoor events. The technique of using multipleref-
`erence imagesis interesting and can be applied to
`other problems, e.g. handling repositioned furni-
`ture in indoor environments. For more detail on
`this method, see [Tserng, 1998].
`
`2.4. Complex events
`
`sequence was
`2.3.5.3. Drop offsequence. This
`filmed in the samelocation as the previous one. In
`this sequence, a car drives up and a person Is
`dropped off. The car drives away with the person
`still standing in the samelocation. Then, the person
`walks off. The system correctly detects the person
`exiting the car and does not report a false enter
`event when the car moves away.
`
`2.3.6. Experiments: videotaped sequences
`
`These sequences were run on the system straight
`from videotape. These were all run at a higher
`threshold to accommodate noise on the videotape.
`However, this tended to decrease the performance
`of the system.
`
`The AVS video monitoring technology enables the
`recognition of specific events such as when a per-
`2.3.6.1. Dark day. This is a 15 minute sequence
`son enters a room, deposits or picks up an object,
`that was recorded from the 3rd floor of a building
`or loiters for a while in a given area. Although
`onafairly dark day. In that time span, 8 cars passed
`these events are more sophisticated than those de-
`through the camera’s field of view. The system de-
`tected via simple motion detection, they are still
`tected 6 cars correctly and one false car (due to
`unstructured events that are detected regardless of
`people grouped together). One car that was not de-
`the context in which they occur. This can result in
`tected was dueto its small size. The other car was
`alarms being generated on events that are not of
`undetected because the system slowed down (due
`interest.
`to multiple events occurring) and missed the imag-
`
`273
`
`Canon Ex. 1051 Page 9 of 19
`
` Canon Ex. 1051 Page 9 of 19
`
`

`

`far, and the objects involved in them. Whenever the
`first sub-event
`in a complex event’s sequence is
`recognized, an activation for that complex event is
`created. The activation contains the /D of the ob-
`ject involved in the event, and an index, which is
`the number of sub-events in the sequence that have
`been recognized thusfar. The indexisinitialized to
`1 when the activation is created, since the activa-
`tion is only created when the first
`sub-event
`matches. The system maintains a list of current ac-
`tivations for each defined complex-event
`type.
`Whenever any new eventis recognized,thelist of
`currentactivations is consulted to see if the newly
`recognized (or incoming) event matches the next
`sub-event in the complex event.If so, the index is
`incremented. If the index reaches the total number
`of sub-events in the sequence, the complete com-
`plex event has been recognized, and any desired
`alarm can be generated. Also, since the complex
`event that was just recognized may also be a sub-
`event of another complex event, the activation lists
`are consulted again (recursively) to see if the indi-
`ces of any other complex eventactivations can be
`advanced.
`
`For example, if the system is monitoring a room or
`store with the intention of detecting theft, the sys-
`tem could be set up to generate an alarm whenever
`an object is picked up (i.e., whenever a REMOVE
`event occurs). However, no theft has occurred un-
`less the person leaves the area with the object. A
`simple, unstructured event
`recognition system
`would generate an alarm every time someone
`picked up an object, resulting in many false alarms:
`whereas a system that can recognize complex
`events could be programmed to only generate an
`alarm when the REMOVEeventis followed by an
`EXIT event. The EXIT eventprovides context for
`the REMOVEeventthat enables the system to fil-
`ter out uninteresting cases in which the person does
`not leave the area with the object they picked up.
`This section describes the design and implementa-
`tion of such a complex-event recognition system.
`
`Weuse the term simple event to mean an unstruc-
`tured atomic event. A complex eventis structured,
`in thatit is madeup of one or more sub-events. The
`sub-events of a complex event may be simple
`events, or they may be complex, enabling the defi-
`nition of event hierarchies. We will simply say
`eventto refer to an event that may beeither simple
`or complex. In our theft example above, REMOVE
`and EXIT are simple events, and THEFT is a com-
`plex event. A user may also define a further event,
`e.g., CRIME-SPREE,which mayhave one or more
`complex THEFT events as sub-events.
`
`the complex
`To return to our THEFT example,
`THEFT event has two sub-events, REMOVEand
`EXIT. When a REMOVEevent occurs, an activa-
`tion for the THEFT eventis created, containing the
`ID of the person involved in the REMOVE event,
`and an indexset to 1. Later, when another eventis
`recognized bythe system,the activation is consult-
`Wecreated a userinterface that enables definition
`ed to see if the event type of this new, incoming
`of a complex event by constructingalist of sub-
`event matches the next sub-event in the sequence
`events. After one or more complex events have
`(in this case, EXIT). If the event type matches, the
`been defined, the sub-events of subsequently de-
`object ID is also checked,in this caseto see if the
`fined complex events can be complex events
`person EXITing is the same as that of the person
`themselves.
`who REMOVE4¢ theobjectearlier. This is to ensure
`that we do not signal a THEFT event when one
`personpicksup an objectand a different person ex-
`its the area. In a closed environment, the IDs used
`may merely be track-IDs, in which each object that
`enters the monitored area is assigned a unique
`track-ID, and the track-ID is discarded when the
`object is no longer being tracked.If both the event
`type and the object ID match, the activation’s index
`is incremented to 2. Since there are only 2 sub-
`events in the complex eventin this example, the en-
`tire complex-event has been recognized, and an
`alarm is generated if desired. Also, since the
`THEFT eventhas been recognized,this newly rec-
`ognized THEFT event may be a sub-event of
`
`2.4.1. Complex-event recognition
`
`Oncethe user has defined the complex events and
`the actions to take when they occur, the event rec-
`ognition system recognizes these events as they
`occur in the monitored area. For the purposes of
`this section, we assume a priori that the simple
`events can be recognized, and that the object in-
`volved
`in
`them can
`be
`tracked.
`In
`the
`implementation we will use the methods discussed
`in [Courtney, 1997, Olson and Brill, 1997] to track
`objects and recognizethe simpleevents. In order to
`recognize a complex event, the system must keep a
`record of the sub-events that have occurred thus
`
`274
`
`Canon Ex. 1051 Page 10 of 19
`
` Canon Ex. 1051 Page 10 of 19
`
`

`

`another complex event. When the complex THEFT
`event is recognized, the current activations are re-
`cursively checked to see if the theft is a part of
`another higher-level event, such as a CRIME-
`SPREE.
`
`2.4.2. Variations and enhancements
`
`We have described the basic mechanism of defin-
`ing and recognizing complex events. There are
`several variations on this basic mechanism. Oneis
`to allow unordered events,
`i.e., complex events
`which are simply the conjunction or disjunction of
`their sub-events. Another is to allow negated sub-
`events, which can be used to cancel an activation
`when the negated sub-event occurs. For example,
`considering the definition for THEFT again,if the
`person paysfor the item, it is not a theft. Also, if
`the person puts the item back downbefore leaving,
`no theft has occurred. A more complete definition
`of theft is one in which “a person picks up an item
`and then leaves without putting it back or paying.”
`Assuming we can recognize the simple events RE-
`MOVE, DEPOSIT, PAY, and EXIT, the complex
`THEFT event can now be expressed as the ordered
`list (REMOVE, ~DEPOSIT, ~PAY, EXIT), where
`“.” indicates negation. Another application of the
`complex event with negated sub-eventsis to detect
`suspicious behaviorin front of a building. The nor-
`mal behavior may be for a person to park the car,
`get out ofit, and then comeup into the building. If
`the person parks the vehicle and leaves the area
`without coming upinto the building, this may be a
`car bombing scenario. If we can detect the sub-
`events for PARK, OUTCAR, ENTER-BUILDING,
`and EXIT,we can define the car-bombing scenario
`as
`(PARK, OUTCAR, ~ENTER-BUILDING,
`EXIT).
`
`Another variation is to allow the user to label the
`objects involved in the events, whichfacilitates the
`ability to specify that two object be different. Con-
`
`sidera different car bombing scenario in which two
`cars pull up in front of the building, and a person
`gets out of one car and into the other, which drives
`away. The event definition must specify that there
`are two different cars involved: the car-bomb and
`the getaway-car. This can be accomplished by la-
`belling the object
`involved when defining the
`event, and giving different labels to objects which
`mustbe different.
`
`Finally, one could allow multiple activations for
`the same event. For example, the desired behavior
`maybethat a separate THEFT eventshould besig-
`nalled for each item stolen by a given person, €.g.,
`if a person goes into a store andsteals three things,
`three THEFT events are recognized. The basic
`mechanism described above
`signals
`a_
`single
`THEFT event no matter how many objects are sto-
`len. We can achieve the alternate behavior by
`creating multiple activations for a given eventtype,
`differing only in the ID’s ofthe objects involved.
`
`2.4.3. Implementation in AVS
`
`We have described a method for defining and rec-
`ognizing complex events. Most of this has been
`implemented and incorporated into the AVS sys-
`tem. This
`subsection
`describes
`the
`current
`implementation.
`
`AVSanalyzes the incoming video stream to detect
`and recognize events such as ENTER, EXIT, DE-
`POSIT, and REMOVE. The primary technique
`used by AVS for event recognition is motion graph
`matching as described in [Courtney, 1997]. The
`AVSsystem recognizes and reports these events in
`real time as illustrated in Figure 10. Whenthe per-
`son enters the monitored area, an ENTEReventis
`recognized as shown in the image on the left.
`Whenthe person picks up an object, a REMOVE
`event is recognized, as depicted in the center image
`
`below. When the person exits the area, the EXIT
`
`Figure 10: A series of simple events
`
`275
`
`Canon Ex. 1051 Page 11 of 19
`
` Canon Ex. 1051 Page 11 of 19
`
`

`

`event is signalled as shown in the image on the
`right
`
`While the AVS system recognizes numerousevents
`as shown above, the user can select which events
`are of interest by providing the dialog box interface
`illustrated in Figure 11. The user selects the event
`type, object type, time, location, and duration of
`the event of interest using a mouse. The user can
`also select an action for the AVS system to take
`whenthe event is recognized. This dialog box de-
`fines one type of simple event; an arbitrary number
`of different simple event types can be defined via
`multiple uses of the dialog box. Theillustration in
`Figure 11 shows a dialog box defining an event
`called “Loiter by the door’ which is triggered
`when a personloiters in the area near the door for
`more than 5 seconds.
`
`AVSwill generate a voice alarm and write a log en-
`try when the specified event occurs. If the eventis
`only being defined in order to be used as a sub-
`event in a complex event, the user might not check
`any action box, and no action will be taken when
`
`the event is recognized exceptto see if it matches
`the next sub-event in a complex-eventactivation, or
`generate a new activation if it matches the first sub-
`event in a complex event.
`
`After one or more simple events have been defined,
`the user can define a complex event via the dialog
`box shown in Figure 12. This dialog box presents
`two lists: on the left is a scrolling list of all the
`event types that have been defined thusfar, and on
`the right is a list of the sub-events of the complex
`event being defined. The sub-eve