ATTACHMENT F
`
`TO REQUEST FOR EX R-éiRTE REEXAMINATiON OF
`US. PATIEN'I‘ \0 "Ffiéflfiiz
`
`AVIGILON EX. 2005
`
`IPR2019-00235
`
`Page 1 of 18
`
`AVIGILON EX. 2005
`IPR2019-00235
`Page 1 of 18
`
`

`

`Moving Object Detection and Event Recognition Algorithms for Smart Cameras
`
`Thomas J. Olson
`
`Frank Z. Brill
`
`Texas Instruments
`
`Research & Development
`
`HO. Box 655303. MS 83%, Dalian, TX 76266
`
`E—maii: olson®csc.ti.com, brill@ti.com
`
`hop:fs‘www.ti.comfresearchfdocsliubalindexmml
`
`Abstract
`
`Swan video cameras analyze the video stream and
`translate it into a description of she scene in terms
`of objects. object motions, nod events. This paper
`describes a set of nigofithms for the core compnm
`{ions needed to build smart cameras. Together
`{hose algorithms make up the Anionomons Video
`Surveillance {AVS} system, a generalnpnrpose
`framework for moving object demotion and event
`mcognition. Moving objects are detected using
`change detection, and are tracked aging first-order
`prediction and nenrost neighbor matching. Events
`are recognized by appiying predicates to the graph
`fomieri by linking corresponding obiecls in sncees»
`sine frames.Tho AVS algorithms have been used to
`create severe} novel video snweiilance applicaw
`tions. 'l‘hese include a video surveillance shell that
`
`allows a human to monitor the outputs of multiple
`cameras, :1 system that takes a singie high-quality
`snapshot of every person who enters its field of
`View, and a system that 'ieamr tho structure of the
`monitored environment by watching humans move
`around in the scene.
`
`1 Introduction
`
`ages and video clips. but the-so will be carefully
`selected to maximizo their useful infomiation con-
`
`tent. The symboiic infomiation and images from
`omen cameras will be filtered by program that ex—
`tract darn mievnnl to particular tasks. This filtering
`process will enable a singie human to monitor hun-
`dreds or thousands of video sitcoms.
`
`in pursuit of our research objeerives I'Hincbbaugh,
`£997}, we are developing the technology needed to
`make smart cams a reality. Two fundamental ca—
`pabiiities are needed. The first in the ability to
`describe scenes in icons of object moons and in-
`teractions. The second is the ability to recognize
`important events that occur in the scene, and to
`pick. out those that no: relevant to the current task.
`Those capabilities make it possibie to develop a va—
`riety of novel and useful video surveillanco
`applications.
`
`1.] Video Surveillance and Monitoring
`Scenarios
`
`Our work is morivatod by a several types of video
`surveillance and monitoring scenarios.
`
`Video cameras today produce images, which must
`he oxnmined by humans in order to be useful. fin»
`tore.
`‘smort‘ video cameras wiii produce infor-
`mation. including descriptions of the environment
`they are monitoring and the events taking place in
`it. The information they produce may inoinde km
`
`The research described in this report was soonsored in {Jan by
`tho DARPA image Understanding Program.
`
`Indoor Surveillance: Indoor mmiliance provides
`information about areas such as. building lobbies,
`hallways. and offices. Monitoring tasks in lobbies
`and hallways include detection of people deposit—
`ing things (ego, unattended luggage in an airport
`lounge}. removing things (e.g.. their}, or loitering.
`Office monitoring tasks typically require informa-
`tion about people’s identities:
`in an office, for
`example, the. office owner may do anything at any
`
`159
`
`AVIGILON EX. 2005
`
`IPR2019-00235
`
`Page 2 of 18
`
`AVIGILON EX. 2005
`IPR2019-00235
`Page 2 of 18
`
`

`

`time. but other people should not open desk draw—
`ers or operate the computer untess the owner is
`present. Cicaning stat? may come in at night to vac—
`uum and empty trash cans, but should not handic-
`ohjects on the desk.
`
`Cmtdoor Surveillance: Untdoor snrveiiinnoc in—
`
`cludes tasks such as monitoring a site perimeter for
`intrusion or threats From vehicies (tag. car bombs).
`In military apptications, video sorveiitance can
`function as a sentry or forward observer, e.g. by
`notifying commanders when
`enemy
`soldiers
`emerge from a wooded area or cross a road.
`
`In order for smart cameras to be practicai for real-
`
`worid tasks, the aigorithms they use must be to-
`bust. Current
`commercial
`video surveiliartce
`
`systems have a high fatse alarm rate {Ringier and
`Hoover, £995], which renders them useiess for
`roost applications, For this reason. our research
`stresses robustness and quantification of detection
`and faise aiarm rates. Smart camera atgorithms
`must also run effectivoiy on Iow—cost piatforms. so
`that they can be implemented in smaii, towpowcr
`packages and can be usad in large numbers. Study~
`ing algorithms that can run in near real time makes
`it practical
`to conduct extensive evaluation and
`testing of systems, and may enable. wonhwhiic
`near-term appiications as well as contributing to
`longvtcrm research goats.
`
`1.2 Approach
`
`The first step in processing a video stream for sur-
`veittance purposes is to identify the important
`objects in the scene. In this paper it is assumed that
`the important objects are thoso that more indepen—
`dentiy. Camera paramters are assumed to be fixed.
`This atiows the use of simple change detection to
`identify moving objects. Whore use of moving
`cameras is necessary, stabilization hardware and
`stabilized moving object detection aigorithms can
`be used {c.g. {Burt et at, 1989, Nelson. 19911.1'he
`use of criteria other than motion te.g., salience
`based on shape or coior, or more genera} object
`recognition) is compatihie with our approach, but
`these criteria
`are not
`used
`in our
`current
`
`apptications.
`
`Our event recognition algorithms are based on
`graph marching. Moving objects in the image are
`
`169
`
`tracked overtime. Observations of an object in sue—
`cossive video frames are linked to form a directed
`
`graph (the motion graph}. Events are defined in
`terms of predicates on the motion graph. For in-
`stance.
`the beginning of a chain of successive
`ohsewations of an object is defined to be an EN-
`TER event. Event detection is described in more
`
`detail bciow.
`
`Our approach to video surveiilance stresses 2D,
`imagotmed aigorithms and simple. tow-tore] ob-
`jecr representations that can be extracted reiiahiy
`from the video sequence. This emphasis yieids a
`high lever of robustness and low computational
`cost- ()bject recognition and other detaited anaiyv
`ses are used onty after the system has determined
`that the objects in question are interesting and mer-
`it further investigation,
`
`1.3 Research Strategy
`
`The primary technical goal of this research is to do
`velop generai-pnrpose algorithms
`for moving
`object detection and event recognition. These algo-
`rithms
`comprise
`the Autonomous
`Video
`Surveiiiance (AVS'j system, a modniar framework
`for miiding video surveiiianco applications. AVS
`is designed to be updated to incorporate better core
`aigorithms or to tone the processing to Specific do
`mains as our research progresses.
`
`In order to evaluate the AVS core algorithms and
`event recognition and tracking framework. we use
`them to develop apptications motivated by the snr~
`vciiiance
`scenarios
`described
`above.
`The
`
`applications are smalioscaie implementations of to
`tote smart camera systems. They are designed for
`tong-tom: operation, and are evaluated by al'towing
`them to run for tong periods (hours or days} and
`analyzing their output.
`
`The remainder of this paper is organized as folv
`tows. The next section discusses related work.
`
`Section 3 presents the core moving object detection
`and event recognition algorithms, and the mocha
`nism used to establish the 39 positions of objects.
`Section 4 presents appiications that have been built
`using the [NS framework. "fire final! section dism
`cusses the current state of the system and our
`future plans.
`
`AVIGILON EX. 2005
`
`IPR2019-00235
`
`Page 3 of 18
`
`AVIGILON EX. 2005
`IPR2019-00235
`Page 3 of 18
`
`

`

`2 Related Work
`
`Oar overaii approach to video sun'eiiiance has
`been influenced by interest in seicctive attention
`and task—oriented processing {Swain and Stricken
`3991, Rintcy and Brown,
`i993, Camus et at,
`1993'}. The fundamental probicm with current vid—
`eo surveiiiance technology is
`that
`the usent
`information density of the images delivered to a
`human is very tow; the vast majority of surveii—
`lance video frames contain no usefni information
`
`at at}. The fundatnentai rote of the smart camera
`described above is to reduce the voiumc of data
`
`produced by the camera. and increase the value of
`that data. It does this by discarding irreievant
`frames, and by expressing the information in the
`relevant frames primariiy in symhoiic form.
`
`2.1 Moving iject Detection
`
`Most algorithms for moving object detection using
`fixed cameras work by comparing incoming video
`frames to a reference image, and attributing signifi»
`cant differences either to motion or to noise. The
`
`aigorithms differ in the form of the comparison op—
`erator they use. and in the way in which the
`
`reference image is maintained. Simple intensity
`diiierencing foiiowed by threshoiding is widciv
`used {Iain ct at, 19?9, Yalarnanchiii at al., 1982,
`Keiiy at 31.. 2995. Bobick and Davis, 1.996. Court“
`nay,
`i9???
`because
`it
`is
`computationaiiy
`inexpensive and works quite wait in many indoor
`
`environments. Some algorithms provide a means of
`adapting the reference image overtime, in order to
`track slow changes in fighting conditions undfor
`changes in the environment {Karmann and von
`Brandt, 1990. Makerov, 1996a}. Some aiso fitter
`
`the image to reduce or remove tow snatiai {requiem
`cy content, which again makes the detector icss
`sensitive to lighting changes {Makarov or at,
`1996!). Kolicr or 31.. 1994}.
`
`Recent work [Pentiani 1996, Kano ct 31., 1996}
`
`has extended the basic change detection paradigm
`by repiacing the reference image with a statistical
`model of the background. The comparison operator
`becomes a Statistical test that estimates the proba—
`
`bility that the observed pixel voice heiongs to the
`background.
`
`Our hasciinc change detection aigoridnn uses
`thresholdcd absolute differencing, since this works
`wet! for our indoor snrvciilance scenarios. For no
`piications when: fighting change is a problem, we
`use the adaptive reference frame aigorithrn of Kar-
`mann and von Brandt {2990]. We are also
`experimenting with a probabilistic change detector
`similar to Pfinder [Pentinod, i996.
`
`Our work assumes fixed cameras. When the cam-
`
`era is not fixed, simple change detection cannot be
`used because of background motion. One approach
`to this problem is to treat the scene as a collection
`of independently moving objects. and to detect and
`ignore the visnai motion due in camera motion
`[c.g. Bun et al, 1989} Other researchers have prov
`posed ways of detecting features of the optical flow
`that are inconsistent with a hypothesis of seif mo~
`tioo Watson. 199?}.
`
`In many of our applications moving object detec—
`tion is a preiude to person detection. There has
`been significant recent progress in the development
`of algorithms to locate and track humans. Pfinder
`(cited above} uses a coarse statisticai modci of hu—
`
`man body geometry and motion to estimate the
`likelihood that a given pixei is pan of a human.
`Several researchers have described methods of
`
`tracking human body and timh movements {Gavri-
`to and Davis, i996, Kakadiaris and Memos, 3996}
`and locating faces in images [Song and Poggio,
`i994. Rowlcy or at.
`i996}. Intilic and 'Bohick
`£1995] describe methods of tracking humans
`through episodes of mutuai occiusion in a highiy
`structured environment. We do not currently make
`use of these techniques in live experiments because
`of their computational cost. However, we expect
`that this type of anaiysis wili eventuaiiy be an im-
`ponzmi part of smart camera processing.
`
`2.2 Event Recognition
`
`Most work on event recognition has focussed on
`events that consist. of a well-defined sequence of
`primitive motions. This ciass of events can be con-
`verted into spatioternporai patterns and recognized
`using statistical patient matching techniques. A
`number of researchers have demonstrated aigo~
`ritnms for recognizing gestures and Sign ianguagc
`{e.g., Sterner and Pendant}.
`i995}. Bohick and
`Davis {1996] describe a method of recognizing ate,
`
`161
`
`AVIGILON EX. 2005
`
`IPR2019-00235
`
`Page 4 of 18
`
`AVIGILON EX. 2005
`IPR2019-00235
`Page 4 of 18
`
`

`

`
`
`
`
`
`
`fiafemnm image
`
`wage» Frame
`
`ammo. image
`
`mmmm image:
`
`Fig-um i: Image. p-mcstsséngmays fig: aiming atfjeiei dais-mikm.
`
`gamma
`nmtimi
`Ifisfiypicgi
`anm-mandifig m.
`amines: suah as skiing {imam}, talking sir wavmg.
`
`{Jar appmamh is: avatar :‘emgnigiflfi is bazaar} on the
`iviidm fiatahase éadmiz‘tg work afi‘i‘iouriney é‘i‘éififl.
`wh’iezzh igmtaénmd the. 135:: af mmmm as me me“
`5333!? gz-‘agtth it} 'ztps'fiiém mama. Mmim graphs an:
`wan sisimii m mwmgming ahgimah ggnerii: mama
`Suit?! m: ‘dewgiting as: ehjgm‘ a; ’mniing gr: 5931“,.
`which an: {iifficuit a: satpzum axing this gammy
`233336 apg‘reaahea: m‘émsd m fihm‘i.
`€39
`ihg mixer
`Mimi. gaftemfiaaed appmachcs star: rcpmxem' mm»
`was mafians such '35 “thawing. an niqitci’ er
`"waning", which wouid he. difiimh {a} expiras‘ss‘r using.
`minim; graphs, Ii is iikfiiy that hath patiem‘imscs}
`am: mama? even: regrggniiiam Mchaiqum wéii he?
`flawed ti} hamfie {he {a}? fimga at“ swam that? am {3f
`matures! in anmaiflimcz: agp'iic-aiigang,
`
`'3 1W3 ’I‘mckiag anti Event Racogniiinn
`.Aiguriihmg
`
`"‘f’hia Stains!) damfibe‘s {ha mm: rflfihrwingifls than
`
`pmvifie the. asides: wrwiiiamra and mm‘nm‘mg <2: ..
`pahiiiiigs zxf my. AVE sysmu. That‘s zetre $12an $325;
`tawhmfis‘sgisrs:
`flaming abject dafigcimn.
`vi'suai
`tracking, ami eve-m meggmfion, ’E’hg'r- making {:hieai
`aim-mien famine-:3 dfiierzjiine wiéea em:- Gr mm}:— ah“
`
`jfiflifi mm 3 manimmd same; flack-1:3 which pimié‘.
`in 3 given vidsm {mm :cmmspumi m the. taming
`irrhgflctéa versus whiirh {kimis flmmapmé to the hackin-
`gmzsm‘i, and form 3 35mph! mpmsttniminn of the.
`sfififiefm image in me: $532.3: fmsm. This mpmséma
`Kim: is zeférwii m are: a maxim; .xggi'zm, am? i; eiim
`it} :1 siagifl vidw framm its fimifigtiith {mm- the:
`u—wréd rdfififl wfiich axis-i in me warm and give- rim
`{3 the. mmiwn ragisyn$
`
`“33:33 {rackmg mtméxm nf dg-ifl-rmimfig terrasgxmm
`{films batsmen the metim; mgiimx {war a
`amamma nf'vsdm frames, am! maimainiag a aifigfe
`ti:{amafifatitm‘ m errfi, far £3“:- wnrkfi (333%; which
`gave rise m 2hr. mxguenw 9f 233mm. regain; i1"? {192:
`acquaint a? fmnms. iximiiig, awn! ramgnixism is a
`mans; at“ imasiyxingihe miiamim} {if {mat-ks i3} 99258?
`to idcniif}? {Emma {if i'fitgtresji iavoiifiag that. wmki
`obicgiis mpikrammi fig xiv:- tracks.
`
`3.1}. firiwiagfihjeci{Magma-
`
`Th6- maxing ehjm: {Eat-fictkm {itchrmkkgy w: em-
`jfis‘i‘f if; a BI.) change-{Estamim {34$}:qu 3311333: in
`:ha: tiemfiimd £9 3.3;}: 8!" a3, mm; and Yaiamann
`chiii a!
`a? {$383}.
`it’riar £0 aciévzuitm cf the
`mmfimxing 5333mm 3:: image. 9:“ 3m: kmkgmzsmi‘
`Sign. m1 image: :35“ {he scent: which Cfifiméfls ms ifi¥§¥*
`mg, m“ eshemise En'iemsIEEng 9133mm; is sapz'umé m
`{NIKE-331513 régfli'mfim image. Wizm} the $323153; :55 in
`u_;xeraiiits.n;,
`the: ahflfiuifif diffs-”mam: a? me— czar-rem
`$391313 frame frat“ Rm mffimam imags‘: i3 mm'gmzazi
`1:: gst'céxm: a (lffifimnw fmagsn 'I'hgs fiii‘fiarmce in»
`age ix than ihreximédw at: an 3:3;wgzpriaic mine it}
`{sismia a himq; image in whim; she: “93%“ giséxsis mp»
`mam. hickgnmmi gawk; and {be “m” gsixeis
`mgr-mm “Lsmwi'ag Miner“ pixels.- The.
`fb<3r=ss<m~
`maid {Emmamms {5f arming sagas: gaimis in the:
`{Ermaimkiafi iimigfi are: me. mmhga mgims {mm Fig--
`uni-1}.
`
`Simpifl gpgiiafitiam {if we sweat. demfi'tism :pi‘mls‘f»
`dart: miiihcd imma- masks: $3: a 533mm: :3? fifl'flffig
`
`iatgei}: (his: 10 the iimiiaiitms: {3f i'fifeshaiding, mm:
`threarheiti aged is toe. km; gamma mix: aging wad-
`ows‘ wéii pmdxxacc. spm‘iums okj‘etziiz Mag-mas: a? gig:
`Ibms‘haid is {at} high, 33mm: 330mm: :3? {3m uhieais-
`in my: scent: wifi f—ai: R} by: segmratefi fmm the: hack.-
`
`'3"-»
`
`'
`3.3. i:
`
`AVIGILON EX. 2005
`
`IPR2019-00235
`
`Page 5 of 18
`
`AVIGILON EX. 2005
`IPR2019-00235
`Page 5 of 18
`
`

`

`in Which a. singie
`gimme, meehiwg we breakup,
`wwrifi {rifled gives rise. to severe} {waxiew re-giwwe;
`were; a ginger- $‘reme. {Tar genera: aepmeeh is? re
`wiiww hgewhng,
`'trw: age gmagiwg heuristic-s 1w
`ewrge rewiriwie err-weeeteri ewrwweeewm we: a sing-Te
`rrreziew regiexr and wraieiaiw e erreeeewe eerie--
`erewirkwee heme—tie wweiew regions; and wwrie'.
`rwhjeeie wrihiw ewe-h frame.
`
`{tine grwu'giwg zeeheiqee ee-erxteiey is 21'} metam-
`'iegieei were-m ne'- rhe ermine regions. Twie ewehiee
`the system 1w- merge mw‘rweegee Cwmgwwrewze sewer»
`rake h}? a few grixeh‘, be: ueieg this reehnzi'eue 2e
`were! Targe- gees {emits in w eewere werfemrewee
`degwwiwriwwr Mace-Wen diiwiiww in the image- space
`may reeeh hr inseam-ti}: wer'ghrg {Regent-whims
`whieh are... nearby in the imwge {a few pixele}, her.
`are in {wet segxareieri by a iatge diameter 'i'w The.
`wage {a T'ew feel}.
`
`if 313 iwi'wrwmtiew is awwiiehie. the {Emmeemd seam“
`
`grower}: greragweg eigwt‘iiwm makes new wf' we
`estimate erf the wire {in wwrid eewrrdiwwtee} wf {he
`ehgeers in the image The. hw-rrzwri‘rwg heiress cf the
`ewwweeted ewmpwwems are expwwiicd wewiewfiy and
`herieereaiéy by a eiweece measureri in feet {rarer-e:
`ihe‘e wixeie}. eeriewwweciee cwrwpwwerws w‘i 1?: {Were
`.i'epwfieg hermeiwg boxes are w—serged inter e siwg-Ee
`marker; region The meh'wirgwe fer estimating {he
`wee ei‘ {we-ehjee‘is "it: {he image is: described in eee~
`tier: '3»? weiww;
`
`3:43- Tiraekiwg
`
`The fwectiew {if rhe 4W8 tree-king mewwe is if: err“
`rahhsh camewwwrfiewees
`between
`the: met-kw:
`regime in the sew-err: frames-2mg: these it: the prewar
`ewe i'mrwe. W use the:
`ieehw'i'tgee wf' {Itrwriwey
`HEW}! Which wwweeds we fifliewe. Fir-teat. assume-
`rwzw we: haee' ewrwpezed 2f) weiwciiy eeiiwreiee fer
`the wmtiow regime i'w- {he preeirwre frame. Tireee we-
`Teehy estirwaree, ragtime: with the int—enema ef the.
`eewtreids in {he erewieee frame. are erred re pwrieew
`The: iemwiiens wt“ fire. ate-wwwiezs w? the wwrwww regime
`wee rhe warmer! {eerie Then a merger: rim-resin-
`
`ewierr'we
`neigfreee
`eeweswendeweew
`
`is:
`
`need
`
`he
`
`eerahiish
`
`ere- current frame. and he
`
`he were set. ef aii errei'r
`
`prefix-{ed ieewi-iews in {we eew'ewi' frerwe. Let {If he
`the we ef «sewer: region eesereie ieerwiene ie the-
`current
`irwwrer. If :he {iiwwwee theme-m p: awe
`Ci €£
`if iii ihe Siweiie'si is}! 331 eierweiwe-‘wf {L were?
`are: eiweeee is 3339 {he ewwtiiest er fire éfisiwwe235'
`trerwwew r"wed 333 ekmewts er P (5i.e. 9’. anti 2.“
`are. meme} weereet werghhiws}, {hen exiwhinh w cw!»
`resmmeerwe heme-nee e were at by ererrweg e-
`hieimctizmai rmrrzg fine. heiweew Thee}, flee the. we
`{memerw iziwre- awe ewece eezweew p 3er (1‘
`£53
`tie-{ermine a filigree}? eetiwewe {weeepreweri er
`grieeis gear sesame if {were is: ew {missing meek e03:
`miwiwg pf. gee £3- “: it. Otherwise, esiwbiwzh a were
`{rec-h, were we {we}: gri- erre a} m if.
`
`The same: iiwke' firm} the few-sis of me weeks with w
`highrwwi‘rdeece oi {heir em.eezeews ‘widew abjems
`Which do; ”we! flew: meme} weereei retigirherie w: the
`wrijweew: em- meg'feii w} fwww earnespwwrieweee
`became: the eseieriying war-tie ehieer e involve-d in
`we event (reg. enter; eew, tie-3mm, :ewwwwe}. In or»
`{fer 3e weeirw in fire- éiiemifieetiww ef' these eeewia
`whieere wewhen! stirring Tinker are given wwiéireeriwm
`wi werfi; {r'w'és {e the we: {mw—wruzwei} wearerw
`neighhwre {he week There rewreseet gramme err}
`{figure}! in {we {racking gmwcesei'i‘he wwwi'ew regime
`w: ail ef me frames, {age-{her wiih their emerge :91th
`week Three, were e merger: graph.
`
`i'w {he fig"
`figure 2?; weeks}; a wimp}: rmi'wrw 35mph.
`are,
`eweh. {were
`is; wwewiiwrwwsieweh
`are}
`is:
`
`mpresensed {r}: w eewiew! iiwe {Fifi ~- ere}. {Tireies-
`repreeem ehieers h}; the. wearer ere-ease mews rep--
`resent wrung weir-Sr am? the gray mews repreeem
`week: fine-s. we; whim eaters rhe were: in frame w ,
`end {he}; mews ihmwgh the scene. mm {were PET,
`where it deweeir‘s- a meme whieseh The fir-w wig-gem
`eewiiwwee {w mews threwgh the. eeewe emf exits we
`{ream $6. The (Segmeiteefi wwjeei remains siaiiwewry,
`.49.: frame 353 whether: {reject ware-re the serene? reww
`power}; eeeieeer:
`the: werierrery whim ex? frame.
`Fifi} {er is weiurieri by it}, awe thew previews: we
`move was: the ewwimetty aged; *i‘hix meme were»
`iwg wifieet reweeees dimeiiene meme frames 5TB
`were! fife-L mama to retrieve 'ihe watiuwwry (were! iw
`frame F113,- enri heat}? ewes in frame} H“, m: new
`iiwwe'i abject emers w {mince-‘3. ewdewwe swimmer
`we Wither}: iwzeeaetzirrg weir-33w ether siege—err
`
`Tea 1‘3 he the Set wt“- wrewww r'egiww emerged Tear»
`titres:
`in fire- gateflme frame, with $9.3 ewe. see}:
`Tandem, Let fi'ii’m the. prwjeeied migration ef gr;
`iw
`
`we endeared by we swiped iii} herteme iw figrwe 2.,
`{he mere-ct ewmewwwdeweee fer {we {make are w._ _*
`
`16.3
`
`AVIGILON EX. 2005
`
`IPR2019-00235
`
`Page 6 of 18
`
`AVIGILON EX. 2005
`IPR2019-00235
`Page 6 of 18
`
`

`

`5mm
`
`5mm
`
`93?
`
`
`
`:- QEFS‘SI? E‘s...
`
`as
`
`$1
`
`F3
`
`F3
`
`94
`
`1:5
`
`#5
`
`3-"?
`
`EXI?"
`
`
`
`
`1:1" ‘15:} iii _ 1 Ti?"
`
`(53% as. m: 1:: F2- 913 :M as 3°35
`
`F1? Fifi
`
`EN £253
`
`Figure} ‘3: £39m: dewmim in the maxim: graph
`
`{ht}
`interactmna sash as:
`GEE-$245?
`higumxs after
`OCGhiSiG?! ins frame Fifi. m: AVS sysiem remims
`{his ambigfity when: possibie. by preferring m
`mam?) aiming (mica: with waving objentisg am?
`maxiexmt‘y {mjecis with stationary nhjmtts‘ Tim dis—
`:gincfim Mmm} naming am: si'atianary Hacks.-
`is
`9:932:3qu mug thresheids 933.
`the: what-y $519
`mama, and hyxtarefiis far .smhiiiz-iag ifiifii‘sii‘ififlfi
`bemtzm waving and 2333599333
`
`$0330w'ing an {wcinaim}. {which may first far seven?
`{mums} the. frames imnmdimsztigy befmg 3:16 Min
`{fie mgigssmn m- wmyamd (tag, frames FE} and
`1313 in ié'igzsm El ‘i‘hg‘: AVE; system emmimtzt mt}:
`aiaiiunary fihjflci
`in dis: pra»mfli§$§0fi imam, zinc}
`maths-s far its camapmcfmt in the gmsL-mcisssion
`frame {which ahiéuki he Miami}; wheat it was ha»-
`faria,. sing-:23. {baa object is mfinaary; “mix pmcgdum
`imxuiw‘es a $3136 pcfiimi of aha imekéng amhiguifiex
`(imam! resaiuiifin a? ambigu'iz‘ies ramming {mm
`tnuifipfia moving {tifiefis in the. mam: is :3 {amt f0:
`further “twitch. The .AVS xyamm may Magi}:
`{ram inclusion a? a ”ciogefi warm 11:21::- king" timiiizy
`with ass
`that maceribmi by {mine am? .Bg'sfiicia
`'3 1:35:53, {9935},
`
`3.3 Event ifificfignitim
`
`Cam-3m Magma a}? mic kit $236 pairs of Ems—ks may
`want} to events. Farr cxamp‘im the 'bfiginning cf 2.1
`Iran}: cermpmds £23 an ENTER 3mm, am} the and
`Eififl‘é‘flwfidfi 1.0 an 'iE—XIT event in an finding Wei-n!
`datmrfim sy‘stmn 3: ii»; pnefembh: {mismat- ma: awn:
`
`:15 mar in aim: 2:33 imam-tibia {a fix * ammai iwfiufifi-fitfi
`of {he meant. “Hm gnaw-Ems swim} whiszh. usud me»
`{Em gi‘afihi far wan: éettfii'im} {Cnmzrmy,_. {9%};
`
`Bpemifid' in a hatch math, and mguimd muiiiph‘;
`
`flaws; We: the- mminu gragaig. precmding {xx-fine.
`epitramm. The ENS system imam {wants it; a xix“
`g§e 3:355 eve-r the mama! 333.3%, as {ha graph is
`Miami. I‘ifiwaficn in flfiifif f0 Edam. {frmfi {his in
`
`misc‘ 133;: wax: system inivodums a: Siigin stem {)5
`A frame, times {#23 in the carmm impifimemaiémg
`ins-imam:
`.regsfirzing mmin rum .Fex‘ swings: in
`Figuri: '3, an ems: mam! «Ream m: frame F}, The:
`.sW'S syzgeam requims ti: °- track is} his “minimum! for
`:‘e frame-s Nifiifi ramming the timer emu. E!“ the
`track nut nmimairmii far we mquércé uumber {32"
`franm. is is ignemi. axis? 1hr: eater awn-2' ézs mm? m-
`-§oflagi.( 2...“; if a 3: $3, the iéhjgut' it: Fignrs: .2 which
`{aims in frame: F5 and writs in {mm £93 wiii no:
`
`gmieraia an} €¥8§if§.
`
`A. mad: that gyms énm ma ireicks‘ «3m: {sf wiéigh ix
`
`moving mid the: mizcr' (32‘ whim '31:: mamas"; gas-m»
`sfi>mfis {its 3 EXEHBSIT mam; If :21 Ma'x‘ing‘ ism-2k
`iniermtts a. summary mask! mad men mafiausg 1:1:-
`snmfi, imi {he $318931;er Rae-3; ends at flit: 'inifirsegv
`
`iim, {his mrmsmmds a: a R'fififlfi’fi exam. The-
`
`rflmm wan! can he gfificraifiii as mm: 3:; £1th mu
`mam: (iismceimka the. incmiijm {if 83;: fli'giimmry
`thefl which was; mmawtde and the S’ffiifii’fl can £331»-
`Ecmfim: that me. staminary ehjexi is; was Esmgfl? m
`that imat‘mn..
`
`iii-f;
`
`AVIGILON EX. 2005
`
`IPR2019-00235
`
`Page 7 of 18
`
`AVIGILON EX. 2005
`IPR2019-00235
`Page 7 of 18
`
`

`

` a
`
`3-: Ei-etiiisiisiiing the. image {e 1313;} meriiinaie transfixmatien
`
`in 3 meme! Simiier his the {meiesiei‘i siteatien fie“
`
`ecriitseii aimve. in semen 3.3-. iihe degmsi'i event 3139
`ggi‘é'ttfi rise. is :iizriisigiiiry as in ieiiiitii abject iii {in} dew
`waiter, and which is tier: degxisitee‘ Fm exampie. i:
`way have men fine. the abject which eziieiefi at
`{mine F? ii? Figure. '31 flagged at frame $34 and tie“
`peeked e mmiflg {ibis-Area and ii"
`is {he {itixiiitt‘fl
`eigeet which then; preacetied in exit the geene- ii?
`81$. Again, {he .WS syiiem ieiies en a aiming Vt?!-
`smiimzity distineiien m remix-e. the iiiiibigiiiiy, and
`ineiais that the. eepesimfi semi-“tin smegma? Mixer .2:-
`{Eegmsii event; This SW3 5ysiem requires; him}: the
`{Eegiegitei and the iiepezsiiee {reeks {e esteem! few :3-
`f-remea‘ past the {wire iii which the. {mike Separaie
`{eigu pest flame $5. in Figure. 3}, am! inst:
`the-deu-
`gmiwfi etfiee? remake Biifiiflfiafyi
`(iiiiemise no
`{fem-eh men! is gesiereied‘
`
`Nee iieieeted {me net iiieeireiex} in Figure. ‘2}, exe-
`RfiSii" {reams {whee a waving shied messes m e-
`atepji, mi Mfifiifi meets {ween a:
`'iiES’iTieg eb‘ieez'
`hegine in mime. again}. Fiiiafiy. ei‘se- fimher even;
`{hie ig iieEeC-ted iii the LIGi-i'TSOEET event. which
`(Bi-”$2.6m whenever .4; 'iiirge change intents ever the es»
`tire image. The maiden graph med em be emeuhed
`so dame: this event.
`
`33:3 image to: Week! Mapping
`
`in awe: in irieete. objects; .aeen iii the image wiiii re“
`sped ti? :5 mega it
`is mmmary is} eetehiish ii
`snagging between image and map eetmiinitces, ‘E‘his‘
`mapping; is estahiishw in the MES sys—‘me by MW“
`iii'g a use: drew Qufifiri'ietemis eii the herizoiiiai
`
`surfaces: e‘iiiihie in as image, ami {he eerie-samediug.
`gi'saiiriiaiemis {in a map, es simmer:
`in Figure-3‘ 3k
`warp immeflmmiae {mm image {e meg) SCfX'HXfiu
`Mitzi:
`is cskeiimieii using five quediiimem!
`meminates‘
`
`{Ema the. iransf‘iamafiiem area-steiziir‘iimd, 1m: 533+
`iem caii eaiimae: {he imetiee iii" {in eeieez. {ex is
`Fififlithhmigi‘! 3m} Bennie! {WW}; 1);: assuming mat
`iii? ebjzeeis mt m: e heme-ma! surface. When. an
`{ib‘ieci is {16:1me is {he seem:a She,- ti‘iiiipeim e? the
`hives: siiie. pf git {Beaming tim: i2: used as the in»
`age. {mini 10 pmieitt inn: the mag: wiedew using me.
`quafiriiifletei warp Hensfbrsmiion ikfi’efimrg. 383993.
`
`.3: Aepiicatims
`
`'I’he- 53‘25'3 wee-- eigemiime tiesei‘ibed in fieiztien '3
`have: been eseifi are the; basis ii}: seemed whim set“
`
`weakness: ewii'cetiims. See-tiers. ii {ieecriiiee flame
`agipiimtiené the: we kiwi: impiemeieeéi simaeieeei
`awarenese, fiestwie ‘ seiectim fix? activity iegging‘
`am? eizeiieemee: Eeeiming,
`
`«$.53 Situatimai Awarencxs
`
`fie: gee; {if the. siteetitmei awareneefi :ipgsiiizaiiim i3
`{is gimdeee- a mai-iime inagx-‘haizeii iiiepiay of liifii io-
`eatiane {if gimme,
`eifieeis
`and avenge
`in a
`metliimmi regime and {e aik‘ms a user '26 smmify
`sienna. estimiitims iizteraerieeiy. Aiem eeniiiiiims
`may be haseii en the iiueatisms {if week 333:? iii}-
`ieeu»; in {he geese. {he {eyes iii @3in in me mean
`{he men-e; is} which 1311? peeiie 331$ {ikjtx'tts em if!"
`
`16.5
`
`AVIGILON EX. 2005
`
`IPR2019-00235
`
`Page 8 of 18
`
`AVIGILON EX. 2005
`IPR2019-00235
`Page 8 of 18
`
`

`

`
`
`Figure 5: Emr interface fer syec-ifyingga monkey it: QVS
`
`wivesai; and that times 32$ Whig}: the-miems (mum.
`
`Furmcrmflm. the use: can sgmcify the ziciien' 10' take
`when at: $333!“??? is tfiggcrmi. 6.3.3., {{3- generate an: my
`(1&0 3:33:37} or write: a ifig iiitzs‘ Em exmugig, 53%;: 859?
`sauna! in: aids”: ta sgm-‘zfy that an amiiu 3mm
`séhouiai he trigggfxmfi if a. 13-??me (iepmits a briefitaw
`an a given m5}: Mime}: 5133311: and "3:60 me am a
`Weakitighi.
`
`The- amhitecium 0f the A135? stimaz'immi awamnt‘s‘s'
`
`syfitfim is dfipigiéii in Figum 6%, The $¥§i§im consists;
`:3? am: a! 1mm: smart mantras mmmunimiing with
`a. Video S'urmi'iiatnse Sinai}. {VSS}. Each camera has
`
`as-Sacimefi with it w mciammim: AVS mm {éfigififi
`{hat parfanm the. gxrtx‘nasaiing fiesuibed in {Ramon '3.
`Tim is: the. mgme fimfis anti tracks smvifig abj‘ecis
`in. the arcane, maps. their ixmge imaiieag :0 Warm
`
`cmrfiéflfiias, mid {samgnim fivmrs immiuing ms
`objactx Each mm engine. midis a sfiremn 0f wean
`iifln and twin: mm)!“ £9 {he V33 whisk fiitm‘ {ha
`Ema-suing ass-mm Sire—mm fm agar-swcfiiad warm
`can-nafisiens-amfi “$33 an: ammgrimr: actions,
`
` 3
`
`
`W almze
`gm;
`mm Camera 3
`
`%_
`512:? ..
`.. >
`
`055% {mam}
`
`Figure q,; 133$ sit‘aiafiwa! sjiwatamzas sygem
`
`166
`
`in Maia}? {a «flaming um:- isiemiiies {if abjecig; {fig
`Mafméec, fiififlbflfik}, {he seitmtim'ai awfifizmss x33-
`tcm minimums-aim; wizh {we at mare abject
`maiysirz- tmdfiififi {HA-Ms}, 3?»: cm engines as?“
`am: mammals af inmmxing 013mm in that scenes.
`am! {award the gampsfimas in ma. 3AM, aieng wiih
`the I95 9:" {ha Hacks cefltaining the cfijccis.. The:
`SAM that pmsscfifimg {he snagxth is; orgies!“ is: timer—
`mine: the 133;)»: at” <§h§$fitfi1¢fif¥§§ pmcassing and.
`fin: ENS gaze mgina campumtims are :isytwhroa
`new.
`:20 the mm angina- may harm pracessm!
`sever-a: mm: {stamens by 'iime ihe‘ {MM Wings-refi-
`its anfiysfi (him. the analysis is mmgieie.
`i'hfi:
`(mm. wads ifie- msnhs {an objmt £353: £3bei} am}
`that trim: if} hack 1mm: mare. engiaefi ‘I‘fw mm 61:»
`gfine um; me: {rat-$1 i13- m 359mm the .iahai with
`the amaze; abjm' in the mumm' imam {assuming
`the: (shied. has; :emineé in {has sag-3R: 8135i {mm may
`eessfitiiy nackad}.
`
`The V35 gmvides a map dixpiay fif the: nnmimmd
`{ma} wizh m»: immims ”inf (in: afiéetis- 5;: {ha mane
`mpcma as mam-m: sin: amp; “I’m: VSS aim wows
`the. use: {:3 spimify aiarm mgifins and candélians.
`Mann mgimm am awe-fiat} by émwing than: {3:}
`{he may :35ng a mum-z, am} nanfifig {Mm a den
`simii‘ The flag-r cm: {him spat-52$ {he madiiitjms mi?
`actiam I'm stigma:- h}? emitting em: 0: max-“é: was?
`mm: ”Figure 3 aepicu me- mm‘s’mr swath»: fiiaiog
`£333. The um; names the: mamiim am was the:
`13mm {:3 misc: shack imam: assmtiaied with an:
`
`.m-miizians that xvii} iriggm‘ the mmiixfir. The use!
`seiaizts the fig“! uf avg-at,
`'the {51:32.
`(31‘ ehjgci ifi“
`waived in the mam, 13m day of wars}: and time at“
`day a? £332 warm, Wham that await News; 5m! what
`:0 dc: when it»: 31am: madiiim mam-m. “this marlin
`
`m: symifimi ix: Figun: 3 specifisxa that a wine 33am:
`
`AVIGILON EX. 2005
`
`IPR2019-00235
`
`Page 9 of 18
`
`AVIGILON EX. 2005
`IPR2019-00235
`Page 9 of 18
`
`

`

`
`
`Figure. 5: Tracking am 3336:: in lite-scam an fix: map
`
`wi§i he maxim Mm: a minimise is dapuaimd {m
`
`5.1 Architecium
`
`Tattle-“A- hez’wen 13:80:;3313 ami “mam on a watch“
`
`nigfit, The voice alarm: am {taste-mam {.0 tin: “mi
`
`and «3&3ij mm, 9:: that when {hm flaws is: My
`gemé, me 3'}: Sui-m wiii Manama “eiepmii hm." via
`
`its audio ism‘pat Figum 6 shew-s $4 perm: mm; m
`irigger‘ihis affirm.
`
`5 fleshfiew Seieciicm fair Activity Logging
`
`in many visim surveiiiamce amficafieng the gag? as?
`auweifiemce is an: m (imam mamas in tea? time and
`
`gamma? fianns, but mthsr to maximal :2 Eng“: au—
`
`ai'ét {mii 0f aii a? the ac-i'ivity- that {aka-s phase: in: the
`
`mama‘s fitfid {sf viw, This mg is cmmimzd its}: in.»
`
`vastiggmm wife: a sac-urn? imidem fag,- a 1338f: a:
`
`iamiris: ait3£k3,_ and is mad 1:}
`
`idmxify gmssih'ic:
`
`saggmcis in? wiimsmx.
`
`3 1r: mfim‘ m gait: ss-gmrience wiih {hi5 lygm cf agkpib
`
`333:3“,
`aw; ham ns'mi
`{hrs Sf; :king mid {went
`'{ififficiiun mfmbiiitime fiascrihcfi in smfém 3 to can»
`mm: a prngs‘am {hm menitms and recasflx ms:
`nwvmmus at' hummus iii-3'15 fiaid of vim“ Fer awn
`
`cry imam m: it sates;
`
`it. creams a tag fiixs him:
`
`summarizes; impamm infommmon ahnuz 21w perv-
`.smx, imimiéng a sngpsiwt tak‘m whmz me: gmmm
`
`mg siege to the gamma and {if massibks} facing it,
`
`The aggxiicaiirm makes mi: :3? {ha AVE mm figm—
`mhm is arse-m $11316 track gawk, Ugo“ dflfiflion
`£31? a mic}: eemsgmmiing it} a 333mm it: the: Mimi,-
`tha {me—Ra: assimiams a. dam mind with the mack,
`
`“fin: {hm recurd'mmains a snmnmr}! of infamaiim‘i
`aims! (in: mm)“, zineiuding a magmas: mas-acted.
`fmm {he cumin: 95:32:) image. ska {he gamma is
`lmckfid fibrough the seem). {hi3 mic-Raf examines
`each 3:ng of that WW this! :1 rem-isms. if that
`113W inmge 1531.33 be—im flaw af thé pram»: mam zhe
`previsaus‘ég 9mm 3n. what. the 313393210: is mpiafied
`mm the rim flew: Wm: {as perm: Rams the
`scam, {ht} dale}. mtmfi is saved to a fikt.
`
`Each Rig :3me flit: tweak {he time- w‘hen the per;
`you cmmxi the! 5mm:- amfi 3. Es: {afwarzfimm pairs
`stawixag {Mir gmséiion in mm}: 'ssitlm fiaxxm. £133}?
`tag may flier aim mutaim; tin: 5:121:11:th ma: wax
`filmed in ihfi track maxim far the; my: what; they
`Mimi ms: ascents... Re. sam- (32" the way gtmpshaza are.
`maimaincd, {h