`
`
`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`——————————
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`——————————
`
`Axis Communications AB, Canon Inc., and Canon U.S.A., Inc.,
`
`Petitioner
`
`v.
`
`Avigilon Fortress Corporation,
`
`Patent Owner
`
`——————————
`
`Case: IPR2019-00236
`
`U.S. Patent No. 7,868,912
`Issue Date: January 11, 2011
`
`Title: Video Surveillance System Employing Video Primitives
`
`——————————
`
`
`
`DECLARATION OF GAD TALMON, B.Sc.EE.
`
`
`

`

`Inter Partes Review No. IPR2019-00236
`United States Patent No. 7,868,912
`
`
`I, Gad Talmon, B.Sc.EE., state and declare as follows:
`
`1.
`
`I have prepared this Declaration in connection with the Petition of
`
`Axis Communications AB, Canon Inc., and Canon U.S.A., Inc. for inter partes
`
`review of U.S. Patent No. 7,868 ,912, which I understand will be filed concurrently
`
`with this Declaration.
`
`2.
`
`I was one of the co-founders of Aspectus Ltd. in 2003. Aspectus Ltd.
`
`later changed its name to Agent Video Intelligence, Inc. (or “Agent VI”) in 2005.
`
`3.
`
`I served as Chief Executive Officer (CEO) at Aspectus Ltd. (and then
`
`Agent VI) from its founding until 2006, at which time I took the role of Vice
`
`President. I served as Vice President of Agent VI until I left in December 2007. As
`
`part of those roles, I attended trade shows and was responsible for the design,
`
`marketing, and development of the Video Intelligence System (“VI-System”).
`
`Consequently, I am intimately familiar with the details of the technical design,
`
`development, marketing, and sales of the VI-System.
`
`4.
`
`In the early 2000s, video analytics was a relatively new or emerging
`
`field. Aspectus/Agent VI was one of the first companies to commercially offer a
`
`video intelligence (Video Analytics) system (the VI-System) for the security
`
`industry.
`
`5.
`
`Attached as Exhibit A (Exhibit 1003 to the Petition) is a true and
`
`correct copy of U.S. Patent No. 8,004,563 titled “Method and System for
`
`
`
`2
`
`

`

`Inter Partes Review No. IPR2019-00236
`United States Patent No. 7,868,912
`
`
`Effectively Performing Event Detection Using Feature Streams of Image
`
`Sequences” to Talmon et al. (“Talmon”). Talmon issued on August 23, 2011 to
`
`Agent VI (previously Aspectus Ltd.).
`
`6.
`
`I am one of the co-inventors of Talmon and my name appears on the
`
`face of the patent. The VI-System was a commercial implementation of the
`
`innovative systems and methods described in Talmon.
`
`7.
`
`Attached as Exhibit B (Exhibit 1005 to the Petition) is a true and
`
`correct copy of “Aspectus Video Intelligence VI-SystemTM” Brochure
`
`(“Aspectus”), which includes a detailed description of the features of the VI-
`
`System. Aspectus bears a copyright date of 2003-2004 and was distributed and
`
`made publicly available to customers in the United States in 2004 (i.e., before
`
`2005).
`
`8.
`
`As disclosed in Aspectus, the VI-System was a distributed image
`
`processing system useful for large-scale video surveillance networks. The VI-
`
`Server processed data streams from all of the camera units in the field. The VI
`
`Sentry of the VI-System provided a friendly wizard on a computer that guided a
`
`user step-by-step how to create event criteria rules. As an example, a user could
`
`define an event based on a unique sequence, such as “vehicle stops for more than 1
`
`minute, and a person leaves the vehicle unattended.” A wide variety of other events
`
`could also be defined through the system.
`
`3
`
`
`
`

`

`Inter Partes Review No. IPR2019-00236
`United States Patent No. 7,868,912
`
`
`As described in Aspectus, the criteria for defining events were based
`
`9.
`
`on attributes extracted from the video (referred to as “feature extraction”). The
`
`feature or attribute extraction was performed by the VI-Agent algorithm embedded
`
`in the VideoOutpost (located at or near each camera). This is illustrated on page 2
`
`and further described on page 3 of Aspectus.
`
`10.
`
`I personally recall emailing Aspectus to prospective customers in the
`
`United States in 2004. Printed copies of Aspectus were also mailed to prospective
`
`customers in 2004, many of whom were within the United States. At the time, the
`
`typical procedure at Aspectus Ltd. was to attach Aspectus to emails or physical
`
`correspondence to market the VI-System.
`
`11. During 2004, we began to develop a newer version of the brochure
`
`describing the VI-System. Aspectus, the version attached as Exhibit B, was the
`
`print version. Sometime during 2004, the information in Aspectus was converted
`
`into our new red and white theme (the “red and white brochure”). The red and
`
`white brochure was made available on Aspectus Ltd.’s website at
`
`www.aspectusvi.com. I personally recall viewing and accessing the red and white
`
`brochure on the company website before 2005 (i.e., during the 2004 calendar year).
`
`12. Attached as Exhibits C and D are printouts from the
`
`WaybackMachine (www.archive.org) that show that the “VI-SystemTM” brochure
`
`was available for download from the website in at least two places. Based on my
`
`
`
`4
`
`

`

`Inter Partes Review No. IPR2019-00236
`United States Patent No. 7,868,912
`
`
`personal knowledge, this referred to red and white brochure. This confirms my
`
`personal recollection that the red and white brochure describing the VI-System was
`
`made available on the company website by at least December 2004, and that its
`
`predecessor brochure (Aspectus) was publicly used and distributed before that date.
`
`13. Aspectus was distributed to potential customers at the ASIS
`
`International 50th Annual Seminar (“ASIS 2004”). ASIS 2004 was an annual
`
`convention for professionals in the security industry that was held in Dallas, Texas
`
`in October 2004.
`
`14. Attached as Exhibit E is an article from Business Wire that bears a
`
`publication date of October 18, 2004. It confirms that ASIS 2004 had record
`
`attendance of more than 19,000 people as well as 842 exhibitors. This is consistent
`
`with my recollection of ASIS 2004, as it was attended by many companies and
`
`professionals in the security industry
`
`15. During ASIS 2004, I personally met with potential U.S. customers,
`
`including On-Net Surveillance Systems, Inc. (OSSI) and others. During these
`
`meetings, Aspectus was publicly used, distributed, and made available for review
`
`to those U.S. customers.
`
`16. Attached as Exhibit F is a copy of OnSSI’s website from 2004. As
`
`shown in Exhibit F, OnSSI was a U.S. company, based in New York. This is
`
`consistent with my recollection of OnSSI’s status as a U.S. company at the time.
`
`
`
`5
`
`

`

`Inter Partes Review No. IPR2019-00236
`United States Patent No. 7,868,912
`
`17.
`
`I personally worked on the VI-System (disclosed in Aspectus), the
`
`development of which was based directly on my co-invention disclosed in Talmon.
`
`I was personally involved in the preparation of Talmon, the brochures describing
`
`the VI-System, including Aspectus, and the design, development, and marketing of
`
`the VI-System. Based on my personal knowledge and experience, Aspectus and
`
`Talmon describe the same system, with the VI-System being the commercial
`
`embodiment of Talmon, and Aspectus being the marketing material for that
`
`commercial embodiment.
`
`I declare under penalty of perjury that the foregoing is true and correct.
`
`Executed on November 10, 2018 in Ramat-HaSharon, Israel.
`
`____________________________________
`
`Gad Talmon, B.Sc.EE.
`
`6
`
`

`

`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`EXHIBIT A
`EXHIBIT A
`
`
`Axis Exhibit 1023, Page 7 of 38
`
`

`

`111111111111111111111111111111111111111111111111111111111111111111111111111
`US008004563B2
`
`c12) United States Patent
`Talmon et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,004,563 B2
`Aug. 23, 2011
`
`(54) METHOD AND SYSTEM FOR EFFECTIVELY
`PERFORMING EVENT DETECTION USING
`FEATURE STREAMS OF IMAGE
`SEQUENCES
`
`6,188,381 B1
`6,266,369 B1 *
`6,349,114 B1
`
`2/2001 van derWal et al.
`7/2001 Wang eta!. ................... 375/240
`2/2002 Mory
`(Continued)
`
`(75)
`
`Inventors: Gad Talmon, Givat Slnnuel (IL); Zvi
`Ashani, Ganei Tikva (IL)
`
`(73) Assignee: Agent Vi, Rosh-Haayin (IL)
`
`DE
`EP
`EP
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 846 days.
`
`(21) Appl. No.:
`
`10/501,949
`
`(22) PCT Filed:
`
`Jul. 3, 2003
`
`(86) PCTNo.:
`
`PCT /IL03/00555
`
`§ 371 (c)(l),
`(2), ( 4) Date:
`
`Jul. 21,2004
`
`(87) PCT Pub. No.: W02004/006184
`
`PCT Pub. Date: Jan. 15, 2004
`
`(65)
`
`Prior Publication Data
`
`US 2005/0036659 Al
`
`Feb. 17, 2005
`
`(51)
`
`Int. Cl.
`H04N 7118
`(2006.01)
`H04N 7112
`(2006.01)
`(52) U.S. Cl. ................................... 348/155; 375/240.08
`(58) Field of Classification Search .................. 348/154,
`348/155; 375/240.08
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`6,069,655 A * 5/2000 Seeley eta!. .................. 348/154
`6,130,707 A
`10/2000 Koller et al.
`
`FOREIGN PATENT DOCUMENTS
`6/1990
`38 42 356 A1
`1 107 609 A1
`6/2001
`1 173 020 A2
`1/2002
`(Continued)
`
`OTHER PUBLICATIONS
`
`XP-010199874: Meyer, M., eta!., "A New System for Video-Based
`Detection of Moving Objects and its Integration into Digital Net(cid:173)
`works", IEEE, pp. 105-110, (1996).
`
`Primary Examiner- Jayanti K Patel
`Assistant Examiner- Richard Torrente
`(74) Attorney, Agent, or Firm- Spilman Thomas & Battle,
`PLLC
`
`ABSTRACT
`(57)
`Method and system for performing event detection and object
`tracking in image streams by installing in field, a set of image
`acquisition devices, where each device includes a local pro(cid:173)
`grammable processor for converting the acquired image
`stream that consist of one or more images, to a digital format,
`and a local encoder for generating features from the image
`stream. These features are parameters that are related to
`attributes of objects in the image stream. The encoder also
`transmits a feature stream, whenever the motion features
`exceed a corresponding threshold. Each image acquisition
`device is connected to a data network through a correspond(cid:173)
`ing data communication channel. An image processing server
`that determines the threshold and processes the feature stream
`is also connected to the data network. Whenever the server
`receives features from a local encoder through its correspond(cid:173)
`ing data communication channel and the data network, the
`server provides indications regarding events in the image
`streams by processing the feature stream and transmitting
`these indications to an operator.
`58 Claims, 5 Drawing Sheets
`
`ENCODER 1
`
`WAN
`
`101
`
`

`

`US 8,004,563 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`6,384,862 B1
`5/2002 Brusewitz et a!.
`6,996,275 B2 *
`2/2006 Edanami ....................... 382/218
`2002/0041626 A1
`4/2002 Yoshioka et al.
`2002/0054210 A1
`5/2002 Glier eta!.
`2002/0062482 A1 *
`5/2002 Bolle eta!. .................... 725/105
`2005/0146605 A1 * 7/2005 Lipton eta!. .................. 348/143
`
`wo
`wo
`wo
`wo
`wo
`
`FOREIGN PATENT DOCUMENTS
`00/01140 A1
`1/2000
`01163937 A2
`8/2001
`1112001
`01191353 A2
`01191353 A3
`1112001
`02/37429 A1
`5/2002
`
`* cited by examiner
`
`

`

`0'1 w = N
`"'Of:) = = "'~ u.
`
`rJl
`d
`
`0 -.
`....
`.....
`rFJ =- ('D
`
`('D
`
`(Jl
`
`N
`~(H
`N
`~
`~
`
`....
`0 ....
`
`00 .
`~
`
`~ = ~
`
`~
`~
`~
`
`104
`
`105\
`
`FIG.l
`
`---------.-----
`
`101
`
`-::::-r
`
`203
`DETECTION
`THRESHOLD~ 201
`STREAM
`FEATURE
`
`207
`
`•
`•
`•
`
`MCIP
`
`209,
`
`II
`
`/
`
`ENCODER 2
`
`WAN
`
`ENCODER 1
`
`

`

`U.S. Patent
`U.S. Patent
`
`Aug.23, 2011
`Aug. 23, 2011
`
`Sheet 2 of 5
`Sheet 2 of 5
`
`US8,004,563 B2
`US 8,004,563 B2
`
`awt
`
`cS
`
`seae
`
`N
`C)
`
`Fig.2
`·-u..
`
`Axis Exhibit 1003, Page 4 of 16
`
`Axis Exhibit 1023, Page 11 of 38
`
`aa
`
`DoS
`_f=
`
`

`

`U.S. Patent
`U.S. Patent
`
`Aug. 23, 2011
`Aug. 23, 2011
`
`Sheet 3 of 5
`Sheet 3 of 5
`
`US 8,004,563 B2
`
`<(
`('f)
`•
`0)
`
`·-LL
`
`US 8,004,563 B2
`
`Axis Exhibit 1003, Page 5 of 16
`
`Axis Exhibit 1023, Page 12 of 38
`
`

`

`U.S. Patent
`U.S. Patent
`
`Aug.23, 2011
`Aug. 23, 2011
`
`Sheet 4 of 5
`Sheet 4 of 5
`
`US 8,004,563 B2
`
`aJ
`C"')
`•
`0)
`
`·-LL
`
`US 8,004,563 B2
`
`Axis Exhibit 1003, Page 6 of 16
`
`Axis Exhibit 1023, Page 13 of 38
`
`

`

`U.S. Patent
`U.S. Patent
`
`Aug.23, 2011
`Aug. 23, 2011
`
`Sheet 5 of 5
`Sheet 5 of 5
`
`US 8,004,563 B2
`US 8,004,563 B2
`
`Fig.3C
`
`(.)
`CVJ .
`·-LL
`
`C)
`
`Axis Exhibit 1003, Page 7 of 16
`
`Axis Exhibit 1023, Page 14 of 38
`
`

`

`US 8,004,563 B2
`
`1
`METHOD AND SYSTEM FOR EFFECTIVELY
`PERFORMING EVENT DETECTION USING
`FEATURE STREAMS OF IMAGE
`SEQUENCES
`
`FIELD OF THE INVENTION
`
`The present invention relates to the field of video process(cid:173)
`ing. More particularly, the invention relates to a method and
`system for obtaining meaningful knowledge, in real time,
`from a plurality of concurrent compressed image sequences,
`by effective prong of a large number of concurrent incoming
`image sequences and/or features derived from the acquired
`images.
`
`BACKGROUND OF THE INVENTION
`
`Many efforts have been spent to improve the ability to
`extract meaning data out of images captured by video and still
`cameras. Such abilities are being used in several applications,
`such as consumer, industrial, medical, and business applica(cid:173)
`tions. Many cameras are deployed in the streets, airports,
`schools, banks, offices, residencies-as standard security
`measures. These cameras are used either for allowing an
`operator to remotely view security events in real time, or for
`recording and analyzing a security event at some later time.
`The introduction of new technologies is shifting the video
`surveillance industry into new directions that significantly
`enhance the functionality of such systems. Several processing
`algorithms are used both for real-time and offline applica(cid:173)
`tions. These algorithms are implemented on a range of plat(cid:173)
`forms from pure software to pure hardware, depending on the
`application. However, these platforms are usually designed to
`simultaneously process a relatively small number of incom(cid:173)
`ing image sequences, due to the substantial computational
`resources required for image processing. In addition, most of
`the common image processing systems are designed to pro(cid:173)
`cess only uncompressed image data, such as the system dis(cid:173)
`closed in U.S. Pat. No. 6,188,381. Modern networked video 40
`environments require efficient processing capability of a
`large number of compressed video steams, collect from a
`plurality of image sources.
`Increasing operational demands, as well as cost constraints
`created the need for automation of event detection. Such 45
`event detection solutions provide a higher detection level,
`save manpower, replace other types of sensors and lower false
`alarm rates.
`Although conventional solutions am available for auto(cid:173)
`matic intruder detection, license plate identification, facial so
`recognition, traffic violations detection and other image
`based applications, they usually support few simultaneous
`video sources, using expensive hardware platforms that
`require field installation, which implies high installation,
`maintenance and upgrade costs.
`Conventional surveillance systems employ digital video
`networking technology and automatic event detection. Digi-
`tal video networking is implemented by the development of
`Digital Video Compression technology and the availability of
`IP based networks. Compression standards, such as MPEG-4 60
`and similar formats allow transmitting high quality images
`with a relatively narrow bandwidth.
`A major limiting factor when using digital video network(cid:173)
`ing is bandwidth requirements. Because it is too expensive to
`transmit all the cameras all the time, networks are designed to 65
`concurrently transmit data, only from few cameras. The trans(cid:173)
`mission of data only from cameras that are capturing impor-
`
`2
`tant events at any given moment is crucial for establishing an
`efficient and cost effective digital video network.
`Automatic video-based event detection
`technology
`becomes effective for this purpose. This technology consists
`of a series of algorithms that are able to analyze the camera
`image in real time and provide notification of a special event,
`if it occurs. Currently available event-detection solutions use
`conventional image processing methods, which require
`heavy processing resources. Furthermore, they allocate a
`1 o fixed processing power (usually one processor) per each cam(cid:173)
`era input. Therefore, such systems either provide poor per(cid:173)
`formance due to resources limitation or are extremely expen(cid:173)
`sive.
`As a result, the needs of large-scale digital surveillance
`15 installations-namely, reliable detection, effective band(cid:173)
`width usage, flexible event definition, large-scale design and
`cost, carmot be met by any of the current automatic event
`detection solutions.
`Video Motion Detection (VMD) methods are disclosed,
`20 for example, in U.S. Pat. No. 6,349,114, WO 02/37429, in
`U.S. Patent Application Publication 2002,041,626, in U.S.
`Patent Application Publication No. 2002,054,210, in WO
`01/63937, in EP1107609, in EP1173020, in U.S. Pat. No.
`6,384,862, in U.S. Pat. No. 6,188,381, in U.S. Pat. No. 6,130,
`25 707, and in U.S. Pat. No. 6,069,655. However, all the methods
`described above have not yet provided satisfactory solutions
`to the problem of effectively obtaining meanings knowledge,
`in real time, from a plurality of concurrent image sequences.
`It is an object of the present invention to provide a method
`30 and system for obtaining meaningful knowledge, from a plu(cid:173)
`rality of concurrent image sequences, in real time.
`It is another object of the present invention to provide a
`method and system for obtaining meaningful knowledge,
`from a plurality of concurrent image sequences, which are
`35 cost effective.
`It is a further object of the present invention to provide a
`method and system for obtaining meaningful knowledge,
`from a plurality of concurrent image sequences, with reduced
`amount of bandwidth resources.
`It is still another object of the present invention to provide
`a method and system for obtaining meaningful knowledge,
`from a plurality of concurrent image sequences, which is
`reliable, and having high sensitivity in noisy environments.
`It is yet another object of the present invention to provide a
`method and system for obtaining meaningful knowledge,
`from a plurality of concurrent image sequences, with reduced
`installation and maintenance costs.
`Other objects and advantages of the invention will become
`apparent as the description proceeds.
`
`SUMMARY OF THE INVENTION
`
`While these specifications discuss primarily video cam(cid:173)
`eras, a person skilled in the art will recognize that the inven-
`55 tion extends to any appropriate image source, such as still
`cameras, computer generated images, prerecorded video
`data, and the like, and that image sources should be equiva(cid:173)
`lently considered. Similarly, the terms video and video
`stream, should be construed broadly to include video
`sequences, still pictures, computer generated graphics) or any
`other sequence of images provided or converted to an elec-
`tronic format that may be processed by a computer.
`The present invention is directed to a method for perform(cid:173)
`ing event detection and object tracking in image streams. A
`set of image acquisition devices is installed in field, such that
`each device comprises a local programmable processor for
`converting the acquired image stream, that consists of one or
`
`

`

`US 8,004,563 B2
`
`3
`more images, to a digital format, and a local encoder, for
`generating features from the mage stream. The features are
`parameters that are related to attributes of objects in the image
`stream. Each device transmits a feature stream, whenever the
`number and type of features exceed a corresponding thresh(cid:173)
`old. Each image acquisition device is connected to a data
`network through a corresponding data communication chan(cid:173)
`nel. An image processing server connected to the data net(cid:173)
`work determines the threshold and processes the feature
`stream. Whenever the server receives features from a local
`encoder through its corresponding data communication chan- 10
`nel and the data network, the server obtains indications
`regarding events in the image streams by processing the fea(cid:173)
`ture stream and transmitting the indications to an operator.
`The local encoder may be a composite encoder, which is a
`local encoder that further comprises circuitry for compress- 15
`ing the image stream. The composite encoder may operate in
`a first mode, during which it generates and transmits the
`features to the server, and in a second mode, during which it
`transmits to the server, in addition to the features, at least a
`portion of the image stream in a desired compression level, 20
`according to commands sent from the server. Preferably, each
`composite encoder is controlled by a command sent from the
`server, to operate in its first mode. As long as the server
`receives features from a composite encoder, that composite
`encoder is controlled by a command sent from the server, to 25
`operate in its second mode. The server obtains indications
`regarding events in the image streams by processing the fea(cid:173)
`ture stream, and transmitting the indications and/or their cor(cid:173)
`responding image streams to an operator.
`Whenever desired one or more compressed image streams 30
`containing events are decoded by the operator station, and the
`decoded image streams are transmitted to the display of an
`operator, for viewing. Compressed image streams obtained
`while their local encoder operates in its second mode may be
`recorded.
`Preferably, additional image processing resources, in the
`server, are dynamically allocated to data communication
`duels that receive image streams. Feature streams obtained
`while operating in the first mode may comprise only a portion
`of the image.
`A graphical polygon that encompasses an object of interest
`being within the fame of an image or anAOI (Area Oflnter(cid:173)
`est) in the image may be generated by the server and dis(cid:173)
`played to an operator for viewing. In addition, the server may
`generate and display a graphical trace indicating the history 45
`of movement of an object of interest, being within the frame
`of an image or an AOI in the image.
`The image stream may be selected from the group of
`images that comprises video streams, still images, computer
`generated images, and pre-recorded digital, analog video 50
`data, or video streams, compressed using MPEG format. The
`encoder may use different resolution and frame rate during
`operation in each mode.
`Preferably, the features may include motion features, color,
`portions of the image, edge data and frequency related infor- 55
`mation.
`The server may perform, using a feature stream, received
`from the local encoder of at least one image acquisition
`device, one or more of the following operations and/or any
`combination thereof:
`License Plate Recognition (LPR);
`Facial Recognition (FR);
`detection of traffic rules violations;
`behavior recognition;
`fire detection;
`traffic flow detection;
`smoke detection.
`
`35
`
`40
`
`4
`The present invention is also directed to a system for per(cid:173)
`forming event detection and object tracking in image streams,
`that comprises:
`a) a set of image acquisition devices, installed in field, each
`of which includes:
`a.l) a local programmable processor for converting the
`acquired image stream, to a digital format
`a.2) a local encoder, for generating, from the image
`stream,
`features, being parameters
`related
`to
`attributes of objects in the image stream, and for trans(cid:173)
`mitting a feature stream, whenever the motion fea(cid:173)
`tures exceed a corresponding threshold;
`b) a data network, to which each image acquisition device
`is connected through a corresponding data communica(cid:173)
`tion charmel; and
`c) an image processing server connected to the data net(cid:173)
`work, the server being capable of determining the
`threshold, of obtaining indications regarding events in
`the image streams by processing the feature stream, and
`of transmitting the indications to an operator.
`The system may further comprise an operator display, for
`receiving and displaying one or more image streams that
`contain events, as well as a network video recorder for record(cid:173)
`ing one or more image streams, obtained while their local
`encoder operates in its first mode.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The above and other characteristics and advantages of the
`invention will be better understood through the following
`illustrative and non-limitative detailed description of pre(cid:173)
`ferred embodiments thereof, with reference to the appended
`drawings, wherein:
`FIG. 1 schematically illustrates the structure of a surveil(cid:173)
`lance system that comprises a plurality of cameras connected
`to a data network, according to a preferred embodiment of the
`invention;
`FIG. 2 illustrates the use of AOI's (Area of Interest) for
`designating areas where event detection will be performed
`and for reducing the usage of system resources, according to
`a preferred embodiment of the invention; and
`FIGS. 3A to 3C illustrates the generation of an object of
`interest and its motion trace, according to a preferred embodi(cid:173)
`ment of the invention.
`
`DETAILED DESCRIPTION OF PREFERRED
`EMBODIMENTS
`
`A significant saving in system resources can be achieved
`by applying novel data reduction techniques, proposed by the
`present invention. In a situation where thousands of cameras
`are connected to a single server, only a small number of the
`cameras actually acquire important events that should be
`analyzed. A large-scale system can function properly only if
`it has the capability of identifYing the inputs that may contain
`useful information and perform further processing only on
`such inputs. Such a filtering mechanism requires minimal
`60 processing and bandwidth resources, so that it is possible to
`apply it concurrently on a large number of image streams. The
`present invention proposes such a filtering mechanism, called
`Massively Concurrent Image Processing (MCIP) technology
`that is based on the analysis of incoming image sequences
`65 and/or feature streams, derived from the acquired images, so
`as to fulfill the need for automatic image detection capabili(cid:173)
`ties in a large-scale digital video network environment.
`
`

`

`US 8,004,563 B2
`
`5
`MCIP technology combines diverse technologies such as
`large scale data reduction, effective server design and opti(cid:173)
`mized image processing algorithms, thereby offering a plat(cid:173)
`form that is mainly directed to the security market and is not
`rivaled by conventional solutions, particularly with vast num(cid:173)
`bers of potential users. MCIP is a networked solution for
`event detection in distributed installations, which is designed
`for large scale digital video surveillance networks that con(cid:173)
`currently support thousands of camera inputs, distributed in
`an arbitrarily large geographical area and with real time per(cid:173)
`formance. MCIP employs a unique feature transmission
`method that consumes narrow bandwidth, while maintaining
`high sensitivity and probability of detection. MCIP is a
`server-based solution that is compatible with modern moni(cid:173)
`toring and digital video recording systems and carries out
`complex detection algorithms, reduces field maintenance and
`provides improved scalability, high availability, low cost per
`channel and backup utilities. The same system provides con(cid:173)
`currently multiple applications such as VMD, LPR and FR. In
`addition, different detection applications may be associated
`with the same camera.
`MCIP is composed of a server platform with various appli(cid:173)
`cations, camera encoders (either internal or eternal to the
`camera), a Network Video Recorder (NVR) and an operator
`station. The server contains a computer that includes propri(cid:173)
`etary hardware and software components. MCIP is based on
`the distribution of image processing algorithms between low(cid:173)
`level feature extraction, which is performed by the encoders
`which are located in field (i.e., in the vicinity of a camera), and
`high-level processing applications, which are performed by a
`remote central server that collects and analyzes these fea(cid:173)
`tures.
`The MCIP system described hereafter solves not only the
`bandwidth problem but also reduces the load from the server
`and uses a unique type of data stream (not a digital video
`stream), and performs an effective process for detecting
`events at real time, in a large scale video surveillance envi(cid:173)
`ronment.
`A major element in MOP is data reduction, which is
`achieved by the distribution of the image processing algo(cid:173)
`rithms. Since all the video sources, which require event detec(cid:173)
`tion, transmit concurrently, the required network bandwidth
`is reduced by generating a reduced bandwidth feature stream
`201 in the vicinity of each camera. In order to detect, track,
`classify and analyze the behavior of objects present in video
`sources, there is no need to transmit full video streams, but
`only partial data, which contains information regarding
`describing basic attributes of each video scene.
`By doing so, a significantly smaller data bandwidth is used,
`which reduces the demands for both the network bandwidth
`and the event detection processing power. Furthermore, if
`only the shape, size, direction of movement and velocity
`should be detected, there is no need to transmit data regarding
`their intensity or color, and thus, a further bandwidth reduc(cid:173)
`tion is achieved. Another bandwidth optimization may be
`achieved if the encoder in the transmitting side filters out all
`motions which are under a motion threshold, determined by
`the remote central server 203. Such threshold may be the
`amount of motion in pixels between two consecutive frames,
`and may be determined and changed dynamically, according
`to the attributes of the acquired image, such as resolution,
`AOI, compression level, etc. Areas of movement which are
`under the threshold are considered either as noise, or non(cid:173)
`interesting motions.
`One method for extracting features at the encoder side is by
`slightly modifYing and degrading existing temporal-based
`video compressors which were originally designed to trans-
`
`6
`mit digital video. The features may also be generated by a
`specific feature extraction algorithm (such as any motion
`vector generating algorithm) that is not related to the video
`compression algorithm. When working in this reduced band(cid:173)
`width mode, the output streams of these encoders are defi(cid:173)
`nitely not a video stream, and therefore cannot not be used by
`any receiving party to produce video images.
`FIG. 1 schematically illustrates the structure of a surveil(cid:173)
`lance system that comprises a plurality of cameras connected
`10 to a data network, according to a preferred embodiment of the
`invention. The system 100 comprises n image sources (in this
`example, n cameras, CAM1, ... , CAMn), each of which
`connected to a digital encoder ENC j, for converting the
`images acquired by CAM} to a compressed digital format.
`15 Each digital encoder ENC j is connected to a digital data
`network 101 at point Pj and being capable of transmitting
`data, which may be a reduced bandwidth feature stream 201
`or a full compressed video stream, through its corresponding
`channel Cj. The data network 101 collects the data transmit-
`20 ted from all channels and forwards them to the MCIP server
`102, through data-bus 103. MCIP server 102 processes the
`data received from each charmel and controls one or more
`cameras which transmit any combination of the reduced
`bandwidth feature stream and the full compressed video
`25 stream, which can be analyzed by MCIP server 102 in real
`time, or recorded by NVR 104 and analyzed by MCIP server
`102 later. An operator station 105 is also connected to MCIP
`server 102, for real time monitoring of selected full com(cid:173)
`pressed video streams 205. Operator station 105 can manu-
`30 ally control the operation of MCIP server 102, whenever
`desired.
`The MCIP (Massively Concurrent Image Processing)
`server is connected to the image sources (depicted as cameras
`in the drawing, but may also be any image source, such taped
`35 video, still cameras, video cameras, computer generated
`images or graphics, and the like.) through data-bus 103 and
`ne