Inter Partes Review
`United States Patent No. 7,932,923
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`Canon Inc., Canon U.S.A., Inc., and Axis Communications AB,
`
`Petitioner
`
`V.
`
`Avigilon Fortress Corporation,
`
`Patent Owner
`
`
`
`Case: Unassigned
`
`U.S. Patent No. 7,932,923
`Issue Date: April 26, 2011
`
`Title: Video Surveillance System Employing Video Primitives
`
`DECLARATION OF EMILY R. FLORIO
`
`Canon Corrected Ex. 1007 Page 1 of 219
`
`Canon Corrected Ex. 1007 Page 1 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`I, Emily R.Florio, state and declare as follows:
`
`1.
`
`I have prepared this Declaration in connection with the Petitions of
`
`Axis Communications AB, CanonInc., and Canon U.S.A., Inc. (collectively
`
`“Petitioner”) for two inter partes reviews of U.S. Patent No. 7,932,923 (“the ’923
`
`patent”), which I understand will be filed concurrently with this Declaration.I also
`
`understand that October 1999is a date that is relevant for determining whatis prior
`
`art to the ’923 patent.
`
`2.
`
`I am currently the Director of Research & Information Servicesat
`
`Finnegan, Henderson, Farabow, Garrett & Dunner LLP, 901 New York Avenue
`
`NW, Washington, DC 20001-4413.
`
`3.
`
`I am over 18 years of age and am competent to makethis Declaration.
`
`I make this Declaration based on my ownpersonal knowledge, based on my
`
`knowledge oflibrary science practices, as well as my knowledge ofthe practices at
`
`the Massachusetts Institute of Technology (“MIT”) Libraries.
`
`4,
`
`I earned a Master’s of Library Science (“MLS”) from Simmons
`
`College in 2006, and I have workedasa librarian for over a decade. I have been
`
`employed in the Research & Information Services (formerly Library) Department
`
`of Finnegan since 2013, and from 2005-2013, I worked in the Library Department
`
`of Fish & Richardson P.C.
`
`Canon Corrected Ex. 1007 Page 2 of 219
`
`Canon Corrected Ex. 1007 Page 2 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`5.
`
`I am currently the Vice-President Elect of the American Association
`
`of Law Libraries and the President of the Law Librarians’ Society of Washington,
`
`DC, and a memberofthe International Legal Technology Association.
`
`Attachments
`
`6.
`
`Attached as Exhibit A (Exhibit 1003 to the Petition) is a true and
`
`correct copy of “Visual Memory,” May 1993, pp. 1-92, by Christopher James
`
`Kellogg (“Kellogg”), obtained from the MITLibraries.
`
`7.
`
`Attached as Exhibit B is a true and correct copy ofthe “Standard”
`
`record from the MIT Libraries’ catalog system (knownas the Barton Catalog) for
`
`its copy of Kellogg.
`
`8.
`
`Attached as Exhibit C is a true and correct copy of the MARCrecord
`
`of the MIT Libraries for its copy of Kellogg.
`
`9.
`
`Attached as Exhibit D (Exhibit 1004 to the Petition)is a true and
`
`correct copy of F. Brill et al., “Event Recognition and Reliability Improvements
`
`for the AutonomousVideo Surveillance System,” Proceedings of the Image
`
`Understanding Workshop, Monterey, CA, Nov. 20-23, 1998, Vol. 1, pp. 267-283
`
`(“Brill”), obtained from the Duderstadt Center, formerly known as the University
`
`of Michigan Media Union (UMMU).
`
`10. Attached as Exhibit E is a true and correct copy of the MARCrecord
`
`of the University of Virginia Library for its copy of Brill.
`
`3
`
`Canon Corrected Ex. 1007 Page 3 of 219
`
`Canon Corrected Ex. 1007 Page 3 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`11. Attached as Exhibit F is a true and correct copy of the MARCrecord
`ofthe North Carolina State University library for its copy ofBrill.
`
`12. Attached as Exhibit G (Exhibit 1006 to the Petition) is a true and
`
`correct copy of N. Dimitrovaet al., “Motion Recovery for Video Content
`
`Classification,” ACM Transactions on Information Systems, October 1995, Vol.
`
`13, No. 4, pp. 408-439 (“Dimitrova’’), obtained from University of California Los
`
`Angeles Science & Engineering Library.
`
`13. Attached as Exhibit H is a true and correct copy of Dimitrova,
`
`obtained from the Library of Congress.
`
`14. Attached as Exhibit I is a true and correct copy of the MARCrecord
`
`of the MIT Libraries for its copy of the ACM Transactions on Information Systems
`
`journal, in which Dimitrova waspublished.
`
`15. Attached as Exhibit J is a true and accurate copy of B. Flinchbaugh et
`
`al., “Autonomous Video Surveillance,” SPIE Proceedings, 25" AIPR Workshop:
`
`Emerging Applications of Computer Vision, Feb. 26, 1997, Vol. 2962,p. 144-151
`
`(“Flinchbaugh’), obtained from the MITLibraries.
`
`16. Attached as Exhibit K is a true and correct copy of the MARC record
`
`of the Library of Congress for its copy of the SPIE Proceedings publication that
`
`includes Flinchbaugh.
`
`Canon Corrected Ex. 1007 Page 4 of 219
`
`Canon Corrected Ex. 1007 Page 4 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`17. Attached as Exhibit L is a true and correct copy of the MARCrecord
`
`of the MIT Libraries for its copy of the SPIE Proceedings publication that includes
`
`Flinchbaugh.
`
`The MARC Cataloging System
`
`18.
`
`The MAchine-Readable Cataloging (“MARC”) system is used by
`
`libraries to catalog materials. The MARC system was developedin the 1960s to
`
`standardize bibliographic records so they could be read by computers and shared
`
`amonglibraries. By the mid-1970’s, MARChad becometheinternational standard
`
`for bibliographic data, anditis still used today.
`
`19.
`
`Each field in a MARCrecord provides information about the
`
`cataloged item. MARCusesa simple three-digit numeric code (from 001-999)to
`
`identify each field in the record.
`
`20.
`
`For example, field 245 lists the title of the work andfield 260lists
`
`publisher information. In addition, field 008 provides the date the item was
`
`cataloged. Thefirst six characters ofthe field 008 are always in the “YYMMDD”
`
`format.
`
`21.
`
`It is standardlibrary practice that once an item is cataloged using the
`
`MARCsystem,it is shelved. This process maytakea relatively nominal amount
`
`oftime (i.e., a few days or weeks). During the time betweenthe cataloging and
`
`Canon Corrected Ex. 1007 Page 5 of 219
`
`Canon Corrected Ex. 1007 Page 5 of 219
`
`

`

`shelving of an item, the public maystill find the item by searching the catalog and
`
`Inter Partes Review
`United States Patent No. 7,932,923
`
`requesting the item from the library.
`
`Kellogg
`
`22. As indicated in Exhibit A (Exhibit 1003 to the Petition), Kellogg has
`
`an MIT Libraries date stamp of “JUL 09 1993”on page1, indicating that the MIT
`
`Libraries received Kellogg on July 9, 1993. Further, as indicated in Exhibit B, the
`
`Standard record of the Barton Catalog confirms that Kellogg is shelved at the MIT
`
`Libraries and was published in 1993. In view of the above and the following,
`
`Kellogg was published and accessible to the public in 1993, years before October
`
`1999.
`
`23. As indicated in Exhibit C, Kellogg has a cataloging date of September
`
`28, 1993 (shown as “930928” in field 008). This confirms that Kellogg was
`
`entered into the OCLC database, in which MIT doesits cataloging, on September
`
`28, 1993. This is also consistent with its noted year of publication in the MARC
`
`record (shown as “1993”in field 260). The OCLC database(alsoreferred to as
`
`“WorldCat”) is the largest online public access catalog (OPAC)in the world.
`
`24.
`
`Soon after Kellogg received a cataloging date, a record ofits existence
`
`would have appeared in and been keyword-searchable through the Barton Catalog
`
`of the MIT Libraries. The Barton Catalog is currently available online to any user
`
`of the World Wide Web. Before it was accessible by Web(i.e., at the time the
`
`6
`
`Canon Corrected Ex. 1007 Page 6 of 219
`
`Canon Corrected Ex. 1007 Page 6 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`Kellogg thesis was received by the MIT Libraries in July 1993), it would have been
`
`accessible to anyone on the MIT campus and anyone whohadaccessto the OCLC
`
`database.
`
`25. During the time period from September 1993 through October 1999,
`
`the Barton Catalog allowed keyword searching for wordsin thethesistitle, and
`
`Kellogg would have appearedin a relevant Barton Catalog search conducted on or
`
`shortly after September 28, 1993.
`
`26. After being cataloged, a document such as Kellogg will undergo a
`
`process of being labeled and then shelved at the MIT Libraries. Based on my
`
`knowledge of MIT Libraries’ current and prior practices, Kellogg would have been
`
`shelvedin a relatively nominal amountof time(i.e., a few days or weeks). Thus,
`
`Kellogg wascataloged and shelvedat the MIT Libraries at least before the end of
`
`1993,
`
`27.
`
`Once shelved, Kellogg can be borrowed by any member of the MIT
`
`community. Furthermore, a copy of Kellogg can be purchased from MITby any
`
`memberof the public. Indeed,thefirst page of Kellogg confirmsthat there were
`
`no restrictions placed on its publication,as it states that “[t]he author hereby grants
`
`to MIT permission to reproduce andto distribute copies ofthis thesis documentin
`
`wholeorin part, and to grant others the right to do so.”
`
`Canon Corrected Ex. 1007 Page 7 of 219
`
`Canon Corrected Ex. 1007 Page 7 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`28.
`
`Further evidence ofthe public availability of Kellogg before October
`
`1999 is provided in Exhibit J, which is a copy of Flinchbaugh. In its Bibliography,
`
`Flinchbaugh cites to Kellogg (reference [4] on p. 151). As addressed below,
`
`Flinchbaugh was published in SPIE Volume 2962, which correspondsto the
`
`Proceedings from the 25“ Annual AIPR Workshop on Emerging Applications of
`
`Computer Vision. The Workshop was held October 16-18, 1996, and the
`
`Proceedings were published by at least 1997. Thus, Kellogg wasat least available
`
`to membersofthe public in 1997, as shownbyits citation in Flinchbaugh.
`
`29.
`
`For the avoidance of any doubt, I note that on June 23, 2001, Kellogg
`
`wasalso cataloged in the MIT Archive Noncirculating Collection 1,
`
`Noncirculating Collection 3, and in microfiche form in the Barker Library, as
`
`indicated in the three entries for PST8 and in the second, third, and fourth instances
`
`of field 008 on page 1 of Exhibit C. However, noneofthis alters the fact that
`
`Kellogg was published and accessible to the public in 1993, as indicated above.
`
`Brill
`
`30. As indicated in Exhibit D, Brill is part of the published Proceedings of
`
`the 1998 Image Understanding Workshop. The Workshop washeld in Monterey,
`
`California during November 20-23, 1998, and the Proceedings were “APPROVED
`
`FOR PUBLIC RELEASE”with “DISTRIBUTION UNLIMITED.” Ex. D at 1. In
`
`Canon Corrected Ex. 1007 Page 8 of 219
`
`Canon Corrected Ex. 1007 Page 8 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`view of the above and the following, the Proceedings, including Brill, was
`
`published and accessible to the public before October 1999.
`
`31.
`
`Evidence of Brill’s publication and availability to the public includes
`
`the hand-written receipt date of “8-13-99”at the top of page 3 of Exhibit D. This
`
`indicates it was received by the UMMU (the University of Michigan Media Union,
`
`now knownas the Duderstadt Center) on August 13, 1999. In my experience as a
`
`librarian and knowledgeof standard library practices, the hand-written information
`
`at the top of p. 2 of Exhibit D appearsto be the catalog record information for
`
`Brill. Based on standardlibrary practices, this reference would have been shelved
`
`shortly after being received and cataloged by UMMU.
`
`32.
`
`Further evidence of the publication and accessibility of Brill to the
`
`public can be found in Exhibit E, which is the MARCrecord for the Proceedings,
`
`including Brill, that was obtained from the University of Virginia Library. As
`
`shownin field 008 near the top of page 2 of Exhibit E, Brill was cataloged by the
`
`library on December 15, 1998. Based on standard library practices, this reference
`
`would have been shelved shortly after(i.e., within a few days or weeks) and been
`
`accessible to the public prior to October 1999,
`
`33.
`
`Further evidence of the publication and public availability of Brill can
`
`be found in Exhibit F, which is the MARCrecord for the Proceedings, including
`
`Brill, that was obtained from North Carolina State University. As showninfield
`
`9
`
`Canon Corrected Ex. 1007 Page 9 of 219
`
`Canon Corrected Ex. 1007 Page 9 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`008 on page 1 of Exhibit F, Bril] was cataloged by the library on December15,
`
`1998. Based on standardlibrary practices, this reference would have been shelved
`
`shortly after (i.e:, within a few days or weeks) and been accessibleto the public
`
`prior to October 1999.
`
`Dimitrova
`
`34. As indicated in Exhibit G, Dimitrova waspublished in a special issue
`
`of the ACM Transactions on Information Systems Journal. Ex. G at 1. In view of
`
`the above andthe following, the ACM Journal, including Dimitrova, was
`
`published and accessible to the public before October 1999.
`
`35.
`
`Evidence of Dimitrova’s publication and availability to the public
`
`includes the “November 17, 1995” date stamp on page | of Exhibit G. This
`
`confirms that Dimitrova wasreceived by the University of California Los Angeles
`
`Science & Engineering Library on November 17, 1995. Based on standardlibrary
`
`practices, this reference would have been shelved shortly after (i.e., within a few
`
`days or weeks) and accessible to the public before October 1999.
`
`36.
`
`Further evidenceof the publication and accessibility of Dimitrova to
`
`the public is found in Exhibit H, whichis a copy of Dimitrova obtained from the
`
`Library of Congress. Page 3 of Exhibit H bears the date stamp of “November 21,
`
`1995” from the Library of Congress. This confirms that Dimitrova was received
`
`by Library of Congress on November 21, 1995. Based on standard library
`
`10
`
`Canon Corrected Ex. 1007 Page 10 of 219
`
`Canon Corrected Ex. 1007 Page 10 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`practices, this reference would have been shelvedshortly after (i.e., within a few
`
`days or weeks) and accessible to the public before October 1999.
`
`37.
`
`Further evidence of the publication and accessibility of Dimitrova to
`
`the public can be found in Exhibit I, which is the MARCrecord of the ACM
`
`Journal, which includes Dimitrova, obtained from the MIT Libraries. As shown in
`
`field 008 near the top of page 1 of Exhibit I, the MIT Libraries began receiving the
`
`ACM Journal in September 1989.
`
`38.
`
`Field 362 (shown as “3620”) on page 1 of Exhibit I indicates that the
`
`MITLibraries has issues dating back to Volume7 ofthe journal, which as noted
`
`above was published in September 1989. There is no end-datein field 362,
`
`indicating that the MIT Libraries has an ongoing subscription, which would have
`
`includedreceipt of the issue that contained Dimitrova in 1995. Based on standard
`
`library practices, the issue containing Dimitrova would have been shelved shortly
`
`after cataloging(i.e., within a few days or weeks) and accessibleto the public
`
`before October 1999.
`
`39.
`
`For the avoidance of any doubt, I note that it appears that on June 23,
`
`2001, online access to Dimitrova was provided to certain MIT-associated
`
`individuals, as indicated by the field 008 onthelast line of page 1, field 8528, and
`
`the URL entry at the top of page 2 of Exhibit I. Also on June 23, 2001, the ACM
`
`Journal, including Dimitrova, was archived at the MIT Library Storage Annex
`
`1]
`
`Canon Corrected Ex. 1007 Page 11 of 219
`
`Canon Corrected Ex. 1007 Page 11 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`(“LSA”), as indicated by the 008 field and subsequent 85271 entry on page 2 of
`
`Exhibit I. However, noneofthis alters the fact that Dimitrova was published and
`
`accessible to the public years before October 1999, as indicated above.
`
`Flinchbaugh
`
`40.
`
`Asindicated in Exhibit J, Flinchbaugh waspublishedin the
`
`Proceedings of the 25 AIPR Workshop: Emerging Applications of Computer
`
`Vision, SPIE Vol. 2962. The Workshop washeld in Washington, D.C. during
`
`October 16-18, 1996, and the Proceedings was published by SPIE (The
`
`International Society for Optical Engineering). Ex. J at 1. In view of the above
`
`and the following, Flinchbaugh waspublished and accessible to the public before
`
`October 1999.
`
`Al.
`
`Page 2 of Exhibit J shows a copyright date of 1997. The edition of the
`
`SPIE Proceedings that was published with Flinchbaugh is Volume 2962,andit
`
`was “Printed in the United States of America.” Ex. J at 2.
`
`42. Although the copyright date of Flinchbaughis listed as 1997,it
`
`appears that Flinchbaugh wasactually published before that, in 1996. First, as
`
`noted above, the Workshop washeld in Washington, D.C. during October 16-18,
`
`1996. Second, a copy of Flinchbaugh wasreceived and cataloged by the Library
`
`of Congress in November 1996. See Ex. K at 1. Exhibit K is the MARC record
`
`for the SPIE Proceedings, including Flinchbaugh,that was obtained from the
`
`12
`
`Canon Corrected Ex. 1007 Page 12 of 219
`
`Canon Corrected Ex. 1007 Page 12 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`Library of Congress. As shown in field 008 nearthe top of page 2 of Exhibit K,
`
`Flinchbaugh wascataloged bythe library on November 21, 1996. Based on
`
`standard library practices, this reference would have been shelved shortly afterit
`
`wascataloged(i.e., within a few days or weeks). Collectively, Exhibits J and K
`
`show that Flinchbaugh was published and accessible to the public years before
`
`October 1999.
`
`43.
`
`Further evidenceof the publication and public availability of
`
`Flinchbaugh can be found in Exhibit L, which is the MARCrecordfor the SPIE
`
`Proceedings, including Flinchbaugh, that was obtained from the MITLibraries.
`
`As shownin field 008 on page1 of Exhibit L, Flinchbaugh wascatalogedby the
`
`library on March 10, 1997. Based onstandard library practices and my
`
`understanding of the practices of the MIT Libraries, this reference would have
`
`been shelved shortly after it was cataloged(i.e., within a few days or weeks) and
`
`accessible to the public before October 1999.
`
`44.
`
`For the avoidanceof any doubt, I note that on April 8, 2011, online
`
`access to Flinchbaugh wasprovidedto certain MIT-associated individuals, as
`
`indicated bythe fields 008 and 8528 and the URL entry at the top of page 2 of
`
`Exhibit L. Also, on June 23, 2001, the SPIE Proceedings, including F,linchbaugh,
`
`wasarchived at the MIT Library Storage Annex (“LSA”), as indicated by the
`
`second 008field and subsequent 8520 entry on page 2 of Exhibit L. However,
`
`13
`
`Canon Corrected Ex. 1007 Page 13 of 219
`
`Canon Corrected Ex. 1007 Page 13 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`noneofthis alters the fact that Flinchbaugh was published and accessible to the
`
`public years before October 1999, as indicated above.
`
`I have been warned and understandthat willful false statements andthelike
`
`are punishable by fine or imprisonment, or both (18 U.S.C. § 1001). I declare that
`
`all statements made in this declaration of my own knowledgeare true andall
`
`statements made on information and belief are believed to be true.
`
`I declare under penalty of perjury that the foregoing is true and correct.
`
`t
`
`Executed on July 9, 2019 in Washington, D.C.
`
`Gk [rt
`
`Emily R. Florio
`
`14
`
`Canon Corrected Ex. 1007 Page 14 of 219
`
`Canon Corrected Ex. 1007 Page 14 of 219
`
`

`

`Visual Memory
`
`by
`Christopher James Kellogg
`
`Submitted to the Department of Electrical Engineering and Computer Science
`in partial fulfillment of the requirements for the degrees of
`
`Bachelor of Science
`and
`Masterof Science in Computer Science
`
`at the
`
`MASSACHUSETTS INSTITUTE OF TECHNOLOGY
`
`May 1993
`
`© Christopher James Kellogg, MCMXCIII. All rights reserved.
`
`The author hereby grants to MIT permission to reproduce and to distribute copies
`of this thesis document in whole or in part, and to grant others the right to do so.
`
`Signature redacted
`Author... ccc ccc cece eee e tenet eee en eens eeene rape oFF eR pee eRe
`.
`Department of Electrical Engineering
`omp
`cience
`247 1993
`
`ril
`
`Signature redacted
`
`Certified by 00... cece ccc eter ee ne eTgATEA cee e em renee ster eee e eee ces
`Alex P. Pentland
`Associate Professor of Media Arts and Sciences
`_ Thesis Supervisor
`
`Signature redacted
`peeewerenceres
`ManagerofImagenein hayTendstaptroments
`“Bruce E. Flinchbaugh
`Vd
`Supervisor
`SIIgnature redacted
`
`YPVCP$eie See
`
`son,
`artmental Committee on Graduate Students
`MASSACHUSETTS INSTITUTE
`QF TFCHNOLOGY
`
`Certified by 0... .... cece cece cece ence een e ete ee
`
`nee eee sem
`
`Accepted by......... 0.07
`
`Chair
`
`‘JUL 09 1993
`usrades. A Page | of 92
`ARCHIVES
`Canon Corrected Ex. 1007 Page 15 of 219
`Canon Corrected Ex. 1007 Page 15 of 219
`
`Ex. A Page 1 of 92
`
`

`

`Visual Memory
`
`by
`
`Christopher James Kellogg
`
`Submitted to the Department of Electrical Engineering and Computer Science
`on April 21, 1993, in partial fulfillment of the
`requirements for the degrees of
`Bachelor of Science
`and
`Master of Science in Computer Science
`
`Abstract
`
`Visual memory supports computer vision applications by efficiently storing and re-
`trieving spatiotemporal information. It is a unique combination of databases, spatial
`representation and indexing, and temporal representation and indexing. This the-
`sis designs a visual memory architecture that meets the requirements of a number
`of computer vision applications.
`It also presents an implementation of part of this
`design in support of a scene monitoring prototype.
`
`Thesis Supervisor: Alex P. Pentland
`Title: Associate Professor of Media Arts and Sciences
`
`Thesis Supervisor: Bruce E. Flinchbaugh
`Title: Manager of Image Understanding Branch at Texas Instruments
`
`Ex. A Page 2 of 92
`
`Canon Corrected Ex. 1007 Page 16 of 219
`Canon Corrected Ex. 1007 Page 16 of 219
`
`Ex. A Page 2 of 92
`
`

`

`Acknowledgements
`
`Myprimary thanks goes to my two thesis supervisors, Bruce Flinchbaugh at Texas
`
`Instruments and Sandy Pentland at MIT. Bruce pointed me to the visual memory
`
`project that he was starting and guided myresearch at Texas Instruments. Sandy
`
`provided useful feedback throughout the research stage. They were both very helpful
`
`in critiquing the thesis document.
`
`I'd also like to thank the other people at Texas Instruments who helped me with
`
`this project. Steve Ford and Tom Bannon wereespecially helpful in developing the
`
`visual memory design. In addition, I don’t think I would have survived the bugs in
`
`PC++ without Steve’s expertise. Tom Bannon and Tom O’Donnell provided a nice
`
`tracking system with which to test the visual memory prototype.
`
`Finally, I’d like to thank my family, Fred, Jeannette, and Mark Kellogg, my fiancée
`
`Christine Bailey, and my brothers at Phi Kappa Sigma for their support throughout
`
`my MIT career.
`
`Ex. A Page 3 of 92
`
`Canon Corrected Ex. 1007 Page 17 of 219
`Canon Corrected Ex. 1007 Page 17 of 219
`
`Ex. A Page 3 of 92
`
`

`

`Contents
`
`1
`
`Introduction
`
`1.1 Needs for Visual Memory... ......... 2.02. ee eee eae
`
`)
`
`9
`
`1.2 Goals... ee ee 10
`
`2 Background
`
`11
`
`2.1 Database Research .. 1... 2. ee ee en 11
`
`2.1.1 DARPA Open OODB ..............0..2006- 11
`
`2.12 POSTGRES... 2... 2... ee ee ee 12
`
`2.2 Spatial Research
`
`. 2... . eee ee es 13
`
`2.2.1 CODGER ........ ee ee eee 13
`
`2.2.2 Core Knowledge System .. 2... 0.0.2... eee eee eee
`
`2.2.3 ISR... ee ee es
`
`2.2.4
`
`Image Understanding Environments..............-.
`
`2.2.5
`
`PROBE ... 2... 2. ee ee ee ee ee es
`
`2.2.6
`
`Spatial Indices ©... 2... 2.0... 2. ee ee ee et ees
`
`13
`
`14
`
`14
`
`14
`
`15
`
`2.3 Temporal Research ... 1... 0... eee ee ee es 15
`
`2.3.1 TQuel ..... ee ee ee 15
`
`2.3.2 Temporal Sequences .... 2.2... 0.020 2 eee eens 16
`
`2.3.3 Temporal Sets... 0... 0. ee ee te ee ee ee 16
`
`2.3.4 Relative Time... 2.2... ee ee ee ee 17
`
`2.3.5 Temporal Indices .. 2... 2... 2. ee ee et ee ee 17
`
`Ex. A Page 4 of 92
`
`Canon Corrected Ex. 1007 Page 18 of 219
`1007 Page 18 of 219
`Canon Corrected Ex.
`
`Ex. A Page 4 of 92
`
`

`

`3 Design
`
`18
`
`3.1 Requirements and Considerations ............00 02 eae 19
`
`3.1.1 Database Considerations ............2.00050 008 19
`
`3.1.2
`
`Spatial and Temporal Considerations ..............
`
`20
`
`3.1.3
`
`Performance Considerations ..............2000- 20
`
`3.2 Design Overview .. 1... ee ee 22
`
`3.3 Spatial Representations... 1... 0... ee ee ee ee ee ns
`
`3.3.1 Core Spatial Classes 2... 2... ee ee es
`
`3.3.2 Relative Spatial Specification ............20 000+
`
`3.3.3 Uncertain Spatial Specification ............2.000-
`
`24
`
`24
`
`29
`
`31
`
`3.4 Temporal Representations ........... 0.0. ee eee eee 36
`
`3.4.1 Core Temporal Classes .. 1... ......- 002 eee eee 36
`
`3.4.2 Relative Temporal Specification ..............0...
`
`3.4.3 Uncertain Temporal Specification ............+024-
`
`3.5 Spatiotemporal Representations ...........050 0002 e eee
`
`40
`
`41
`
`45
`
`3.6 Object Storage 2... ee ne 50
`
`3.6.1
`
`Identity .. 2... ... ee ee ee en 50
`
`3.6.2
`
`Storage Mechanism ..........-...2.0-0 20000 ee
`
`3.6.3 Time... .. ee ee ee ee ns
`
`51
`
`52
`
`3.7 Queries. 2... et ee es 53
`
`3.7.1 Query Mechanism. ..........2. 00000 ee ee eeee
`
`53
`
`3.7.2
`
`Spatial Queries .. 1... ee 54
`
`3.7.3 Temporal Queries... 1... . ee ee ee eee ee ee 57
`
`3.7.4
`
`Spatiotemporal Queries... 2... ee ee 59
`
`3.8 Indices... ee ee 64
`
`3.8.1 Mechanism .. 1... ... eee ee ee 64
`
`3.8.2
`
`Spatial Indices
`
`.. 2.2...
`
`. 0... eee ee ee ee 65
`
`3.8.3 Temporal Indices ...... 2... 2... eee ee ee ee ee 66
`
`3.8.4
`
`Spatiotemporal Indices... 2... .. 2.2.2.0... ee eee 66
`
`Ex. A Page 5 of 92
`
`Canon Corrected Ex. 1007 Page 19 of 219
`Canon Corrected Ex. 1007 Page 19 of 219
`
`Ex. A Page 5 of 92
`
`

`

`4
`
`Implementation
`
`68
`
`4.1 Database... 2... ee ee ee 68
`
`4.2 Spatiotemporal Representations ............205 002s ee
`
`71
`
`4.3
`
`Indices. 2... ee ee 71
`
`4.3.1 Mechanism .. 1... 2... pee eee et ete ee ee ns
`
`4.3.2
`
`Spatial Indices 2... ... ee ee ee es
`
`4.3.3 Temporal Indices ... 1... ... 0.2... ee eee ee eens
`
`71
`
`72
`
`75
`
`4.4 Queries. 2...ee 77
`
`4.5 Input... ee ee ee ee ee 17
`
`4.6 Graphical Query Interface... 2... . ee ee ee es
`
`78
`
`5 Performance
`
`5.1 Spatiotemporal Object Storage and Retrieval. .............
`
`81
`
`82
`
`5.2
`
`Index Comparison... 1... . ee ee 83
`
`6 Conclusion
`
`88
`
`Ex. A Page 6 of 92
`
`Canon Corrected Ex. 1007 Page 20 of 219
`Canon Corrected Ex. 1007 Page 20 of 219
`
`Ex. A Page 6 of 92
`
`

`

`List of Figures
`
`3-1 Spatial objects 2... tee 25
`
`3-2 Discrete point set... 2. ee ee es 26
`
`3-3 Abstract point set... ee ee ee ee ee 27
`
`3-4 Coordinate systems... 1... ee te es 28
`
`3-5 Relative spatial objects... 1... ee eee 30
`
`3-6 Breaking a relative spatial specification, partl.............
`
`3-7 Breaking a relative spatial specification, part 2.............
`
`32
`
`32
`
`3-8 Uncertain edges... 0... ee ee ee 33
`
`3-9 Uncertain location ©... 0... 2. ee ee ee es 34
`
`3-10 Conflicting information... 1... ee ee ee ee 35
`
`3-11 Temporal element... ..... 2.2... 0.2 eee ee eee ee ee es 38
`
`3-12 Overlapping temporal elements .............02020 20s
`
`38
`
`3-13 Temporal resolution in favor of version A... .....2.2.020005 39
`
`3-14 Temporal resolution in favor of version B...........-..0005 39
`
`3-15 Relative temporal specification... 2... 2... ee ee ee ee 40
`
`3-16 Probabilistic temporal interval... .........2. 20. e eae 42
`
`3-17 Overlapping probabilistic temporal intervals .............- 43
`
`3-18 Probabilistic conjunction by minimization ...............
`
`43
`
`3-19 Probabilistic disjunction by maximization ...............- 44
`
`3-20 Discrete spatiotemporal information. .............6...-5 46
`
`3-21 Interpolated spatiotemporal state... 2.2... .. 0. eee ee eee 46
`
`3-22 Point set trajectory .
`
`6 6. ee ee ns 47
`
`3-23 Coordinate system trajectory .. 2... 1... eee ee eee eee 48
`
`Ex. A Page 7 of 92
`
`Canon Corrected Ex. 1007 Page 21 of 219
`Canon Corrected Ex. 1007 Page 21 of 219
`
`Ex. A Page 7 of 92
`
`

`

`3-24 Spatial queries... 1... es 55
`
`3-25 Temporal queries .. 2... 1... ee ee et 58
`
`3-26 States of a spatiotemporal object
`
`. 2... 0. ee te ee 60
`
`3-27 Joint spatial and temporal queries... 1... .....0 00. e ee 61
`
`3-28 Spatiotemporal queries... 2... ee ee es 62
`
`4-1 Scene monitoring prototype
`
`.. 2... 2.2... eee eee ee es
`
`69
`
`4-2 Fixed grid spatial index 2... 1... eee ee ee ee ee 73
`
`4-3 Segmented space for bucket PR quadtree. ...............
`
`74
`
`4-4 Data structure for bucket PR quadtree ..............2-- 74
`
`4-5 Temporal segment tree... 2.2... ee ee ee 76
`
`4-6 Temporal B+ tree... . 2... te te te es
`
`76
`
`4-7 Graphical query interface viewing region ..............4.- 78
`
`4-8 Specification of query times and classes ...........+2000% 79
`
`4-9 Graphical query results... 2... 2 ee ee ee 80
`
`5-1 Spatiotemporal update performance. ............--20005 82
`
`5-2 Spatial update performance ............ 0.00 eee eee 84
`
`5-3 Temporal update performance ..........0 200 ee eee nee
`
`5-4 Spatial query performance ... 2... 1... 0. eee ee te ees
`
`85
`
`86
`
`5-5 Temporal query performance... .......-.0 2.0 ee ene 87
`
`Ex. A Page 8 of 92
`
`Canon Corrected Ex. 1007 Page 22 of 219
`Canon Corrected Ex. 1007 Page 22 of 219
`
`Ex. A Page 8 of 92
`
`

`

`Chapter 1
`
`Introduction
`
`Visual memory supports computer vision applications by efficiently storing and re-
`
`trieving spatiotemporal information. It is a unique combination of databases, spatial
`
`representation and indexing, and temporal representation and indexing. Visual mem-
`
`ory provides representational flexibility and high-performance information access to
`
`meet the requirements of a variety of computer vision applications.
`
`1.1 Needs for Visual Memory
`
`Applications use spatiotemporal data in manydifferent ways and place manydifferent
`
`demands on a visual memory. Studying possible uses helps to clarify the concept of
`
`a visual memoryand toidentify the functionality it provides.
`
`Visual memory could serve as the repository for static information, such as ob-
`
`ject descriptions, maps, and environment models, that applications reference during
`
`execution. For example, a vehicle navigator could store maps and images to help it
`
`later recognize its location. A large amount of such information could be established
`
`prior to application execution, and the visual memory would subsequently provide an
`
`application with efficient access to desired pieces of information.
`
`An application could store dynamic information in the visual memory. For ex-
`
`ample, a vehicle navigator’s input systems could maintain in the visual memory a
`
`description of the vehicle’s local environment, updating it as the vehicle moved. The
`
`Ex. A Page 9 of 92
`
`Canon Corrected Ex. 1007 Page 23 of 219
`Canon Corrected Ex. 1007 Page 23 of 219
`
`Ex. A Page 9 of 92
`
`

`

`visual memory could provide the navigator’s planning processes with information
`
`about the vehicle’s latest state and could analyze its progress to help determine a
`
`course of action. The high performance of the visual memoryallowsit to handle the
`
`frequent updates and queries needed by such dynamic,real-time systems.
`
`Visual memory could manipulate spatiotemporal information about objects and
`
`collections of objects too large to fit into volatile memory. For example, a computer-
`
`aided design and modeling system could use the visual memoryin building up a large
`
`design layout and simulating its execution over time; a photo interpretation system
`
`could similarly construct in the visual memory a complex representation of a scene.
`
`The visual memory would retrieve into main memory only a manageable part of a
`
`large representation at a time.
`
`Visual memory could act as the interface between inputs and applications in a
`
`computer vision system. For example, computer vision algorithms for a security
`
`system could analyze data provided by various cameras and store information in the
`
`visual memory. Applications could then retrieve this data to track objects, watch for
`
`suspicious events, and respond to user queries. The visual memory would coordinate
`
`the information from its inputs and eliminate the need for full connectivity between
`
`inputs and applications.
`
`Finally, visual memory could serve as a means for data transfer. A computer
`
`vision application could store spatiotemporal information in the visual memoryfor
`
`other applications to retrieve at any time in the future. To run comparative studies,
`
`different algorithms could use common data stored in the visual memory.
`
`1.2 Goals
`
`This thesis explores visual memory design and implementation. The primary goal
`
`of the thesis is to design a visual memoryarchitecture that meets the requirements
`
`of various computer vision applications. A secondary goal is to implement a visual
`
`memory prototype to support a real-time scene monitoring prototype.
`
`10
`
`Ex. A Page 10 of 92
`Canon Corrected Ex. 1007 Page 24 of 219
`Canon Corrected Ex. 1007 Page 24 of 219
`
`Ex. A Page 10 of 92
`
`

`

`Chapter 2
`
`Background
`
`Visual memory builds on research in database design, spatial representation and
`
`indexing, and temporal representation and indexing. While there has been significant
`
`research i