Inter Partes Review
`United States Patent No. 7,932,923
`
`UNITED STATES PATENT AND TRADEMARKOFFICE
`
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`
`
`Canon Inc., Canon U.S.A., Inc., and Axis Communications AB,
`
`Petitioner
`
`V.
`
`Avigilon Fortress Corporation,
`
`Patent Owner
`
`
`
`Case: Unassigned
`
`U.S. Patent No. 7,932,923
`Issue Date: April 26, 2011
`
`Title: Video Surveillance System Employing Video Primitives
`
`
`DECLARATION OF EMILY R. FLORIO
`
`Canon Ex.1007 Page 1 of 219
`
`Canon Ex. 1007 Page 1 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`I, Emily R. Florio, state and declare as follows:
`
`l.
`
`I have prepared this Declaration in connection with the Petitions of
`
`Axis Communications AB, CanonInc., and Canon U.S.A., Inc. (collectively
`
`“Petitioner”) for two inter partes reviews of U.S. Patent No. 7,932,923 (“the °923
`
`patent”), which I understand will be filed concurrently with this Declaration. I also
`
`understand that October 1999 is a date that is relevant for determining whatis prior
`
`art to the °923 patent.
`
`2.
`
`[am currently the Director of Research & Information Servicesat
`
`Finnegan, Henderson, Farabow, Garrett & Dunner LLP, 901 New York Avenue
`
`NW, Washington, DC 20001-4413.
`
`3.
`
`lam over 18 years of age and am competent to makethis Declaration.
`
`I make this Declaration based on my ownpersonal knowledge, based on my
`
`knowledge oflibrary science practices, as well as my knowledgeof the practices at
`
`the Massachusetts Institute of Technology (“MIT”) Libraries.
`
`4,
`
`J earned a Master’s of Library Science (“MLS”) from Simmons
`
`College in 2006, and I have worked asa librarian for over a decade.
`
`I have been
`
`employed in the Research & Information Services (formerly Library) Department
`
`of Finnegan since 2013, and from 2005-2013, I worked in the Library Department
`
`of Fish & Richardson P.C.
`
`Canon Ex. 1007 Page 2 of 219
`
`Canon Ex. 1007 Page 2 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`I am currently the Vice-President Elect of the American Association
`
`5.
`
`of Law Libraries and the President of the Law Librarians’ Society of Washington,
`
`DC, and a memberofthe International Legal Technology Association.
`
`Attachments
`
`6.
`
`Attached as Exhibit A (Exhibit 1003 to the Petition) is a true and
`
`correct copy of “Visual Memory,” May 1993, pp. 1-92, by Christopher James
`
`Kellogg (“Kellogg”), obtained from the MITLibraries.
`
`7.
`
`Attached as Exhibit Bis a true and correct copy ofthe “Standard”
`
`record from the MITLibraries’ catalog system (knownas the Barton Catalog) for
`
`its copy of Kellogg.
`
`8.
`
`Attached as Exhibit C is a true and correct copy of the MARCrecord
`
`of the MIT Libraries for its copy of Kellogg.
`
`9.
`Attached as Exhibit D (Exhibit 1004 to the Petition) is a true and
`correct copy of F. Brill et al., “Event Recognition and Reliability Improvements
`
`for the Autonomous Video Surveillance System,” Proceedings of the Image
`
`Understanding Workshop, Monterey, CA, Nov. 20-23, 1998, Vol. 1, pp. 267-283
`
`(“Brill”), obtained from the Duderstadt Center, formerly known as the University
`
`of Michigan Media Union (UMMU).
`
`10. Attached as Exhibit Eis a true and correct copy of the MARCrecord
`
`of the University of Virginia Library forits copy of Brill.
`
`3
`
`Canon Ex. 1007 Page 3 of 219
`
`Canon Ex. 1007 Page 3 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`11. Attached as Exhibit Fis a true and correct copy of the MARCrecord
`
`of the North Carolina State University library forits copy of Brill.
`
`12. Attached as Exhibit G (Exhibit 1006to the Petition) is a true and
`
`correct copy of N. Dimitrovaet al., “Motion Recovery for Video Content
`
`Classification,” ACM Transactions on Information Systems, October 1995, Vol.
`
`13, No. 4, pp. 408-439 (“Dimitrova’”), obtained from University of California Los
`
`Angeles Science & Engineering Library.
`
`13. Attached as Exhibit His a true and correct copy of Dimitrova,
`
`obtained from the Library of Congress.
`
`14. Attached as Exhibit I is a true and correct copy of the MARCrecord
`
`of the MITLibraries for its copy of the ACM Transactions on Information Systems
`
`journal, in which Dimitrova was published.
`
`15. Attached as Exhibit J is a true and accurate copy of B. Flinchbaughet
`
`al., “Autonomous Video Surveillance,” SPIE Proceedings, 25" AIPR Workshop:
`
`Emerging Applications of ComputerVision, Feb. 26, 1997, Vol. 2962, p. 144-151
`
`(“Flinchbaugh”), obtained from the MITLibraries.
`
`16. Attached as Exhibit Kis a true and correct copy of the MARC record
`
`of the Library of Congress forits copy of the SPIE Proceedings publication that
`
`includes Flinchbaugh.
`
`Canon Ex. 1007 Page 4 of 219
`
`Canon Ex. 1007 Page 4 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`17. Attached as Exhibit Lis a true and correct copy of the MARCrecord
`
`of the MITLibraries forits copy of the SPIE Proceedings publication that includes
`
`vy
`
`The MARC Cataloging System
`
`18.
`
`The MAchine-Readable Cataloging (“MARC”) system is used by
`
`libraries to catalog materials. The MARC system was developed in the 1960sto
`
`standardize bibliographic records so they could be read by computers and shared
`
`amonglibraries. By the mid-1970’s, MARChad becomethe international standard
`
`for bibliographic data,andit is still used today.
`
`19.
`
`Each field ina MARCrecord provides information aboutthe
`
`cataloged item. MARCusesa simple three-digit numeric code (from 001-999) to
`
`identify each field in the record.
`
`20.
`
`For example,field 245lists the title of the work and field 260 lists
`
`publisher information.
`
`In addition, field 008 provides the date the item was
`
`cataloged. The first six characters of the field 008 are always in the “YYMMDD”
`
`format.
`
`21.
`
`It is standard library practice that once an item is cataloged using the
`
`MARCsystem,it is shelved. This process may take a relatively nominal amount
`
`of time (i.e., a few days or weeks). During the time between the cataloging and
`
`Canon Ex. 1007 Page 5 of 219
`
`Canon Ex. 1007 Page 5 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`shelving ofan item, the public maystill find the item by searching the catalog and
`requesting the item from thelibrary.
`
`Kellogg
`
`22.
`As indicated in Exhibit A (Exhibit 1003 to the Petition), Kellogg has
`an MIT Libraries date stamp of“JUL 09 1993”on page 1, indicating that the MIT
`
`Libraries received Kellogg on July 9, 1993. Further, as indicated in Exhibit B, the
`
`Standard record of the Barton Catalog confirms that Kellogg is shelved at the MIT
`Libraries and was published in 1993.
`In view of the above and the following,
`Kellogg was published andaccessible to the public in 1993, years before October
`
`1999.
`
`23.
`Asindicated in Exhibit C, Kellogg has a cataloging date of September
`28, 1993 (shown as “930928”in field 008). This confirms that Kellogg was
`
`entered into the OCLC database, in which MITdoesits cataloging, on September
`
`28, 1993. This is also consistent with its noted yearof publication in the MARC
`
`record (shown as “1993”in field 260). The OCLC database(also referred to as
`
`“WorldCat”) is the largest online public access catalog (OPAC)in the world.
`
`24.
`
`Soonafter Kellogg received a cataloging date, a record ofits existence
`
`would have appeared in and been keyword-searchable through the Barton Catalog
`of the MITLibraries. The Barton Catalog is currently available online to any user
`
`of the World Wide Web. Before it was accessible by Web(i.e., at the timethe
`
`6
`
`Canon Ex. 1007 Page 6 of 219
`
`Canon Ex. 1007 Page 6 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`Kellogg thesis wasreceived by the MITLibraries in July 1993), it would have been
`
`accessible to anyone on the MIT campus and anyone who hadaccess to the OCLC
`
`database.
`
`25. During the time period from September 1993 through October 1999,
`
`the Barton Catalog allowed keyword searching for wordsin the thesistitle, and
`
`Kellogg would have appearedin a relevant Barton Catalog search conductedon or
`
`shortly after September 28, 1993.
`
`26. After being cataloged, a document such as Kellogg will undergo a
`
`process of being labeled and then shelved at the MIT Libraries. Based on my
`knowledge ofMITLibraries’ current and priorpractices, Kellogg would have been
`
`shelved in a relatively nominal amountoftime(i.e., a few days or weeks). Thus,
`
`Kellogg was cataloged and shelved at the MITLibraries at least before the end of
`
`1993.
`
`27.
`
`Once shelved, Kellogg can be borrowed by any memberof the MIT
`
`community. Furthermore, a copy of Kellogg can be purchased from MITby any
`
`member ofthe public.
`
`Indeed,thefirst page of Kellogg confirmsthat there were
`
`no restrictions placed onits publication,as it states that “[t]he author hereby grants
`
`to MITpermission fo reproduce and to distribute copies ofthis thesis document in
`
`whole orin part, and to grantothers the right to do so.”
`
`Canon Ex. 1007 Page 7 of 219
`
`Canon Ex. 1007 Page 7 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`28.
`
`Further evidence ofthe public availability of Kellogg before October
`
`1999 is provided in Exhibit J, which is a copy of Flinchbaugh.
`Inits Bibliography,
`Flinchbaughcites to Kellogg (reference [4] on p. 151). Asaddressed below,
`
`Flinchbaugh waspublished in SPIE Volume 2962, which correspondsto the
`
`Proceedings from the 25" Annual AIPR Workshop on Emerging Applications of
`
`Computer Vision. The Workshop was held October 16-18, 1996, and the
`
`Proceedings were published by at least 1997. Thus, Kellogg wasat least available
`
`to members of the public in 1997, as shownbyits citation in Flinchbaugh.
`
`29.
`
`For the avoidance of any doubt, | note that on June 23, 2001, Kellogg
`
`wasalso cataloged in the MIT Archive Noncirculating Collection 1,
`
`Noncirculating Collection 3, and in microfiche form in the Barker Library, as
`
`indicated in the three entries for PST8 andin the second,third, and fourth instances
`
`of field 008 on page 1 of Exhibit C. However, noneofthis alters the fact that
`
`Kellogg was published and accessible to the public in 1993, as indicated above.
`
`30.
`
`Asindicated in Exhibit D, Brill is part of the published Proceedingsof
`
`the 1998 Image Understanding Workshop. The Workshop washeldin Monterey,
`
`California during November 20-23, 1998, and the Proceedings were “APPROVED
`
`FOR PUBLIC RELEASE”with “DISTRIBUTION UNLIMITED.” Ex. D at 1.
`
`In
`
`Canon Ex. 1007 Page 8 of 219
`
`Canon Ex. 1007 Page 8 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`view of the above and the following, the Proceedings, including Brill, was
`
`published andaccessible to the public before October 1999.
`
`31.
`
`Evidence of Brill’s publication and availability to the public includes
`
`the hand-written receipt date of “8-13-99”at the top of page 3 of Exhibit D. This
`
`indicates it was received by theUMMU(the University of Michigan Media Union,
`
`now knownas the Duderstadt Center) on August 13, 1999.
`
`In my experience as a
`
`librarian and knowledgeofstandard library practices, the hand-written information
`
`at the top of p. 2 of Exhibit D appearsto be the catalog record informationfor
`
`Brill. Based on standard library practices, this reference would have been shelved
`
`shortly after being received and cataloged by UMMU.
`
`32.
`
`Further evidence of the publication and accessibility of Brill to the
`
`public can be found in Exhibit E, which is the MARCrecord forthe Proceedings,
`
`including Brill, that was obtained from the University of Virginia Library. As
`
`shownin field 008 nearthe top of page 2 of Exhibit E, Brill was cataloged bythe
`
`library on December15, 1998. Based on standard library practices, this reference
`
`would have been shelvedshortly after(i.e., within a few days or weeks) and been
`
`accessible to the public prior to October 1999,
`
`33.
`
`Further evidence of the publication and public availability of Brill can
`
`be found in Exhibit F, which is the MARC record forthe Proceedings, including
`
`Brill, that was obtained from North Carolina State University. As shownin field
`
`9
`
`Canon Ex. 1007 Page 9 of 219
`
`Canon Ex. 1007 Page 9 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`008 on page 1 of Exhibit F, Brill was cataloged bythe library on December 15,
`
`1998. Based on standardlibrary practices, this reference would have been shelved
`
`shortly after (i.e., within a few days or weeks) and been accessible to the public
`
`prior to October 1999,
`
`Dimitrova
`
`34.
`
`As indicated in Exhibit G, Dimitrova was publishedin a special issue
`
`of the ACM Transactions on Information Systems Journal. Ex. G at 1.
`In view of
`the above and the following, the ACM Journal, including Dimitrova, was
`
`published and accessible to the public before October 1999.
`
`35.
`
`Evidence of Dimitrova’s publication and availability to the public
`
`includes the “November 17, 1995” date stamp on page | of Exhibit G. This
`
`confirms that Dimitrova was received by the University of California Los Angeles
`
`Science & Engineering Library on November 17, 1995. Based on standard library
`
`practices, this reference would have been shelved shortly after (i.e., within a few
`
`days or weeks) and accessibleto the public before October 1999.
`
`36.
`
`Further evidenceofthe publication and accessibility of Dimitrova to
`
`the public is found in Exhibit H, which is a copy of Dimitrova obtained from the
`
`Library of Congress. Page 3 of Exhibit H bears the date stamp of “November 21,
`
`1995” from the Library of Congress. This confirms that Dimitrova was received
`
`by Library of Congress on November 21, 1995. Based on standard library
`
`10
`
`Canon Ex. 1007 Page 10 of 219
`
`Canon Ex. 1007 Page 10 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`practices, this reference would have been shelved shortly after(i.e., within a few
`
`days or weeks) and accessible to the public before October 1999,
`
`37.
`
`Further evidence ofthe publication and accessibility of Dimitrova to
`
`the public can be found in Exhibit I, which is the MARCrecord of the ACM
`
`Journal, which includes Dimitrova, obtained from the MIT Libraries. As shown in
`
`field 008 nearthe top of page 1 of Exhibit I, the MIT Libraries began receiving the
`
`ACM Journal in September 1989.
`
`38.
`
`Field 362 (shownas “3620”) on page | of Exhibit I indicatesthat the
`
`MITLibraries has issues dating back to Volume 7 of the journal, which as noted
`
`above was published in September 1989. There is no end-date in field 362,
`
`indicating that the MITLibraries has an ongoing subscription, which would have
`
`included receipt of the issue that contained Dimitrova in 1995. Based on standard
`
`library practices, the issue containing Dimitrova would have beenshelved shortly
`
`after cataloging (i.e., within a few days or weeks) and accessible to the public
`
`before October 1999.
`
`39.
`
`For the avoidance of any doubt, I note that it appears that on June 23,
`
`2001, online access to Dimitrova was provided to certain MIT-associated
`
`individuals, as indicated by the field 008 on thelast line of page 1, field 8528, and
`
`the URLentry at the top of page 2 of Exhibit I. Also on June 23, 2001, the ACM
`
`Journal, including Dimitrova, was archived at the MITLibrary Storage Annex
`
`1]
`
`Canon Ex. 1007 Page 11 of 219
`
`Canon Ex. 1007 Page 11 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`(“LSA”), as indicated by the 008 field and subsequent 85271 entry on page 2 of
`
`Exhibit I. However, noneofthis alters the fact that Dimitrova was published and
`
`accessible to the public years before October 1999, as indicated above.
`
`Flinchbaugh
`
`40.
`
`As indicated in Exhibit J, F/inchbaugh was published in the
`
`Proceedings of the 25" AIPR Workshop: Emerging Applications of Computer
`
`Vision, SPIE Vol. 2962. The Workshop was held in Washington, D.C. during
`
`October 16-18, 1996, and the Proceedings was published by SPIE (The
`
`International Society for Optical Engineering). Ex. J at 1.
`
`In view of the above
`
`and the following, Flinchbaugh was published andaccessible to the public before
`
`October 1999.
`
`41.
`
`Page 2 of Exhibit J shows a copyright date of 1997. The edition of the
`
`SPIE Proceedings that was published with Flinchbaugh is Volume 2962, andit
`
`was “Printed in the United States of America.” Ex. J at 2.
`
`42.
`
`Althoughthe copyright date of Flinchbaughislisted as 1997, it
`
`appears that /’linchbaugh wasactually published before that, in 1996. First, as
`
`noted above, the Workshop was held in Washington, D.C. during October 16-18,
`
`1996. Second, a copy of Flinchbaugh wasreceived and cataloged by the Library
`
`of Congress in November 1996. See Ex. K at 1. Exhibit K is the MARC record
`
`for the SPIE Proceedings, including linchbaugh, that was obtained from the
`
`12
`
`Canon Ex. 1007 Page 12 of 219
`
`Canon Ex. 1007 Page 12 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`Library of Congress. As shownin field 008 nearthe top of page 2 of Exhibit K,
`Flinchbaugh wascataloged bythe library on November 21, 1996. Based on
`
`standard library practices, this reference would have beenshelvedshortly afterit
`wascataloged (i.e., within a few days or weeks). Collectively, Exhibits J and kK
`
`show that Flinchbaugh waspublished and accessible to the public years before
`
`October 1999,
`
`43.
`
`Further evidence ofthe publication and public availability of
`
`Flinchbaugh can be found in Exhibit L, which is the MARCrecord for the SPIE
`
`Proceedings, including F/inchbaugh, that was obtained from the MITLibraries.
`
`As shown in field 008 on page 1 of Exhibit L, Flinchbaugh was cataloged by the
`
`library on March 10, 1997. Based on standard library practices and my
`
`understanding of the practices of the MITLibraries, this reference would have
`
`been shelved shortly after it was cataloged (i.e., within a few days or weeks) and
`
`accessible to the public before October 1999.
`
`44.
`
`For the avoidance of any doubt, I note that on April 8, 2011, online
`
`access to Flinchbaugh wasprovided to certain MIT-associated individuals,as
`
`indicated by the fields 008 and 8528 and the URLentry at the top of page 2 of
`
`Exhibit L. Also, on June 23, 2001, the SPIE Proceedings, including Flinchbaugh,
`
`was archived at the MIT Library Storage Annex (“LSA”), as indicated by the
`
`second 008 field and subsequent 8520 entry on page 2 of Exhibit L. However,
`
`13
`
`Canon Ex. 1007 Page 13 of 219
`
`Canon Ex. 1007 Page 13 of 219
`
`

`

`Inter Partes Review
`United States Patent No. 7,932,923
`
`ay
`
`public years before October 1999, as indicated above.
`
`I declare underpenalty of perjury that the foregoing is true and correct.
`
`Executed on November6, 2018 in Washington, D.C.
`
`fn No fy/
`
`/ mA | [ (
`
`\
`SA
`Emily R. Florio
`
`Canon Ex. 1007 Page 14 of 219
`
`Canon Ex. 1007 Page 14 of 219
`
`

`

`Visual Memory
`
`by
`Christopher James Kellogg
`
`Submitted to the Department of Electrical Engineering and Computer Science
`in partial fulfillment of the requirements for the degrees of
`
`Bachelor of Science
`and
`Master of Science in Computer Science
`
`at the
`
`MASSACHUSETTS INSTITUTE OF TECHNOLOGY
`
`May 1993
`
`@ Christopher James Kellogg, MCMXCIII. All rights reserved.
`
`The author hereby grants to MIT permission to reproduce and to distribute copies
`of this thesis document in whole or in part, and to grant others the right to do so.
`
`Signature redacted
`Author ................ D~~~~~~~~~ ~f· El~~~;i~·~. E~~i~~;Jn~ ~~~;---flci~~~~
`/ ) /
`j
`ril
`, 1993
`Signature redacted
`
`.
`
`Certified by .................................. -.-.r• ... ·v •.•• • ·- .................. .
`Alex P. Pentland
`Associate Professor of Media Arts and Sciences
`. Thesis Supervi§or
`Signature redacted
`Certified by ...................................... ·r · · · · · · · ... ,. · ·~· · · · · · · · · ..... , -· r
`Bruce E. Flinchbaugh
`Manager of Image Understandin~ra~h a~ruments
`. >-:? /J
`/"/ / // / ~s/supervisor
`
`Signature redacted
`Accepted by ........... ~; . . 1 ;/· .) .. 07 .. ·r· ·v .. . 6~~~];~n·L.· ~i~~ri~
`
`Ch7on, ~rtmental Committee on Graduate Students
`MASSACHUSETIS INSTITUTE
`OF TFr.HNOLOGY
`rJUL 09 1993
`LIBRARIES
`ARCHIVE~
`
`Ex. A Page 1 of 92
`
`Canon Ex. 1007 Page 15 of 219
`
`

`

`Visual Memory
`
`by
`
`Christopher James Kellogg
`
`Submitted to the Department of Electrical Engineering and Computer Science
`on April 21, 1993, in partial fulfillment of the
`requirements for the degrees of
`Bachelor of Science
`and
`Master of Science in Computer Science
`
`Abstract
`
`Visual memory supports computer vision applications by efficiently storing and re(cid:173)
`trieving spatiotemporal information. It is a unique combination of databases, spatial
`representation and indexing, and temporal representation and indexing. This the(cid:173)
`sis designs a visual memory architecture that meets the requirements of a number
`of computer vision applications. It also presents an implementation of part of this
`design in support of a scene monitoring prototype.
`
`Thesis Supervisor: Alex P. Pentland
`Title: Associate Professor of Media Arts and Sciences
`
`Thesis Supervisor: Bruce E. Flinchbaugh
`Title: Manager of Image Understanding Branch at Texas Instruments
`
`Ex. A Page 2 of 92
`
`Canon Ex. 1007 Page 16 of 219
`
`

`

`Acknowledgements
`
`My primary thanks goes to my two thesis supervisors, Bruce Flinchbaugh at Texas
`
`Instruments and Sandy Pentland at MIT. Bruce pointed me to the visual memory
`
`project that he was starting and guided my research at Texas Instruments. Sandy
`
`provided useful feedback throughout the research stage. They were both very helpful
`
`in critiquing the thesis document.
`
`I'd also like to thank the other people at Texas Instruments who helped me with
`
`this project. Steve Ford and Tom Bannon were especially helpful in developing the
`
`visual memory design. In addition, I don't think I would have survived the bugs in
`
`PC++ without Steve's expertise. Tom Bannon and Tom O'Donnell provided a nice
`
`tracking system with which to test the visual memory prototype.
`
`Finally, I'd like to thank my family, Fred, Jeannette, and Mark Kellogg, my fiancee
`
`Christine Bailey, and my brothers at Phi Kappa Sigma for their support throughout
`
`my MIT career.
`
`3
`
`Ex. A Page 3 of 92
`
`Canon Ex. 1007 Page 17 of 219
`
`

`

`Contents
`
`1 Introduction
`
`1.1 Needs for Visual Memory.
`1.2 Goals .. ..........
`
`2 Background
`2.1 Database Research ......
`2.1.1 DARPA Open OODB
`
`2.1.2 POSTGRES.
`
`2.2 Spatial Research
`
`2.2.1 CODGER
`
`2.2.2 Core Knowledge System
`
`2.2.3
`
`ISR . . . . . . . . . . . .
`
`2.2.4
`Image Understanding Environments .
`2.2.5 PROBE ....
`2.2.6 Spatial Indices
`
`2.3 Tern poral Research
`2.3.1 TQuel ...
`2.3.2 Temporal Sequences
`
`2.3.3 Temporal Sets .
`
`2.3.4 Relative Time .
`
`2.3.5 Temporal Indices
`
`4
`
`9
`
`9
`
`10
`
`11
`
`11
`
`11
`
`12
`
`13
`
`13
`
`13
`
`14
`
`14
`
`14
`
`15
`
`15
`
`15
`
`16
`
`16
`
`17
`
`17
`
`Ex. A Page 4 of 92
`
`Canon Ex. 1007 Page 18 of 219
`
`

`

`3 Design
`
`3.1 Requirements and Considerations
`
`3.1.1 Database Considerations
`
`3.1.2 Spatial and Temporal Considerations
`
`3.1.3 Performance Considerations
`3.2 Design Overview ....
`3.3 Spatial Representations .
`
`3.3.1 Core Spatial Classes
`
`3.3.2 Relative Spatial Specification
`
`3.3.3 Uncertain Spatial Specification
`3.4 Temporal Representations ..
`3.4.1 Core Temporal Classes
`
`3.4.2 Relative Temporal Specification
`
`3.4.3 Uncertain Temporal Specification
`
`3.5 Spatiotemporal Representations
`
`3.6 Object Storage
`
`3.6.1
`
`Identity
`
`3.6.2 Storage Mechanism
`
`3.6.3 Time
`
`3.7 Queries . . .
`
`3.7.1 Query Mechanism .
`
`3.7.2 Spatial Queries
`
`3.7.3 Tern poral Queries
`
`3.7.4 Spatiotemporal Queries .
`
`3.8
`
`Indices . . . . . . .
`
`3.8.1 Mechanism
`
`3.8.2 Spatial Indices
`
`3.8.3 Temporal Indices
`
`3.8.4 Spatiotemporal Indices
`
`5
`
`18
`
`19
`
`19
`
`20
`
`20
`
`22
`
`24
`
`24
`
`29
`
`31
`
`36
`
`36
`
`40
`
`41
`
`45
`
`50
`
`50
`
`51
`
`52
`
`53
`
`53
`
`54
`
`57
`
`59
`
`64
`
`64
`
`65
`
`66
`
`66
`
`Ex. A Page 5 of 92
`
`Canon Ex. 1007 Page 19 of 219
`
`

`

`4 Implementation
`4.1 Database . . . ..........
`4.2 Spatiotemporal Representations
`
`4.3
`
`Indices . . . . . . .
`
`4.3.1 Mechanism
`
`4.3.2 Spatial Indices
`
`4.3.3 Temporal Indices
`
`4.4 Queries.
`
`4.5
`
`Input ..
`
`4.6 Graphical Query Interface
`
`5 Performance
`
`5.1 Spatiotemporal Object Storage and Retrieval .
`Index Comparison . ...............
`
`5.2
`
`6 Conclusion
`
`68
`
`68
`
`71
`
`71
`
`71
`
`72
`
`75
`
`77
`
`77
`
`78
`
`81
`
`82
`
`83
`
`88
`
`6
`
`Ex. A Page 6 of 92
`
`Canon Ex. 1007 Page 20 of 219
`
`

`

`List of Figures
`
`3-1 Spatial objects
`
`3-2 Discrete point set
`
`3-3 Abstract point set .
`
`3-4 Coordinate systems
`
`3-5 Relative spatial objects
`
`3-6 Breaking a relative spatial specification, part 1
`
`3-7 Breaking a relative spatial specification, part 2 .
`
`3-8 Uncertain edges . .
`
`3-9 Uncertain location
`
`3-10 Conflicting information
`
`3-11 Temporal element ...
`
`3-12 Overlapping temporal elements
`
`3-13 Temporal resolution in favor of version A
`
`3-14 Temporal resolution in favor of version B
`
`3-15 Relative temporal specification .
`
`3-16 Probabilistic temporal interval .
`
`3-17 Overlapping probabilistic temporal intervals
`
`3-18 Probabilistic conjunction by minimization
`
`3-19 Probabilistic disjunction by maximization
`
`3-20 Discrete spatiotemporal information .
`
`3-21 Interpolated spatiotemporal state
`
`3-22 Point set trajectory . . . . . .
`
`3-23 Coordinate system trajectory
`
`7
`
`25
`
`26
`
`27
`
`28
`
`30
`
`32
`
`32
`
`33
`
`34
`
`35
`
`38
`
`38
`
`39
`
`39
`
`40
`
`42
`
`43
`
`43
`
`44
`
`46
`
`46
`
`47
`
`48
`
`Ex. A Page 7 of 92
`
`Canon Ex. 1007 Page 21 of 219
`
`

`

`55
`
`58
`
`60
`
`61
`
`62
`
`69
`
`73
`
`74
`
`74
`
`76
`
`76
`
`78
`
`79
`
`80
`
`82
`
`84
`
`85
`
`86
`
`87
`
`3-24 Spatial queries ..
`
`3-25 Temporal queries
`
`3-26 States of a spatiotemporal object
`
`3-27 Joint spatial and temporal queries .
`
`3-28 Spatiotemporal queries . . .
`
`4-1 Scene monitoring prototype
`4-2 Fixed grid spatial index ..
`4-3 Segmented space for bucket PR quadtree
`
`4-4 Data structure for bucket PR quadtree
`
`4-5 Temporal segment tree
`
`4-6 Temporal B+ tree . . .
`
`4-7 Graphical query interface viewing region
`
`4-8 Specification of query times and classes
`
`4-9 Graphical query results . . . . . . . .
`
`5-1 Spatiotemporal update performance .
`
`5-2 Spatial update performance
`
`5-3 Temporal update performance
`5-4 Spatial query performance ..
`5-5 Temporal query performance .
`
`8
`
`Ex. A Page 8 of 92
`
`Canon Ex. 1007 Page 22 of 219
`
`

`

`Chapter 1
`
`Introduction
`
`Visual memory supports computer vision applications by efficiently storing and re(cid:173)
`
`trieving spatiotemporal information. It is a unique combination of databases, spatial
`
`representation and indexing, and temporal representation and indexing. Visual mem(cid:173)
`
`ory provides representational flexibility and high-performance information access to
`
`meet the requirements of a variety of computer vision applications.
`
`1.1 Needs for Visual Memory
`
`Applications use spatiotemporal data in many different ways and place many different
`
`demands on a visual memory. Studying possible uses helps to clarify the concept of
`
`a visual memory and to identify the functionality it provides.
`
`Visual memory could serve as the repository for static information, such as ob(cid:173)
`
`ject descriptions, maps, and environment models, that applications reference during
`
`execution. For example, a vehicle navigator could store maps and images to help it
`
`later recognize its location. A large amount of such information could be established
`
`prior to application execution, and the visual memory would subsequently provide an
`
`application with efficient access to desired pieces of information.
`
`An application could store dynamic information in the visual memory. For ex(cid:173)
`
`ample, a vehicle navigator's input systems could maintain in the visual memory a
`
`description of the vehicle's local environment, updating it as the vehicle moved. The
`
`9
`
`Ex. A Page 9 of 92
`
`Canon Ex. 1007 Page 23 of 219
`
`

`

`visual memory could provide the navigator's planning processes with information
`
`about the vehicle's latest state and could analyze its progress to help determine a
`
`course of action. The high performance of the visual memory allows it to handle the
`
`frequent updates and queries needed by such dynamic, real-time systems.
`
`Visual memory could manipulate spatiotemporal information about objects and
`
`collections of objects too large to fit into volatile memory. For example, a computer(cid:173)
`
`aided design and modeling system could use the visual memory in building up a large
`
`design layout and simulating its execution over time; a photo interpretation system
`
`could similarly construct in the visual memory a complex representation of a scene.
`
`The visual memory would retrieve into main memory only a manageable part of a
`
`large representation at a time.
`
`Visual memory could act as the interface between inputs and applications m a
`
`computer vision system. For example, computer vision algorithms for a security
`
`system could analyze data provided by various cameras and store information in the
`
`visual memory. Applications could then retrieve this data to track objects, watch for
`
`suspicious events, and respond to user queries. The visual memory would coordinate
`
`the information from its inputs and eliminate the need for full connectivity between
`
`inputs and applications.
`
`Finally, visual memory could serve as a means for data transfer. A computer
`
`vision application could store spatiotemporal information in the visual memory for
`
`other applications to retrieve at any time in the future. To run comparative studies,
`
`different algorithms could use common data stored in the visual memory.
`
`1.2 Goals
`
`This thesis explores visual memory design and implementation. The primary goal
`
`of the thesis is to design a visual memory architecture that meets the requirements
`
`of various computer vision applications. A secondary goal is to implement a visual
`
`memory prototype to support a real-time scene monitoring prototype.
`
`10
`
`Ex. A Page 10 of 92
`
`Canon Ex. 1007 Page 24 of 219
`
`

`

`Chapter 2
`
`Background
`
`Visual memory builds on research in database design, spatial representation and
`
`indexing, and temporal representation and indexing. While there has been significant
`
`research in each of these areas, no previous project has combined them in this manner.
`
`The visual memory design uses knowledge gained from research projects in all these
`
`areas. This chapter summarizes and discusses some especially relevant projects.
`
`2.1 Database Research
`
`Visual memory must address concerns that a great deal of database research has
`
`already investigated. It must provide everything from information storage techniques
`
`to concurrency control for multiple inputs and outputs. Visual memory should build
`
`on the results of research into these topics. Presented here are two databases that
`
`address a number of the issues important to visual memory and that could be the
`
`basis for a visual memory system.
`
`2.1.1 DARPA Open OODB
`
`The DARPA Open Object~Oriented Database (Open OODB) project at Texas In~
`
`struments outlines an extensible architecture that allows " ... tailoring database func~
`
`tionality for particular applications in the framework of an incrementally improvable
`
`system .... " [25] The architecture meets functional requirements such as an object
`
`11
`
`Ex. A Page 11 of 92
`
`Canon Ex. 1007 Page 25 of 219
`
`

`

`data model and concurrent access, along with "meta requirements" including open(cid:173)
`
`ness and reusability. The open architecture lets separate modules handle extensions
`
`to the basic storage mechanism. These extensions cover standard database issues
`
`such as transactions, versions, and queries.
`
`The Open OODB architecture is very suitable for visual memory. The object(cid:173)
`
`oriented model can flexibly and intuitively represent the information used by computer
`
`vision applications. Following the Open OODB architecture, visual memory could
`
`avoid confronting standard database issues by letting other modules support those
`
`features. Instead, visual memory would consist only of those extensions necessary to
`
`support efficient manipulation of spatiotemporal information. If new features were
`
`needed, extra modules could easily be added to the architecture.
`
`2.1.2 POSTGRES
`
`The POSTGRES database [23] expands the relational database model to meet the
`
`needs of complex applications. Because it builds on traditional relational databases, it
`
`provides a number of standard features, such as transactions, a query language, and
`
`recovery processing. In addition, it allows applications to specify new data types,
`
`operators, and access methods. POSTGRES supports active databases and rules,
`
`letting applications set up daemons in the database that react to changes in the data.
`
`A versioning mechanism keeps track of old data and works with the query language
`
`to let applications retrieve this information. Finally, the POSTGRES st