Inter Partes Review
`
`United States Patent No. 7,932,923
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`Canon Inc., Canon USA, Inc., and Axis Communications AB,
`
`Petitioner
`
`V.
`
`Avigilon Fortress Corporation,
`
`Patent Owner
`
`
`
`Case: Unassigned
`
`US. Patent No. 7,932,923
`Issue Date: April 26, 2011
`
`Title: Video Surveillance System Employing Video Primitives
`
`DECLARATION OF EMILY R. FLORIO
`
`Canon Corrected EX. 1007 Page 1 of 219
`
`Canon Corrected Ex. 1007 Page 1 of 219
`
`
`
`United States Patent No. 7,932,923
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`Canon Inc., Canon USA, Inc., and Axis Communications AB,
`
`Petitioner
`
`V.
`
`Avigilon Fortress Corporation,
`
`Patent Owner
`
`
`
`Case: Unassigned
`
`US. Patent No. 7,932,923
`Issue Date: April 26, 2011
`
`Title: Video Surveillance System Employing Video Primitives
`
`DECLARATION OF EMILY R. FLORIO
`
`Canon Corrected EX. 1007 Page 1 of 219
`
`Canon Corrected Ex. 1007 Page 1 of 219
`
`
`
`Inter Partes Review
`
`United States Patent No. 7,932,923
`
`I, Emily R. Florio, state and declare as follows:
`
`1.
`
`I have prepared this Declaration in connection with the Petitions of
`
`Axis Communications AB, Canon Inc., and Canon U.S.A., Inc. (collectively
`
`“Petitioner”) for two inter partes reviews of US. Patent No. 7,932,923 (“the ’923
`
`paten ”), which I understand will be filed concurrently with this Declaration. I also
`
`understand that October 1999 is a date that is relevant for determining what is prior
`
`art to the ’923 patent.
`
`2.
`
`I am currently the Director of Research & Information Services at
`
`Finnegan, Henderson, Farabow, Garrett & Dunner LLP, 901 New York Avenue
`
`NW, Washington, DC 20001-4413.
`
`3.
`
`I am over 18 years of age and am competent to make this Declaration.
`
`I make this Declaration based on my own personal knowledge, based on my
`
`knowledge of library science practices, as well as my knowledge of the practices at
`
`the Massachusetts Institute of Technology (“MIT”) Libraries.
`
`4.
`
`I earned a Master’s of Library Science (“MLS”) from Simmons
`
`College in 2006, and I have worked as a librarian for over a decade. I have been
`
`employed in the Research & Information Services (formerly Library) Department
`
`of Finnegan since 2013, and from 2005-2013, I worked in the Library Department
`
`of Fish & Richardson P.C.
`
`Canon Corrected EX. 1007 Page 2 of 219
`
`Canon Corrected Ex. 1007 Page 2 of 219
`
`
`
`Inter Partes Review
`
`United States Patent No. 7,932,923
`
`5.
`
`I am currently the Vice-President Elect of the American Association
`
`of Law Libraries and the President of the Law Librarians’ Society of Washington,
`
`DC, and a member of the International Legal Technology Association.
`
`Attachments
`
`6.
`
`Attached as Exhibit A (Exhibit 1003 to the Petition) is a true and
`
`correct copy of “Visual Memory,” May 1993, pp. 1-92, by Christopher James
`
`Kellogg (“Kellogg”), obtained from the MIT Libraries.
`
`7.
`
`Attached as Exhibit B is a true and correct copy of the “Standard”
`
`record from the MIT Libraries’ catalog system (known as the Barton Catalog) for
`
`its copy of Kellogg.
`
`8.
`
`Attached as Exhibit C is a true and correct copy of the MARC record
`
`of the MIT Libraries for its copy of Kellogg.
`
`9.
`
`Attached as Exhibit D (Exhibit 1004 to the Petition) is a true and
`
`correct copy of F. Brill et al., “Event Recognition and Reliability Improvements
`
`for the Autonomous Video Surveillance System,” Proceedings of the Image
`
`Understanding Workshop, Monterey, CA, Nov. 20-23, 1998, Vol. 1, pp. 267-283
`
`(“Brill”), obtained from the Duderstadt Center, formerly known as the University
`
`of Michigan Media Union (UMMU).
`
`10. Attached as Exhibit E is a true and correct copy of the MARC record
`
`of the University of Virginia Library for its copy of Brill.
`
`3
`
`Canon Corrected Ex. 1007 Page 3 of 219
`
`Canon Corrected Ex. 1007 Page 3 of 219
`
`
`
`Inter Partes Review
`
`United States Patent No. 7,932,923
`
`11. Attached as Exhibit F is a true and correct copy of the MARC record
`
`ofthe North Carolina State University library for its copy ofBrill.
`
`12. Attached as Exhibit G (Exhibit 1006 to the Petition) is a true and
`
`correct copy of N. Dimitrova et al., “Motion Recovery for Video Content
`
`Classification,” ACM Transactions on Information Systems, October 1995, Vol.
`
`13, No. 4, pp. 408—439 (“Dimitrova”), obtained from University of California Los
`
`Angeles Science & Engineering Library.
`
`13. Attached as Exhibit H is a true and correct copy of Dimitrova,
`
`obtained from the Library of Congress.
`
`14. Attached as Exhibit I is a true and correct copy of the MARC record
`
`of the MIT Libraries for its copy of the ACM Transactions on Information Systems
`
`journal, in which Dimitrova was published.
`
`15. Attached as Exhibit J is a true and accurate copy of B. Flinchbaugh et
`
`al., “Autonomous Video Surveillance,” SPIE Proceedings, 25th AIPR Workshop:
`
`Emerging Applications of Computer Vision, Feb. 26, 1997, Vol. 2962, p. 144—151
`
`(“Flinchbaugh”), obtained from the MIT Libraries.
`
`16. Attached as Exhibit K is a true and correct copy of the MARC record
`
`of the Library of Congress for its copy of the SPIE Proceedings publication that
`
`includes Flinchbaugh.
`
`Canon Corrected Ex. 1007 Page 4 of 219
`
`Canon Corrected Ex. 1007 Page 4 of 219
`
`
`
`Inter Partes Review
`
`United States Patent No. 7,932,923
`
`17. Attached as Exhibit L is a true and correct copy of the MARC record
`
`of the MIT Libraries for its copy of the SPIE Proceedings publication that includes
`
`Flinchbaugh.
`
`The MARC Cataloging System
`
`18.
`
`The MAchine—Readable Cataloging (“MARC”) system is used by
`
`libraries to catalog materials. The MARC system was developed in the 19603 to
`
`standardize bibliographic records so they could be read by computers and shared
`
`among libraries. By the mid-1970’s, MARC had become the international standard
`
`for bibliographic data, and it is still used today.
`
`19.
`
`Each field in a MARC record provides information about the
`
`cataloged item. MARC uses a simple three-digit numeric code (from 001-999) to
`
`identify each field in the record.
`
`20.
`
`For example, field 245 lists the title of the work and field 260 lists
`
`publisher information. In addition, field 008 provides the date the item was
`
`cataloged. The first six characters of the field 008 are always in the “YYMMDD”
`
`format.
`
`21.
`
`It is standard library practice that once an item is cataloged using the
`
`MARC system, it is shelved. This process may take a relatively nominal amount
`
`of time (i.e., a few days or weeks). During the time between the cataloging and
`
`Canon Corrected Ex. 1007 Page 5 of 219
`
`Canon Corrected Ex. 1007 Page 5 of 219
`
`
`
`Inter Partes Review
`
`United States Patent No. 7,932,923
`
`shelving of an item, the public may still find the item by searching the catalog and
`
`requesting the item from the library.
`
`M 2
`
`2. As indicated in Exhibit A (Exhibit 1003 to the Petition), Kellogg has
`
`an MIT Libraries date stamp of “JUL 09 1993” on page 1, indicating that the MIT
`
`Libraries received Kellogg on July 9, 1993. Further, as indicated in Exhibit B, the
`
`Standard record of the Barton Catalog confirms that Kellogg is shelved at the MIT
`
`Libraries and was published in 1993. In view of the above and the following,
`
`Kellogg was published and accessible to the public in 1993, years before October
`
`1999.
`
`23. As indicated in Exhibit C, Kellogg has a cataloging date of September
`
`28, 1993 (shown as “930928” in field 008). This confirms that Kellogg was
`
`entered into the OCLC database, in which MIT does its cataloging, on September
`
`28, 1993. This is also consistent with its noted year of publication in the MARC
`
`record (shown as “1993” in field 260). The OCLC database (also referred to as
`
`“WorldCat”) is the largest online public access catalog (OPAC) in the world.
`
`24.
`
`Soon after Kellogg received a cataloging date, a record of its existence
`
`would have appeared in and been keyword—searchable through the Barton Catalog
`
`of the MIT Libraries. The Barton Catalog is currently available online to any user
`
`of the World Wide Web. Before it was accessible by Web (i.e., at the time the
`
`6
`
`Canon Corrected Ex. 1007 Page 6 of 219
`
`Canon Corrected Ex. 1007 Page 6 of 219
`
`
`
`Inter Partes Review
`
`United States Patent No. 7,932,923
`
`Kellogg thesis was received by the MIT Libraries in July 1993), it would have been
`
`accessible to anyone on the MIT campus and anyone who had access to the OCLC
`
`database.
`
`25. During the time period from September 1993 through October 1999,
`
`the Barton Catalog allowed keyword searching for words in the thesis title, and
`
`Kellogg would have appeared in a relevant Barton Catalog search conducted on or
`
`shortly after September 28, 1993.
`
`26. After being cataloged, a document such as Kellogg will undergo a
`
`process of being labeled and then shelved at the MIT Libraries. Based on my
`
`knowledge of MIT Libraries’ current and prior practices, Kellogg would have been
`
`shelved in a relatively nominal amount of time (i.e., a few days or weeks). Thus,
`
`Kellogg was cataloged and shelved at the MIT Libraries at least before the end of
`
`1993.
`
`2 7.
`
`Once shelved, Kellogg can be borrowed by any member of the MIT
`
`community. Furtherrnore, a copy of Kellogg can be purchased from MIT by any
`
`member of the public. Indeed, the first page of Kellogg confirms that there were
`
`no restrictions placed on its publication, as it states that “[t]he author hereby grants
`
`to MIT permission to reproduce and to distribute copies ofthis thesis document in
`
`whole or in part, and to grant others the right to do so.”
`
`Canon Corrected EX. 1007 Page 7 of 219
`
`Canon Corrected Ex. 1007 Page 7 of 219
`
`
`
`Inter Partes Review
`
`United States Patent No. 7,932,923
`
`28.
`
`Further evidence of the public availability of Kellogg before October
`
`1999 is provided in Exhibit J, which is a copy of Flinchbaugh. In its Bibliography,
`
`Flinchbaugh cites to Kellogg (reference [4] on p. 151). As addressed below,
`
`Flinchbaugh was published in SPIE Volume 2962, which corresponds to the
`
`Proceedings from the 25th Annual AIPR Workshop on Emerging Applications of
`
`Computer Vision. The Workshop was held October 16-18, 1996, and the
`
`Proceedings were published by at least 1997. Thus, Kellogg was at least available
`
`to members of the public in 1997, as shown by its citation in Flinclzbaugh.
`
`29.
`
`For the avoidance of any doubt, I note that on June 23, 2001, Kellogg
`
`was also cataloged in the MIT Archive Noncirculating Collection 1,
`
`Noncirculating Collection 3, and in microfiche form in the Barker Library, as
`
`indicated in the three entries for PST8 and in the second, third, and fourth instances
`
`of field 008 on page 1 of Exhibit C. However, none of this alters the fact that
`
`Kellogg was published and accessible to the public in 1993, as indicated above.
`
`M
`
`30. As indicated in Exhibit D, Brill is part of the published Proceedings of
`
`the 1998 Image Understanding Workshop. The Workshop was held in Monterey,
`
`California during November 20-23, 1998, and the Proceedings were “APPROVED
`
`FOR PUBLIC RELEASE” with “DISTRIBUTION UNLIMITED.” Ex. D at 1. In
`
`Canon Corrected Ex. 1007 Page 8 of 219
`
`Canon Corrected Ex. 1007 Page 8 of 219
`
`
`
`Inter Partes Review
`
`United States Patent No. 7,932,923
`
`View of the above and the following, the Proceedings, including Brill, was
`
`published and accessible to the public before October 1999.
`
`31.
`
`Evidence of Brill ’s publication and availability to the public includes
`
`the hand-written receipt date of “8-13-99” at the top of page 3 of Exhibit D. This
`
`indicates it was received by the UMMU (the University of Michigan Media Union,
`
`now known as the Duderstadt Center) on August 13, 1999. In my experience as a
`
`librarian and knowledge of standard library practices, the hand-written information
`
`at the top of p. 2 of Exhibit D appears to be the catalog record information for
`
`Brill. Based on standard library practices, this reference would have been shelved
`
`shortly after being received and cataloged by UMMU.
`
`32.
`
`Further evidence of the publication and accessibility of Brill to the
`
`public can be found in Exhibit E, which is the MARC record for the Proceedings,
`
`including Brill, that was obtained from the University of Virginia Library. As
`
`shown in field 008 near the top of page 2 of Exhibit E, Brill was cataloged by the
`
`library on December 15, 1998. Based on standard library practices, this reference
`
`would have been shelved shortly after (i.e., within a few days or weeks) and been
`
`accessible to the public prior to October 1999.
`
`33.
`
`Further evidence of the publication and public availability of Brill can
`
`be found in Exhibit F, which is the MARC record for the Proceedings, including
`
`Brill, that was obtained from North Carolina State University. As shown in field
`
`9
`
`Canon Corrected Ex. 1007 Page 9 of 219
`
`Canon Corrected Ex. 1007 Page 9 of 219
`
`
`
`Inter Partes Review
`
`United States Patent No. 7,932,923
`
`008 on page 1 of Exhibit F, Brill was cataloged by the library on December 15,
`
`1998. Based on standard library practices, this reference would have been shelved
`
`shortly after (i.e., within a few days or weeks) and been accessible to the public
`
`prior to October 1999.
`
`Dimitrova
`
`34. As indicated in Exhibit G, Dimitrova was published in a special issue
`
`of the ACM Transactions on Information Systems Journal. Ex. G at 1. In view of
`
`the above and the following, the ACM Journal, including Dimitrova, was
`
`published and accessible to the public before October 1999.
`
`35.
`
`Evidence of Dimitrova ’s publication and availability to the public
`
`includes the “November 17, 1995” date stamp on page 1 of Exhibit G. This
`
`confirms that Dimitrova was received by the University of California Los Angeles
`
`Science & Engineering Library on November 17, 1995. Based on standard library
`
`practices, this reference would have been shelved shortly after (i.e., within a few
`
`days or weeks) and accessible to the public before October 1999.
`
`36.
`
`Further evidence of the publication and accessibility of Dimitrova to
`
`the public is found in Exhibit H, which is a copy of Dimitrova obtained from the
`
`Library of Congress. Page 3 of Exhibit H bears the date stamp of “November 21,
`
`1995” from the Library of Congress. This confirms that Dimitrova was received
`
`by Library of Congress on November 21, 1995. Based on standard library
`
`10
`
`Canon Corrected Ex. 1007 Page 10 of 219
`
`Canon Corrected Ex. 1007 Page 10 of 219
`
`
`
`Inter Partes Review
`
`United States Patent No. 7,932,923
`
`practices, this reference would have been shelved shortly after (i.e., within a few
`
`days or weeks) and accessible to the public before October 1999.
`
`37.
`
`Further evidence of the publication and accessibility of Dimitrova to
`
`the public can be found in Exhibit 1, which is the MARC record of the ACM
`
`Journal, which includes Dimitrova, obtained from the MIT Libraries. As shown in
`
`field 008 near the top of page 1 of Exhibit I, the MIT Libraries began receiving the
`
`ACM Journal in September 1989.
`
`38.
`
`Field 362 (shown as “3620”) on page 1 of Exhibit I indicates that the
`
`MIT Libraries has issues dating back to Volume 7 of the journal, which as noted
`
`above was published in September 1989. There is no end—date in field 362,
`
`indicating that the MIT Libraries has an ongoing subscription, which would have
`
`included receipt of the issue that contained Dimitrova in 1995. Based on standard
`
`library practices, the issue containing Dimitrova would have been shelved shortly
`
`after cataloging (i.e., within a few days or weeks) and accessible to the public
`
`before October 1999.
`
`39.
`
`For the avoidance of any doubt, I note that it appears that on June 23,
`
`2001, online access to Dimitrova was provided to certain MIT—associated
`
`individuals, as indicated by the field 008 on the last line of page 1, field 8528, and
`
`the URL entry at the top of page 2 of Exhibit 1. Also on June 23, 2001, the ACM
`
`Journal, including Dimitrova, was archived at the MIT Library Storage Annex
`
`11
`
`Canon Corrected Ex. 1007 Page 11 of 219
`
`Canon Corrected Ex. 1007 Page 11 of 219
`
`
`
`Inter Partes Review
`
`United States Patent No. 7,932,923
`
`(“LSA”), as indicated by the 008 field and subsequent 85271 entry on page 2 of
`
`Exhibit 1. However, none of this alters the fact that Dimitrova was published and
`
`accessible to the public years before October 1999, as indicated above.
`
`F[inchbaugh
`
`40. As indicated in Exhibit J, Flinchbaugh was published in the
`
`Proceedings of the 25th AIPR Workshop: Emerging Applications of Computer
`
`Vision, SPIE Vol. 2962. The Workshop was held in Washington, DC. during
`
`October 16-18, 1996, and the Proceedings was published by SPIE (The
`
`International Society for Optical Engineering). EX. J at 1. In View of the above
`
`and the following, Flinchbaugh was published and accessible to the public before
`
`October 1999.
`
`41.
`
`Page 2 of Exhibit J shows a copyright date of 1997. The edition of the
`
`SPIE Proceedings that was published with Flinchbaugh is Volume 2962, and it
`
`was “Printed in the United States of America.” Ex. J at 2.
`
`42. Although the copyright date of Flinchbaugh is listed as 1997, it
`
`appears that Flinchbaugh was actually published before that, in 1996. First, as
`
`noted above, the Workshop was held in Washington, DC. during October 16-18,
`
`1996. Second, a copy of Flinchbaugh was received and cataloged by the Library
`
`of Congress in November 1996. See Ex. K at 1. Exhibit K is the MARC record
`
`for the SPIE Proceedings, including Flinchbaugh, that was obtained from the
`
`12
`
`Canon Corrected Ex. 1007 Page 12 of 219
`
`Canon Corrected Ex. 1007 Page 12 of 219
`
`
`
`Inter Partes Review
`
`United States Patent No. 7,932,923
`
`Library of Congress. As shown in field 008 near the top of page 2 of Exhibit K,
`
`Flinchbaugh was cataloged by the library on November 21, 1996. Based on
`
`standard library practices, this reference would have been shelved shortly after it
`
`was cataloged (i.e., within a few days or weeks). Collectively, Exhibits J and K
`
`show that Flinchbaugh was published and accessible to the public years before
`
`October 1999.
`
`43.
`
`Further evidence of the publication and public availability of
`
`Flinchbaugh can be found in Exhibit L, which is the MARC record for the SPIE
`
`Proceedings, including Flinchbaugh, that was obtained from the MIT Libraries.
`
`As shown in field 008 on page 1 of Exhibit L, Flinchbaugh was cataloged by the
`
`library on March 10, 1997. Based on standard library practices and my
`
`understanding of the practices of the MIT Libraries, this reference would have
`
`been shelved shortly after it was cataloged (i.e., within a few days or weeks) and
`
`accessible to the public before October 1999.
`
`44.
`
`For the avoidance of any doubt, I note that on April 8, 2011, online
`
`access to Flinchbaugh was provided to certain MIT-associated individuals, as
`
`indicated by the fields 008 and 8528 and the URL entry at the top of page 2 of
`
`Exhibit L. Also, on June 23, 2001, the SPIE Proceedings, including Flinchbaugh,
`
`was archived at the MIT Library Storage Annex (“LSA”), as indicated by the
`
`second 008 field and subsequent 8520 entry on page 2 of Exhibit L. However,
`
`13
`
`Canon Corrected Ex. 1007 Page 13 of 219
`
`Canon Corrected Ex. 1007 Page 13 of 219
`
`
`
`Inter Partes Review
`
`United States Patent No. 7,932,923
`
`none of this alters the fact that Flinchbaugh was published and accessible to the
`
`public years before October 1999, as indicated above.
`
`I have been warned and understand that willfiil false statements and the like
`
`are punishable by fine or imprisonment, or both (18 U.S.C. § 1001). I declare that
`
`all statements made in this declaration of my oWn knowledge are true and all
`
`statements made on information and belief are believed to be true.
`
`1
`
`I declare under penalty of perjury that the foregoing is true and correct.
`
`Executed on July 9, 2019 in Washington, DC.
`
`gm
`
`Emily R. Florio
`
`14
`
`Canon Corrected EX. 1007 Page 14 of 219
`
`Canon Corrected Ex. 1007 Page 14 of 219
`
`
`
`Visual Memory
`
`by
`
`Christopher James Kellogg
`
`Submitted to the Department of Electrical Engineering and Computer Science
`in partial fulfillment of the requirements for the degrees of
`
`Bachelor of Science
`and
`
`Master of Science in Computer Science
`
`at the
`
`MASSACHUSETTS INSTITUTE OF TECHNOLOGY
`
`May 1993
`
`© Christopher James Kellogg, MCMXCIII. All rights reserved.
`
`The author hereby grants to MIT permission to reproduce and to distribute copies
`of this thesis document in whole or in part, and to grant others the right to do so.
`
`Signaturelnredacted“
`Author .................................................7mvw.
`.. ”0......
`Department of Electrical Engineerng
`cience
`,1993
`Signatureomredacted
`
`Certified by ..................................................................
`Alex P. Pentland
`
`Associate Professor of Media Arts and Sciences
`
`_
`_ Thesis Supervisor
`Signature redacted
`Certified by ............................................................ v., w.
`Bruce E. Flinchbaugh
`h a
`ruments
`
`Manager of ImageyUnderstarWr
`
`“8'
`Acceptedby...........
`...g/../....
`
`
`Chair
`
`/
`
`Supervisor
`
`'V .....................
`WV Campbell L. Searle
`artmental Committee on Graduate Students
`
`MASSACHUSETTS iNSTlTUTE
`0F TFGHNULOGY
`
`'JUL 09 1993
`
`LiBRAFFfi. A Page 1 of 92
`ARCHIVES
`Canon Corrected Ex. 1007 Page 15 of 219
`Canon Corrected EX. 1007 Page 15 of 219
`
`Ex. A Page 1 of 92
`
`
`
`Visual Memory
`
`by
`
`Christopher James Kellogg
`
`Submitted to the Department of Electrical Engineering and Computer Science
`on April 21, 1993, in partial fulfillment of the
`requirements for the degrees of
`Bachelor of Science
`
`and
`
`Master of Science in Computer Science
`
`Abstract
`
`Visual memory supports computer vision applications by efficiently storing and re-
`trieving spatiotemporal information. It is a unique combination of databases, spatial
`representation and indexing, and temporal representation and indexing. This the-
`sis designs a visual memory architecture that meets the requirements of a number
`of computer vision applications.
`It also presents an implementation of part of this
`design in support of a scene monitoring prototype.
`
`Thesis Supervisor: Alex P. Pentland
`Title: Associate Professor of Media Arts and Sciences
`
`Thesis Supervisor: Bruce E. Flinchbaugh
`Title: Manager of Image Understanding Branch at Texas Instruments
`
`EX. A Page 2 of 92
`
`Canon Corrected Ex. 1007 Page 16 of 219
`Canon Corrected EX. 1007 Page 16 of 219
`
`Ex. A Page 2 of 92
`
`
`
`Acknowledgements
`
`My primary thanks goes to my two thesis supervisors, Bruce Flinchbaugh at Texas
`
`Instruments and Sandy Pentland at MIT. Bruce pointed me to the visual memory
`
`project that he was starting and guided my research at Texas Instruments. Sandy
`
`provided useful feedback throughout the research stage. They were both very helpful
`
`in critiquing the thesis document.
`
`I’d also like to thank the other people at Texas Instruments who helped me with
`
`this project. Steve Ford and Tom Bannon were especially helpful in developing the
`
`visual memory design. In addition, I don’t think I would have survived the bugs in
`
`PC++ without Steve’s expertise. Tom Bannon and Tom O’Donnell provided a nice
`
`tracking system with which to test the visual memory prototype.
`
`Finally, I’d like to thank my family, Fred, Jeannette, and Mark Kellogg, my fiancee
`
`Christine Bailey, and my brothers at Phi Kappa Sigma for their support throughout
`
`my MIT career.
`
`EX. A Page 3 of 92
`
`Canon Corrected Ex. 1007 Page 17 of 219
`Canon Corrected EX. 1007 Page 17 of 219
`
`Ex. A Page 3 of 92
`
`
`
`Contents
`
`1
`
`Introduction
`
`1.1 Needs for Visual Memory .........................
`
`9
`
`9
`
`1.2 Goals ....................................
`
`10
`
`2 Background
`
`2.1 Database Research ............................
`
`2.1.1 DARPA Open OODB ......................
`
`2.1.2
`
`POSTGRES ............................
`
`2.2 Spatial Research .............................
`
`2.2.1 CODGER .............................
`
`2.2.2 Core Knowledge System .....................
`
`2.2.3
`
`ISR .................................
`
`2.2.4
`
`Image Understanding Environments ...............
`
`2.2.5
`
`PROBE ..............................
`
`2.2.6
`
`Spatial Indices ..........................
`
`2.3 Temporal Research ............................
`
`2.3.1 TQuel ...............................
`
`2.3.2 Temporal Sequences .......................
`
`2.3.3 Temporal Sets ...........................
`
`2.3.4 Relative Time ...........................
`
`2.3.5 Temporal Indices .........................
`
`11
`
`11
`
`11
`
`12
`
`13
`
`13
`
`13
`
`14
`
`14
`
`14
`
`15
`
`15
`
`15
`
`16
`
`16
`
`17
`
`17
`
`4
`
`EX. A Page 4 of 92
`
`Canon Corrected Ex. 1007 Page 18 of 219
`Canon Corrected EX. 1007 Page 18 of 219
`
`Ex. A Page 4 of 92
`
`
`
`3 Design
`
`3.1 Requirements and Considerations ....................
`
`3.1.1 Database Considerations .....................
`
`3.1.2
`
`Spatial and Temporal Considerations ..............
`
`3.1.3
`
`Performance Considerations ...................
`
`3.2 Design Overview .............................
`
`3.3 Spatial Representations ..........................
`
`3.3.1 Core Spatial Classes .......................
`
`3.3.2 Relative Spatial Specification ..................
`
`3.3.3 Uncertain Spatial Specification .................
`
`3.4 Temporal Representations ........................
`
`3.4.1 Core Temporal Classes ......................
`
`3.4.2 Relative Temporal Specification .................
`
`3.4.3 Uncertain Temporal Specification ................
`
`3.5 Spatiotemporal Representations .....................
`
`3.6 Object Storage ..............................
`
`3.6.1
`
`Identity ..............................
`
`3.6.2
`
`Storage Mechanism ........................
`
`3.6.3 Time ................................
`
`3.7 Queries ...................................
`
`3.7.1 Query Mechanism .........................
`
`3.7.2
`
`Spatial Queries ..........................
`
`3.7.3 Temporal Queries .........................
`
`3.7.4
`
`Spatiotemporal Queries ......................
`
`3.8
`
`Indices ...................................
`
`3.8.1 Mechanism ............................
`
`3.8.2
`
`Spatial Indices ..........................
`
`3.8.3 Temporal Indices .........................
`
`3.8.4
`
`Spatiotemporal Indices ......................
`
`18
`
`19
`
`19
`
`20
`
`20
`
`22
`
`24
`
`24
`
`29
`
`31
`
`36
`
`36
`
`40
`
`41
`
`45
`
`50
`
`50
`
`51
`
`52
`
`53
`
`53
`
`54
`
`57
`
`59
`
`64
`
`64
`
`65
`
`66
`
`66
`
`EX. A Page 5 of 92
`
`Canon Corrected Ex. 1007 Page 19 of 219
`Canon Corrected EX. 1007 Page 19 of 219
`
`Ex. A Page 5 of 92
`
`
`
`4
`
`Implementation
`
`4.1 Database ..................................
`
`4.2 Spatiotemporal Representations .....................
`
`4.3
`
`Indices ...................................
`
`4.3.1 Mechanism ............................
`
`4.3.2
`
`Spatial Indices ..........................
`
`4.3.3 Temporal Indices .........................
`
`4.4 Queries ...................................
`
`4.5
`
`Input ....................................
`
`4.6 Graphical Query Interface ........................
`
`5 Performance
`
`5.1 Spatiotemporal Object Storage and Retrieval ..............
`
`5.2
`
`Index Comparison .............................
`
`6 Conclusion
`
`68
`
`68
`
`71
`
`71
`
`71
`
`72
`
`75
`
`77
`
`77
`
`78
`
`81
`
`82
`
`83
`
`88
`
`6
`
`EX. A Page 6 of 92
`
`Canon Corrected Ex. 1007 Page 20 of 219
`Canon Corrected EX. 1007 Page 20 of 219
`
`Ex. A Page 6 of 92
`
`
`
`List of Figures
`
`3-1 Spatial objects ..............................
`
`3-2 Discrete point set .............................
`
`3-3 Abstract point set .............................
`
`3-4 Coordinate systems ............................
`
`3-5 Relative spatial objects ..........................
`
`3-6 Breaking a relative spatial specification, part 1 .............
`
`3-7 Breaking a relative spatial specification, part 2 .............
`
`3-8 Uncertain edges ..............................
`
`3—9 Uncertain location ............................
`
`3-10 Conflicting information ..........................
`
`3-11 Temporal element .............................
`
`3-12 Overlapping temporal elements .....................
`
`3-13 Temporal resolution in favor of version A ................
`
`3-14 Temporal resolution in favor of version B ................
`
`3—15 Relative temporal specification ......................
`
`3-16 Probabilistic temporal interval ......................
`
`3-17 Overlapping probabilistic temporal intervals ..............
`
`3-18 Probabilistic conjunction by minimization ...............
`
`3—19 Probabilistic disjunction by maximization ...............
`
`3—20 Discrete spatiotemporal information ...................
`
`3-21 Interpolated spatiotemporal state ....................
`
`25
`
`26
`
`27
`
`28
`
`30
`
`32
`
`32
`
`33
`
`34
`
`35
`
`38
`
`38
`
`39
`
`39
`
`40
`
`42
`
`43
`
`43
`
`44
`
`46
`
`46
`
`3-22 Point set trajectory ............................ 47
`
`3-23 Coordinate system trajectory ......................
`
`48
`
`EX. A Page 7 of 92
`
`Canon Corrected Ex. 1007 Page 21 of 219
`Canon Corrected EX. 1007 Page 21 of 219
`
`Ex. A Page 7 of 92
`
`
`
`3—24 Spatial queries ...............................
`
`3-25 Temporal queries .............................
`
`3-26 States of a spatiotemporal object ....................
`
`3-27 Joint spatial and temporal queries ....................
`
`3-28 Spatiotemporal queries ..........................
`
`4—1 Scene monitoring prototype .......................
`
`4-2 Fixed grid spatial index .........................
`
`4-3 Segmented space for bucket PR quadtree ................
`
`4-4 Data structure for bucket PR quadtree .................
`
`4-5 Temporal segment tree ..........................
`
`4-6 Temporal B+ tree .............................
`
`4—7 Graphical query interface viewing region ................
`
`4-8 Specification of query times and classes .................
`
`4—9 Graphical query results ..........................
`
`5—1 Spatiotemporal update performance ...................
`
`5-2 Spatial update performance .......................
`
`5-3 Temporal update performance ......................
`
`5-4 Spatial query performance ........................
`
`5-5 Temporal query performance .......................
`
`55
`
`58
`
`60
`
`61
`
`62
`
`69
`
`73
`
`74
`
`74
`
`76
`
`76
`
`78
`
`79
`
`80
`
`82
`
`84
`
`85
`
`86
`
`87
`
`EX. A Page 8 of 92
`
`Canon Corrected Ex. 1007 Page 22 of 219
`Canon Corrected EX. 1007 Page 22 of 219
`
`Ex. A Page 8 of 92
`
`
`
`Chapter 1
`
`Introduction
`
`Visual memory supports computer vision applications by efficiently storing and re-
`
`trieving spatiotemporal information. It is a unique combination of databases, spatial
`
`representation and indexing, and temporal representation and indexing. Visual mem~
`
`ory provides representational flexibility and high-performance information access to
`
`meet the requirements of a variety of computer vision applications.
`
`1.1 Needs for Visual Memory
`
`Applications use spatiotemporal data in many different ways and place many different
`
`demands on a visual memory. Studying possible uses helps to clarify the concept of
`
`a visual memory and to identify the functionality it provides.
`
`Visual memory could serve as the repository for static information, such as ob—
`
`ject descriptions, maps, and environment models, that applications reference during
`
`execution. For example, a vehicle navigator could store maps and images to help it
`
`later recognize its location. A large amount of such information could be established
`
`prior to application execution, and the visual memory would subsequently provide an
`
`application with efficient access to desired pieces of information.
`
`An application could store dynamic information in the visual memory. For ex-
`
`ample, a vehicle navigator’s input systems could maintain in the visual memory a
`
`description of the vehicle’s local environment, updating it as the vehicle moved. The
`
`EX. A Page 9 of 92
`
`Canon Corrected Ex. 1007 Page 23 of 219
`Canon Corrected EX. 1007 Page 23 of 219
`
`Ex. A Page 9 of 92
`
`
`
`visual memory could provide the navigator’s planning processes with information
`
`about the vehicle’s latest state and could analyze its progress to help determine a
`
`course of action. The high performance of the visual memory allows it to handle the
`
`frequent updates and queries needed by such dynamic, real-time systems.
`
`Visual memory could manipulate spatiotemporal information about objects and
`
`collections of objects too large to fit into volatile memory. For example, a computer-
`
`aided design and modeling system could use the visual memory in building up a large
`
`design layout and simulating its execution over time; a photo interpretation system
`
`could similarly construct in the visual memory a complex representation of a scene.
`
`The visual memory would retrieve into main memory only a manageable part of a
`
`large representation at a time.
`
`Visual memory could act as the interface between inputs and applications in a
`
`computer vision system. For example, computer vision algorithms for a security
`
`system could analyze data provided by various cameras and store information in the
`
`visual memory. Applications could then retrieve this data to track objects, watch for
`
`suspicious events, and respond to user queries. The visual memory would coordinate
`
`the information from its inputs and eliminate the need for full connectivity between
`
`inputs and applications.
`
`Finally, visual memory could serve as a means for data transfer. A computer
`
`vision application could store spatiotemporal information in the visual memory for
`
`other applications to retrieve at any time in the future. To run comparative studies,
`
`different algorithms could use common data stored in the visual memory.
Accessing this document will incur an additional charge of $.
After purchase, you can access this document again without charge.
Accept $ Charge
Still Working On It
This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.
Give it another minute or two to complete, and then try the refresh button.
A few More Minutes ... Still Working
It can take up to 5 minutes for us to download a document if the court servers are running slowly.
Thank you for your continued patience.
This document could not be displayed.
We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.
Your account does not support viewing this document.
You need a Paid Account to view this document. Click here to change your account type.
Your account does not support viewing this document.
Set your membership
status to view this document.
With a Docket Alarm membership, you'll
get a whole lot more, including:
- Up-to-date information for this case.
- Email alerts whenever there is an update.
- Full text search for other cases.
- Get email alerts whenever a new case matches your search.
One Moment Please
The filing “” is large (MB) and is being downloaded.
Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.
Your document is on its way!
If you do not receive the document in five minutes, contact support at support@docketalarm.com.
Sealed Document
We are unable to display this document, it may be under a court ordered seal.
If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.
Access Government Site
