Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`
`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`
`
`
`NIANTIC, INC.,
`Petitioner
`
`v.
`
`NANT HOLDINGS IP, LLC, and NANTWORKS, LLC,
`Patent Owners
`
`
`
`U.S. Patent No. 10,664,518 B2
`Issue Date: May 26, 2020
`
`
`
`Title: Wide Area Augmented Reality Location-Based Services
`
`PETITION FOR INTER PARTES REVIEW
`OF U.S. PATENT NO. 10,664,518 B2
`
`
`
`
`

`

`Table of Contents
`
`
`Page
`
`
`I. Mandatory Notices under §42.8(A)(1) ........................................................... 1 
`A.
`Real Party-In-Interest under §42.8(b)(1) .............................................. 1 
`B.
`Related Matters under §42.8(b)(2) ....................................................... 1 
`C.
`Lead and Back-Up Counsel under §42.8(b)(3) .................................... 1 
`D.
`Service Information .............................................................................. 3 
`E.
`Power of Attorney ................................................................................ 3 
`Fee Payment .................................................................................................... 4 
`II.
`III. Requirements under §§ 42.104 and 42.108 and Considerations under
`§§ 314(a) and 325(d) ...................................................................................... 4 
`A. Grounds for Standing under 37 C.F.R. § 42.104(a) ............................. 4 
`B.
`Identification of Challenge Under 37 C.F.R. § 42.104(b) and
`Statement of Precise Relief Requested ................................................ 4 
`Sections 314 and 325(d) ....................................................................... 5 
`C.
`IV. The ’518 Patent ............................................................................................... 5 
`V.
`Level of Ordinary Skill in the Art .................................................................. 7 
`VI. Claim Construction ......................................................................................... 7 
`VII. The Challenged Claims are Unpatentable ...................................................... 8 
`A.
`Summary and Date Qualification of the Prior Art ............................... 8 
`B. Ground 1: Obviousness of Claims 1-9, 11-14, 18-20, 23-32,
`and 34-36 Over Altman and Langseth ............................................... 11 
`C. Ground 2: Obviousness of Claims 1-9, 11-14, 18-20, 23-32,
`and 34-36 Over Altman, Langseth, and CaveLie ............................... 43 
`D. Ground 3: Obviousness of Claims 1-9, 11-20, 23-32, and 34-36
`Over Altman, Langseth, CaveLie, and Gelfand ................................. 51 
`Ground 4: Single Reference Obviousness of Claims 1-8, 11-14,
`18-20, 23-24, 26-32, and 34-36 Over Sterkel .................................... 55 
`VIII. Conclusion .................................................................................................... 69 
`
`
`E.
`
` 
`
`
`
`-i-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`List of Exhibits
`
`
`
`
`Description of Document
`Ex. No.
`1001 U.S. Patent No. 10,664,518 (“’518 Patent” or “’518”)
`1002 Declaration of Dr. Michael Zyda (“Zyda”)
`1003 U.S. Patent Application Publication No. 2008/0132251 (“Altman”)
`1004 U.S. Patent Application Publication No. 2013/0124563 (“CaveLie”)
`1005 U.S. Patent Application Publication No. 2008/0268876 (“Gelfand”)
`1006 U.S. Patent Application Publication No. 2013/0178257 (“Langseth”)
`1007 U.S. Patent No. 8,762,047 (“Sterkel”)
`1008
`
`Excerpts from Oliver Bimber and Ramesh Raskar, Chapter 1: A Brief
`Introduction to Augmented Reality, in Spatial Augmented Reality:
`Merging Real and Virtual Worlds, CRC Press (2005)
`
`1009
`
`Paul Milgram and Fumio Kishino, A Taxonomy of Mixed Reality
`Visual Displays, IEICE Transactions on Info. Sys., E77-D (12):1321-
`29 (1994)
`
`1010
`
`Jean-Marie Normand et al., A new typology of augmented reality
`applications, AH ’12 Proceedings of the 3rd Augmented Human Int’l
`Conference (Mar. 2012)
`1011 U.S. Patent No. 3,050,870 (“Heilig”)
`1012
`
`Ivan E. Sutherland, A head-mounted three dimensional display,
`AFIPS ’68 (Fall, Part I): Proceedings of the December 9-11, 1968
`Fall Joint Comput. Conf., Part I, pp. 757-64 (1968)
`1013 Grégory Maubon, A little bit of history from 2006: Nokia MARA
`project, https://www.augmented-reality.fr/2009/03/un-petit-peu-
`dhistoire-de-2006-projet-mara-de-nokia/
`1014 U.S. Patent No. 6,553,310 (“Lopke”)
`
` 
`
`
`
`‐i‐ 
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`List of Exhibits
`
`
`
`Description of Document
`Ex. No.
`1015 U.S. Patent Application Publication No. 2002/0044152 (“Abbott”)
`1016
`
`Excerpts from Lester Madden, Professional Augmented Reality
`Browsers for Smartphones, Programming for junaio, Layar, and
`Wikitude (2011)
`
`1017
`
`1018
`
`1019
`
`Jonathan Raper et al., Applications of location-based services: a
`selected review, J. Location Based Servs., 1(2):89-111 (2007)
`
`Paul G. Savage, Blazing Gyros: The Evolution of Strapdown Inertial
`Navigation Technology for Aircraft, J. of Guidance, Control, and
`Dynamics, 36(3):637-55 (2013)
`
`Jeong Won Kim et al., A Step, Stride and Heading Determination for
`the Pedestrian Navigation System, J. Global Positioning Sys., 3(1-
`2):273-79 (2004)
`
`1020
`
`Press Release, Apple Reinvents the Phone with iPhone (Jan. 9, 2007),
`https://www.apple.com/newsroom/2007/01/09Apple-Reinvents-the-
`Phone-with-iPhone/
`1021 U.S. Patent No. 8,745,494 (“Spivak”)
`1022 U.S. Patent No. 5,848,373 (“DeLorme”)
`1023 Michael R. Macedonia et al., Exploiting Reality with Multicast
`Groups, Proceedings of the Virtual Reality Annual Int’l Symposium
`(VRAIS ’95) (1995)
`1024 U.S. Patent No. 6,882,933 (“Kondou”)
`1025
`Reserved
`1026 U.S. Patent No. 9,317,133 (“Korah”)
`1027 Duy-Nguyen Ta et al., SURFTrac: Efficient Tracking and Continuous
`Object Recognition using Local Feature Descriptors, 2009 IEEE
`
` 
`
`
`
`‐ii‐ 
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`List of Exhibits
`
`
`
`Ex. No.
`
`1028
`
`1029
`
`Description of Document
`Conference on Computer Vision and Pattern Recognition, pp. 2937-
`44 (2009)
`
`Carsten Magerkurth et al., Pervasive Games: Bringing Computer
`Entertainment Back to the Real World, ACM Comput. in Entm’t,
`3(3):1-19 (July 2005)
`
`Bruce Thomas et al., ARQuake: An Outdoor/Indoor Augmented
`Reality First Person Application, IEEE Int’l Symposium on Wearable
`Comput., pp. 139-46 (2000)
`
`1030
`
`Bruce Thomas et al., First Person Indoor/Outdoor Augmented Reality
`Application: ARQuake, Personal and Ubiquitous Computing, 6:75-86
`(2002)
`1031 U.S. Patent Applicaton Publication No. 2004/0164897 (“Treadwell”)
`1032 Michael Zyda, From Visual Simulation to Virtual Reality to Games,
`IEEE 38(9):25-32 (Sept. 2005)
`1033 Michael Zyda, Creating a Science of Games, Commc’ns of the ACM
`50(7):27-29 (July 2007)
`1034 U.S. Patent No. 8,519,844 (“Richey”)
`1035
`
`International Patent Application Publication No. WO 2013/023705
`(“Hoffman”)
`1036 U.S. Patent Application Publication No. 2013/0073988 (“Groten”)
`1037 U.S. Patent Application Publication No. 2013/0124326 (“Huang”)
`1038
`
`Excerpts from NantWorks’ Disclosure of Asserted Claims and
`Infringement Contentions and Ex. 3, dated 1/28/2021, served in
`Nantworks, LLC v. Niantic, Inc., Case No. 3:20-cv-06262-LB (N.D.
`Cal.)
`
` 
`
`
`
`‐iii‐ 
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`List of Exhibits
`
`Description of Document
`
`
`
`Ex. No.
`1039
`
`Excerpts from Microsoft Corporation, Microsoft Computer
`Dictionary (2002)
`1040 David Rumsey Historical Map Collection: “San Francisco street map”
`(1953), https://www.davidrumsey.com/luna/servlet/s/or3ezx
`
`1041
`
`1042
`
`Florida Department of Transportation, Official Transportation Map
`(2010),
`https://www.fdot.gov/docs/default-source/geospatial/past_statemap/m
`aps/FLStatemap2010.pdf
`
`Patent L.R. 4-3 Joint Claim Construction and Prehearing Statement,
`ECF No. 61, dated 5/18/2021, filed in Nantworks, LLC v. Niantic,
`Inc., Case No. 3:20-cv-06262-LB (N.D. Cal.)
`
`1043
`
`Peter Krogh, GPS, American Society of Media Photographers (Mar.
`22, 2010)
`1044 Ads of the World, Tigo: Smartphone, 2,
`https://www.adsoftheworld.com/media/print/tigo_smartphone_2
`
`1045
`1046
`
`1047
`
`Reserved
`
`Proof of Service of Summons and Complaint, ECF No. 12, dated
`Sept. 8, 2020, filed in Nantworks, LLC v. Niantic, Inc., Case No.
`3:20-cv-06262-LB (N.D. Cal.)
`
`Third Amended Complaint for Patent Infringement, ECF No. 50,
`dated Feb. 23, 2021, filed in Nantworks, LLC v. Niantic, Inc., Case
`No. 3:20-cv-06262-LB (N.D. Cal.)
`
` 
`
`
`
`‐iv‐ 
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`This is a petition for inter partes review of claims 1-9, 11-20, 23-32, and 34-
`
`36 of U.S. Patent No. 10,664,518 (Ex. 1001) (“’518 Patent” or “’518”).
`
`I. MANDATORY NOTICES UNDER §42.8(A)(1)
`A. Real Party-In-Interest under §42.8(b)(1)
`Niantic, Inc. (“Niantic” or “Petitioner”) is the real party-in-interest to this
`
`Petition.
`
`B. Related Matters under §42.8(b)(2)
`The ’518 Patent is the subject of pending litigation involving Petitioner:
`
`NantWorks, LLC, and Nant Holdings IP, LLC v. Niantic, Inc., Case No. 3:20-cv-
`
`06262-LB (N.D. Cal.). No trial date has been set.
`
`Petitioner was first served on September 8, 2020. (Ex. 1046.) The Third
`
`Amended Complaint in that action alleges that Niantic infringes the ’518 Patent.
`
`(Ex. 1047, ¶¶1, 35-45, 87-109.)
`
`C. Lead and Back-Up Counsel under §42.8(b)(3)
`Petitioner provides the following designation of counsel.
`
`LEAD COUNSEL
`
`BACK-UP COUNSEL
`
`Heidi L. Keefe (Reg. No. 40,673)
`hkeefe@cooley.com
`
`COOLEY LLP
`ATTN: Patent Group
`1299 Pennsylvania Ave. NW, Suite 700
`Washington, DC 20004
`Tel: (650) 843-5001
` 
`
`
`
`Lauren J. Krickl (Reg. No. 70,261)
`lkrickl@cooley.com
`
`COOLEY LLP
`ATTN: Patent Group
`1299 Pennsylvania Avenue NW
`Suite 700
`Washington D.C. 20004
`
`-1-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`
`LEAD COUNSEL
`
`BACK-UP COUNSEL
`
`Fax: (650) 849-7400
`
`
`
`
`
`
`
`
`
`
`
` 
`
`Tel: (650) 843-5065
`Fax: (650) 849-7400
`Dena Chen (Admission pro hac vice to be
`requested)
`dchen@cooley.com
`
`COOLEY LLP
`ATTN: Patent Group
`1299 Pennsylvania Ave. NW, Suite 700
`Washington D.C. 20004
`Tel: (650) 843-5135
`Fax: (650) 849-7400
`Alexandra D. Leeper (Admission pro hac
`vice to be requested)
`aleeper@cooley.com
`
`COOLEY LLP
`ATTN: Patent Group
`1299 Pennsylvania Avenue NW
`Suite 700
`Washington D.C. 20004
`Tel: (650) 843-5376
`Fax: (650) 849-7400
`Patrick W. Lauppe (Admission pro hac
`vice to be requested)
`plauppe@cooley.com
`
`COOLEY LLP
`ATTN: Patent Group
`1299 Pennsylvania Avenue NW
`Suite 700
`Washington D.C. 20004
`Tel: (650) 843-5833
`Fax: (650) 849-7400
`
`-2-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`D.
`Service Information
`This Petition is being served by Federal Express to the attorneys of record for
`
`the ’518 patent, Mauriel Kapouytian Woods LLP, 15 W. 26th Street, Floor 7, New
`
`York, NY 10010. This Petition is also being served on litigation counsel for Patent
`
`Owners1 identified in the Certificate of Service. Petitioner consents to electronic
`
`service at the addresses provided above for lead and back-up counsel.
`
`E.
`Power of Attorney
`Filed concurrently per 37 C.F.R. § 42.10(b).
`
`
`1 NantWorks asserts that Nant Holdings IP, LLC “solely owns the Asserted Patents,”
`
`with NantWorks, LLC as “the exclusive licensee of the Asserted Patents.” (Ex.
`
`1047, ¶19.) The assignment records for the ’518 Patent however show assignments
`
`to both Nant Holdings IP, LLC and NantWorks, LLC. Accordingly, Petitioner has
`
`named both Nant Holdings IP and NantWorks as Patent Owners. To the extent any
`
`ambiguity exists, Petitioner is serving this Petition on counsel of record for Nant
`
`Holdings IP, LLC and NantWorks, LLC (collectively, “NantWorks”). Petitioner
`
`reserves the right to correct any clerical error.
`
` 
`
`
`
`
`
`-3-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`II.
`FEE PAYMENT
`This Petition requests review of claims 1-9, 11-20, 23-32, 34-36 of the ’518
`
`patent. A payment of $55,000 is submitted herewith. See 37 C.F.R. § 42.15(a). This
`
`Petition meets the fee requirements of 35 U.S.C. §312(a)(1).
`
`III. REQUIREMENTS UNDER §§ 42.104 AND 42.108 AND CONSIDERATIONS UNDER
`§§ 314(A) AND 325(D)
`A. Grounds for Standing under 37 C.F.R. § 42.104(a)
`Petitioner certifies that the ’518 patent is available for IPR and Petitioner is
`
`not barred or otherwise estopped.
`
`B.
`
`Identification of Challenge Under 37 C.F.R. § 42.104(b) and
`Statement of Precise Relief Requested
`Petitioner requests institution of IPR based on:
`
`Ground
`
`Claims
`
`Basis for Challenge under §103(a)
`
`1
`
`2
`
`3
`
`4
`
`1-9, 11-14,
`18-20, 23-32,
`34-36
`1-9, 11-14,
`18-20, 23-32,
`34-36
`1-9, 11-20,
`23-32, 34-36
`1-8, 11-14,
`18-20, 23-24,
`26-32, 34-36
`
`Altman (Ex. 1003) and Langseth (Ex. 1006)
`
`Altman (Ex. 1003), Langseth (Ex. 1006) and
`CaveLie (Ex. 1004)
`
`Altman (Ex. 1003), Langseth (Ex. 1006), CaveLie
`(Ex. 1004), and Gelfand (Ex. 1005)
`Sterkel (Ex. 1007)
`
`Submitted with this Petition is the Declaration of Dr. Michael Zyda (Ex. 1002)
`
`(“Zyda”), a qualified technical expert. (Zyda, ¶¶1-9, Ex. A.)
`
` 
`
`
`
`-4-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`C.
`Sections 314 and 325(d)
`Petitioner is aware of no prior petitions challenging the ’518 Patent. Petitioner
`
`filed this Petition shortly after the parties’ filing of their joint claim construction
`
`statement, and before NantWorks’ opening brief is due (July 2). The Board should
`
`exercise its discretion to institute this Petition under §314(a) because the litigation
`
`is in relatively early stages, no trial date has been set, and there is no substantial
`
`overlap of the issues and arguments presented in this Petition with those presented
`
`at the district court. Apple v. Fintiv, IPR2020-00019, Paper 15 (P.T.A.B. May 13,
`
`2020). For example, this Petition challenges three claims (Claims 9, 25, and 27) not
`
`asserted in district court, and presents grounds for obviousness that rely exclusively
`
`on prior art not asserted in district court.
`
`None of the prior art references relied upon in Petitioner’s Grounds were cited
`
`during prosecution of the ’518 Patent. Petitioner’s Grounds rely on new art and
`
`grounds of unpatentability different from any presented during prosecution.
`
`IV. THE ’518 PATENT
`The ’518 Patent, entitled “Wide Area Augmented Reality Location-Based
`
`Services,” issued from a series of continuation of applications, the earliest of which
`
`is U.S. Patent Application No. 14/517,728 (“the ’728 Application”), filed on October
`
`17, 2014. The ’728 Application, claims priority to U.S. Provisional Patent
`
` 
`
`
`
`-5-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`Application No. 61/892,238 which was filed on October 17, 2013. The ’518 Patent
`
`specification discloses admitted prior art. (Zyda, ¶¶61-62; ’518, 1:26-67.)
`
`The ’518 Patent specification discloses “systems and methods in which AR
`
`content is provided to one or more user devices based on at least one of location
`
`identification and object recognition.” (’518, 2:15-18.) It also discloses a content
`
`management system that fetches content from databases for a mobile device, which
`
`can be a “cell phone” or a “vehicle.” (Id., 2:23-29, 3:54-62.) Such content can
`
`include map image data for an area (id., 2:30-37), and augmented reality content
`
`including images, which can be “associated with one or more real world objects[.]”
`
`(Id., 2:38-44.) AR content objects can be used to “generate an augmented-
`
`reality…environment” by “overlaying the content on real-world imagery (preferably
`
`in real-time) … such that the user of the computing device sees a combination of the
`
`real-world imagery with the AR content seamlessly.” (Id., 7:16-23.)
`
`Further, the ’518 Patent discloses that the system fetches data to the mobile
`
`device including AR content objects associated with map tiles that are in an area of
`
`interest around the user. (Id., 3:34-62, 2:30-44, 6:3-9, 6:24-34.) “Tiles can be
`
`constructed at varying levels of fidelity and resolution[.]” (Id., 20:46-49.) Location
`
`can be determined from location sensor data, including GPS. (Id., 11:25-28.) The
`
`mobile device shows objects from a particular “view of interest,” which can be an
`
`area or a particular viewing angle. (Id., 3:10-17; 2:66-3:4.)
`
` 
`
`
`
`-6-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`V. LEVEL OF ORDINARY SKILL IN THE ART
`A person of ordinary skill in the art (“POSA”) as of October 2013 would have
`
`possessed at least a Master’s of Science in the areas of electrical engineering or
`
`computer science, with a working knowledge of augmented reality, mobile mapping,
`
`and the associated technologies; or, alternatively, a Bachelor’s of Science in
`
`computer science with at least two years of experience in the aforementioned areas
`
`(or equivalent degree or experience). (Zyda, ¶¶11-15.)
`
`VI. CLAIM CONSTRUCTION
`As of the Petition’s filing, claim construction in the underlying litigation is
`
`ongoing, and there has been no claim construction ruling.
`
`The parties agreed to construe “tessellated tiles” as “tiles fitted together to
`
`cover an area without gaps.” (Ex. 1042 at 002.) The mappings herein satisfy this
`
`construction.
`
`The parties identified additional terms with disputed constructions. (Ex. 1042
`
`at 003-08.) For purposes of the Petition, Petitioner has identified these potential
`
`disputes in the below analysis for the Board’s reference, but respectfully submits
`
`that they need not be resolved by the Board as the prior art would satisfy all proposed
`
`constructions.
`
` 
`
`
`
`-7-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`VII. THE CHALLENGED CLAIMS ARE UNPATENTABLE
`This Petition presents four Grounds of obviousness. Ground 1 relies on the
`
`combination of Altman and Langseth. Ground 2 adds CaveLie. Ground 3 further
`
`adds Gelfand. Ground 4 relies on Sterkel.
`
`A.
`Summary and Date Qualification of the Prior Art
`
`Altman [Ex. 1003]
`Altman is a U.S. patent application publication entitled “Geo-Tagged Journal
`
`System for Location-Aware Mobile Communication Devices.” Altman discloses a
`
`mobile application that shows a user a tile map surrounding the user’s location
`
`overlaid with icons (objects) indicating positions of the user’s friends, places of
`
`interest, and other location-tagged objects. Altman qualifies as prior art because it
`
`was published on June 5, 2008, before the October 17, 2013 filing date of the ’518
`
`Patent’s provisional application. Petitioner relies on Altman as the primary
`
`reference for Grounds 1-3.
`
`Altman discloses a “location-based social network manager process” that
`
`“displays a map representation of the area around the mobile communication device
`
`on a graphical user interface of the mobile communication device, and superimposes
`
`on the map the respective locations of one or more other users of mobile
`
`communication devices,” as well as other types of content, as shown below in Figure
`
`2A. (Altman, ¶¶0006, 0050.) Altman discloses that map data of a real-world
`
` 
`
`
`
`-8-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`location is stored as “map tiles that are image files of maps with varying degrees of
`
`granularity.” (Id., ¶0043.)
`
`
`
`
`Langseth [Ex. 1006]
`Langseth is a U.S. patent application publication entitled “System and
`
`Method for Interacting with Virtual Objects in Augmented Realities.” It discloses a
`
`system and method for “augmented reality interaction” by “enabl[ing] users to create
`
`and deploy virtual objects having custom visual designs and embedded content or
`
`other virtual items that can be shared with other users[.]” (Langseth, ¶0004.)
`
`Langseth teaches that virtual objects can be displayed in an augmented reality either
`
`on a “map view” as objects overlaid on a map or in a “live view” as objects overlaid
`
`on a camera image. (Id., ¶¶0043-0044, 0046, 0035, 0037.) Langseth qualifies as
`
` 
`
`
`
`-9-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`prior art because it was filed on January 6, 2012 and published on July 11, 2013,
`
`both dates before the October 17, 2013 filing date of the ’518 Patent’s provisional
`
`application.
`
`
`CaveLie [Ex. 1004]
`CaveLie is a U.S. patent application publication entitled “Controlling Pre-
`
`Fetching of Map Data Tiles Based on Selectable Parameters.” CaveLie discloses a
`
`system and method of pre-fetching map tiles from a server to a mobile device based
`
`on the device location, and displaying a map of surrounding area. CaveLie qualifies
`
`as prior art because it was published on May 16, 2013, before the October 17, 2013
`
`filing date of the ’518 Patent’s provisional application.
`
` Gelfand [Ex. 1005]
`Gelfand is a U.S. patent application publication entitled “Method, Device,
`
`Mobile Terminal, and Computer Program Product for a Point of Interest Based
`
`Scheme for Improving Mobile Visual Searching Functionalities.” Gelfand discloses
`
`a “visual searching” system that can recognize real-world points of interest by
`
`pointing a mobile device camera at a recognized location such as a restaurant or store
`
`and showing similar locations (e.g., other restaurants near the user) on a map by
`
`overlaying icons indicating those locations. Gelfand qualifies as prior art because it
`
`was published on October 30, 2008, before the October 17, 2013 filing date of the
`
`’518 Patent’s provisional application.
`
` 
`
`
`
`-10-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`
`Sterkel [Ex. 1007]
`Sterkel is a U.S. patent entitled “Location-Based In-Route Entertainment and
`
`Information.” Sterkel discloses a “moving map” system that provides users with
`
`location-relevant content, including a system for presenting information to airline
`
`passengers via icons overlaid on a tiled map display that loads map and content
`
`information as the plane flies over map regions, including cities. Sterkel qualifies
`
`as prior art because it was filed on March 5, 2012 and its application was published
`
`on September 13, 2012, both dates before the October 17, 2013 filing date of the
`
`’518 Patent’s provisional application. Petitioner relies on Sterkel as the primary
`
`reference for Ground 4.
`
`B. Ground 1: Obviousness of Claims 1-9, 11-14, 18-20, 23-32, and 34-
`36 Over Altman and Langseth
`1.
`Claim 1
`(a)
`“A device capable of rendering augmented reality
`(AR), the device comprising:” (Preamble)
`To the extent this preamble is limiting, it is satisfied by Altman and rendered
`
`obvious by Altman with Langseth. Altman discloses a device capable of rendering
`
`augmented reality (“AR”), namely, “a location-aware mobile communication
`
`device” configured to overlay virtual icons representing virtual content—like journal
`
`entries, events, points of interest, and user locations—on a map of a real-world area.
`
`(Altman, Abstract, ¶0108.) A POSA would recognize Altman’s map display as an
`
` 
`
`
`
`-11-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`AR experience and Altman’s virtual icons and virtual content displayed on the map
`
`as AR content objects. (Zyda, ¶¶93, 95-97, 104-107.) This is consistent with
`
`NantWorks’ Infringement Contentions, wherein NantWorks alleges that virtual
`
`elements depicted on a graphical map of the area around the user are “AR content
`
`objects.” (See, e.g., Ex. 1038, Ex. 3 at 023.)
`
`To the extent Patent Owners argue that Altman’s virtual icons and objects
`
`overlaid on a map of the real-world are not on their own considered AR, the
`
`combination of Altman with Langseth results in rendering AR and AR content
`
`objects. Langseth discloses “a system and method for interacting with virtual objects
`
`in augmented realities,” where the virtual objects “may include any text, pictures,
`
`graphics, audio, video, icons, games, … offers, coupons, or other content or virtual
`
`items[.]” (Langseth, ¶¶0001, 0008.)2 Langseth provides a specific example of using
`
`virtual game objects, such as a ball and goal, to enable users of the AR application
`
`to play a virtual ball game. (Id. ¶¶0043-44.) Langseth’s virtual objects are
`
`analogous to the AR content objects discloses by the ’518 patent: “a virtual object,
`
`… digital image, digital video, audio data, … promotion, advertisement,
`
`game[.]” (’518, 7:24-27; Zyda, ¶104.)
`
`
`2 Unless noted otherwise, all emphasis has been added.
`
` 
`
`
`
`-12-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`Langseth additionally discloses that AR content objects can be viewed either
`
`on a “map view” as images overlaid on a real-world street map, or in a “live view,”
`
`over a “physical reality image” from a mobile device’s camera (i.e. a picture or
`
`video). (Langseth, ¶¶0011-0012, 0032, 0035, 0036-0037, 0043-0044, Fig. 4B.)
`
`Langseth gives an example for a virtual ball, which can be superimposed on either a
`
`street-map image (map view) or a camera (“physical reality”) image (live view).
`
`It would have been obvious to a POSA to implement Altman’s application
`
`using Langseth’s wider selection of virtual objects as AR content objects. (Zyda,
`
`¶¶108-112, 98-104.) It would further have been obvious to incorporate into Altman
`
`the ability to view the AR content objects on either a map view or a live view, as
`
`taught by Langseth. (Zyda, ¶¶108-112, 99-103.) The resulting system would allow
`
`the user to choose whether to display Altman and Langseth’s AR content objects
`
`superimposed on either a map (as taught by Altman and Langseth) or on a live view
`
`(as taught by Langseth). A POSA would have recognized the virtual objects as AR
`
`content objects, and both the map view and live view as forms of AR. (Zyda, ¶¶93-
`
`107; see, e.g., Langseth, ¶0034 (describing the map view as an “augmented reality”);
`
`id. ¶0012 (describing the live view as “an augmented reality”); ’518, 10:35-37 (the
`
`“view of interest” in which AR content objects are rendered “is a digital
`
`representation of a physical location in real-world space”).)
`
` 
`
`
`
`-13-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`Manner, Rationale, and Motivation to Combine: It would have been
`
`obvious to combine Altman with Langseth. (Zyda, ¶¶108-112, 99-102.) The same
`
`manner, rationale, and motivation to combine described below applies to all
`
`combinations of Altman with Langseth in this Petition.
`
`Altman and Langseth are analogous references in the field of location-based
`
`social networking, both of which describe displaying virtual objects used in the
`
`social networking application on a map display of the area around the user. (Altman,
`
`¶¶0006, 0051; Langseth, Abstract, Fig. 9; Zyda, ¶¶110-112.) Additionally, both are
`
`implemented on a mobile device including a camera and, as discussed below, one or
`
`more location sensors, such as GPS. (Altman, ¶¶0103, 0004, 0044; Langseth,
`
`¶0006.) Accordingly, a POSA would have had confidence that Langseth’s virtual
`
`objects could be implemented within Altman’s system. (Zyda, ¶¶111, 108, 100.)
`
`The combination would have predictably resulted in Altman’s location-based
`
`social networking application, including the map view and its use of icons/objects,
`
`being enhanced with Langseth’s virtual objects being shared with a user’s friends
`
`through Altman’s social network manager. (Langseth, ¶¶0008, 0028, 0031, 0034,
`
`0044; Altman ¶¶0047-0056; Zyda, ¶¶108, 111, 100, 103-104.) Altman provides a
`
`social network for sharing AR content objects, and Langseth provides additional
`
`types of AR content objects for sharing via Altman’s social network. (Zyda, ¶111)
`
`The combination further would have predictably resulted in the addition of a “live
`
` 
`
`
`
`-14-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`view” as an alternative way to view AR content objects. (Langseth, ¶¶0028, 0031,
`
`0044; Altman, ¶¶0105, 0006, 0051; Zyda, ¶¶109, 112.)
`
`A POSA would have been motivated to incorporate Langseth’s virtual objects,
`
`including games and game objects, into Altman’s location-based social network
`
`application to provide users more ways to connect with each other through a wider
`
`set of virtual objects and virtual object interactions. (Altman, ¶¶0005, 0048.)
`
`Langseth teaches that its virtual objects can be used to “stay in touch with friends or
`
`meet new people via interaction with the virtual objects,” making them an obvious
`
`choice to incorporate in Altman’s social network. (Langseth, Abstract.) A POSA,
`
`having incorporated Langseth’s virtual objects into Altman’s system, would have
`
`been motivated further to incorporate Langseth’s “live view” by displaying virtual
`
`objects superimposed on camera images of a user’s surroundings in order to create
`
`a more immersive environment and enable additional forms of user interaction with
`
`the AR content objects. (Zyda, ¶¶109, 112, 99, 101-103.) For example, Altman
`
`discloses that one use of its map display is a “street level map to aid the navigation
`
`of the user within the region displayed by the map.” (Altman, ¶0050.) A POSA
`
`would have been motivated to incorporate Langseth’s “live view” to further
`
`Altman’s goal to aid navigation, as Langseth teaches that its live view may display
`
`arrows and other direction indicators. (Langseth, ¶0044.) A POSA would have
`
`appreciated that Altman’s sensors, including location sensors and camera, would be
`
` 
`
`
`
`-15-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`compatible with implementing Langseth’s live view since Langseth teaches using
`
`the same types of sensors. (Zyda, ¶¶109, 116; Langseth, ¶0006; Altman, ¶0090.)
`
`(b)
`“at least one sensor, including a location sensor;” (1[a])
`Altman’s mobile device contains at least one sensor, including a location
`
`sensor. (Altman, ¶¶0044 (describing “position determination unit 166, such as a
`
`Global Positioning System (GPS) method”), 0036; Zyda, ¶¶113-116.) A POSA
`
`would have understood a mobile device’s GPS system must include a sensor,
`
`particularly given Langseth’s disclosure of a “GPS sensor.” (Langseth, ¶0006; Zyda,
`
`¶¶115-116; see also ’518 Patent Claim 2 (listing GPS as type of location sensor).)
`
`(c)
`“a display;” (1[b])
`Altman’s mobile device contains a display:
`
`(Altman, Fig. 2A, ¶¶0044, 0051; Zyda, ¶¶117-118.)
`
`
`
`(d)
`
`“a non-transitory computer readable memory storing
`-16-
` 
`
` 
`
`
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`software instructions; and” (1[c])
`Altman discloses a non-transitory computer readable memory storing
`
`software instructions. (Zyda, ¶¶119-125.) Altman discloses its functionality could
`
`be
`
`implemented as “instructions embodied
`
`in…computer-readable media.”
`
`(Altman, ¶¶0124, 0123, 0046, 0049, 0107; Zyda, ¶¶119-125.)
`
`(e)
`
`“at least one processor coupled with the non-transitory
`computer readable memory, the at least one sensor,
`and the display; and, upon execution of the software
`instructions, is configurable to:” (1[d])
`Altman’s mobile device contains at least one processor coupled with the at
`
`least one sensor and the display:
`
`(Altman, Fig. 1B (red outlines added); Zyda, ¶¶126-127.) Further, a POSA would
`
`
`
` 
`
`
`
`-17-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,664,518 B2
`
`have understood that the processor module is coupled to a non-transitory computer
`
`readable memory. (Zyda, ¶¶128-130; Altman, ¶¶0123-0124.)
`
`(f)
`
`“obtain sensor data from the at least one sensor
`wherein the sensor data includes a device location
`obtained from the location sensor;” (1[d][i])
`Altman discloses obtaining sensor data from the at least one sensor
`
`wherein the sensor data includes a device location obtained from the location
`
`sensor. Altman discloses that the location sensor sends the mobile device’s latitude
`
`and longitude information to the processor. (Altman, ¶¶0044-0045, 0049, 0090;
`
`Zyda, ¶¶131-132.)3
`
`(g)
`
`“obtain an area of interest via an area database based
`on at least the device location within the sensor data;”
`(1[d][ii])
`Altman’s mobile device obtain[s] an area of interest