US009317133B2
`
`(12) United States Patent
`Korah et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9,317,133 B2
`Apr. 19, 2016
`
`(54) METHOD AND APPARATUS FOR
`GENERATINGAUGMENTED REALITY
`CONTENT
`
`(75) Inventors: Thommen Korah, Marina Del Rey, CA
`(US); Ronald Azuma, Santa Monica,
`CA (US)
`
`(73) Assignee: Nokia Technologies Oy, Espoo (FI)
`-
`(*) Notice:
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 477 days.
`
`3, 2005 Okuda et al.
`2005/0051623 A1
`4/2005 Onishi
`2005/0O85999 A1
`1/2006 Borak
`2006,0002590 A1
`3/2007 Kubo et al.
`2007/0053551 A1
`7/2007 Hamynen et al.
`2007/0162942 A1
`8/2007 Brulle-Drews et al.
`2007/0192020 A1
`2007/024208.6 A1* 10/2007 Tsujimoto et al. ............ 345/632
`2008/0181507 A1* 7/2008 Gope et al..................... 382,190
`2008, 0211809 A1
`9, 2008 Kim et al.
`2008/0240573 A1 10, 2008 Nakamura et al.
`2008/0310757 A1 12/2008 Wolberg et al.
`2009.0109216 A1
`4, 2009 Uetabira
`Continued
`(Continued)
`FOREIGN PATENT DOCUMENTS
`
`(21) Appl. No.: 12/900,990
`22) Filed
`Oct. 8, 2010
`(22) Filed:
`ct. 8,
`(65)
`Prior Publication Data
`
`AU
`CA
`
`T 2009
`2007 362729 A1
`T 2009
`2709 740 A1
`(Continued)
`OTHER PUBLICATIONS
`
`Apr. 12, 2012
`
`(2006.01)
`(2006.01)
`(2011.01)
`
`Heinrich et al., ARWeather—An Augmented Reality Weather Sys
`tem, 2008, IEEE International Symposium on Mixed and Augmented
`Reality, 187-188.*
`
`(Continued)
`
`US 2012/OO86727 A1
`(51) Int. Cl
`G09G 5/00
`G06F 3/03
`G06T 9/00
`Primary Examiner — M. Good Johnson
`(52) U.S. Cl.
`CPC ................ G06F 3/03 (2013.01); ong I so (74) Attorney, Agent, or Firm — Ditthavong & Steiner, P.C.
`2013.O1
`(58) Field of Classification Search
`(57)
`ABSTRACT
`S. licati - - - - - file? - - - - - - - - - - - - -i - - - - - - - E. 632, 629
`An approach is provided for providing augmented reality
`ee application file for complete search history.
`based on tracking. Information, including location informa
`References Cited
`tion, orientation information, or a combination thereof of a
`device is determined. A representation of a location indicated
`based, at least in part, on the information is determined. One
`or more items are selected to associate with one or more
`points within the representation. Display information is deter
`mined to be generated, the display information including the
`one or more items overlaid on the representation based, at
`least in part, on the one or more points.
`16 Claims, 9 Drawing Sheets
`
`(56)
`
`U.S. PATENT DOCUMENTS
`
`6,711,293 B1
`3, 2004 Lowe
`SE R : $2. St. al. 3.? 9.
`OSall Cal. ................
`8,207,964 B1* 6/2012 Meadow et al. .............. 345/419
`9/2002 Miyaki
`2002fO130906 A1
`2003. O1699.03 A1
`9, 2003 Yoshikawa et al.
`2004/O194129 A1
`9, 2004 Carlbom et al.
`
`
`
`Niantic's Exhibit No. 1026
`Page 001
`
`

`

`US 9,317,133 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2009/0289955 A1* 11/2009 Douris et al. ................. 345,630
`2010, 0161658 A1
`6/2010 Hamynen et al.
`2010/0214291 A1
`8, 2010 Muller et al.
`2011 OO63295 A1
`3/2011 Kuo et al.
`2011/0103651 A1* 5, 2011 Nowak et al. ................. 382/106
`2011/0164163 A1* 7/2011 Bilbrey et al. .
`348,333.01
`2011/0279445 A1* 11/2011 Murphy et al.
`... 345,419
`2011/0302527 A1* 12/2011 Chen et al. .................... T15,800
`
`
`
`FOREIGN PATENT DOCUMENTS
`
`3, 2007
`1292.8494. A
`CN
`101275841 A 10, 2008
`CN
`101379.369 A
`3, 2009
`CN
`101903745. A 12/2010
`CN
`20 2007 O18 604 U1
`4/2009
`DE
`20 2007 O18 605 U1
`4/2009
`DE
`1594 322 A2 11/2005
`EP
`1612707 A2
`1, 2006
`EP
`1681 537 A1
`T 2006
`EP
`1796 048 A2
`6, 2007
`EP
`1971 825 A1
`9, 2008
`EP
`1975 S65 A2 10/2008
`EP
`2183 724 A1
`5, 2010
`EP
`2223 041 A1
`9, 2010
`EP
`2413 O21 A 10/2005
`GB
`2002-340588 A 11, 2002
`JP
`2003-263.104 A
`9, 2003
`JP
`2005-083941 A
`3, 2005
`JP
`2005-121650 A
`5, 2005
`JP
`2006-030208 A
`2, 2006
`JP
`2007-074326 A
`3, 2007
`JP
`2008-0250687. A 10, 2008
`JP
`2010-238.098 A 10, 2010
`JP
`2011-506983. A
`3, 2011
`JP
`20080076611 A
`8, 2008
`KR
`20080084859. A
`9, 2008
`KR
`2002099 C2
`6, 2009
`NL
`2002105 A1
`6, 2009
`NL
`2002105 C
`4/2011
`NL
`2008 132293. A
`2, 2010
`RU
`WO96,31047 A2 10, 1996
`WO
`WO WO 2005/098362 A1 10/2005
`WO WO 2007/080473 A1
`7/2007
`WO WO 2007/1284.52 A2 11/2007
`WO WO 2008. 143523 A1 11, 2008
`WO WO 2009/O155O1 A1
`2, 2009
`WO WO 2009/08007O A1
`T 2009
`
`OTHER PUBLICATIONS
`
`Madsen, Claus B., Tommy Jensen, and Mikkel S. Andersen. “Real
`time image-based lighting for outdoor augmented reality under
`dynamically changing illumination conditions.” International con
`ference on Graphics. Theory and Applications, GRAPP 2006, pp.
`364-371.
`International Search Report for related International Patent Applica
`tion No. PCT/FI2011/050853 dated Feb. 13, 2012, pp. 1-7.
`International Search Report for related International Patent Applica
`tion No. PCT/FI2011/050864 dated Feb. 20, 2012, pp. 1-7.
`International Written Opinion for related International Patent Applia
`tion No. PCT/FI2011/050853 dated Feb. 13, 2012, pp. 1-9.
`
`International Written Opinion for related International Patent Appli
`cation No. PCT/FI2011/050864 dated Feb. 20, 2012, pp. 1-9.
`U.S. Appl. No. 12/790,288, filed May 28, 2010, Wither, Jason et al.
`3D SSD tracking with estimated 3D planes, Cobzas et al., Image and
`Vision Computing 27 (2009) 69-79.
`Augmented Reality Scouting for Interactive 3D Reconstruction,
`Reitinger et al., abstract only, in Virtual Reality Conference, IEEE,
`2007, 1 page.
`Augmented Reality Scouting for Interactive 3D Reconstruction,
`Reitinger et al., accessed on: Nov.30, 2010, http://www.icg...tu-graZ.
`ac.at/publications/pdfreitingerarScouting07/at download/file, pp.
`1-4.
`Estimating Natural Illumination from a Single Outdoor Image,
`Lalonde et al., accessed on: Dec. 3, 2010, http://graphics.cs.cmu.edu/
`projects/outdoorillumination/lalonde iccV 09.pdf, pp. 1-8.
`Fusing Online and Offline Information for Stable 3D Tracking in
`Real-Time, Vacchetti et al., accessed on Nov.30, 2010, http://cvlab.
`epfl.ch/publications/publications/2003/VacchettiLF03.pdf, pp. 1-8.
`Image Fusion for Context Enhancement and Video Surrealism,
`Raskar et al., TR2004-039, Jun. 2004. International Symposium on
`Non-Photorealistic Animation and Rendering, pp. 1-12.
`Location-Based Games, accessed on: Dec. 3, 2010, http://en.
`wikipedia.org/wiki/Location-based game, pp. 1-4.
`Markerless Tracking using Planar Structures in the Scene, Simon et
`al., accessed on Nov. 30, 2010, http://www.robots.ox.ac.uk/-vgg?
`publications/papers/Simon00.pdf, pp. 1-9.
`Parallel Tracking and Mapping for Small ARWorkspaces, Klein et
`al., accessed on: Dec. 3, 2010, http://www.robots.ox.ac.uk/~gk/pub
`lications/KleinMurray2007ISMAR.pdf, pp. 1-10.
`Parallel Tracking and Mapping on a Camera Phone, Klein et al.,
`accessed on: Dec. 3, 2010, http://www.robots.ox.ac.uk/~gk/publica
`tions/KleinMurray2009ISMAR.pdf, pp. 1-4.
`Pose Tracking from Natural Features on Mobile Phones, Wagner et
`al., accessed on: Dec. 3, 2010, http://www.icg...tugraZ.at/publications/
`pdf WAGNER ISMAR08 NFT.pdf, pp. 1-10.
`Robust Spatiotemporal Analysis of Architectural Imagery, Korah,
`Thommen, University of Delaware, 2007, pp. 1-190.
`Seeing the World Behind the Image: Spatial Layout for 3D Scene
`Understanding, Hoiem, Derek, Aug. 2007, Robotics Institute,
`Carnegie Mellon University, pp. 1-153.
`SURFTrac: Efficient Tracking and Continuous Object Recognition
`using Local Feature Descriptors, Ta et al., accessed on: Dec. 3, 2010,
`http://people.csail.mit.edu/kapu/papers/nguyen cvpr09.pdf,
`pp.
`1-8.
`The Westwood Experience: Connecting Story to Locations Via
`Mixed Reality, Wither et al...IEEE International Symposium on
`Mixed and Augmented Reality 2010, Arts, Media and Humanities
`Proceedings, Oct. 13-16, Seoul, Korea, pp.39-46.
`Tracking with Reference Images: A Real-Time and Markerless
`Tracking Solution for Out-Door Augmented Reality Applications,
`Stricker, Didier, abstract only, in Proceedings of the 2001 conference
`on Virtual reality, archeology, and cultural heritage, 2001, pp. 1-2.
`Tracking with Reference Images: A Real-Time and Markerless
`Tracking Solution for Out-Door Augmented Reality Applications,
`Stricker, Didier, accessed on: Nov. 30, 2010, http://archeoguide.
`intranet.gr/papers/publications/ARCHEOGUIDE-VAST01-2.pdf.
`pp. 1-5.
`Office Action for related U.S. Appl. No. 12/939,558 dated Sep. 18,
`2012, pp. 1-49.
`* cited by examiner
`
`Niantic's Exhibit No. 1026
`Page 002
`
`

`

`U.S. Patent
`
`Apr. 19, 2016
`
`Sheet 1 of 9
`
`US 9,317,133 B2
`
`
`
`
`
`¿383 §§§§§§§§§§§§§
`
`Niantic's Exhibit No. 1026
`Page 003
`
`

`

`U.S. Patent
`
`Apr. 19, 2016
`
`Sheet 2 of 9
`
`US 9,317,133 B2
`
`
`
`§§§
`
`Niantic's Exhibit No. 1026
`Page 004
`
`

`

`U.S. Patent
`
`Apr. 19, 2016
`
`Sheet 3 of 9
`
`US 9,317,133 B2
`
`
`
`
`
`§§§§§§§§§§§§§
`
`arvoossvorsnämnoatas || ~ |º
`
`§§§§§§§ §§
`
`Niantic's Exhibit No. 1026
`Page 005
`
`

`

`U.S. Patent
`
`Apr. 19, 2016
`
`Sheet 4 of 9
`
`US 9,317,133 B2
`
`
`
`
`
`s
`
`s
`
`Niantic's Exhibit No. 1026
`Page 006
`
`

`

`U.S. Patent
`
`Apr. 19, 2016
`
`Sheet 5 Of 9
`
`US 9,317,133 B2
`
`are Es:
`
`
`
`s
`
`Niantic's Exhibit No. 1026
`Page 007
`
`

`

`U.S. Patent
`
`Apr. 19, 2016
`
`Sheet 6 of 9
`
`US 9,317,133 B2
`
`
`
`Niantic's Exhibit No. 1026
`Page 008
`
`

`

`U.S. Patent
`
`Apr. 19, 2016
`
`Sheet 7 Of 9
`
`US 9,317,133 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`lanean 4–1 aoû?aml-º xeometnooi
`
`********
`§§§
`
`
`
`30183ggaeod
`
`Niantic's Exhibit No. 1026
`Page 009
`
`

`

`U.S. Patent
`
`Apr. 19, 2016
`
`Sheet 8 of 9
`
`US 9,317,133 B2
`
`
`
`s
`
`Niantic's Exhibit No. 1026
`Page 0010
`
`

`

`. Patent
`
`Apr
`. 19, 2016
`
`Sheet 9 Of 9
`
`133 B2 317
`
`US 9
`
`
`
`??
`
`is six
`
`g:
`
`88:
`
`{ { * # } } } } }
`
`Åž}}}}}}}}
`
`aoaei &s
`
`§§§ §§
`
`e
`
`&
`
`Niantic's Exhibit No. 1026
`Page 0011
`
`

`

`US 9,317,133 B2
`
`1.
`METHOD AND APPARATUS FOR
`GENERATINGAUGMENTED REALITY
`CONTENT
`
`BACKGROUND
`
`Service providers and device manufacturers (e.g., wireless,
`cellular, etc.) are continually challenged to deliver value and
`convenience to consumers by, for example, providing com
`pelling network services. One area of interest has been the
`10
`development of location-based services (e.g., navigation ser
`vices, mapping services, augmented reality applications, etc.)
`that have greatly increased in popularity, functionality, and
`content. Augmented reality applications allow users to see a
`view of the physical world merged with virtual objects in real
`time. However, with the increase in the available content and
`functions such services and applications, service providers
`and device manufacturers face significant technical chal
`lenges to efficiently track the location of a device while the
`device is moving to provide Such augmented reality content.
`
`15
`
`SUMMARY
`
`Therefore, there is a need for an approach for providing
`augmented reality based on tracking.
`According to one embodiment, a method comprises deter
`mining information including location information, orienta
`tion information, or a combination thereof of a device. The
`method also comprises determining a representation of a
`location indicated based, at least in part, on the information.
`The method further comprises selecting one or more items to
`associate with one or more points within the representation.
`The method additionally comprises determining to generate
`display information including the one or more items overlaid
`on the representation based, at least in part, on the one or more
`points.
`According to another embodiment, an apparatus comprises
`at least one processor, and at least one memory including
`computer program code, the at least one memory and the
`computer program code configured to, with the at least one
`processor, cause, at least in part, the apparatus to determine
`information including location information, orientation
`information, or a combination thereof of a device. The appa
`ratus is also caused to determine a representation of a location
`indicated based, at least in part, on the information. The
`apparatus is further caused to select one or more items to
`associate with one or more points within the representation.
`The apparatus is additionally caused to determine to generate
`display information including the one or more items overlaid
`on the representation based, at least in part, on the one or more
`points.
`According to another embodiment, a computer-readable
`storage medium carries one or more sequences of one or more
`instructions which, when executed by one or more proces
`sors, cause, at least in part, an apparatus to determine infor
`mation including location information, orientation informa
`tion, or a combination thereof of a device. The apparatus is
`also caused to determine a representation of a location indi
`cated based, at least in part, on the information. The apparatus
`is further caused to select one or more items to associate with
`one or more points within the representation. The apparatus is
`additionally caused to determine to generate display informa
`tion including the one or more items overlaid on the repre
`sentation based, at least in part, on the one or more points.
`According to another embodiment, an apparatus comprises
`means for determining information including location infor
`mation, orientation information, or a combination thereof of
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`a device. The apparatus also comprises means for determin
`ing a representation of a location indicated based, at least in
`part, on the information. The apparatus further comprises
`means for selecting one or more items to associate with one or
`more points within the representation. The apparatus addi
`tionally comprises means for determining to generate display
`information including the one or more items overlaid on the
`representation based, at least in part, on the one or more
`points.
`Still other aspects, features, and advantages of the inven
`tion are readily apparent from the following detailed descrip
`tion, simply by illustrating a number of particular embodi
`ments and implementations, including the best mode
`contemplated for carrying out the invention. The invention is
`also capable of other and different embodiments, and its
`several details can be modified in various obvious respects, all
`without departing from the spirit and scope of the invention.
`Accordingly, the drawings and description are to be regarded
`as illustrative in nature, and not as restrictive.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The embodiments of the invention are illustrated by way of
`example, and not by way of limitation, in the figures of the
`accompanying drawings:
`FIG. 1 is a diagram of a system capable of providing
`augmented reality based on tracking, according to one
`embodiment;
`FIG. 2 is a diagram of the components of user equipment,
`according to one embodiment;
`FIG.3 is a flowchart of a process for providing augmented
`reality based on tracking, according to one embodiment;
`FIGS. 4-5 are diagrams of processing of image represen
`tations based on the processes of FIG. 3, according to various
`embodiments;
`FIG. 6 is a diagram of a user interface utilized in the
`processes of FIG. 3, according to one embodiment;
`FIG. 7 is a diagram of hardware that can be used to imple
`ment an embodiment of the invention;
`FIG. 8 is a diagram of a chip set that can be used to
`implement an embodiment of the invention; and
`FIG.9 is a diagram of a mobile terminal (e.g., handset) that
`can be used to implement an embodiment of the invention.
`
`DESCRIPTION OF SOME EMBODIMENTS
`
`Examples of a method, apparatus, and computer program
`for providing augmented reality based on tracking are dis
`closed. In the following description, for the purposes of
`explanation, numerous specific details are set forth in order to
`provide a thorough understanding of the embodiments of the
`invention. It is apparent, however, to one skilled in the art that
`the embodiments of the invention may be practiced without
`these specific details or with an equivalent arrangement. In
`other instances, well-known structures and devices are shown
`in block diagram form in order to avoid unnecessarily obscur
`ing the embodiments of the invention.
`FIG. 1 is a diagram of a system capable of providing
`augmented reality based on tracking, according to one
`embodiment. It is noted that mobile devices and computing
`devices in general are becoming ubiquitous in the world today
`and with these devices, many services are being provided.
`These services can include augmented reality (AR) and
`mixed reality (MR) services and applications. AR allows a
`user's view of the real world to be overlaid with additional
`visual information. MR allows for the merging of real and
`virtual worlds to produce visualizations and new environ
`
`Niantic's Exhibit No. 1026
`Page 0012
`
`

`

`3
`ments. In MR, physical and digital objects can co-exist and
`interact in real time. Thus, MR can be a mix of reality, AR,
`virtual reality, or a combination thereof. A key challenge in
`AR is to accurately track the camera pose so that the virtual
`objects appear registered with objects in the real environment.
`Image-based tracking traditionally relied on artificial markers
`or fiducials placed in a controlled scene. More recently, com
`puter vision techniques have been used to track point features
`through an image sequence in order to estimate Six Degrees
`of Freedom (6DOF) pose on a mobile device. Although vari
`ous embodiments are described with respect to AR, it is
`contemplated that the approach described herein may be used
`with other user interface situations such as MR and virtual
`reality.
`Mixed Reality is particularly compelling for user experi
`ences like virtual tourism, location-based games, and inter
`active story-telling, etc. Many of these occur in populated,
`urban environments with rich history and culture. To create an
`immersive experience that mixes elements of the virtual with
`the real, conventional AR requires a tracking system to be
`stable under extreme conditions.
`Frame-to-frame tracking allows the virtual content to
`appear registered with a video stream from a camera of a
`device. Tracking technologies can treat the scene as a rigid set
`of features that are first detected and then tracked between
`frames. Features may be points, lines, curves, or other para
`metric shapes with unique intensity distributions that allow
`making correspondences. To make tracking on a device real
`time, systems may simplify proven computer vision tech
`nologies to make it less processing intensive. However, this
`30
`can adversely affect the tracking quality. Because these com
`puter vision technologies are processing intensive, a signifi
`cant amount of computation may be expended for low-level
`registration tasks (e.g., detecting or matching certain portions
`of an Streaming camera feed), which prevents more contex
`tual and higher-level processing that could enhance a user's
`overall experience.
`In uncontrolled outdoor environments, reliable tracking in
`real-time on a device with limited processing capability intro
`duces many additional challenges. Typical tracking systems
`rely on a smooth camera motion to restrict the search for
`visual feature correspondences. But these assumptions may
`not hold in the presence of shadows, occlusion, variable light
`ing conditions, and multiple distracting motions, signifi
`cantly affecting the stability of tracking. Traditionally, AR
`45
`content is overlaid on a raw camera stream. Even when stable
`tracking is feasible, there is an obvious visual discrepancy
`between virtual content and the low resolution camera
`stream. For example, the device's camera may not be as sharp
`as or may have unlit pixels due to lighting while the virtual
`content is completely separate and looks vibrant. Seeing low
`resolution images in bright outdoor conditions can also be
`challenging. Effectively, such Mixed Reality is forced to
`make many technical compromises that hamper the quality of
`an experience.
`To address these problems, a system 100 of FIG. 1 intro
`duces the capability to provide augmented reality based on
`tracking, according to one embodiment. A combination of
`approximate tracking with semantic knowledge of a scene
`and its elements can be utilized to provide AR. This allows the
`system 100 to blend elements of the real world into a virtual
`canvas, increasing a user's sense of presence and immersion.
`Because the geometry of the virtual world is known before
`hand, a consistent visual representation can be presented to
`the user.
`Further, a modeled environment can act as the virtual can
`vas on which virtual objects are overlaid. This allows shifting
`
`50
`
`35
`
`40
`
`55
`
`60
`
`65
`
`US 9,317,133 B2
`
`10
`
`15
`
`25
`
`4
`computation from transient frame-to-frame tracking towards
`more permanent recognition and understanding modules.
`Because virtual content is not overlaid on a real-time camera
`feed, approximate tracking (at reduced computation) can be
`utilized to infer a user's approximate pose. By utilizing
`approximate tracking, a higher error tolerance can be met
`while still presenting a compelling user experience. As such,
`large-scale outdoor experiences can be presented to one or
`more users utilizing the system 100.
`User equipment (UEs) 101a-101 in can communicate with a
`location services platform 103 via a communication network
`105 to utilize world data 107 to generate and present location
`based information to users of the UEs 101. Applications 109,
`Such as a map application, navigation application, AR appli
`cation, or the like executing on the UE 101 can be utilized
`process world data 107 and present the world data 107 to the
`user. The location services platform 103 can be implemented
`using one or more computers or servers, for example using the
`chipset described in FIG. 8.
`A data collection module 111 of the UE 101 can collect
`location information (e.g., Global Positioning System (GPS)
`information based on one or more GPS satellites 113,
`Assisted GPS (A-GPS), cell of origin information, etc.), ori
`entation information (e.g., magnetometer information, accel
`erometer information, etc.), combinations thereof, etc. to
`determine information utilized in requesting world data 107
`from the location services platform 103. In certain embodi
`ments, world data 107 or subsets of the world data 107 can be
`stored on the UE 101. The world data 107 can include map
`information. Map information may include maps, satellite
`images, street and path information, item information Such as
`point-of-interest (POI) information, signing information
`associated with maps, objects and structures associated with
`the maps, information about people and the locations of
`people, coordinate information associated with the informa
`tion, etc., or a combination thereof. An item may be an object
`(e.g., a representation of a physical object) or a virtual object
`(e.g., images, audio, video, icons, advertisements, etc.) that
`can be associated with a location. Further, world data can
`include information associated with Light Detection and
`Ranging (LIDAR) pointclouds acquired by one or more ser
`vice providers (e.g., NAVTEQ, GOOGLE MAPS, etc.).
`A POI can be a specific pointlocation that a person may, for
`instance, find interesting or useful. Examples of POIs can
`include an airport, a bakery, a dam, a landmark, a restaurant,
`a hotel, a building, a park, the location of a person, or any
`point interesting, useful, or significant in Some way. In some
`embodiments, the map information and the maps presented to
`the user may be a simulated 3D environment. In certain
`embodiments, the simulated 3D environment is a 3D model
`created to approximate the locations of Streets, buildings,
`features, etc. ofan area. This model can then be used to render
`the location from virtually any angle or perspective for dis
`play on the UE 101. Further, in certain embodiments, the
`graphical user interface (GUI) presented to the user may be
`based on a combination of real world images (e.g., a camera
`view of the UE 101 or a panoramic image) and the 3D model.
`The 3D model can include one or more 3D object models
`(e.g., models of buildings, trees, signs, billboards, lampposts,
`etc.). Each 3D object model can be associated with a particu
`lar location (e.g., GPS coordinates or other location coordi
`nates, which may or may not be associated with the real
`world) and can be identified using one or more identifier. A
`data structure can be utilized to associate the identifier and the
`location with a comprehensive 3D map model of a physical
`environment (e.g., a city, the world, etc.). A Subset or the set
`of data can be stored on a memory of the UE 101. Further,
`
`Niantic's Exhibit No. 1026
`Page 0013
`
`

`

`5
`items can be associated with the 3D model and placed based
`on location coordinates. This can additionally take into
`account occluding parts of the 3D model.
`A representation of a location can be determined based on
`the collected information. This representation can be gener- 5
`ated at the location services platform 103 and/or the UE 101.
`The representation can include an image (e.g., a panoramic
`image), a model (e.g., a 3D model), combinations thereof, etc.
`Further, the representation can be preprocessed at the location
`services platform 103 according to one or more criteria. For 10
`example, the criteria can be based on time parameters and/or
`weather parameters. A time parameter may be utilized to
`preprocess a 3D model and/or image based on a time of day.
`In this manner, the 3D model and/or the image may be light
`ened or darkened based on one or more algorithms to coincide 15
`with lighting in a real world situation. The location services
`platform 103 and/or another platform may be utilized to track
`Sunlight based on time. Further, an algorithm may be utilized
`in processing the image or model according to the time
`parameters. Additionally or alternatively, weather informa- 20
`tion can be tracked and/or utilized instead or in addition to the
`time. As such, if it is snowing at a location that the UE 101 is
`being used, the representation can be preprocessed to include
`Snow. In certain embodiments, the preprocessing can occur
`before any request for a representation while in other embodi- 25
`ments, the preprocessing can occur at the location services
`platform 103 in response to a request for the representation.
`The representation can be processed at the UE 101.
`The UE 101 can request and receive preprocessed world
`data 107 from the location services platform 103. The world 30
`data 107 can include the representation, items, POI informa
`tion, combinations thereof, etc. One or more items are then
`associated with one or more points within the representation.
`For example, a POI item can be associated, based on a spatial
`layout, with particular points of an image or model. Then, the 35
`application 109 can generate display information including
`the POI item (or additional items) overlaid on the represen
`tation based on one or more of the points.
`Further, with this approach, pre-computed representations
`(e.g., the full 3D models, panoramic images, user-generated 40
`images, etc.) can be leveraged to incorporate elements of
`scene understanding and approximate camera tracking to
`improve the quality of an AR experience. For example, Scene
`understanding of an environment can be improved by pro
`cessing the representation by incorporating time and/or 45
`weather elements. As such, a virtual representation (e.g., from
`the world data 107) can be augmented instead of raw camera
`information. Thus, the UE 101 need not act as the lens
`through which a scene is viewed. Instead, the user's periph
`eral vision can provide context to a rich or augmented repre- 50
`sentation of the real world that is shown on the UE 101. With
`this approach, world data 107 (e.g., panoramic images,
`LIDAR, 3D models, maps, user images, other associated
`images, etc.) can be restyled to match real-conditions associ
`ated with a location. In one scenario, other associated images 55
`can be associated with a story, game, historical narrative, or a
`combination thereof. Thus, one or more of the associated
`images can be tagged to a particular location in the world data
`107 and later utilized in one or more applications 109. These
`associated images can be a photorealistic rendering of the 60
`environment or a non-photorealistic rendering of the environ
`ment. With either approach, the renderings can be augmented
`based on the parameters to match current conditions (e.g.,
`include weather or lighting/shadowing in an image generated
`utilizing drawings based on the parameters).
`Moreover, approximate localization and tracking can be
`utilized to limit the processing needed at the UE 101. As such,
`
`65
`
`US 9,317,133 B2
`
`6
`image feature can be associated with information about their
`geometry in a 3D scene. A source of such information can rely
`on fast semantic classification of an image into ground, Ver
`tical Surfaces, sky regions, etc. as further detailed in FIG. 4.
`Another Such source of information can include information
`regularly found in urban environments to detect grid patterns
`of rectangular elements that often signal the presence of pla
`nar Surfaces. Tracking of multiple planes can be accom
`plished in 3D with constraints on perpendicularity or co
`planarity as further detailed in FIG. 5. Thus pixel accurate
`camera tracking under uncontrolled circumstances need not
`be utilized to generate a good user experience for users of
`augmented reality applications 109.
`Additionally or alternatively, generating an augmented
`reality experience can involve embedding relevant aspects of
`the real world into a canvas (e.g., an urban canvas, a Suburban
`canvas, a rural canvas, etc.) or restyling a virtual world to
`better match real-time conditions. As noted above, weather
`and/or time parameters can be utilized to augment represen
`tations. Further, other elements of a real-time environment
`can be incorporated in the scene generated by the augmented
`representation. For example, moving elements (e.g., cars,
`people, etc.) can be detected using an algorithm and incorpo
`rated into the scene.
`With these approaches, natural feature tracking need not
`occur in isolation with the sole intent of estimating a camera
`pose every frame. The feature tracking can be combined with
`high-level domain knowledge (e.g., pre-processed world data
`107) of the scene that provides strong context about the geo
`metric relationships between image features. This knowledge
`may come from assumptions based on characteristics of out
`door urban environments, or it may be garnered from visual
`data previously collected at the specific location of interest
`(e.g., collected by a service such as NAVTEQ, GOOGLE
`MAPS, etc.). Man-made environments are strongly struc
`tured and contain a number of regularities. The dominant
`orientations of line and plane features within the image can
`thus be consistent with the geometry of the scene. Many
`urban structures exhibit symmetry and repetition, a cue that
`can be exploited to overcome occlusions and other distrac
`tions.
`Two dimensional approaches to feature tracking (e.g.,
`based on real-time streaming images) cannot adequately
`address the problems posed by occlusions, clutter, and light
`ing changes. More Sophisticated features such as Scale Invari
`ant Feature Transform (SIFT) and Speeded Up Robust Fea
`tures (SURF) that are robust to some of these effects are not
`practical for real-time implementation (e.g., on a mobile UE
`101 Such as a mobile phone). The current tracking system
`offers advantages from standard feature-based techniques
`because the system augments tracks with hierarchical seman
`tic information Such as the Surface a feature belongs to
`(ground, building, sky, etc.). It can also identify features that
`adhere to Some global constraint and move cohesively, Such
`as windows that form a lattice or grid pattern. Image process
`ing can further benefit from assumptions that are modeled
`statistically based on training data. For example, most people
`take pictures with their camera held upright. An estimate of
`the ground plane, on which virtual objects can be placed,
`should therefore be biased towards appearing lower in the
`image. With these approaches, there is a higher margin for
`error with a less than 100% recognition rate for tracking
`items. As such, the system 100 is less computation intensive
`and allows for approximations when, for example, the UE
`101 moves.
`In general, registering a cam