throbber
(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2011/0199479 A1
`
`
` Waldman (43) Pub. Date: Aug. 18, 2011
`
`US 20110199479A1
`
`(54) AUGMENTED REALITY MAPS
`
`(52) US. Cl. .................. 348/116; 701/201; 348/E07.085
`
`(75)
`
`Inventor:
`
`Jaron Waldman, Palo Alto, CA
`(US)
`
`(73) Ass1gnee:
`
`Apple Inc., Cupertmo, CA (US)
`
`(21) Appl. No.:
`
`12/705,558
`
`(22)
`
`Filed:
`
`Feb. 12, 2010
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`H04N 7/18
`G01C 21/00
`
`(2006.01)
`(2006.01)
`
`(57)
`
`ABSTRACT
`.
`.
`.
`.
`A user pomts a handheld communicatlon dev1ce to capture
`and display a real-time Video stream. The handheld commu-
`nication device detects geographic position, camera direc-
`tion, and tilt of the image capture device. The user sends a
`search request to a server for nearby points of interest. The
`handheld communication device receives search results
`based on the search request, geographic position, camera
`direction, and tilt ofthe handheld communication device. The
`handheld communication device visually augments the cap-
`tured video stream with data related to each point of interest.
`The user then selects a point of interest to visit. The handheld
`communication device visually augments the captured video
`stream with a directional map to a selected point of interest in
`response to the user input.
`
`
`
`
`
`Processor
`Memory
`Graphics accelerator ’
`52d ’
`@29
`$2.2.
`
`
`
`
`
`Accelerometer
`Cemrrlunicafion
`626
`
`
`
`.
`5
`Camera
`
`iererface
`
`
`
`28
`
`
`
`
`
`
`
`
`Compass
`Display
`,
`Input Device
`
`egg
`% we}
`gee
`
`
`
`
`
`602
`
`Snap Inc. EX. 1011 Page 0001
`
`Snap Inc. Ex. 1011 Page 0001
`
`

`

`Patent Application Publication
`
`Aug. 18, 2011 Sheet 1 0f 6
`
`US 2011/0199479 A1
`
`
`
`Snap Inc. EX. 1011 Page 0002
`
`Snap Inc. Ex. 1011 Page 0002
`
`

`

`Patent Application Publication
`
`Aug. 18, 2011 Sheet 2 0f 6
`
`US 2011/0199479 A1
`
`FIG.2
`
`Snap Inc. EX. 1011 Page 0003
`
`Snap Inc. Ex. 1011 Page 0003
`
`

`

`Patent Application Publication
`
`Aug. 18, 2011 Sheet 3 0f6
`
`US 2011/0199479 A1
`
`New
`
`NS“
`
`m.QE
`
`Snap Inc. EX. 1011 Page 0004
`
`Snap Inc. Ex. 1011 Page 0004
`
`

`

`Patent Application Publication
`
`Aug. 18, 2011 Sheet 4 0f 6
`
`US 2011/0199479 A1
`
`402
`
`404
`
`406
`
`408
`
`410
`
`412
`
`Capturing & displaying a video
`stream on a handheld
`electronic device
`
`Detecting geographic position,
`direction, and/or tilt of the
`handheld electronic device
`
`
`
`Sending a request for nearby
`POIs based on a search term
`
`Receiving nearby POI in
`response to the request
`
`Visually augmenting the
`captured video stream
`with POI data
`
`Visually augmenting the captured
`video stream with a directional
`map to a selected point of interest
`
`in response to a user input
`
`FIG. 4
`
`Snap Inc. EX. 1011 Page 0005
`
`Snap Inc. Ex. 1011 Page 0005
`
`

`

`Patent Application Publication
`
`Aug. 18, 2011 Sheet 5 0f 6
`
`US 2011/0199479 A1
`
`f
`
`“’71517
`
`
`
`Keybaard
`Q345-
`‘
`‘
`Microphgne iiiiiiii
`52.342
`
`‘
`Dispiay
`
`549
`
`N.V.St0rage
`
`550
`
`‘

`
`‘
`RAM
`
`570
`
`Painting
`Device
`
`§§§
`
`Snap Inc. EX. 1011 Page 0006
`
`Snap Inc. Ex. 1011 Page 0006
`
`

`

`Patent Application Publication
`
`Aug. 18, 2011 Sheet 6 0f 6
`
`US 2011/0199479 A1
`
`
`
`
`
`Carmnunicaiions
`interface
`r
`
`828
`
`’
`
`Snap Inc. EX. 1011 Page 0007
`
`Snap Inc. Ex. 1011 Page 0007
`
`

`

`US 2011/0199479 A1
`
`Aug. 18, 2011
`
`AUGMENTED REALITY MAPS
`
`FIELD
`
`[0001] The following relates to searching for nearby points
`of interest, and more particularly to displaying information
`related to nearby points of interest overlaid onto a Video feed
`of a surrounding area.
`
`BACKGROUND
`
`[0002] Augmented reality systems supplement reality, in
`the form of a captured image or Video stream, with additional
`information. In many cases, such systems take advantage of a
`portable electronic device’s imaging and display capabilities
`and combine a Video feed with data describing objects in the
`Video. In some examples, the data describing the objects in
`the Video can be the result of a search for nearby points of
`interest.
`
`For example, a user Visiting a foreign city can point
`[0003]
`a handheld communication device and capture a Video stream
`of a particular View. A user can also enter a search term, such
`as museums. The system can then augment the captured Video
`stream with search term result information related to nearby
`museums that are within the View of the Video stream. This
`
`allows a user to supplement their View of reality with addi-
`tional information available from search engines.
`[0004] However, if a user desires to Visit one of the muse-
`ums, the user must switch applications, or at a minimum,
`switch out of an augmented reality View to learn directions to
`the museum. However, such systems can fail to orient a user’s
`with a poor sense of direction and force the user to correlate
`the directions with objects in reality. Such a transition is not
`always as easy as it might seem. For example, an instruction
`that directs a user to go north on Main St. assumes that the
`user can discern which direction is north. Further, in some
`instances, street signs might be missing or indecipherable,
`making it difficult for the user to find the directed route.
`
`SUMMARY
`
`Such challenges can be overcome using the present
`[0005]
`technology. Therefore, a method and system for displaying
`augmented reality maps are disclosed. By interpreting the
`data describing the surrounding areas, the device can deter-
`mine what objects are presently being Viewed on the display.
`The device can further overlay information regarding the
`presently Viewed objects, thus enhancing reality. In some
`embodiments, the device can also display search results over-
`laid onto the displayed Video feed. Search results need not be
`actually Viewable by a user in real life. Instead, search results
`can also include more-distant objects.
`[0006] The user can interact with the display using an input
`device such as a touch screen. Using the input device, the user
`can select from among objects represented on the screen,
`including the search results.
`[0007]
`In one form of interaction, a device can receive an
`input from the user requesting directions from a present loca-
`tion to a selected search result. Directions can be overlaid
`
`onto the presently displayed Video feed, thus showing a
`course and upcoming turns. As the user and associated device
`progress along a route, the overlaid directions can automati-
`cally update to show the updated path.
`[0008]
`In some embodiments the display can also include
`indicator graphics to point the user in a proper direction. For
`example, if the user is facing south but a route requires the
`
`user to progress north, “no route” would be displayed in the
`display because the user would be looking to the south but the
`route would be behind him or her. In such instances, an
`indicator can point the user in the proper direction to find the
`route.
`
`In some embodiments, multiple display Views can
`[0009]
`be presented based on the orientation of the device. For
`example, when the device is held at an angle with respect to
`the ground of 45 degrees to 180 degrees, the display View can
`present the augmented reality embodiments described herein.
`However, when the device is held at an angle less than 45
`degrees, an illustrated or schematic View can be represented.
`In such embodiments, when the device is held at an angle with
`respect to the ground of less than 45 degrees, the device is
`likely pointed at the ground, where few objects of interest are
`likely to be represented in the displayed Video. In such
`instances, a different map View is more likely to be useful. It
`should be appreciated that precise range oftilt can be adjusted
`according the actual environment or user preferences.
`[0010]
`In practice, a user points a handheld communication
`device to capture and display a real-time Video stream of a
`View. The handheld communication device detects a geo-
`graphic position, camera direction, and tilt of the image cap-
`ture device. The user sends a search request to a server for
`nearby points of interest. The handheld communication
`device receives search results based on the search request,
`geographic position, camera direction, and tilt of the hand-
`held communication device. The handheld communication
`
`device visually augments the captured Video stream with data
`related to each point of interest. The user then selects a point
`of interest to visit. The handheld communication device Visu-
`
`ally augments the captured video stream with a directional
`map to a selected point of interest in response to the user
`input.
`[0011] A method of augmenting a Video stream of a
`device’s present surrounding with navigational information is
`disclosed. The user can instruct the device to initiate a live
`
`Video feed using an onboard camera and display the captured
`Video images on a display. By polling a Global Positioning
`System (GPS) device, a digital compass, and optionally, an
`accelerometer, location, camera direction, and orientation
`information can be determined. By using the location, camera
`direction, and orientation information, the device can request
`data describing the surrounding areas and the objects therein.
`In some embodiments, this data includes map vector data.
`The can be requested from an onboard memory or a server.
`The data describing surrounding areas can further be
`requested in conjunction with a search request. The search
`request can also include a request for information about
`nearby places of interest.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 illustrates an exemplary visually augmented
`[0012]
`captured image with data related to a search for points of
`interest;
`FIG. 2 illustrates a the results of a field-of—view and
`[0013]
`point-of-interest search;
`[0014]
`FIG. 3 illustrates an exemplary captured image Visu-
`ally augmented with a route to a selected point of interest;
`[0015]
`FIG. 4 is a flow chart illustrating an exemplary
`method of preparing and displaying an augmented reality
`map;
`FIG. 5 is a schematic illustration of an exemplary
`[0016]
`system embodiment; and
`
`Snap Inc. EX. 1011 Page 0008
`
`Snap Inc. Ex. 1011 Page 0008
`
`

`

`US 2011/0199479 A1
`
`Aug. 18, 2011
`
`FIG. 6 is a schematic illustration of an exemplary
`[0017]
`system embodiment.
`
`DESCRIPTION
`
`[0018] The technology described herein Visually augments
`a captured image or Video stream with data for points of
`interest related to search terms entered by the user. The tech-
`nology also Visually augments the captured image or Video
`stream with a directional map to a selected point of interest.
`[0019]
`FIG. 1 is a screenshot illustrating an augmented
`reality embodiment as described herein. As illustrated, a
`handheld communication deVice has captured an image 102
`of the northwest corner of the intersection of Dolores St and
`
`17th St. using its image-capturing deVice and displayed the
`image on its display. In this way, the display can function as a
`Viewfinder. As illustrated, the captured image 102 has been
`augmented with information corresponding to points of inter-
`est 104, 106 and street labels 110, 112.
`[0020]
`FIG. 1 illustrates a captured and presented image
`102 using an image capture deVice, i.e., the camera of a smart
`phone, which is but one type of handheld communication
`deVice to which the present disclosure can be applied. In this
`illustrated embodiment, the user has entered a search term
`“parks” in search bar 108 to conduct a search for nearby
`parks, i.e., a specific type of point of interest. Using map data
`that describes the area surrounding the present location of the
`deVice and the points of interest located in the surrounding
`area, the deVice augments the displayed image with addi-
`tional information. In this instance, the smart phone or hand-
`held communication deVice displays points of interest
`described by the data that are displayed in the Viewfinder
`(such as Dolores St. 110 and 17th St. 112) or within a field of
`View and range from the geographic position ofthe deVice but
`that are obstructed by other in-screen objects, e.g., Golden
`Gate Park 104 and Buena Vista Park 106. While other parks
`might also be nearby, they are not shown because they fall
`outside the field of View of the deVice. HoweVer, the user
`could locate these parks by panning the deVice around the
`intersection, in which case those parks would appear on the
`screen.
`
`In the captured image 102, the handheld communi-
`[0021]
`cation deVice augments the captured image with bubbles
`showing the relatiVe geographic position of “Golden Gate
`Par ” 104 and “Buena Vista Park” 106 within the captured
`image 102. This allows the user to determine a general direc-
`tion to a point of interest. A user can then select a point of
`interest, e.g., by selecting the “Buena Vista Park” 106 point of
`interest information bubble, e.g., by touching the point of
`interest information bubble with a finger or stylus if the smart
`phone employs a touch screen. In other implementations, a
`cursor and mouse can be used to select a desired point of
`interest.
`
`Points of interest can be any map feature, but most
`[0022]
`often a point of interest can be a map feature that identified as
`result of a search for a category of such map features. For
`example, a point ofinterest can be a park when a user searches
`for nearby parks. Likewise a point of interest can be places,
`buildings, structures, eVen friends that can be located on a
`map, when the point of interest is searched for. In some
`instances a point of interest is not necessarily identified as a
`result of a search. A point of interest can also be a map feature
`that is identified by the present system because it can be
`Viewed in the captured image. In short, a point of interest can
`be any map feature for which the user has an interest.
`
`FIG. 2 illustrates search results for point of interest
`[0023]
`results for nearby parks based on geographic position and
`also illustrates how a range and field ofView correspond to the
`results displayed in the Viewfinder. A handheld communica-
`tion deVice captures a Video stream of the View as shown in
`FIG. 1. The handheld communication deVice detects the geo-
`graphic position, camera direction, and tilt of the handheld
`communication deVice.
`
`[0024] The geographic position of the handheld communi-
`cation deVice can be determined using GPS coordinates or
`using triangulation methods using cell phone towers. In yet
`another example, a blend of GPS coordinates and triangula-
`tion information can be used to determine the position of the
`deVice.
`
`[0025] The camera direction is a direction relatiVe to a
`planet’s magnetic field (i.e., Earth’s magnetic field) in which
`the camera is pointing. The camera direction can be consid-
`ered a direction that canbe identifiedusing a compass, such as
`a digital compass. The camera direction can be used to iden-
`tify the direction in which the camera is pointing as it acquires
`an image to be augmented using the present technology.
`[0026] The tilt direction is a direction that determines the
`direction in which either the camera deVice or display deVice
`is pointing relatiVe to a horizontal or Vertical axis. The tilt
`direction can most commonly be determined using an accel-
`erometer.
`
`[0027] The user can enter a search request for nearby points
`of interest based on a search term. In this example, upon entry
`by the user of a search for nearby “Parks” the handheld
`communication deVice sends a request for data related to
`nearby parks to a map database.
`[0028] Either the request itself, or the database being que-
`ried can determine a releVant range from within which search
`results must be encompassed. Upon receipt ofthe request, the
`database will return search results for points of interest
`related to the search term that are also within a defined radius
`of the handheld communication deVice as illustrated in FIG.
`
`2. As shown in this example, the server returned points of
`interest “Golden Gate Park” 208, “Buena Vista Park” 206,
`“Midtown Terrace Playground” 210, and “Mission Dolores
`Par ” 212. The handheld communication deVice determines
`
`that ofthe point-of-interest search results, only “Golden Gate
`Park” 208 and “Buena Vista Park” 206 are within the field of
`
`View of the handheld communication deVice. The point-of-
`interest results “Golden Gate Park” 208 and “Buena Vista
`
`Park” 206 are displayed with their relatiVe spatial relationship
`to the handheld communication deVice. In the example shown
`in FIG. 2, the camera direction of the handheld communica-
`tion deVice is northwest.
`
`[0029] A field of View can be determined using a digital
`compass to inform the deVice ofthe camera direction in which
`the camera is facing or, altematiVely, the user could enter in a
`heading. As explained ab0Ve, in FIGS. 1 and 2, the camera is
`facing northwest and its theoretical line of sight is represented
`as 214 in FIG. 2.Any search results that are to be displayed on
`the Viewfinder must be within a certain angle of line 214. For
`example, a camera on a handheld communication deVice
`might only be able to display range of View encompassing 30
`degrees. In such an instance, a giVen display would represent
`those items encompassed within 15 degrees in each direction
`from the center ofthe field ofView. This concept is illustrated
`in FIG. 2 wherein 214 illustrates the center ofthe field ofView
`
`and angles 01 216:02 218 and they represent angles from the
`center of the field of View to the outer limits of the field of
`
`Snap Inc. EX. 1011 Page 0009
`
`Snap Inc. Ex. 1011 Page 0009
`
`

`

`US 2011/0199479 A1
`
`Aug. 18, 2011
`
`view. A distance from the device’s geographic location can
`also be used to define a field of view. As discussed above, a
`distance or range can be defined by the device in its request for
`search results or by the database serving the request. Only
`search results encompassed in this field of view will be dis-
`played on the display.
`[0030]
`In some embodiments, a device can also use an
`accelerometer to inform the device of what objects are dis-
`played in its viewfinder. For example, ifthe device is in a hilly
`location, the accelerometer can tell the device that it is point-
`ing downhill. In another example, the device can determine
`that, due to the topography surrounding its present location
`(described by map data) an object viewed at a certain angle
`from the horizon must be a neighboring hill or mountain peak
`in the distance. In yet another example, an angle from a
`horizon can indicate that the user is viewing a multiple story
`building having places of interest in multiple stories of the
`building. An accelerometer can inform the device ofthe angle
`at which the device is pointed.
`[0031]
`FIG. 3 illustrates a captured image that has been
`Visually augmented with route data to a selected point of
`interest. In this example, a user has selected the “Buena Vista
`Park” point of interest and, in response, the smart phone has
`Visually augmented the captured image 302 with a directional
`map 310 to the selected point of interest, i.e., “Buena Vista
`Park”. The route shows a direction 312 that the user must
`
`travel on Dolores St. to begin travelling to reach “Buena Vista
`Park.” The directional map 310 further indicates a turn 314
`that the user must take, i.e., a turn left onto Duboce Ave. from
`Dolores St. In the illustrated example, the map is shown
`overlaid onto Dolores St.
`
`[0032] The route 310 guides the user with complete navi-
`gation illustrations to reach “Buena Vista Park,” including
`any required turns. In some embodiments, the route can be
`represented as a schematic map, i.e., a simplified map that
`includes only relevant information for the user in an easy-to-
`read format.
`
`[0033] A schematic map can be thought of as similar to a
`subway map one would see on a subway train. While the
`subway track itself might wind and turn, a typical subway
`map represents the subway route as a mostly straight line.
`Further, the subway map often does not have any particular
`scale and frequently shows every destination approximately
`evenly dispersed along the route. Thus, a schematic map as
`discussed below is one that does not adhere to geographic
`“reality,” but rather represents map features in a schematic
`fashion by illustrating directions as a route made of one or
`more roads, trails, or ways that can be represented as substan-
`tially straight lines instead of by their actual shapes (which
`would be represented in a non-schematic map by adhering to
`geographic reality). The schematic map can also be devoid of
`uniform scale. Thus, in some parts of the map, such as an area
`of the map representing a destination, such area can be “dis-
`torted” somewhat to clearly illustrate important details, while
`map areas that represent portions of a route where there are no
`turns or other significant features can be very condensed. In
`short, the map can be a schematic of the real world that can
`provide a simple and clear representation that is sufficient to
`aid the user in guidance or orientation without displaying
`unnecessary map features or detail that could otherwise clut-
`ter a small display space.
`[0034]
`FIG. 4 is a flow chart illustrating an exemplary
`method of preparing and displaying an augmented reality
`map. As shown at block 402, the method includes capturing
`
`and displaying a video stream on a handheld communication
`device. Although described here in reference to a video
`stream, another embodiment of the disclosed technology
`includes capturing and displaying a single still image or a
`series of still images.
`[0035] As shown at block 404, the method includes detect-
`ing geographic position, camera direction, and/or tilt of the
`handheld communication device. This allows the device to
`
`determine features, such as streets, buildings, points of inter-
`est, etc., that are within a field of view for the captured video
`stream.
`
`[0036] As shown at block 406, the method includes sending
`a request for nearby points of interest based on one or more
`search terms. For example, the user can search for nearby
`hotels, parks, or restaurants. The request can be sent to a
`database located on a server that is separate from the handheld
`communication device and communicate via a wireless pro-
`tocol. In another embodiment, the database can be stored
`locally on the device and the search request remains internal
`(sometimes termed “onboard” the device) to the handheld
`communication device.
`
`In block 408, the method includes receiving nearby
`[0037]
`points of interest in response to the request. The server can
`filter point of interest results in one example. In this example,
`if the number of returned points of interest exceeds a set
`threshold, the server can filter the results to only return a fixed
`number of the best results. Various algorithms can be
`employed to filter points of interest to a desired number for
`visual augmentation of a captured video stream. In another
`embodiment, the handheld communication device can filter
`point-of-interest results received from the server for optimal
`display on a handheld communication device.
`[0038]
`In block 410, the handheld communication device
`visually augments the captured video stream with data related
`to each point of interest. As shown in FIG. 2, the handheld
`communication device can visually augment a captured video
`stream with a bubble for each point of interest within the field
`of view for the handheld communication device. The hand-
`
`held communication device determines which points of inter-
`est are within its field of view by analyzing the geographic
`position, camera direction, and/or tilt of the handheld com-
`munication device in concert with the known geographic
`position of the returned points of interest.
`[0039]
`In block 412, the handheld communication device
`visually augments the captured video stream with a direc-
`tional map to a selected point of interest in response to the
`user input. For example, as described in connection with FIG.
`3, the smart phone now visually augments the captured image
`302 with a directional map 310 to the selectedpoint ofinterest
`in response to the user input. The user input can be a selection
`of a displayed point of interest to indicate that the user wishes
`to view navigation data for reaching the selected point of
`interest.
`
`In some embodiments, the display can also include
`[0040]
`indicator graphics to point the user in a proper direction. For
`example, if the user is facing south but a route requires the
`user to progress north, “no route” would be shown in the
`display because the route would be behind him or her. In such
`instances, an indicator can point the user in the proper direc-
`tion to find the displayed route.
`[0041]
`In some embodiments, multiple display views can
`be presented based on the orientation of the device. For
`example, when the device is held at an angle with respect to
`the ground of 45 degrees to 180 degrees, the display view can
`
`Snap Inc. EX. 1011 Page 0010
`
`Snap Inc. Ex. 1011 Page 0010
`
`

`

`US 2011/0199479 A1
`
`Aug. 18, 2011
`
`present the augmented reality embodiments described herein.
`However, when the device is held at an angle less than 45
`degrees, an illustrated or schematic view can be presented. In
`such embodiments, when the device is held at an angle with
`respect to the ground of less than 45 degrees, the device is
`likely pointed at the ground, where few objects of interest are
`likely to be represented in the displayed Video. In such
`instances, a different map view than the augmented reality
`map is more likely to be useful. It should be appreciated that
`precise range of tilt can be adjusted according to the actual
`environment or user preferences.
`[0042]
`FIG. 5 illustrates a computer system 500 used to
`execute the described method and generate and display aug-
`mented reality maps. Computer system 500 is an example of
`computer hardware, software, and firmware that can be used
`to implement the disclosures above. System 500 includes a
`processor 520, which is representative of any number of
`physically and/or logically distinct resources capable of
`executing software, firmware, and hardware configured to
`perform identified computations. Processor 520 communi-
`cates with a chipset 522 that can control input to and output
`from processor 520. In this example, chipset 522 outputs
`information to display 540 and can read and write informa-
`tion to non-volatile storage 560, which can include magnetic
`media and solid state media, for example. Chipset 522 also
`can read data from and write data to RAM 570. A bridge 535
`for interfacing with a variety of user interface components
`can be provided for interfacing with chipset 522. Such user
`interface components can include a keyboard 536, a micro-
`phone 537, touch-detection-and-processing circuitry 538, a
`pointing device such as a mouse 539, and so on. In general,
`inputs to system 500 can come from any of a variety of
`machine-generated and/or human-generated sources.
`[0043] Chipset 522 also can interface with one or more data
`network interfaces 525 that can have different physical inter-
`faces 517. Such data network interfaces can include inter-
`faces for wired and wireless local area networks, for broad-
`band wireless networks, as well as personal area networks.
`Some applications ofthe methods for generating and display-
`ing and using the augmented reality user interface disclosed
`herein can include receiving data over physical interface 517
`or be generated by the machine itself by processor 520 ana-
`lyzing data stored in memory 560 or 570. Further,
`the
`machine can receive inputs from the user via devices key-
`board 536, microphone 537, touch device 538, and pointing
`device 539 and execute appropriate functions, such as brows-
`ing functions by interpreting these inputs using processor
`520.
`
`[0044] While FIG. 5 illustrates an example of a common
`system architecture, it should also be appreciated that other
`system architectures are known and can be used with the
`present technology. For example, systems wherein most or all
`of the components described within FIG. 5 can be joined to a
`bus, or the peripherals could write to a common shared
`memory that is connected to a processor or a bus can be used.
`Other hardware architectures are possible and such are con-
`sidered to be within the scope of the present technology.
`[0045]
`FIG. 6 illustrates an exemplary system embodiment.
`A server 602 is in electronic communication with a handheld
`
`communication device 618 having functional components
`such as a processor 620, memory 622, graphics accelerator
`624, accelerometer 626, communications interface 628, com-
`pass 630, GPS 632, display 634, input device 636, and camera
`
`638. None of the devices are limited to the illustrated com-
`
`ponents. The components may be hardware, software, or a
`combination of both.
`
`In some embodiments, the server can be separate
`[0046]
`from the handheld communication device. The server and
`handheld communication device can communicate wire-
`
`lessly, over a wired-connection, or through a mixture ofwire-
`less and wired connections. The handheld communication
`device can communicate with the server over a TCP/IP con-
`nection. In another embodiment, the handheld communica-
`tion device can be directly connected to the server. In another
`embodiment, the handheld communication device can also
`act as a server and store the points of interest locally.
`[0047]
`In some embodiments, instructions are input to the
`handheld electronic device 618 through an input device 636
`that instructs the processor 620 to execute functions in an
`augmented reality application. One potential instruction can
`be to generate an augmented reality map of travel directions
`to a point of interest. In that case, the processor 620 instructs
`the camera 638 to begin feeding video images to the display
`634. In some embodiments, video images recorded by the
`camera are first sent to graphics accelerator 624 for process-
`ing before the images are displayed. In some embodiments,
`the processer can be the graphics accelerator. The image can
`be first drawn in memory 622 or, ifavailable, memory directly
`associated with the graphics accelerator 624.
`[0048] The processor 620 can also receive location and
`orientation information from devices such as a GPS device
`
`632, communications interface 628, digital compass 630 and
`accelerometer 626. The GPS device can determine GPS coor-
`
`dinates by receiving signals from Global Positioning System
`(GPS) satellites and can communicate them to the processor
`Likewise, the processor can determine the location of the
`device through triangulation techniques using signals
`received by the communications interface 628. The processor
`can determine the orientation ofthe device by receiving direc-
`tional information from the digital compass 620 and tilt infor-
`mation from the accelerometer.
`
`[0049] The processor can also direct the communications
`interface to send a request to the server 602 for map data
`corresponding to the area surrounding the geographical loca-
`tion of the device. In some embodiments, the processor can
`receive signals from the input device, which can be inter-
`preted by the processor to be a search request for map data
`including features of interest.
`[0050] The processor can interpret the location and orien-
`tation data received from the accelerometer 626, compass
`630, or GPS 632 to determine the direction in which the
`camera 638 is facing. Using this information, the processor
`can further correlate the location and orientation data with the
`
`map data and the video images to identify objects recorded by
`the camera 638 and displayed on the display 634.
`[0051] The processor can receive other inputs via the input
`device 636 such as an input that can be interpreted as a
`selection of a point of interest displayed on the display 634
`and a request for directions. The processor 620 can further
`interpret the map data to generate and display a route over the
`displayed image for guiding the user to a destination (selected
`point of interest).
`[0052] As the user follows the specified direction to the
`selected points of interest, the processor can continue to
`receive updated location and directional
`information and
`video input and update the overlaid route.
`
`Snap Inc. Ex. 1011 Page 0011
`
`Snap Inc. Ex. 1011 Page 0011
`
`

`

`US 2011/0199479 A1
`
`Aug. 18, 2011
`
`above-described
`the
`to
`according
`[0053] Methods
`examples can be implemented using computer-executable
`instructions that are stored or otherwise available from com-
`
`for
`puter-readable media. Such instructions comprise,
`example, instructions and data which cause or otherwise con-
`figure a general-purpose computer, a special-purpose com-
`puter, or a special-purpose processing device to perform a
`certain function or group of functions. Portions of computer
`resources used can be accessible over a network. The com-
`
`puter-executable instructions may be, for example, binaries,
`intermediate format instructions such as assembly language,
`firmware, or source code. E

This document is available on Docket Alarm but you must sign up to view it.


Or .

Accessing this document will incur an additional charge of $.

After purchase, you can access this document again without charge.

Accept $ Charge
throbber

Still Working On It

This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.

Give it another minute or two to complete, and then try the refresh button.

throbber

A few More Minutes ... Still Working

It can take up to 5 minutes for us to download a document if the court servers are running slowly.

Thank you for your continued patience.

This document could not be displayed.

We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.

Your account does not support viewing this document.

You need a Paid Account to view this document. Click here to change your account type.

Your account does not support viewing this document.

Set your membership status to view this document.

With a Docket Alarm membership, you'll get a whole lot more, including:

  • Up-to-date information for this case.
  • Email alerts whenever there is an update.
  • Full text search for other cases.
  • Get email alerts whenever a new case matches your search.

Become a Member

One Moment Please

The filing “” is large (MB) and is being downloaded.

Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.

Your document is on its way!

If you do not receive the document in five minutes, contact support at support@docketalarm.com.

Sealed Document

We are unable to display this document, it may be under a court ordered seal.

If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.


Access Government Site

We are redirecting you
to a mobile optimized page.





Document Unreadable or Corrupt

Refresh this Document
Go to the Docket

We are unable to display this document.

Refresh this Document
Go to the Docket