(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2010/0066750 A1
`(43) Pub. Date:
`Mar. 18, 2010
`Yu et al.
`
`US 20100.066750A1
`
`(54) MOBILE VIRTUAL ANDAUGMENTED
`REALITY SYSTEM
`
`(75) Inventors:
`
`Han Yu, Carpentersville, IL (US);
`Eric R. Buhrke, Clarendon Hills,
`IL (US); Julius S. Gyorfi, Vernon
`Hills, IL (US); Juan M. Lopez,
`Chicago, IL (US)
`Correspondence Address:
`MOTOROLA, INC.
`1303 EAST ALGONQUINROAD, IL01/3RD
`SCHAUMBURG, IL 60196
`(73) Assignee:
`MOTOROLA, INC. Schaumburg,
`IL (US)
`12/211,417
`
`(21) Appl. No.:
`
`(22) Filed:
`
`Sep. 16, 2008
`
`Publication Classification
`
`(51) Int. Cl.
`(2006.01)
`G09G 5/00
`(52) U.S. Cl. ........................................................ 34.5/581
`
`ABSTRACT
`(57)
`A user can create “virtual graffiti” (203) that will be left for
`a particular device to view as part of an augmented-reality
`scene. The virtual graffiti will be assigned to a particular
`physical location or a part of an object that can be mobile. The
`virtual graffiti is then uploaded to a network server (101),
`along with the location and individuals who are able to view
`the graffiti as part of an augmented-reality Scene. When a
`device that is allowed to view the graffiti is near the location,
`the graffiti will be downloaded to the device and displayed as
`part of an augmented-reality scene. To further enhance the
`user experience, the virtual graffiti can be dynamic, changing
`based on ambient-light conditions.
`
`
`
`509
`
`Context
`Aware
`Circuitry
`
`circuitry
`
`Graffiti
`Database
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Logic
`Circuitry
`
`
`
`User
`Interface
`
`511
`
`Transmitter
`
`Receiver
`
`Niantic's Exhibit No. 1003
`Page 001
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 1 of 7
`
`US 2010/0066750 A1
`
`10
`
`Virtual
`Graffiti
`
`105
`
`device
`
`FIG. 1
`
`100
`
`s,
`
`User // 107
`
`device
`
`1
`O9
`
`User
`device
`
`Global
`Ambient;
`Light modification—- Object
`----- sey - - --
`:
`Manager
`
`Personal
`Object
`Manager
`
`Transmitter
`
`409
`-V
`
`N/
`
`A.
`
`411
`
`A/
`401
`
`Fig. 4
`
`Database
`y
`
`403
`
`407
`
`Receiver
`
`Niantic's Exhibit No. 1003
`Page 002
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 2 of 7
`
`US 2010/0066750 A1
`
`
`
`205
`
`201
`Real-world door
`
`2O1
`Real-world door having
`Virtual graffiti upon it
`
`2O1
`Real-world door having
`Virtual graffiti upon it
`
`FG. 2
`
`Niantic's Exhibit No. 1003
`Page 003
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 3 of 7
`
`US 2010/0066750 A1
`
`
`
`Virtual Graffiti Created by
`A first user and left for a
`USer to View.
`
`3O2
`Door as viewed by the
`user at a first time of day
`
`Door as viewed by the
`user at a second time of day
`
`FIG. 3
`
`Niantic's Exhibit No. 1003
`Page 004
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 4 of 7
`
`US 2010/0066750 A1
`
`Context
`AWare
`Circuitry
`
`Ambient
`Light modification
`circuitry
`
`507
`
`
`
`
`
`
`
`
`
`
`
`
`
`Transmitter
`
`Receiver
`
`
`
`
`
`
`
`
`
`
`
`
`
`LOgic
`Circuitry
`
`User
`Interface
`
`Niantic's Exhibit No. 1003
`Page 005
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 5 of 7
`
`US 2010/0066750 A1
`
`global object manager receives information representing
`virtual graffiti, a location of the virtual graffiti and a list of
`users able to view the virtual graffiti
`
`603
`
`FIG. 6
`
`personal object manager periodically receives locations
`for devices
`
`
`
`
`
`
`
`O
`
`
`
`OCation of a device
`near any stored virtual
`Graffiti?
`
`- - - - - - - - - - - - - - - - - - - - - - -t. - - - - - - - - - - - - - - - - - - - - - - - - -
`
`W
`yes
`: Modify the virtual graffiti to account :
`For ambient light source
`
`
`
`Wirelessly transmit the graffiti
`
`Niantic's Exhibit No. 1003
`Page 006
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 6 of 7
`
`US 2010/0066750 A1
`
`user interface receives virtual graffiti input from
`a user, along with a list of individuals with
`privileges to view the graffiti.
`
`
`
`This information is passed to logic circuitry
`
`retrieve a current location for the virtual graffiti
`
`701
`
`703
`
`705
`
`
`
`
`
`
`
`
`
`
`
`transmit the location, ambient light information
`a digital representation of the graffiti, and the
`list of users with privileges to view the graffiti
`to the Server
`
`707
`
`FIG. 7
`
`------------------
`Periodically update graffiti W
`
`Niantic's Exhibit No. 1003
`Page 007
`
`

`

`Patent Application Publication
`
`Mar. 18, 2010 Sheet 7 of 7
`
`US 2010/0066750 A1
`
`901
`Determine if device is indoors or W
`outdoors
`
`902
`Determine position data for the sun W
`
`Obtain local Weather data
`
`
`
`903
`
`Modify graffiti
`
`FIG. 9
`
`periodically access location circuitry
`and provide location to transmitter to
`be transmitted to Server
`
`801
`
`803
`
`805
`
`receive information necessary to view
`the virtual graffiti
`
`analyze the virtual graffiti and accesses
`Ambient-light modification circuitry in
`order to determine if the graffiti should
`be modified
`
`y
`
`807
`
`
`
`Virtual graffiti database is provided
`with the virtual graffiti and information
`necessary to display the virtual graffiti.
`
`display the virtual graffiti as part of an
`augmented reality scene.
`
`811
`
`813
`
`FIG. 8
`
`Niantic's Exhibit No. 1003
`Page 008
`
`

`

`US 2010/0066750 A1
`
`Mar. 18, 2010
`
`MOBILE VIRTUAL AND AUGMENTED
`REALITY SYSTEM
`
`0013 FIG. 9 is a flow chart showing operation of the
`ambient light modification circuitry.
`
`RELATED APPLICATIONS
`0001. This application is related to application Ser. No.
`1 1/844538, entitled MOBILE VIRTUAL AND AUGMENTED
`REALITY SYSTEM, filed Aug. 24, 2007, application Ser. No.
`117858997, entitled MOBILE VIRTUAL AND AUGMENTED
`REALITY SYSTEM, filed Sep. 21, 2007, to application Ser. No.
`11/93.0974 entitled MOBILE VIRTUAL AND AUGMENTED REAL
`ITY SYSTEM, filed Oct. 31, 2007, and to application Ser. No.
`11/962139 entitled MOBILE VIRTUAL AND AUGMENTED REAL
`ITY SYSTEM, filed Dec. 21, 2007.
`
`FIELD OF THE INVENTION
`0002 The present invention relates generally to messag
`ing, and in particular, to messaging within a mobile virtual
`and augmented reality system.
`
`BACKGROUND OF THE INVENTION
`0003 Messaging systems have been used for years to let
`users send messages to each other. Currently, one of the
`simplest ways to send a message to another individual is to
`send a text message to the individual's cellular phone.
`Recently, it has been proposed to expand the capabilities of
`messaging systems so that users of the network may be given
`the option of leaving “virtual graffiti for users of the system.
`For example, the system described in application Ser. No.
`1 1/844538, entitled MOBILE VIRTUAL AND AUGMENTED
`REALITY SYSTEM, allows users to post and retrieve various
`types of virtual content from their mobile devices as the next
`generation of messaging system to enhance their mobile com
`munication experiences. All virtual content are associated
`with a physical location, and are Superimposed onto the real
`images captured by phone camera when they are displayed on
`the screen.
`0004 Although the appearance of real objects captured by
`the camera reflects the lighting conditions of the environment
`(e.g., they look darker in poor lighting conditions), the virtual
`objects are rendered using a predetermined illumination that
`is not related to the real world lighting conditions. Therefore,
`an effective method of adapting the appearance of virtual
`objects to various lighting conditions of the real environment
`is needed for improving the viewing experience for users in a
`mobile augmented reality messaging system.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`0005 FIG. 1 is a block diagram of a context-aware mes
`Saging System.
`0006 FIG. 2 illustrates an augmented-reality scene.
`0007 FIG. 3 illustrates an augmented-reality scene.
`0008 FIG. 4 is a block diagram of the server of FIG. 1.
`0009 FIG. 5 is a block diagram of the user device of FIG.
`1
`0010 FIG. 6 is a flow chart showing operation of the
`server of FIG. 1.
`0011
`FIG. 7 is a flow chart showing operation of the user
`device of FIG. 1 when creating graffiti.
`0012 FIG. 8 is a flow chart showing operation of the user
`device of FIG. 1 when displaying graffiti.
`
`DETAILED DESCRIPTION OF THE DRAWINGS
`0014. In order to address the above-mentioned need, a
`method and apparatus for messaging within a mobile virtual
`and augmented reality system is provided herein. During
`operationauser can create “virtual graffiti that will be left for
`a particular device to view as part of an augmented-reality
`scene. The virtual graffiti will be assigned to either a particu
`lar physical location or a part of an object that can be mobile.
`The virtual graffiti is then uploaded to a network server, along
`with the location and individuals who are able to view the
`graffiti as part of an augmented-reality Scene.
`(0015. When a device that is allowed to view the graffiti is
`near the location, the graffiti will be downloaded to the device
`and displayed as part of an augmented-reality scene. To fur
`ther enhance the user experience, the virtual graffiti can be
`dynamic, changing based on an ambient light source. For
`example, in an outdoor environment, the context available to
`the mobile device (time, location, and orientation) can be
`acquired in order to determine the Source and intensity of
`natural light and apply it to appropriate surfaces of the virtual
`objects. As the location of a device is already available to
`GPS-enabled phones, and the locations of the virtual objects
`are also known to the system, the viewing direction of each
`virtual object can be calculated in the scene from the device.
`The direction of sunlight, on the other hand, is determined by
`the current date and time as well as the latitude and longitude
`of the device. The position of the sun can be determined from
`Solar ephemeris data and used to position a “virtual Sun' (i.e.,
`an omni-directional light source) in the virtual coordinate
`system used by the rendering software. The intensity of Sun
`light can be adjusted through known attenuation calculations
`that can further be modified based on current local weather
`conditions. Simultaneously, a light sensor could be used to
`determine the ambient light intensity which could also be
`replicated in the virtual environment to give an even more
`accurately illuminated Scene.
`0016. In an augmented reality system, computer generated
`images, or "virtual images' may be embedded in or merged
`with the user's view of the real-world environment to enhance
`the user's interactions with, or perception of the environment.
`In the present invention, the user's augmented reality system
`merges any virtual graffiti messages with the user's view of
`the real world.
`0017. As an example, Ed could leave a message for his
`friends Tom and Joe on a restaurant door Suggesting they try
`the chili. At various times of the day the intensity of the image
`left would be modified based on how much ambient light was
`falling on the restaurant door.
`0018. The present invention encompasses a method for
`modifying a virtual graffiti object. The method comprises the
`steps of obtaining Sun location data, obtaining virtual graffiti,
`and modifying the virtual graffiti based on the Sun location
`data.
`0019. The present invention encompasses a method for
`receiving and displaying virtual graffiti as part of an aug
`mented-reality scene. The method comprises the steps of
`providing a location, receiving virtual graffiti in response to
`the step of providing the location, obtaining ambient-light
`information, modifying the virtual graffiti based on the ambi
`ent-light information, and displaying the modified virtual
`graffiti as part of an augmented-reality Scene.
`
`Niantic's Exhibit No. 1003
`Page 009
`
`

`

`US 2010/0066750 A1
`
`Mar. 18, 2010
`
`0020. The present invention additionally encompasses an
`apparatus for receiving and displaying virtual graffiti as part
`of an augmented-reality Scene. The apparatus comprises a
`transmitter providing a location, a receiver receiving virtual
`graffiti in response to the step of providing the location,
`circuitry determining ambient-light information and modify
`ing the virtual graffiti based on the ambient-light information,
`and an augmented reality system displaying the modified
`virtual graffiti as part of an augmented-reality scene.
`0021 Turning now to the drawings, wherein like numerals
`designate like components, FIG. 1 is a block diagram of
`context-aware messaging system 100. System 100 comprises
`virtual graffiti server 101, network 103, and user devices
`105-109. In one embodiment of the present invention, net
`work 103 comprises a next-generation cellular network,
`capable of high data rates. Such systems include the enhanced
`Evolved Universal Terrestrial Radio Access (UTRA) or the
`Evolved Universal Terrestrial Radio Access Network (UT
`RAN) (also known as EUTRA and EUTRAN) within 3GPP
`along with evolutions of communication systems within other
`technical specification generating organizations (such as
`Phase 2 within 3GPP2, and evolutions of IEEE 802.11,
`802.16, 802.20, and 802.22). User devices 105-109 comprise
`devices capable of real-world imaging and providing the user
`with the real-world image augmented with virtual graffiti.
`0022. During operation, a user (e.g., a user operating user
`device 105) determines that he wishes to send another user
`virtual graffiti as part of an augmented-reality Scene. User
`device 105 is then utilized to create the virtual graffiti and
`associate the virtual graffiti with a location. The user also
`provides device 105 with a list of user(s) (e.g., user 107) that
`will be allowed to view the virtual graffiti. Device 105 then
`utilizes network 103 to provide this information to virtual
`graffiti server 101.
`0023 Server 101 periodically monitors the locations of all
`devices 105-109 along with their identities, and when a par
`ticular device is near a location where it is to be provided with
`virtual graffiti, server 101 utilizes network 103 to provide this
`information to the device. When a particular device is near a
`location where virtual graffiti is available for viewing, the
`device will notify the user, for example, by beeping. The user
`can then use the device to view the virtual graffiti as part of an
`augmented-reality scene. Particularly, the virtual graffiti will
`be embedded in or merged with the user's view of the real
`world. It should be noted that in alternate embodiments, no
`notification is sent to the user. It would then be up to the user
`to find any virtual graffiti in his environment.
`0024 FIG. 2 illustrates an augmented-reality scene. In this
`example, a user has created virtual graffiti 203 that states,
`“Joe, try the porter” and has attached this graffiti to the loca
`tion of a door. As is shown in FIG. 2, the real-world door 201
`does not have the graffiti existing upon it. However, if a user
`has privileges to view the virtual graffiti, then their augmented
`reality viewing system will show door 201 having graffiti 203
`upon it. Thus, the virtual graffiti is not available to all users of
`system 100. The graffiti is only available to those designated
`able to view it (preferably by the individual who created the
`graffiti). Each device 105-109 will provide a unique aug
`mented-reality scene to their user. For example, a first user
`may view a first augmented-reality Scene, while a second user
`may view a totally different augmented-reality Scene (e.g.,
`the user may have left another message 205 for another user).
`This is illustrated in FIG. 2 with graffiti 205 being different
`than graffiti 203. Thus, a first user, looking at door 201 may
`
`view graffiti 203, while a second user, looking at the same
`door 201 may view graffiti 205.
`0025. Although the above example was given with virtual
`graffiti 203 displayed on a particular object (i.e., door 201), in
`alternate embodiments of the present invention, virtual graf
`fiti may be displayed unattached to any object. For example,
`graffiti may be displayed as floating in the air, or simply in
`front of a person's field of view. Additionally, although the
`virtual graffiti of FIG. 2 comprises text, the virtual graffiti
`may also comprise a “virtual object’. Such as images, audio
`and video clips, etc.
`0026. As discussed above, to further enhance the user
`experience, the virtual graffiti can be dynamic, changing
`based on the ambient light. For example, the shadowing of a
`virtual object may be allowed to change based on, for
`example, the position of the Sun.
`0027. This is illustrated in FIG. 3. As shown in FIG. 3 a
`first user creates virtual graffiti 301. Virtual graffiti 301 com
`prises at least two parts; a first virtual object (scroll) along
`with virtual text (“try the chili"). Virtual graffiti 301 is
`attached to door 302 and left for a second user to view. As is
`evident, virtual graffiti 301 is displayed with a shadow 303
`that changes with the time of day. For example, door 302
`viewed at a first time of day will have shadow 303 displayed
`to the lower right of graffiti 301. However, door 302 viewed at
`a second time of day will have shadow 303 displayed to the
`lower left of graffiti 301.
`0028. It should be noted that the above example was given
`with respect to the virtual graffiti changing its shadow in
`response to ambient light, however, in alternate embodiments
`of the present invention virtual graffiti 301 may change any
`combination of shadow, brightness, contrast, color, specular
`highlights, or texture maps in response to the ambient light.
`Additionally, in one embodiment of the present invention, the
`virtual graffiti is modified in response to ambient light by the
`device 105-109 viewing the virtual graffiti, however in
`another embodiment, the virtual graffiti is modified by server
`101 prior to being transmitted to devices 105-109.
`(0029. As is evident, for any particular device 105-109 to
`be able to display virtual graffiti attached to a particular “real
`object, the device must be capable of identifying the object's
`location, and then displaying the graffiti at the object's loca
`tion. There are several methods for accomplishing this task.
`In one embodiment of the present invention, this is accom
`plished via the technique described in US2007/0024527,
`METHOD AND DEVICE FOR AUGMENTED REALITY MESSAGE
`HIDING AND REVEALING by the augmented reality system
`using vision recognition to attempt to match the originally
`created virtual graffiti to the user's current environment. For
`example, the virtual graffiti created by a user may be uploaded
`to server 101 along with an image of the graffiti's surround
`ings. The image of the graffiti's Surroundings along with the
`graffiti can be downloaded to a user's augmented reality sys
`tem, and when a user's Surroundings match the image of the
`graffiti's Surroundings, the graffiti will be appropriately dis
`played.
`0030. In another embodiment of the present invention the
`attachment of the virtual graffiti to a physical object is accom
`plished by assigning the physical coordinates of the physical
`object (assumed to be GPS, but could be some other system)
`to the virtual graffiti. The physical coordinates must be con
`verted into virtual coordinates used by the 3D rendering sys
`tem that will generate the augmented-reality Scene (one Such
`3D rendering system is the Java Mobile 3D Graphics, or
`
`Niantic's Exhibit No. 1003
`Page 0010
`
`

`

`US 2010/0066750 A1
`
`Mar. 18, 2010
`
`M3G, API specifically designed for use on mobile devices).
`The most expedient way to accomplish the coordinate con
`version is to set the virtual X coordinate to the longitude, the
`virtual y coordinate to the latitude, and the virtual Z coordi
`nate to the altitude thus duplicating the physical world in the
`virtual world by placing the origin of the virtual coordinate
`system at the center of the earth so that the point (0,0,0) would
`correspond the point where the equator and the prime merid
`ian cross, projected onto the center of the earth. This would
`also conveniently eliminate the need to perform computation
`ally expensive transformations from physical coordinates to
`virtual coordinates each time a virtual graffiti message is
`processed.
`0031. As previously mentioned, the physical coordinate
`system is assumed to be GPS, but GPS may not always be
`available (e.g., inside buildings). In Such cases, any other
`Suitable location system can be substituted, such as, for
`example, a WiFi-based indoor location system. Such a system
`could provide a location offset (Xoyo.Z.o.) from a fixed refer
`ence point (x,y,z) whose GPS coordinates are known.
`Whatever coordinate system is chosen, the resultant coordi
`nates will always be transformable into any other coordinate
`system.
`0032. After obtaining the virtual coordinates of the virtual
`graffiti, a viewpoint must be established for the 3D rendering
`system to be able to render the virtual scene. The viewpoint
`must also be specified in virtual coordinates and is completely
`dependent upon the physical position and orientation (i.e.,
`viewing direction) of the device. If the viewpoint faces the
`virtual graffiti, the user will see the virtual graffiti from the
`viewpoint's perspective. If the user moves toward the virtual
`graffiti, the virtual graffiti will appear to increase in size. If the
`user turns 180 degrees in place to face away from the virtual
`graffiti, the virtual graffiti will no longer be visible and will
`not be displayed. All of these visual changes are automati
`cally handled by the 3D rendering system based on the view
`point.
`0033) Given a virtual scene containing virtual graffiti (at
`the specified virtual coordinates) and a viewpoint, the 3D
`rendering system can produce a view of the virtual scene
`unique to the user. This virtual scene must be overlaid onto a
`view of the real world to produce an augmented-reality Scene.
`One method to overlay the virtual scene onto a view of the real
`world from the mobile device's camera is to make use of an
`M3G background object which allows any image to be placed
`behind the virtual scene as its background. Using the M3G
`background, continuously updated frames from the camera
`can be placed behind the virtual scene, thus making the scene
`appear to be overlaid on the camera output.
`0034. Given the above information, a device's location is
`determined and sent to the server. The server determines what
`messages, if any, are in proximity to and available for the
`device. These messages are then downloaded by the device
`and processed. The processing involves transforming the
`physical locations of the virtual messages into virtual coor
`dinates. The messages are then placed at those virtual coor
`dinates. At the same time, the device's position and its orien
`tation are used to define a viewpoint into the virtual world also
`in virtual coordinates. If the downloaded virtual message is
`visible from the given viewpoint, it is rendered on a mobile
`device's display on top of live video of the scene from the
`device's camera.
`0035. Thus, if the user wants to place a virtual message on
`the top of an object, the user must identify the location of the
`
`point on top of the object where the message will be left. In the
`simplest case, the user can place his device on the object and
`capture the location. He then sends this location with the
`virtual object and its associated content (e.g., a beer Stein with
`the text message “try the porter” applied to the southward
`facing side of the stein) to the server. The user further speci
`fies that the message be available for a particular user. When
`the particular user arrives at the bar and is within range of the
`message, they will see the message from their location (and,
`therefore, their viewpoint). If they are looking toward the
`eastward-facing side of the message, they will see the Stein,
`but will just be able to tell that there is some text message on
`the Southern side. If a user wishes to read the text message,
`they will have to move their device (and thus their viewpoint)
`so that it is facing the Southern side of the Stein.
`0036 FIG. 4 is a block diagram of a server of FIG.1. As is
`evident, server 101 comprises a global object manager 401,
`database 403, personal object manager 405, and optional
`ambient light modification circuitry 411. During operation,
`global object manager 401 will receive virtual graffiti from
`any device 105-109 wishing to store graffiti on server 101.
`This information is preferably received wirelessly through
`receiver 407. Global object manager 401 is responsible for
`storing all virtual graffiti existing within system 100. Along
`with the virtual graffiti, global object manager 401 will also
`receive a location for the graffiti along with a list of devices
`that are allowed to display the graffiti. Again, this information
`is preferably received wirelessly through receiver 407. If the
`graffiti is to be attached to a particular item (moving or sta
`tionary), then the information needed for attaching the virtual
`graffiti to the object will be received as well. For the first
`embodiment, a digital representation of a stationary item's
`surroundings will be stored; for the second embodiment, the
`physical location of moving or stationary virtual graffiti will
`be stored. All of the above information is stored in database
`403.
`0037 Although only one personal object manager 405 is
`shown in FIG. 4, it is envisioned that each user device will
`have its own personal object manager 405. Personal object
`manager 405 is intended to serve as an intermediary between
`its corresponding user device and global object manager 401.
`Personal object manager 405 will periodically receive a loca
`tion for its corresponding user device. Once personal object
`manager 405 has determined the location of the device, per
`Sonal object manager 405 will access global object manager
`401 to determine if any virtual graffiti exists for the particular
`device at, or near the device's location. Personal object man
`ager 405 filters all available virtual graffiti in order to deter
`mine only the virtual graffiti relevant to the particular device
`and the device's location. Personal object manager 405 then
`provides the device with the relevant information needed to
`display the virtual graffiti based on the location of the device,
`wherein the relevant virtual graffiti changes based on the
`identity and location of the device. This information will be
`provided to the device by instructing transmitter 409 to trans
`mit the information wirelessly to the device. It should be
`noted that if server 101 is to modify the graffiti based on
`ambient light, circuitry 411 will modify the graffiti before
`being transmitted.
`0038 FIG. 5 is a block diagram of a user device of FIG.1.
`As shown, the user device comprises augmented reality sys
`tem 515, context-aware circuitry 509, ambient light modifi
`cation circuitry 507, graffiti database 508, logic circuitry 505,
`transmitter 511, receiver 513, and user interface 517. Con
`
`Niantic's Exhibit No. 1003
`Page 0011
`
`

`

`US 2010/0066750 A1
`
`Mar. 18, 2010
`
`text-aware circuitry 509 may comprise any device capable of
`generating a current context for the user device. For example,
`context-aware circuitry 509 may comprise a GPS receiver
`capable of determining a location of the user device. Alterna
`tively, circuitry 509 may comprise such things as a clock, a
`thermometer capable of determining an ambient temperature,
`an internet connection capable of determining the current
`weather, a Sun position calculator, a light detector, a biometric
`monitor Such as a heart-rate monitor, an accelerometer, a
`barometer, a connection to an application that determines if
`the user is indoors or outdoors, etc.
`0039. During operation, a user of the device creates virtual
`graffiti via user interface 517. The virtual graffiti preferably,
`but not necessarily, comprises at least two parts, a virtual
`object and content. The virtual object is a 3D object model
`that can be a primitive polygon or a complex polyhedron
`representing an avatar, for example. The content is preferably
`either text, pre-stored images Such as clip art, pictures, pho
`tos, audio or video clips,..., etc. The virtual object and its
`associated content comprise virtual graffiti that is stored in
`graffiti database 508. In one embodiment of the present inven
`tion, user interface 517 comprises an electronic tablet capable
`of obtaining virtual objects from graffiti database 508 and
`creating handwritten messages and/or pictures.
`0040. Once logic circuitry 505 receives the virtual graffiti
`from user interface 517 or graffiti database 508, logic cir
`cuitry 505 accesses context-aware circuitry 509 and deter
`mines a location where the graffiti was created (for stationary
`graffiti) or the device to which the virtual graffiti will be
`attached (for mobile graffiti). Logic circuitry 505 also
`receives a list of users with privileges to view the graffiti. This
`list is also provided to logic circuitry 505 through user inter
`face 517.
`0041. In one embodiment of the present invention the vir
`tual graffiti is associated with a physical object. When this is
`the case, logic circuitry 505 will also receive information
`required to attach the graffiti to an object. Finally, the virtual
`graffiti is provided to virtual graffiti server 101 by logic cir
`cuitry 505 instructing transmitter 511 to transmit the virtual
`graffiti, the location, the list of users able to view the graffiti,
`and if relevant, the information needed to attach the graffiti to
`an object. As discussed above, server 101 periodically moni
`tors the locations of all devices 105-109 along with their
`identities, and when a particular device is near a location
`where it is to be provided with virtual graffiti, server 101
`utilizes network 103 to provide this information to the device.
`0042. When a particular device is near a location where
`virtual graffiti is available for viewing, the device will notify
`the user, for example, by instructing user interface 517 to
`beep. The user can then use the device to view the virtual
`graffiti as part of an augmented-reality Scene. Thus, when the
`device of FIG. 5 is near a location where virtual graffiti is
`available for it, receiver 513 will receive the graffiti and the
`location of the graffiti from server 101. If relevant, receiver
`513 will also receive information needed to attach the graffiti
`to a physical object. This information will be passed to logic
`circuitry 505.
`0043 Receiver 513 will receive virtual graffiti and its loca
`tion. Logic circuitry 505 will store this graffiti within graffiti
`database 508. Logic circuitry 505 periodically accesses con
`text-aware circuitry 509 to get updates to its location and
`provides these updates to server 101. When logic circuitry
`505 determines that the virtual graffiti should be displayed, it
`will access ambient light modification circuitry 507, causing
`
`circuitry 507 to update the virtual graffiti based on the ambi
`ent light. The user can then use augmented reality system 515
`to display the updated graffiti. More particularly, imager 503
`will image the current background and provide this to display
`501. Display 501 will also receive the virtual graffiti from
`graffiti database 508 and provide an image of the current
`background with the graffiti appropriately displayed. Thus,
`the virtual graffiti will be embedded in or merged with the
`user's view of the real-world.
`
`Modification of Virtual Graffiti Based on Ambient Light
`0044 As discussed above, to further enhance the user
`experience, the virtual graffiti can be dynamic, changing
`based on the ambient light. When modification to the virtual
`graffiti is to take place via a user device, each user device will
`comprise ambient light modification circuitry 507 to perform
`this task. However, when modification to the virtual graffiti is
`to take place via server 101, server 101 will modify the graffiti
`via ambient light modification circuitry 411 prior to sending
`the virtual graffiti to the user device. Regardless of where the
`virtual graffiti gets modified based on the ambient light; cir
`cuitry will perform the following steps in order to make the
`modification.
`0.045
`Optionally determining if the device is indoors or
`outdoors. This determination can be made in several
`ways. In one embodiment of the present invention this
`determination can be made by accessing context-aware
`circuitry 509 and determining GPS coordinates for the
`device. From the GPS coordinates a point