(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2013/0124326A1
`HUANG et al.
`(43) Pub. Date:
`May 16, 2013
`
`US 2013 0124326A1
`
`(54) PROVIDING ADVERTISEMENTS IN AN
`AUGMENTED REALITY ENVIRONMENT
`(75) Inventors: Wyatt (Ling-Wei) HUANG, Taipei
`(TW); Balduran (Chia-Chun)
`CHANG, Hsinchu (TW); Connie
`(Shih-Ting) HUANG, Taipei (TW)
`(73) Assignee: Yahoo! Inc., Sunnyvale, CA (US)
`(21) Appl. No.: 13/296,236
`(22) Filed:
`Nov. 15, 2011
`Publication Classification
`
`(51) Int. Cl.
`G06O 30/02
`G09G 5/00
`
`(2012.01)
`(2006.01)
`
`(52) U.S. Cl.
`USPC ........................ 705/14.64; 345/633; 705/14.4
`
`(57)
`
`ABSTRACT
`
`A computer-implemented method for providing advertise
`ments in an augmented reality environment to a user includes
`receiving data related to a marker, the marker placed amongst
`one or more physical objects captured by the video camera.
`The computer-implemented method also includes retrieving
`dynamic digital content associated with the marker. Further,
`the computer-implemented method includes displaying the
`dynamic digital content amongst the one or more physical
`objects. Furthermore, the computer-implemented method
`includes receiving a user interaction with the dynamic digital
`content. Moreover, the computer-implemented method
`includes performing an action based on the user interaction.
`
`310
`
`Augmented
`Reality
`
`S
`
`S
`
`&
`
`XXXX: ...' assas
`M
`wV
`
`S
`s
`
`R
`W
`
`RSR
`
`S&S S.
`
`SS:
`&:
`
`&
`
`&: S:
`...'.::::S8
`
`S.
`
`88: 8
`
`s
`
`s
`
`
`
`W
`
`Niantic's Exhibit No. 1037
`Page 001
`
`

`

`Patent Application Publication May 16, 2013 Sheet 1 of 6
`
`US 2013/O12432.6 A1
`
`
`
`
`
`RECEIVING DATA RELATED TO A MARKER THE MARKER
`PLACED AMONGST ONE OR MORE PHYSICAL OBJECTS
`CAPTURED BY THE WIDEO CAMERA
`
`RETRIEVING DYNAMIC. DIGITAL CONTENT ASSOCIATED
`WITH THE MARKER
`
`DISPLAYING THE DYNAMIC. DIGITAL CONTENT AMONGST
`THE ONE OR MORE PHYSICAL OBJECTS
`
`RECEIVING A USER INTERACTION WITH THE DYNAMIC
`DIGITAL CONTENT
`
`PERFORMING AN ACTION BASED ON THE USER
`INTERACTION
`
`110
`
`115
`
`120
`
`125
`
`130
`
`FIG. 1
`
`Niantic's Exhibit No. 1037
`Page 002
`
`

`

`Patent Application Publication May 16, 2013 Sheet 2 of 6
`
`US 2013/O12432.6 A1
`
`
`
`SS $8 SS &
`
`Niantic's Exhibit No. 1037
`Page 003
`
`

`

`Patent Application Publication
`
`May 16, 2013 Sheet 3 of 6
`
`US 2013/O12432.6 A1
`
`{ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * ;
`
`
`
`
`
`W
`
`X
`
`Augmented
`Reality
`
`arrara-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-
`
`32O
`
`EIG
`
`3
`
`Niantic's Exhibit No. 1037
`Page 004
`
`

`

`Patent Application Publication May 16, 2013 Sheet 4 of 6
`
`US 2013/O12432.6 A1
`
`
`
`
`
`
`
`Mass storage
`and AR frame
`work
`
`Download AR 3D
`model and digital
`Content
`
`NA
`V-N
`
`
`
`High reliability
`image feature
`
`N
`
`410
`
`OR-Code
`or marker
`
`FIG. 4
`
`Niantic's Exhibit No. 1037
`Page 005
`
`

`

`Patent Application Publication May 16, 2013 Sheet 5 of 6
`
`US 2013/O12432.6 A1
`
`
`
`
`
`Mass storage
`and AR frame
`work
`
`2. response content
`
`
`
`sarayaraaraaraaraaraaraaraaraaraarara.
`
`1. N
`
`SES
`$888
`
`
`
`Sas
`
`FIG 5
`
`Niantic's Exhibit No. 1037
`Page 006
`
`

`

`Patent Application Publication May 16, 2013 Sheet 6 of 6
`
`US 2013/O12432.6 A1
`
`... "...Y
`
`HARD
`PROCESS OR
`m
`61 O
`o DRTVE
`
`62O
`
`COMPUTTNG DEVICE
`61 O
`
`MEMORY
`64 O
`
`OS MODULE
`642
`
`T/O PORT
`63 O
`
`T NETWORK
`CARD
`632
`
`MOUSE
`KEYBOARD
`CONTROTTER CONTROTTER
`634
`636
`
`f
`
`6,351
`
`GPS CARD
`638
`
`-
`
`FIG. 6
`
`Niantic's Exhibit No. 1037
`Page 007
`
`

`

`US 2013/0124326 A1
`
`May 16, 2013
`
`PROVIDING ADVERTISEMENTS IN AN
`AUGMENTED REALITY ENVIRONMENT
`
`TECHNICAL FIELD
`0001 Embodiments of the disclosure relate generally, to
`targeted advertising, and more specifically, advertisements in
`an augmented reality environment.
`
`BACKGROUND
`0002 Augmented Reality (AR) enhances a person’s cur
`rent perception of reality with computer generated output
`with, for example, Sound, video and graphics. AR gives a live
`view of a physical, real-world environment where objects in
`the environment are augmented. Artificial information about
`environment and associated physical objects can be overlaid
`on the real world. With the help ofAR, information around the
`real world becomes interactive. AR technologies have been
`enhancing interaction of users with the real world using com
`puter generated virtual objects. Moreover, usage of AR to
`promote products with interactive AR applications is becom
`ing common. Shopping environment using AR, however,
`poses more challenges.
`0003 Various traditional methods of advertising exist, for
`example, light box advertising, posterior or exterior wall
`advertising, and out-of-home advertising. However, the tra
`ditional methods have only fixed content that are displayed to
`the user. Further, advertisements are not interactive with the
`user. Interactive advertisements stills remain a challenge.
`0004 Inlight of the foregoing discussion, there is a need to
`dynamically provide advertisements in an augmented reality
`environment.
`
`SUMMARY
`0005. The above-mentioned needs are met by a method,
`system, and computer program product for providing inter
`active advertisements in an augmented reality environment.
`0006 An example of a computer-implemented method for
`providing advertisements in an augmented reality environ
`ment includes receiving data related to a marker, the marker
`placed amongst one or more physical objects captured by the
`Video camera. The computer-implemented method also
`includes retrieving dynamic digital content associated with
`the marker. Further, the computer-implemented method
`includes displaying the dynamic digital content amongst the
`one or more physical objects. Furthermore, the computer
`implemented method includes receiving a user interaction
`with the dynamic digital content. Moreover, the computer
`implemented method includes performing an action based on
`the user interaction.
`0007 An example of a computer program product stored
`on a non-transitory computer-readable medium that when
`executed by a processor, performs a method for providing
`advertisements in an augmented reality environment,
`includes data related to a marker, the marker placed amongst
`one or more physical objects captured by the video camera.
`The computer program product also includes retrieving
`dynamic digital content associated with the marker. Further,
`the computer program product includes displaying the
`dynamic digital content amongst the one or more physical
`objects. Furthermore, the computer program product includes
`receiving a user interaction with the dynamic digital content.
`Moreover, the computer program product includes perform
`ing an action based on the user interaction.
`
`0008. An example of a system for providing advertise
`ments in an augmented reality environment includes a video
`camera to receive data related to a marker, the marker placed
`amongst one or more physical objects captured by the video
`camera. The system also includes a network interface to
`retrieve dynamic digital content associated with the marker.
`Further, the system includes a display device to display the
`dynamic digital content amongst the one or more physical
`objects. Furthermore, the system includes a user interface to
`receive a user interaction with the dynamic digital content.
`Moreover, the system includes an interaction module to per
`form an action based on the user interaction
`0009 Advantageously, a user can view and interact with
`advertisements in an augmented reality environment.
`0010. The features and advantages described in this sum
`mary and in the following detailed description are not all
`inclusive, and particularly, many additional features and
`advantages will be apparent to one of ordinary skill in the
`relevant art in view of the drawings, specification, and claims
`hereof. Moreover, it should be noted that the language used in
`the specification has been principally selected for readability
`and instructional purposes, and may not have been selected to
`delineate or circumscribe the inventive subject matter, resort
`to the claims being necessary to determine Such inventive
`Subject matter.
`
`BRIEF DESCRIPTION OF THE FIGURES
`0011. In the following drawings like reference numbers
`are used to refer to like elements. Although the following
`figures depict various examples of the invention, the inven
`tion is not limited to the examples depicted in the figures.
`0012 FIG. 1 is a flow diagram illustrating a method for
`providing advertisements in an augmented reality environ
`ment, in accordance with one embodiment;
`0013 FIG. 2 is a schematic diagram illustrating interac
`tion of an augmented reality model with physical world, in
`accordance with one embodiment;
`0014 FIG. 3 is a schematic diagram illustrating markers
`and corresponding dynamic digital content, in accordance
`with one embodiment;
`0015 FIG. 4 is a schematic diagram illustrating a method
`to retrieve augmented reality contents, in accordance with
`one embodiment;
`0016 FIG. 5 is an exemplary representation of user inter
`action on dynamic digital content, in accordance with one
`embodiment; and
`0017 FIG. 6 is a block diagram illustrating a mobile com
`puting device, in accordance with one embodiment.
`
`DETAILED DESCRIPTION OF THE
`EMBODIMENTS
`0018. A method, system, and computer program product
`for providing advertisements in an augmented reality envi
`ronment to a user is disclosed. The augmented reality envi
`ronment refers, in one embodiment, to a live view of a physi
`cal, real-world environment which is modified by a device
`having a video camera and display.
`0019. The following detailed description is intended to
`provide example implementations to one of ordinary skill in
`the art, and is not intended to limit the invention to the explicit
`disclosure, as one or ordinary skill in the art will understand
`that variations can be substituted that are within the scope of
`the invention as described.
`
`Niantic's Exhibit No. 1037
`Page 008
`
`

`

`US 2013/0124326 A1
`
`May 16, 2013
`
`0020 FIG. 1 is a flow diagram illustrating a method for
`providing advertisements in an augmented reality environ
`ment, in accordance with one embodiment.
`0021. At step 110, data related to a marker is received. The
`marker is placed amongst one or more physical objects and
`then captured by a video camera, a photograph, a video
`stream, or just a still frame.
`0022. A user of the device places the video camera over the
`one or more physical objects in real world. The physical
`objects are fitted with markers. The markers are specific pat
`terns that are detected by augmented reality. The video cam
`era captures the physical object and data related to the marker
`is received. The data includes, but is not limited to, position,
`rotation and scale of the marker.
`0023. In one embodiment, the marker includes a Quick
`Response (QR) code marker. The QR code marker follows a
`specific pattern in the physical object and is represented as
`black and white squares. The black and white squares repre
`sent an encoded string of characters. The characters are first
`encoded as bits (Zeros and ones) and laterprinted as the black
`and white squares.
`0024. A marker can be recognized by, for example, a
`mobile application executing on a cellphone or PDA. In one
`embodiment, the application is always detecting markers, and
`in another embodiment, the application detects markers upon
`request by a user.
`0025. At step 115, dynamic digital content associated with
`the marker, such as an interactive advertisement, is retrieved.
`0026. The marker is converted to a network address. Using
`the network address the dynamic digital content is retrieved.
`In one embodiment, a Universal Resource Locator (URL) is
`retrieved which leads to a website. The URL can be accessed
`and downloaded over a data network, a cellular network, or
`the like. The dynamic digital content can be stored locally in
`RAM for immediate display, or stored in memory as a
`prefetch for a later display.
`0027. The digital content is considered dynamic because,
`in one embodiment, the digital content is constantly updated.
`As a result, dynamic data retrieved for a first user can differ
`from dynamic data retrieved for a second user. Many factors
`can be used to determine which dynamic data to send respon
`sive to a request. Exemplary factors include user profile, time,
`day, location (e.g., as determined by an integrated GPS),
`mood, preferences, and other context factors.
`0028. At step 120, the dynamic digital content is displayed
`amongst the one or more physical objects. In one embodi
`ment, a user can see real objects integrated in a video display
`with augmented reality objects. One example is discussed
`below with respect to FIG. 2. The video display can be at any
`resolution Such as Standard, high definition, or 3-dimen
`sional, and of any format such as MP4.
`0029. A mobile application can include a video renderer.
`In operation, the video renderer combines, or overlays mul
`tiple video streams into a single stream. One stream is the real
`world video and another stream is the augmented reality
`Video generated from the dynamic digital content. Based on
`the movements of the video camera, in relation to the mark
`ers, display of the dynamic digital content is adjusted. As the
`user moves the video camera across the physical objects, step
`110, step 115 and step 120 are performed. For example,
`consider the video camera placed over an image of spacetti in
`a magazine. An advertisement related to different types of
`spaghetti is displayed. At this point, the video camera is
`moved to another image of sports shoes in the magazine.
`
`Consequently, an advertisement related to various brands of
`the sports shoes is displayed to the user.
`0030 A variation of the dynamic digital content is selected
`in accordance with at least one of time, date, location, infor
`mation about a user, a user preference, and display capability
`of a device.
`0031. At step 125, a user interaction with the dynamic
`digital content is received.
`0032. The dynamic digital content includes a mechanism
`to make a purchase. For example, the user can buy a product
`displayed in the advertisement using a BUY button.
`0033. At step 130, an action is performed based on the user
`interaction.
`0034 Examples of the action includes, but are not limited
`to, purchasing a product, posting a comment related to an
`advertisement and a commenting on Twitter.
`0035 FIG. 2 is a schematic diagram illustrating interac
`tion of an augmented reality model with physical world, in
`accordance with one embodiment.
`0036. A mobile device 210 is placed in the augmented
`reality environment 220. A camera in the mobile device 210
`captures markers from the augmented reality environment
`220 and, in response, displays a girl 230 sitting on a physical
`box.
`0037 FIG. 3 is a schematic diagram illustrating markers
`and corresponding dynamic digital content, in accordance
`with one embodiment.
`0038 A plurality of markers 310 represents specific pat
`terns of images that are detected by augmented reality. For
`each of the markers 310 corresponding augmented reality
`dynamic digital content is shown. The dynamic digital con
`tent includes advertisements 320. Further, each of the mark
`ers 310 is unique and corresponds to one of the advertise
`ments 320.
`0039. The advertisements 320 are dynamic. Content of the
`advertisements 320 can change based on a plurality of factors,
`for example, time and location of video camera.
`0040 FIG. 4 is a schematic diagram illustrating a method
`to retrieve augmented reality contents, in accordance with
`one embodiment.
`0041. A mobile device 410 captures a marker or a (Quick
`Response) QR code marker in an augmented environment.
`The QR code marker follows a specific pattern in the physical
`object and is represented as black and white squares. The
`black and white squares represent an encoded string of char
`acterS.
`0042. In one embodiment, the mobile device 410 can also
`be a Personal Computer (PC), a laptop or notebook computer,
`a tablet computer, a Smartphone or Personal Digital Assistant
`(PDA), a Smart appliance, a video gaming console, an Internet
`television, a set-up box, or any other Suitable processor-based
`device that can view advertisements.
`0043 Based on the captured marker or QR code marker, a
`high reliability image feature is sent to a mass storage and
`augmented reality framework. At this point, the mobile
`device 410 retrieves augmented reality 3D model and
`dynamic digital content from the mass storage and aug
`mented reality framework.
`0044 FIG. 5 is an exemplary representation of user inter
`action on dynamic digital content, in accordance with one
`embodiment.
`0045. The dynamic digital content is retrieved by the
`mobile device 410 as explained in FIG. 4. At this point, a user
`of the mobile device 410 performs an action. Examples of the
`
`Niantic's Exhibit No. 1037
`Page 009
`
`

`

`US 2013/0124326 A1
`
`May 16, 2013
`
`action include, but are not limited to, movements of the video
`camera, change in location of the mobile device 410, and
`capturing of a new marker.
`0046 Data related to the action performed by the user is
`then sent to a mass storage and Augmented Reality frame
`work. Based on the data, response content is sent to the
`mobile device 410. The response content refers to an interac
`tion that the user performs. For example, the user can now buy
`product 510.
`0047 FIG. 6 is a block diagram illustrating a mobile com
`puting device 610, in accordance with one embodiment.
`0048. The computing device 610 includes a processor 610,
`a hard drive 620, an I/O port 630, and a memory 640, coupled
`by a bus 699.
`0049. The bus 699 can be soldered to one or more moth
`erboards. Examples of the processor 610 includes, but is not
`limited to, a general purpose processor, an application-spe
`cific integrated circuit (ASIC), an FPGA (Field Program
`mable Gate Array), a RISC (Reduced Instruction Set Control
`ler) processor, or an integrated circuit. There can be a single
`core or multiple cores processor. In one embodiment, the
`processor 610 is specially Suited for processing demands of
`location-aware reminders (for example, custom micro-code,
`and instruction fetching, pipelining or cache sizes). The pro
`cessor 610 can be disposed on silicon or any other suitable
`material. In operation, the processor 610 can receive and
`execute instructions and data stored in the memory 640 or the
`hard drive 620. The hard drive 620 can be a platter-based
`storage device, a flash drive, an external drive, a persistent
`memory device, or other types of memory.
`0050. The hard drive 620 provides persistent (long term)
`storage for instructions and data. The I/O port 630 is an
`input/output panel including a network card 632 with an
`interface 633, along with a keyboard controller 634, a mouse
`controller 636, and a GPS card 638. The network card 632 can
`be, for example, a wired networking card (for example, a USB
`card, or an IEEE 802.3 card), a wireless networking card (for
`example, an IEEE 802.11 card, or a Bluetooth card), and a
`cellular networking card (for example, a 3G card). The inter
`face 633 is configured according to networking compatibility.
`For example, a wired networking card includes a physical
`port to plugin a cord, and a wireless networking card includes
`an antennae. The network card 632 provides access to a
`communication channel on a network. The keyboard control
`ler 634 can be coupled to a physical port 635 (for example,
`PS/2 or USB port) for connecting a keyboard. The keyboard
`can be a standard alphanumeric keyboard with 101 or 104
`keys (including alphabetic, numerical and punctuation keys, a
`space bar, modifier keys, etc.), a laptop or notebook keyboard,
`a thumb-sized keyboard, a virtual keyboard, or the like. The
`mouse controller 636 can also be coupled to a physical port
`637 (e.g., mouse or USB port). The GPS card 638 provides
`communication to GPS satellites operating in space to receive
`location data. An antenna 639 provides radio communica
`tions (or alternatively, a data port can receive location infor
`mation from a peripheral device).
`0051. The memory 640 can be a RAM (Random Access
`Memory), a flash memory, a non-persistent memory device,
`or other devices capable of storing program instructions being
`executed. The memory 640 comprises an Operating System
`(OS) module 642 along with a timer unit 610. In other
`embodiments, the memory 640 comprises a calendar appli
`cation that manages a plurality of appointments. The OS
`module 642 can be one of Microsoft Windows(R family of
`
`operating systems (for example, Windows 95, 98, Me, Win
`dows NT, Windows 2000, Windows XP, Windows XP x64
`Edition, Windows Vista, Windows CE, Windows Mobile),
`Linux, HP-UX, UNIX, Sun OS, Solaris, Mac OS X, Alpha
`OS, AIX, IRIX32, or IRIX64.
`0052. As described herein, computer software products
`can be written in any of various Suitable programming lan
`guages, such as C, C++, C#, Pascal, Fortran, Perl, Matlab
`(from MathWorks), SAS, SPSS, JavaScript, AJAX, and Java.
`The computer Software product may be an independent appli
`cation with data input and data display modules. Alterna
`tively, the computer Software products can be classes that can
`be instantiated as distributed objects. The computer software
`products can also be component Software, for example Java
`Beans (from Sun Microsystems) or Enterprise Java Beans
`(EJB from Sun Microsystems). Many functionalities
`described herein can be implemented in computer software,
`computer hardware, or a combination.
`0053. Furthermore, a computer that is running the previ
`ously mentioned computer Software can be connected to a
`network and can interface to other computers using the net
`work. The network can be an intranet, internet, or the Internet,
`among others. The network can be a wired network (for
`example, using copper), telephone network, packet network,
`an optical network (for example, using optical fiber), or a
`wireless network, or a combination of such networks. For
`example, data and other information can be passed between
`the computer and components (or steps) of a system using a
`wireless network based on a protocol, for example Wi-Fi
`(IEEE standards 802.11, 802.11a, 802.11b, 802.11e, 802.
`11g, 802.11i, and 802.11n). In one example, signals from the
`computer can be transferred, at least in part, wirelessly to
`components or other computers.
`0054 As will be understood by those familiar with the art,
`the invention may be embodied in other specific forms with
`out departing from the spirit or essential characteristics
`thereof. Likewise, the particular naming and division of the
`portions, modules, agents, managers, components, functions,
`procedures, actions, layers, features, attributes, methodolo
`gies and other aspects are not mandatory or significant, and
`the mechanisms that implement the invention or its features
`may have different names, divisions and/or formats.
`0055. Furthermore, as will be apparent to one of ordinary
`skill in the relevant art, the portions, modules, agents, man
`agers, components, functions, procedures, actions, layers,
`features, attributes, methodologies and other aspects of the
`invention can be implemented as Software, hardware, firm
`ware or any combination of the three. Of course, wherever a
`component of the present invention is implemented as Soft
`ware, the component can be implemented as a script, as a
`standalone program, as part of a larger program, as a plurality
`of separate scripts and/or programs, as a statically or dynami
`cally linked library, as a kernel loadable module, as a device
`driver, and/or in every and any other way known now or in the
`future to those of skill in the art of computer programming.
`Additionally, the present invention is in no way limited to
`implementation in any specific programming language, or for
`any specific operating system or environment.
`0056 Furthermore, it will be readily apparent to those of
`ordinary skill in the relevant art that where the present inven
`tion is implemented in whole or in part in software, the
`Software components thereof can be stored on computer read
`able media as computer program products. Any form of com
`puter readable medium can be used in this context, such as
`
`Niantic's Exhibit No. 1037
`Page 0010
`
`

`

`US 2013/0124326 A1
`
`May 16, 2013
`
`magnetic or optical storage media. Additionally, Software
`portions of the present invention can be instantiated (for
`example as object code or executable images) within the
`memory of any programmable computing device.
`0057 Accordingly, the disclosure of the present invention
`is intended to be illustrative, but not limiting, of the scope of
`the invention, which is set forth in the following claims.
`What is claimed is:
`1. A computer-implemented method for providing adver
`tisements in an augmented reality environment, comprising:
`receiving, by a computing device from a video camera
`device, data related to a marker, the marker placed
`amongst one or more physical objects captured by the
`Video camera;
`retrieving dynamic digital content associated with the
`marker;
`displaying the dynamic digital content amongst the one or
`more physical objects;
`receiving a user interaction with the dynamic digital con
`tent; and
`performing an action based on the user interaction.
`2. The method of claim 1, wherein the dynamic digital
`content comprises an advertisement.
`3. The method of claim 1, wherein the dynamic digital
`content comprises a mechanism to make a purchase, and
`wherein performing the action comprises making a purchase.
`4. The method of claim 1, wherein a variation of the
`dynamic digital content is selected in accordance with at least
`one of time, date, location, information about a user, a user
`preference, and display capability of a device.
`5. The method of claim 1, further comprising: converting
`the marker to a network address, wherein the dynamic digital
`content is retrieved using the network address.
`6. The method of claim 1, wherein the marker comprises a
`QR code marker.
`7. The method of claim 1, wherein displaying the dynamic
`digital content comprises rendering the dynamic digital con
`tent.
`8. The method of claim 1, further comprising:
`adjusting display of the dynamic digital content based on
`movements of the video camera.
`9. A computer program product stored on a non-transitory
`computer-readable medium that when executed by a proces
`Sor, performs a method for providing advertisements in an
`augmented reality environment, comprising:
`receiving, by a computing device from a video camera
`device, data related to a marker, the marker placed
`amongst one or more physical objects captured by the
`Video camera;
`retrieving dynamic digital content associated with the
`marker;
`
`displaying the dynamic digital content amongst the one or
`more physical objects;
`receiving a user interaction with the dynamic digital con
`tent; and
`performing an action based on the user interaction.
`10. The computer program product of claim 9, wherein the
`dynamic digital content comprises an advertisement.
`11. The computer program product of claim 9, wherein the
`dynamic digital content comprises a mechanism to make a
`purchase, and wherein performing the action comprises mak
`ing a purchase.
`12. The computer program product of claim 9, wherein a
`variation of the dynamic digital content is selected in accor
`dance with at least one of time, date, location, information
`about a user, a user preference, and display capability of a
`device.
`13. The computer program product of claim 9, further
`comprising:
`converting the marker to a network address, wherein the
`dynamic digital content is retrieved using the network
`address.
`14. The computer program product of claim 9, wherein the
`marker comprises a QR code marker.
`15. The computer program product of claim 9, wherein
`displaying the dynamic digital content comprises rendering
`the dynamic digital content.
`16. The computer program product of claim 9, further
`comprising:
`adjusting display of the dynamic digital content based on
`movements of the video camera.
`17. A computing device to providing advertisements in an
`augmented reality environment, comprising:
`a video camera to receive data related to a marker, the
`marker placed amongst one or more physical objects
`captured by the video camera;
`a network interface to retrieve dynamic digital content
`associated with the marker,
`a display device to display the dynamic digital content
`amongst the one or more physical objects;
`a user interface to receive a user interaction with the
`dynamic digital content; and
`an interaction module to perform an action based on the
`user interaction.
`18. The computing device of claim 17, wherein the
`dynamic digital content comprises an advertisement.
`19. The computing device of claim 17, wherein a variation
`of the dynamic digital content is selected in accordance with
`at least one of time, date, location, information about a user,
`a user preference, and display capability of a device.
`20. The computing device of claim 17, wherein the com
`puting device comprises a mobile computing device.
`
`k
`
`k
`
`k
`
`k
`
`k
`
`Niantic's Exhibit No. 1037
`Page 0011
`
`