Methods for acquiring, generating and displaying ranked
`
`
`
`
`
`
`
`
`points of interests for use in an augmented reality service
`
`
`provisioning system are described. The method for generat­
`
`
`
`ing such ranked points of interest comprises: receiving feed­
`
`
`back information comprising information on users interac­
`
`
`tions associated with one or more points of interest displayed
`
`
`
`as selectable graphical indicators on a screen of a mobile
`
`
`
`
`
`device and assigning popularity scores to points of interests
`
`
`
`
`identified in said feedback information on the basis of said
`
`
`
`user interactions; associating a point ofinterest in a recording
`
`
`
`
`list, comprising points of interests sent to one or more mobile
`
`
`
`
`
`devices in said augmented reality service provisioning sys­
`
`
`
`tem, with a popularity score assigned to points of interests
`
`
`
`
`identified in said feedback information; and, storing said
`
`
`
`point of interest and said popularity score in a points of
`
`Jun. 2 , 2010 (EP) .................................. 10005743.9
`interest database.
`
`(30)
`
`
`
`1111111111111111 IIIIII IIIII 1111111111 11111 11111 1111111111 lllll lllll 111111111111111 11111111
`c19) United States
`
`
`c12) Patent Application Publication
`US 2013/0073988 Al
`c10) Pub. No.:
`
`(43) Pub. Date: Mar. 21, 2013
`Groten et al.
`
`US 20130073988Al
`
`
`
`Publication Cl assification
`
`(2006.01)
`(2006.01)
`
`(54)ACQUIRING, RANKING AND DISPLAYING
`
`POINTS OF INTEREST FOR USE IN AN
`(51)Int. Cl.
`AUGMENTED REALITY SERVICE
`
`PROVISIONING SYSTEM AND GRAPHICAL
`G06F 3101
`
`USER INTERFACE FOR DISPLAYING SUCH
`RANKED POINTS OF INTEREST
`
`G06F 17130
`(52)U.S. Cl.
`
`
`
`
`USPC .................... 715/753; 707/741; 707/E17.002
`
`(57)
`
`ABSTRACT
`
`(75) Inventors: Dirk Groten, Amsterdam (NL);
`
`
`
`
`Anthony Martin Yann Maes,
`
`Amsterdam (NL); Cari Serene
`
`D avidson, Amsterdam (NL)
`
`
`
`
`
`(73)Assignee: LAYAR B.V., Amsterdam (NL)
`
`
`
`
`
`(21) Appl. No.:13/700,731
`
`(22)PCT Filed:Jun.1, 2011
`
`(86)PCT No.: PCT/EP2011/059155
`
`§ 371 (c)(l),
`(2), ( 4) Date:Nov. 30, 2012
`
`
`
`
`
`Foreign Application Priority Dat a
`
`
`
`204 DutchReal Estate
`
`224
`
`222
`
`208c 208b
`
`206
`
`210
`
`220
`
`216
`
`
`
`Woonoppervlak 20P m2
`
`212
`
`www.RayalMakelaren.com
`
`214
`
`(B)
`
`218
`
`Niantic's Exhibit No. 1036
`Page 001
`
`

`

`Patent Application Publication
`
`Mar. 21, 2013 Sheet 1 of 9
`
`US 2013/0073988A1
`
`
`
`Niantic's Exhibit No. 1036
`Page 002
`
`

`

`Patent Application Publication
`
`Mar. 21, 2013 Sheet 2 of 9
`
`US 2013/0073988A1
`
`ZZZ
`
`990Z
`
`
`
`eZOZ
`
`CIZOZ
`
`OZOZ
`
`pZOZ
`
`Niantic's Exhibit No. 1036
`Page 003
`
`

`

`Patent Application Publication
`
`Mar. 21, 2013 Sheet 3 of 9
`
`US 2013/0073988A1
`
`CO
`CN
`cy
`
`N
`v cy
`
`o
`cy
`
`O
`CN
`cy
`
`co
`cy
`
`CN
`CN
`cy
`
`r
`CN
`cy
`
`r
`v cy
`
`O
`C
`cy
`
`C
`
`cy
`
`cC
`Crd
`cy
`
`r
`Crd
`cy
`
`CN
`o
`cy
`
`S.
`
`C
`O
`CY)
`
`i
`
`Niantic's Exhibit No. 1036
`Page 004
`
`

`

`Patent Application Publication
`
`Mar. 21, 2013 Sheet 4 of 9
`
`US 2013/0073988A1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`LET] [TOT]
`
`- - - - - - - - - -|Z ??7
`
`Niantic's Exhibit No. 1036
`Page 005
`
`

`

`Patent Application Publication
`
`Mar. 21, 2013 Sheet 5 of 9
`
`US 2013/0073988A1
`
`
`
`
`
`
`
`
`
`
`
`Á?ueIndod IOB 139 || 719
`
`Niantic's Exhibit No. 1036
`Page 006
`
`

`

`Patent Application Publication
`
`Mar. 21, 2013 Sheet 6 of 9
`
`US 2013/0073988A1
`
`
`
`| co || || HS || || EG Od || LÈT DOEST DOET DOET
`
`
`
`
`
`
`
`Niantic's Exhibit No. 1036
`Page 007
`
`

`

`Patent Application Publication
`
`US 2013/0073988A1
`
`
`
`ZZ/
`
`O
`O
`n
`
`/ ?un 31H
`
`Niantic's Exhibit No. 1036
`Page 008
`
`

`

`Patent Application Publication
`
`Mar. 21, 2013 Sheet 8 of 9
`
`US 2013/0073988A1
`
`90
`8
`
`808
`
`808
`
`
`
`8 ?In8|+
`
`Niantic's Exhibit No. 1036
`Page 009
`
`

`

`Patent Application Publication
`
`Mar. 21, 2013 Sheet 9 of 9
`
`US 2013/0073988A1
`
`
`
`»uedpuenera
`
`(
`
`)
`
`Niantic's Exhibit No. 1036
`Page 0010
`
`

`

`US 2013/0073988 A1
`
`Mar. 21, 2013
`
`ACQUIRING, RANKING AND DISPLAYING
`POINTS OF INTEREST FOR USE IN AN
`AUGMENTED REALITY SERVICE
`PROVISIONING SYSTEMAND GRAPHICAL
`USER INTERFACE FOR DISPLAYING SUCH
`RANKED POINTS OF INTEREST
`0001 Acquiring, ranking and displaying points of interest
`for use in an augmented reality service provisioning system
`and graphical user interface for displaying Such ranked points
`of interests
`
`FIELD OF THE INVENTION
`0002 The invention generally relates to acquiring, rank
`ing and displaying points of interests for use in an augmented
`reality service provisioning system and a graphical user inter
`face for displaying Such ranked points of interest and, in
`particular, though not necessarily, to a method and a system
`for generating ranked points of interests for use in an aug
`mented reality service provisioning system, a client and a
`server for use in Such system, and a computer product pro
`gram using Such method.
`
`BACKGROUND OF THE INVENTION
`0003. The recent convergence of mobile telecommunica
`tions, imaging systems and multimedia techniques enable the
`realisation of mobile services wherein real-world scenery
`seen by a user is enhanced with computer-generated imagery.
`These services, which are generally referred to as augmented
`reality (AR) services, are currently implemented on mobile
`multimedia devices.
`0004 Typically such services involve retrieval of digital
`data from a network server on the basis of the geographical
`location of the mobile device on which an AR client is
`executed. The digital data may be displayed to the user in the
`form of a graphical layer overlaying the real-world Scenery
`seen by the user on a graphical user interface (GUI). Known
`GUI implementations include a screen of a mobile device,
`which displays a graphical layer over the scenery of a screen
`shot generated by a digital camera of the mobile device, or a
`head mounted display.
`0005. The graphical layer may include visual information
`indicators associated with real-world objects and locations in
`the scenery. Such an indicator, which hereafter is generally
`referred to as a Point Of Interests (POI), may include graphi
`cal information and/or selectable links allowing a user to
`access further sources, e.g. web pages, audio and media files,
`etc.
`0006 Currently, the first systems hosting such mobile AR
`services are set up and rapidly grow in popularity. One key
`feature for rapid adoption by users is the use of an open
`architecture. For example the Layar R AR platform uses a
`standardized public layer definition allowing third party con
`tent providers to design their own layers each comprising a
`particular set of POIs and add this layer to the existing pool of
`layers which is managed by a layer proxy. Using a layer
`database comprising URLs of the content providers and layer
`metadata, the layer proxy enables an AR client to retrieve
`POIs associated with a user-selected layer.
`0007. In order to maintain an open and manageable AR
`system, the layer metadata typically comprise global infor
`mation on a layer and no detailed information on the POIs
`associated with the layer. The layer database thus does not
`contain Sufficient information for enablinga user to search for
`
`relevant POIs, which are available within a certain range
`around the user. Hence, with the rapid increase of the number
`of layers and associated POIs, it is desirable for a user to be
`able to search for “local POIs, i.e. relevant POIs within a
`particular range around the location of the user. POIs however
`are not linked documents such as HTML pages so that
`schemes for ranking search results known from the Internet
`(e.g. PageRank used by the Google search engine) cannot be
`used. Hence, news way for searching and ranking POIs
`according to their relevance are needed.
`0008. Using a suitable scheme for searching POI, the num
`ber of relevant POIs in a search result within a predetermined
`range of the user may become very large and spatial visuali
`sation of the search results on a mobile screen is of limited
`size may become problematic. In particular, POIs may be
`located close together so that the displayed POI are cluttered
`together on the screen. Especially when a mobile device uses
`a display screen with touchscreen functionality, selection of a
`POI from a large group of POIs may become very difficult.
`0009 Hence, there is a need in the art for methods and
`systems allowing ranking and searching points of interest for
`use in an augmented reality service provisioning. Moreover,
`there is a need in the art for controlling user interface opera
`tion of a mobile device displaying Such ranked and searched
`POIs such that a POI user interaction is achieved which
`allows a user to select and examine the POI search results in
`a simple and intuitive way.
`
`SUMMARY OF THE INVENTION
`0010. It is an object of the invention to reduce or eliminate
`at least one of the drawbacks known service provisioning
`systems. In a first aspect the invention may relate to a method
`for generating ranked points of interests for use in an aug
`mented reality service provisioning system. The method may
`comprise the steps of receiving feedback information com
`prising information on users interactions associated with one
`or more points of interest displayed as selectable graphical
`indicators on a screen of a mobile device and assigning popu
`larity scores to points of interests identified in said feedback
`information on the basis of said user interactions; associating
`a point of interest in a recording list, comprising points of
`interests sent to one or more mobile devices in said aug
`mented reality service provisioning system, with a popularity
`score assigned to points of interests identified in said feed
`back information; and, storing said point of interest and said
`popularity score in a points of interest database.
`0011. In one embodiment, said method may further com
`prise: associating said points of interest in said recording list
`with distance interaction information comprised in said feed
`back information, said distance interaction information deter
`mining the distance between a mobile device and the point of
`interest at the moment of a user interaction; and storing said
`distance interaction information associated with said point of
`interest in said points of interest database.
`0012. In another embodiment a points of interest may be
`associated with a data structure comprising at least a unique
`identifier, location information and metadata for allowing a
`mobile device to display a point of interest as selectable
`graphical indicators on a graphical user interface.
`0013. In yet another embodiment said method may com
`prise: storing one or more points of interest identified in said
`feedback information and their associated popularity in a
`popularity table; determining on the basis of the points of
`
`Niantic's Exhibit No. 1036
`Page 0011
`
`

`

`US 2013/0073988 A1
`
`Mar. 21, 2013
`
`interest and the popularity scores in said popularity table, the
`popularity score of a point of interest in said recording list.
`0014. In a further embodiment said feedback list is gener
`ated by storing events associated with a user interacting with
`a point of interest displayed on a screen of a mobile device.
`0015. In one variant said popularity score may be deter
`mined on the basis of a popularity algorithm using user inter
`action information as an input parameter.
`0016. In another variant said method further may further
`comprise: receiving a request from a mobile device for points
`of interests associated with a predetermined graphical over
`lay; on the basis of a resource database, relaying said request
`to a server comprising said points of interests; receiving one
`or more points of interest associated with said graphical over
`lay; relaying said points of interests to said mobile a points of
`interest recorder for storing said points of interest in said
`recording list.
`0017. In yet another variant said method may further com
`prise indexing the information in the points of interest data
`base using an indexing scheme.
`0018. In another aspect the invention may relate to a
`method for searching in an indexed list of points of interests,
`wherein each point of interest is at least associated with
`location information and ranking information, said ranking
`information comprising at least a popularity score determined
`on the basis of information on user interactions associated
`with one or more points of interest displayed as selectable
`graphical indicators on a screen of a mobile device. The
`method may comprise: receiving a search request from a
`mobile device, said request at least comprising location infor
`mation associated with said mobile device; selecting points of
`interest in said indexed list on the basis of said location
`information; ranking said selected points of interest on the
`basis of said ranking information; sending said ranked points
`of interest to said mobile device.
`0019. In one embodiment said ranking information may
`further comprise an average distance interaction of a point of
`interest, the distance between a point of interest and/or the
`location information in said search request.
`0020. In yet another embodiment the method may com
`prise if said mobile device has moved over a predetermined
`distance, sending a Subsequent search request for retrieving a
`list of ranked points of interests on the basis of the new
`location information of said mobile device.
`0021. In further aspect the invention may relate to a rank
`ing system for generating ranked points of interests for use in
`an augmented reality service provisioning system, said points
`of interests allowing a mobile device to display selectable
`graphical indicators on a graphical user interface, wherein the
`system may comprise: a feedback recorder for receiving feed
`back information generated by a feedback function executed
`on a mobile device, said feedback information comprising
`information on users interactions associated with one or more
`points of interest displayed as selectable graphical indicators
`on a screen of a mobile device; a points of interest recorder for
`generating recoding list comprising points of interests which
`are sent to sent to one or more mobile devices in said aug
`mented reality service provisioning system; a processor
`executing a popularity algorithm for assigning popularity
`scores to points of interests identified in said feedback infor
`mation on the basis of said user interactions; and, a processor
`for associating points of interest in said recording list with a
`popularity score assigned to points of interests identified in
`said feedback information.
`
`0022. In yet another aspect the invention may relate to a
`client for use with a ranking system as described above,
`wherein said client may be configured to: send a request for
`points of interest to a content server in a network; to receive
`from said content server one or more points of interest and to
`display said points of interests as location-dependent select
`able graphical indicators in a graphical overlay on the graphi
`cal user interface of a mobile device; to generate a feedback
`information of user interactions associated with one or more
`points of interests displayed on said graphical user interface
`of said mobile device; and, to send said feedback information,
`preferably periodically, to said ranking system.
`0023. In a further aspect the invention may relate to a
`graphical user interface for displaying search results compris
`ing ranked points of interests on a mobile device comprising
`a digital compass, wherein said graphical user interface may
`comprise: a first window for displaying a scrollable list of
`selectable items, each item being associated with a ranked
`point of interest and identified by a predetermined graphical
`indicator, the size of said first window allowing displaying of
`a predetermined part of said of list of items; a second window
`for displaying a linear movable compass scale and a first set of
`graphical elements representing a point of interest associated
`with displayed items in said first window and each graphical
`element being presented along said compass scale in accor
`dance with the direction of said points of interest relative to
`the mobile device; said second window further comprising: a
`fixed direction pointer wherein if the mobile device is pointed
`in a certain direction, the linear compass scale and the graphi
`cal elements are moved by a signal of the digital compass in
`the mobile device such that when the fixed direction pointer
`coincides with a graphical element, the mobile device is
`pointing towards the location of the point of interest associ
`ated with said graphical element.
`0024. The invention may also relate to a computer pro
`gram product, wherein the computer program product may
`comprise software code portions configured for, when run a
`computer, executing the method steps according to any of the
`methods described above.
`0025. The invention will be further be illustrated with
`reference to the attached drawings, which schematically
`show embodiments according to the invention. It will be
`understood that the invention is not in any way restricted to
`these specific embodiments.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0026 FIG. 1 depicts a schematic of a known AR service
`provisioning system.
`(0027 FIG. 2 depicts exemplary GUIs of a known AR
`service.
`0028 FIG. 3 depicts a schematic of an AR service provi
`Sioning system according to one embodiment of the inven
`tion.
`0029 FIG. 4 depicts an exemplary flow diagram of a POI
`feedback process according to one embodiment of the inven
`tion.
`0030 FIG. 5 depicts an exemplary flow diagram of gen
`erating a ranked POI database according to one embodiment
`of the invention.
`0031
`FIG. 6 depicts an exemplary flow diagram of the
`execution of a POI search according to one embodiment of
`the invention.
`0032 FIG.7 depicts an exemplary GUI for displaying POI
`search results according to one embodiment of the invention.
`
`Niantic's Exhibit No. 1036
`Page 0012
`
`

`

`US 2013/0073988 A1
`
`Mar. 21, 2013
`
`0033 FIG.8 depicts an exemplary GUI for displaying POI
`results according to another embodiment of the invention.
`FIG.9 depicts an exemplary GUI for displaying POI results
`according to yet another embodiment of the invention.
`
`DETAILED DESCRIPTION
`0034 FIG. 1 (a) depicts a schematic of a known AR ser
`vice provisioning system 100 for providing mobile AR ser
`vices. The system comprises one or more mobile user equip
`ment (UEs) 102a-102c connected to a wireless network 104.
`Such wireless networks typically include networks which are
`implemented in accordance with the 2G, 3G or UMTS-based
`technologies. A wireless network may include a number of
`network nodes, e.g. a Base Station Controller 106 (BSC) for
`controlling a number of access nodes 108, typically referred
`to as base stations, each covering a certain area (cell), a
`Mobile Switching Centre (MSC) 110 for connecting UEs to
`fixed line telecommunications network 112, e.g. a PSTN, a
`Home Location Register (HLR) comprising 114 information
`associated with subscribers to the mobile services offered by
`the wireless network and a Serving General Support Node
`(SGSN) 116 for connecting UEs to one or more public or
`private data networks 118 such as the Internet. Alternatively
`and/or in addition UEs may be wirelessly connected to public
`or private data networks through a local Wi-Fi or WiMax
`network (not shown).
`0035 Each UE, schematically shown in more detail in
`FIG. 1 (b)), may generally comprise a processor 120 for
`executing and managing an Operating System (OS) 122, a
`User Interface (UI) including a selectable display 124 and
`Software applications, which may be stored in a memory 126.
`The OS may execute client software such as HTTP and/or SIP
`clients for setting up web-based services and/or streaming
`services. The UE may comprise a radio card module 128
`comprising a base band processor (BP) for controlling the
`radio communications between the ME and an access node of
`a wireless network using a RF communications interface.
`Network access and authentication may be controlled using a
`SIM card connected to the processor.
`0036. The UE may further comprise a digital imaging
`system 130 comprising a lens system, an image sensor and an
`imaging processor connected to the GUI which is configured
`to generate a camera view and sensor modules for generating
`positional information associated with the UE, i.e. the geo
`coordinates and the attitude. Such sensor modules, which are
`known per se, may include a GPS receiver module 132 for
`generating the geo-coordinates longitude and latitude of the
`mobile device, a magnetometer 134 for determining direction
`(rotation around the vertical axis) and an accelerometer 136
`for determining the tilt (the angle with respect to the earth's
`gravitation vector). In one embodiment, the tilt parameter
`generated by the accelerometer may be used for determining
`and displaying the horizontal plane in order to display objects
`correctly in the camera view.
`0037. An AR client stored in the memory of the UE may be
`activated by the user in order to provide AR services to the
`UE. In one particular AR service, the AR client may generate
`on the basis of the location information provided by the GPS
`module and the digital imaging system, a so-called AR cam
`era view wherein a digitally generated graphical overlay, i.e.
`a layer comprising geo-coded information in the form of one
`or more POIs, i.e. indicators, typically graphical indicators,
`associated with an geo-located object, person, action or loca
`tion in the camera view. A POI may provide information, e.g.
`
`text, e.g. a tweet, images or (3D) virtual objects, or an indi
`cation for an action, e.g. selectable links or buttons allowing
`a user to view further digital sources, e.g. video and/or audio
`ora webpage, or to execute an application, sending an SMS or
`starting a call. Further, a POI comprises a geo-location, i.e.
`geo-coordinates locating it on the Earth's Surface (e.g. lat,
`long) and altitude information (Z-coordinate) that may deter
`mine a POI to be located on the actual Earth's surface or
`anywhere above or below this surface.
`0038 Alayer may be defined according to certain publicly
`available standard rules, thereby allowing third parties, typi
`cally content providers, to define layers associated with dif
`ferent Subjects, objects or services, e.g. lifestyle, restaurants,
`shopping, banks, housing, etc. Hence, a content provider may
`design its own layer inaccordance with the standard rules and
`submit metadata associated with the layer, i.e. an URL for
`retrieving the POIs, a layer subject, layer layout information,
`etc., to the AR system so that it is known from which server
`the layer data (the actual POIs) may be retrieved.
`0039. The AR system thus stores the metadata associated
`with the available layers in a layer database 140. If a user
`selects a particular layer, the AR client transmits request, e.g.
`a HTTP GET request GetPOI, to a layer proxy 142. On the
`basis of the information in the request, the layer proxy may
`retrieve the URL associated with the selected layer and sub
`sequently relays the request to a server of one of the content
`providers 144-148. On the basis of the geo-information in the
`request, the relevant POIs, including metadata associated
`with the POIs, e.g. the geo-coordinates of each POI, are
`determined and returned in a response message to the AR
`client. The AR client uses the layer metadata in order to
`generate a graphical overlay including the various POIs and
`to display the graphical overlay in the camera view.
`0040. When using the AR service, a user may first select a
`layer from the layers in a layer gallery view 200 as depicted by
`the GUI layout in FIG. 2(a). When using the AR service, the
`AR client may retrieve layers, e.g. the most frequently
`selected layers, from the layer database and present the avail
`able layers 202a-202e to the user. Upon selection of a layer by
`the user, e.g. a real estate layer as depicted in the example of
`FIG.2(a), the AR client transmits a request via the layer proxy
`to a content provider in order to retrieve the POIs associated
`with the selected layer. Further, the AR client may switch over
`to the AR camera view mode wherein the user uses the UEas
`a camera and points the camera towards the POIs of interest.
`0041. On the basis of the information in the response, the
`AR client may generate an AR camera view as depicted in
`FIG. 2(b). The GUI may comprise a graphical layer 204, in
`this example layer of a real estate organisation, Superimposed
`over a real-world window 206 for displaying the scenery to
`which the user points the camera of the UE. The layer may
`comprise a number of POIs 208a–208c wherein each POI
`may be associated with a number of POI elements, e.g. text
`and picture windows, URLs, a select button for activating a
`multimedia or messaging service, etc. A POI element may
`become visible when the user selects a POI or when a user is
`within a certain distance of the POI. The POIs, e.g. a disc
`shaped POI, may be projected onto a 3D grid 210 and may
`change in size, shape and/or colour as a function of distance
`and/or time. For example, in FIG. 2(b) the disc-shaped POIs
`become smaller in size when the distance between the POI
`and the AR client gets larger thereby producing a visual depth
`effect. Further, the AR client may automatically lock onto a
`POI 208a, which is at a distance closest to the UE and within
`
`Niantic's Exhibit No. 1036
`Page 0013
`
`

`

`US 2013/0073988 A1
`
`Mar. 21, 2013
`
`the angle of view of the camera. Alternatively, a user may lock
`onto a POI by selecting it. Whenlocking onto a POI a window
`212 may appear comprising further information regarding the
`POI, including e.g. a picture window 214, a text window 216,
`an URL 218 and/or a selectable link 220.
`0042. The AR camera view is limited by the angle of view
`of the camera, so not all retrieved POIs, which are within a
`predetermined distance from the UE, are visible in the AR
`camera view. For that reasons the GUI may further comprise
`a small radar Screen 222 providing a two-dimensional view of
`the POIs associated with the selected layer, which are avail
`able in the area around the UE. A triangular area 224 in the
`radar screen depicts the area, which is covered by the AR
`camera view, and the POIs that are within that area (and thus
`visible in the AR camera view). For example in FIG. 2(b) one
`can determine on the basis of the radar view that three of the
`four POIs are displayed in the AR camera view.
`0043. The GUIs and the associated AR services described
`with reference to FIGS. 2 and 1 are known from the Layar R
`mobile AR service platform. These figures illustrate that the
`access to layer information, in particular the access to relevant
`POIs, is still rather limited. In particular, it illustrates that
`there is a need for functionality allowing a user to search
`relevant POIs, which may be associated with different layers
`and which are available within a predetermined area around
`the UE. SuchPOI search function should not negatively influ
`ence the open architecture of the platform. Further, such
`search function should be scalable to large AR systems com
`prising a large number of different types of POIs.
`0044) Moreover, with the steady increase of the number of
`layers, each comprising an increasing number of POIs, the
`search results will comprise more and more irrelevant or
`unwanted POIs. As the AR GUI is using actual real space to
`represent the POIs, that space may become “filled in the
`sense that it may not be possible to represent all POIs at the
`same time, if they are all located on the same spatial coordi
`nates. Hence, a desired POI search scheme should preferably
`rank the search results according to a scheme wherein for
`instance it is presumed that POIs which are frequently
`selected by users are of a higher relevance than the ones not so
`often selected. Further, from the GUIs as described with
`reference to FIG. 2, it is apparent that there is a need for a
`simple and intuitive presentation and manipulation of the
`search results such that a user may directly associate a ranked
`POI in the search result with its geo-information and manipu
`late and access this information Such that problems related to
`selecting a POI from many POIs displayed on a small touch
`screen are mitigated.
`0045 FIG. 3 depicts a schematic of an AR service provi
`sioning system 300 for providing mobile AR services accord
`ing to one embodiment of the invention. The AR system
`depicted in FIG.3 allows a user to search POIs present within
`a certain distance from the user in accordance to a ranking
`scheme.
`0046. The system comprises one or more mobile devices
`(user equipment or UE) 302a, 302b which are configured to
`access fixed public and/or private data networks via a wireless
`access network (not shown) in a similar was as described with
`reference to FIG. 1. Each UE may comprise an AR client
`304a, 304b configured to generate an AR camera view on the
`basis of POI information from content providers 306. Further,
`the AR client is configured to receive location and direction
`information from sensors in the UE. In order to retrieve the
`relevant POI information, a layer proxy 308 may manage the
`
`request and response messages exchanged between an AR
`client and a server of a content provider on the basis of
`resource information in a layer database 310.
`0047. The layer database may further comprise a list of
`URLs associated with the content providers, user informa
`tion, e.g. in the form of user profiles, and layer information
`(i.e. layer metadata such as information used to display a layer
`on the screen of a UE: title, publisher, description text, icons,
`color schemes; and information used by the AR system: list of
`countries where the layer is relevant, keywords for searching
`the layer, location information in the form of bounding geo
`boxes to restrict places where layer should be shown, flags for
`opting out of certain services (e.g. the search service as
`described in more detail below), average time-to-live of the
`POIs and expiration date of the layer, etc.).
`0048. The AR system may be configured to record POI
`information that is exchanged between the AR client and the
`content providers. To that end, the proxy server may comprise
`(or may be connected to) a POI recorder 312 and a POI
`feedback recorder 314. The POI recorder monitors POI
`responses originating from the content providers and gener
`ates on the basis of the monitored POI responses a list of POIs
`316, i.e. a POI queue representing a recorded list of tempo
`rally ordered POIs, which were sent by the content providers
`via the layer proxy to the AR clients. In one embodiment, on
`the basis of the metadata in the layer database and on the basis
`of the request parameters, the POI recorder may add contex
`tual information, e.g. layer information and/or user informa
`tion and/or distance information to each recorded POI. The
`POI feedback recorder may generate a list of POI events, i.e.
`a POI feedback queue 318, representing information regard
`ing the “POI browsing history' of users of the AR system.
`0049. In order to correlate POI information recorded by
`the POI recorder and the POI feedback recorder, each POI in
`the AR system may be assigned to a unique identifier POI ID,
`which may be easily calculated on the basis of the layer
`metadata and the POI ID included in the POI response. In
`one embodiment, the POI ID may be defined as the hash of a
`layer identifier, which is unique within the layer database, and
`a POI identifier, which is unique within a particular layer
`definition.
`0050. The AR client may comprise a POI feedback func
`tion 320, which is configured to monitor and store a group of
`POI events and to periodically send the thus collected POI
`event information to the POI feedback recorder. POI events
`may generally relate to any type of information associated
`with a user interacting with POIs presented in the AR view of
`the mobile device. A POI event may include metadata asso
`ciated with a selected POI, e.g. the POI ID, and metadata
`associated with “user actions', e.g. selection of a link asso
`ciated with the POI, activation of a user selectable program,
`e.g. a media player or a widget, time information (e.g. time
`stamps) associated with Such user actions and “local user
`actions associated with a POI, e.g. selection of a POI as a
`favourite or tagging the POI with a user-defined tag. A POI
`event may al