(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2008/0268876A1
`(43) Pub. Date:
`Oct. 30, 2008
`Gelfand et al.
`
`US 20080268876A1
`
`(54) METHOD, DEVICE, MOBILE TERMINAL,
`AND COMPUTER PROGRAMI PRODUCT FOR
`A POINT OF INTEREST BASED SCHEME
`FOR IMPROVING MOBILE VISUAL
`SEARCHING FUNCTIONALITIES
`
`(22) Filed:
`
`Apr. 23, 2008
`Related U.S. Application Data
`(60) Provisional application No. 60/913,733, filed on Apr.
`24, 2007.
`
`(76) Inventors:
`
`Natasha Gelfand, Sunnyvale, CA
`(US); Ramakrishna Vedantham,
`Sunnyvale, CA (US); C. Philipp
`Schloter, San Francisco, CA (US);
`Radek Grzeszczuk, Menlo Park,
`CA (US); Wei-Chao Chen, Los
`Altos, CA (US): Suresh Chitturi,
`Plano, TX (US); Jiang Gao,
`Sunnyvale, CA (US); Markus
`Kahari, Helsinki (FI); David
`Murphy, Helsinki (FI); Kari Pulli,
`Palo Alto, CA (US); Ramin
`Vatanparast, Redwood City, CA
`(US); Yingen Xiong, Mountain
`View, CA (US)
`
`Correspondence Address:
`ALSTON & BRD LLP
`BANK OF AMERICA PLAZA, 101 SOUTH
`TRYON STREET, SUITE 4000
`CHARLOTTE, NC 28280-4000 (US)
`
`(21) Appl. No.:
`
`12/108,281
`
`
`
`Publication Classification
`
`(51) Int. Cl.
`(2006.01)
`H04O 7/20
`(2006.01)
`H04M I/00
`(52) U.S. Cl. ...................................... 455/457; 455/556.1
`(57)
`ABSTRACT
`Systems, methods, devices and computer program products
`which relate to utilizing a camera of a mobile terminal as a
`user interface for search applications and online services to
`perform visual searching are provided. The system consists of
`an apparatus that includes a processor that is configured to
`capture an image of one or more objects and analyze data of
`the image to identify an object(s) of the image. The processor
`is further configured to receive information that is associated
`with at least one object of the images and display the infor
`mation that is associated with the image. In this regard, the
`apparatus is able to simplify access to location based services
`and improve a user's experience. The processor of the appa
`ratus is configured to combine results of robust visual
`searches with online information resources to enhance loca
`tion based services.
`
`53
`
`55
`
`57
`
`s
`
`an Drive Sea
`Stanfore
`zoom in
`
`EW
`
`Niantic's Exhibit No. 1005
`Page 001
`
`

`

`Patent Application Publication
`
`Oct. 30, 2008 Sheet 1 of 11
`
`US 2008/0268876 A1
`
`36
`
`A 12
`
`CAMERA
`MODULE
`
`TRANSMITTER
`
`- 10
`
`
`
`-
`Mapping Module
`
`97
`Mobile Visual
`Search Module
`
`GPS
`MODULE
`
`DISPLAY
`
`70
`
`28
`
`RECEIVER
`RINGER
`22
`SPEAKE
`
`24
`MICROPHONEO
`26
`
`CONTROLLER
`
`i
`
`40
`VOLATILE
`MEMORY
`
`NON-
`VOLATILE
`O
`MEMORY
`
`38
`
`UM
`
`KEYPAD
`30
`
`34
`
`42
`
`A
`-T-
`
`Niantic's Exhibit No. 1005
`Page 002
`
`

`

`Patent Application Publication
`
`Oct. 30, 2008 Sheet 2 of 11
`
`US 2008/0268876 A1
`
`12
`
`10
`
`62
`
`48
`
`GTW
`
`54
`
`VISUAL MAP
`SERVER
`
`52
`
`TERMINAL
`
`COMPUTING
`
`-1 SYSTEM
`
`M
`AP
`
`48
`
`51
`
`POSHOP
`
`SERVER
`
`60
`
`12
`
`10
`
`TERMINAL
`
`10
`
`2
`
`48
`
`AP
`
`GTW
`
`50
`
`GGSN
`
`46
`
`TERMINAL
`
`I
`
`MSC
`
`12
`
`10
`
`44
`
`SGSN
`
`TERMINAL
`
`56
`
`FIG. 2.
`
`Niantic's Exhibit No. 1005
`Page 003
`
`

`

`Patent Application Publication
`
`Oct. 30, 2008 Sheet 3 of 11
`
`US 2008/0268876 A1
`
`12
`
`10
`
`TERMINAL
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`VISUAL MAP SERVER
`72
`74
`
`Mobile Visual
`Search Server
`
`Centralized
`PO
`Database
`Sever
`
`PO| SHOP
`SERVER
`
`98
`Local POI
`Database
`Server
`
`MAP
`SERVER
`
`FIG. 3.
`
`Niantic's Exhibit No. 1005
`Page 004
`
`

`

`Patent Application Publication
`
`Oct. 30, 2008 Sheet 4 of 11
`
`US 2008/0268876 A1
`
`CAPTURE ANIMAGE OF A POl(s)
`
`---....Y.-- 405
`SELECT AN ACTION FROM LIST OF ACTIONS THAT ARE RELATED /
`TO THE CAPTURED PO
`
`TRANSMIT THE CAPTURE IMAGE OF THE POl(s) TO SERVER
`
`41 O
`
`DISPLAY LINKS OR OTHER INFORMATION RELATED
`TO POl(s)
`
`415
`
`
`
`
`
`RETRIEVE OVERHEAD MAP OF SURROUNDING AREA 420
`WITH SUPERMPOSED VISUALTAGS
`CORRESPONDING TO POI(S)
`
`
`
`SEND OVERHEAD MAP TO MOBILE TERMINAL
`
`425
`
`
`
`
`
`
`
`LAUNCH VIEW TO DISPLAY SURROUNDING AREA OF POI
`AND SUPERIMPOSED SET OF VISUALTAGS THAT
`CORRESPOND TO SET OF POS
`
`
`
`430
`
`SWITCH THE VIEW OF MAPF THE POIDATA
`ISTOO DENSE TO DISPLAY CLEARLY
`
`435
`
`Niantic's Exhibit No. 1005
`Page 005
`
`

`

`Patent Application Publication
`
`Oct. 30, 2008 Sheet 5 of 11
`
`US 2008/0268876 A1
`
`
`
`-94
`
`INTERFACE
`
`PROCESSOR
`
`FIG. 5.
`
`Niantic's Exhibit No. 1005
`Page 006
`
`

`

`Patent Application Publication
`
`Oct. 30, 2008 Sheet 6 of 11
`
`US 2008/0268876 A1
`
`
`
`g
`
`Niantic's Exhibit No. 1005
`Page 007
`
`

`

`Patent Application Publication
`
`Oct. 30, 2008 Sheet 7 of 11
`
`US 2008/0268876 A1
`
`
`
`Niantic's Exhibit No. 1005
`Page 008
`
`

`

`Patent Application Publication
`
`Oct. 30, 2008 Sheet 8 of 11
`
`US 2008/0268876 A1
`
`
`
`69
`
`Niantic's Exhibit No. 1005
`Page 009
`
`

`

`Patent Application Publication
`
`Oct. 30, 2008 Sheet 9 of 11
`
`US 2008/0268876 A1
`
`
`
`
`
`ASSOCATE OR TAG PRODUCT INFORMATION TO ANIMAGE
`& STORE TAGGED PRODUCT INFORMATION
`
`
`
`
`
`
`
`
`
`
`
`TRANSMIT TAGGED PRODUCT INFORMATION TO
`DEVICE WHICH SENDS SENDS TAGGED PRODUCT
`INFORMATION TO TERMINAL
`
`DISPLAY RESULTS ASSOCATED WITH THE TAGGED
`PRODUCT INFORMATION WHICH MAY BEAN
`ADVERTISEMENT
`
`915
`
`FIG. 9.
`
`Niantic's Exhibit No. 1005
`Page 0010
`
`

`

`Patent Application Publication
`
`Oct. 30, 2008 Sheet 10 of 11
`
`US 2008/0268876 A1
`
`
`
`
`
`
`
`
`
`CAPTURE IMAGE(S) WITHMOBILE TERMINAL WHICH MAYBE
`TRANSMITTED TO SERVER
`
`REMOVE FEATURE OUTLERS IN MAGES
`
`CLUSTER THE IMAGES BASED ON FEATURES SHARED BY
`IMAGES
`
`EXTRACT CLUSTERS THAT BELONG TO SINGLE OBJECT OR POI |
`
`800
`
`805
`
`810
`
`815
`
`COMPUTE & STORE EXTENT OF EACH CLUSTER
`
`
`
`820
`
`
`
`
`
`
`
`IDENTIFY CLUSTERS REPRESENTATIVE OF STORE OR BUSINESS
`l
`PROCESS THE POIS IN CAPTURED IMAGES & ASSOCATE EACH
`CLUSTER WITH A POl(s)
`
`
`
`
`
`
`
`RECEIVE GEOGRAPHIC EXTENT INFORMATION OF STORE OR
`BUSINESS
`
`FIND POINTS THAT FALL WITHIN THE EXTENT OF THE BUSINESS
`POS
`m
`
`DETERMINE CORREPSPONDING GROUPS OF 3D POINTS
`
`DETERMINE ACLUSTER OF POINTS WITH A CENTER CLOSEST TO
`THE GIVENPO & ASSOCATE THE POINTS WITH THE POTO
`DETERMINE THE LOCATION OF THE POI
`
`825
`
`830
`
`835
`
`840
`
`845
`
`850
`
`FIG. 10.
`
`Niantic's Exhibit No. 1005
`Page 0011
`
`

`

`Patent Application Publication
`
`Oct. 30, 2008 Sheet 11 of 11
`
`US 2008/0268876 A1
`
`//
`
`9/
`
`d)
`f
`f
`s CD
`
`o
`
`X
`
`S
`
`C
`
`to 8 32
`
`C
`
`S
`
`CO
`
`S...
`
`O.
`C
`
`?e
`
`is
`sis
`
`/9
`
`
`
`
`
`
`
`
`
`Niantic's Exhibit No. 1005
`Page 0012
`
`

`

`US 2008/0268876 A1
`
`Oct. 30, 2008
`
`METHOD, DEVICE, MOBILE TERMINAL,
`AND COMPUTER PROGRAMI PRODUCT FOR
`A POINT OF INTEREST BASED SCHEME
`FOR IMPROVING MOBILE VISUAL
`SEARCHING FUNCTIONALITIES
`
`CROSS REFERENCE TO RELATED
`APPLICATION
`0001. This application is related to and claims the benefit
`of U.S. Provisional Patent application Ser. No. 60/913,733
`filed Apr. 24, 2007, which is hereby incorporated by refer
`CCC.
`
`FIELD OF THE INVENTION
`0002 Embodiments of the present invention relate gener
`ally to mobile visual search technology and, more particu
`larly, relate to methods, devices, mobile terminals and com
`puter program products for utilizing points-of-interest (POI).
`locational information and images captured by a camera of a
`device to perform visual searching, to facilitate mobile adver
`tising, and to associate point-of-interest data with location
`tagged images.
`
`BACKGROUND OF THE INVENTION
`0003. The modern communications era has brought about
`a tremendous expansion of wireline and wireless networks.
`Computer networks, television networks, and telephony net
`works are experiencing an unprecedented technological
`expansion, fueled by consumer demands, while providing
`more flexibility and immediacy of information transfer.
`0004 Current and future networking technologies con
`tinue to facilitate ease of information transfer and conve
`nience to users by expanding the capabilities of mobile elec
`tronic devices. One Such expansion in the capabilities of
`mobile electronic devices relates to modern mobile devices
`possessing the promise of making Augmented Reality (AR)
`(which deals with the combination of real-world and com
`puter generated data) practical and universal. There are sev
`eral characteristics that make mobile devices the platform of
`choice for developing AR applications. First, new mobile
`devices are being developed and equipped with broadband
`wireless connectivity, providing the users of the mobile
`devices access to vast amounts of information via the World
`WideWeb anywhere, at anytime. Second, the need for AR is
`at its highest in a mobile setting since current mobile devices
`utilize video clips, images and various forms of multimediato
`enhance a user's experience. Third, the physical location of
`the mobile device can be accurately estimated, either through
`a global positioning system (GPS) or through cell tower loca
`tion triangulation. The above features make mobile devices
`an ideal platform for implementing and deploying AR appli
`cations and in fact, examples of Such applications are cur
`rently available and gaining in popularity. A good example is
`a GPS-based navigation system for smart mobile phones. The
`software of the smart mobile phone not only provides a user
`with driving directions, but also uses real-time traffic infor
`mation to find the quickest way to a destination, and enables
`a user to find points-of-interest, such as restaurants, gas sta
`tions, coffee shops, or the like based on proximity to the
`current location. A similar application of AR consists of a
`computer-generated atlas of the Earth that enables a user to
`Zoom in to street level and find point of interests in his/her
`proximity.
`
`0005. Notwithstanding the fact that mobile devices are
`implementing and deploying AR applications and that there is
`a natural progression of the AR applications towards a general
`mobile search capability, a limiting factor in the adoption of
`mobile searching relates to difficult and inefficient user-inter
`facing. Hence a major challenge in developing mobile visual
`search applications is to enable the search to be easy and
`simple to use by incorporating non-standard input devices,
`Such as cameras and location sensors into intuitive and robust
`user interfaces applicable in a mobile setting.
`0006 Current versions of mobile visual search applica
`tions utilize a centralized database that stores predefined POI
`images, their corresponding features and the related metadata
`(textual tags). While current versions of mobile visual search
`client devices show textual tags corresponding to an image
`pointed at by a mobile phone's camera, a user may not be
`interested only in these textual tags, but also in the informa
`tion about other points of interest (POI) in the surrounding
`area. This is particularly relevant when the object in the user's
`immediate vicinity does not have any visual tags, and the user
`is interested in finding where the visual tags are. Currently,
`there is no easy way for visualization of the POI data in a
`mobile visual search client other than displaying of the visual
`tags that are visible by the mobile phone's camera. As such,
`the user may have to switch between the mobile visual search
`client and an external mapping application or a web browser
`to see the Surrounding areas and other tags/POI’s.
`0007 Another drawback of current mobile visual search
`clients relates to the POI data displayed on a mobile device
`when using either an online mapping application (e.g.,
`Smart2Go) or a web browser-based mapping application
`(e.g., Google Maps, Yahoo Maps) is typically not dynamic.
`Information resulting from the online mapping application
`has limited usefulness without a complementing mobile
`visual search application. Furthermore, existing mapping
`applications are targeted to only display information about
`points of interest to the user. In this regard, there exists a need
`to make use of the fact that a phone is a communication device
`with a broadband connectivity to expand the scope of visual
`tags beyond information display to a communication tool. As
`Such, there exists a need to utilize visual tags to communicate
`with web sites, e-mail clients, online and shared calendars
`and even other mobile visual search users. There also exists a
`need to utilize the various online information resources that
`are available and in order to combine this online information
`with the results of the mobile visual search applications to
`generate the next generation mobile device services.
`0008. Additionally, as known to those skilled in the art,
`innovation generates marketing opportunities as well as chal
`lenges. In this regard, advances in mobile technology have
`changed the business environment considerably. As noted
`above, devices and systems based on mobile technologies are
`commonplace in our everyday lives and have changed the
`way we communicate and interact. Phones and multimedia
`devices increase the accessibility, frequency and speed of
`communication. As a result, mobile media goes beyond tra
`ditional communication and advances one-to-one, many-to
`many and fosters mass communication. Today's development
`in information technology helps marketers to keep track of
`customers and provide new communication venues for reach
`ing Smaller customer segments more cost effectively with
`more personalized messages. Gradually many more compa
`nies are redirecting marketing spending to interactive mar
`
`Niantic's Exhibit No. 1005
`Page 0013
`
`

`

`US 2008/0268876 A1
`
`Oct. 30, 2008
`
`keting, which can be focused more effectively on targeted
`individual consumer and trade segments.
`0009 Forecasts concerning growth of mobile advertising
`have been quite enthusiastic. Mobile advertising holds strong
`promises to become the best targeted, one-to-one, and most
`powerful digital advertising medium offering new ways to
`aim messages to users that existing advertising channels are
`not able to achieve. The mobile advertising market is esti
`mated to grow to over S600 million during 2007 and is
`expected to increase to S11.35 billion in 2011. By utilizing
`mobile advertising, companies can implement marketing
`campaigns targeted to tens of thousands of people with a
`fragment of the costs injust a few seconds of time.
`0010 Advertising is a strategic marketing tool for busi
`nesses, and recently the Internet is becoming a very popular
`medium for advertising. Current advertising models relating
`to the Internet are based on traditional search systems which
`are typically based on text or keyword searches, wherein the
`text provided by the user with specific criteria is typically
`used to retrieve a list of items that match those criteria. The
`results are usually sorted with respect to some measure of
`relevance to the input provided by the user. Search engines
`using the text or keyword search concepts are based on fre
`quently updated indexed sets of data for fast and efficient
`information retrieval. Oftentimes, as the engine is providing
`relevant information to the user, based on the typed key or
`content of information, a series of advertisements accompa
`nies the information. The advertisements may also accom
`pany the web-pages which the user is reviewing. This is the
`most basic form of Internet based advertising.
`0011. In contrast, unlike the keyword searches, visual
`search systems are based on analyzing the perceptual content
`Such as images or video data (e.g. video clips) using an input
`sample image as the query. The visual search system is dif
`ferent from the so-called image search commonly employed
`by the Internet, where keywords entered by users are matched
`to relevant image files on the Internet. Visual search systems
`are typically based on Sophisticated algorithms that are used
`to analyze the input image against a variety of image features
`or properties of the image such as color, texture, shape, com
`plexity, objects and regions within an image. The images
`along with their properties are usually indexed and stored in a
`database to facilitate efficient visual search.
`0012. As noted above, in mobile devices, the concept of
`visual searches is gaining popularity as more and more
`devices are being equipped with digital cameras. This pro
`vides the ability to generate high quality input query images
`almost anywhere at anytime, which is by far more advanta
`geous and usable than the visual search systems designed for
`desktop or personal computer (PC) systems wherein multiple
`steps are required to generate a query image. For example, in
`order for a user to perform a visual search of an image on the
`PC, first the user would need to capture the image from a
`digital camera, then transfer it to a PC and Subsequently
`perform the search. However, all of these multiple steps can
`be avoided when using a mobile device equipped with a
`digital camera.
`0013 Currently, Internet advertising models fall mainly
`into three categories: (1) Impressions, (2) Click-Through's,
`and (3) Affiliate sales. Impressions consist of a model
`whereby an advertiser creates a banner advertisement and
`pays for this banner advertisement to be displayed on another
`site, for example, on search engine websites. Regarding the
`Click-Through's model, the seller or advertiser only pays
`
`when a visitor clicks on the banner advertisement and goes to
`the advertiser's site. If the user ignores the banner, then the
`advertiser is not charged. Affiliate sales model consists of
`situations in which a seller only pays for advertising when a
`particular sales target is met.
`0014. Although the above-mentioned models are quite
`successful they come with limitations as they are limited to
`keyword searches and do not take into account the visual
`search system and related contextual information Such as
`geo-location, and time including mobility that a wireless
`terminal offers.
`0015. In a dynamic world with constantly evolving adver
`tising media, advertisers need to find new ways to break
`through the clutter and reach their target consumers. Given
`the advantages of visual searches to a user/consumer, in the
`future, consumers are going to use more Visual Search-based
`advertising as away to retrieve relevant information. As such,
`there is a need to create a new system to find relevant adver
`tisements based on searched images/videos. The new system
`should impact existing advertisement delivery systems and
`also enable modified existing advertisement delivery systems
`to effectively target relevant consumers and thereby increase
`an advertiser's return on investment (ROI) for advertising
`campaigns. In this regard, visual searches require a unique
`approach to advertising, highly different from traditional
`Internet marketing. For the foregoing reasons, the concept of
`mobile visual searches coupled with contextual information
`provides various advantages for an end-user and as such,
`there exists a need to enable advertising in mobile visual
`search systems, thereby enabling relevant advertisements to
`be associated with the image/video search.
`0016 Point-of-interest (POI) databases are also relevant
`to mobile visual search systems. For instance, point-of-inter
`est (POI) databases are an integral component of systems for
`car navigation, computation of directions, on-line yellow
`pages, and virtual tour guide applications. POI databases
`typically consist of locations, coupled together with some
`associated information Such as names of businesses, contact
`information, and web links. A GPS location associated with a
`given POI is typically computed by interpolating the location
`of a given Street address within a given block. As a result, the
`location of a POI can often be imprecise. Given the increasing
`availability of GPS-equipped camera devices, it is now com
`monplace to acquire geo-tagged images (i.e., images with
`associated GPS information) of various points-of-interest.
`(For instance, geo-tagged images may contain geographical
`identification metadata to various media Such as websites,
`RSS feeds, or images which may consist of latitude and
`longitude coordinates, though it can also include altitude and
`place names as well as addresses which can be related to
`geographic coordinates.) However, there exists a need to be
`able to automatically associate POI data with geo-tagged
`images. Automatic association of POI data with geo-tagged
`images is needed to enable new camera-based user interfaces
`that retrieve information from POI databases using geo
`tagged image matching. Additionally, Such an association
`could be used to correct errors in GPS location present in POI
`databases and to augment the error correction with informa
`tion consisting of richer geometric information than is cur
`rently available.
`0017. When an image is geo-tagged, typically only the
`position of the camera is given, however, the position of the
`object that is depicted in the image is typically not provided.
`Therefore, in cases where there are several objects that can be
`
`Niantic's Exhibit No. 1005
`Page 0014
`
`

`

`US 2008/0268876 A1
`
`Oct. 30, 2008
`
`photographed from the same location (e.g. businesses on two
`sides of the street photographed from the street median), the
`position of the camera cannot be used as the position of the
`object in the image. The imprecision in the GPS position of
`both the camera and the POIs makes it difficult to associate
`POI data with images by the naive method of directly match
`ing the GPS coordinates of images and POIs, as is done
`conventionally.
`0018. In view of the foregoing, there also exists a need for
`a system enabling automatic association of point-of-interest
`data (POI) with their corresponding images and visual fea
`tures extracted from the respective images. In conventional
`systems, skilled artisans are faced with a challenge pertaining
`to location of images that are geo-tagged which are not nec
`essarily the true physical location of an object(s), or the
`location associated with this object in a POI database. As
`Such, there exists a need for a mechanism to enable proper
`association between these different entities so as to improve
`the accuracy and descriptiveness of the location information
`in a POI database.
`
`BRIEF SUMMARY OF THE INVENTION
`0019 Systems, methods, devices and computer program
`products of the exemplary embodiments of the present inven
`tion relate to utilizing a camera (e.g., a camera module) of a
`mobile terminal as a user interface for search applications and
`online services to perform visual searching. These systems,
`methods, devices and computer program products simplify
`access to location based services and improve a mobile users
`experience, which in turn can increase the sales of camera
`phones and also facilitates the launch of new mobile Internet
`based services. In this regard, new mobile location based
`services can be created by combining the results of robust
`mobile visual searches with online information resources.
`0020 Systems, methods, devices and computer program
`products of exemplary alternative embodiments of the
`present invention provide robust mobile visual search appli
`cations displaying relevant information regarding points-of
`interest pointed to by a camera of a mobile terminal. The
`systems, methods, devices and computer program products of
`the exemplary alternative embodiments of the present inven
`tion also provide mapping applications for a mobile terminal
`and can display relevant visual tags on a map view of a camera
`of the mobile terminal. Additionally, systems, methods,
`devices and computer program products of exemplary alter
`native embodiments of the present invention provide a hybrid
`of visual searching applications and online web-based appli
`cations which are capable of providing a user of a mobile
`terminal both a global view (of a relevant point-of-interest on
`a map) and a local view (of the point-of-interest from the
`camera of the mobile terminal).
`0021 Systems, methods, devices and computer program
`products of another exemplary alternative embodiment of the
`present invention provide advertising based on mobile visual
`search systems as opposed to keyword and PC-based search
`ing systems and enables an advertiser(s) to convey informa
`tion to a consumer on a daily basis, regardless of time of day
`and location of the user of the mobile terminal. The systems,
`methods, devices and computer program products of the
`exemplary alternative embodiments of the present invention
`also enable advertisers to place tags or associate information
`with images or one or more categories of images in a visual
`search database as well as creation of a relevancy link(s)
`between the information sent by a user of a mobile terminal to
`
`a server relating to products and service information. Addi
`tionally, the systems, methods, devices and computer pro
`gram products of the exemplary alternative embodiments of
`the present invention provide exclusive access or control to
`advertisers based on a particular region or through global
`objects/links as well as ease of use with the concept of a
`"point-through' business model with zero input from a key
`board of a user's terminal, (for e.g., the user is not required to
`use his/her keyboard to type relating to a keyword search)
`which reduces the number of steps required by a user/con
`Sumer to reach or find relevant information.
`0022. In one exemplary embodiment, a method for switch
`ing between camera and map views of a terminal is provided.
`The method includes capturing an image of one or more
`objects and analyzing data associated with the image to iden
`tify an object of the image. The method further includes
`receiving information that is associated with an object of the
`images and displaying the information that is associated with
`the object.
`0023. In yet another exemplary embodiment, a method for
`enabling advertising in mobile visual search systems is pro
`vided. The method includes defining and associating meta
`information to one or more objects and receiving one or more
`captured images of objects from a device. The method further
`includes automatically sending media data associated with an
`object to the device when the captured images received from
`the device include data that corresponds to one of the objects.
`0024. In another exemplary embodiment, another method
`of enabling advertising in mobile visual search systems is
`provided. The method includes defining and storing one or
`more objects and receiving one or more captured images
`objects from a device. The method further includes automati
`cally sending media data to the device when the captured
`images received from the device include data that is associ
`ated with one of the defined objects.
`0025. In yet another exemplary embodiment, a method for
`associating images with one or more points-of-interest to
`determine the location of the point-of-interest is provided.
`The method includes receiving one or more captured images
`of objects, removing features from the images and generating
`a group of images that share one or more features. Each of the
`images of the group are associated with a point. The method
`further includes determining whether the group is associated
`with a shape of an object captured in an image based on a
`predetermined number of points corresponding to the images
`of the group, associating the group to a single object when the
`determination reveals that there are a predetermined number
`of points and determining the location of at least one object in
`the images on the basis of the points.
`0026. In one exemplary embodiment, an apparatus for
`Switching between camera and map views of a terminal is
`provided. The apparatus comprises a processing element con
`figured to capture an image of one or more objects and ana
`lyze data associated with the image to identify an object of the
`image. The processing element is further configured to
`receive information that is associated with an object of the
`images and display the information that is associated with the
`object.
`0027. In yet another exemplary embodiment, an apparatus
`for enabling advertising in mobile visual search systems is
`provided. The apparatus includes a processing element con
`figured to define and associate meta-information to one or
`more objects and receive one or more captured images of
`objects from a device. The processing element is configured
`
`Niantic's Exhibit No. 1005
`Page 0015
`
`

`

`US 2008/0268876 A1
`
`Oct. 30, 2008
`
`to automatically send media data associated with an object to
`the device when the captured images received from the device
`include data that corresponds to one of the objects.
`0028. In another exemplary embodiment, an apparatus for
`facilitating advertising in mobile visual search systems is
`provided. The apparatus comprises a processing element con
`figured to define and store one or more objects and receive
`captured images of objects from a device. The apparatus is
`further configured to automatically send media data to the
`device, when the captured images received from the device
`include data that is associated with one of the defined objects.
`0029. In yet another exemplary embodiment, an apparatus
`for associating images with one or more points-of-interest to
`determine the location of the point-of-interest is provided.
`The apparatus comprises a processing element configured to
`receive captured images of one or more objects, remove fea
`tures from the images and generate a group of images that
`share features. Each of the images of the group are associated
`with a point. The processing element is further configured to
`determine whether the group is associated with a shape of an
`object captured in one of the images based on a predetermined
`number of points corresponding to the images of the group,
`associate the group to a single object when the determination
`reveals that there are a predetermined number of points and
`determine the location of the at least one object of the images
`on the basis of the points.
`
`BRIEF DESCRIPTION OF THE SEVERAL
`VIEWS OF THE DRAWINGS
`
`0030 Having thus described the invention in general
`terms, reference will now be made to the accompanying
`drawings, which are not necessarily drawn to scale, and
`wherein:
`0031
`FIG. 1 is a schematic block diagram of a mobile
`terminal according to an exemplary embodiment of the
`present invention;
`0032 FIG. 2 is a schematic block diagram of a wireless
`communications system according to an exemplary embodi
`ment of the present invention;
`0033 FIG.3 illustrates a visual search system according to
`an exemplary embodiment of the invention;
`0034 FIG. 4 illustrates a flowchart of a method of switch
`ing between camera and map views of a terminal according to
`an exemplary embodiment of the invention;
`0035 FIG. 5 illustrates a server according to exemplary
`embodiments of the present invention:
`0036 FIG. 6 illustrates a map view with superimposed
`visual tags according to an exemplary embodiment of the
`invention;
`0037 FIG. 7 illustrates a map view with overcrowded
`visual tags of points-of-interest according to an exemplary
`embodiment of the present invention;
`0038 FIG. 8A illustrates a camera view of a mobile ter
`minal with visual search results according to an exemplary
`embodiment of the present invention;
`0039 FIG.8B illustrates a map view of a mobile terminal
`having visual tags according to an exemplary embodiment of
`the present invention;
`0040 FIG. 9 illustrates a flowchart of a method of
`enabling advertising in mobile visual search systems accord
`ing to an exemplary embodiment of the invention;
`
`FIG. 10 illustrates a flowchart for associating
`0041
`images with one or more POI(s) to determine the location of
`the POI according to an exemplary embodiment of the inven
`tion; and
`0042 FIG. 11 illustrates a system for associating images
`with points-of-interest.
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`0043 Embodiments of the present invention will now be
`described more fully hereinafter with reference to the accom
`panying drawings, in which some, but not all embodiments of
`the invention are shown. Indeed, the invention may be embod
`ied in many different forms and should not be construed as
`limited to the embodiments set forth herein; rather, these
`embodiments are provided so that this disclosure will satisfy
`applicable legal requirements. Like reference numerals refer
`to like elements throughout.
`0044 FIG. 1 illustrates a block diagram of a mobile ter
`minal 10 that would benefit from the present invention. It
`should be understood, however, that a mobile telephone as
`illustrated and hereinafter