`
`I IIIII IIIIIIII II llllll lllll lllll lllll lllll lllll lllll lllll lllll lllll lllll 111111111111111111
`US 20070005363Al
`
`c19) United States
`02) Patent Application Publication (10) Pub. No.: US 2007/0005363 Al
`Jan. 4, 2007
`Cucerzan et al.
`(43) Pub. Date:
`
`(54) LOCATION AWARE MULTI-MODAL
`MULTI-LINGUAL DEVICE
`
`(75) Inventors: Silviu-Petru Cucerzan, Redmond, WA
`(US); David Joshua Kurlander,
`Seattle, WA (US); Michael J. Sinclair,
`Kirkland, WA (US); Yuan Kong,
`Kirkland, WA (US); Zhengyou Zhang,
`Bellevue, WA (US); David W.
`Williams, Woodinville, WA (US); Eric
`D. Brill, Redmond, WA (US)
`
`Correspondence Address:
`AMIN. TUROCY & CALVIN, LLP
`24TH FLOOR, NATIONAL CITY CENTER
`1900 EAST NINTH STREET
`CLEVELAND, OH 44114 (US)
`
`(73) Assignee: Microsoft Corporation, Redmond, WA
`
`(21) Appl. No.:
`
`11/169,456
`
`(22) Filed:
`
`Jun.29, 2005
`
`Publication Classification
`
`(51) Int. Cl.
`GJOL 15/14
`(2006.01)
`(52) U.S. Cl .
`
`
`.............................................................. 704/256
`
`(57)
`
`ABSTRACT
`
`Location-based technologies (e.g., global position system
`(GPS)) can be employed to facilitate providing multi-modal,
`multi-lingual location-based services. Identification ofloca(cid:173)
`tion can provide significant context as to identifying user
`state and intentions. Thus, location identification can facili(cid:173)
`tate providing/augmenting data and services (e.g., location(cid:173)
`aware based suggestions, truncating contact lists based upon
`location, location-based reminders as a user approaches a
`predetermined location, truncating pre-loaded tasks, sug(cid:173)
`gesting routes to accomplish pre-loaded tasks in a PIM). Still
`other aspects can augment GPS location identification with
`a compass, accelerometer, azimuth control, cellular triangu(cid:173)
`lation, SPOT services of telephone, etc. Effectively, these
`alternative aspects can facilitate determination of a target
`location by detecting movement and direction of a user
`and/or portable device.
`
`r 100
`
`FROM POSITIONING
`SYSTEM
`
`INFORMATION STORE,
`INTERNET, etc.
`
`LOCATION CONTEXT
`COMPONENT
`
`106
`LOCATION
`DETECTION
`ENGINE
`
`-----,
`
`I
`I
`I
`108 I
`:,,.--l02
`LOCATION
`ANALYZER
`:
`COMPONENT :
`
`-
`
`-
`
`- - --
`
`------
`
`I
`
`_I
`
`MUL Tl-LANGUAGE
`COMPONENT
`
`104
`
`USER, PORT ABLE DEVICE,
`APPLICATION, etc.
`
`IPR2020-01192
`Apple EX1040 Page 1
`
`

`

`Patent Application Publication
`
`Jan. 4, 2007 Sheet 1 of 11
`
`US 2007 /0005363 Al
`
`r 100
`
`FROM POSITIONING
`SYSTEM
`
`INFORMATION STORE,
`INTERNET, etc.
`
`-----.,
`
`I
`I
`I
`
`.
`
`---------------
`LOCATION CONTEXT
`COMPONENT
`
`r----
`
`1
`:
`I 106
`:
`LOCATION
`: DETECTION
`ENGINE
`1
`
`,...__ ____ _
`
`108 I
`t,- I02
`LOCATION
`ANALYZER
`:
`COMPONENT :
`I
`- - _I
`
`I
`L-------------
`
`-
`
`-
`
`- - -
`
`- - - - -
`
`MULTI-LANGUAGE
`COMPONENT
`
`104
`
`USER, PORTABLE DEVICE,
`APPLICATION, etc.
`
`FIG. 1
`
`IPR2020-01192
`Apple EX1040 Page 2
`
`

`

`Patent Application Publication
`
`Jan. 4, 2007 Sheet 2 of 11
`
`US 2007 /0005363 Al
`
`START
`
`DETECT LOCATION
`
`. ANALYZE LOCATION
`
`IDENTIFY LOCATION(cid:173)
`BASED SERVICE
`
`202
`
`204
`
`206
`
`210
`
`TRANSLATE CONTENT
`
`212
`
`COMMUNICATE
`LOCATION-BASED
`SERVICE
`
`STOP
`
`FIG. 2
`
`IPR2020-01192
`Apple EX1040 Page 3
`
`

`

`Patent Application Publication
`
`Jan. 4, 2007 Sheet 3 of 11
`
`US 2007 /0005363 Al
`
`r3oo
`
`PORTABLE DEVICE
`
`304
`
`PERSONAL
`INFORMATION
`MANAGER (PIM)
`
`LOCATION
`ANALYZER
`COMPONENT
`
`LOCATION
`DETECTION
`ENGINE
`
`108
`
`106
`
`104
`
`302
`
`FROM POSITIONING
`SYSTEM
`
`MULTI-LANGUAGE
`COMPONENT
`
`USER,
`APPLICATION, etc.
`
`FIG. 3
`
`IPR2020-01192
`Apple EX1040 Page 4
`
`

`

`Patent Application Publication
`
`Jan. 4, 2007 Sheet 4 of 11
`
`US 2007 /0005363 Al
`
`~400
`
`PORT ABLE DEVICE
`408
`
`PIM DATA
`
`406
`
`LOCATION
`ANALYZER
`COMPONENT
`r ________ r __ l
`410
`QUERY
`I
`I
`: COMPONENT :
`
`INFORMATION
`STORE, INTERNET, etc.
`
`404
`
`402
`
`LOCATION
`DETECTION ENGINE
`r _______ _L 41;
`
`GPS ENGINE
`
`:
`I
`
`I
`
`r ____ J __ ,e!~6
`
`FROM POSITIONING
`SYSTEM
`
`1 MOVEMENT :
`: DETECTOR
`-------------
`
`1
`
`412
`
`MULTI-LANGUAGE
`COMPONENT
`
`IPR2020-01192
`Apple EX1040 Page 5
`
`

`

`Patent Application Publication
`
`Jan. 4, 2007 Sheet 5 of 11
`
`US 2007 /0005363 Al
`
`rsoo
`
`PORT ABLE DEVICE
`408
`
`PIM DATA
`COMPONENT
`
`504
`
`INFORMATION
`STORE, INTERNET, etc.
`
`502
`
`FROM POSITIONING
`SYSTEM
`
`LOCATION
`ANALYZER
`COMPONENT
`r--------r
`so~
`FILTER
`I
`I
`: COMPONENT 1
`
`r _____ l _£: _?~~
`
`QUERY
`I
`I
`: COMPONENT :
`
`404
`
`41;
`
`LOCATION
`DETECTION ENGINE
`r-------.L:
`: GPS ENGINE
`I
`
`:
`I
`
`r ____ J __ £:!~6
`
`1 MOVEMENT :
`: DETECTOR
`
`412
`
`MULTI-LANGUAGE
`COMPONENT
`
`FIG. 5
`
`IPR2020-01192
`Apple EX1040 Page 6
`
`

`

`Patent Application Publication Jan. 4, 2007 Sheet 6 of 11
`
`US 2007 /0005363 Al
`
`r6oo
`
`----------
`r-------------------------------
`MULTI-MODAL PORTABLE DEVICE
`
`LOCATION DETECTION
`ENGINE
`
`604
`
`606
`
`ANALYZER COMPONENT
`
`616
`LOCATION-BASED
`SERVICE1
`
`614
`LOCATION-BASED
`DATA1
`
`•••
`
`•••
`
`616
`LOCATION-BASED
`SERVICEN
`
`614
`LOCATION-BASED
`DATAM
`
`v-602
`
`610
`
`618
`
`PIM DATA COMPONENT
`
`MULTI-LANGUAGE
`COMPONENT
`
`FIG. 6
`
`IPR2020-01192
`Apple EX1040 Page 7
`
`

`

`~ --· 0 =
`""O = O"' --· ~
`~ --· 0 =
`~ "Cl --· ~
`
`704
`DEVICE
`INPUT
`
`OPERATOR
`
`726
`
`708
`
`706
`
`,100
`
`724
`
`PIM DATA
`
`716
`
`738
`
`COMMUNICATIONS
`
`DATA
`
`714
`
`STORAGE
`MEMORY&
`---~--
`
`.Jio,J
`
`FIG. 7
`
`PROCESSOR
`
`____
`
`720
`
`718
`
`702
`
`712
`
`710
`
`POWER SOURCE
`SUPPLEMENT AL
`
`POWER SOURCE
`
`ONBOARD
`
`DRIVER
`DISPLAY
`
`DISPLAY
`
`736
`
`DETECTOR
`MOVEMENT
`
`GLOBAL POSITION ----.
`734
`732
`
`VOICE REC.
`
`ENGINE
`
`TRANSLATOR
`
`AUDIOI/0
`
`730
`
`728
`
`IPR2020-01192
`Apple EX1040 Page 8
`
`

`

`~ --· 0 =
`""O = O"' --· ~
`~ --· 0 =
`~ "Cl --· ~
`
`~704
`
`DEVICE
`INPUT
`
`OPERATOR
`
`.
`
`-
`
`r726
`
`r708
`
`DRIVER
`DISPLAY
`
`r
`
`!
`
`~706
`
`DISPLAY
`
`,soo
`
`,...
`...
`...
`
`-
`-
`-
`
`702
`
`PROCESSOR
`!
`
`'"
`
`,i,
`
`r
`
`~
`
`r
`
`~
`
`,..
`
`-,
`
`I
`
`""·
`
`736_)
`
`DETECTOR
`MOVEMENT
`
`PIM DATA
`
`724
`
`.
`
`-
`
`716 _)
`
`r13s
`
`COMMUNICATIONS
`
`r714
`
`DATA
`
`1
`
`STORAGE
`MEMORY&
`
`.
`
`-
`
`l
`
`~ TXI
`
`t \
`
`FIG. 8
`
`122-L
`
`I
`
`720 _J
`
`718-......
`
`1
`7 .~ +
`
`802 _)
`
`-...
`
`COMPONENT
`
`AI
`
`POWER SOURCE
`SUPPLEMENTAL
`
`POWER SOURCE I\__ 710
`
`~712
`
`t
`
`GLOBAL POSITION
`734\
`732-H VOICE REC. I ~
`
`-
`
`~
`
`-
`
`ONBOARD
`
`ENGINE
`
`iTRANSLATORI
`
`AUDIOI/0
`
`730-
`
`728-......
`
`IPR2020-01192
`Apple EX1040 Page 9
`
`

`

`PIM DATA
`
`724
`
`,.
`
`-
`
`716J
`
`r13s
`
`COMMUNICATIONS
`
`r714
`
`DATA
`
`!
`
`STORAGE
`MEMORY&
`
`-
`
`,
`
`~ --· 0 =
`""O = O"' --· ~
`~ --· 0 =
`~ "Cl --· ~
`
`~704
`
`DEVICE
`INPUT
`
`OPERATOR
`
`-
`
`,
`
`r726
`
`r1os
`
`v-706
`
`r9oo
`
`t
`DRIVER
`DISPLAY
`
`!
`
`DISPLAY
`
`a
`
`736 _)
`
`DETECTOR
`MOVEMENT
`
`732-H VOICE REC. I ,;
`l TRANSLATOR I
`
`730-
`
`AUDIOI/0
`
`728--.
`
`FIG. 9
`
`t \
`ITI(I
`l
`
`722~
`
`I
`
`720 __,/
`
`RX
`
`718--~
`
`902-_;
`COMPONENT
`
`~
`
`RULES
`
`POWER SOURCE
`SUPPLEMENTAL
`
`I
`
`-
`-
`-
`-
`-
`-
`
`!
`
`'I,
`
`~~
`
`_/ ... t
`... PROCESSOR
`. ,
`
`702
`
`,
`~
`
`I
`
`:\.._ 710
`
`--
`
`~712
`
`POWER SOURCE
`
`ONBOARD
`
`t
`
`-
`
`-
`
`ENGINE
`
`GLOBAL POSITION
`734\
`
`IPR2020-01192
`Apple EX1040 Page 10
`
`

`

`Patent Application Publication
`
`Jan. 4, 2007 Sheet 10 of 11
`
`US 2007 /0005363 Al
`
`rlOOO
`
`I ·------------
`
`.... --
`
`.... - .......... 1
`
`,--------------r
`1030 ______ •
`: _ OPERATING_SYSTEM _:
`,-----···------r
`I 032 _
`: APPLICATIONS !
`
`,-------....... r ______ _
`1034
`: MODULES
`:
`I --------·---------------1
`
`______________ r_ I 036
`I!
`DATA
`
`I ...................................................
`
`I
`I
`I
`
`!
`
`1
`
`1002
`
`1004
`
`1006
`
`PROCESSING
`UNIT
`
`1008
`SYSTEM
`MEMORY
`
`RAM
`
`ROM
`
`----------,
`-----+----,
`1014
`~-
`-"'r
`: ixriwi.
`(
`\ - - - _l-!QQ - - - )
`
`1026
`INTERFACE
`
`en
`::,
`o::i
`
`FDD
`
`DISK
`
`OPTICAL
`DRIVE
`DISK
`
`VIDEO
`ADAPTOR
`
`INPUT
`DEVICE
`INTERFACE
`
`NETWORK
`ADAPTOR
`
`042
`l
`(WIRED/WIRELESS)
`1058
`
`MODEM
`
`1056
`
`(WIRED/WIRELESS)
`
`FIG. 10
`
`1044
`
`MONITOR
`
`1038
`
`KEYBOARD
`
`1040
`
`MOUSE
`
`1048
`
`REMOTE
`COMPUTER(S)
`
`1050
`
`MEMORY/
`STORAGE
`
`IPR2020-01192
`Apple EX1040 Page 11
`
`

`

`Patent Application Publication
`
`Jan. 4, 2007 Sheet 11 of 11
`
`US 2007 /0005363 Al
`
`r 1100
`
`1104
`
`SERVER(S)
`
`1102
`
`CLIENT(S)
`
`COMMUNICATION
`FRAMEWORK
`
`1108
`
`1110
`
`CLIENT DATA STORE(S)
`
`SERVER DATA STORE(S)
`
`FIG. 11
`
`IPR2020-01192
`Apple EX1040 Page 12
`
`

`

`US 2007 /0005363 Al
`
`Jan. 4, 2007
`
`I
`
`LOCATION AWARE MULTI-MODAL
`MULTI-LINGUAL DEVICE
`
`BACKGROUND
`
`[0001] Both enterprises and individuals are increasingly
`interested in using handheld devices. Most modem handheld
`devices are equipped with multiple sensors (e.g., micro(cid:173)
`phone, wireless transmitter, global positioning system (GPS)
`engine, camera, stylus, etc.). However, there are no appli(cid:173)
`cations available that make full use of multiple sensors. In
`other words, multi-sensory technologies that make handheld
`devices a multi-modal, multi-lingual mobile assistant are not
`available.
`
`[0002] Today, cellular telephones nmning on state-of-the(cid:173)
`art operating systems have increased computing power in
`hardware and increased features in software in relation to
`earlier technologies. For instance, cellular telephones are
`often equipped with built-in digital image capture devices
`(e.g., cameras) and microphones together with computing
`functionalities of personal digital assistants (PDAs). Since
`these devices combine the functionality of cellular tele(cid:173)
`phones with the functionality of PD As, they are commonly
`referred to as "smartphones." The hardware and software
`features available in these smartphones and similar techno(cid:173)
`logically capable devices provide developers the capability
`and flexibility to build applications through a versatile
`platform. The increasing market penetration of these por(cid:173)
`table devices (e.g., PDAs) inspires programmers to build
`applications, Internet browsers, etc. for these smartphones.
`
`[0003] The Internet continues to make available ever(cid:173)
`increasing an1ounts of information which can be stored in
`databases and accessed therefrom. Additionally, with the
`proliferation of portable terminals (e.g., notebook comput(cid:173)
`ers, cellular telephones, PDAs, smartphones and other simi(cid:173)
`lar communications devices), users are becoming more
`mobile, and hence, trackable with respect to buying habits
`and locations that they tend to frequent. For example, many
`devices are being designed with a geographic location
`tracking technology such as GPS for reasons of safety,
`finding travel destinations, etc. Thus, it now becomes pos(cid:173)
`sible to determine the location of the user.
`
`[0004] Location identification systems are used in many
`aspects of everyday life. By way of exan1ple, it has become
`increasingly more common for GPS to be integrated into
`automobiles to assist in navigation. Generally, a GPS system
`can, by triangulation of signals from three satellites, pinpoint
`a current location virtually anywhere on earth to within a
`few meters.
`
`[0005] Knowledge of where the user has traveled, cur(cid:173)
`rently is, and is heading in an urban canyon, which includes
`structures such as multi-story buildings (principally, and
`whether the user is inside or outside of the building), but also
`include trees, hills, and tum1els (generally), can be of value
`to the user and to companies that seek to benefit economi(cid:173)
`cally by knowledge of the user location by providing loca(cid:173)
`tion-based data and services to the user.
`
`SUMMARY
`
`[0006] The following presents a simplified summary of the
`invention in order to provide a basic understanding of some
`aspects of the invention. This sullllliary is not an extensive
`
`overview of the invention. It is not intended to identify
`key/critical elements of the invention or to delineate the
`scope of the invention. Its sole purpose is to present some
`concepts of the invention in a simplified form as a prelude
`to the more detailed description that is presented later.
`
`[0007] The invention disclosed and claimed herein, in one
`aspect thereof, comprises a system that facilitates multi(cid:173)
`modaL multilingual location-based data and/or services.
`More particularly, aspects can employ a global position
`system (GPS) that identifies a current location of a user
`and/or portable device. Additionally, the system can main(cid:173)
`tain a log of locations with respect to the portable device
`and/or user. This log can be employed by an artificial
`intelligence (AI) and/or mies-based logic to infer a destina(cid:173)
`tion location. Accordingly, location-based data and/or ser(cid:173)
`vices can be generated with respect to the current or inferred
`location.
`
`[0008] A multi-language component can be provided that
`facilitates comprehendible commm1ication. For example,
`the multi-language component can translate text and/or
`voice communication of the location-based data and services
`into a language comprehendible by a user or recipient. As
`well, the multi-language component can be employed to
`translate location-based data and/or services into a language
`( or dialect) that corresponds to the current ( or destination)
`location. Moreover, aspects employ the multi-language
`component to translate any desired communication into any
`desired language and/or dialect.
`
`[0009] The invention can employ a local and/or remote
`store to identify and render location-based data and/or
`services. For exan1ple, a local personal infomiation manager
`(PIM) can be employed whereby contacts and calendar
`entries can be filtered and/or sorted based upon a location.
`As well, remote sources (e.g., remote server, Internet) can be
`employed to identify location-based information thereafter,
`rendering the identified information to a user, application,
`etc.
`
`[0010] Location-based technologies (e.g., GPS, wireless,
`etc.) can be employed to facilitate providing multi-modal,
`multi-lingual location-based services. Identification ofloca(cid:173)
`tion can provide significant context as to identifying user
`state and intentions. Thus, location identification can facili(cid:173)
`tate providing/augmenting data and services (e.g., location(cid:173)
`aware based suggestions, trnncating contact lists based upon
`location, location-based reminders such as pick up dry
`cleaning as a user approaches a location of the cleaners).
`
`[0011] Still other aspects can augment GPS location iden(cid:173)
`tification with a compass, accelerometer, azimuth control,
`cellular triangulation, etc. Effectively, these alternative
`aspects can facilitate determination of a target location by
`detecting movement and direction of a user and/or portable
`device. For example, aspects can employ GPS or other
`suitable technology to determine a current location of a
`device. Additionally, motion and direction sensor technolo(cid:173)
`gies can be employed to detennine a relevant movement of
`the device. Accordingly, the aspect can infer and/or calculate
`a probable target location and time corresponding therewith.
`As a result, location-based data and services can be provided
`with respect to a current and/or target location.
`
`[0012] In yet another aspect thereof, an AI component is
`provided that employs a probabilistic and/or statistical-
`
`IPR2020-01192
`Apple EX1040 Page 13
`
`

`

`US 2007 /0005363 Al
`
`Jan. 4, 2007
`
`2
`
`based analysis to prognose or infer an action that a user
`desires to be automatically perfonned. The AI reasoning
`and/or learning logic can be provided to facilitate inferring
`and/or predicting a location of a user and/or device. Rules(cid:173)
`based logic can also be provided in addition to or in place of
`the AI component. The rules-based logic component can
`facilitate automating functionality in accordance with a
`predefined or preprogrammed rnle.
`
`[0013] To the accomplishment of the foregoing and related
`ends, certain illustrative aspects of the invention are
`described herein in connection with the following descrip(cid:173)
`tion and the annexed drawings. These aspects are indicative,
`however, of but a few of the various ways in which the
`principles of the invention can be employed and the subject
`invention is intended to include all such aspects and their
`equivalents. Other advantages and novel features of the
`invention will become apparent from the following detailed
`description of the invention when considered in conjunction
`with the drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`[0014] FIG. l illustrates a system that facilitates providing
`multi-lingual location-based functionality in accordance
`with an aspect of the invention.
`
`[0015] FIG. 2 illustrates an exemplary flow chart of pro(cid:173)
`cedures that facilitate providing multi-lingual location-based
`functionality in accordance with an aspect of the invention.
`
`[0016] FIG. 3 illustrates a block diagram of a portable
`device having a location detection system and a multi(cid:173)
`language component in accordance with an aspect of the
`invention.
`
`[0017] FIG. 4 illustrates a portable device that employs a
`global position system (GPS) and a movement detector to
`identify location-based data and services in accordance with
`an aspect of the invention.
`
`[0018] FIG. 5 illustrates a portable device that includes a
`query component and a filter component that facilitates
`identifying location-based data and services in accordance
`with an aspect of the invention.
`
`[0019] FIG. 6 illustrates a multi-modal, multi-lingual por(cid:173)
`table device that facilitates rendering multiple location(cid:173)
`based data and service components in accordance with an
`aspect of the invention.
`
`[0020] FIG. 7 is a schematic block diagram of a portable
`handheld device according to one aspect of the subject
`invention.
`
`[0021] FIG. 8 illustrates an architecture of a portable
`handheld device including an artificial intelligence-based
`component that can automate functionality in accordance
`with an aspect of the invention.
`
`[0022] FIG. 9 illustrates an architecture of a portable
`handheld device including a rules-based logic component
`that can automate functionality in accordance with an aspect
`of the invention.
`
`[0023] FIG. 10 illustrates a block diagram of a computer
`operable to execute the disclosed architectnre.
`
`[0024] FIG. 11 illustrates a schematic block diagram of an
`exemplary computing environn1ent in accordance with the
`subject invention.
`
`DETAILED DESCRIPTION
`
`[0025] The invention is now described with reference to
`the drawings, wherein like reference numerals are used to
`refer to like elements throughout. In the following descrip(cid:173)
`tion, for purposes of explanation, numerous specific details
`are set forth in order to provide a thorough understanding of
`the subject invention. It may be evident, however, that the
`invention can be practiced without these specific details. In
`other instances, well-known structures and devices are
`shown in block diagram form in order to facilitate describing
`the invention.
`
`[0026] As used in this application, the terms "component"
`and "system" are intended to refer to a computer-related
`entity, either hardware, a combination of hardware and
`software, software, or software in execution. For example, a
`component can be, but is not limited to being, a process
`rum1ing on a processor, a processor, an object, an executable,
`a thread of execution, a program, and/or a computer. By way
`of illustration, both an application rnmling on a server and
`the server can be a component. One or more components can
`reside within a process and/or thread of execution, and a
`component can be localized on one computer and/or dis(cid:173)
`tributed between two or more computers.
`
`[0027] As used herein, the tenns to "infer" and "inference"
`refer generally to the process of reasoning about or inferring
`states of the system, enviromnent, and/or user from a set of
`observations as captnred via events and/or data. Inference
`can be employed to identify a specific context or action, or
`can generate a probability distribution over states, for
`example. Inference can also refer to techniques employed
`for composing higher-level events from a set of events
`and/or data. Such inference results in the construction of
`new events or actions from a set of observed events and/or
`stored event data, whether or not the events are correlated in
`close temporal proximity, and whether the events and data
`come from one or several event and data sources.
`
`[0028] Referring initially to the drawings, FIG. 1 illus(cid:173)
`trates a multi-lingual system 100 having location identifi(cid:173)
`cation functionality in accordance with an aspect of the
`invention. The multi-lingual system 100 of FIG. 1 can affect
`identification and/or generation of location-based data and
`services. Generally, system 100 can include a location
`context component 102 and a multi-language component
`104. These components and their corresponding functional(cid:173)
`ity are described in greater detail below.
`
`[0029] The location context component 102 can facilitate
`identifying and providing content that corresponds to a
`location of a portable device. In accordance therewith, the
`location context component 102 can facilitate identifying a
`location of a device and thereafter can identify data and/or
`services that correspond to the location. In one aspect, the
`location context component 102 can include a location
`detection engine 106 and a location analyzer component
`108.
`
`[0030] As illustrated in FIG. 1, the location detection
`engine 106 can interface with a positioning system (e.g.,
`global position system (GPS)) to determine a current loca(cid:173)
`tion of a device. Once the location is determined, the
`location analyzer component 108 can facilitate identification
`and/or generation of location-based content. Additionally,
`the multi-language component 104 can facilitate formatting
`
`IPR2020-01192
`Apple EX1040 Page 14
`
`

`

`US 2007 /0005363 Al
`
`Jan. 4, 2007
`
`3
`
`the content ( e.g., data, service) into a language comprehend(cid:173)
`ible to a user or group of users and/or recipients.
`[0031] By way of example, in accordance with an aspect,
`suppose that a user intends to purchase a headset. Further,
`suppose that the user receives a catalog which has a set of
`headset collections available from a particular store. In
`accordance with the novel functionality described herein,
`the user can employ a suitably equipped portable device to
`capture an image of the catalog page that illustrates the
`desired product. As well, voice recognition fonctionality can
`be employed to correct or augment interpretation or recog(cid:173)
`nition of the image. In one example, the catalog listing
`contains both telephone number and address information of
`the store.
`[0032] Once the image is scanned and analyzed, the user
`can instruct the device to "call the store." Thus, the tele(cid:173)
`phone conmrnnication can be automatically established. In
`the event that the callee does not speak the same language
`as the caller (e.g., user), the device can determine a native
`dialect either through location detection or telephone num(cid:173)
`ber analysis. In other words, the device can employ location
`detection techniques to determine a current location of the
`telephone thereby inferring the native spoken language.
`Accordingly. the multi-lingual fonctionality of the device
`can be employed to translate incoming and outgoing speech
`signals thereby enabling comprehendible commtmication.
`[0033] Continuing with the example, the user can inquire
`to the receptionist when the store will close. Thus, a deter(cid:173)
`mination can be made if time permits to visit the store. This
`determination can be made by automatically querying cal(cid:173)
`endar appointments with respect to the current time and
`distance to the store location. It will be appreciated that a
`distance to the store from the device location can be auto(cid:173)
`matically calculated using satellite location detection, cel(cid:173)
`lular triangulation or the like.
`[0034] Upon an affirmative determination that time exists
`to visit the store, the novel functionality can automatically
`access directions to the store. Since the device can be
`wirelessly-connected to the Internet, the user can instruct
`(e.g., audibly) the device to contact a service provider
`thereby generating specific directions to the store. Again, the
`satellite positioning system can facilitate identification of the
`reference ( e.g., current) location. This current location can
`be employed together with the target location ( e.g., store
`address) in order to establish directions to the store.
`[0035] With continued reference to the example, while in
`the store, the user sees a product called "WITTY Wireless
`Stereo Headset". In response thereto, the user scans (or
`captures an image of) the product name and reads it at the
`same time in order to get accurate recognition. Again, the
`device, through a suitable wireless counection, can generate
`product reviews from the Internet or other network/server.
`Additionally, in accordance with the location detection sys(cid:173)
`tem, the device can notify the user of nearby locations to
`purchase the same, or similar, product at a better price. It is
`to be understood that this scenario is provided to detail some
`of the novel fonctionalities described herein. It will be
`appreciated that other multi-modal, multi-lingual aspects
`exist and are to be included in the scope of this specification
`as well as the claims appended hereto.
`[0036] FIG. 2 illustrates a procedure flow of providing
`location-based information and/or services in accordance
`
`with an aspect of the invention. While, for purposes of
`simplicity of explanation. the one or more methodologies
`shown herein, e.g., in the form of a flow chart, are shown and
`described as a series of acts, it is to be understood and
`appreciated that the subject invention is not limited by the
`order of acts, as some acts may, in accordance with the
`invention, occur in a different order and/or concurrently with
`other acts from that shown and described herein. For
`example, those skilled in the art will understand and appre(cid:173)
`ciate that a methodology could alternatively be represented
`as a series of interrelated states or events, such as in a state
`diagram. Moreover, not all illustrated acts may be required
`to implement a methodology in accordance with the inven(cid:173)
`tion.
`
`[0037] At 202, a location of a user and/or portable device
`can be detected. It is to be appreciated that any suitable
`method of identifying the location can be employed in
`accordance with the invention. By way of example, but not
`limitation, the location can be detected using methods such
`as GPS and cellular triangulation. It is to be appreciated that
`other aspects can be employed to additionally determine if
`the user and/or device is in motion. For example, services
`such as accelerometers, compasses, azimuth controls. and
`SPOT services can be employed to augment the location.
`
`[0038] Once the location is determined, at 204, it can be
`analyzed with respect to an information store, Internet
`content, personal infonnation manager (PIM) or the like.
`Accordingly, a location-based service (or data) can be iden(cid:173)
`tified at 206. At 208, a determination is made if, based upon
`user identity, inference, historical and/or statistical analysis,
`etc., translation is necessary. If, at 208. a detennination is
`made that that translation is necessary, the content can be
`translated into an appropriate language and/or dialect of a
`user ( or group of users). As described above, location
`detection can be employed to determine an appropriate
`language for which to translate.
`
`[0039] On the other hand, if at 208 a determination is
`made that the identified service and/or data is fonnatted in
`an appropriate language and/or dialect corresponding to the
`instant user, the identified location-based service can be
`connnunicated to the user in lieu of re-formatting (e.g.,
`translation). It is to be appreciated that this conmrnnication
`can occur in either an audible (e.g., spoken) or textual
`manner.
`
`[0040] Referring now to FIG. 3, a system 300 that facili(cid:173)
`tates providing a user with location-based information is
`illustrated. Generally, the system 300 can include a portable
`device 302 capable of conummicating with a location iden(cid:173)
`tification system (e.g., GPS) as shown. As illustrated, por(cid:173)
`table device 302 can include a multi-language component
`104, a location detection engine 106, a location analyzer
`component 108 and a PIM component 304.
`
`[0041] In operation, the location detection engine 106 can
`connnunicate with a positioning system to determine a
`location of the portable device 302. Once determined, the
`location analyzer component 108 can conununicate with the
`PIM component 304 to obtain location-based information.
`For example, location analyzer component 108 can query
`and/or filter data of the PIM component 304 based upon the
`identified location. In one specific example, contact lists
`maintained within the PIM component 304 can be truncated
`or sorted based upon the location or distance from the
`
`IPR2020-01192
`Apple EX1040 Page 15
`
`

`

`US 2007 /0005363 Al
`
`Jan. 4, 2007
`
`4
`
`determined location. In other words, the analyzer component
`108 can truncate and/or sort a contact list based upon a
`distance relative to/from the current location. It will be
`appreciated that this data manipulation can be based upon
`any predefined criteria ( e.g., rule) and/or artificial intelli(cid:173)
`gence (AI) inference based analysis.
`
`[0042] As stated supra, many modem handheld devices
`( e.g .. cellular telephones, smartphones, pocket computers,
`personal data assistants (PDAs)) are equipped with multiple
`sensors. For exan1ple, it is c01lll11011 for a portable device to
`include a microphone, wireless transmitter, GPS, image
`capture device (e.g., camera, scanner), stylus, etc. In one
`aspect because the invention employs a multi-language
`component 104, the fimctionality described herein can be
`particularly useful to international travelers to assist in
`communication in view of potential language barriers. For
`example, foreign travelers can employ the device 302 for
`understanding foreign signs and restaurant menus as well as
`for asking non-English speakers for directions using
`English. It is to be appreciated that the invention can be
`employed to conununicate using any native or base language
`and/or dialect.
`
`[0043] Additionally, as described above, the invention can
`be employed to regionally organize and/or truncate PIM data
`( or other data) based upon a location. Other aspects exist
`whereby the invention can infer a target location thus PIM
`data can be organized and/or truncated accordingly. Effec(cid:173)
`tively, the invention can provide intelligent assistance to a
`user by leveraging redundancy and complimentarily of
`multi-modal information. Therefore, it can be possible to
`produce significantly better results than if one single modal(cid:173)
`ity is used.
`
`[0044] The following scenarios are provided merely to add
`context to the invention and are not intended to limit the
`invention in any way. In other words, the scenarios included
`herein are provided to illustrate exemplary situations with
`regard to the novel functionality of the invention. As such,
`the scope of this disclosure and claims appended hereto are
`not to be limited by these exemplary scenarios.
`
`[0045] In a first scenario, suppose a user is on a trip to
`Beijing from his native United States of America. Upon
`arriving at Beijing Airport, the user recognizes that all signs
`are in the native language of Beijing, Chinese. He cam10t ask
`his fellow travelers to translate the signs because they do not
`speak English. Therefore, he takes out his portable device
`and snaps a photo of the signs with the built-in camera.
`Based upon GPS information, the portable device can iden(cid:173)
`tify that he is in China. Accordingly, the device can imme(cid:173)
`diately translate the Chinese signs into English. Thus, he is
`able to locate the Customer Services and Baggage Claim
`areas without any difficulty even though he does not speak
`the Chinese language.
`
`[0046] In addition to this scenario, it is to be appreciated
`that the location detection engine 106 together with the
`location analyzer component 108 can be employed to deter(cid:173)
`mine a location and regional dialect. The multi-language
`( e.g., multi-lingual) component can be employed to convert
`or translate the information captured in a foreign dialect into
`a comprehendible dialect.
`
`[0047] Referring no