Page 1 of 36
`
`Google Inc., Nest Labs, Inc., and Dropcam, Inc.
`GOOG 1007
`IPR of US Pat. No. 8,311,524
`
`

`

`WO 2009/043020
`
`PCT/US2008/078148
`
`SYSTEM AND METHOD FOR INJECTING SENSED PRESENCE INTO
`
`SOCIAL NETWORKING APPLICATIONS
`
`RELATED APPLICATIONS
`
`[0001]
`
`This application claims benefit of priority to United States
`
`Provisional Patent Application Serial Number 60/976,371 filed 28 September 2007,
`
`which is incorporated herein by reference.
`
`BACKGROUND
`
`10
`
`15
`
`20
`
`25
`
`[0002]
`
`Presence is currently limited to a user’s contactable status. For
`
`example, a first user’s status and availability is provided by presence servers in a
`
`network environment. A second user needing to contact the first user may thereby
`
`determine the optimal way (and likelihood of success) in contacting the second user.
`
`A presence server may determine this status by detecting events made by the first
`
`user. For example, if the first user is typing on a keyboard of a computer, these key
`
`input events indicate that the first user is sitting at, and using, the computer. The
`
`presence server thus displays the status of the first user as ‘using the computer’.
`
`Using this presence information, the second user may decide to send an instant
`
`message to the first user’s computer to initiate communication. Alternatively, the
`
`second user may decide to call a telephone located near the first user’s computer,
`
`knowing that the first user will probably answer.
`
`[0003]
`
`Thus, presence is a useful tool for making informed decisions
`
`about other users. However, this presence information is limited to locating and
`
`communicating with its users.
`
`[0004]
`
`Social networking applications; such as MySpace, Facebook,
`
`Skype, allow users to interactively define their presence in greater detail, thereby
`
`allowing other permitted users to view this additional presence information.
`
`However, the burden of updating the presence information interactively typically
`
`results in infrequent updates and therefore often old (and Often incorrect) presence
`
`information.
`
`Page 2 of 36
`
`Page 2 of 36
`
`

`

`WO 2009/043020
`
`PCT/US2008/078148
`
`SUMMARY
`
`[0005]
`
`A method for injecting sensed presence into social networking
`
`applications includes receiving sensor data associated with a user, inferring a presence
`
`status of the user based upon analysis of the sensor data, storing the sensor data and
`
`presence status within a database, and sending the presence status to a social
`
`networking server to update the user’s presence information for the social networking
`
`applications based upon the user’s preferences.
`
`[0006]
`
`A software product includes instructions, stored on computer—
`
`readable media, where the instructions, when executed by a computer, perform steps
`
`10
`
`for injecting sensed presence into social networking applications. The instructions
`
`include instructions for receiving sensor data associated with a user, instructions for
`
`inferring a presence status of the user based upon analysis of the sensor data,
`
`instructions for storing the sensor data and presence status within a database, and
`
`instructions for sending the presence status to a social networking server to update the
`
`15
`
`user’s presence information for the social networking applications based upon the
`
`user’s preferences.
`
`[0007]
`
`A system for injecting sensed presence into social networking
`
`applications includes at least one sensor proximate to a user, the at least one sensor
`
`being used for collecting sensor data associated with the user, a presence server for
`
`2O
`
`receiving and storing the sensor data, an inference engine for analyzing the stored data
`
`and to infer a presence status for the user, and a presentation engine for presenting the
`
`information to the user and other users.
`
`BRIEF DESCRIPTION OF THE FIGURES
`
`[0008]
`
`FIG. 1 shows a system for injecting sensed presence into social
`
`25
`
`networking applications, according to an embodiment.
`
`[0009]
`
`FIG. 2 shows the presence server of FIG. 1 interacting with
`
`applications, a cell phone, a PDA, embedded sensors, and a notebook computer.
`
`[0010]
`
`FIG. 3 is a flowchart illustrating one exemplary method for
`
`injecting sensed presence into social networking applications.
`
`30
`
`[0011]
`
`FIG. 4 shows an example of a snapshot of a user’s data page as
`
`accessed from a portal of the system of FIG. 1.
`
`Page 3 of 36
`
`Page 3 of 36
`
`

`

`WO 2009/043020
`
`PCT/US2008/078148
`
`DETAILED DESCRIPTION OF THE EMBODIMENTS
`
`[0012]
`
`To make a presence system useful for a social network, additional
`
`sensor input and activity determination is used. That is, presence outside a work
`
`environment is detected to provide a user’s status that is useful in the social
`
`environment. Once the user‘s status is determined, it may be shared with other users,
`
`as permitted by the user.
`
`[0013]
`
`FIG. 1 shows a system 100 for injecting sensed presence into social
`
`networking applications. A user 102 has, for example, a cell phone 106, a personal
`
`digital assistant (PDA) 108 and an embedded sensor unit 110. Cell phone 106 may
`
`10
`
`include a camera for sensing images around user 102 and a global positioning sensor
`
`(GPS) unit for determining the location of user 102. PDA 108 may include a
`
`temperature sensor for sensing temperature near user 102. Embedded sensors 110
`
`may include one or more accelerometers for determining activity of user 102.
`
`Embedded sensors 1 10 may include a transceiver to enable wireless communication
`
`with cell phone 106, PDA 108 and/or network 120. Embedded sensors 1 10 may be
`
`attached to a user’s body (e.g., a user’s running shoes) as illustrated in FIG. 1.
`
`However, embedded sensors may also be used in other manners, such as carried by
`
`another user, attached to personal property (e.g., a user’s bicycle, car, ski boot), or
`
`embedded in the ecosystem of a city or town (e.g., a carbon dioxide sensor or a pollen
`
`20
`
`sensor attached to a structure in a city). Network 120 may include the Internet and
`
`other wired and/or wireless networks.
`
`[0014]
`
`A second user 104 is shown with a notebook computer 112 that
`
`may include one or more sensors that collect information of user 104. Such sensors
`
`are becoming common amongst items carried by people during everyday activities.
`
`25
`
`These sensors are periodically interrogated and resulting sensor data sent to a
`
`presence server 116 via network 120. Presence server 116 analyzes this sensor data to
`
`infer activity of users 102 and 104. For example, sensor data received from sensors
`
`associated with user 102 is used to define a presence status 122 of user 102.
`
`Similarly, sensor data from sensors associated with user 104 is used to define a
`
`30
`
`presence status 124 of user 104.
`
`[0015]
`
`Through analysis of historical server data, presence server 116 may
`
`determine behavioral patterns based on movements of users 102 and 104. For
`
`Page 4 of 36
`
`Page 4 of 36
`
`

`

`WO 2009/043020
`
`PCT/US2008/078148
`
`example. presence server 1 l 6 may infer that user 102 and user 104 frequent the same
`
`coffee shop by determining that user 102 and user 104 visit the coffee shop each
`
`morning when traveling to their respective places of work.
`
`[0016]
`
`Different sensors may be used to collect data associated with a
`
`user, such as one or more of the following: cameras, microphones, accelerometers,
`
`GPS locators, radio sensors (e.g., Bluetooth device contact logs. 80215.4 ranging,
`
`802.1 1 localization), temperature sensors, light sensors, humidity sensors,
`
`magnetometers, button click sensors and device status sensors (e.g., cell phone ringer
`
`status sensors). As noted above, such sensors may be included within devices carried
`
`10
`
`by a user.
`
`[0017]
`
`In another example, wireless sensors external to cell phone 106
`
`may relay information to cell phone 106, which then sends the information to
`
`presence server 116 for processing. For example, embedded sensor unit 110 may
`
`include one or more sensor types that periodically provide sensor data to cell phone
`
`106 via a wireless communication link 1 1 1, such as Bluetooth.
`
`[0018]
`
`Where sensor operation is resource taxing, sensor operation may be
`
`done on demand via an explicit query from presence server 1 l6. Sensor data is
`
`pushed from consumer devices 106, 108, 110 and 112 to presence server 116 where it
`
`is analyzed to infer activity of the associated user.
`
`[0019]
`
`Another type of sensor that may be used in system 100 is a
`
`software sensor that measure artifacts of other software running on a computing
`
`platform to determine a user’s behavior, mood, etc. Since no actual sensor is being
`
`read, these software sensors may also be termed virtual sensors. Where processing
`
`allows, inference may occur within the computing device (e. g, cell phone 106, PDA
`
`108, notebook computer 1 l2), otherwise data is sent to presence server 1 16 for
`
`15
`
`2O
`
`25
`
`inference.
`
`[0020]
`
`1n another example, a sensor may not be immediately associated
`
`with a user, but may be indirectly associated to the user by locality. For example, to
`
`determine air quality associated with user 102, sensor information from a statically
`
`30
`
`deployed sensor infrastructure 114 that measures air quality may be used if that data is
`
`obtained from one or more sensors of the infrastructure that are near to the location of
`
`user 102. Such sensor infrastructures may operate independently of user 102 and
`
`Page 5 of 36
`
`Page 5 of 36
`
`

`

`WO 2009/043020
`
`PCT/US2008/078148
`
`presence server 1 16, but may be matched to user 102 through time and location
`
`information ofuser 102.
`
`In another example, where users 102 and 104 are near one
`
`another and user 104 has one or more different sensor types to those ofuser 102,
`
`while users 102 and 104 are proximate, sensor data from user 104 may be applicable
`
`to user 102 and vice versa. Thus, sensors may be shared.
`
`[0021]
`
`As more and mode consumer devices include sensors, these device
`
`may be used to unobtrusively collect information about a user. By collecting
`
`information related to a user, presence server 1 16 (via data fusion and analysis) may
`
`determine characteristics and life patterns (e. g., presence status 122, 124) of the user
`
`10
`
`and/or of particular groups of users. These characteristics and life patterns may be fed
`
`back to the users in the form of application services 118, such as on a display ofa
`
`consumer device (e.g., cell phone 106). These characteristics (e.g., presence status
`
`122, 124) may also be sent to a social networking server 126 that supports one or
`
`more social network applications 119, such as F acebook and MySpace, and instant
`
`messaging. For example, user 102 may also have an account with a social networking
`
`application such as Facebook, and therefore configures presence server 1 16 to send
`
`presence information of user 102 to social networking server 126 for use by social
`
`networking application 1 19, thereby alleviating the need for user 102 to continually
`
`update presence information within that social networking application. That is,
`
`presence information for one or more associated social networking applications may
`
`be updated with presence information (e. g., presence status 122, 124) for the
`
`associated user (e.g., user 102) by presence server 1 16. Thus, as user 104 accesses
`
`social networking application 119, shared presence status of user 102 is automatically
`
`updated by presence server 116 via social networking server 126.
`
`[0022]
`
`Presence server 1 16 may consist of a set of servers that: (1) hold a
`
`database of users and their associated presences 122, 124; (2) implement a web portal
`
`that provides access to presence information via per—user accounts; and (3) contain
`
`algorithms to draw inferences about many objective and subjective aspects of users
`
`15
`
`20
`
`25
`
`based upon received and stored sensor data. Presence server 116 may include one or
`
`30
`
`more interfaces (e. g., using one or more application programming interfaces (APIs))
`
`to clients (e. g., thin clients) running on consumer computing and communication
`
`devices, such as cell phone 106, PDA 108, and notebook computer 112. The clients
`
`Page 6 of 36
`
`Page 6 of 36
`
`

`

`WO 2009/043020
`
`PCT/US2008/078148
`
`may send via push operation information about the user to presence server 1 16 and
`
`receive from presence server 1 16 inferred information about the user and the user’s
`
`life patterns based on sensed data. Data sensing clients may make use ofsensor
`
`sharing and calibration functions to improve the quality ofdata gathered for each user,
`
`thereby enhancing the performance of presence server 1 16.
`
`[0023]
`
`While processed user information is available (both for individual
`
`review and group sharing) via the web portal, APIs for the retrieval and presentation
`
`of (a subset of) this information are available, through one or more plug-ins (i.e., a
`
`pull operation) to popular social network applications such as Skype, Pidgin,
`
`10
`
`Facebook, and MySpace.
`
`[0024]
`
`FIG. 2 shows (in an embodiment) presence server 1 16 of FIG. 1
`
`interacting with applications 118, cell phone 106, PDA 108, embedded sensors 110
`
`and notebook computer 1 12 , collectively called consumer devices 220, in further
`
`detail. Presence server 1 16 is shown with storage 210, an inference engine 212 and a
`
`presentation engine 214. Storage 210 is used to store received sensor data and user’s
`
`presence status 122, 124. Applications 118 may include instant messaging 202, social
`
`networks 204, multimedia 206, and blogosphere 208.
`
`[0025]
`
`Inference engine 212 analyzes received sensor data and determines
`
`presence information (e. g, 122 and 124) for each user. This presence information is,
`
`for example, presented to designated applications running on one or more user
`
`devices (e. g, phone 106, PDA 108 and notebook computer 112). A user’s presence
`
`information may be pushed to certain devices based upon the user’s permission. That
`
`is, in certain embodiments, the user must allow others to view their presence
`
`information for the information to be made available to, and/or pushed to, other users.
`
`[0026]
`
`Certain embodiments of system 100 support opportune peer-to-
`
`peer communication between consumer devices using available short range radio
`
`technologies, such as Bluetooth and IEEE 802.154. Communication between
`
`consumer devices and presence server 116 takes place according to the availability of
`
`the 802.11 and cellular data channels, which can be impacted both by the device
`
`feature set and by radio coverage. For devices that support multiple communication
`
`modes, communication can be attempted first using a TCP/IP connection over open
`
`15
`
`20
`
`25
`
`30
`
`Page 7 of 36
`
`Page 7 of 36
`
`

`

`WO 2009/043020
`
`PCT/US2008/078148
`
`802.1 1 channels. second using GPRS/EDGE—enabled bulk or stream transfer, and
`
`finally SMS/MMS is used as a fallback, in certain embodiments of system 100.
`
`A.
`
`Sensing
`
`[0027]
`
`lllustratively, a sensing client (e.g., thin sensing client) is installed
`
`on a consumer device (e.g., cell phone 106, PDA 108, and computer 112) to
`
`periodically poll on—board sensors (both hardware and software) and to push the
`
`collected sensor data, via an available network connection (wired or wireless, e.g.,
`
`107, 109, 113), to presence server 116 for analysis and storage. For sensing
`
`modalities that are particularly resource taxing (especially for mobile devices), sensor
`
`10
`
`sampling may be done on demand via an explicit query from presence server 116.
`
`Sampling rates and durations for each of the sensors are set in accordance with the
`
`needs of inference engine 212. Typically, the sensing clients use low rate sampling to
`
`save energy and switch to a higher sampling rate upon detection of an interesting
`
`event (i.e., set of circumstances) thereby improving sampling resolution for periods of
`
`15
`
`interest while preserving power for periods ofless interest. Given the pricing
`
`schemes of MMS/SMS and the battery drain implied by 802.1 1 or cellular radio
`
`usage, data may be compressed before sending to presence server 1 16, using standard
`
`generic compression techniques on raw data and/or domain-specific run-length
`
`encoding (e. g., for a stand/walk/run classifier, only send updates to presence server
`
`20
`
`116 when the state changes) to reduce communication cost and power usage. When
`
`using SMS, the maximum message size may be used to minimize the price per bit.
`
`Furthermore, preliminary data analysis (e. g, filtering, inference) may be migrated to
`
`consumer devices 220. Given the computational power of many new cellular phones,
`
`significant processing may be done on these mobile devices to reduce communication
`
`25
`
`costs. However, aggregate (trans—users) analysis may be done within presence server
`
`116.
`
`1)
`
`Hardware Sensors:
`
`[0028]
`
`Cell phone 106 may represent one or more of a Nokia 5500 Sport,
`
`N80, N800 and N95 cell phones. PDA 108 may represent one or more of a Nokia
`
`30
`
`N800. Cell phone 106 and PDA 108 may be combined as in a phone/FDA hybrids
`
`such as an Apple iPhone. Embedded sensors 110 may represent one or more of a
`
`Page 8 of 36
`
`Page 8 of 36
`
`

`

`WO 2009/043020
`
`PCT/US2008/078148
`
`Nike+ system, a recreational sensor platform such as a Garmin Edge. a SkiScape. and
`
`a BikeN et,
`
`[0029]
`
`The SkiScape platform is, for example, used at a ski area to sense
`
`information on skiers and/or the ski area environment. Sensing devices may be
`
`attached to skiers, and fixed nodes communicate with the skiers‘ sensing devices.
`
`The fixed nodes may also capture data such as images, sound, and radar information
`
`including snow depth. Additional information on SkiScape may be found in the
`
`following paper which is incorporated herein by reference: Eisenman et al., SkiScape
`
`Sensing (Poster Abstract), in Proc. of Fourth ACM Conf. on Embedded Network
`
`10
`
`Sensor Systems, (SenSys 2006), Boulder, Nov. 2006.
`
`[0030]
`
`The BikeNet system includes a mobile networked sensing system
`
`for bicycles. Sensors collect cyclist and environmental data along a cycling route.
`
`Application tasking and sensed data uploading occur when sensors come within radio
`
`range of a static sensor access point or via a mobile sensor access point along the
`
`15
`
`cycling route. Additional information on BikeNet may be found in the following
`
`paper which is incorporated herein by reference: Eisenman et al., The BikeNet Mobile
`
`Sensing System/0r Cyclist Experience Mapping, in Proc. of Fifth AC M Conf. on
`
`Embedded Networked Sensor Systems, (SenSys 2007) Sydney, Australia, Nov. 2007.
`
`[003]]
`
`Another example of embedded sensor 1 10 is a BlueTooth enabled
`
`20
`
`3D accelerometer, sometimes refen‘ed to as a BlueCel. The BlucCel may be attached
`
`to a user or to another entity, such as a bicycle. Notebook computer 112 may
`
`represent one or more of Toshiba and Hewlett Packard laptop and/or desktop
`
`computers. Through a survey of the commonly available commercial hardware,
`
`including the examples listed above, the following hardware sensors are currently
`
`25
`
`available on one or more commercial—of-the—shelf(COTS) devices: embedded
`
`cameras, laptop/desktop web cameras, microphone, accelerometer, GPS, radio (e.g.,
`
`Bluetooth device contact logs, 80215.4 ranging, 802.1 1 localization), temperature
`
`sensors, light sensors, humidity sensors, magnetometers, button click sensors, and
`
`device state sensors (e. g., ringer off detectors). System 100 may exploit the
`
`availability of these sensors.
`
`Page 9 of 36
`
`Page 9 of 36
`
`

`

`WO 2009/043020
`
`PCT/US2008/078148
`
`2)
`
`Virtual Software Sensors:
`
`[0032]
`
`Software sensors are for example those that measure artifacts of
`
`other software that runs on the computing platform in an effort to understand the
`
`context ofthe user’s behavior, mood, etc. They are "virtual” in that they do not sense
`
`physical phenomena, but rather sense electronic evidence (“breadcrumbs") left as the
`
`user goes about his daily routine. Examples of virtual software sensors include the
`
`following: a trace of recent/current URLs loaded by the web browser; a trace of recent
`
`songs played on the music player to infer mood or activity; and mobile phone call log
`
`mining for structure beyond what a cell phone bill commonly provides.
`
`10
`
`[0033]
`
`As an example of how hardware and software sensor samples are
`
`combined to infer activity or status, if recent web searches show access to a movie
`
`review site (e. g., moviefonecom), and if a call was recently made to a particular
`
`friend, and if the time of day and the day of week is consistent with movie theatre
`
`times, and if the cell phone ringer is turned off, it is highly probably that the user is at
`
`15
`
`a movie theatre.
`
`B.
`
`Sensor Calibration
`
`[0034]
`
`Sensor calibration is a fundamental problem in all types of sensor
`
`networks. Without proper calibration, sensor-enabled mobile devices (e. g., cell
`
`phones, PDAs and embedded sensor platforms, etc.) produce data that may not be
`
`useful or may even be misleading. There are many possible sources of error
`
`introduced into sensed data, including those caused by the device hardware itself.
`
`This hardware error can be broken down into error caused by irregularities of the
`
`physical sensor itself due to its manufacturing, sensor drift (sensors characteristics
`
`change because of age or damage), and errors resulting from integration of the
`
`physical sensor with the consumer device (cg, cell phone 106). Physical sensor
`
`irregularities are compensated for at the factory, where a set of known stimuli is
`
`applied to the sensor to produce a map of the sensor output. To correct the device
`
`integration error, post—factory calibration of the sensor—enabled mobile device is
`
`required.
`
`[0035]
`
`One example of a calibration protocol that may be used in system
`
`100 is CaliBree. CaliBree is a distributed, scalable, and lightweight protocol that may
`
`20
`
`25
`
`30
`
`Page 10 of 36
`
`Page 10 of 36
`
`

`

`WO 2009/043020
`
`PCT/US2008/078148
`
`be used to automatically calibrate mobile sensor devices.
`
`It is assumed that. in mobile
`
`people-centric sensor networks, there will be two classes ofsensors with respect to
`
`calibration: calibrated nodes that are either static or mobile; and un-calibrated mobile
`
`nodes.
`
`In the following discussion, the nodes belonging to the former class are
`
`referred to as ground truth nodes. These ground truth nodes may exist as a result of
`
`factory calibration, or end user manual calibration.
`
`ln CaliBree, un-calibrated nodes
`
`opportunistically interact with calibrated nodes to solve a discrete average consensus
`
`problem, leveraging cooperative control over their sensor readings. The average
`
`consensus algorithm measures the disagreement of sensor samples between the un—
`
`10
`
`calibrated node and a series of calibrated neighbors. The algorithm eventually
`
`converges to a consensus among the nodes and leads to the discovery of the actual
`
`disagreement between the un-calibrated node sensor and calibrated nodes sensors.
`
`The disagreement is used by the un—calibrated node to generate (using a best fit line
`
`algorithm) the calibration curve of the newly calibrated sensor. The calibration curve
`
`15
`
`is then used to locally adjust the newly calibrated nodes sensor readings.
`
`[0036]
`
`in order for the consensus algorithm to succeed, the un-calibrated
`
`sensor devices compare data when sensing the same environment as the ground truth
`
`nodes. Given the limited amount of time mobile nodes may experience the same
`
`sensing environment during a particular rendezvous, and the fact that even close
`
`20
`
`proximity does not guarantee that the un—calibrated sensor and the ground truth sensor
`
`experience the same environment, the consensus algorithm is run over time when un—
`
`calibrated nodcs encounter different ground truth nodes. Additional information on
`
`CaliBree may be found in the following technical report which is incorporated herein
`
`by reference: E. Miluzzo et al., Calibree #A Self Calibration Experimental System/0r
`
`25
`
`Wireless Sensor Networks, in 5067/2008 Lecture Notes in Computer Science 314
`
`(Springer Berlin/Heidelberg, 2008).
`
`C.
`
`Sensor Sharing
`
`[0037]
`
`In system 100, mobile sensing devices (e. g., cell phone 106, PDA
`
`108, embedded sensors 110) locally manage sensing requests both on behalf of the
`
`30
`
`sensing clients (e. g., thin sensing clients) running on the mobile device and/or
`
`services running remotely on presence server 116. These sensing requests may be
`
`-10-
`
`Page 11 of 36
`
`Page 11 of 36
`
`

`

`WO 2009/043020
`
`PCT/US2008/078148
`
`multidimensional. with each requested sensor sample comprising a primary sensor
`
`type and a metadata or context vector. Context is the set of circumstances or
`
`conditions that surround a particular sensor sample. This context vector included in a
`
`request may indicate a location or time at which a sample is to be taken, and may
`
`include more sophisticated context tags such as sensor orientation (e. g, facing north,
`
`mounted on hip), and inferred custodian activity (e. g, walking, running, sitting). This
`
`context may also include weights that indicate the relative importance of one or more
`
`context elements.
`
`[0038]
`
`Whether a given request is injected into the system locally by a
`
`10
`
`sensing client running on a mobile sensor node, or by presence server 116, difficulties
`
`may arise in tasking an appropriate mobile sensor node with that given request.
`
`Considered strictly, a mobile sensor node is only appropriate to handle a given request
`
`ifit is equipped with the proper sensor, and if it has a context vector that matches that
`
`ofthe request. Strict handling ofa request may lead to excessive delay and/or a loss
`
`15
`
`of data fidelity in successfully servicing the request due to three connected issues: the
`
`uncontrolled mobility of mobile sensing device users; the location (or effective reach)
`
`ofthe request injection point with respect to the sensing target location; and the
`
`mismatch between the equipped sensors and context of available mobile sensor nodes
`
`and the sensors and context required by a request. Sensor sharing addresses this last
`
`20
`
`difficulty.
`
`[0039]
`
`Within system 100, not every consumer device 220 is equipped
`
`with every sensor type. Node heterogeneity arises due to a number of reasons,
`
`including cost (some sensors are more expensive, or rare), interest (some sensors are
`
`of more importance in different interest groups, or to individual users), and hardware
`
`25
`
`evolution (hardware evolves and many platform generations will be in use
`
`simultaneously, e. g. many newer mobile phone models are equipped with a camera
`
`and/or accelerometer but many older/cheaper models are not). Due to this
`
`heterogeneity, mismatches between the sensors required by sensing client requests
`
`and the sensors with which a given mobile node is equipped are likely to arise.
`
`30
`
`[0040]
`
`Sensor equipment and context constraints that a mobile sensing
`
`device must meet to be tasked for a given application request are loosened by
`
`allowing sensor sharing between mobile devices (e.g., a buddy’s device). At a high
`
`-11-
`
`Page 12 of 36
`
`Page 12 of 36
`
`

`

`WO 2009/043020
`
`PCT/US2008/078148
`
`level. in—situ sensor sharing is a function that allows an under-qualified node to
`
`borrow sensor data from a qualified node in an effort to satisfy a sensing request
`
`while maintaining or improving on the data fidelity, and in a more timely manner.
`
`[0041]
`
`In an example scenario, a student walks from his dorm to school to
`
`attend a mid-day class. The student carries a mobile phone equipped with a suite of
`
`simple sensors (e. g, accelerometer, temperature, microphone, camera), a GPS (e. g,
`
`Nokia N95), and a low power radio such as Bluetooth or IEEE 80215.4, in his
`
`pocket. As he leaves the dorm building, his phone is queried via the cellular data
`
`10
`
`15
`
`channel to measure outdoor light intensity based upon a request made by the student’s
`
`mother accessing a portal page of system 100. Recognizing that the phone is in the
`
`pocket using data inferencing, as the student walks along the sensor sharing service on
`
`the phone periodically broadcasts to discover a node that is out of the pocket (e. g, a
`
`hand-held or hip-mounted device) that can answer the light sensing request. If such a
`
`node responds and shares its light sensor reading, the outdoor light intensity query
`
`may be answered. Further along, the student receives an Ozone Alert text message on
`
`his phone. Curious about the ozone level in his vicinity, he launches an applet on his
`
`phone that is programmed to request this information from presence server 1 16.
`
`However, since the phone is not equipped with an ozone sensor, a request for the local
`
`ozone information is broadcast by the sensor sharing service. A bus mounted
`
`20
`
`platform (e. g., UMass DieselNet) with an attached ozone sensor replies, sharing the
`
`ozone reading which is displayed locally on the cell phone screen.
`
`[0042]
`
`The sensor sharing primitive comprises a distributed sharing
`
`decision algorithm running on the mobile user devices (e. g., cell phone), resource and
`
`context discovery protocols, in-situ sensor sharing protocols and algorithms that adapt
`
`25
`
`according to the radio and sensed environment, and a context analysis engine that
`
`provides the basis for sharing decision making.
`
`D.
`
`Analysis
`
`[0043]
`
`Sensed data is sent from device clients in consumer devices 220 to
`
`presence server 116 and is processed by one or more analysis component (eg,
`
`30
`
`inference engine 212) resident within presence server 116. This analysis component
`
`may combine historical per-user information with inferences derived from
`
`-12-
`
`Page 13 of 36
`
`Page 13 of 36
`
`

`

`WO 2009/043020
`
`PCT/US2008/078148
`
`combinations of current data derived from multiple sensors to determine the presence
`
`status of the user. This presence status may include objective items such as location
`
`and activity, and subjective items such as mood and preference. While a number of
`
`data fusion, aggregation, and data processing methods are possible, the following are
`
`examples of analysis/inference outputs that are used to generate the sensed presence
`
`within system 100.
`
`[0044]
`
`Location is a key function in any sensing system for providing
`
`geographical context to raw sensor readings. When explicit localization services like
`
`GPS are not available, due to hardware limitation or issues with satellite coverage,
`
`10
`
`15
`
`location of the client devices may be inferred from observed WiFi (e.g., access point
`
`identifiers), Skyhook service, Bluetooth (e. g., identifying location from static devices)
`
`cellular base station neighborhoods, and/or other unique sets of sensed data in a
`
`manner similar to ambient beacon localization. Ambient beacon localization may be
`
`used to determine a sensor’s location by use of supervised learning algorithms that
`
`allow the sensor to recognize physical locations that are sufficiently distinguishable in
`
`terms of sensed data from the other sensors in a field. Additional information on
`
`ambient beacon localization may be found in the following paper which is incorporate
`
`herein by reference: Nicholas D. Lane et al., Ambient Beacon Localization: Using
`
`Sensed Characteristics oft/2e Physical World to Localize Mobile Sensors, in 2007
`
`20
`
`Proceedings of the 41h Workshop on Embedded Networked Sensors 38 (2007).
`
`[0045]
`
`Human activity-inferring algorithms are incorporated within
`
`inference engine 212 to log and predict a users’ behavior. A simple classifier to
`
`determine whether a user is stationary or mobile may be built from several different
`
`data inputs, alone or in combination (e.g., changes in location by any possible means,
`
`accelerometer data). Accelerometer data may be analyzed to identify a number of
`
`physical activities, including sitting, standing, using a mobile phone, walking,
`
`running, stair climbing, and others.
`
`[0046]
`
`Human behavior is often a product of the environment. To better
`
`understand a user’s behavior, it is useful to quantify the user’s environment. Image
`
`and sound data may be collected and analyzed to derive the noisiness/brightness of
`
`the environment. Conversation detection and voice detection algorithms may be used
`
`25
`
`30
`
`-13-
`
`Page 14 of 36
`
`Page 14 of 36
`
`

`

`WO 2009/043020
`
`PCT/US2008/078148
`
`to identify the people in a user‘s Vicinity that may impact behavior and mood ofthe
`
`user.
`
`[0047]
`
`Part of a person's daily experience is the environment where the
`
`person lives and spends most of the time. For example, health related issues of
`
`5
`
`interest may include the level of an individual‘s exposure to particulates (e.g., pollen)
`
`and pollution. By incorporating mechanisms that enable air quality monitoring
`
`around the individual through opportunistic interaction with mobile sensors and/or
`
`static pre—deployed infrastructure (e. g., sensors 1 14), these health related issues of
`
`interest may be predicted and possible prevented.
`
`10
`
`E.
`
`Pres