Multi-Sensor Context-Awareness in Mobile Devices and Smart Artefacts
`
`Hans-W. Gellersen1, Albrecht Schmidt1 and Michael Beigl2
`1Department of Computing, Lancaster University, Lancaster LA1 4YR, UK
`2TecO, University of Karlsruhe, Vinc.-Priessnitz-Str. 1, 76131 Karlsruhe, GERMANY
`
`
`
`
`
`
`Abstract
`The use of context in mobile devices is receiving increasing attention in mobile and ubiquitous computing
`research. In this article we consider how to augment mobile devices with awareness of their environment and
`situation as context. Most work to date has been based on integration of generic sensor, in particular for
`positioning and for vision. We propose the integration of diverse simple sensors as alternative, aimed at
`awareness of situational context that can not be inferred from location, and targeted at resource constraint device
`platforms that typically do not permit processing of visual context. We have investigated multi-sensor context-
`awareness in a series of project and report experience from development of a number of device prototypes.
`These include development of an awareness module used for augmentation of a mobile phone, of the Mediacup
`exemplifying context-enabled everyday artefacts, and of the Smart-Its platform for aware mobile devices. The
`prototypes have been explored in a range of applications, to validate the multi-sensor approach to awareness, but
`moreover to develop new perspectives of how embedded context-awareness can be applied in mobile and
`ubiquitous computing.
`
`
`1
`
`Google Inc., Nest Labs, Inc., and Dropcam, Inc.
`GOOG 1009
`IPR of US Pat. No. 8,315,618
`
`

`

`1. Introduction
`Mobile computing technology has facilitated the use of computer-based devices in different and changing
`settings. In this article we consider augmentation of mobile devices with awareness of their environment and
`situation as context. More specifically we look at sensor integration in mobile devices, at abstraction of sensor
`data to more general characterizations of device situation, and at use of such context to improve user interaction
`and to support new types of application. Our work is grounded in ubiquitous computing research, and the
`devices we consider are mobile and computer-based in the widest sense. Specifically, they include smart devices
`and artefacts in which computing is secondary to a primary purpose, for instance mobile phones and everyday
`objects augmented with embedded computing.
`Context is what surrounds, and in mobile and ubiquitous computing the term is primarily used in reference to
`the physical world that surrounds the use of a mobile device. This has also been referred to as physical context,
`to distinguish it from other kinds of context in mobile computing, such as conditions in the surrounding systems
`and network infrastructure [3,7]. Other suggested terminology includes situational context, stressing the aim to
`reflect the situated nature of mobile device use beyond the capture of physical conditions [how-to-build, more-
`than-location]. Some confusion occasionally arises from the use of the term context at different levels of
`abstraction to denote (i) the real world situation surrounding a device, (ii) an aspect of a situation, such as
`location, and (iii) a specific occurrence of an aspect, such as a specific place. In our work we distinguish
`situation as what is given in the real world from context as what we can acquire through sensors and processing
`of sensor data to capture aspects of the surrounding world. We also use the term situational context for multi-
`faceted characterizations of a situation that typically require substantial analysis and fusion of data from
`individual sensors. Examples for situational context are being in a meeting, driving in a car, and user activities
`such as sleeping, watching TV, cycling, and so on.
` The use of context in mobile devices is receiving considerable attention in various fields of research including
`mobile computing, wearable computing, augmented reality, ubiquitous computing and human-computer
`interaction. A general motivation is that context-awareness can serve to compensate for the abstraction that is
`required in the first place to make systems accessible in changing environments and situations. The actual utility
`of context-awareness in mobile systems has been demonstrated in a wide range of application examples, in
`obvious domains such as fieldwork [17,19] and tourism [3,6,18], as well as in emerging areas like affective
`computing based on bio-sensing [15,20]. Also, it has been shown, that context is useful at different levels within
`a mobile device. At systems level, it can be exploited for example for context-sensitive resource and power
`management. At application level, context-awareness enables both adaptive applications and explicitly context-
`based services. And at the user interface level, the use of context facilitates a shift from explicit to implicit
`human-computer interaction, toward less visible if not invisible user interfaces [27,35].
`The focus of this article is on how to facilitate context-awareness in mobile devices. Context as we consider it
`here originates in the physical surroundings of a device, is captured through sensors, and finally results from
`analysis of sensor data. Devices may have direct or indirect awareness of context. In the case of indirect
`awareness, the entire sensing and processing occurs in the infrastructure while the mobile device obtains its
`context by means of communication. In contrast, a device has direct awareness if it is able to obtain context
`autonomously, (more or less) independent of any infrastructure. We will not further consider indirect awareness
`in this article, as the reliance on sensing infrastructure limits mobile device use to specifically equipped smart
`environments. Another problem of indirect awareness is the dislocation of context acquisition from context use.
`For instance, if a device embodies adaptation of its behavior to its temperature, then it is problematic to rely on
`temperature readings received from the environment.
`Mobile devices with direct awareness by definition embody one or multiple sensors, and models or algorithms
`for computation of more abstract context from raw sensor data. Most research into context-aware mobile
`devices has considered the use of single but powerful sensors, specifically position sensors and cameras.
`Position sensors provide access to location as rather specific but particularly useful context. Often it is not the
`position itself that constitutes immediately useful context but additional information that can be inferred, for
`instance resources registered at a location. Cameras similarly provide access to potentially rich information that
`
`2
`
`

`

`can be derived by means of feature extraction and video analysis. While positional context and visual context
`are powerful for augmentation of devices with some awareness of their situation, they both also have distinct
`shortcomings. Position is a static description of an environment and does not capture dynamic aspects of a
`situation. Its usefulness as context also largely depends on pre-captured knowledge about locations. Vision on
`the other hand can be employed to capture activity and other dynamic aspects, but extraction of specific context
`is computationally expensive and problematic in mobile and uncontrolled environments.
`In this article we discuss multi-sensor context-awareness as an alternative approach toward aware mobile
`devices. In this approach, the single powerful sensor is replaced by a collection of diverse simple sensors, and
`context is derived from multi-sensor data. We first introduced this approach in the European project TEA on
`mobile situation awareness, in search of comparatively cheap technology both regarding processing
`requirements and component cost [25]. In TEA and in follow-up projects, we have focused on the integration of
`deliberately simple and cheap sensors, and on comparatively inexpensive processing of multi-sensor data to
`obtain context, to meet the constraints typically found in mobile and embedded devices. In this series of
`projects, we have investigated the approach in different settings, to develop an understanding of how multi-
`sensor context-awareness can be applied in mobile and ubiquitous computing. A variety of device prototypes
`have emerged from this research, and we will discuss three of these in this article: the TEA awareness module
`used for augmentation of mobile phones; the Mediacup, a coffee cup with embedded awareness of its own state;
`and the Smart-Its device for augmentation of everyday artefacts. The investigation of these prototypes provides
`new perspectives of aware mobile devices, and insights into the kind of context that can be obtained from multi-
`sensor embedding. It further contributes an exploration of architectures for embedded awareness, and a rich
`design experience in trading off context-awareness against energy and memory constraints on microcontroller
`platforms.
`Following this introduction to our work we will reflect on related research in section 2. In doing so, we will
`keep the focus on mobile devices with direct awareness and look further into both positional and visual context-
`awareness. In addition we will discuss related work on use of other sensors and types of context in mobile
`devices, and in particular other research on integration of multiple and diverse sensors. In section 3, we will
`return to the discussion of multi-sensor context-awareness, and provide an overview of our program of research
`on this concept. This will be followed by a section each on a series of projects, starting with the TEA experience
`in section 4, followed by section 5 on the Mediacup development, and section 6 on the design of the Smart-Its
`device. In the final section we will summarize and discuss our overall experience regarding the augmentation of
`mobile devices with context-awareness, and point out issues and directions for further research.
`
`2. Background and Related Work
`In distributed computing, the interest in context has emerged in parallel with mobility support for users and
`devices. Groundbreaking contributions have been the Active Badge system as it integrated a positioning
`technique with a distributed computing infrastructure [14,33], and the Ubiquitous Computing vision as it
`pointed out the importance of location and context for the next era of computing [35]. These contributions were
`also underlying the ParcTab experiment, an early investigation into context-aware computing in which palm-
`size personal devices were augmented with locality for mobile access to location-based services [34]. One of the
`outcomes of the ParcTab work was a taxonomy of context-aware applications that has inspired much further
`work [24]. However, these pioneering projects all employed indirect awareness, with sensors located in the
`infrastructure, listening for beacons sent out from the mobile devices. In the case of the ParcTab, location
`information is actually a by-product of the cell-based communication infrastructure in which the base stations
`double as device locators. Many other context-aware mobile systems have likewise used cell-of-origin as
`context for location-based services, for example the GUIDE system deployed in Lancaster on the basis of a
`wireless LAN [6].
`More recent work has increasingly considered embedding of direct awareness in mobile devices. This has been
`boosted by rapid advances in sensor technologies, such as piezo-materials, VLSI video, optical gyros and
`MEMS (Micro Electro-Mechanical Systems) [22]. The trend is leading to sensors in ever smaller packages that
`increasingly integration feature extraction in the sensor hardware, enabling powerful sensing at very low cost. In
`
`3
`
`

`

`the following sections we will look now at related research on integration of sensors and context-awareness in
`mobile devices.
`
`2.1. Position Sensing and Location-based Context
`Location as position or area in space is a context of particular importance, and has received more attention in
`mobile computing than any other type of context. Like time, spatial location is an inherent attribute of other
`types of physical context, and often used implicitly for filtering nearby observations as relevant context from
`remote and hence less relevant observations [11]. Location is a well understood context for which powerful
`models are available to support querying and processing in a variety of ways. For example, geometric models
`support location representations on which simple arithmetic can be applied, while symbolic models support the
`use of set theoretical expressions, such as being contained in. However, it is typically not a location as such that
`is of interest in location-aware mobile computing. Instead, “location becomes a useful indexing device” from
`which to infer further context [7, p.292].
`Most available position sensing technologies are based on sensor infrastructure rather than sensor integration
`into the mobile device. This is in particular the case for indoor positioning systems, which are typically based on
`small radio or infrared cells, or on sensor arrays in the environment for higher accuracy. However, alternatives
`have been proposed, for example indoor positioning based on visual context (see below). For outdoor
`positioning, the situation is different. In the GPS system (Global Positioning System) it is the infrastructure that
`sends out signals while the sensor is located in the client device. GPS sensors have become available in very
`small package sizes enabling their integration in mobile devices. A variety of projects have shown how
`integration of GPS with mobile computing devices can support new kinds of application. For example, in the
`stick-e-notes system GPS is used in conjunction with palm computers to support context-based access to shared
`notes in fieldwork [2,19].
`In contrast to approaches based on positioning technology, one of our explicit objectives is to achieve awareness
`of context that can not be inferred from location. With the use of diverse sensors we further aim at more direct
`acquisition of multi-faceted information from the local environment of a device.
`
`2.2. Auditory and Visual Context
`Computer vision is by its very definition concerned with the capture of aspects of the physical world for use in
`computers. The field is rooted in artificial intelligence and the tradition of building computer models of human
`capabilities. However, with decreasing cost and size of camera modules and emergence of wearable computers,
`computer vision is now considered as technology option for augmentation of devices with context. Visual
`context typically constitutes a rich resource from which more specific context can be derived by means of video
`analysis and feature extraction. Visual Augmented Memory is for example a wearable system that embodies a
`camera and face recognition software to obtain highly specific visual context [9]. In other wearable systems,
`vision has been used to determine location and to support navigation tasks [31].
`More closely related to our work though is that of Clarkson et al. who investigated the recognition of user
`situations from a wearable camera and microphone [5], respectively from only a microphone [4]. The
`commonality with our work is in the focus on situation rather than location. However while Clarkson is
`concerned with recognition technology only, we are interested in the wider picture of how awareness technology
`can be integrated in mobile devices, and in particular in less powerful devices than wearable high-end
`computers.
`
`2.3. Integration of Other Sensors in Mobile Devices
`Besides technologies for positioning and vision, a range of other sensors have been investigated for
`augmentation of mobile devices. In many cases this is aimed to facilitate capture of very specific context as
`input to likewise specific applications. An example is the StartleCam application, based on a wearable computer
`augmented with bio-sensors for recognition of extreme user situations [15]. Another perspective on sensor
`
`4
`
`

`

`integration in mobile interactive devices is to extend the user interface with new interaction techniques. For
`example, Rekimoto added tilt sensors to a palm computer to enable one-handed operations [21]. In a similar
`way, we have integrated sensors in a handheld computer to automatically adapt display orientation
`(landscape/portrait) to device orientation [26]. In this line of work, sensors are used for new user interface
`techniques, whereas in this paper we focus on sensor integration for awareness of device situation.
`
`2.4. Integration of Diverse Sensors
`An important aspect of our approach is the integration of different kinds of sensor. This aspect has also been
`investigated in the Smart Badge, a device inspired by the Active Badge for positioning but in addition equipped
`with a large variety of sensors to capture additional context [30]. The Smart Badge provides integrated access to
`diverse types of sensor data, with the research focus on device architecture rather than sensor fusion and
`inference of generic situational context. A similar device development is the Sensor Badge, however only
`integrating a movement sensor to infer information about the user’s activity [8]. The Sensor Badge particularly
`relates to our early work on the TEA awareness module, as both were conceived as peripheral components to
`provide other mobile devices with context. The integration of diverse sensors has also been considered for an
`indoor location technique that does not require any infrastructure [13]. The research prototype integrated sensors
`for motion, orientation, light, and temperature, and relates to our own work with its emphasis on small,
`lightweight, low-power and cheap components.
`
`3. Multi-Sensor based Context-Awareness
`The combination of comparatively simple sensors is an interesting alternative to the use of single powerful
`sensors as in position- and vision-based systems. The combination of multiple diverse sensors that individually
`capture just a small aspect of an environment may result in a total picture that better characterizes a situation
`than location- or vision-based context. The rationale for our approach is to advance beyond location-based
`systems to achieve awareness of context that can not be inferred from location, for which we see the diversity of
`sensors as key. Table 1 lists a few examples of how situations and data from different sensors may relate.
`
`
`Table 1. Real world situations related to sensor data (adapted from [29]).
`
`Situation
`User sleeps
`
`User is watching TV
`
`User is cycling
`
`Sensor Data
`It is dark, room temperature, silent, type of location is indoors, time is “night-
`time”, user is horizontal, specific motion pattern, absolute position is stable
`Light
`level/color
`is changing, certain audio
`level (not silent), room
`temperature, type of location is indoors, user is mainly stationary
`Location type is outdoors, user is sitting, specific motion pattern of legs,
`absolute position is changing.
`
`
`Our work is further motivated by the objective to devise awareness technology that can be embedded in
`resource constraint mobile devices, and even in simple everyday artefacts. Through the use of multiple diverse
`sensors we expect to gain access to rich data from which useful context can be inferred with comparatively little
`computation. As there are multiple sensors, they each only have to contribute a part to the whole picture which
`means that pre-processing of sensor data will be more focused than for example in vision, where substantially
`more processing is required to get the big picture from just a single data source. Figure 1 illustrates the
`difference between use of one generic sensor and use of multiple simple sensors.
`
`5
`
`

`

`
`
`Figure 1. Use of a single generic sensor vs. multiple specific sensors for context-awareness (adapted from
`Van Laerhoven et al [32])
`
`Over the last years, we have investigated our approach in a series of projects to develop an understanding of its
`utility for mobile and ubiquitous computing. Our research approach is to build fully integrated sensor-enabled
`device prototypes and to deploy them in mobile environments and where possible in everyday use. This stresses
`our interest in advancing both the understanding of how mobile devices can be augmented with awareness, and
`how aware mobile devices can be applied in the real world. This approach, characteristic for ubiquitous
`computing research, is aimed at collection of overall design and use experience, with lesser attention to aspects
`such as optimization of specific sensor fusion and context recognition techniques.
`The projects that we will discuss in the remainder of the article cover the development of the TEA awareness
`module, the Mediacup experience, and our current work on the Smart-Its platform. Each of these projects
`explores different perspectives of how multi-sensor context-awareness may be applied in conjunction with
`mobile devices and artifacts:
`(cid:120) TEA – Technology Enabling Awareness: in this initial project we investigated the realization of multi-
`sensor context-awareness in a self-contained device that would be available as peripheral or plug-in for
`mobile host devices. The general application perspective was to supply situational context to a mobile host
`device to improve the device’s service to its user. We explored this perspective in the application domain of
`mobile telephony.
`(cid:120) Mediacup: in this project we looked at how non-computational artifacts can be augmented with awareness
`technology. The application perspective is entirely different from that underlying TEA. Awareness is not
`employed to improve the immediate function of the augmented device, but to create a digital presence for
`it. New functionality is not expected to emerge in the device itself but in the surrounding system
`environment.
`(cid:120) Smart-Its: this recently started project is aimed at moving beyond the study of design examples such as
`Mediacup, toward platforms for aware mobile devices. With this project we also shift our attention from
`individual devices to ad hoc networking of aware devices, and scenarios of collective awareness.
`A general question underlying the research in these projects is what kind of situational context we can obtain
`from multi-sensor context-awareness. Closely related is the question of how data from multiple sensors can be
`related effectively to situational context. In the following three sections we will report findings from the
`individual projects, to be followed by a discussion in which we will summarize what we have learned so far,
`drawing some conclusion for further investigation of context-awareness in mobile devices.
`
`6
`
`

`

`
`
`Figure 2. TEA is based on a layered architecture for abstraction from raw sensor data to multi-sensor
`based context.
`
`
`
`4. The TEA Context-Awareness Module
`The general motivation underlying the TEA project is to make personal mobile devices smarter. The assumption
`is that the more a device knows about its user, its environment and the situations in which it is used the better it
`can provide user assistance. The objective of TEA is to arrive at a generic solution for making devices smarter,
`and the approach taken is to integrate awareness technology – both hardware and software – in a self-contained
`device conceived as plug-in for any personal appliance which from a TEA perspective is called host. The
`cornerstones of the TEA device concept are:
`(cid:120)
`Integration of diverse sensors, assembled for acquisition multi-sensor data independently of any particular
`application.
`(cid:120) Association of multi-sensor data with situations in which the host device is used, for instance being in a
`meeting.
`Implementation of hardware, i.e. sensors and processing environment, and software, i.e. methods for
`computing situational context from sensor data, in an embedded device.
`A specific objective of TEA is to address the kind of context that can not be derived from location information
`at all, for example situations that can occur anywhere. Another specific issue investigated in TEA is sensor
`fusion. The aim is to derive more context from a group of sensors than the sum of what can be derived from
`individual sensors.
`
`(cid:120)
`
`4.1. TEA architecture
`TEA is based on a layered architecture for sensor-based computation of context as illustrated in figure 2, with
`separate layers for raw sensor data, for features extracted from individual sensors (‘cues’), and for context
`derived from cues.
`The sensor layer is defined by an open-ended collection of sensors. The data supplied by sensors can be very
`different, ranging form slow sensors that supply scalars (e.g. temperature) to fast and complex sensors that
`provide larger volume data (e.g. microphone). Also the update time can vary largely from sensor to sensor.
`The cue layer introduces cues as abstraction from raw sensor data. Each cue is a feature extracted from the data
`stream of a single sensor, and many diverse cues can be derived from the same sensor. This abstraction from
`sensors to cues serves to reduce the data volume independent of any specific application, and is also referred to
`as ‘cooking the sensors’ [13]. Just as the architecture does not prescribe any specific set of sensors, it also does
`not prescribe specific methods for feature extraction in this layer. However, in accordance with the idea of
`shifting complexity from algorithms to architecture it is assumed that cue calculation will be based on
`comparatively simple methods. The calculation of cues from sensor values may for instance be based on simple
`
`7
`
`

`

`statistics over time (e.g. average over the last second, standard deviation of the signal, quartile distance, etc.) or
`slightly more complex mappings and algorithms (e.g. calculation of the main frequencies from a audio signal
`over the last second, pattern of movement based on acceleration values).
`The cue layer hides the sensor interfaces from the context layer it serves, and instead provides a smaller and
`uniform interface defined as set of cues describing the sensed system environment. This way, the cue layer
`strictly separates the sensor layer and context layer which means context can be modeled in abstraction from
`sensor technologies and properties of specific sensors. Separation of sensors and cues also means that both
`sensors and feature extraction methods can be developed and replaced independently of each other. This is an
`important requirement in context-aware systems and has motivated the development of architectures such as the
`Context Toolkit [23].
`The context layer introduces a set of contexts which are abstractions of real world situations, each as function of
`available cues. It is only at this level of abstraction, after feature extraction and data reduction in the cue layer,
`that information from different sensors is combined for calculation of context. While cues are assumed to be
`generic, context is considered to be more closely related to the host device and the specific situations in which it
`is used. Again, the architecture does not prescribe the methods for calculating context from cues; rule-based
`algorithms, statistical methods and neural networks may for instance be used. Conceptually, context is
`calculated from all available cues. In a rule set however, cues known to be irrelevant may simply be neglected,
`and in neural networks their weight would be reduced accordingly. The mapping from cues to context may be
`explicit, for instance when certain cues are known to be relevant indicators of a specific context, or implicit in
`the result of a supervised or unsupervised learning technique.
`
`4.2. Initial exploration of the approach
`To study the TEA approach, we have developed two generations of prototype devices and used them for
`exploration of multi-sensor data, and for a validation of TEA as add-on device for mobile phones. In parallel to
`development of the first prototype we have also conducted scenario-based requirements analysis to investigate
`our assumption that there is useful context for personal mobile devices that can not be derived from location but
`from multi-sensor input. In this analysis, a range of scenarios were developed for both mobile phones and
`personal digital assistants (PDA), and it was found that the potential for context beyond location was higher in
`communication-related scenarios than in typical PDA applications which led us to focus further studies on the
`domain of mobile telephony.
`The first generation TEA module was developed for exploration of a wide range of sensors and their
`contribution to context-awareness. It contained common sensors such as microphone, light sensor and
`accelerometers but also sensors for example for air pressure, certain gas concentration and so on. With several
`implementations of the device, large amounts of raw sensor data were collected independently at different sites
`for further analysis of multi-sensor fusion following two strategies:
`(cid:120) Analysis of the contribution of a sensor or group of sensors to perception of a given context, i.e. a specific
`real-world situation: For this study a number of situations that we considered relevant for personal mobile
`devices were selected (e.g. user is walking, user is in a conversation, other people are around, user is
`driving a car, etc.) for data collection at three different sites. The data was subjected to statistical analysis to
`determine for each sensor or sensor group whether its inclusion increased the probability of recognizing
`situations.
`(cid:120) Analysis of clusters in collected multi-sensor data: Here the strategy was to carry the device over a longer
`period of time so it accompanies a user in different situations. Over the whole period of time, raw sensor
`data was recorded and to be later analyzed to identify clusters corresponding to situations that occurred
`during recording time, e.g. the user is sitting at their desk, walking over to a colleague, chatting, walking
`back, engaging in a phone conversation and so on. This process was aimed at identifying the sensors
`relevant to situations, and at development of clustering algorithms.
`
`8
`
`

`

`4.3. Implementation of TEA in a Self-Contained Awareness Device
`The initial exploration of sensors and their contribution to awareness of typical real-world situations served to
`inform development of the second generation device optimized for smaller packaging, and shown in figure 3.
`The device integrates two light sensors, two microphones, a two-axis accelerometer, a skin conductance sensor
`and a temperature sensor. The sensors are read by a micro-controller, that also calculates the cues and in some
`applications also the contexts. The system is designed to minimize the energy consumption of the component.
`The micro-controller (PIC16F877) has a number of analog and digital inputs and communicates via serial line
`with the host device. The calculation of cues and contexts is very much restricted due to the limitations of the
`micro-controller. Programs have to fit into 8K of EEProm, and have only about 200 Byte RAM available.
`The cue extraction algorithms have been designed to accommodate these limitations. Data that has to be read
`with high speed such as audio is directly analyzed and not stored. Typical cues for audio that are calculated on
`the fly are the number of zero crossings of the signal in a certain time (indicator of the frequency) and number
`of direction changes of the signal (together with the zero crossings this is indicative of the noise in the signal).
`For acceleration and light basic statistical methods and an estimation of the first derivative are calculated.
`Slowly changing values – temperature and skin conductance – are not further processed in the cue layer (the cue
`function is the identity). The contexts are calculated based on rules that were extracted off-line from data
`recorded with the sensor board in different situations.
`The prototype is independent of any specific host and at times has been used in conjunction with a palmtop
`computer, a wearable computer and mobile phones, connected via the serial interface. Primarily however the
`prototype is being applied in the area of mobile telephony.
`
`4.4. Application in mobile telephony
`State of the art mobile phones support so-ca