`
`
`DECLARATION OF JUNE ANN MUNFORD
`DECLARATION OF JUNE ANN MUNFORD
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`1
`
`APPLE 1017
`
`APPLE 1017
`
`1
`
`

`

`1. My name is June Ann Munford. I am over the age of 18, have personal
`
`knowledge of the facts set forth herein, and am competent to testify to the
`
`same.
`
`2. I earned a Master of Library and Information Science (MLIS) from the
`
`University of Wisconsin-Milwaukee in 2009. I have over ten years of
`
`experience in the library/information science field. Beginning in 2004, I
`
`have served in various positions in the public library sector including
`
`Assistant Librarian, Youth Services Librarian and Library Director. I have
`
`attached my Curriculum Vitae as Appendix CV.
`
`3. During my career in the library profession, I have been responsible for
`
`materials acquisition for multiple libraries. In that position, I have cataloged,
`
`purchased and processed incoming library works. That includes purchasing
`
`materials directly from vendors, recording publishing data from the material
`
`in question, creating detailed material records for library catalogs and
`
`physically preparing that material for circulation. In addition to my
`
`experience in acquisitions, I was also responsible for analyzing large
`
`collections of library materials, tailoring library records for optimal catalog
`
`2
`
`

`

`search performance and creating lending agreements between libraries
`
`during my time as a Library Director.
`
`4. I am familiar with the Internet Archive, a digital library formally certified by
`
`the State of California as a public library. Among other services that the
`
`Internet Archive makes available to the general public is the Wayback
`
`Machine, an online archive. The Internet Archive’s Wayback Machine
`
`service archives webpages as of a certain capture date to track changes in the
`
`web over time. The Internet Archive has been in operation as a nonprofit
`
`library since 1996 and has hosted the Wayback Machine service since its
`
`inception in 2001. During my time as a librarian, I frequently used the
`
`Internet Archive’s Wayback Machine for research and instruction purposes.
`
`This includes teaching instructional classes on using the Wayback Machine
`
`to library patrons and using the Wayback Machine to research reference
`
`inquiries that require hard-to-find online resources. I consider the Internet
`
`Archive’s record-keeping to be as rigorous and detailed as other formal
`
`library recordkeeping practices such as MARC records, OCLC records and
`
`Dublin Core.
`
`5. I have reviewed Exhibit APPLE-1007, a document entitled “Multimodal
`
`Sensing for Pediatric Obesity Applications” by M. Annavaram, N.
`
`3
`
`

`

`Medvidovic, U. Mitra, S. Narayanan, G. Sukhatme, Z. Meng, S. Qiu, R.
`
`Kumar, G. Thatte and D. Spruijt-Metz as presented at the UrbanSense08
`
`conference.
`
`6. Attached hereto as Appendix ANNAVARAM01 is a true and correct copy
`
`of “Multimodal Sensing for Pediatric Obesity Applications” within a PDF
`
`file entitled ”annavaram_urbansense08.pdf”. I secured this file myself from
`
`the UrbanSense08 conference website at
`
`http://sensorlab.cs.dartmouth.edu/urbansensing/papers/annavaram_urbansen
`
`se08.pdf. In comparing Appendix ANNAVARAM01 to Exhibit APPLE-
`
`1007, it is my determination that Exhibit APPLE-1007 is a true and correct
`
`copy of “Multimodal Sensing for Pediatric Obesity Applications”.
`
`7. Attached hereto as Appendix ANNAVARAM02 is a true and correct copy
`
`of a file tree captured from the UrbanSense08 website. I secured this
`
`appendix myself from
`
`http://sensorlab.cs.dartmouth.edu/urbansensing/papers/. The file tree
`
`captured within ANNAVARAM02 indicates that
`
`”annavaram_urbansense08.pdf” was first uploaded to this page as of
`
`October 27, 2008.
`
`4
`
`

`

`8. Attached hereto as Appendix ANNAVARAM03 is a screen capture of the
`
`Internet Archive Wayback Machine entry for
`
`http://sensorlab.cs.dartmouth.edu/urbansensing/papers/annavaram_urbansen
`
`se08.pdf. I secured these screen captures myself from
`
`http://web.archive.org/web/2010*/http://sensorlab.cs.dartmouth.edu/urbanse
`
`nsing/papers/annavaram_urbansense08.pdf. This Internet Archive record
`
`contains an archived copy of ”annavaram_urbansense08.pdf” dated to July
`
`1, 2010.
`
`9. Attached hereto as Appendix ANNAVARAM04 is a screen capture of the
`
`Internet Archive Wayback Machine entry for
`
`http://sensorlab.cs.dartmouth.edu/urbansensing/. I secured these screen
`
`captures myself from
`
`http://web.archive.org/web/20080201000000*/http://sensorlab.cs.dartmouth.
`
`edu/urbansensing/. This record is an archived copy of the call for papers for
`
`the UrbanSense08 conference captured as of May 30, 2008. This call for
`
`papers advertises the date of the conference as November 4, 2008.
`
`10. Appendix ANNAVARAM03 indicates that the conference organizers
`
`published a full PDF copy of as “Multimodal Sensing for Pediatric Obesity
`
`Applications” shortly before the conference in October 2008. Appendix
`
`5
`
`

`

`ANNAVARAM03, the Wayback Machine’s capture of this PDF, was first
`
`preserved by the Internet Archive as of July 1, 2010. Appendix
`
`ANNAVARM 04 indicates the paper was presented and circulated at the
`
`UrbanSense08 conference as of November 4, 2008. Considering this
`
`information, it is my determination that “Multimodal Sensing for Pediatric
`
`Obesity Applications” was first made available by the UrbanSense08
`
`conference organizers as of October 2008, broadly published and distributed
`
`in November 2008 via the conference and archived to ensure continuous
`
`access as of July 1, 2010.
`
`11. I have been retained on behalf of the Petitioner to provide assistance in the
`
`above-illustrated matter in establishing the authenticity and public
`
`availability of the documents discussed in this declaration. I am being
`
`compensated for my services in this matter at the rate of $100.00 per hour
`
`plus reasonable expenses. My statements are objective, and my
`
`compensation does not depend on the outcome of this matter.
`
`12. I declare under penalty of perjury that the foregoing is true and correct. I
`
`hereby declare that all statements made herein of my own knowledge are
`
`true and that all statements made on information and belief are believed to
`
`be true; and further that these statements were made the knowledge that
`
`6
`
`

`

`willful false statements and the like so made are punishable by fine or
`
`imprisonment, or both, under Section 1001 of Title 18 of the United States
`
`Code.
`
`Dated: 6/20/2022
`
`June Ann Munford
`
`7
`
`

`

`APPENDIX CV
`APPENDIX CV
`
`8
`
`

`

`J. Munford
`Curriculum Vitae
`
`Education
`
`University of Wisconsin-Milwaukee - MS, Library & Information Science, 2009
`Milwaukee, WI
`
`
`● Coursework included cataloging, metadata, data analysis, library systems,
`management strategies and collection development.
`● Specialized in library advocacy, cataloging and public administration.
`
`
`Grand Valley State University - BA, English Language & Literature, 2008
`Allendale, MI
`
` ●
`
` Coursework included linguistics, documentation and literary analysis.
`● Minor in political science with a focus in local-level economics and
`government.
`
`
`
`Professional Experience
`
`Researcher / Expert Witness, October 2017 – present
`Freelance ● Pittsburgh, Pennsylvania & Grand Rapids, Michigan
`
`
`● Material authentication and public accessibility determination.
`Declarations of authenticity and/or public accessibility provided upon
`research completion. Experienced with appeals and deposition process.
`
` ●
`
` Research provided on topics of public library operations, material
`publication history, digital database services and legacy web resources.
`
` ●
`
` Past clients include Alston & Bird, Arnold & Porter, Baker Botts, Fish &
`Richardson, Erise IP, Irell & Manella, O'Melveny & Myers, Perkins-Coie,
`Pillsbury Winthrop Shaw Pittman and Slayden Grubert Beard.
`
`Library Director, February 2013 - March 2015
`Dowagiac District Library ● Dowagiac, Michigan
`
`
`● Executive administrator of the Dowagiac District Library. Located in
`
`9
`
`

`

`Southwest Michigan, this library has a service area of 13,000, an annual
`operating budget of over $400,000 and total assets of approximately
`$1,300,000.
`
`● Developed careful budgeting guidelines to produce a 15% surplus during
`the 2013-2014 & 2014-2015 fiscal years while being audited.
`
`
`
` ●
`
` Using this budget surplus, oversaw significant library investments
`including the purchase of property for a future building site, demolition of
`existing buildings and building renovation projects on the current facility.
`
` Led the organization and digitization of the library's archival records.
`
` ●
`
` ●
`
` Served as the public representative for the library, developing business
`relationships with local school, museum and tribal government entities.
`
` ●
`
` Developed an objective-based analysis system for measuring library
`services - including a full collection analysis of the library's 50,000+
`circulating items and their records.
`
`November 2010 - January 2013
`Librarian & Branch Manager, Anchorage Public Library ● Anchorage, Alaska
`
`
`● Headed the 2013 Anchorage Reads community reading campaign
`including event planning, staging public performances and creating
`marketing materials for mass distribution.
`
` ●
`
` Co-led the social media department of the library's marketing team,
`drafting social media guidelines, creating original content and instituting
`long-term planning via content calendars.
`
` ●
`
` Developed business relationships with The Boys & Girls Club, Anchorage
`School District and the US Army to establish summer reading programs for
`children.
`
`
`June 2004 - September 2005, September 2006 - October 2013
`Library Assistant, Hart Area Public Library
`Hart, MI
`
`
`● Responsible for verifying imported MARC records and original MARC
`
`10
`
`

`

`
`
`cataloging for the local-level collection as well as the Michigan Electronic
`Library.
`
`● Handled OCLC Worldcat interlibrary loan requests & fulfillment via
`ongoing communication with lending libraries.
`
`
`
`Professional Involvement
`
`Alaska Library Association - Anchorage Chapter
`● Treasurer, 2012
`
`
`Library Of Michigan
`● Level VII Certification, 2008
`● Level II Certification, 2013
`
`
`Michigan Library Association Annual Conference 2014
`● New Directors Conference Panel Member
`
`
`Southwest Michigan Library Cooperative
`● Represented the Dowagiac District Library, 2013-2015
`
`
`
`Professional Development
`
`Library Of Michigan Beginning Workshop, May 2008
`Petoskey, MI
`● Received training in cataloging, local history, collection management,
`children’s literacy and reference service.
`
`
`Public Library Association Intensive Library Management Training, October 2011
`Nashville, TN
`● Attended a five-day workshop focused on strategic planning, staff
`management, statistical analysis, collections and cataloging theory.
`
`
`Alaska Library Association Annual Conference 2012 - Fairbanks, February 2012
`Fairbanks, AK
`● Attended seminars on EBSCO advanced search methods, budgeting,
`cataloging, database usage and marketing.
`
`11
`
`

`

`Depositions
`
`2019 ● Fish & Richardson
`
`IPR Petitions of 865 Patent, Apple v. Qualcomm (IPR2018-001281 /
`
`39521-00421IP & IPR2018-01282 / 39521-00421IP2)
`
`2019 ● Erise IP
`
`Implicit, LLC v. Netscout Systems, Inc (Civil Action No. 2:18-cv-53-JRG)
`
`2019 ● Perkins-Coie
`
`Adobe Inc. v. RAH Color Technologies LLC (Cases IPR2019-00627,
`
`IPR2019-00628, IPR2019-00629 and IPR2019-00646)
`
`2020 ● O’Melveny & Myers
`
`Maxell, Ltd. v. Apple Inc. (Case 5:19-cv-00036-RWS)
`
`2021 ● Pillsbury Winthrop Shaw Pittman LLP
`
`Intel v. SRC (Case IPR2020-1449)
`
`
`Limited Case History & Potential Conflicts
`
`Alston & Bird
`
`● Nokia (v. Neptune Subsea, Xtera)
`
`Arnold & Porter
`
`● Ivantis (v. Glaukos)
`
`Erise I.P.
`
`● Apple
`
`
`v. Future Link Systems (IPRs 6317804, 6622108, 6807505, and
`
`
`7917680)
`
`
`v. INVT
`
`
`v. Navblazer LLC (Case No. IPR2020-01253)
`
`12
`
`

`

`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`v. Qualcomm (IPR2018-001281, 39521-00421IP, IPR2018-01282,
`39521-00421IP2)
`v. Quest Nettech Corp, Wynn Technologies (Case No. IPR2019-
`00XXX, RE. Patent Re38137)
`
`● Fanduel (v CGT)
`
`● Garmin (v. Phillips North America LLC, Case No. 2:19-cv-6301-AB-KS
`Central District of California)
`
`● Netscout
`
`v. Longhorn HD LLC)
`
`v. Implicit, LLC (Civil Action No. 2:18-cv-53-JRG)
` ● Sony Interactive Entertainment LLC
`v. Bot M8 LLC
`v. Infernal Technology LLC
`● Unified Patents (v GE Video Compression, Civil Action No. 2:19-cv-248)
`
`
`Fish & Richardson
`
`● Apple
`
`
`v. LBS Innovations
`
`
`v. Masimo (IPR 50095-0012IP1, 50095-0012IP2, 50095-0013IP1,
`
`
`50095-0013IP2, 50095-0006IP1)
`
`
`v. Neonode
`
`
`v. Qualcomm (IPR2018-001281, 39521-00421IP, IPR2018-01282,
`
`
`39521-00421IP2)
`
`
`
`
`● Dish Network
`
`v. Realtime Adaptive Streaming, Case No 1:17-CV-02097-RBJ)
`
`13
`
`

`

`v. TQ Delta LLC
`
` Huawei (IPR 76933211)
`
` Kianxis
`
`
`
` ●
`
` ●
`
` ●
`
` LG Electronics (v. Bell Northern Research LLC, Case No. 3:18-cv-2864-
`CAB-BLM)
`
` ●
`
` ●
`
` Samsung (v. Bell Northern Research, Civil Action No. 2:19-cv-00286-
`JRG)
`
` Texas Instruments
`
` ●
`
`
`Irell & Manella
`
`● Curium
`
`O’Melveny & Myers
`
`● Apple (v. Maxell, Case 5:19-cv-00036-RWS)
`
`Perkins-Coie
`
`● TCL Industries (v. Koninklijke Philips NV, PTAB Case Nos. IPR2021-
`
`00495, IPR2021-00496, and IPR2021-00497)
`
`Pillsbury Winthrop Shaw Pittman
`
`● Intel (v. FG SRC LLC, Case No. 6:20-cv-00315 W.D. Tex)
`
` Metaswitch
`
` MLC Intellectual Property (v. MicronTech, Case No. 3:14-cv-03657-SI)
`
` Realtek Semiconductor
`
` Quectel
`
` ●
`
` ●
`
` ●
`
`14
`
`

`

`APPENDIX ANNAVARAM01
`APPENDIX ANNAVARAMOI
`
`15
`
`

`

`Multimodal Sensing for Pediatric Obesity
`Applications
`M. Annavaram†,N. Medvidovic†, U. Mitra†, S. Narayanan† G. Sukhatme†,
`Z. Meng‡, S. Qiu‡, R. Kumar†, G. Thatte†, D. Spruijt-Metz§
`† Viterbi School of Engineering, University of Southern California, Los Angeles, CA 90089, USA
`Email: {annavara,ubli,thatte}@usc.edu, {neno,gaurav}@cs.usc.edu, shri@sipi.usc.edu
`‡ Tsinghua University, Beijing China
`Email: {third,forth}@institution.edu
`§ Keck School of Medicine, University of Southern California, Los Angeles, CA 90089, USA
`Email: metz@usc.edu
`
`Abstract
`
`In this paper, a wireless body area network comprised of heterogeneous sensors is developed for wearable
`health monitoring applications. The ultimate application space is in the context of pediatric obesity. The specific
`task examined herein is activity detection based on heart rate monitor and accelerometer data. Based on statistical
`analysis of experimental data for different key states (lying down, sitting, standing, walking and running), a multi-
`modal detection strategy is proposed. The resulting detector can achieve 85-95% accuracy in state detection. It
`is observed that the accelerometer is more informative for the active states, while the heart rate monitor is more
`informative for the passive states.
`
`I. INTRODUCTION
`Wearable health monitoring systems coupled with wireless communications are the bedrock of an emerging
`class of sensor networks: wireless body area networks (WBAN). The objectives of such WBANs are manifold
`from diet monitoring [14], activity detection [3], [4], and health crisis support[6]. These new networks demand
`significant technological advances from sensor development to novel software engineering, signal processing,
`wireless communications and networking. Importantly, WBANs must be designed with application-specific design
`and end-use requirements in mind. These advancements are necessary to cope with the unique challenges introduced
`by deployment on people, such as: unpredictable mobility, heterogeneous sensor nodes, new wireless channels, very
`low power requirements, non-invasive sensing and the need for sensors with small footprints. Furthermore, drawing
`robust inference from sensor streams requires information from multiple, often disparate, sources. In the current
`work, we provide preliminary results from the construction of a WBAN which we will use to drive the development
`of assessments and interventions for pediatric obesity applications.
`Pediatric obesity has emerged as a major national and international health crisis. National collected data from
`2003-2006 show 11.3% of adolescents aged 12 - 19 years by some measures could be designated as obese; a further
`16% would be classified as overweight and 32% considered at risk for being overweight [13]. While physical activity
`(PA) is tightly related to lower obesity rates in children [11], [7], there are additional factors leading to obesity. The
`increasing environmental stress may promote both general obesity (through lifestyle behaviors such as decreased
`physical activity) and visceral obesity (through hypothalamic-pituitary-adrenal axis activation and increased cortisol
`secretion)[5]. Current monitoring systems validated for research in children typically monitor physical activity only
`(such as the much-used Actigraph accelerometer). However, in order to truly understand and reverse childhood
`obesity, we need a multimodal system that will track stress levels, PA levels, blood glucose levels and other vital
`signs simultaneously, as well as anchor these levels to context such as time of day and geographical location. Our
`preliminary KNOWME network is a first step towards such a system.
`A key aspect of our work is the unified design and evaluation of multimodal sensing and interpretation, for
`automatically recognizing, predicting and reasoning about human physical activity and socio-cognitive behavior
`states. On the one hand, this meets the needs of traditional observational research practices in the obesity and
`metabolic health domain (based on, and validated through, careful expert human coding of data) while on the other,
`
`UrbanSense08 - Nov. 4, 2008, Raleigh, NC, USA
`
`21
`
`16
`
`

`

`this enables new analysis capabilities that have not been possible before such as providing information on user
`emotional state in conjunction with physical activity and energy expenditure.
`Many aspects of human behavior are inherently multimodal or require multimodal processing. For example,
`measuring and understanding energy expenditure and its etiology requires processing not only activity from ac-
`celerometers but other data such as pulse rate, ECG, oxygen intake, as well as contextual information such as
`emotions that are marked by humans through their voice, body posture and through physiological signals skin
`conductance measures (electro dermal response). Hence, to model human behavior and task-specific activity, both
`in terms of what people do, how they do it, and why they do it, it is critical to understand and capture the interplay
`between such multimodal streams. Multi-modal coverage of our approach enables cross-channel comparison and
`verification (allowing us, for example, to capture relationships between increased heart rate, increased emotional
`activity, and changes in physical activity). Our approach to this problem is grounded in statistical signal processing.
`In the current work, we summarize preliminary results on activity assessment. We consider a mix of low mobility
`(lying down, sitting, standing) and higher mobility (walking, running) states. Features of our problem and approach
`do appear in the prior literature. Much work on activity detection appears to center on accelerometer data alone
`(e.g.[8], [3], [10]) with some systems employing many accelerometer packages. On the other hand, multi-sensor
`WBANs have been implemented and deployed (see e.g. [12], [9], [6]); however in those works, the emphasis
`was on the higher layer communication network processing and hardware design – signals from each sensor
`were transmitted directly to a central decision making unit. Our focus is on a modest number of heterogeneous
`sensors and the utilization of multi-modal signal processing methods; we wish to design decision making and data
`interpretation methods that will reside within the WBAN and allow for interaction with the WBAN wearer. For
`our pediatric obesity application, activity detection is an indirect measure of energy expenditure quantification as
`discussed above. In [4], multi-modal classification is considered. There are some key differences to the approach
`taken herein. First, while different sensors are employed, they are similar in the types of measurements taken (e.g.
`accelerometers, gyroscopes and tilt measurements), herein we use sensors which measure fundamentally different
`quantities that are correlated, but the statistical relationships are unclear a priori. The goal of [4] is to determine
`a sampling scheme (with respect to frequency of sampling and sleeping/waking cycles) for multiple sensors to
`minimize power consumption. The authors show that their new methods achieve reduced power relative to classical
`joint schemes. Our goal is on classifier performance with heterogeneous sensors – future versions of our methods
`could incorporate power minimization strategies of [4]. An important question to address is how the correlation
`between measurements affects power minimization. We conjecture that the sensors employed in [4] have more highly
`correlated observations with regards to the states of interest than our sensors and thus greater power minimization
`is possible through the use of their methods.
`As our WBAN must be used for a diverse set of decision making processes, all sensors may not be uniformly
`useful for each task. We, in fact, see this with the activity detection problem considered herein.
`
`II. KNOWME NETWORK ARCHITECTURE
`The basic foundation of the KNOWME network is our three
`tier network architecture as depicted in Figure 1. The first tier’s
`goal is data collection based on the heterogeneous sensors that
`are coupled to a mobile phone which acts as a “base station,”
`equipped with data transmission and processing capabilities.
`The second tier is a web server that receives data and can
`perform additional processing; the web server transmits the
`data to the final tier: a back-end database server that stores the
`information. In the sequel, we shall discuss the specific sensors
`employed.
`Currently, the primary focus of this research is to perform multi-modal sensing and interpretation of data to
`serve some of the end-user needs. As such, significant effort has been spent in integrating heterogeneous sensors
`to a mobile phone. One challenge in integrating heterogeneous sensors is that these sensors have different APIs,
`packaging, and data collection methods. In addition to integrating multiple sensors, synchronization of the data
`
`Fig. 1.
`Three-tier architecture overview of wireless body
`area network sensor system.
`
`22
`
`17
`
`

`

`received from multiple sensors in the phone is critical for statistical correlation of sensor data and to perform the
`multi-modal data processing. Sensor information is continuously recorded on the local storage on the mobile phone.
`Our mobile device platform has a 8GB in-built flash memory that can be used for storing sensor information. Sensor
`data rates vary from 300bps for the accelerometers to 100 bps for the heart rate monitor. Using these data rates,
`we estimate that our 8GB local storage can store 1000 days worth of data. As the Bluetooth wireless link is a
`bottleneck for our current data collection, we use time-division multiple-access to schedule the data from different
`sensors (equal time share).
`The software development phase uses well-known unit testing to extensively test the mobile software suite. In
`order to minimize errors in configuring the software, our software has several built-in checks to advise the user if
`any of the sensor readings do not match expected sensor behavior. Since the mobile device has to transmit the data
`to the backend servers, we are currently developing an opportunistic data transfer mechanism that uses an open WiFi
`network where available to transfer data both efficiently and cheaply. In the absence of WiFi networks, the mobile
`software is configured to automatically use the cellular data network to transmit the data. Our initial deployment
`is mostly with graduate and undergraduate student test subjects with limited (on-going) pilot experiments with
`children in the Exercise Physiology Lab at the USC Keck School of Medicine.
`
`A. Sensor Systems
`The sensor layer is a collection of off-the-
`shelf devices that measure features which can
`provide insight about metabolic activity; most
`(with the exception of galvanic skin response)
`are also capable of wirelessly transmitting this
`data over a Bluetooth interface. The current
`study employs an Alive Technologies[1] elec-
`Fig. 2.
`(a) ECG monitor, (b) pulse oximeter, (c) Nokia Smartphone (GPS and
`trocardiograph (ECG). The ECG is a single
`accelerometer).
`channel device with 8 bit resolution and a peak sampling rate of 300 samples/second. The pulse-oximeter, also from
`Alive, provides non-invasive monitoring of oxygen saturation (SpO2) and pulse rate. The oximeter is a Bluetooth
`slave device that supports the Bluetooth Serial Port Profile (SPP). We also have BodyMedia WMS sensors [2] to
`measure Galvanic Skin Response (GSR) 1 and motion estimation using accelerometers. We use feature rich Nokia
`N95 as the mobile phone platform. N95 supports Bluetooth 2.0 + EDR for quick pairing with external Bluetooth
`sensors, and has 3G and WiFi radios for high bandwidth data transfer. In addition to the high bandwidth radio
`capabilities, the N95 mobile phone platform has a highly accurate built-in assisted GPS unit that uses a combination
`of GPS satellites, cellular tower and WiFi scanning to obtain a GPS position lock in less than 10 seconds. The
`stated location accuracy of GPS unit is 30 meters. We have observed accuracy at less than 3 meters in practice.
`The data collected from multiple sensors is geo-tagged using the location data collected from the in-built GPS.
`Furthermore, our system is also capable of audio and video tagging to assist users to supplement the automatically
`collected sensor data (as in [14]). Some WBAN components are depicted in Figure 2.
`
`III. ACTIVITY MODELING
`Data collected from our experimental system setup can be used in multiple contexts, for instance by the users
`to regularly monitor their physical well being as well as by medical practitioners in assessing the physical health
`of their patients. Here, we describe one such application of using the data to automatically derive the activity of a
`person with data collected from multiple sensors. Statistical modeling of various test subject states was undertaken
`based on the data collected from the WBAN. We examined five different states: lying down, sitting, standing,
`walking and running. Again, to reiterate, activity detection has been previously considered with an emphasis on the
`use of many accelerometers, yielding a cumbersome network to wear. We conjecture that multimodal data analysis
`will enable the achievement equal or even better accuracy and robustness in activity detection with fewer sensors.
`
`1The data of the WMS GSRs are not currently included due to issues with time synchronization.
`
`(a) (b) (c)
`
`23
`
`18
`
`

`

`In this research, multiple distributions
`were considered to fit the data which
`for each sensor was predominantly uni-
`modal in nature. After extensive exper-
`imentation, the use of the pulse oxime-
`ter sensor was abandoned due to lim-
`ited change in readings for any of the
`states of interest for our activity de-
`tection problem. Thus, we focused on
`ECG and accelerometer data. The dis-
`tributions under consideration were: T
`log-logistic, one-side Gaussian and Laplacian. Where possible,
`location-scale, Gaussian,
`log-normal,
`logistic,
`Gaussian distributions were selected to facilitate the determination of joint densities. The ECG data were pre-
`processed as follows: peak detection was performed and the inter-peak time collected. The inter-peak time was
`modeled as a Gaussian random variable. An average of the empirical variance for each of the axes over a pre-
`specified window of time for the accelerometer data was employed. The walking and running state data were
`modeled as Gaussian; however, the lower-activity level data (lying down, sitting and standing) was modeled as a
`Laplacian to achieve a better fit. Figure 3 (L) and (R) shows the ECG and accelerometer data for the running and
`sitting modes, respectively. We see that both states are relatively well distinguished from each other with significant
`differences in the accelerometer data.
`
`(L) ECG and (R) accelerometer data from the heart-rate monitor for sitting and
`
`Fig. 3.
`running.
`
`Fig. 4.
`(L) Statistical fitting for higher activity states (accelerometer data): sitting,
`walking, and running. (R) Statistical fitting for lower activity states (ECG data): lying
`down, sitting, and standing.
`
`Not surprisingly, ECG and accelerom-
`eter data had different discriminatory
`properties for the various states, un-
`derscoring the benefits of multi-modal
`sensing and signal processing. In Fig-
`ure 4, we see the statistical fits for
`the accelerometer data for high activity
`states and the statistical fits for the ECG
`data for low activity fits. To develop
`bivariate models (joint densities) for the
`ECG and accelerometer data, additional
`processing (resampling) was required to
`determine the correlation between the ECG statistic and the accelerometer statistic in the high-activity levels.
`In the low-activity level
`cases, the ECG and accelerom-
`eter statistics were assumed to
`be independent. The resulting
`bivariate densities for each of
`the five hypotheses are shown
`in Figure 5(L) and (R). For
`clarity, the low activity states
`are shown separate from the
`higher activity states. Bivari-
`ate testing yielded state detec-
`tion rates on the order of 85%
`to 95% – achieving detection
`rates with two heterogeneous
`sensors comparable to the rates found in [3], where nine single mode (accelerometer) sensors were employed.
`
`Fig. 5. Bivariate distributions for (L) running, walking and sitting and for (R) lying down, sitting
`and standing.
`
`300
`
`250
`
`200
`
`150
`
`100
`
`50
`
`0
`
`300
`
`250
`
`200
`
`150
`
`100
`
`50
`
`0
`
`100
`
`200
`
`300
`Running ECG Raw Data
`
`400
`
`500
`
`600
`
`200
`
`400
`
`600
`Sitting ECG Raw Data
`
`800
`
`1000
`
`1200
`
`150
`
`100
`
`50
`
`0
`
`−50
`
`−100
`
`−150
`
`0
`
`150
`
`100
`
`50
`
`0
`
`−50
`
`−100
`
`−150
`0
`
`500
`
`1000
`
`1500
`
`Running
`
`2000
`
`2500
`
`3000
`
`3500
`
`500
`
`1000
`
`1500
`
`2000
`Sitting
`
`2500
`
`3000
`
`3500
`
`Sitting
`
`Walking
`
`
`
`Sitting Data
`Walking Data
`Running Data
`(Sitting) One−sided Normal Fit
`(Walking) Normal Fit
`(Running) Normal Fit
`
`Running
`
`20
`
`40
`
`60
`
`80
`
`100
`
`120
`
`Standard Deviation of Accelerometer Data
`
`0.18
`
`0.16
`
`0.14
`
`0.12
`
`0.1
`
`0.08
`
`0.06
`
`0.04
`
`0.02
`
`
`
`Amplitude
`
`
`
`Lying Data
`Lying Fit
`Sitting Data
`Sitting Fit
`Standing Data
`Standing Fit
`
`Sitting
`
`Lying
`
`10
`
`Standing
`
`8
`
`6
`
`4
`
`2
`
`Density
`
`0
`
`
`
`0.6
`
`0.7
`
`0.8
`
`0.9
`Time (Sample)
`
`1
`
`1.1
`
`1.2
`
`x 10−3
`
`Sitting
`
`Walking
`
`Running
`
`0
`
`20
`
`40
`60
`Std Dev of Accelerometer
` Data
`
`0
`
`100
`
`200
`
`80
`
`400
`
`100
`
`300
`
`Inter−peak Time of ECG
`
`012345678
`
`PDF
`
`0.2
`
`0.15
`
`Lying
`
`0.1
`Sitting
`
`0.05
`
`200
`
`250
`
`300
`
`Standing
`
`0
`
`150
`
`0.6
`
`0.8
`
`1
`
`350
`
`400
`
`1.2
`
`1.4
`
`Std Dev of Acceleromter
` Data
`
`Inter-peak Time
` of ECG
`
`24
`
`19
`
`

`

`IV. OBSERVATIONS AND ONGOING WORK
`Our preliminary system successfully collects data and transmits it to the cellular phone. We conjecture from
`our experiments that a few heterogeneous sensors may offer better discrimination and robustness than many
`homogeneous sensors. Our preliminary data for activity detection in comparison to [3] appears to bear this out this
`conjecture. There are however important engineering challenges associated with WBANs, especially for activity
`detection. For our particular set up, we are limited by the mobile phone platform which can only accommodate a
`maximum of eight different sensors. If all sensors sample at their maximum sampling rate, the expected throughput
`would exceed the capabilities of the Bluetooth link leading to dropped packets. The battery power of the cellular
`phone is another bottleneck for the system. Finally, for activity detection, high activity/mobility can impair a sensor’s
`ability to sense. This fact can be viewed two ways: it is detrimental in that we lose sensor accuracy, on the other
`hand, new features are introduced into the signal which are still indicative of high activity. Our preliminary results
`suggest that sensor selection and prioritization will be important to ensure that packets are not lost; furthermore
`energy aware sensor management will be critical.
`We have recently conducted a pilot study with two pre-adolescent girls following an observation protocol typical
`for pediatric obesity studies. We are curre