as) United States
`a2) Patent Application Publication co) Pub. No.: US 2011/0066383 A1
`Jangleetal. Mar.17, 2011 (43) Pub. Date:
`
`
`
`US 20110066383A1
`
`(54)
`
`(75)
`
`INDENTIFYING ONE OR MOREACTIVITIES
`OF AN ANIMATE OR INANIMATE OBJECT
`Inventors:
`Jeetendra Jangle, Fremont, CA
`(US); Vijay Nadkarni, San Jose,
`CA (US)
`
`Publication Classification
`
`(51)
`
`Int.Cl
`ooeol,
`np fen
`(
`01)
`(52) US. CD. ec ceceeeecneteneeseeenenees 702/19; 702/141
`
`(73) Assignee:
`
`.
`(21) Appl. No.:
`:
`(22)
`Filed:
`
`Wellcore Corporation, San Jose,
`CA (US)
`12/883,304
`.
`16,
`Sep. 16, 2010
`
`ABSTRACT
`(57)
`Methods, systems and apparatusfor identifying an activity of
`an animate or inanimate object are disclosed. One method
`includes identifying each elemental motion of a sequence of
`elemental motions of a device attached to the animate or
`inanimate object. The activity of the animate or inanimate
`object can be identified by matching the sequence ofidenti-
`fied elemental motions of the device with a library of stored
`Related U.S.Application Data
`sequences of elemental motions, wherein each stored
`(63) Continuation-in-part of application No. 12/560,069,
`filed on Sep. 15, 2009.
`sequence of elemental motions corresponds with an activity.
`
`
`Acc.
`4412
`
`4
`i
`|
`
`400
`.
`:
`.
`Motion Detection Device
`
`
`
`|
`{|
`/
`
`Compare
`Le
`. mpareor
`
`130
`
`
`
`
`
`
`149
`
`
`
`
`
`
`
`Centrolier
`
`we
`
`a
`
`
`
`
`
`Sequences of
`Elemental
`Motions
`i
`ee
`5
`Library
`io12
`
`
`
`hee
`it4
`id4
`
`Acc,
`4468
`
`
`
`Aro
`a
`420
`
`Audio.
`180)
`
`
`
`
`
`GPS
`490
`
`
`4
`
`4
`
`>
`
`>
`
`VY
`
`“
`
`/
`
`a
`
`‘“
`
`4
`Contraller can be internal or
`
`inciude external controlier(s) of
`the Network
`
`
`
`
`
`
`
`
`
`
`Sequences of
`Activities
`Library
`16 ia)
`
`Communication
`
`Connection to
`Network
`
`—» Wireless
`
`yw
`
`1
`
`APPLE 1012
`
`APPLE 1012
`
`1
`
`

`

`Patent Application Publication Mar. 17,2011 Sheet 1 of 8
`
`US 2011/0066383 Al
`
`
`Ae.
`f\
`406
`112
`Motion Detection Device
`|
`|
`
`
`
`
`Aes
`‘
`
`114
`
`AL
`
`ae
`
`Campare
`430
`
`Acceleration
`
`Signature
`Library
`
`
`
`Ag,
`116
`
`
`
`
`Cantroller
`
`
`
`
`
`GPS
`, <<?‘
`790
`
`170
`
`y
`
`YO
`
`y /
`
`Y
`
`7
`
`7
`
`/
`Controller can be internal ar
`
`a
`Sequences of
`Elemental
`Motions
`Library
`189
`
`include external contratier(s} of
`the Network
`
`Sequences of
`Activities
`Library
`
`Communication
`
`Connection to
`Network
`
`——, Wireless y
`
`FIGURE1
`
`2
`
`

`

`Patent Application Publication Mar. 17,2011 Sheet 2 of 8
`
`US 2011/0066383 Al
`
`identifying cach elemental motion of a sequence of elemental motions of a device atlached to
`the animate or inanimate objoct
`
`210
`
`240
`
`Identifying the behavior of the animate or inanimate objeci, comprising matching the sequence
`ofidentified activitics of the animate or inanimate objoct with a stored soquonces ofactivities,
`wherein each stored sequence ofactivities corresponds with an identified behavior
`
`identifying the activity of the animate or inanimate object, comprising matching the sequence
`of identified elemental motions of the device with a siored sequences of elemental motions,
`wherein each stored sequence of elemental motions corresponds wilh an activity
`
`identifying each activity of a sequenceofactivities of the animate or inanimate object
`
`30
`
`FIGURE 2
`
`3
`
`

`

`Patent Application Publication Mar. 17,2011 Sheet 3 of 8
`
`US 2011/0066383 Al
`
`identified
`Behaviors
`
`~
`
`/
`
`¢
`
`.
`Location
`
`Identified
`Activities
`
`320
`
`{
`
`a Identify Behavior |
`
`\
`
`\ (signature) /
`
`é
`|
`\,
`
`Identify Activity
`
`|
`
`/
`
`ir ~ ~
`\
`
`~
`
`Elemental
`Motion
`
`>
`identified
`/
`\
`ideniifie
`an
`i sates
`\E mantal Mation /
`
`s
`ues
`co
`‘.
`tdentified
`f
`Elemental Motion|
`\
`SS
`Ls
`
`“
`oN
`(Atomic Motia
`\.
`(signature)
`
`a
`~
`/Atomic Motion\
`
`FIGURE 3
`
`4
`
`

`

`Patent Application Publication Mar. 17,2011 Sheet 4 of 8
`
`US 2011/0066383 Al
`
`Established Daily Pattern
`Current Day Pattern
`
`FIGURE 4A
`
`FIGURE 4B
`
`5
`
`

`

`Patent Application Publication Mar. 17,2011 Sheet 5 of 8
`
`US 2011/0066383 Al
`
`100 +
`
`}
`
`Acceleration
`
`||fo ;
`NeharncnctePetraran
`
`-106
`
`1
`
`Slow Falling and lying down summirsauit
`
`
`
`
`
`
`-100 T
`. poyan
`he
`
`450
`
`250
`
`x Axis
`
`350
`
`FIGURE 5A
`
`Slipping and falling on back on a bouncy surface (air matiress)
`
`Acceleration
`
`100 +
`
`
`
`
`
`150
`
`250 ave
`
`350
`
`FIGURE 5B
`
`6
`
`

`

`Patent Application Publication Mar. 17,2011 Sheet 6 of 8
`
`US 2011/0066383 Al
`
`Generating an acceleration signature based on sensed acceleration of the object
`
`10
`
`matching of the acceleration signature
`
`Matching the acceleration signature with at least one of a plurality of stored
`acceleration signatures, wherein each stored acceleration signatures corresponds with
`a type of motion
`
`20
`
`identifying the type of mation of the oblect based on the statistical matching or exact
`
`FIGURE 6
`
`7
`
`

`

`Patent Application Publication Mar. 17,2011 Sheet 7 of 8
`
`US 2011/0066383 Al
`
`The motion detection device determining what network connections are available to
`the motion detection device
`
`He
`
`£20
`
`The motion detection device distributing at least some of the acceleration signature
`matching processing if processing capability is available to the motion detection device
`though available network connections
`
`FIGURE 7
`
`8
`
`

`

`Patent Application Publication Mar. 17,2011 Sheet 8 of 8
`
`US 2011/0066383 Al
`
`
`
`300
`
`
`a a
`
`
`
`
`
`Processor
`
`
`
`
`
`
`610
`Bluetooth ®
`
`ee
`
`aor te
`
`
`
`ae a
`
`a -
`
`845
`ZigBee
`
`
`
`
`
`Processor
`
`830
`
`we
`
`No Network
`Available
`
`4
`
`Cellular
`\
`820
`WZ tego
`
`
`Processor
`
`
`
`
`
`350
`
`Horne
`Base Station
`3840
`
`FIGURE 8
`
`9
`
`

`

`US 2011/0066383 Al
`
`Mar. 17, 2011
`
`INDENTIFYING ONE OR MOREACTIVITIES
`OF AN ANIMATE OR INANIMATE OBJECT
`
`RELATED APPLICATIONS
`
`[0001] This patent application is a continuation in part
`(CIP) of U.S. patent application Ser. No. 12/560,069 filed on
`Sep. 15, 2009, which is incorporated by reference.
`
`FIELD OF THE DESCRIBED EMBODIMENTS
`
`[0002] The described embodiments relate generally to
`monitoring motion. More particularly, the described embodi-
`ments relate to a method, system and apparatus for identify-
`ing one or moreactivities of an animate or inanimate object.
`
`BACKGROUND
`
`of identified elemental motions of the object with stored
`sequences of elemental motions, wherein each stored
`sequence of elemental motions corresponds with an activity.
`[0009] Another embodimentincludes a system for identi-
`fying an activity of a animate or inanimate object. The system
`includes means for identifying each elemental motion of a
`sequence of elemental motions of an device attached to the
`animate or inanimate object, and means for identifying the
`activity of the animate or inanimate object, comprising
`matching the sequenceof identified elemental motionsof the
`device with a library of stored sequences of elemental
`motions, wherein each stored sequence of elemental motions
`corresponds with an activity. The meansfor identifying each
`elemental motion includes means for generating an accelera-
`tion signature based on sensed acceleration of the device,
`meansfor matching the acceleration signature with at least
`one of a plurality of stored acceleration signatures, wherein
`each stored acceleration signatures corresponds witha type of
`motion, and meansfor identifying the type of motion of the
`device based on the matching of the acceleration signature
`with the stored acceleration signature.
`[0010] Other aspects and advantages of the described
`embodiments will become apparent from the following
`detailed description, taken in conjunction with the accompa-
`nying drawings,illustrating by way of example the principles
`of the described embodiments.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0003] There is an increasing need for remote monitoring
`ofindividuals, animals and inanimate objectsin their daily or
`natural habitats. Many seniors live independently and need to
`have their safety and wellness tracked. A large percentage of
`society is fitness conscious, and desire to have, for example,
`workouts and exercise regimen assessed. Public safety offic-
`ers, such as police and firemen, encounter hazardoussitua-
`tions ona frequentbasis, and need their movements, activities
`and location to be mappedoutprecisely.
`[0004] The value in such knowledge is enormous. Physi-
`cians, for example, like to know their patients sleeping pat-
`terns so they can treat sleep disorders. A senior living inde-
`pendently wants peace of mindthatif he hasafall it will be
`[0011]
`FIG. 1 shows an example of a block diagram of a
`detected automatically and help summoned immediately. A
`motion-detection and tracking device.
`fitness enthusiast wants to track her daily workout routine,
`[0012]
`FIG. 2 is a flow chart that includes steps of an
`capturing the various types of exercises, intensity, duration
`example ofa method of identifying an activily ofa animate or
`and caloric burn. A caregiver wants to know that herfatheris
`inanimate object.
`living an active, healthy lifestyle and taking his daily walks.
`[0013]
`FIG. 3 shows an example of hierarchical relation-
`The police would like to know instantly when someonehas
`ships between elemental motions, activities, behaviors and
`been involved in a car collision, and whether the victims are
`behavioral patterns.
`moving or not.
`[0014]
`FIGS. 4A, 4B are plots that show examples of an
`[0005] Existing products for the detection of animate and
`established activity pattern and a daily activity pattern for an
`inanimate motionsare simplistic in nature, and incapable of
`animate or inanimate object, allowing for detection of
`interpreting anything more than simple atomic movements,
`changes in behavior.
`such as jolts, changes in orientation and thelike. It is not
`[0015]
`FIGS. 5A, 5B shows examplesof time-lines of sev-
`possible to draw reliable conclusions about human behavior
`eral different acceleration curves (signatures), wherein each
`from these simplistic assessments.
`signature is associated with a different type of sensed or
`[0006]
`It is desirable to have an apparatus and methodthat
`detected motion.
`can accurately identify and monitoractivities of an animate or
`inanimate object
`
`SUMMARY
`
`[0007] An embodimentincludes a method ofidentifying an
`activity of an animate or inanimate object. The method
`includes identifying each elemental motion of a sequence of
`elemental motions of a device attached to the animate or
`inanimate object. The activity of the animate or inanimate
`object can be identified by matching the sequence of identi-
`fied elemental motions of the device with a stored sequences
`of elemental motions, wherein each stored sequence of
`elemental motions corresponds with an activity.
`[0008] Another embodiment
`includes an apparatus for
`identifying an activity of an animate or inanimate object. The
`apparatus includes a controller operative to identify each
`elemental motion of a sequence of elemental motions of
`device attached to the animate or inanimate object. The con-
`troller is further operative to identify the activity of the ani-
`mate or inanimate object, comprising matching the sequence
`
`FIG. 6 is a flow chart that includes the steps of one
`[0016]
`example of a method of identifying a type of motion of an
`animate or inanimate object.
`[0017]
`FIG. 7 is a flow chart that includes steps of one
`example of a method of a motion detection device checking
`network availability for improvements in speed and/or pro-
`cessing powerof acceleration signature matching.
`[0018]
`FIG. 8 shows an example of a motion detection and
`tracking device that can be connected to one of multiple
`networks.
`
`DETAILED DESCRIPTION
`
`[0019] The described embodiments include methods, sys-
`tems and apparatusesthat provide human activity and motion
`pattern recognition, allowing a determination of granular
`level activities of daily living being performed by a user.
`Embodiments of these granular feature determinations pro-
`vide the capability to identify user safety. For example, a
`comprised safety situation, suchas, the user falling down can
`
`10
`
`10
`
`

`

`US 2011/0066383 Al
`
`Mar. 17, 2011
`
`be identified. By combining the granular motion actions and
`features with data from other sensors such as GPS (global
`positioning system), vital stats sensors and otherinferred data
`such as time,it is possible to establish the high level activity
`being performedby the user. Knowledge of high level activi-
`ties being performed during time periods such as a day allows
`for the building of various interesting applications that are
`useful for improving the quality oflife of the users and their
`caregivers and to customize and optimize care plans. Armed
`with the knowledge of variation of peoples’ behavior, repeti-
`tive and variant patterns across people, age, gender, location
`and time, systems can provide customized services for indi-
`viduals and categories of people.
`[0020] The monitoring of humanactivities generally falls
`into three categories: safety, daily lifestyle, and fitness. By
`carefully interpreting human movementsit is possible to draw
`accurate and reasonably complete inferences aboutthestate
`of well being of individuals. A high degree of sophistication
`is required in these interpretations. Simplistic assessments of
`human activity lead to inaccurate determinations, and ulti-
`mately are of questionable value. By contrast, a comprehen-
`sive assessment leads to an accurate interpretation and can
`prove to be indispensable in tracking the well being andsafety
`of the individual.
`
`To draw accurate inferences about the behavior of
`[0021]
`humans,it turns out that the atomic movements become sim-
`ply alphabets that include elemental motions. Furthermore,
`specific sequences of elemental motions becomethe vocabu-
`lary that comprises human behavior. As an example, take the
`case of a person who leaves the home and drives to the
`shopping center. In such a scenario, the behavioral pattern of
`the person is walking to the door or the house, opening and
`closing the door, walking further to the car, settling down in
`the car, starting the engine, accelerating the car, going
`through a series of stops, starts and turns, parking the car,
`getting out and closing the car door, and finally walkingto the
`shopping center. This sequence of human behavior is com-
`prised of individual motions such as standing, walking,sit-
`ting, accelerating (in the car), decelerating, and turningleft or
`right. Each individual motion, for example walking, is com-
`prised of multiple atomic movements such as acceleration in
`an upwarddirection, acceleration in a downwarddirection, a
`modest forward acceleration with each step, a modest decel-
`eration with each step, and so on.
`[0022] With written prose, letters by themselves convey
`almost no meaningat all. Words taken independently convey
`individual meaning, but do not provide the context to com-
`prehendthesituation. It takes a complete sentence to obtain
`that context. Along the sameline of reasoning, it requires a
`comprehension of a complete sequence of movements to be
`able to interpret human behavior.
`[0023] Although there is an undeniable use for products
`that are able to detect complex human movementsaccurately,
`the key to the success of such technologies lies in whether
`users adopt them or not. The technology needs to capture a
`wide range of human activities. The range of movements
`should ideally extendto all types ofdaily living activities that
`a human being expects to encounter—sleeping, standing,
`walking, running, aerobics, fitness workouts, climbingstairs,
`vehicular movements, falling, jumping and colliding, to name
`some of the more commonones.
`
`Itis important to detect humanactivities with a great
`[0024]
`deal of precision. In particular, activities that relate to safety,
`fitness, vehicular movements, and day to daylifestyle pat-
`
`terns such as walking,sleeping, climbing stairs, are important
`to identify precisely. For example,it is not enough to know
`that a person is walking. One needs to know the pace and
`duration of the walk, and additional knowledge of gait,
`unsteadiness, limping, cadence andthe like are important.
`[0025]
`Itis critical that false positives as well as false nega-
`tives be eliminated. This is especially important for cases of
`safety, such as falls, collisions, and the like. Human beings
`comein all types—short, tall, skinny, obese, male, female,
`athletic, couch potato, people walking with stick/rolator,
`people with disabilities, old and young. The product needs to
`be able to adapt to their individuality andlifestyle.
`[0026] The embodiments described provide identification
`of types of motion of an animate or inanimate object. Motion
`is identified by generating acceleration signatures based on
`the sensed acceleration of the object. The acceleration signa-
`tures are compared with a library of motionsignature, allow-
`ing the motion of the object
`to be identified. Further,
`sequences of the motions can be determined, allowing iden-
`tification of activities of, for example, a person the objectis
`attachedto.
`
`Just as the handwritten signatures of a given human
`[0027]
`being are substantively similar from one signature instance to
`the next, yet have minordeviations with each new instance, so
`too will the motion signatures of a given humanbe substan-
`tively similar from one motion instance to the next, yet have
`minordeviations.
`
`[0028] Algorithms used for pattern recognition (signature
`matching) should havethe sophisticationto accurately handle
`a wide range of motions. Such algorithms should have the
`ability to recognizethe identical characteristics ofa particular
`motion by a given humanbeing, yet allow for minor varia-
`tions arising from human randomness. Additionally,
`the
`devices used to monitor peoples’ movement need to be min-
`iature and easy to wear. These two objectives are fundamen-
`tally opposed. However, the described embodiments provide
`a single cohesive device and system that is both sophisticated
`enough to detect a wide range of motions.
`[0029]
`FIG. 1 shows an example of a block diagram of a
`motion-detection and tracking device. The motion detection
`device can be attached to an animate or inanimate object, and
`therefore, motion of the object that can be detected and iden-
`tified. Based on the identified motion, estimates ofthe behav-
`ior and conditions of the object can be determined.
`[0030] The motion detection device includes sensors (such
`as, accelerometers) that detect motion of the object. One
`embodimentofthe sensors includes accelerometers 112, 114,
`116 that can sense, for example, acceleration of the object in
`X,Y and Z directional orientations. It is to be understood that
`other types of motion detection sensors can alternatively be
`used.
`
`[0031] An analogto digital converter (ADC) digitizes ana-
`log accelerometer signals. The digitized signals are received
`by compare processing circuitry 130 that comparesthe digi-
`tized accelerometer signals with signatures that have been
`stored within a library of signatures 140. Each signature
`corresponds with a type of motion. Therefore, when a match
`between the digitized accelerometer signals and a signature
`stored in the library 140, the type of motion experienced by
`the motion detection device can determined.
`
`[0032] An embodimentincludesfiltering the accelerometer
`signals before attempting to match the signatures. Addition-
`ally, the matching process can be made simpler by reducing
`the possible signature matches.
`
`11
`
`11
`
`

`

`US 2011/0066383 Al
`
`Mar. 17, 2011
`
`[0033] An embodiment includes identifying a previous
`human activity, context. That is, forexample, by knowing that
`the previous humanactivity was walking, certain signatures
`can intelligently be eliminated from the possible matches of
`the present activity that occurs subsequent to the previous
`human activity (walking).
`[0034] An embodimentincludes additionally reducing the
`numberofpossible signature matches by performinga time-
`domain analysis on the accelerometer signal. The time-do-
`main analysis can be used to identify a transient or steady-
`state signature of the accelerometer signal. That
`is, for
`example, a walk may have a prominent steady-state signature,
`whereas a fall may have a prominent transient signature.
`Identification of the transient or steady-state signature of the
`accelerometer signal can further reduceor eliminate the num-
`ber of possible signature matches, and therefore, make the
`task ofmatching the accelerometer signature with a signature
`within the library of signature simpler, and easier to accom-
`plish. More specifically, the required signal processing is
`simpler, easier, and requires less computing power.
`[0035] A controller 170 manages the signature matching
`andidentification. As will be described, the controller 170 can
`be connected to an external network. The processing of the
`controller 170 can be performed locally or distributed
`amongst other controller through the network. Determination
`of where processing takes place (that is, what controller or
`processor) can be based on a balanceof speedof the process-
`ing, and powerofthe local controller(that is, power required
`of a controller within a mobile device). The controller 170
`also managesthe activity identification based on sequences of
`motion, and managesthe identifications of behaviors based
`on the identified activities as will be described. A sequences
`ofelemental motionslibrary 150 can be used for matching the
`sequences of motion to a particular activity. A sequences of
`activities library 160 can be used for matching sequences of
`activities to a particular behavior. Again, the processing ofthe
`controller 170, as well as the libraries 150, 160, 170 can be
`distributed across the network through a wired or wireless
`connection.
`
`[0036] Upon detection of certain types of motion, an audio
`device 180 and/or a global positioning system (GPS) 190 can
`engaged to provide additional informationthat can be used to
`determine the situation of, for example, a human being the
`motion detection device is attachedto.
`
`[0037] Thecondition,or informationrelating to the motion
`detection device can be communicated through a wired or
`wireless connection. A receiver of the information can pro-
`cess it, and make a determination regarding the status of the
`human being the motion detection deviceis attached to. Infor-
`mation and history of a user of the motion detection device
`can be utilized to characterize the user/FIG. 2 is a flow chart
`
`that includes steps of an example of a method ofidentifying
`an activity of an animate or inanimate object. A first step 210
`includes identifying each elemental motion of a sequence of
`elemental motions of a device attached to the animate or
`
`inanimate object. A second step 220 includes identifying the
`activity of the animate or inanimate object, comprising
`matching the sequenceof identified elemental motionsof the
`device with stored sequences of elemental motions, wherein
`each stored sequence of elemental motions corresponds with
`an activity.
`[0038] A plurality or sequenceof identified activities of, for
`example, a human being, can be usedto identify a behavior of
`the human being.As such,a third step 230 includes identify-
`
`ing each activity of a sequenceofactivities of the animate or
`inanimate object. A fourth step 240 includes identifying the
`behavior of the animate or inanimate object, comprising
`matching the sequenceofidentified activities of the animate
`or inanimate object with a stored sequences of activities,
`wherein each stored sequence ofactivities corresponds with
`an identified behavior.
`
`[0039] The animate or inanimate object can be many
`things, such as, a human being or an animal. Alternatively or
`additionally, the animate or inanimate object can be an object
`associated with a humanbeing,such as, a vehicle. The device
`can be attached to the animate or inanimate object in many
`different ways. For example, the device can be attached to a
`human being, or clothing (pants, shirt, jacket, and/or hat)
`being worn by the human being. The device can be within a
`pendant or necklace being worn by the human being. The
`device can be attached, for example, to a vehicle being oper-
`ated by the humanbeing.
`identifying each elemental
`[0040]
`For an embodiment,
`motion includes generating an acceleration signature based
`on sensed acceleration of the device, matching the accelera-
`tion signature with at least one of a plurality of stored accel-
`eration signatures, wherein each stored acceleration signa-
`tures corresponds with a type of motion, and identifying the
`type of motion of the device based on the matching of the
`acceleration signature with the stored acceleration signature.
`[0041] Other factors can be used to refine ((mprove) the
`identification of the activity. These factors can include, for
`example, analyzing timing of the identified activity. For an
`embodiment, the timing includesat least one of an hour of a
`day, a day of a week, a week of a month, a month ofa year.
`Other factors include analyzing at least one identified loca-
`tion ofthe identified activity, analyzing a rate of change of a
`location of the animate or inanimate object, analyzing pat-
`terns ofa plurality of identified activities, and/or analyzing an
`age of the animate or inanimate object.
`[0042] As previously mentioned, behaviors can be identi-
`fied based on sequencesof identified activities. Embodiments
`further include tracking at least one behavior ofthe animate or
`inanimate object over time. One embodimentincludes iden-
`tifying patterns of the at least one behavior. An embodiment
`includes grouping the patterns of the animate or inanimate
`objects based on a commonparameter between the animate or
`inanimate objects. Embodimentsinclude identifying changes
`in atleast one behavior ofthe animate or inanimate object. An
`embodimentfurther includes sending an alert upon identifi-
`cation of predeterminedset of behavior changes.
`[0043]
`FIG. 3 shows an example of hierarchical relation-
`ships between elemental motions, activities, behaviors and
`behavioral patterns. At the lowest level of the hierarchy are
`the identified elemental motions 310. As described,
`the
`elemental motions can be identified by sensing signatures of
`motion (by, for example, accelerometers within a device
`attached to a user) and matching the signatures within known
`signatures. At the next higher level of the hierarchy are the
`identified activities 320. As described, the activities can be
`identified by matching determined sequences of elemental
`motions with previously known sequences of elemental
`motions. At the next higher level of the hierarchy are identi-
`fied behaviors 330. As described, the behaviors can be iden-
`tified by matching determined sequencesof activities with
`previously known sequencesofactivities. Each of the levels
`of hierarchy can be aided with additional information. For
`example, the identified behaviors can be moreintelligently
`
`12
`
`12
`
`

`

`US 2011/0066383 Al
`
`Mar. 17, 2011
`
`identified with time, location andor ageofthe user. Addition-
`ally, this information can be used for grouping andidentified
`behavior patterns. Once a behavior pattern has been associ-
`ated with a user, much more useful information can be asso-
`ciated with the user.
`
`[0044] The described embodiments can correlate the
`sequences of activity data being generated along with the
`ambient information like location, time, etc to generate daily
`patterns of the user. These daily patterns then emerge as
`behavioral patterns of the person. Behavioral patterns allow
`the system to determine how people spendtheir time, recre-
`ational and buying habits, interests of people, and pattern
`variations across demographics etc. Based on the behavioral
`patterns, how habits ofpeople vary in relationship to time and
`their physical wellbeing can be deducedorinferred.
`[0045] The described embodiment includes systems that
`can detect critical conditions based on the previous knowl-
`edge obtained by the systems for an individual and help
`prevent and aid safety situations. Additionally, the systems
`can detect carly signs of conditions that enablecarly attention
`from experts. Additionally, the systems can learn from obser-
`vation and capture behavior patterns that cannot be deter-
`mined with generic real-time monitoring. The systems adapt
`to the observed person using the data being collected through
`monitoring.
`[0046] Descriptively, an analogy can be drawn between a
`person’s motions and languages. For example person has
`minute motions,activities, daily lifestyle, behavioral patterns
`and analogous to word, sentences, paragraph, chapters and
`books. As there are words in the vocabulary, vocabulary can
`be created of elemental motions. The way sentencesare cre-
`ated with putting words into certain order, the elemental
`motions can be put into certain order and form activities.
`Activities can be put in succession along with ambient param-
`eters and form contextual activities. Series of contextual
`activities or data mining ofactivities per day/week can form
`alifestyle ofa person, which can lead to behavioral patterns of
`a person.
`[0047] Analogies include, for example, Sound—Atomic
`motion, Alphabets—-Elemental Motion, Orientation,
`Words—>Basic Movements, Sentences+Compound Move-
`ments,
`Paragraph—Contextual
`Movements,
`ChaptersActivity Pattern, Books—>Behavioral Patterns.
`[0048] A person’s lifestyle or behavior can be determined
`based on his/her movement patterns. Small movement(el-
`emental motion) patterns can be determined by the 3 dimen-
`sional acceleration signals and orientation. Examples of
`elemental motions include, for example, arm movement, sit-
`ting sedentary in the chair, getting up from the chair, standing,
`walking, running, falling. Putting the movement patterns
`basic componentsin series (varied combinations) provides a
`high degree of inference of the activity. Inference can made
`using the metrics of elemental motions and metrics of ambi-
`ent variables. Examples of ambient variables include time of
`the day, GPS location, environment, and/or physical nature.
`[0049] The following is an example of a series (sequence)
`of higher-level contextual activities includes that each
`includes a sequence of elemental motions. A first example of
`a higherlevel activity includes a user going to a grocery store.
`First, the user leaves his/her house which can include the
`following elemental motions/activities. Leaving the house
`can be detected as including the following sequence of
`elemental motions: getting up from chair, walking few steps,
`stopping briefly at the door, and walking to the car. Next, can
`
`include: identifying the user driving a car, including thefol-
`lowing sequence of elemental motions: sitting into the car,
`driving the car, car movements, parking ofcar, getting out of
`car, with the additional inputs of, for example, location and
`time ofthe day. The next step can include identifying that the
`user walked to the grocery store at the location.
`[0050] Other identified activities can include identifying
`the user getting up in the morning, by identifying the follow-
`ing sequence of elemental motions andinputs: identifying the
`time ofthe day (night), identifying sleeping, or long seden-
`tary activity, identifying going to the bathroom androlling
`over in the sleep. The activity of sitting in a car can include
`identifying the following sequence of elemental motions:
`opening the door, sitting down in the car, closing the door,
`pulling on the belt, putting on theseatbelt, and sitting back in
`the seat. The activity of driving can be identified by identify-
`ing the following sequence of elemental motions(influence,
`for example, by location and time of the day): sitting in the
`car, starting the car, the car moving, stopping the car, opening
`the car door, getting out of the car, closing the car door.
`Identification of car movement can include identifying the
`following sequence of elemental motions: going forward,
`going backward, driving, braking, turning left, and turning
`right.
`[0051] Higher-level activity identification can include, for
`example, monitoring a duration of sleep including identifying
`anumberoftimes wokeup andleaving thebed, duration away
`from bed, duration ofdeep sleep and numberof deep sleeps at
`night, start and end time of sleep. This information can be
`used to establish long term trends of sleep including deter-
`mining whenthe person (user) goes to bed, how longit takes
`to fall asleep, quality of sleep, and/or length ofsleep. If there
`is a sudden departure from the established sleep pattern, for
`example, frequent visits to the bathroom at night,this could
`indicate signs of distress or unease requiring a visit from
`caregivers or visit to physician. Additional sensors like tem-
`perature and/or blood pressure could be triggered automati-
`cally to take vital stats at certain intervals and transmitted to
`a physician for study prior to visit in such scenarios. Such a
`system can also predict the change in sleep cycles and when
`the person should move in his/her sleep. If this does not
`happen, the system could againtriggerthevital stats sensors
`and breathing motion sensor to determine if the person is