(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2011/0081634 A1
`Kurata et al.
`(43) Pub. Date: Apr. 7, 2011
`
`
`US 20110081634A1
`
`(54) BEHAVIOUR PATTERN ANALYSIS SYSTEM,
`MOBILE TERMINAL, BEHAVIOUR PATTERN
`ANALYSIS METHOD, AND PROGRAM
`
`(76)
`
`Inventors:
`
`Masatomo Kurata, Tokyo (JP);
`Makoto Murata, Tokyo (JP);
`Takashi Ogata, Tokyo (JP);
`Yasutaka Fukumoto, Tokyo (JP);
`Masanori Katsu, Tokyo (JP)
`
`(21) Appl. No.:
`
`12/868,302
`
`(22)
`
`Filed:
`
`Aug. 25, 2010
`
`(30)
`
`Foreign Application Priority Data
`
`Oct. 2, 2009
`
`(JP) ............................... P2009-230579
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`G09B 19/00
`
`(2006.01)
`
`(52) US. Cl. ........................................................ 434/236
`
`(57)
`
`ABSTRACT
`
`Provided is a mobile terminal including a movement sensor
`that detects a movement of a user and outputs movement
`information, acquires information on a building existing at a
`current location or information on buildings existing in a
`vicinity of the current location, analyses the movement infor-
`mation output from the movement sensor, and detects a first
`behaviour pattern corresponding to the movement informa-
`tion from multiple first behaviour patterns obtained by clas—
`sifying behaviours performed by the user over a relatively
`short period of time, and analyses the information on a build-
`ing or buildings and the first behaviour pattern, and detects a
`second behaviour pattern corresponding to the information
`on a building or buildings and the first behaviour pattern from
`multiple second behaviour patterns obtained by classifying
`behaviours performed by the user over a relatively long
`period of time.
`
`& BEHAVIOUR/ SITUATION ANALYSIS SYSTEM
`
`106
`
`TIME/ CALENDAR
`INFORMATION
`ACQUISITION UNIT
`
`112
`
`
`
`
`BEHAVIOUR/
`
`
`SITUATION
`
`MOVEMENT/ STATE
`RECOGNITION
`
`
`
`RECOGNITION UNIT
`UNIT
`
`
`
`RECOMMENDED
`
`
`GEO-
`SERVICE IN
`
`
`CATEGORISATION
`ACCORDANCE WITH
`
`
`UNIT
`BEHAVIOUR/ SITUATION
`
`
`
`-PERSONAL PROFILE INFORMATION PR
`
`- BEHAVIOUR] SITUATION FEEDBACK FB
`
`IIII|II
`
`_:
`
`
`
`- MAP INFORMATION MP
`- GEO CATEGORY
`INFORMATION GC
`
`APPLICATION IN
`
`ACCORDANCE WITH
`
`I
`:
`|
`LEEHAYIQLLRELTE’AIIE’LII
`
`1
`
`APPLE 1014
`
`APPLE 1014
`
`1
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 1 of 53
`
`US 2011/0081634 A1
`
`
`
`mu.xo<mommmzo_._<3._._mE:O_><Im_m.
`
`
`
`
`
`
`
`mm.zo_._.<s_mon_z_m.__u_omm._<zowmmm.
`
`
`
`
`
`SEES“:.29230820_uz_ESEm“-80n_r--m_.mm_zm§%mmm-L_o:_ unzoiacmaaosémfi:2:n1:;
`>m)\.m:2:
`
`
`
`:z:zo_._._zooomm
`
`
`
`
`
`.1||||||||||||||J
`
`ZOEZGOOME
`
`
`zoF<z_s_mm.EoE:o_><Imm
`whimszsmSo—z“zO_.F<:.:mwo—“No—
`
`
`__
`
`«2mo—
`
`m<ozm._<om2:
`
`zo_.r<_>_mon_z_
`
`:23zoEwSOo<
`
`9.0.".
`
`
`
`_>_m._.m>mm_m>._<z<zo_._.<3:wE:o_><Im_mg
`
`
`
`
`
`_____
`
`J
`
`“133%mwangfiw
`owzO_._.<s_mOn_z_
`
`
`
`1:2,moz<omooo<
`
`
`
`>moomfi<o0mm.
`
`IIIIL
`
`z_zoEo:n_n_<
`n3zo_._.<s_mon_z_n22.
`
`2
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 2 of 53
`
`US 2011/0081634 A1
`
`_______
`
`Fl
`
`w.
`
`omozmsioomm
`
`z_mo_>mmw
`
`
`
`I._._>>moz<omooo<
`
`
`
`zO_._.<:._._wE:O_><Immi__
`
`L
`
`:23zoEzooomm
`
`.95
`
`zo_._.<w_moom_._.<o
`
`:2:
`
`
`
`
`
`zoEzooommm._.<._.wszsmSO—z
`
`E30_><Imm
`
`29.5.2.5
`
`«.67.
`
`
`
`rmdzsmzop<%w_u.zoo555
`
`mm>mmw
`
`._.zm_._o
`
`m<ozm._<o
`
`zo_._.<_>_mon_z_
`
`
`
`._._z:zoEmSOo<
`
`ESE.
`
`u_
`
`2&3wa9%»?me
`
`
`TE;moz<omooo<
`owzo_._.<s_mon_z_
`
`
`
`>moom:.<o0mm.
`
`z_2922?:
`n3zo_._.<s_mon_z_m<s_.
`
`3
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 3 of 53
`
`US 2011/0081634 A1
`
`_______
`
`1
`
`r
`
`
`
`233mm335‘er
`
`
`
`:2::23zoEzooomm
`
`
`
`
`
`zOEzooommm._.<._.wFzmsmSOE
`
`
`
`zO_.r<z__>_mmhmoEDO_><Imm
`
`zo:<::m
`
`flat
`
`
`
`
`
`NEmszxmzozéaemzoo:55
`
`Hzmjo
`
`m<ozm._<o
`
`zOF<_>_mon_z_
`
`
`
`:2:zO:._w_DOo<
`
`EEC.
`
`It;moz<omooo<:2:
`
`omeszEoomm-omo
`
`z_mo_>mmmzo_.r<w_w_00m._.<o
`
`_
`
`n_<\/\_r.lllllllllllll
`
`L
`
`
`z_zo_Eo_._&<
`
`
`as.29258;9%..
`
`
`"1333,0”9%?sz8292582.
`
`
`E;moz<omooo<E0820So.
`
`___
`
`4
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 4 of 53
`
`US 2011/0081634 A1
`
`_______
`
`rl
`
`.I
`
`Qmozwfizoomm
`
`z_mo_>mmm
`
`
`
`zO_._.<3:mEDO_><ImmJ“
`
`
`
`I._._>>moz<omooo<_
`
`-Omo
`
`zOP<wEOGMH<o
`
`:2:
`
`10.".
`
`
`
`mBasiazocézoazoo55$
`
`mm>mmw
`
`EDO_><Imm
`
`zO:.<::w
`
`
`
`zo_._._zooommHEM\._.2m=>_w>o_>_
`
`
`
`
`
`.223:2:zO_._._zooomm
`
`
`
`m<ozmfi<o
`
`zO_H<§mOu_z_
`
`
`
`._._z:20Ew500<
`
`\m_>=._. Hzmjo
`
`
`
`
`
`n_s_zo_h<§mon_z_n_<s_.
`
`ow20:.<_>_mon_z_>m00m._.<oOmo.
`
`
`fizmpabmmm.:mv._\m<.z.m.mi_
`15>moz<omooo<“z_zo:<o_#_n_<_lllllllllllL
`
`5
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 5 of 53
`
`US 2011/0081634 A1
`
`_r--.om_mw_m§mmmmlL
`
`
`
`
`
`
`EQZEWEamawxmfi:2:n1:;mozémoonz_zoEmEoomEo“z_55%-80
`
`>m7N
`
`de
`
`
`
`qwig/5zocésmzzoo55$
`
`Hzmjo
`
`
`
`m<ozm._<o\msE.
`
`zo_._.<s_mou_z_
`
`EDO_><Imm
`
`zoijtm
`
`zO_._._zooom_m
`
`:23
`
`with
`
`
`
`:2:zOEzooomE
`
`szsmSO—a :23zoEm500<
`
`
`
`L
`
`n_<\/&Flllllllllllll
`
`
` EQZEW9%»?me_TE>>moz<omooo<"_
`
`2_20:5an
`m2zo_.r<_>_mon_z_n22.
`
`owzo_.r<_>_mou_z_
`
`>m00m_.<oOmo.
`
`6
`
`
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 6 of 53
`
`US 2011/0081634 A1
`
`
`
`2922?ngansémm“
`
`zoEzonmmE<5pzm2m>oz”mowzmm2952
`
`292252:aNE
`
`
`:2::2:zoEzmoomm
`
`
`
`902mm295me
`
`90.".
`
`
`
`mwaszfizocéawazoo55$
`
`Hzmjo
`
`m<ozm._<oEs:
`
`zO:.<_>_mon_z_
`
`:23zO_._._w_:Oo<
`
`
`
`
`
` 2m_§._3.a._>§§"It>>moz<omooo<":2:Wz_mo_>mmmZOF<mEOOMH<o_mowzmmzoF<oo._
`.ragéfimL$0
`
`
`vow
`
`
`n—<\L_IllllllIllllll
`
`EmEmbmflawimrmfl
`
`__.:._>>moz<omooo<“_
`z_2922?:
`
`
`n=>_zo_._.<s_m_ou_z_n_<s_.
`
`owZOF<_>_mon_z_
`
`>moom:.<oOmo.
`
`L
`
`7
`
`
`
`
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 7 of 53
`
`US 2011/0081634 A1
`
`m2:
`
`m2:
`
`AzmmEaHim:zmzméé
`
`
`
`9255:52320228a.
`
`8%;
`
`
`
`EpzéoNEE/m:EEm:zmzmsoé
`
`
`
`$5.3m“—
`
`>thz<30
`
`
`
`mo_m._._z:zO_._._zoOommwhimszs—mSOE
`
`b.67—
`
`
`
`
`
`
`
`Eszmw20:.02ZONE.SnPDO<H<oEmou_m_><>>vzo_._.<mm._moo<
`
`
`
`
`
`
`
`8
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 8 0f 53
`
`US 2011/0081634 A1
`
`FIG.8
`
`
`EXAMPLES OF MOVEMENT/ STATE PATTERN
`
`MOVEMENT PATTERN
`
`(INCLUDING POSTURE)
`
`
`
`
`1
`
`
`
`STATE PATTERN
`
`(VEHICLE)
`
`
`
`“_
`
`SUN
`
`“l—
`
`
`
`
`
`
`
`Jumping
`
`Posture Changing
`
`
`
`
`
`9
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 9 0f 53
`
`US 2011/0081634 A1
`
`FIG.9
`
`x-acc
`
`y-acc
`
`z—acc
`
`I
`
`FIFO
`
`I
`
`31000
`
`CALc:ULATE VARIANCES
`
`31002
`
`CALCULATE LARGEST VALUE
`
`$1004
`
`det
`
`31006
`
`NO
`
`NO
`
`PAUSING
`
`81012
`
`6 Y
`
`ES
`
`,
`SI 008
`
`
`
` CONTINUE
`FOR LONGER THAN
`
`
`
`
`TIME T?
`
`YES
`
`STAYING STILL
`
`$1010
`
`10
`
`10
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 10 0f 53
`
`US 2011/0081634 A1
`
`FIG.1O
`
`y-acc
`
`z-acc
`
`I
`
`BPF
`
`I
`
`81100
`
`81102
`
`I
`
`FIFO I
`I
`FIFO I
`_ $1104
`--
`CALCULATE
`CALCULATE
`AUTOCORRELATION
`\(ARIANCES
`__ FUNCTION
`
`$1106
`
`31 108
`
`$1110
`
`31 1 12
`
`$1 1 1 4
`
`$1128
`
`CALCULATE: NUMBER
`OF STEPS + freq x time
`
`NUMBER OF STEPS
`
`$1130
`
`[IIII
`
`
`
`
`
`
`CALCULATE LARGEST
`VALUE
`
`SACF
`
`var
`
`CALCULATE ENHANCED
`
`AUTOCORRELATION
`FUNCTION
`
`ESACF
`
`CALCULATE PEAK
`
`CALCULATE: freq X var
`
`SI 1 16
`
`LPF
`
`det
`
`81118
`
`$1120
`YES
`
`Dz<det<D3?
`
`$1122
`
`NO
`
`31124
`YES
`
`WALKING
`
`81126I
`
`NO
`
`RUNNING
`
`NEITHER WALKING
`NOT RUNNING
`
`51128
`
`A A
`
`11
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 11 0f 53
`
`US 2011/0081634 A1
`
`FIG.11
`
`CALCULATE MAGNITUDE
`
`S1200
`
`BPF
`
`S1202
`
`CALCULATE ABSOLUTE VALUE
`
`81204
`
`LPF
`
`S1206
`
`det
`
`$1208
`
`.No
`
`YES
`
`S1210
`
`$1212
`
`JUMPING
`
`NOT JUMPING
`
`12
`
`12
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 12 0f 53
`
`US 2011/0081634 A1
`
`FIG.12
`
`i
`
`LPF
`
`:
`
`$1300
`
`CALCULATE CHANGES IN
`DIREQTIONS OF GRAVITY
`
`31302
`
`CALCULATE MAGNITUDE
`
`S1304
`
`S1306
`
`LPF
`
`det
`
`S1308
`
`N0
`
`$1316
`
`N0 CHANGE
`IN STATE
`
`YES
`
`S1310
`
`S1312
`
` CURRENT STATE?
`
`
`SITTING
`
`STANDING
`
`SITTING
`
`STANDING
`
`$1314
`
`13
`
`13
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 13 0f 53
`
`US 2011/0081634 A1
`
`
`CALCULATE
`
`ACCELERATION IN
`GRAVITY DIRECTION
`
`
`NO
`
`81412
`
`acc
`
`
`81420
`
`A1 < 8V9
`
`
`
`
`
`$1422
`
`det
`
`81424
`
`
`
`YES
`
`< A 2?
`
`DOES NOT
`UPDATE GRAVITY
`
`31416
`
`DOES NOT
`UPDATE GRAVITY
`
`
`
`S1418
`
`NO
`
`FIRST
`OBSERVATION?
`
`NO
`
`31425.
`
`YES
`
`$1428
`
`ASCENDING IN
`ELEVATOR
`
`$1432
`
`DESCENDING IN
`ELEVATOR HAS ENDED
`
`$1430
`
`NO
`
`FIRST
`OBSERVATION?
`
`NO
`
`31434
`YES
`
`31436
`DESCENDING IN
`ELEVATOR
`
`$1440
`
`ASCENDING IN
`ELEVATOR HAS ENDED
`
`31433
`
`$1442
`
`CURRENT
`STATE?
`
`IN ELEVATOR
`
`NOT IN ELEVATOR
`
`sum
`
`14
`
`N0 CHANGE IN
`STATE
`
`14
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 14 0f 53
`
`US 2011/0081634 A1
`
`
`
`$1528
`
`SET TO ZERO
`
`
`
`81520
`
`CALCULATE:
`INTEGRATION OF h-var
`
`CALCULATE:
`INTEGRATION OF v-var
`
`81522
`
`NO
`
`sYE
`
`NOT ON TRAIN
`
`31526
`
`ON TRAIN
`
`SI 524
`
`15
`
`15
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 15 0f 53
`
`US 2011/0081634 A1
`
`FIG.15
`
`CALCULATE ANGULAR
`VELOCITY IN GRAVITY
`
`DIRECTION
`
`S1602
`
`BPF
`
`81604
`
`81606
`
`YES
`
`81610
`
`YES
`
`~o
`
`No
`
`NO TURNING
`
`81614
`
`16
`
`$1608
`
`$1612
`
`16
`
`

`

`
`
`mmméfiéas
`
`a29258;93:3
`
`,___.__.__._.__.______________.
`
`
`
`
`
`.om.crmoomzoézoEnsoog$55as
`
`
`
`Eamooz§5mTE;ommflmamm
`
`228moEa
`
`.oZSSmto$3520292582;_
`
`
`
`<29Emsmdmnzxmas
`
`
`
`\\|lllllllllllllllllllllllllII
`
`/
`
`
`
`Patent Application Publication
`
`m
`
`m
`
`US 2011/0081634 A1
`
`
`
`lllllllllllllllllllllllllllllllII\/Vnom,_..........................l\m,.Hzngozno5852S\I.
`.moEazoanpfi89nm995Egan.205585:028szu\i/MmoéozofioztbnE8205;20:59h\\NIB//
`
`
`
`
`
`
`bxOz_o.__3m“.0mm>p02_H<o_oz_maoomm>p
`
`
`
`mooo>m00mk<oOmocm:
`
`___i
`
`0.36.".
`
`
`
`mo::020._\mo3._.;<._>mom~<o_oz_zO:.<oo._
`
`0:
`
`notzasmi2_muzésmnommmssz“:2:-\32628/_$585:E0820Soam:"292308208w,/KMQEEHK“Ix
`
`.I'"
`
`17
`
`17
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 17 0f 53
`
`US 2011/0081634 A1
`
`FIG.17
`
`CAkiéJeong
`
`NAME
`
`_--
`———
`
`--—
`
`
`
`
`
`
`-——
`———
`—-_
`
`——
`
`
`
`
`
`
`
`
`
`
`_-—
`—--
`-—_
`-—-
`
`
`
`
`
`——_
`—-—
`——_
`
`
`
`1405000
`
`CHESS ROOM
`
`
`
`-_
`
`18
`
`
`
`
`
`
`
`
`18
`
`

`

`Patent Application
`
`Publ
`
`lcat
`
`ion
`
`Apr. 7, 2011 Sheet 18 of 53
`
`US 2011/0081634 A1
`
`
`
`35:55.:$585689
`
`
`
`>mowmh<o
`
`E.
`0...—
`
`29595.90
`
`19
`
`19
`
`

`

`Patent Application
`
`Publication
`
`Apr. 7, 2011
`
`Sheet 19 0f 53
`
`US 2011/0081634 A1
`
`\\ 2
`
`
`
`24200.er>moowk<oOmo
`
`20
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 20 of 53
`
`US 2011/0081634 A1
`
`
`
`
`
`xo<mommuzo_._.<3._._wEDO_><Imm.
`
`
`
`man—om;4<zommwm.
`
`zo_._.<z_s_mm._.mn_
`
`._m_QO_>_
`
`
`
`
`
`
`
`zO_._.<o:n_n_<._.zw:o>._._oO._w>,JmonthoA
`
`r|||llllllll||l\
`
`_ENE:
`
`29:55“FE:o_><Imm20.225E250minim:
`
`
`.1............:2:1.................
`zoEzooomm<20momzmw
`EDO_><Im_mFf2mm._.._.<n_whim\._.2m=>_m_>Os_
`
`«I
`
`cud—u.
`
`m29:58;_,$3256mg:
`
`{hem@fiflfibfl
`
`n_<Emsm>oznoH2852
`
`
`
`month}_>_<m_oo._.m_I
`
`21
`
`J__
`L_
`
`H\_______/
`
`21
`
`
`
`

`

`Patent App]
`
`lcat
`
`ion
`
`lcat
`PubF
`
`ion
`
`Apr
`
`7,
`
`2011 Sheet 21 0f 53
`
`US 2011/0081634 A1
`
`
`
`
`
`5.0—".
`
`
`
`
`
`
`
`02$me@255;Vac;52m$2swmwmwo:mmmEomofififiw.,EEv32,50%£22flmmmfio
`
`
`
`
`
`
`
`IIIIIIIIIIII§§§
`
`émjéE<.23920858
`
`Illlllllllllgggéw
`
`
`
`:Eamo:immzmov
`
`Illlllllllllgéfi
`
`IIIIIIIIIIIIgéé
`
`mrFm60:8EmwmazF.szémmoz:
`
`IIIIIIIIIIII§§E§N
`
`
`
`IIHIIIIIIIIIgage
`
`
`
`IIIIIIIIIIIIExgg
`
`IIIIIIIIIIII5g3
`
`IIIIIIIIIIII
`IIIIIIIIIIIIEif
`
`
`
`IIIIIIIIIIIIEoér
`
` IIIIIIIIIIII<§§§V
`
`605mEEEEME
`
`ooowomm
`
`ooomomm
`
`ooovomm
`
`22
`
`22
`
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 22 0f 53
`
`US 2011/0081634 A1
`
`FIG.22
`
`810
`
`812
`
`814
`
`SELECT COMBINATIONS OF
`
`SCORES IN ACCORDANCE
`
`WITH GEO CATEGORY
`
`SELECT SCORE MAPS IN
`
`ACCORDANCE WITH BEHAVIOUR!
`
`CALENDAR INFORMATION
`
`SITUATION PATTERN
`
`SELECT SCORE MAP IN
`
`ACCORDANCE WITH TIME/
`
`23
`
`23
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 23 of 53
`
`US 2011/0081634 A1
`
`
`
`r1[LEémavmtwuuwalml.TamL_rx_§>.___rm
`3%4444444
`fillIIJ1J7IIJ“.26u"25a:a::7_Em:_“mzuagmi_Fl
`
`
`_
`
`1
`
`L__J
`_
`
`_n
`
`Fl
`
`
`
`
`
`5852EmmaEmgmsoé
`
`SEES:no
`
`mud—u.
`
`EMEE29:3528.21%
`
`
`
`
`zo_.r<z_s_mmEQzmmE/EzOF<3tm
`
`
`
`EDO_><Imm”—0mezsfi
`
`229:5\Eng“;Jgm:53J.8\2°55
`
`
`
`$259:25ELEV?gagaw
`
`24
`
`
`
`xmmgNIH“.0><oESEOhozEzOmwmmmoomzw-22wwmsomon25—mmoom61-5v.A.
`
`24
`
`
`
`
`

`

`Patent Application Publication
`
`2
`
`US 2011/0081634 A1
`
`\\dzEgm;
`
` \x5m:_:5ILI.__saga9:"29535iulmifllfllflIfllllflsou".Emmoom_.22$08
`
`
`$32528888.02”
`
`
`
`cud—u.
`
`
`
`
`
`5585:Sooza:8:52292535295959056%
`
`_.\_W7.,\32wuUm\oszom;"..A.a
`
`.onm.
`
`3832w8;-Ilul._«w559%4_292%iii-IIIHIIIIHIEOL5$58_.mmoom._n22mmoomn\mmzémoz
`
`
`
`
`aa:M3:35;
`
`4__
`
`E0820 So80SoSo80So
`55'NHHIHHIH
`
`
`
`
`
`
`
`
`E0820$5856$855E0820E0820E0820E0820E0820$0856$0820
`see30959$09$0959:09zomfimmgo
`
`
`25
`
`\
`
`i__EEIJJIIIHIIIIIIIIn".a?mmoom\/\ammoB.mwsémoz5882.on
`
`25
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 25 0f 53
`
`US 2011/0081634 A1
`
`mN.0_u_
`
`MFEmaumzfioz
`
`25282
`
`Hvz+mznNZ“mo:+
`
`qwz+mzu«2“no:9588
`
`5&8SS-u+oz
`
`
`
`
`
`amigo.-u.0290152Iom<mm
`
`—————_———_——_———_—J
`
`
`
`2mm._.._.<n_\zo_k<3._._mRDO_><Immoz_Io._.<s__
`
`”E28u:4no:u858oz_zm<m:mzioémeam.
`
`_-owzEEEEfim:zmzmszE58-mzso:
`
`
`a:Emaumz“mo:3:23258>toqqsmvs_>m
`”EEmanoz“maczo_mmmmommm<mz_._mashed“.whimPzwsm>oz
`
`
`
`
`
`
`K2462umzac:zoammmommm<m22202E559”E550
`$288.gm
`
`
`
`mOH0m>mm2<mmn+0._.:n_z_
`
`
`
`<._.<n_mm10<m._.
`
`
`
`Ammo—2zo:.<z_s_mm:mo
`
`
`
`mokom>mmakmm
`
`
`
`”<55xmmz.n_o-><om2:
`
`.242u02so:fizz/SommDEE
`
`_onzmo>mzo_._.om._mm
`
`
`Hm_02m_m_mu_n__n_montmuzgDeath”:
`
`
`
`R:O_><ImmHommmoo
`
`
`
`m._m<_m<>wszmwwm
`
`
`
`2mm._.._.<n_zo:.<3.:m
`
`me;x0m
`
`
`
`EEOOHQI>moom:.<oOwo
`
`$3853momzmm
`
`
`
`mooo>m00mk<oOmo
`
`26
`
`
`
`
`
`oz_zm<m.#mz_Io<s_0253Q01522953302029.249?ermmOOw
`
`
`
`
`
`
`
`26
`
`

`

`Patent App]
`
`lcat
`
`ion
`
`lcat
`PubF
`
`ion
`
`Apr
`
`7,
`
`2011 Sheet 26 of 53
`
`US 2011/0081634 A1
`
`mNdE
`
`mic:a0235:a
`
`0532N8
`
`._.2m=>_m_>Os_
`
`H00;20
`
`z_<m._.
`
`
`
`:é5mmmohwmozm_2m>zoo.%Ill
`
`
`
` mmm1.5-zo_m_>m..m._.now
`
`
`
`m._o>o_m_
`
`mOH<>m4m
`
`mHmOmm
`
`ZOIH<m<2
`
`”:00
`
`j<mmw<m
`
`wmmxhoa
`
`$95EmmmEoa mmmIHO
`
`
`
`27
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 27 0f 53
`
`US 2011/0081634 A1
`
`FIG.27
`
`BEHAVIOUR/ SITUATION RECOGNITION FLOW
`
`BEHAVIOUR OF USER
`
`320
`
`ACQUISITION OF SENSOR
`DATA BY SENSOR
`
`322
`
`MOVEMENT/ STATE DETECTION,
`GEO-CATEGORISATION
`
`324
`
`DETECTION OF BEHAVIOUR!
`SITUATION PATTERN
`
`526
`
`NOTIFICATION TO APPLICATION
`
`328
`
`28
`
`28
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 28 0f 53
`
`US 2011/0081634 A1
`
`FIG.28
`
`(A: RULE-BASED DETERMINATION)
`
`826: DETECTION OF BEHAVIOUR/
`
`SITUATION PATTERN
`
`USER PROFILE IS
`REGISTERED?
`
`
`
`
`WORKPLACE?
`
`
`LATITUDE/
`LONGITUDE IS HOME OR
`
`
`
`
` SELECT SCORE MAPS
`
`CORRESPONDING TO
`
`SELECT, TAKING PROFILE
`HOME] WORKPLACE
`SELECT COMBINATION OF
`INTO CONSIDERATION,
`SCORES IN ACCORDANCE
`COMBINATION OF SCORES
`
`IN ACCORDANCE
`WITH GEO CATEGORY
`
`
`WITH GEO CATEGORY
`
`
`NARROW DOWN TYPES
`
`
` S112
`
`
` SELECT BEHAVIOUR/
`
`
`
` $114
`
`
`
`
`OF SCORE MAPS BASED
`
`ON MOVEMENT/ STATE
`
`PATTERN AND TIME/
`
`CALENDAR INFORMATION
`
`SITUATION PATTERN
`
`CORRESPONDING TO
`
`HIGHEST SCORE FROM
`
`COMBINATION OF SCORES
`
`OF TYPE OBTAINED BY
`
`NARROWING DOWN
`
`
`
`29
`
`29
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 29 0f 53
`
`US 2011/0081634 A1
`
`FIG.29
`
`(B: MACHINE LEARNING-BASED DETERMINATION)
`
`826: DETECTION OF BEHAVIOUR/
`
`SITUATION PATTERN
`
`
`
`
`USER PROFILE IS
`
`REGISTERED?
`
`SELECT
`
`
`SELECT DETERMINATION
`MODEL OBTAINED WITH
`
`USER PROFILE TAKEN INTO
`CONSIDERATION
`3125
`
`
`GENERAL-PURPOSE
`DETERMINATION MODEL
`
`INPUT INFORMATION TO
`
`SELECTED DETERMINATION
`MODEL AND DETECT
`
`
`8128
`
`
`
`
`MATCHING BEHAVIOUR/
`
`SITUATION PATTERN
`
`OUTPUT DETECTED
`
`
`BEHAVIOUR/ SITUATION
`
`$130
`
`PATTERN
`
`30
`
`30
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 30 0f 53
`
`US 2011/0081634 A1
`
`FIG.30A
`
`(C: COMBINATION OF RULE/ MACHINE LEARNING)
`
`$26: DETECTION OF BEHAVIOUR/
`
`SITUATION PATTERN
`
`USER PROFILE IS
`REGISTERED?
`
`
`
`
`
`LATITUDE/
`LONGITUDE IS HOME OR
`
`WORKPLACE?
`
`
`
`
`SELECT SCORE MAPS
`CORRESPONDING TO
`
`
`HOME/ WORKPLACE
`
`
`
`SELECT, TAKING PROFILE
`
`SELECT COMBINATION OF
`INTO CONSIDERATION,
`
`
`SCORES IN ACCORDANCE
`COMBINATION OF SCORES
`
`
`WITH GEO CATEGORY
`IN ACCORDANCE
`
`
` WITH GEO CATEGORY
`
`
`NARROW DOWN TYPES
`OF SCORE MAPS BASED
`
`ON MOVEMENT/ STATE
`
`
`
`
`CALENDAR INFORMATION
`
`
`
`
`
`SELECT BEHAVIOUR/
`
`
`
`
`
`PATTERN AND TIME/
`
`8152
`
`SITUATION PATTERN
`
`CORRESPONDING TO
`
`$154
`
`HIGHEST SCORE FROM
`
`COMBINATION OF SCORES
`
`OF TYPE OBTAINED BY
`
`NARROWING DOWN
`
`
`
`31
`
`31
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 31 0f 53
`
`US 2011/0081634 A1
`
`FIG.SOB
`
` 8156
`
`SCORE OF
`
`
`SELECTED BEHAVIOUR/
`
`
`YES
`SITUATION PATTERN IS A
`
`
`SPECIFIC VALUE
`OR MORE?
`
` USER PROFILE IS
`
`REGISTERED?
`
`
` SELECT DETERMINATION
`
`DETERMINATION MODEL
`
`MODEL OBTAINED WITH
`
`USER PROFILE TAKEN INTO
`
`CONSIDERATION
`
`
`
`SELECT
`
`GENERAL-PURPOSE
`
`
`INPUT INFORMATION TO
`SELECTED DETERMINATION
`
`
`MODEL AND DETECT
`3164
`MATCHING BEHAVIOUR/
`
`SITUATION PATTERN
`
`
`
`
`
`OUTPUT SELECTED OR
`
`DETECTED BEHAVIOUR/
` SITUATION PATTERN
`
`8166
`
`32
`
`32
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 32 of 53
`
`US 2011/0081634 A1
`
`
`
`BEDOk.._OzOEfizmmmmE
`
`
`
`zO_H<EmOn_z_hOmm
`
`zO:.<_2w_O..._z_nozo_.r<._.zmwmmm
`
`mOImm_zm>DOwzo
`
`zO_H<_zmou_z_nozo_.r<._.zmmmmn_
`
`HOawEEDOHmez20
`
`.ESEDOO“.0ZO.._.<Hzmmmmn_
`
`ZO_H<S_mOn_z_
`
`mjmafiwsEn5szfizmmmmn.
`
`
`
`Fzm2m>02kxmzmo”.
`
`622.23
`
`GZEmOIm
`
`27$:
`
`oz_x._<>>
`
`._<m_>_
`
`
`
`moSo22w:
`
`0222‘s
`
`
`
`mo;oz_o<mI
`
`2925
`
`
`
`02_>_mm<.5022
`
`
`
`HOmw._.m_mDOHk<
`
`
`
`5:59.D230m<
`
`bOmm
`
`
`
`ozmmxkozzom<
`
`oz_mmm:._o_mno
`
`
`
`mEOI><>>.
`
`no828208
`
`20.258;20.23.52
`
`20.535Enossmmv
`
`53%zofimmo
`
`62:2:zoEorEozV
`
`OHOED>T<K<Inxowzwm
`
`22:23
`
`Tokozimzoxi2Exp
`
`O
`0..
`(D
`1—
`
`._<m_>_
`
`$58028%
`
`ozA§<I§oozmmv
`Ram.51
`
`O
`.0.
`V
`1—
`
`OC
`
`)
`..
`CO‘—
`
`3580230;;
`
`02.3336
`
`O
`0..
`N
`‘—
`
`IanA
`
`mSszomV
`
`652.;
`-mzovzx
`
`TaxaziezEm;
`
`goggmzoxé
`
`ozaoms
`
`oz_mmwt._o_w
`
`O
`9.
`O
`Q'-
`
`O
`9.
`r-
`1—
`
`33
`
`
`
`
`
`
`
`zo_._.<o:n_n_<mSQmIomTE>>zO_._.<._mm_moozOQmw<m001.52zo_._.<Hzmwm_mmzo_._.<s_mon_z_m>_._.omn_n_m
`
`
`
`
`
`
`
`
`
`5.0.".
`
`33
`
`
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 33 of 53
`
`US 2011/0081634 A1
`
`
`
`
`
`zmmEmm2021,3200n_O><._n_w_ov
`
`_\R
`
`
`
`
`
`E30_><Im_m._<3._.o<oz<mDO_><Immomgsomrow
`
`and."—
`
`
`
` \\\:zma858:8
`
`V
`
`\
`
`023%HO:
`
`\52mm2;
`
`
`
`\._<m=>_oz_><I
`
`
`
`zo_k<:.:mEDO_><Imm
`
`
`
`ZOFQmEQ2mm._.._.<n_
`
`65:2-m2921
`
`.122;me
`
`652:»-mzov2:
`
`ozmmeQw
`
`ozaoms
`
`uzmmmEQm
`
`$58ozaoms
`
`._<m_s_
`
`5:8ozzoms
`
`oboziTééaxoozmm
`
`mam
`
`552$
`
`Tohozimzoxiz_<w:.
`
`02QO.6:
`
`Aé<zax82me
`
`@5859
`
`34
`
`34
`
`
`
`
`
`
`
`

`

`P
`
`m
`
`3
`
`3
`
`US 2011/0081634 A1
`
`
`
`
`
`zo_._.<_>_mon_z_zO_._.<.EOn_mz<mHijsm.
`
`
`
`EnosémmQm8.9".
`.m2_2:m555@2558;20.258;555m_m:<z<29:35
`
`.m.5
`
`29258;m@3255mg:w_||:2:zo:_w58<m_n22mmoom
`
`
`
`
`.mNow
`
`mom
`
`
`
`A.EDO_><Imm
`
`__we“NS__
`
`___
`
`m:2:_s20.22%820"I__m80nX:n7,$55.:
`
`
`
`
`:2:$555E82:zoEzooommEfiw:zmzmézmowzmm20:59:2:o:HHr.:2:295.8%589:maosémmv:2:zoEzooomm_I__Mm:o_>$m_m295.2"20.225_
`
`
`
`
`
`
`35
`
`
`
`A0.as.292538;93.
`
`5E0820Bo.
`
`oozo_._.<s_mou_z_
`
`mao_><Iwm
`
`
`
`:2:zO_._.<o_u=m_m_>
`
`vow
`
`
`
`zo_.r<s_mon_z_Hmzmzoo.
`
`oE
`
`Esomrom
`
`
`
`29259.2.Hzm>m.zofi<ojmm<
`
`:23>5mm5:2:m0<mOHm
`
`
`35
`
`
`
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 35 of 53
`
`US 2011/0081634 A1
`
`3.6.".
`
`205%E55:no
`
`
`
`mo$5530zmmzfim
`
`
`
`$92325E5235292582.
`
`20.28.:
`
`x
`
`no$530<22:;
`
`
`fl.2952,»:GE..W\‘va.
`2953335%VVW\,\'AVl/x.Eaméoma.1i\
`
`
`
`
`AzmmEE20535
`
`ansémmnoE05:_
`
`
`
`H32:22E:52.535
`
`m2:
`
`
`
`,.um.
`
`éEmaomE530j$5..
`
`22‘HW.W..,.,,.I.W\\\\W“.Ano32.80/;EozEEmao
`
`,VZW’.‘20:59
`‘x.52:<2,
`
`szmfimao
`
`0mmhwao
`.><._.w“51.525
`
`
`
`
`
`m30_><Im_mz_<s__m__>=._.
`
`mm53whzmhzoo
`
`
`
`9.2.2529
`
`zOEEQOzoo
`
`.zOF<oo._
`
`36
`
`1I?!6.
`
`36
`
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 36 0f 53
`
`US 2011/0081634 A1
`
`FIG.35A
`
`PRE-PROCESSING FOR
`BEHAVIOUR VERIFICATION UNIT
`
`I" ________________ "T
`_____ _{I
`SENSOR DATA,
`I
`I LOCATON INFORMATION }
`
`DETECT MOVEMENT/
`
`STATE PATTERN FROM
`
`SENSOR DATA
`
`CALCULATION GEO
`
`CATEGORY INFORMATION
`
`(CODE, HISTOGRAM) FROM
`LOCATION INFORMATION
`
`INFORMATION
`
`DETECT BEHAVIOUR/
`
`SITUATION PATTERN FROM
`
`MOVEMENT/ STATE PATTERN
`
`AND GEO CATEGORY
`
`8202
`
`3204
`
`8206
`
`$208
`
` SCHEDULE
`NO
`
`INFORMATION
`EXISTS?
`
`8212
`
`
`
`
`
`PREDICT NEXT BEHAVIOUR
`SCHEDULED EVENT AND
`BASED ON HISTORY OF
`
`
`NEXT SCHEDULED EVENT
`
`
`
`BEHAVIOUR/ SITUATION
`
`
`BASED ON SCHEDULE
`PATTERNS
`
`
`
`
`INFORMATION
`
`
`CHECK CURRENT
`
`
`
`TO PROCESSING BY BEHAVIOUR
`
`VERIFICATION UNIT
`
`37
`
`37
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 37 0f 53
`
`US 2011/0081634 A1
`
`FIG.35B
`
`PROCESSING BY BEHAVIOUR
`VERIFICATIONUNIT
`
`I
`CURRENT LOCATON’
`I
`__
`x
`BEHAVIOURNEXT
`‘" “I
`L__59'iE_D_UEEP_E_V§IiT__J
`
`5222
`
`
`
`CURRENT
`No
`
`LOCATON/ BEHAVIOUR AND
`SCHEDULED EVENT
`
`MATCH?
`
`
`MATCH NEXT
`SCHEDULED EVENT?
`
`3232
`
`
`
`
`MATCH PAST
`CURRENT
`
`
`SCHEDULED EVENT?
`SCHEDULED BEHAVIOUR IS
`
`YET TO BE OVER?
`
`
`8224
`
`NO
`
`
`
`
`
`ACQUIRE BEHAVIOUR
`
`
`PREDICTION DATA FOR
`VICINITY OF CURRENT
`
`LOCATION
`
`
`
`
`ACQUIRE BEHAV|OUR
`PREDICTION DATA FOR
`
`PLACE LIKELY TO BE
`
`
`VISITED NEXT
`
`CHECK TYPE OF
`SCHEDULED BEHAVIOUR
`
`
`(GOURMET,
`
`
`ENTERTAINMENT, VEHICLE)
`
`
`
`ACQUIRE NEXT
`SCHEDULED EVENT
`
`ACQUIRE CURRENT
`SCHEDULED EVENT
`
`ACQUIRE PAST
`SCHEDULED EVENT
`
`8240
`
`
`
`
`
`ACQUIRE INFORMATION IN
`ACCORDANCE WITH TYPE
`
`3242
`
`DISPLAY ACQUIRED
`INFORMATION ON SCREEN
`
`8244
`
`OF SCHEDULED BEHAV|OUR CASE WHERE THERE IS
`OF APPLICATION
`
`SCHEDULE INFORMATION
`
`38
`
`38
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 38 0f 53
`
`US 2011/0081634 A1
`
`FIG.35C
`
`CASE WHERE THERE IS NO SCHEDULE INFORMATION
`
`PROCESSING BY BEHAVIOUR
`VERIFICATION UNIT
`
`ACQUIRE BEHAVIOUR
`
`I— ______________ _|
`_______: CURRENT LOCATON/
`I
`:
`BEHAVIOUR. NEXT
`I
`L_S_C_H_EPI_JEEQ EVEN};
`
`PREDICTION DATA FOR VICINITY
`
`OF CURRENT LOCATION
`
`ACQUIRE BEHAVIOUR
`
`PREDICTION DATA FOR PLACE
`
`LIKELY TO BE VISITED NEXT
`
`
`
`CHECK TYPE OF SCHEDULED
`
`BEHAVIOUR (GOURMET,
`ENTERTAINMENT, VEHICLE)
`
`ACQUIRE INFORMATION IN
`
`ACCORDANCE WITH TYPE
`
`OF SCHEDULED BEHAVIOUR
`
`DISPLAY ACQUIRED
`
`INFORMATION ON SCREEN
`
`OF APPLICATION
`
`8252
`
`8254
`
`$256
`
`8258
`
`8260
`
`End
`
`39
`
`39
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 39 of 53
`
`US 2011/0081634 A1
`
`mad—u.
`
`
`
`”one. zmwmom><._n_m_ozmmmomzo_._.<mhw_omm
`
`
`<wscmz“.0._.m_._Home.<22E:”.0._.m_._
`
`
`
`
`
`
`
`
`mZOI><>>20Pm:m30_><Im_m
`
`
`
`Ego:><>>20n5onOHmoz__.:.l
`
`
`
`Fm:oz_n_n_on
`
`
`
`
`
`E20:><>>20Pm:m:o_><Im_m
`
`
`
`
`
`.mwom«3:2n2.25200kin—ma
`
`$0220.?
`
`wmothOOmOH00I
`
`
`
`29:35Enos/Ema
`
`
`
`5:mezopowmo
`
`
`
`Uw._.Zm_._.zOo><._n_m_n_
`
`S\m\moom
`
`”mzjoamo
`
` XOOm<mmomonw_.zm_.zoo><._n_m_o
`
`oz_m<Iwnomat.
`
`
`
`
`
`mgu%002.”.0mDO_><Imm
`
`wz_v_m0>>mgn%
`5_SE“.
`
`
`
`macoz_.r<m_
`
`
`
`s_<m._.d<mmm<m
`
`omkfimmimoi
`
`wh<2xm0>>
`
`901mm301
`
`mmmOIQ
`
`Oz_n_n_ozm
`
`mz_>>m_>
`
`40
`
`40
`
`
`
`
`
`
`
`
`

`

`Patent Application Publication
`
`9
`
`Apr. 7
`
`2011 Sheet 40 of 53
`
`US 2011/0081634 A1
`
`oao._. had."—
`ozEaOImu,.mEOI><>>zo.Am._o>o_m_m<o.me.22va02302.5:ozEmOIm
`
`
`
`
`
`
`2mmt<n_ZOE/5H5E:o_><Im_mw._.2m_._.zoo><._n_w_o
`
`
`
`
`oz_>>m=>>._..mmmoroQAOIMmDOIoz_>>m__>EéoommDmamoomm
`
`
`
`
`
`
`
`
`
`wz_v_._<>>.Cbom202v0230—2.5:mo<mwm2
`
`
`
`
`szmOImmm._..._<.ozEmOIm9.2300”?99.595:
`
`
`
`33m.z_<m._.vwz_>os_oz_zm._.w_._Iw_._02m_
`
`
`
`
`
`._<m=>_mm._.n_<J<ms_xomrom__w_o._<o
`
`
`
`
`
`
`
`
`
`41
`
`41
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 41 0f 53
`
`US 2011/0081634 A1
`
`wad—u.
`
`
`
`
`
`
`
`2mpw>mwz_n__>omn_zo:.<s_mon_z_2m._.w>wm_m>4<z<zOfi<3tmE:o_><_._mma
`
`
`
`
`
`
`
`
`_n_m:2:$555"mgn-n:23_8n"zoEmEoomEo"mmimw
`
`\zofiépwammone.
`
`zo:.<o_.._FOz
`
`zO:.<o:mn_<
`
`
`
`n=>_zo_H<§mOn_z_n22.
`
`00zo:.<s_mou_z_
`
`>moom:.<oOmo.
`
`Em
`
`_—n22mmoow
`
`«2we
`
`m<mzm4<om2:
`
`zO:.<§mOn_z_
`
`:23zoEmSOo<
`
`E52232__-80mx:m8822:;no:__r.................L:2::2:zoEzmoomm"__
`
`___Enosémmmeu_
`
`
`
`«2flzoEzooommmemzmm295.2_m2;:zmsmB:_29:35
`
`
`___
`
`42
`
`42
`
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 42 0f 53
`
`US 2011/0081634 A1
`
`and."—
`
`
`
`ZOEEmEOZO:.<o_u_:.Oz\zo_._.<E.m_omE008.
`
`
`
`
`
`zoozazmmmoEmmi
`
`
`
`
`
`2920::02mzsmmmoDEEE5:52
`
`
`
`no5526asosémm
`
`$3$15macaw9:52
`
`2920::on
`
`E88E52
`
`
`
`39m:Emméz
`
`:7588no
`
`43
`
`
`
`
`
`
`
`Hzm2m0<z<2cookAZO_H<mHm_OmmVzO_.F<me_om_m
`
`43
`
`
`
`

`

`Patent Application Publication
`
`4a
`
`US 2011/0081634 A1
`
`OTOE
`
`
`
`
`
`zoEEmEOZOF<0E_._.OZzo_._.m.dn__>_oo\ZO:.<E.w_0mmzo_._.m.E_>_oocook
`
`
`
`
`
`
`
`
`
`
`
`3mAMwmmx\m4mm2=iwmmm20p>gkozm20:55.522052,58
`
`_n_,
`
`__8mhv8
`
`m\\...................1/s..1__mnan,nn:2:88nozoEEmammmu“Hzmzmo<z<zoook
`
`Azoc<mkgommzozm4mzooA“zozm4mzoov
`
`44
`
`zoEEmamm20:;EE2.
`
`
`
`mewzo:<2l:ozzo:mzzooozymmoH>EHoz
`
`Emwzumm2:zoEorEozgnome8E52
`
`44
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 44 0f 53
`
`US 2011/0081634 A1
`
`momo<mwm2one.
`
`
`
`zOF<m_zomIoz>wcoo»
`
`:2:
`
`N5
`
`
`
`zo_._.<E.m_omEeach
`
`.2z:
`
`
`
`29.5.5200cook
`
`
`
`:2:zO:.<m.rw_om_m
`
`3.0.".
`
`
`
`:23Hzm2m6<z<2cook
`
`vow :2:
`
`
`
`
`
`moHzmzmo<z<§anomoEmma
`
`
`
`mowhimm30_><Im_mwkmm:
`
`
`
`zo_._.<o_n=.52one.
`
`
`
`zO_._._._.m_n_m_m-zOz\zo_.:.rm_n_mm.
`
`
`
`m30_><ImmMmooEH.
`
`
`
`mum:mmmmmmonz.
`
`mw<mmm=2.
`
`
`
`EmizooH02\mkmdzoo.
`
`Q.
`
`45
`
`
`
`zO_.E.m_n_mm-zoz\zo_._._._.mmmm.
`
`
`
`mhmfiEoo.52E55200.
`
`
`
`m30_><Im_mmmowip.
`
`mo<wwm§.
`
`n:.
`
`45
`
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 45 0f 53
`
`US 2011/0081634 A1
`
`FIG.42A
`
`REGISTRATION UNIT
`
`OPERATION OF ToDo
`
`
`
`USER EXISTS
`
`3302
`
`REGISTER DB
`
`S304
`
`
`
`
`
`TO OPERATION OF ToDo
`
`SYNCHRONISATION UNIT
`
`ID FROM WHICH REGISTRATION L
`NOTIFICATION IS RECEIVED
`
`USER/ GROUP
`
`MANAGEMENT DB
`
`ToDo MESSAGE DB
`
`46
`
`46
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 46 0f 53
`
`US 2011/0081634 A1
`
`FIG.4ZB
`
`ID FOR WHICH COMPLETION L
`
`NOTIFICATION IS RECEIVED
`
`|
`
`
`
`OPERATION OF ToDo
`
`COMPLETION
`
`REGISTRATION UNIT
`
`REGISTERED ID?
`
`ToDo MESSAGE DB
`
`SET STATUS OF MESSAGE
`
`TO "COMPLETE“
`
`S314
`
`
`
`
`
`
`TO OPERATION OF ToDo
`
`SYNCHRONISATION UNIT
`
`47
`
`47
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 47 0f 53
`
`US 2011/0081634 Al
`
`FIG.42C
`
`E ToDOo RECENTLY L___
`I
`UPDATED ToDo
`:
`
`OPERATION OF ToDo
`SYNCHRONISATION UNIT
`
`
`
`USER/ GROUP
`
`MANAGEMENT DB
`
`S322
`
` THERE IS ToDo
`
`FOR WHICH NOTIFICATION
`PROCESS IS NOT
`
`PERFORMED?
`
`ToDo MESSAGE DB
`
`SELECT ToDo THAT
`
`IS NOT NOTIFIED
`
`3324’
`
`
`
`
`TO OPERATION OF ToDo
`
`NOTIFICATION UNIT
`
`
`
`48
`
`48
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 48 0f 53
`
`US 2011/0081634 A1
`
`FIG.42D
`
`E
`I
`
`I ___
`TODO THAT
`IS NOT NOTIFIED I
`____________ _.J
`
`
`
`OPERATION OF TODO
`NOTIFICATION UNIT
`
` BEHAVIOUR
`
`USER'S BEHAVIOUR
`
`STATE DB
`
`USER/ GROUP
`
`MANAGEMENT DB
`
`STATE OF USER MATCH
`
`TRIGGER OF TODO?
`
`
`
`
`NOTIFY USER OF T000
`
`8334
`
`49
`
`49
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 49 of 53
`
`US 2011/0081634 A1
`
`EN
`
`Hz<m3<._.mm_m
`
`
`
`.2zofi<>mwmmmH>n=._.oz
`
`med—u.
`
`
`
`
`
`
`
`
`2mm.E<n_zo_.—<:._._mE:O_><Immoz_Io._.<s_I._._>>mum:o._.coopwz_>u=.52n5mm<o
`
`
`
`
`
`
`
`Fomjmm@2502\h
`
`022m0>>
`
`‘
`
`x.\
`
`AH”.J!EésimmmEzo:<>mmmmm
`
`
`
`022m0>>\Oz_>0_>_nZOEQZOU
`
`
`
`NwHZm_._.zoozo_._.<oI:.Oz
`
`50
`
`50
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 50 of 53
`
`US 2011/0081634 A1
`
`Sud—u.
`
`02;..EOZnomm<o
`mmmsSzzoszzmmmowmmmnth._._<9.008.
`
`
`
`
`
`
`mzmmtxizo_._.<3:mE:O_><ImmOzfobxéI._._>>mmmm:n5
`
`
`
`
`
`
`
`mzo>mm>mOH>..=.roz
`
`>momzobsz
`
`5<I23.;mmoz
`
`
`Aflmfimon.Ow._<my?.05.mmmmfimé
`
`
`
`j<mooz__._o._.<s_.mmmZmEmzono0250522MA;
`
`02305.w
`
`025.20%
`
`
`
`
`
`nw._.ZMHzOoZO_H<o_u_FOz
`
`
`
`mmzzEmo“..50OOw.._.m._
`
`
`
`02502nzoEono
`
`51
`
`51
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 51 0f 53
`
`US 2011/0081634 A1
`
`i§2m88____
`
`
`
`02_n_n_OImoz_n_n_OIw
`
`
`
`NFzo_._.<o:n_n_<zo_._.<03n_n_<
`
`oz_n.n_OIw
`
`mzo:.<o_._n_n_<
`
`med—u—
`
`
`
`zo_._.<oo._wm500<
`
`
`
`
`
`<._.<n_mowzwm.zo_._.<_>_mon_z_
`
`szn—OIm
`
`
`
`mzoF<oan<moz§m<mm
`
`
`
`EDO_><ImmHomEo
`
`
`
`szF/Ezo_._.<3._._m
`
`52
`
`52
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 52 0f 53
`
`US 2011/0081634 A1
`
`Syd—u.
`
`
`
`
`
`
`
`2m._.m>w405.200><._n_m_oZmfiw>mw_w>._<z<zo_.r<:._._wEDO_><Imma
`
`
`
`
`
`
`
`2m
`
`
`
`—_n25.mmoow
`
`
`
`N:oo—
`
`m<ozm4<oE22.
`
`zo_.r<s_mon_z_
`
`
`
`:2:zoEmSOo<
`
`
`
`
`
`._._z:53me><._n_m_o
`
`:23Jomhzoo
`
`vow
`
`N9q
`
`
`
`n__>_zo:.<_2mon_z_n2:.
`
`00zO:.<_>_mou_z_
`
`>m00m._.<oOmo.
`
`:2:_
`
`
`
` 29230820“nommu__ux:no:nH:2:_.zoEzooomm:2:zoEzooomm_momzmm2952_Him:zmzmSoz_29225ume"Ezosémmmenu__
`
`
`
`
`
`53
`
`53
`
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 7, 2011 Sheet 53 of 53
`
`US 2011/0081634 A1
`
`>44<zmMHXm
`
`omhomzzoo
`
`mo_>mn_
`
`J_<>Os_mm
`
`0283083
`
`235%;
`
`2958232200
`zoFomzzOo
`
`Hmom
`
`
`:2:
`
`:23magmoéoESE8
`
`._.3n_z_
`
`:z:
`
`0%mao
`
`m—
`
`o5mac—mm
`
`mam._<zmm_.Xm
`
`moEmmHz.
`
`54
`
`c
`
`omvow8
`
`N
`
`Ned—u—
`
`mom
`
`
`
`54
`
`
`
`
`
`
`
`
`

`

`US 2011/0081634 A1
`
`Apr. 7, 2011
`
`BEHAVIOUR PATTERN ANALYSIS SYSTEM,
`MOBILE TERMINAL, BEHAVIOUR PATTERN
`ANALYSIS METHOD, AND PROGRAM
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`[0001]
`[0002] The present invention relates to a behaviour pattern
`analysis system, a mobile terminal, a behaviour pattern analy-
`sis method, and a program.
`[0003]
`2. Description of the Related Art
`[0004] A technology for installing a motion sensor on a
`mobile terminal such as a mobile phone and for enabling
`automatic detection and recording of a user’s behaviour his-
`tory is gaining attention. For example, JP-A-2008-3655 dis-
`closes a technology for detecting a walking movement, a
`running movement, a movement of turning left or right and a
`still state by using a motion sensor such as an acceleration
`sensor and a gyro sensor. This patent document describes a
`method of calculating a walking speed, a walking power and
`a rotation angle around a gravity axis from output data of the
`motion sensor, and detecting the walking movement, the run-
`ning movement, the movement of turning left or right and the
`still state by using the calculation result. Furthermore, this
`patent document describes a method of detecting a user’s
`behaviour pattern by statistical processing which has, as
`inputs, the pattern of the movement or the state such as the
`type of the movement or the state, the duration of the move-
`ment or the state and the number of times of the movement.
`
`SUMMARY OF THE INVENTION
`
`[0005] Using the method described above, a behaviour pat—
`tern such as “slow walking” and “busy movement” can be
`obtained as time series data. However, the behaviour pattern
`obtained by this method mainly expresses a relatively short-
`term movement or state of a user. Accordingly, it is difficult to
`estimate, from a behaviour pattern history, a specific content
`of a behaviour such as “today, went shopping at a department
`store” or “yesterday, had a meal at a restaurant in a hotel.” The
`behaviour pattern obtained by using the method described in
`the patent document described above is an accumulation of
`behaviours performed in a relatively short period of time.
`Also, each of the behaviours constituting the behaviour pat-
`tern is not purposefully performed by a user. On the other
`hand, a specific content of a behaviour is, in many cases, that
`of a behaviour that is purposefully performed by a user and
`that is performed for entertainment over a relatively long
`period of time. Accordingly, it is difficult to know such spe-
`cific content of a behaviour from an accumulation of behav-
`
`iours performed in a short period of time.
`[0006]
`In light of the foregoing, it is desirable to provide a
`behaviour pattern analysis system, a mobile terminal, a
`behaviour pattern analysis method, and a program, which are
`new and improved, and which are capable of detecting, from
`a relatively short-term behaviour pattern obtained by using a
`motion sensor, a relatively long-term highly-entertaining
`behaviour pattern.
`[0007] According to an embodiment of the present inven-
`tion, there is provided a behaviour pattern analysis system
`which includes a mobile terminal including a movement sen-
`sor that detects a movement of a user and outputs movement
`information, a current location information acquisition unit
`that acquires information on a current location, a building
`information acquisition unit that acquires information on a
`
`building existing at a location indicated by the information
`acquired by the current location information acquisition unit
`or information on buildings existing at the current location
`and in a vicinity of the current location, a first behaviour
`pattern detection unit that analyses the movement informa-
`tion output from the movement sensor, and detects a first
`behaviour pattern corresponding to the movement informa-
`tion from multiple first behaviour patterns obtained by clas-
`sifying behaviours performed by the user over a relatively
`short period of time, and a transmission unit that transmits, to
`a server, the information on a building or buildings acquired
`by the building information acquisition unit and the first
`behaviour pattern detected by the first behaviour pattern
`detection unit, and a server including a reception unit that
`receives, from the mobile terminal, the information on a
`building or buildings and the first behaviour pattern, and a
`second behaviour pattern detection unit that analyses the
`information on a building or buildings and the first behaviour
`pattern received by the reception unit, and detects a second
`behaviour pattern corresponding to the information on a
`building or buildings and the first behaviour pattern from
`multiple second behaviour patterns obtained by classifying
`behaviours performed by the user over a relatively long
`period of time.
`[0008] The second behaviour pattern detection unit may
`create, by using a specific machine learning algorithm, a
`detection model for detecting the second behaviour pattern
`from the information on a building or buildings and the first
`behaviour pattern, and may detect, by using the created detec-
`tion model, the second behaviour pattern corresponding to the
`information on a building or buildings and the first behaviour
`pattern received by the reception unit.
`[0009] The mobile terminal may further include a time
`information acquisition unit that acquires information on a