US008207832B2
`
`US 8,207,832 B2
`(10) Patent No.:
`a2) United States Patent
`’
`Yunet al.
`45) Date of Patent:
`Jun. 26, 2012
`
`
`(54) HAPTIC EFFECT PROVISIONING FOR A
`MOBILE COMMUNICATION TERMINAL
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`(75)
`
`Inventors: Jue Byung Yun, Seoul (KR); Jong
`Hwan Kim,Seoul (KR)
`
`2008/0024459 Al*
`2009/0167509 Al*
`2010/0102941 Al*
`
`1/2008 Poupyrev etal.
`............ 345/173
`
`... 340/407.2
`7/2009 Fadell etal.
`4/2010 Richteretal. oo... 340/407.2
`
`(73) Assignee: LG Electronics Inc., Seoul (KR)
`(*) Notice:
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`US.C. 154(b) by 432 days.
`
`(21) Appl. No.: 12/436,068
`yg.
`Filed:
`
`May5, 2009
`
`(22)
`
`(65)
`
`Prior Publication Data
`
`US 2009/0322498 Al
`
`Dec. 31, 2009
`
`BD
`EP
`WO
`
`FOREIGN PATENT DOCUMENTS
`heoctea
`9500
`1548549
`6/2005
`2004081776
`9/2004
`OTHER PUBLICATIONS
`E. Hoggan et al., “Investigating the Effectiveness ofTactile Feedback
`for Mobile Touchscreens,” CHI Conference Proceedings—Tactile
`and Haptic User
`Interfaces,
`pp.
`1573-1582, Apr.
`2008,
`XP-002525519.
`
`* cited by examiner
`
`.
`Primary Examiner — Jennifer Mehmood
`Assistant Examiner — Hongmin Fan
`(74) Attorney, Agent, or Firm —Lee, Hong, Degerman,
`Kang & Waimey
`
`
`Jun. 25,2008=(KR) wee ee eeeeeee 10-2008-0060358 (57) ABSTRACT
`A mobile terminal and a method of controlling the mobile
`terminal are provided. The method includes displaying at
`least one object, which is divided into a plurality of haptic
`regions, on a touch screen; receiving a touch signal for any
`one of the haptic regions of the object; and outputting a
`control signal for controlling a haptic module to generate a
`haptic effect. Therefore, it is possible for a user to easily
`determine whichofthe haptic regions of the object has been
`touched with the sense of touch.
`
`(30)
`
`Foreign Application Priority Data
`
`(51)
`
`Int. Cl.
`GO8B 6/00
`
`(2006.01)
`
`(52) US.CL won 340/407.2; 340/407.1; 345/173;
`345/165; 715/702; 715/701
`
`(58) Field of Classification Search............... 340/407.2
`340/407.1; 345/173, 165; 715/700-702
`See application file for complete search history.
`
`18 Claims, 45 Drawing Sheets
`
`—————————_
`—)
`
`
`
`
`
`
`val
`
`aoc] {e251
`(A)
`KEYBOARD MODE
`
`456
`
`
`
`
`oon
`
`a
`
`-~
`
`aoe
`
`987
`
`Jinstrumint Pianoi L “ae” (bossa noba
`
`
`
`ENU
`
`OK PREVIOUS
`
`980
`
`981
`
`983
`
`APPLE 1015
`
`APPLE 1015
`
`1
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 1 of 45
`
`US 8,207,832 B2
`
`FIG. 1
`
`190
`
`POWER SUPPLY UNIT
`
`105
`
`110
`
`150
`
`WIRELESS
`COMMUNICATION UNIT
`BROADCAST
`
`111CEPTION MODULE
`MOBILE
`COMMUNICATION
`MODULE
`WIRELESS
`115-s~
`INTERNET MODULE
`SHORT-RANGE
`MODULE
`COMMUNICATION
`GPS MODULE
`CONTROLLER
`
`113-—Y
`
`119-~—
`
`V7
`
`
`
`
`
`
`
`OUTPUT UNIT
`
`DISPLAY MODULE
`
`
`AUDIO OUTPUT
`
`MODULE
`
`
`ALARM MODULE
`
`
`
`
`
`
`
`
`HAPTIC MODULE
`
`
`
`
`
`154
`
`153
`
`155
`
`157
`
`
`
`MULTIMEDIA
`PLAY MODULE
`
`INTERFACE UNIF
`
`120-4
`
`AN INPUT UNIT
`
`CAMERA MODULE
`121
`
`123 “~~ MICROPHONE
`
`
`
`
`MODULE
`
`130 --—
`
`140
`
`USER INPUT UNIT
`
`SENSING UNIT
`
`{4{--—~ | PROXIMITY SENSOR
`
`
`
`180
`
`181
`
`170
`
`160
`
`MEMORY
`
`2
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 2 of 45
`
`US 8,207,832 B2
`
`FIG. 2
`
`
`
`3
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 3 of 45
`
`US 8,207,832 B2
`
`FIG. 3
`
`100A-2 100A-1
`
`150
`
`125
`
`121b
`
`126
`100B en
`
`190
`
`
`
`
`SN1306
`
`(tr 30f
`
`+—~___100B-1
`
`++-~_100B-2
`
`4
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 4 of 45
`
`US 8,207,832 B2
`
`FIG. 4
`
`
`SET MOBILE TERMINAL
`
`
`~~ 5200
`TO HAPTIG MODE
`
`
`DISPLAY FIRST SCREEN
`
`$202
`
`
`
`
`
`
`_
`IS CHANGE
`IN FIRST SCREEN
`
`DETECTE3S?
`
`
`$206
`a
`OUTPUT CONTROL SIGNAL
`CORRESPONDING TO CHANGE
`IN FIRST SCREEN
`
`
`
`
`
`
`
`$210
`S208
`
`
`OUTPUT CONTROLSIGNAL
`IS POPUPSeg
`CORRESPONDING TO
`Sm SCREEN DISPLAYED? >>
`
`POPUP SCREEN
`
`
`
`$214
`
`
`
`OUTPUT CONTROL SIGNAL
`CORRESPONDING TO
`
`SECOND SCREEN
`
`
`$218
`
`
`
` OUTPUT CONTROL SIGNAL
`IS CHANGE IN
`YES
`BRIGHTNESS DETECTED or
`CORRESPONDING TO CHANGE
`
`IN BRIGHTNESS
`
`
`
`$222
`
`
`
`OUTPUT CONTROL SIGNAL
`IS CHANGE IN
`
`
`SOUND OR SOUND VOLUME
`CORRESPONDING TO CHANGE
`DETECTED?
`IN SOUND OR SOUND VOLUME
`
`
`
`
`
` 3224
`
`RESPONSE TO CONTRCL SIGNAL
` GENERATE HAPTIC EFFECT IN
`
`
`
`
`
`__---“WISH TO TERMINATE>
`HAPTIC MODE?
`
`NO
`
`5
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 5 of 45
`
`US 8,207,832 B2
`
`FIG. 5
`
`DISPLAY MOVING
`IMAGE SCREEN
`
`S HAPTIC MODE CHOSEN?
`
`
`
`
`
`
`
`
`
`
`
`
`DETECT CHANGE IN
`MOVING IMAGE SCREEN
`
`OUTPUT CONTROL SIGNAL
`CORRESPONDING TO CHANGE
`IN MOVING IMAGE SCREEN
`
`GENERATE HAPTIC EFFECT
`
`WISH TO
`TERMINATE PLAYING
`MOVING IMAGE?
`
`6
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 6 of 45
`
`US 8,207,832 B2
`
`FIG. 6
`
`START
`
`
`
`DISPLAY MOVING
`IMAGE SCREEN
`
`S300
`
`5306
`
`5304
`
`$308
`
`S302
`
` IS VIBRATION
` INPUT MODE CHOSEN?
`
`
`
`
`YES
`
`DISPLAY VIBRATION
`INPUT MENU
`
`INPUT VIBRATION INTENSITY
`AND VIBRATION PATTERN
`
`STORE INPUT DATA
`
`
`
`
`5310
`
`
`
`
`WISH TO
`TERMINATE VIBRATION
`INPUT MODE?
`
`YES
`
`NO
`
`$312
`
`
`WISH TO
`MOVING IMAGE?
`
`TERMINATE PLAYING
`
`
`YES
`
`END
`
`7
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 7 of 45
`
`US 8,207,832 B2
`
`FIG. 7
`
`START
`
`INPUT MODE
`
`CHOOSE CHARACTER
`
`
`DISPLAY CHARACTER
`INPUT WINDOW
`
`$354
`
`
`
`
`IS CHARACTER INPUT?
`
`YES
`
`DISPLAY INPUT CHARACTER
`
`GENERATE HAPTIC EFFECT
`CORRESPONDING TO
`INPUT CHARACTER
`
`$352
`
`$356
`
`$358
`
`
`
`
`
`S360
`
`NO
`
`
` S ANOTHER
`
`OPERATING MO
`DE
`
`CHOSEN?
`
`
`YES
`
`
`
`
`ENTER CHOSEN
`OPERATING MODE
`
`
`
`8
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 8 of 45
`
`US 8,207,832 B2
`
`aileal
`
`3°Old
`
`
`
`“4;20bg7TWOWHOi
`
`
`
`TtSddeoo
`
`
`
`@_1OdgdsOUVeSEVEINIO
`
`SSHHOO{VvHINSE Iga
`
`SaHqwOO
`
`SuvbINaS
`
`9
`
`
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 9 of 45
`
`US 8,207,832 B2
`
`FIG. 9
`
`START
`
`
`
`DISPLAY MENU ITEMS
`
`$450
`
`9452
`
`
`IS ANY ONE OF MENU
`ITEMS TOUCHED?
`
`
`
`
`
`GENERATE HAPTIC EFFECT
`
`
`CORRESPONDING TO TOUCHED
`MENU ITEM
`
`$454
`
`S456
`
`IS TOUCHED MENU
`
`
`
`TEM TOUCHED AGAIN?
`
`YES
`
`
`
`
`
`
`ENTER MENU ITEM CORRESPONDING
`
`TO TOUCHED MENU ITEM
`
`10
`
`10
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 10 of 45
`
`US 8,207,832 B2
`
`FIG. 10
`
`START
`
`
`
`
`
`DISPLAY OPERATION SCREEN
`
` S500
`
`
`
`
`OUTPUT VIBRATION SIGNAL CORRESPONDING
`TO TOUCHED OR APPROACHED OBJECT
`
`
`
`
`
`
`
`
` IS ANY ONE OF
`OBJECTS DISPLAYED ON OPERATION
`
`SCREEN TOUCHED OR
`APPROACHED?
`
`
`
`
`
` iS TOUCHED OR APPROACHED
`
`OBJECT CHOSEN?
`
`
`
`
`
`
`
`ORM OPERATION CORRESPONDING
`TO CHOSEN OBJECT
`
`
`
`
`
`E CHOSEN?
`
`
`ER OPERATING
`
`
` ES
`
`
`
` ENTER CHOS
`
` RATING MOD
`
`
`
`
`
`$512
`
`11
`
`11
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet11 of 45
`
`US 8,207,832 B2
`
`FIG. 11
`
`
`
`
`
`
`
`
`b3
`De
`Di
`Do
`
`
`
`
`
`
`
`!
`.
`!
`
`th
`
`
`
`
`
` CET aT Me
`
`
`|
`
`12
`
`12
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet12 of 45
`
`US 8,207,832 B2
`
`FIG. 12A
`
`OBJECT REGION
`
`eee
`
`FIRST
`OBJECT REGION
`
`SECOND
`
`FIG. 12B
`
`eeeee
`
`13
`
`13
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 13 of 45
`
`US 8,207,832 B2
`
`FIG. 12C
`
`2 We
`
`(a)
`
`(b)
`
`(Cc)
`
`(d)
`
`FIG. 12D
`
`
`
`14
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 14 of 45
`
`US 8,207,832 B2
`
`FIG. 13
`
`OBJECT REGION
`
`
`
`AVAVAN
`
`INTENSITY
`
`INTENSITY
`
`INTENSITY
`
`\SPLPYI
`|eee|TIME TIME TIME
`
`
`
`oN|
`[A]
`
`15
`
`15
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 15 of 45
`
`US 8,207,832 B2
`
`FIG. 14A
`
`(a)
`
`FIG. 14B
`
`16
`
`16
`
`
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet16 of 45
`
`US 8,207,832 B2
`
`FIG. 15
`
`
`
`
`SFO) so KI Eid 12:34am
`
`
`
`
`
`
`Mea
`
`12:34
`
`(a)
`
`(6)
`
`(c)
`
`
`
`
`[Mal
`1 13:34
`
`
`
`
`
`(d) a
`12:34
`
`(e)
`
`Wall
`
`
`112:34
`
`
`
`f)
`
`Sal
`12:34am
`
`
`
`
`
`
`g) Mel 12:34 O
`
`
`
`(h)
`
`Sal
`
`12:34am
`
`17
`
`17
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 17 of 45
`
`US 8,207,832 B2
`
`FIG. 16
`
`INTENSITY
`
`[FET
`
`
`
`FIG. 17
`
`
`
`Co
`
`
`ruil
`(R)
`aoc] (22:51
`
`Edit
`35/i60[{ ||~~~ 600
`
`
`
`
`
`
`
`
`
`£2
`tay’
`meZF|_ 603
`63%
`Transfering file
`Transfering file
`
`
`
`18
`
`18
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 18 of 45
`
`US 8,207,832 B2
`
`FIG. 18
`
`eeoee
`
`
`
`
`RATE OF CHANGE
`OF AN IMAGE
`
`QPERATION TYPE
`
`
`
`TYPES OF OPERATION
`
`OPERATION TYPE
`
`CONNECTING
`BLUETOOTH
`
`SENDINGFILE
`
`
`
`
`
`
`
`
`
`
`SEEPD OF CHANGE
`OF AN IMAGE
`
`OPERATION TYPE
`
`2
`
`4
`
`19
`
`19
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 19 of 45
`
`US 8,207,832 B2
`
`co)
`
`Yul
`(RB) aoc
`
`Send to
`
`
`22:51
`
`Send this file?
`
`
`
`[fo
`
`
`
`
`
`
`
`565656563
`
`Options
`
`Back
`
`
`
`
`
`
`
`
`Cc
`
`fwd (R)
`
`Edit
`
`eo oc] Qio2-51
`(abe)
`
`
`
`
`
`
`1+ B30
`
`20
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 20 of 45
`
`US 8,207,832 B2
`
`Gro"
`
`
`STSESESESESESESSSESEOSSESEOSSEEOSISSEOUISSEQUISISGESEONOY:20S)SSSS)=SNSSISINNOSINNOUININGUTSINSEOSSSISGIOSOSSION/SINN}UININOUININGUISSEOE}SSIOEOSSNEOUGNGINU}SINSU}SINSEOS}SSEOS)SSEIEOSSSEOESIOED
`
`
`=
`
`peusin®yBurBuu
`
`\ee
`
`Y
`
`02“Dls
`
`
`
`
`
`
`
`
`gOG2I91Ud43Saeet:
`
`
`
`}U0-43877
`
`OZS71WoOIst
`
`
`
`Og@Z13Oeuordo1Sk
`
`
`
`azUoIdo1s
`
`21
`
`
`
`
`
`=
`
`a
`
`21
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 21 of 45
`
`US 8,207,832 B2
`
`
`ir
`~)
`(7
`S\
`
`coo
`
`vel
`
`(R})
`
`eo] 22:51
`
`
`a)
`
`=
`*
`N
`Co
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`q°
`2
`a
`
`© C
`
`wBse
`
`Eaw
`ERBO
`QBS5
`a
`
`incomming call
`
`& Maramara
`
`
`
`
`‘201055898
`
`(cx)
`— 3
`Snooze
`Quit
`-
`
`
`
`22
`
`
`
`22
`
`

`

`U.S. Patent
`
`US 8,207,832 B2
`
`
`
`(9)(a)ix
`
`OSae
`
`BdforoudAy°___aMbie(oO}q:=
` "|!+|Smt|°|a?“N|}—\'e'v:=a:MLLEENbbLgQGolfopouctA)
`
`
`eoOla
`
`
`
`
`
`
`
`
`
`
`23
`
`23
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 23 of 45
`
`US 8,207,832 B2
`
`FIG. 23
`
`val
`
`(A)
`
`ew o) f22.51
`
`HPI
`
`ce
`
`AE
`
`itii '
`
`i Hi
`
`SLEEPING
`
`PLAYING
`
`
`
`
`
`
`AG
`ANGRY --a2-
`
`LEG
`
`ti
`ON BUSINESS!
`TRIP
`
`7
`!
`
`
`—-—
`
`a)
`
`
`~\
`
`Yul
`
`eo oj 22:51
`(R)
`
`Crazy in Love
`90/000
`
`
`BBeatles
`@ Abbey Road
`
`02:45
`
`Kd [][] po
`
`‘Menu
`
`OK
`
`Se
`04:55
`
`Back
`
`
`XS
`LY)
`
`24
`
`24
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 24 of 45
`
`US 8,207,832 B2
`
`
`
`02:45
`
`File name.xxx
`
`+]
`=
`
`FIG. 26
`
`
`
`eoeoen
`
`25
`
`25
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 25 of 45
`
`US 8,207,832 B2
`
`FIG. 27
`
`RATE OF CHANGE
`
`CLASSIFY
`
`
`
`
`
`OBJECT SIZE
`
`CLASSIFY
`
`26
`
`26
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 26 of 45
`
`US 8,207,832 B2
`
`
`
`Tall
`
`enw I) (22-51
`
`ot
`
`==cti==coc
`=cta=iota=
`
`27
`
`710
`
`27
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 27 of 45
`
`US 8,207,832 B2
`
`FIG. 29
`
`02:44
`02:48
`
`
`AT
`
`|
`
`
`
`+
`
`+
`
`02:45
`
`ia oa
`
`+-
`
`++
`
`
`
`——» Dance.xxx
`
`
`
`
`28
`
`28
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 28 of 45
`
`US 8,207,832 B2
`
`
`
`Tall
`
`720
`
`29
`
`29
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 29 of 45
`
`US 8,207,832 B2
`
`
`
`
`
`
`
`
`
`
`LONG TOUCH 2
`
`eee
`
`BUTTON NO
`
`CLASSIFY
`
`| T
`
`OUCH PATTERN
`
`CLASSIFY
`
`SHORT TOUCH
`
`LONGTOUCH1
`
`4
`
`ae
`
`4
`
`ay
`
`(a)
`
`(b)
`
`(c)
`
`30
`
`30
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 30 of 45
`
`US 8,207,832 B2
`
`
`
`
`
`cc) UE
`
`
`
`Tull
`
`(R)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`31
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 31 of 45
`
`US 8,207,832 B2
`
`FIG. 33
`
`
`INTENSITY
`
`
`INTENSITY
`
`
`INTENSITY
`
`
`TIME
`
`TIME
`
`TIME
`
`FIG. 34
`
`
`c=
`
`yal
`
`(A)
`
`eo] 22:51
`
`Te__-830
`
`
`
`
`
`
`
`
`&,
`
`Bang
`12345
`
`<p
`
`End
`
`INTENSITY
`—- — eevee Lo
`
`
`ei
`
`TIME
`
`
`
`"
`
`()
`
`32
`
`32
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 32 of 45
`
`US 8,207,832 B2
`
`FIG. 35
`
`ral
`
`sc] (eo-51
`(2)
`Page 1/10 Pos
`
`TIME
`
`840
`
`INTENSITY
`
`> |7\/
`
`33
`
`33
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 33 of 45
`
`US 8,207,832 B2
`
`FIG. 36
`
`
`
`INTENSITY
`
`| S\ a |
`
`
`
`aTIME
`
`
`
`
`
`~
`
`
`
` —
`
`Fal
`
`(B]
`
`a0 c) (22-47
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`© Os
`() ) &
`
`
`
`
`val
`Pi
`
`A
`BS
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 37
`
`860
`
`INTENSITY
`
`
`
`TIME
`
`34
`
`
`
`
`
`—EEEEEEEEEE)
`
`—
`
`(R} we) (22-47
`fe
`
`
`34
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 34 of 45
`
`US 8,207,832 B2
`
`SN
`<p
`
`sect (io2-.47
`
`FIG. 38
`
`860
`
`INTENSITY
`
`
`
` TIME
`
`fet
`
`(R)
`
`
`
`
`
`
`
`
`
`(a)
`
`(b)
`
`35
`
`35
`
`

`

`Jun. 26, 2012
`
`Sheet 35 of 45
`
`US 8,207,832 B2
`
`NLL nnnSE.
`
`
`
` \\iener30wee|\he‘‘ust
`
`SULOUIES)
`
`USGLLO9OVSO/rE|
`
`
`
`naanoccannnnsnnn
`
`
`
`36
`
`U.S. Patent
`
`62“Oi
`
`
`
`hewneweeeeneeneee
`
`aO
`
`JRAtyzzfhbea)as
`
`a w
`
`oDa
`
`e>ca
`
`ed
`y
`mbee
`
`3S
`
`=e
`
`y
`
`g==S
`
`O
`Li)
`74
`™
`ea~~
`panne
`e
`
`
`36
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet36 of 45
`
`US 8,207,832 B2
`
`FIG. 40
`
`
`cc
`
`
`Yel
`
`(R)
`
`ee] GB 22-47
`
`
`
`eee
`
`TIME
`
`
`
`
`
` INTENSITY
`Connecting oS
`
`
`
`
`FIG. 41
`
`vel
`
`(BR)
`
`0c] (22-47
`
`Korea's figure skatung
`star Kim Yuna grabbed
`her second
`conseculive title at the
`
`GrandPri {)
`
`940
`
`INTENSITY
`
`—
`
`37
`
`37
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 37 of 45
`
`US 8,207,832 B2
`
`epDid
`
`No
`
`IC3010=3s53
`
`
`
`38
`
`38
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 38 of 45
`
`US 8,207,832 B2
`
`FIG. 43
`
`OBJECT1
`
`OBJECT2
`
`I
`WEAK
`
`STRONG
`
`
`
`
`WEAK
`
`STRONG
`
`39
`
`39
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 39 of 45
`
`US 8,207,832 B2
`
`FIG. 44
`
`OBJECT1
`
`OBJECT2
`
`
`
`40
`
`40
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 40 of 45
`
`US 8,207,832 B2
`
`FIG. 45
`
` f sooo ~
`
`
`
`
`
`™ HAPTIC EFFECT BECOMES STRONGER
`
`Control Wheel
`
`41
`
`41
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 41 of 45
`
`US 8,207,832 B2
`
`FIG
`
`. 46
`
`
`
`42
`
`42
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 42 of 45
`
`US 8,207,832 B2
`
`FIG. 47
`
`(R)
`
`Tul
`
`-953
`
`
`
`43
`
`43
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 43 of 45
`
`US 8,207,832 B2
`
`FIG. 48
`
`960
`
`> 961
`
`|_---- » 963
`
`44
`
`44
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 44 of 45
`
`US 8,207,832 B2
`
`CHS
`
`876.9MHz
`
`Digitally Imported
`
`45
`
`45
`
`

`

`U.S. Patent
`
`Jun. 26, 2012
`
`Sheet 45 of 45
`
`US 8,207,832 B2
`
`
`
`Cc
`
`*ail
`
`aot) ffj2o.54
`KEYBOARD MODE
`
`456
`
`
`
`
`
`
`
`
`
`co
`
`oo
`
`980
`
`981
`
`987
`
`“_
`ano
`7 lineturnrn: [Piano
`
`B¥<4
`“sye’ [bossa noba
`ENU
`OK PREVIOUS
`
`
`
`
`
`
`
`46
`
`46
`
`

`

`US 8,207,832 B2
`
`1
`HAPTIC EFFECT PROVISIONING FORA
`MOBILE COMMUNICATION TERMINAL
`
`CROSS-REFERENCE TO RELATED
`APPLICATION
`
`This application claims the priority benefit of Korean
`Patent Application No. 10-2008-0060358, filed on Jun. 25,
`2008 in the KoreanIntellectual Property Office, the content of
`whichis incorporated herein in entirety.
`
`RELATED FIELDS
`
`The present invention relates to a mobile terminal, which
`provides various haptic effects according to the content of
`information displayed by the mobile terminal or the setting
`state of the mobile terminal, and a method ofcontrolling the
`mobile terminal.
`
`BACKGROUND
`
`Mobile terminals are portable devices, which can provide
`users with various services such as a voice calling service, a
`video calling service, an information input/output service,
`and a data storage service. As the types of services provided
`by mobile terminals diversify, an increasing number of
`mobile terminals have been equipped with various compli-
`cated functions such as capturing photos or moving pictures,
`playing music files or moving imagefiles, providing game
`programs, receiving broadcast programs and providing wire-
`less Internet services and have thus evolved into multimedia
`players.
`Various user interface (UI) environments, in which users
`are allowedto easily search for and choose desired functions,
`have been developed. In addition, the demand for various
`designs for mobile terminals such as a double-sided liquid
`crystal display (LCD) or a full touch screen has steadily
`grown dueto a growing tendency of considering mobileter-
`minals as personal itemsthat can represent users’ personality.
`In a limited display device, such as a mobile communica-
`tion terminal, there is a restriction in allocating sufficient
`space for a UI device (such as a display device or a keypad) of
`a mobile terminal without compromising the mobility and the
`portability of a mobile terminal. In addition, conventional
`mobile terminals provide operating state information or
`respond to user commands merely using a limited UI space or
`simple soundeffects.
`Therefore, systems and methods are neededto controlthe
`operation of mobile terminals using through an efficient and
`easy-to-use user interface.
`
`SUMMARY
`
`In accordance with one embodiment, a mobile terminalis
`provided in which a haptic effect is generated when a certain
`inputor output is detected. A user ofthe mobile terminal may
`configure the manner in which the haptic effect is produced.
`For example, if the haptic effect is generated by way of a
`vibration mechanism,the user maysetthe level ofvibration to
`be generated when a certain input is provided to the mobile
`terminal, or when a certain output is generated by the mobile
`terminal. The input and output to the mobile terminalis thus
`monitored so that when, for example, a message including a
`certain word or a certain output, for example, playback of a
`moving picture having a certain level of lighting or motion is
`detected, then a particular haptic effect associated with the
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`detected input our output is generated. The user may set
`different levels or types of haptic effects with different levels
`or types of input and output.
`In one embodiment, a mobile terminal comprising an input
`mechanism and an output mechanism, wherein the input
`mechanism has a surface such that said surface is virtually
`divided into at least two separate regions, wherein the two
`separate regions provide a tactile sensation such thatthe first
`region providesa first tactile sensation and the second region
`provides a second tactile sensation, wherein a user in
`response to touchingthefirst region and the second region
`recognizes a difference in the first and secondtactile sensa-
`tions, wherein said difference in sensations allows a user to
`detect a change in functionality of the mobile terminal or an
`output generated by the mobile terminal.
`In one embodiment, the mobile terminal divides an object
`(such as an icon or an image for controlling the operation of
`the mobile terminal) into one or more haptic regions and
`provides different haptic effects for the haptic regions, and a
`methodof controlling the mobile terminal. A method of con-
`trolling a mobile terminal comprises displaying at least one
`object, which is divided into a plurality of haptic regions, on
`a touch screen; receiving a touch signal for any one of the
`haptic regions of the object; and outputting a control signal
`for controlling a haptic module to generate a haptic effect.
`In one embodiment, a mobile terminal comprises a touch
`screen configured to display at least one object, which is
`divided into a plurality of haptic regions; a haptic module
`configured to generate a haptic effect; and a controller which
`receives a touch signal for any one ofthe haptic regionsofthe
`object and controls the haptic module to generate a haptic
`effect correspondingto the haptic region for which the touch
`signal is received.
`Accordingto another aspect, amethod comprising display-
`ing at least one object, which is divided into a plurality of
`haptic regions, on a touch screen is provided. The method
`comprises receiving a touch signal for any oneofthe haptic
`regions of the object; and outputting a control signal for
`controlling a haptic module to generate a haptic effect.
`The above and other features and advantages of the present
`invention will become more apparent by describing in detail
`preferred embodiments thereofwith reference to the attached
`drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 illustrates a block diagram of a mobile terminal
`according to an exemplary embodiment;
`FIG.2 illustrates a front perspective view of the mobile
`terminal shown in FIG.1;
`FIG. 3 illustrates a rear perspective view of the mobile
`terminal shownin FIG.1;
`FIG.4 illustrates a flowchart of a method of controlling a
`mobile terminal according to an exemplary embodiment;
`FIG.5 illustrates a flowchart of a method of controlling a
`mobile terminal according to another exemplary embodi-
`ment;
`FIG.6 illustrates a flowchart of a method of controlling a
`mobile terminal according to another exemplary embodi-
`ment;
`FIG.7 illustrates a flowchart of a method of controlling a
`mobile terminal according to another exemplary embodi-
`ment;
`FIG.8 illustrates a flowchart of a method of controlling a
`mobile terminal according to another exemplary embodi-
`ment;
`
`47
`
`47
`
`

`

`US 8,207,832 B2
`
`4
`the broadcast signal include a TV broadcast signal, a radio
`broadcast signal, a data broadcast signal, or the combination
`ofa data broadcastsignal and either a TV broadcastsignal or
`a radio broadcast signal. The broadcast-related information
`maybe providedto the mobile terminal 100 through a mobile
`communication network. In this case, the broadcast-related
`information may bereceived by the mobile communication
`module 113, rather than by the broadcast reception module
`111. The broadcast-related information may comein various
`forms, for example, electronic program guide (EPG)ofdigi-
`tal multimedia broadcasting (DMB)orelectronic service
`guide (ESG)of digital video broadcast-handheld (DVB-H).
`The broadcast reception module 111 may receive the
`broadcast signal using various broadcasting systems such as
`digital multimedia broadcasting-terrestrial (DMB-T), digital
`multimedia broadcasting-satellite (DMB-S), media forward
`link only (MediaFLO), DVB-H,andintegrated services digi-
`tal broadcast-terrestrial (ISDB-T). In addition, the broadcast
`reception module 111 may be configured to be suitable for
`nearly all types of broadcasting systems other than those set
`forth herein. The broadcast signal and/or the broadcast-re-
`lated information received by the broadcast reception module
`1110 maybe stored in the memory 160.
`The mobile communication module 113 transmits wireless
`
`3
`FIG.9 illustrates a flowchart of a method of controlling a
`mobile terminal according to another exemplary embodi-
`ment;
`FIG.10 illustrates a flowchart of a methodof controlling a
`mobile terminal according to another exemplary embodi-
`ment;
`FIG. 11 illustrates a diagram for explaining the operation
`of a proximity sensor;
`FIGS. 12 through 14 illustrate diagrams for explaining a
`proximity region and a haptic region;
`FIGS. 15 through 24 illustrate diagramsfor explaining how
`to generate a haptic effect in accordance with a changein a
`screen image displayed by a display unit of the mobile termi-
`nal shown in FIG.1;
`FIGS. 25 through 31 illustrate diagramsfor explaining how
`to generate and store a haptic effect during the play of a
`moving imagefile;
`FIGS. 32 through 42illustrate diagramsfor explaining how
`to generate a haptic effect according to the content of infor-
`mation displayed on the screen of the display module of the
`mobile terminal shownin FIG. 1; and
`FIGS. 43 through 50 illustrate diagramsfor explaining how
`to generate different haptic effects for different object
`regions.
`
`DETAILED DESCRIPTION OF EXEMPLARY
`EMBODIMENTS
`
`10
`
`25
`
`30
`
`40
`
`45
`
`55
`
`signals to or receives wireless signals from at least one of a
`base station, an external terminal, and a server through a
`mobile communication network. The wireless signals may
`include varioustypes of data according to whether the mobile
`terminal 100 transmits/receives voice call signals, video call
`The term ‘mobile terminal’, as used herein, may indicate a
`signals, or text/multimedia messages. The mobile communi-
`mobile phone, a smart phone, a laptop book computer, a
`cation module 113 may be a modulefor wirelessly accessing
`digital broadcast receiver, a personaldigital assistant (PDA),
`the Internet. The mobile communication module 113 may be
`a portable multimedia player (PMP), or a navigation device.
`embeddedin the mobile terminal 100 or may beinstalled in an
`FIG.1 illustrates a block diagram of a mobile terminal 100
`external device. The short-range communication module 117
`according to an embodimentof the present invention. Refer-
`may be a module for short-range communication. The short-
`ring to FIG. 1, the mobile terminal 100 may include a wireless
`range communication module 117 may use various short-
`communication unit 110, an audio/video (A/V) input unit
`range communication techniques such as Bluetooth, radio
`120, a user input unit 130, a sensing unit 140, an output unit
`frequency identification (RFID), infrared data association
`150, a memory 160, an interface unit 170, a controller 180,
`(IrDA), ultra wideband (UWB), and ZigBee.
`and a power supply unit 190. Two or more of the wireless
`The GPS module 119 may receive position information
`communication unit 110, the A/V input unit 120, the user
`
`input unit 130, the sensing unit 140, the output unit 150, the fromaplurality ofGPSsatellites. The A/V input unit 120 may
`memory 160, the interface unit 170, the controller 180, and
`be used to receive audio signals or video signals. The A/V
`the power supply unit 190 maybe incorporated into a single
`input unit 120 may include a camera module 121 and a micro-
`unit, or some ofthe wireless communication unit 110, the A/V
`phone module 123. The camera module 121 processes vari-
`input unit 120, the user input unit 130, the sensing unit 140,
`ous image frames such asstill images or moving images
`the output unit 150, the memory 160, the interface unit 170,
`captured by an image sensor during a video call mode or an
`the controller 180, and the power supply unit 190 may be
`image capturing mode. The image frames processed by the
`divided into two or more smallerunits.
`camera module 121 may be displayed by a display module
`The wireless communication unit 110 may include a
`151. The image frames processed by the camera module 121
`broadcast reception module 111, a mobile communication
`may be stored in the memory 160 or may be transmitted
`module 113, a wireless Internet module 115, a short-range
`outside the mobile terminal 100 through the wireless com-
`communication module 117, and a global positioning system
`munication unit 110. The mobile terminal 100 may include
`two or more camera modules 121.
`(GPS) module 119. The broadcast reception module 111
`receives a broadcast signal and/or broadcast-related informa-
`The microphone module 123 receives external soundsig-
`tion from an external broadcast managementserver through a
`nals during a call mode, a recording mode, or a voice recog-
`broadcast channel. Examples of the broadcast channel
`nition mode with the use of a microphone and converts the
`include a satellite channel and a terrestrial channel. The
`sound signals into electrical sound data. In the call mode, the
`mobile communication module 1130 may convert the elec-
`trical sound data into data that can be readily transmitted to a
`mobile communication base station and then output the data
`obtained by the conversion. The microphone module 123 may
`use various noise removal algorithms to remove noise that
`may be generated during the reception of external sound
`signals. The user input unit 130 generates key input data
`based on user input for controlling the operation ofthe mobile
`terminal 100. The user input unit 130 may be implementedas
`akeypad, a domeswitch, a touch pad(eitherstatic pressure or
`
`broadcast managementserver maybe a server which gener-
`ates broadcast signals and/or broadcast-related information
`and transmits the generated broadcast signals and/or the gen-
`erated broadcast-related information or a server which
`
`receives and then transmits previously-generated broadcast
`signals and/or previously-generated broadcast-related infor-
`mation.
`
`Examples of the broadcast-related information include
`broadcast channel information, broadcast program informa-
`tion and broadcastservice provider information. Examples of
`
`48
`
`48
`
`

`

`US 8,207,832 B2
`
`5
`constant electricity), a jog wheel, or a jog switch. In particu-
`lar, if the user input unit 130 is implemented as a touch pad
`and forms a mutual layer structure along with the display
`module 151, the user input unit 130 and the display module
`151 may becollectively referred to as a touch screen.
`The sensing unit 140 determines a current state of the
`mobile terminal 100 such as whether the mobile terminal 100
`
`is opened up or closed,the position ofthe mobile terminal 100
`and whether the mobile terminal 100 is placed in contact with
`a user, and generates a sensing signal for controlling the
`operation of the mobile terminal 100. For example, when the
`mobile terminal 100 is a slider-type mobile phone,the sens-
`ing unit 140 may determine whether the mobile terminal 100
`is opened up or closed. In addition, the sensing unit 140 may
`determine whether the mobile terminal 100 is powered by the
`power supply unit 190 and whetherthe interface unit 170 is
`connected to an external device.
`
`The sensing unit 140 may include a proximity sensor 141.
`The proximity sensor 141 may determine whetherthere is an
`entity nearby and approaching the mobile terminal 100 with-
`out any mechanical contact with the entity. More specifically,
`the proximity sensor 141 may detect an entity that is nearby
`and approaching by detecting a change in an alternating mag-
`netic field or the rate of change of static capacitance. The
`sensing unit 140 mayinclude two or more proximity sensors
`141. The output unit 150 may output audio signals, video
`signals and alarm signals. The output unit 150 may include
`the display module 151, an audio output module 153, an
`alarm module 155, and a haptic module 157 incorporating
`haptic technology.
`Haptic technologyrefers to technology whichinterfaces to
`the user via the sense of touch by applying forces, vibrations
`and/or motionsto the user. This mechanical stimulation may
`be usedto assist in the creation ofvirtual objects(e.g., objects
`existing in a computing environment), for control of such
`virtual objects, and to enhance the control of machines and
`devices (e.g., mobile terminals). For example, haptic technol-
`ogy makesit possible to investigate in detail how the human
`sense of touch works, by allowing the creation of carefully-
`controlled haptic virtual objects. These objects may be used
`to systematically probe human haptic capabilities.
`The display module 151 may display various information
`processed by the mobile terminal 100. For example, if the
`mobile terminal 100 is in a call mode,the display module 151
`may display a user interface (UJ) or a graphic user interface
`(GUI) for making orreceiving a call. If the mobile terminal
`100 is in a video call mode or an image capturing made, the
`display module 151 maydisplay a UI or a GUIfor capturing
`or receiving images. As described above,if the display mod-
`ule 151 and the user input unit 130 form a mutual layer
`structure and are thus implemented as a touch screen, the
`display module 151 may be usednotonly as an output device
`but also as an input device. The display module 151 may
`include at least one of a liquid crystal display (LCD), a thin
`film transistor (TFT)-LCD, an organic light-emitting diode
`(OLED), a flexible display, and a three-dimensional (3D)
`display. The mobile terminal 100 may include two or more
`display modules 151. For example, the mobile terminal 100
`may include an external display module (not shown) and an
`internal display module (not shown).
`The audio output module 153 may output audio data
`received by the wireless communication unit 110 during a call
`reception mode,a call mode, a recording mode, a voice rec-
`ognition mode,or a broadcast reception mode or may output
`audio data present in the memory 160. In addition, the audio
`output module 153 may output various soundsignals associ-
`ated with the functions of the mobile terminal 100 such as
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`receiving a call or a message. The audio output module 153
`may include a speaker and a buzzer.
`The alarm module 155 mayoutputan alarm signal indicat-
`ing the occurrence of an event in the mobile terminal 100.
`Examplesof the event include receiving a call signal, receiv-
`ing a message, and receiving a key signal. Examples of the
`alarm signal output by the alarm module 155 include an audio
`signal, a video signal and a vibration signal. The alarm mod-
`ule 155 mayoutput a signal upon receiving a call signal ora
`message. In addition, the alarm module 155 mayreceive a key
`signal and may output a signal as feedbackto the key signal.
`Once a signal is output by the alarm unit, the user may
`recognize that an event has occurred. A signal for notifying
`the user of the occurrence of an event may be output by the
`display module 151 or the audio output module 153.
`The haptic module 157 mayprovide various haptic effects
`(such as vibrations) that can be perceived by the user.If the
`haptic module 157 generates vibration as a haptic effect, the
`intensity and the pattern of vibration generated by the haptic
`module 157 may be altered in various manners. The haptic
`module 157 may synthesize different vibration effects and
`may outputthe result of the synthesization. Alternatively, the
`haptic module 157 may sequentially output different vibra-
`tion effects.
`
`The haptic module 157 mayprovide varioushapticeffects,
`other than vibration, such as a haptic effect obtained using a
`pin array that moves perpendicularly to a contactskin surface,
`a haptic effect obtained by injecting or suckingin air through
`an injection hole or a suction hole, a haptic effect obtained by
`giving a stimulus to the surface of the skin, a haptic effect
`obtained through contact with an electrode, a haptic effect
`obtained using an electrostatic force, and a haptic effect
`obtained by realizing the sense ofheat or cold using a device
`capable of absorbing heat or generating heat. The haptic
`module 157 may be configured to enable the user to recognize
`a haptic effect using the kinesthetic senseofthe fingers or the
`arm