111111
`
`(19) United States
`(12) Patent Application Publication
`Hinckley et al.
`
`11111111111111111111111111111111111111111111111111111111111111
`US 20010011995Al
`
`(10) Pub. No.: US 2001/0011995 A1
`Aug. 9, 2001
`(43) Pub. Date:
`
`(54) METHOD FOR PROVIDING FEEDBACK
`RESPONSIVE TO SENSING A PHYSICAL
`PRESENCE PROXIMATE TO A CONTROL
`OF AN ELECTRONIC DEVICE
`
`(76)
`
`Inventors: Kenneth Hinckley, Seattle, WA (US);
`Steven Bathiche, Redmond, WA (US);
`William Vong, Seattle, WA (US)
`
`Correspondence Address:
`BANNER & WITCOFF
`1001 G STREET N W
`SUITE 1100
`WASHINGTON, DC 20001 (US)
`
`(21) Appl. No.:
`
`09/804,383
`
`(22) Filed:
`
`Mar. 13, 2001
`
`Related U.S. Application Data
`
`(63) Continuation-in-part of application No. 09/152,434,
`filed on Sep. 14, 1998, which is a continuation-in-part
`of application No. 09/152,432, filed on Sep. 14, 1998,
`which is a continuation-in-part of application No.
`09/152,443, filed on Sep. 14, 1998, which is a con(cid:173)
`tinuation-in-part of application No. 09/200,325, filed
`
`on Nov. 25, 1998, which is a continuation-in-part of
`application No. 09/200,321, filed on Nov. 25, 1998,
`now Pat. No. 6,232,957, which is a non-provisional of
`provisional application No. 60/100,261, filed on Sep.
`14, 1998.
`
`Publication Classification
`
`Int. Cl? ....................................................... G09G 5/00
`(51)
`(52) U.S. Cl. .............................................................. 345/156
`
`(57)
`
`ABSTRACT
`
`In a computer system, a touch sensitive input device having
`touch sensitive auxiliary controls can be used to provide a
`user with information regarding the functionality of an
`auxiliary control. When a user's hand approaches a touch
`sensitive input device, feedback can be displayed on a
`display screen. A user can receive feedback without activat(cid:173)
`ing the input device by touching or being extremely proxi(cid:173)
`mate to an auxiliary control. The feedback may take the form
`of a tool tip related to the auxiliary control or a combination
`of auxiliary controls simultaneously touched. When the
`hand of a user is moved away from the touch sensitive input
`device, the feedback brought on by sensing the user's hand
`may disappear.
`
`OPTICAL
`DISK
`
`COMPUTER
`
`1-------1--1 MONITOR
`
`47
`
`20
`
`43
`
`INPUT
`DEVICE
`
`52
`
`MICROSOFT EXHIBIT 1005
`
`

`

`'"""'
`>
`Ul
`~
`'"""' ~
`'"""'
`
`'"""' -c c
`N c c
`
`'JJ.
`Cj
`
`-..J
`N
`
`WAN
`
`MODEM
`
`52
`
`(54
`
`~
`
`INTERFACE
`SERIAL PORT
`
`FIG._1
`
`TRANSDUCER
`
`SENSOR
`
`DEVICE
`
`INPUT I TOUCH l MOVEMENT
`
`/'42
`
`l
`40'""
`
`(46
`
`PROGRAM vsa
`MODULE
`PROGRAM vs7
`
`DATA
`
`PROGRAMS
`36\ APPLICATION
`
`SYSTEM
`
`35\ OPERATING
`
`j 60
`
`~22
`
`DRIVER
`DEVICE
`
`RAM
`
`BIOS
`ROM
`
`26\..
`24\
`
`MEMORY (/5
`32
`
`43\
`
`SPEAKER
`
`(45
`
`20 "'--
`
`'"""' 0 ......,
`~ .....
`'JJ. =-~
`'"""'
`N c c
`> = ~
`=
`.... 0
`~ .....
`""C = 0' -....
`.... 0 =
`'t:l -....
`~
`~ =
`.....
`~ .....
`""C
`
`I")
`
`~ .....
`
`I")
`
`~~
`
`(51
`
`l MEMORY
`
`COMPUTER
`
`REMOTE
`
`(41
`
`LAN
`
`51-
`
`{23
`
`ADAPTER
`NETWORK
`
`(53
`
`47
`
`l!
`
`MONITOR
`
`~
`
`ADAPTER I
`I
`(48
`
`I VIDEO
`
`CPU
`
`(21
`
`INTF l:
`INTF ~
`
`277
`
`DISK DRIVE
`
`HARD
`
`DISK DRIVE
`MAGNETIC
`
`,-s3
`
`(28
`
`INTF ~
`
`DISK DRIVE
`
`OPTICAL
`
`r-34
`
`(30
`
`/ COMPUTER
`
`REMOVABLE
`
`STORAGE
`
`ID I
`
`c
`r29
`
`Cl
`
`DISK (31
`
`OPTICAL
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 2 of 27
`
`US 2001!0011995 Al
`
`108
`
`110
`
`112
`
`114
`
`AID
`MULTIPLEXER
`
`MICRO(cid:173)
`CONTROLLER
`
`124
`
`122
`
`FIG._2
`
`150~
`
`FIG._3
`
`152
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 3 of 27
`
`US 2001!0011995 Al
`
`r11a
`
`FIG._4A
`
`FIG._48
`
`192
`
`..--r 170
`
`202
`
`FIG._4C
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 4 of 27
`
`US 2001!0011995 Al
`
`602
`
`600
`
`606
`
`604
`
`FIG._5
`
`FIG._6A
`
`FIG._6B
`
`603
`
`605
`
`607
`
`FIG._6C
`
`FIG._6D
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 5 of 27
`
`US 2001/0011995 Al
`
`610
`
`610
`
`608
`
`FIG._7A
`
`FIG._7B
`
`614
`
`614
`
`612
`
`608
`
`616
`
`F/G .. _BA
`
`FJG._BB
`
`694
`
`694
`
`FIG._9A
`
`F/G._9B
`
`FIG._9C
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 6 of 27
`
`US 2001!0011995 Al
`
`FIG._10A
`
`FJG._10B
`
`FIG._10C
`
`FIG._10D
`
`FIG._10E
`
`FIG._10F
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 7 of 27
`
`US 2001/0011995 Al
`
`FIG._10G
`
`FIG._10H
`
`FIG._11A
`
`722
`
`710
`
`720
`
`724
`
`726
`
`728
`
`FIG._11B
`
`FIG._12A
`
`FJG._12B
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 8 of 27
`
`US 2001/0011995 Al
`
`734
`
`736
`
`738
`
`740
`
`742
`
`746
`
`744
`
`748
`
`FIG._13A FIG._13B FIG._13C FIG._13D
`
`760
`
`FIG._14A
`
`FIG._148
`
`FIG._14C
`
`FIG._14D
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 9 of 27
`
`US 2001!0011995 Al
`
`FIG._15
`
`FIG .. _16
`
`FIG._17
`
`~260
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 10 of 27
`
`US 2001/0011995 A1
`
`301
`
`309
`
`306
`
`FIG._ f/5 A
`
`810
`
`MESSAGE
`HOOK
`PROCEDURES
`
`20
`
`FIG._ Zo
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 11 of 27
`
`US 2001/0011995 Al
`
`,.....
`0
`(")
`
`('\J
`0
`(")
`
`co
`CX)
`~ .
`<.9
`LL
`
`(!)
`N
`,.....
`(")
`
`LL
`N
`'<'"""
`(")
`
`w
`N ,.....
`
`(")
`
`0
`N ,.....
`(")
`
`u
`N ,.....
`(")
`
`co
`N ,.....
`
`(")
`
`<(
`N ,.....
`
`(")
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 12 of 27
`
`US 2001/0011995 A1
`
`(.)
`co
`~ .
`(9
`LL
`
`...-
`0
`(")
`
`(9
`N
`"<'"""
`(")
`
`LL
`N
`"<'"""
`(")
`
`LlJ
`N
`"<'"""
`(")
`
`0
`N
`"<'"""
`(")
`
`u
`N
`"<'"""
`(")
`
`co
`N
`"<'"""
`(")
`
`<(
`N ...-
`
`(")
`
`N
`0
`(")
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 13 of 27
`
`US 2001!0011995 A1
`
`f/6. 19
`
`2000
`TOUCH PAD
`
`

`

`'"""'
`>
`Ul
`'0
`'"""'
`'0
`'"""'
`
`'"""' -c c
`N c c
`
`'JJ.
`Cj
`
`-..J
`N
`0 ......,
`'"""' ~
`~ .....
`'JJ. =(cid:173)~
`'"""'
`N 8
`::0
`~
`~
`.... 0 =
`~ -a -....
`.... 0 =
`~ 't:l -....
`~ = .....
`~ .....
`""C
`
`~ .....
`
`I")
`
`~ .....
`
`I")
`
`[0 __ 1:43 PM!
`
`[I !lniffl Start ij I W Microsoft Word • Doc... Ill [l Find Files containing text..~]
`
`FIG._2/
`
`II REC I :,-0
`I[~
`
`Ln 2 Col22
`
`1/1 II At 1.2"
`I
`
`Sec 1·
`l@lt Elj\l = Jll .. Jl
`
`I Page {
`
`
`
`----
`
`~
`
`Tha Internet
`
`Q
`®
`
`Recycle Bin
`
`Neighborhood
`
`Network
`
`320
`
`326
`
`328
`
`'.
`
`__ ......,.
`that·tli ·shareholders·will·be· AutoShapes...,..
`I'I;he·~·corporate·earning Draw..,.. ~ <e>
`
`~30
`
`334
`
`J
`=~~g[j)
`t:J...,..J..,..A.
`~ \~0 0~ ~
`
`lXI
`
`I Drawing
`
`--332
`
`Inserted
`Bill G, 7/15/98 4:41 PM:
`
`D
`I If ....
`
`I 3 I
`
`I
`
`I If 21 I If I
`
`I
`
`I
`
`I 111 I
`
`I
`
`I
`
`....
`L ~~I I
`
`-
`----
`-
`-
`-
`
`,. ~ ,. I ~I CD
`I u I ~ ~ ))
`f:!elp !_Jel[g]
`l_lol~
`
`))
`
`-
`
`Iools FQrmat T2_ble Window
`
`'j)ew
`
`Jidll
`
`ll D ~ LEl 16 cs v I ~ ~ ~ ~ ! n
`It I Normal · !Til Times New Roman
`, ... ,,14 , ... , I B
`II [j Eire
`!nsert
`IW Microsoft Word· Doc~;;e~t4~-···--
`
`

`

`'"""'
`>
`Ul
`~
`'"""' ~
`'"""'
`'"""' c c
`N c c
`
`'JJ.
`Cj
`
`-..J
`N
`0 ......,
`'"""' Ul
`~
`
`N
`~~
`
`'JJ. =(cid:173)~
`'"""'
`8
`> = ~
`~ .... 0 =
`""C g. -....
`~ .... 0 =
`'t:l -....
`>
`~ = .....
`~
`""C
`
`I")
`
`I")
`
`'t:l
`
`10 4:44PMj
`
`Find Files containing text ...
`
`II ·~'f:B St~rt] I W Microsoft Word " Doc .. ._ I
`
`FIG._ Z 2.
`
`328
`
`II ~E~ l.-0
`I[~
`
`Ln 2 Col22
`
`I
`
`1/1 :II At 1.2"
`I
`
`I Page 1
`Sec 1'
`-~[El]~[~]
`
`;
`0
`t
`~
`..,
`
`~
`
`=
`
`that·the·shareholders ·will· be·pleased.<J(
`l The·yearly·corporate·earnings·report·will·com
`
`
`
`---
`-
`-
`--
`-
`
`»
`~indow J:!elp LIOJI~
`
`Iools FQrmat T!!_ble
`
`~ -~--~~···-~~ ~-~--1:.101~1
`
`jjjNormal
`II rJ file
`!W Microsoft Word-Do~u;,ent4~--
`
`j.-jj Times New Roman
`!nsert
`.§:dit
`
`Yfew
`
`II D ~ s I d [). ~ I M, ~ ~ ~ I ~ " ~ " I ~I GJ
`, .. ! [EJ!I I B 1 u I ~ i:
`
`0
`I ...
`
`))
`
`I
`
`I 31 I
`
`I
`
`I
`
`I
`
`I
`
`I 21 I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I 11 I
`
`I
`
`I
`
`....
`L ~1,1 l
`
`._
`
`---- ~
` --
`
`The Internet
`
`Q
`
`®
`
`Recycle Bin
`
`Neighborhood
`
`Network
`
`320
`
`326
`
`

`

`'"""
`>
`Ul
`~
`'""" ~
`'"""
`'""" c c
`N c c
`
`'JJ.
`Cj
`
`-..J
`N
`0 ......,
`'""" 0'1
`~
`
`N
`~~
`
`'JJ. =(cid:173)~
`'"""
`8
`> = ~
`~ .... 0 =
`"'C g. -....
`~ .... 0 =
`'t:l -....
`>
`~ = .....
`~
`"'C
`
`I")
`
`I")
`
`'t:l
`
`ro 4:47P~n
`
`II ~JB Startfll W Microsoft Word-Doc ... I !fll Find Flies containing text...~
`
`FIG._23'
`
`352
`
`356
`
`II>-
`
`;
`
`Qptions ...
`
`Qustomize ...
`
`Templates and Add-!ns ...
`
`~
`
`,Macro
`
`Letter Wi~ard ...
`
`Sec 1'
`
`Page 1
`
`fuat·fue·sharehO}d 1-d'l ,Envelopes and Labels ...
`The·yearly·corpo1
`
`Mail Merge ...
`
`rOI
`
`~
`
`I
`
`IliA.
`
`I
`
`I
`
`II
`
`»
`
`~
`
`E,rotect Document...
`
`'
`
`Merge Qocuments ...
`I rack Changes
`
`h,utoCorrect ...
`
`~ A!,!lOSummarize ...
`
`!:Yord Count ...
`
`!:_anguage
`
`I 1 I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`II o (;~) ~ I 6 r1 v I
`Ill Normal .
`II [j file
`IW Microsoft Word· Document4
`
`The Internet
`
`Q
`
`Recycle Bin
`
`®
`
`l
`
`350
`
`ib
`a My Computer
`
`Network
`
`Neighborhood
`
`358
`
`DD1811
`
`~indow U-•-~
`
`IT II Times New Ramal v §palling and Grammar... F7
`
`!nsert I roofs I F£rmat T!!ble
`
`~iew
`
`£dit
`
`

`

`'"""'
`>
`Ul
`~
`'"""' ~
`'"""'
`'"""' c c
`N c c
`
`'JJ.
`Cj
`
`I 0 4:48PM]
`
`II r~ Start··~ I W Microsoft Word~ Doc... I [ll. Find Files containing text..]
`
`FIG._.2lf
`
`'JJ. =(cid:173)~
`'"""'
`8
`> = ~
`~ .... 0 =
`""C g. -....
`~ .... 0 =
`'t:l -....
`>
`~ = .....
`~
`""C
`
`-..J
`N
`0 ......,
`'"""'
`-..J
`~
`
`N
`~~
`
`I")
`
`I")
`
`'t:l
`
`352
`
`tH ~
`
`I
`
`»
`
`))
`
`GJ
`=
`
`~ BacjgJround ...
`
`§tyle ....
`
`Style §aflery .. .
`~AutoFormat .. .
`
`Change Cas!!_ ...
`!~t1 Te~t Direction ...
`
`Qrop Cap ...
`
`Iabs ...
`
`~~folumns ...
`
`!!orders and Shading .. .
`
`i = Bullets and ,[iumbering .. .
`A font ...
`
`.t!elp
`
`358
`
`Ioofs I FQrmat I T~ble Window
`
`that·the·shareholders·
`The·yearly·corporate·
`
`I J I
`
`I
`
`I 1 I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`II [j ~ ~ 1 d !). \1' \ ~ ~ =~1 f.aragraph ...
`!I! Normal
`
`ITI Times New Roman
`_!;dit
`Insert
`
`'f_iew
`
`350
`
`The Internet
`
`Q
`®
`
`Recycle Bin
`
`~
`a My Computer
`
`Neighborhood
`
`Network
`
`

`

`""""
`>
`Ul
`~
`"""" ~
`""""
`
`"""" -c c
`N c c
`
`'JJ.
`Cj
`
`-..J
`N
`0 ......,
`"""" 00
`~ .....
`'JJ. =(cid:173)~
`""""
`8
`
`N
`~~
`~
`~
`.... 0 =
`~ -a -....
`~ ..... .... 0 =
`~ 't:l -....
`~ = .....
`~ .....
`""C
`
`~ .....
`
`I")
`
`I")
`
`r<:r4:49FiMI
`
`iOO!ll Start J! I W Microsoft Word-Doc... IIIJl FindFil~s contai~inQte~T...jj
`
`II
`
`FIG._ 2 5-
`
`1··--·--~--·---~···-·----·-------------_,
`1/1
`
`Page 1
`
`Sec 1
`
`,...--352
`
`1~11
`
`I
`
`~
`
`I ~ Bactground ...
`
`I Style §allery ...
`
`§lyle,.
`
`I II -I
`
`I I!: I
`
`I I ---360
`
`I
`
`I
`
`Change cas~ ...
`' UA Te~ Direction.,.
`
`:::} Q.rop Cap ...
`
`366
`
`that·the·shareholders·
`-I TPe·~earl:;t·corporate
`~
`
`I
`
`II
`
`I
`
`354
`
`350 ...._ I
`
`The Internet
`
`Q
`
`Recycle Bin
`
`@
`
`1-lul I ---362
`
`---"-'·-~~-
`
`':
`
`1-1
`
`M
`
`Neighborhood
`
`Network
`
`t=(8•••11111111(1
`
`L.
`
`II D ~ ~ I 0 [). V I ~ ~ =~r f.aragraph ...
`
`/•/
`
`!Ill/Normal
`
`My Computer
`
`))
`
`IGJ
`
`§orders and Shading ...
`
`i = Bullets and Jiumbering •..
`
`_
`
`IIFillvlll _,.---358
`
`1"\ __ , ___ .. ,
`
`ib
`9 lhu .. ~-----u .. , __ _,
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 19 of 27
`
`US 2001!0011995 A1
`
`~370
`
`371
`
`FIG._ 2-h
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 20 of 27
`
`US 2001/0011995 Al
`
`co
`N
`CJ -LL
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 21 of 27
`
`US 2001/0011995 A1
`
`0)
`
`co
`N .
`(9
`LL
`
`<(
`0)
`N .
`<.9
`LL
`
`<(
`0
`M .
`<.9
`LL
`
`co
`0
`M .
`<.9
`LL
`
`.
`(.9
`LL
`
`

`

`""""
`>
`Ul
`'0
`""""
`'0
`""""
`"""" c c
`N c c
`
`'JJ.
`Cj
`
`-..J
`N
`0 ......,
`N
`N
`~
`
`'JJ. =(cid:173)~
`""""
`8
`N
`::0
`> = ~
`~ .... 0 =
`""C g. -....
`~ .... 0 =
`'t:l -....
`>
`~ = .....
`~
`""C
`
`I")
`
`I")
`
`'t:l
`
`FIG._J"f! ..
`
`~--T~~ Microsoft ... I I 0 4:56 PM]
`
`[w Microsoft w?rd~;.~mo Flnd: Files contairL .. ij
`
`4
`
`Shortcut to default.htm'
`
`462
`
`I There are two simple ways
`Products
`Microsoft
`
`Games
`Development Tools
`& Serv~r
`Operating Systems
`Business Software
`
`... Product Type
`
`The Internet
`
`Q
`
`~
`
`Recycle Bin
`
`460
`
`ib
`~ My Computer
`
`Neighborhood
`
`Network
`
`

`

`'"""'
`>
`Ul
`~
`'"""' ~
`'"""'
`'"""' c c
`N c c
`
`'JJ.
`Cj
`
`[0 .. 4:~ .FJMJ
`
`r@L Micro~o!f ... l
`
`!!!Iii Start -~ tlw-Microsoft w ... Jl [!l Find: Files contain~ .. ~
`
`FIG._ 33'
`
`-..J
`N
`0 ......,
`~
`N
`~
`
`N
`~~
`
`'JJ. =(cid:173)~
`'"""'
`8
`> = ~
`~ .... 0 =
`& -....
`~ .... 0 =
`'t:l -....
`>
`~ = ......
`~
`""C
`
`I")
`
`""C
`
`I")
`
`'t:l
`
`472
`
`CarPoint for a look ahead at the new 1999
`What's in the cars for the coming year? Visit
`Car Point
`Get a Sneak Peek at 1999's Models in
`
`• Top Story
`
`470
`
`L-.J)If
`
`Where do you want to go today?
`
`Microsoft
`
`:
`
`Is Here!
`ii§il!Il 98
`w i n d Q w S
`
`Free Newsletters
`
`You
`
`• Recommended For
`
`~Contents
`
`H~
`
`Read All About It!
`
`A MICROSOFT ii.IJJJhlJ£1 DSJ.UJ I Z ZJII.I.I.U2 LSIIJ.CCI.II.Ib§§ ii.r
`tl Address: I http://www.microsoft.com/
`IYI!I Unks
`II
`.dJ ~
`
`Fa\forltes _Print l.S2.J
`[!]....
`
`Search
`Refresh
`[Ej ~ Q
`
`Home
`
`.tfelp
`
`F!jvorites
`
`Stop
`0
`go
`
`Back Forward
`<?
`
`S>
`Y..iew
`
`.!;dit
`
`file
`
`Microsoft Corooratlon Home Page; Welcom to M ... -Microsoft ln •••
`
`468
`
`The Internet
`
`G)
`®
`
`Recycle Bln
`
`~
`9 My Computer
`
`Network
`
`Neighborhood
`
`

`

`""""
`>
`Ul
`'0
`""""
`'0
`""""
`"""" c c
`N c c
`
`'JJ.
`Cj
`
`1@1_ Tll_e Microsoft ... II 0 4:58PM]
`
`I! il!lfl!Start] fit Microsoft Word· ... 11 ~ [J Find: Files contain ... ~
`
`FIG._ 34-
`
`-..J
`N
`0 ......,
`~
`N
`~
`
`'JJ. =(cid:173)~
`""""
`8
`N
`::0
`> = ~
`~ .... 0 =
`""C g. -....
`~ .... 0 =
`'t:l -....
`>
`~ = .....
`~
`""C
`
`I")
`
`I")
`
`'t:l
`
`478
`
`476
`
`HomeP
`Microsoft
`
`Kids
`Games
`Development Tools
`& Server ...
`Operating Systems
`Business Software
`
`, Product Type
`
`I
`
`480
`
`The Internet
`
`Q
`
`Recycle Bin
`
`®
`
`ib
`a My Computer
`
`Network
`
`Neighborhood
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 25 of 27
`
`US 2001/0011995 Al
`
`<(
`L()
`("()
`<.9
`LL
`
`ca
`L()
`("')
`<.9
`lL
`
`()
`LO
`(V)
`C.9
`LL
`
`t~
`
`t~
`
`<!.)
`
`"' "' " <i
`~ r -
`
`' - - -
`
`~t
`
`<l t
`
`+
`r -
`
`" ;q
`" ~
`
`,.----
`
`t ___
`
`r -
`
`r - -
`
`L----
`
`k-~t
`<l t
`
`+
`, - -
`
`-
`
`0
`
`- 0
`
`6-5
`±:j
`::J
`::I 0
`c:ll-
`
`0..
`i=..!!d
`-
`.0
`0 ·-
`~5
`
`- 0
`
`al.~
`::I >
`0 0
`2:2:
`
`-
`
`0
`
`- 0
`
`C.s;::
`0 u
`±:j::J
`::I 0
`c:ll-
`
`0..
`i=..9d
`- .0 g :!Zi
`1->
`
`- 0
`
`Q)Ol
`~ -~
`0 0
`2:2:
`
`-
`
`0
`
`-
`
`0
`
`C.s;::
`0 u
`:t:l::J
`::I 0
`c:ll-
`
`0..
`i=-.9:!
`- .0
`0 ·-
`~5
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 26 of 27
`
`US 2001/0011995 Al
`
`(9
`LL
`
`. ..-- ""
`
`...
`
`· r - - .
`
`" 0
`
`"" ;:;·t"' '---
`................................ ! ; .. : ...
`
`" ' r - - '
`
`.... L - " "" r - - "
`
`" 0
`::: "
`
`;; 0
`
`"
`
`~r ~- ----~r .... -.........
`
`.... ! . - - . . . . . . .
`
`L . - . . . . . . .
`
`0
`
`C.s:;:
`0 u
`......
`......
`::J
`::J 0
`!Xlf-
`
`0
`
`c
`
`c
`0
`.....,..>::
`..... u
`::J=
`!XlU
`
`Q_
`
`f- ~
`-..a
`0 ·-
`0 -~
`I->
`
`C.s:;:
`.....
`0 u
`.....
`::J
`::J 0
`!Xl/-
`
`c
`.8..l.::
`..... u
`::J:.=
`!XlU
`
`Cl.
`f- ~
`-..a
`0 ·-
`0.~
`f->
`
`

`

`Patent Application Publication Aug. 9, 2001 Sheet 27 of 27
`
`US 2001/0011995 A1
`
`t;.t'~#,rA-t. H./EI!.-
`,Nrr~;:-A-1!&
`q ID
`r/J~t: ,111;
`p,qVE ¢7
`.fNI}fl/fi~J
`
`V"qt5
`7 ;b f".U4
`f+L- y,P-/'
`~e"fl'?d4-S
`
`9~a 1
`( -np/ ~ 14a } qz. 2.
`f "1"681 r'f .r/(~) jfn1
`
`1---913
`
`[ 07/IP!. PI.Sf'.tAtt q]_.b
`WI ilt!{Er.J'
`.
`Br.77'1Jtj? E;E..<IE/ZR,_.t:.
`
`1- 30
`
`i \
`
`9"D3
`
`A-?/LI?R://0-t/
`
`} 1:?o
`
`ojs
`
`3b
`
`{jtZtfr?t/ICf Af:Z.
`
`IJ
`
`J..- 155
`
`I nPlf7
`I)EI/!C£ ~ ::r/F
`qDS
`\ too
`
`i)£11/t:I'Zf
`
`J-- 'f{:,
`
`1 9L}-D
`
`Cj{) '1
`
`l
`
`Vt1J£o
`
`!FlJrtP T£R.
`
`1
`l olf .P~ .-?Y
`
`('MCA~IrP~)
`
`---Lf8
`
`Jf-7
`
`/ L/-3
`/,.!/{):/
`P..CIIIc'l£ I
`
`rove;.!-
`SE..Vf11~
`
`~
`
`!fo
`
`-
`
`/ 'f-3
`Nl'vr
`j)E,,c.E .2.
`
`r~u--::1'/-
`.r E "-if iJ /Z.
`
`I-
`
`\
`4-o
`
`f;6. 37
`
`

`

`US 2001/0011995 A1
`
`Aug. 9, 2001
`
`1
`
`METHOD FOR PROVIDING FEEDBACK
`RESPONSIVE TO SENSING A PHYSICAL
`PRESENCE PROXIMATE TO A CONTROL OF AN
`ELECTRONIC DEVICE
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`[0001] The instant application is a continuation-in part of
`commonly assigned, copending U.S. patent application
`entitled "Proximity Sensor in a Computer Input Device" to
`Hinckley et al., filed Sep. 14, 1998 and assigned Ser. No.
`09/152,434, which is herein incorporated by reference. Also,
`the instant application is a continuation-in part of commonly
`assigned, copending U.S. patent application entitled "A
`Technique For Implementing a Two-Handed Desktop Use
`Interface For a Computer" to Hinckley, filed Sep. 14, 1998
`and assigned Ser. No. 09/152,432, which is herein incorpo(cid:173)
`rated by reference. Further, the instant application is a
`continuation-in part of commonly assigned, copending U.S.
`patent application entitled "Method of Interacting With a
`Computer Using a Proximity Sensor in a Computer Input
`Device" to Hinckley et al., filed Sep. 14, 1998 and assigned
`Ser. No. 09/152,443, which is herein incorporated by refer(cid:173)
`ence.
`
`In addition, the instant application is a continua(cid:173)
`[0002]
`tion-in part of commonly assigned, copending U.S. patent
`application entitled "A Technique For Implementing an
`On-Demand Display Widget Through Controlled Fading
`Initiated By User Contact With a Touch Sensitive Input
`Device" to Hinckley, filed Nov. 25, 1998 and assigned Ser.
`No. 09/200,325, which is herein incorporated by reference
`and which claims priority to U.S. provisional patent appli(cid:173)
`cation entitled "Toolglass Improvements-On-Demand
`Tool Sheet; Gesturing Through Tool Sheets", filed Sep. 14,
`1998 and assigned Ser. No. 60/100,261. The instant appli(cid:173)
`cation is also a continuation-in part of commonly assigned,
`copending U.S. patent application entitled "A Technique For
`Implementing an On-Demand Tool Glass For Use in a
`Desktop User Interface" to Hinckley, filed Nov. 25, 1998
`and assigned Ser. No. 09/200,321, which is herein incorpo(cid:173)
`rated by reference and which claims priority to United States
`provisional patent application entitled "Toolglass Improve(cid:173)
`ments-On-Demand Tool Sheet; Gesturing Through Tool
`Sheets", filed Sep. 14, 1998 and assigned Ser. No. 60/100,
`261.
`
`[0003] The present application is related to commonly
`assigned, copending U.S. patent application entitled
`"Method of Interacting With a Computer Using a Proximity
`Sensor in a Computer Input Device" to Hinckley et al., filed
`Apr. 5, 2000, and assigned Ser. No. 09/543,723, which is
`herein incorporated by reference and which is a continuation
`of U.S. patent application Ser. No. 09/152,443 identified
`above.
`
`BACKGROUND OF THE INVENTION
`
`[0004] A Technical Field
`
`[0005] The present invention relates to input devices for
`computer systems. More particularly, the present invention
`relates to the use of touch sensitive input devices in com(cid:173)
`puter systems.
`
`[0006] B. Related Art
`
`Input devices for computerized systems such as
`[0007]
`keyboards, touchpads, trackballs, game controllers and
`mice, often include one or more auxiliary controls such as
`joysticks, touchpads, trackballs, headsets, microphones, but(cid:173)
`tons, knobs, rocker switches, triggers, sliders, wheels, bio(cid:173)
`metric sensors, etc. Auxiliary controls may be used alone or
`in conjunction with input devices and other auxiliary con(cid:173)
`trols.
`
`[0008] Most input devices create input signals using trans(cid:173)
`ducers, or switches. Switches are typically found in the
`buttons of mice, joysticks, and game controllers, as well as
`in the keys of keyboards. Transducers are found in mice and
`trackballs and create electrical signals based on the move(cid:173)
`ment of balls in those devices or by optically detecting
`movement of the device across a surface. Transducers are
`also found in headsets where they convert speech signals
`into electrical signals. Touchpads create input signals using
`sensors that provide an electrical signal when the user
`contacts the touchpad that signal including the location
`within the touchpad where contact is made.
`
`[0009] Although it is desirable to increase the amount of
`information that an input device and its auxiliary controls
`can provide to the computer, the number of transducers and
`switches that can be added to an input device is limited by
`the user's ability to remember all of the functions that a
`particular transducer or switch performs and by the practi(cid:173)
`calities of the available or at least feasibly available real
`estate for the switches and transducers on the input device
`(e.g., keyboard or mouse). The ability of a user to determine
`the functionality of each auxiliary control is also generally
`limited to a static label on or near the auxiliary control,
`activating each auxiliary control, clicking a button to select
`a menu option to request help file or reading a user manual.
`Furthermore, the functionality of an auxiliary control may
`vary from one game or application to another game or
`application. Thus, a static label on an auxiliary control is of
`little value when the function and status of the input device
`and its auxiliary controls change from application to appli(cid:173)
`cation. In this regard, even if a user learns the assignment of
`the buttons and other controls in one game or application,
`another game or application may assign the same buttons or
`controls different features. Similarly, the status of the feature
`controlled by the auxiliary control generally can only be
`determined by activating the control.
`
`[0010] Thus, there is a need to provide a user with the
`ability to better determine the functionality and status of
`auxiliary controls regardless of their context, (e.g., active
`application or game).
`
`[0011] To address this need, indicators such as LEDs or
`small LCDs may be integrated directly with the input device
`and its auxiliary controls to provide feedback or state
`information. However, association of LEDs or LCDs with
`each auxiliary control increases both the amount of power
`consumed and the cost of the input device. Furthermore,
`LEDs and LCDs have very limited output forms and are
`separated from the main display (e.g., a computer monitor)
`causing the user to constantly have to look away from the
`screen and at the auxiliary control to determine the func(cid:173)
`tionality or status of the control. Such constant shifts of
`attention can mentally tire the user as he or she is forced to
`repetitively reacquire the current context. Accordingly, there
`
`

`

`US 2001/0011995 Al
`
`Aug. 9, 2001
`
`2
`
`is a need to provide a more efficient, convenient and/or cost
`effective way to determine the functionality and/or status of
`auxiliary controls of input devices in a computer system.
`
`SUMMARY OF THE INVENTION
`
`[0012] The present invention overcomes many of the
`shortcomings of existing input devices by providing touch
`sensitive input devices which have touch sensitive auxiliary
`controls that sense a physical presence and provide visual
`feedback on an on-screen display or on the control itself,
`acoustic feedback (e.g., voice or music) or tactile feedback
`(e.g., vibration).
`
`In an aspect of the present invention, keys, buttons,
`[0013]
`knobs, rocker switches or other auxiliary controls of an input
`device such as a keyboard, touchpad, trackball, game con(cid:173)
`troller, monitor, joystick, steering wheel, headset or mouse
`can be augmented with sensors that detect contact or
`extreme proximity of a user's hand. According to another
`aspect, a sensed signal of the auxiliary control can provide
`the user with an on-screen display of status, state informa(cid:173)
`tion, tool tips, help text or other feedback relevant to the
`control the user has touched without the user having acti(cid:173)
`vated (e.g., depress, turn, roll or otherwise activate) the
`control. Alternatively, acoustic or tactile feedback may be
`provided to the user instead of or in addition to the feedback
`provided by the on-screen display. Hence, a user can simply
`touch different buttons or other controls to explore their
`function assignment or status, or obtain other feedback.
`Such feedback can allow the user to better understand the
`consequences of their action, should he or she subsequently
`chooses to activate the control. Also, a user may quickly and
`casually be able to view status information.
`
`[0014] Aspects of the invention include providing context
`sensitive feedback for an auxiliary control in an on-screen
`display, such as a display widget (e.g., graphical user inter(cid:173)
`face (GUI)), responsive to detection of a user's hand. Hence,
`touch sensitive controls can be used to predict the context of
`a user's action. The context of the situation can be estab(cid:173)
`lished by preparatory actions by the user such as grabbing,
`touching or approaching a control. Knowing the context of
`the action, the computer can begin to execute and predict the
`will of the user.
`
`[0015] According to another aspect of the invention, con(cid:173)
`texts of an auxiliary control may include, but are not limited
`to, different types of applications such as games, utility, and
`productivity applications. Also, contexts may change within
`various portions of an application or game.
`
`In another aspect of the invention, when a GUI is
`[0016]
`displayed responsive to detection of a physical presence
`proximate to or contacting an auxiliary control, a user may
`interact with the GUI using another auxiliary control or an
`input device including, but not limited to, a mouse, touchpad
`or keypad. For example, if a volume control GUI is dis(cid:173)
`played, a user may adjust the volume with a mouse.
`
`In another aspect of the invention, input devices
`[0017]
`including their auxiliary controls may have a touch sensor
`which directly detects when a physical presence (e.g., user)
`touches the device or its controls, so as to provide display
`and dismissal of feedback on an "on-demand" basis, e.g.,
`whenever the user establishes or breaks physical hand
`contact with the device.
`
`[0018] For example, a transition in a touch indication
`provided by the device reflective of the user then making
`contact with the device or control, such as by touching the
`device with a finger may cause a tool tip to be displayed. A
`transition indicative of a user breaking physical contact with
`the device, such as by lifting his finger off the device, can
`cause the tool tip to be dismissed from the display. In one
`aspect of the invention, to prevent user distraction, these
`detected transitions initiate corresponding predefined ani(cid:173)
`mation sequences that occur over preset time intervals in
`which the feedback either begins to fade into view (typically
`from an invisible, i.e., totally transparent state, to eventually
`a predefined semi-transparent state or non-transparent state)
`as soon as user contact begins, and then begins to fade out
`from view (i.e., eventually back to its invisible state) as soon
`as user contact with the device is broken, e.g., as soon as the
`user lifts his or her hand away from the device.
`
`In another aspect of the invention, toolbars, scroll(cid:173)
`[0019]
`bars and the like may only be displayed on a display screen
`in response to detection of a physical presence. Illustratively,
`touching, and not activating, an auxiliary control may cause
`on-screen display of a toolbar and breaking contact with the
`control may cause dismissing of the toolbar. In one aspect of
`the invention, a user may interface with the toolbar using an
`auxiliary control or input device as desired with one hand
`while continuing to touch the auxiliary control which caused
`the on-screen display of the toolbar. According to this
`aspect, display clutter can be reduced and available appli(cid:173)
`cation screen area increased at appropriate times during
`program execution consistent with and governed by user
`action but without imposing any significant cognitive bur(cid:173)
`den on the user to do so. This, in turn, is likely to signifi(cid:173)
`cantly improve the "user experience".
`
`[0020] These and other novel advantages, details, embodi(cid:173)
`ments, features and objects of the present invention will be
`apparent to those skilled in the art from following the
`detailed description of the invention, the attached claims and
`accompanying drawings, listed herein, which are useful in
`explaining the invention.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0021] FIG. 1 is a block diagram of a computer system
`according to an illustrative embodiment of the present
`invention.
`
`[0022] FIG. 2 is a more detailed block diagram of an
`exemplary embodiment of an input device according to the
`present invention.
`
`[0023] FIG. 3 is a perspective view of a headset according
`to an illustrative embodiment of the present invention.
`
`[0024] FIG. 4A is a perspective view of a mouse accord(cid:173)
`ing to an illustrative embodiment of the present invention.
`
`[0025] FIG. 4B is a bottom view of the mouse of FIG. 4A.
`
`[0026] FIG. 4C is a perspective view of an exemplary
`circuit board of the mouse of FIG. 4A.
`
`[0027] FIG. 5 is a top view of another illustrative embodi(cid:173)
`ment of a mouse according to the present invention.
`
`[0028] FIG. 6A is a left side view of another illustrative
`embodiment of a mouse according to the present invention.
`
`

`

`US 2001/0011995 Al
`
`Aug. 9, 2001
`
`3
`
`[0029] FIG. 6B is a left side view of another illustrative
`embodiment of a mouse according to the present invention.
`
`[0030] FIG. 6C is a right side view of another illustrative
`embodiment of a mouse according to the present invention.
`
`[0031] FIG. 6D is a right side view of another illustrative
`embodiment of a mouse according to the present invention.
`
`[0032] FIGS. 7A and 7B are a left side view and a top
`view, respectively, of another illustrative embodiment of a
`mouse according to the present invention.
`
`[0033] FIGS. SA and 8B are a left side view and a top
`view, respectively, of another illustrative embodiment of a
`mouse according to the present invention.
`
`[0034] FIGS. 9A-9C are a left side view, a top view, and
`a right side view, respectively, of another illustrative
`embodiment of a mouse according to the present invention.
`
`[0035] FIGS. 10A-10H, are top views of different exem(cid:173)
`plary embodiments for a mouse button according to the
`present invention.
`
`[0036] FIG. llA is a top view of another illustrative
`embodiment of a mouse according to the present invention.
`
`[0037] FIG. llB is a top view of another illustrative
`embodiment of a mouse according to the present invention.
`
`[0038] FIGS. 12A and 12B are right side views of dif(cid:173)
`ferent exemplary embodiments of mice according to the
`present invention.
`
`[0039] FIGS. 13A-13D are left side views of different
`exemplary embodiments of mice according to the present
`invention.
`
`[0040] FIGS. 14A-14D are top views of different exem(cid:173)
`plary embodiments of mice showing a touch sensor proxi(cid:173)
`mate to a wheel according to the present invention.
`
`[0041] FIG. 15 is a perspective view of an illustrative
`trackball according to the present invention.
`
`[0042] FIG. 16 is a perspective view of an exemplary
`game controller having a joystick according to the present
`invention.
`
`[0043] FIG. 17 is a perspective view of an exemplary
`game controller according to the present invention.
`
`[0044] FIG. 18A is a perspective view of an illustrative
`keyboard according to the present invention.
`
`[0045] FIGS. 18B and 18C show portions of illustrative
`keyboards according to the present invention.
`
`[0046] FIG. 19 is a perspective view of an illustrative
`touchpad according to the present invention.
`
`[0047] FIG. 20 is a more detailed block diagram of the
`computer of FIG. 1.
`
`[0048] FIG. 21 is an illustrative screen display that may
`appear before an input device of the present invention is
`touched.
`
`[0049] FIG. 22 is an image of an illustrative screen
`display after an input device of the present invention has
`been touched.
`
`[0050] FIG. 23 is an image of an illustrative screen
`display showing a pull-down menu activated according to
`the present invention.
`[0051] FIG. 24 is an image of an illustrative screen
`display showing a second pull-down menu opened accord(cid:173)
`ing to the present invention.
`[0052] FIG. 25 is an image of an illustrative screen
`display showing an item selected in a pull-down menu
`according to the present invention.
`[0053] FIG. 26 is an image of an illustrative screen
`display showing a radial menu according to the present
`invention.
`[0054] FIG. 27 shows an illustrative graphical user inter(cid:173)
`face that may appear on a display screen according to an
`exemplary embodiment of the present invention.
`[0055] FIG. 28 shows an illustrative on-screen display
`that may appear on a display screen according to an exem(cid:173)
`plary embodiment of the present invention.